基于DDPG的变外形航天飞行器碰撞规避的轨迹规划方法

doi:10.12382/bgxb.2023.1182

摘要/Abstract

摘要：

针对变外形航天飞行器制导与变形决策强耦合问题,提出了基于深度确定性策略梯度(Deep Deterministic Policy Gradient, DDPG)变外形碰撞规避的轨迹规划方法。依托变形参量建立变外形航天飞行器运动学模型,设计具有射程误差校正功能的纵向制导律和基于视线角偏差的横向制导律,实现绕飞障碍物并保证制导精度。建立适用于连续变外形的马尔可夫决策模型,以攻角、马赫数以及飞行器与障碍物的相对距离为状态空间,设计考虑碰撞的势场惩罚函数及满足制导精度的奖励函数,并构建DDPG网络实现状态空间到动作的尺度变换,得到最优外形决策指令。仿真结果表明:与固定外形航天飞行器相比,通过对外形最优决策,提高了航天飞行器制导精度和横向避障能力,降低了对机载雷达感知能力的要求,节省了感知成本。

关键词: 变外形航天飞行器, 深度确定性策略梯度, 智能决策, 轨迹规划, 碰撞规避

Abstract:

To address the problem of the strong coupling between the guidance and morphing decision of morphing spacecraft, a morphing collision avoidance trajectory planning method of considering obstacle constraint based on deep deterministic policy gradient (DDPG) is proposed. A kinematic model of morphing aerospace craft is established according to morphing parameter. A longitudinal guidance law with a range error correction function and a lateral guidance law based on line-of-sight angle deviation are designed to realize the obstacle circumvention and ensure the terminal guidance accuracy. Then a Markov decision model is constructed to facilitate a continuous morphing. The angle of attack, Mach, and relative distance from the spacecraft to the obstacle are taken as the state space. The potential field penalty function considering collision and the smallest terminal guidance error reward function is considered in the design. The DDPG network is then trained to generate a map of decision instruction from the state space and obtain the optimal shape decision instruction. The simulated results show that, compared with configuration-fixed spacecraft, the guidance accuracy and lateral obstacle avoidance ability of morphing spacecraft are improved by optimizing the shape, and the requirement for the detection ability of air borne radar is reduced to save the detection cost.

Key words: morphing spacecraft, deep deterministic policy gradient, intelligent decision-making, trajectory planning, collision avoidance

中图分类号:

V448.235

丁天雲, 夏逸, 梅泽伟, 邵星灵, 刘俊. 基于DDPG的变外形航天飞行器碰撞规避的轨迹规划方法[J]. 兵工学报, 2024, 45(11): 3903-3914.

DING Tianyun, XIA Yi, MEI Zewei, SHAO Xingling, LIU Jun. A DDPG-based Trajectory Planning Method for Collision Avoidance of Morphing Spacecraft[J]. Acta Armamentarii, 2024, 45(11): 3903-3914.

图/表 20

图1 变外形智能决策轨迹规划技术结构框图

Fig.1 Structure block diagram of trajectory planning technology for morphing intelligent decision

图2 障碍物示意图

Fig.2 Schematic diagram of no-fly zone

图3 DDPG算法流程图

Fig.3 Training process of DDPG algorithm

图4 DDPG神经网络结构图

Fig.4 Structure diagram of neural network of DDPG

表1 部分仿真初始化参数设置

Table 1 Initialization parameters setting for simulation

训练参数	数值
Critic网络学习率	0.001
Actor网络学习率	0.0001
折扣因子	0.99
抽样样本大小	128
经验数据集大小	1000000
探索噪声方差	0.1
软更新率	0.01
航向角阈值ψ_des/(°)	5

图5 飞行航迹对比图

Fig.5 Comparison of flight paths

图6 变外形训练奖励曲线

Fig.6 Reward curves for variable configuration training

图7 飞行过程变外形相关参数变化图

Fig.7 Variation of related morphing parameters during flying

图8 不同攻角剖面图

Fig.8 Profiles of different angles of attack

表2 场景1下的仿真结果对比

Table 2 Simulation results in Scenario 1

类别	经纬度偏差/(°)	剩余航程/km
变外形	(0.0115,-0.0158)	2.17
外形1	(0.0468,-0.0672)	9.101
外形2	(0.041,-0.0122)	4.754
外形3	(0.0581,-0.0345)	7.514

图9 不同速度节点单一外形航迹图

Fig.9 Single shape track diagram of different speed nodes

图10 不同速度节点变外形航迹图

Fig.10 Variable shape track diagram of different speed nodes

图11 不同攻角最值单一外形航迹图

Fig.11 Single shape track chart with maximum value of different angles of attack

图12 不同攻角最值变外形航迹图

Fig.12 Variable shape track chart with maximum value of different angles of attack

图13 应急避障能力对比图

Fig.13 Comparison of emergency obstacle avoidance

表3 允许的最小感知半径

Table 3 The minimum allowable detection radius

变外形	外形1	外形2	外形3
83%	87%	100%	93%

图14 两种方法飞行航迹图

Fig.14 Flight path diagram of two methods

图15 两种方法飞行过程相关参数变化图

Fig.15 Variation diagram of related parameters of two methods

图16 两种方法飞行航迹图

Fig.16 Flight path diagram of two methods

图17 两种方法飞行过程相关参数变化图

Fig.17 Variation diagram of related parameters of two methods

参考文献 24

[1]	杜万闪, 周洲, 拜昱, 等. 组合式飞行器多体动力学建模与飞行力学特征[J]. 兵工学报, 2023, 44(8): 2245-2262. doi: 10.12382/bgxb.2022.0282
	DU W S, ZHOU Z, BAI Y, et al. Multi-body dynamics modeling and flight mechanics characteristics of combined aircraft[J]. Acta Armamentarii, 2023, 44(8): 2245-2262. (in Chinese)
[2]	张晚晴, 余文斌, 李静琳, 等. 基于纵程解析解的飞行器智能横程机动再入协同制导[J]. 兵工学报, 2021, 42(7): 1400-1411.
	ZHANG W Q, YU W B, LI J L, et al. Intelligent cross-range maneuvering reentry cooperative guidance for air-craft based on longitudinal analytic solution[J]. Acta Armamentarii, 2021, 42(7):1400-1411. (in Chinese)
[3]	姜斌, 孟庆开, 杨浩. 航天器姿轨控制研究综述:微分几何控制方法[J]. 控制与决策, 2023, 38(8): 2079-2092.
	JIANG B, MENG Q K, YANG H. A survey on spacecraft attitude and orbit control: differential geometric control approaches[J]. Control and Decision, 2023, 38(8): 2079-2092. (in Chinese)
[4]	陈树生, 贾苜梁, 刘衍旭, 等. 变体飞行器变形方式及气动布局设计关键技术研究进展[J]. 航空学报, 2024, 45(5): 629595.
	CHEN S S, JIA M L, LIU Y X, et al. Development status on deformation modes and key technologies of aerodynamic layout design about morphing aircraft[J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(5):629595. (in Chinese)
[5]	吴康盛. 变体飞行器飞行控制与外形决策研究[D]. 长沙: 湖南大学, 2022.
	WU K S. The research on flight control and shape decision-making of morphing aircraft[D]. Changsha: Hunan University, 2022. (in Chinese)
[6]	张尧, 张婉, 别大卫, 等. 智能变体飞行器研究综述与发展趋势分析[J]. 飞航导弹, 2021(6):14-23.
	ZHANG Y, ZHANG W, BIE D W, et al. Research summary and development trend analysis of intelligent variable aircraft[J]. Aerodynamic Missile Journal, 2021(6):14-23. (in Chinese)
[7]	冉茂鹏, 王成才, 刘华华, 等. 变体飞行器控制技术发展现状与展望[J]. 航空学报, 2022, 43(10):432-449.
	RAN M P, WANG C C, LIU H H, et al. Research status and future development of morphing aircraft control technology[J]. Acta Aeronautica et Astronautica Sinica, 2022, 43(10): 432-449. (in Chinese)
[8]	甄子洋, 刘攀, 陆宇平. 变体飞行器智能变形与飞行控制技术研究进展[J]. 南京航空航天大学学报, 2022, 54(6): 995-1006.
	ZHEN Z Y, LIU P, LU Y P. Research progress on intelligent deformation and flight control technology of morphing aircraft[J]. Journal of Nanjing University of Aeronautics & Astronautics, 2022, 54(6): 995-1006. (in Chinese)
[9]	王子健. 变体飞行器的变形辅助机动控制[D]. 哈尔滨: 哈尔滨工业大学, 2022.
	WANG Z J. Morphing-aided maneuver control of morphing aircraft[D]. Harbin: Harbin Institute of Technology, 2022. (in Chinese)
[10]	LITTMAN M L. Reinforcement learning improves behaviour from evaluative feedback[J]. Nature, 2015, 521:445-451.
[11]	王青, 刘华华. 变体飞行器智能自主决策与控制[J]. 现代防御技术, 2020, 48(6): 5-11. doi: 10.3969/j.issn.1009-086x.2020.06.002
	WANG Q, LIU H H. Intelligent autonomous decision-making and control of morphing aircraft[J]. Modern Defence Technology, 2020, 48(6):5-11. (in Chinese)
[12]	徐小野, 李爱军, 张丛丛, 等. 基于Q学习的变体无人机控制系统设计[J]. 西北工业大学学报, 2012, 30(3):340-344.
	XU X Y, LI A J, ZHANG C C, et al. Applying Q-learning to designing feasible morphing UAV control system[J]. Journal of Northwestern Polytechnical University, 2012, 30(3): 340-344. (in Chinese)
[13]	闫斌斌, 李勇, 戴沛, 等. 基于增强学习的变体飞行器自适应变体策略与飞行控制方法研究[J]. 西北工业大学学报, 2019, 37(4): 656-663.
	YAN B B, LI Y, DAI P, et al. Adaptive wing morphing strategy and flight control method of a morphing aircraft based on reinforcement learning[J]. Journal of Northwestern Polytechnical University, 2019, 37(4):656-663. (in Chinese)
[14]	LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning[J]. Computing Research Repository, 2015, abs/1509.02971.
[15]	温暖, 刘正华, 祝令谱, 等. 深度强化学习在变体飞行器自主外形优化中的应用[J]. 宇航学报, 2017, 38(11):1153-1159.
	WEN N, LIU Z H, ZHU L P, et al. Deep reinforcement learning and its application on autonomous shape optimization for morphing aircraft[J]. Journal of Astronautics, 2017, 38(11): 1153-1159. (in Chinese)
[16]	XU D, HUI Z, LIU Y Q, et al. Morphing control of a new bionic morphing UAV with deep reinforcement learning[J]. Aerospace Science and Technology, 2019, 92: 232-243.
[17]	桑晨, 郭杰, 唐胜景, 等. 基于DDPG算法的变体飞行器自主变形决策[J]. 北京航空航天大学学报, 2022, 48(5):910-919.
	SANG C, GUO J, TANG S J, et al. Autonomous deformation decision making of morphing aircraft based on DDPG algorithm[J]. Journal of Beijing University of Aeronautics and Astronautics, 2022, 48(5): 910-919. (in Chinese)
[18]	张远, 黄万伟, 聂莹, 等. 一种高速可变形飞行器智能变形决策方法[J]. 宇航学报, 2022, 43(12):1665-1675.
	ZHANG Y, HUANG W W, NIE Y, et al. An intelligent deformation decision-making method for high-speed morphing flight vehicle[J]. Journal of Astronautics, 2022, 43(12): 1665-1675. (in Chinese)
[19]	吴则良, 叶建川, 王江, 等. 基于深度自动编码器神经网络的飞行器翼型参数降维与优化设计[J]. 兵工学报, 2022, 43(6):1326-1336. doi: 10.12382/bgxb.2021.0346
	WU Z L, YE J C, WANG J, et al. Parameter dimensionality reduction and optimal design of aircraft airfoil based on deep autoencoder neural network[J]. Acta Armamentarii, 2022, 43(6): 1326-1336. (in Chinese) doi: 10.12382/bgxb.2021.0346
[20]	田若岑, 张庆振, 郭云鹤, 等. 基于禁飞区规避的高超声速飞行器再入制导律设计[J]. 空天防御, 2022, 5(2): 65-74.
	TIAN R C, ZHANG Q Z, GUO Y H, et al. Design of hypersonic vehicle reentry guidance law based on no-fly zone avoidance[J]. Air-space Defense Force Structure, 2022, 5(2): 65-74. (in Chinese)
[21]	周亮, 王昊宇, 尚海滨, 等. 基于高斯伪谱法的天基再入飞行器滑翔轨迹优化设计研究[J]. 空天防御, 2020, 3(3): 89-95.
	ZHOU L, WANG H Y, SHANG H B, et al. Optimization design of glide trajectory of space-based reentry vehicle based on Gaussian pseudo-spectrum method[J]. Air-space Defense Force Structure, 2020, 3(3): 89-95. (in Chinese)
[22]	PENG Z H, WANG D, LI T S, et al. Output-feedback cooperative formation maneuvering of autonomous surface vehicles with connectivity preservation and collision avoidance[J]. IEEE Transactions on Cybernetics, 2020, 50(6): 2527-2535. doi: 10.1109/TCYB.2019.2914717 pmid: 31180878
[23]	YADOLLAH R, AMIR K, KEN S C, et al. A potential field-based model predictive path-planning controller for autonomous road vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2017, 18(5):1255-1267.
[24]	ZHANG H P, WANG H L, LI N, et al. Time-optimal memetic whale optimization algorithm for hypersonic vehicle reentry trajectory optimization with no-fly zones[J]. Neural Computing & Applications, 2020, 32(7):2735-2749.