高超声速飞行器末制导段协同避障决策方法

doi:10.12382/bgxb.2023.0831

摘要/Abstract

摘要：

针对高超声速飞行器集群编队控制中飞行器飞行速度快、环境参数变化剧烈导致的集群控制困难问题,提出基于强化学习的集中式训练、分布式执行智能集群控制方法。建立高超声速飞行器集群运动学模型,以协同打击、编队保持作为典型目标,将过载、通信、障碍作为约束,设计了观测空间与动作空间。考虑飞行器间、飞行器与目标、飞行器与障碍物相对位置速度关系设计奖励函数,通过不断调整奖励函数权值并对飞行器进行训练,实现飞行器集群的编队控制。为了考核新算法的性能,建立评价指标,开展大量有扰动随机环境下的仿真分析。仿真结果表明,新的智能控制方法在障碍物随机位置增加的测试环境中仍能完成编队控制,为高速复杂飞行环境下的编队控制提供了一种新的解决方法。

关键词: 高超声速飞行器, 强化学习, 编队控制, 协同避障

Abstract:

In addressing the issues of cluster control caused by the rapid flight speeds of aerial vehicles and the drastic changes in environmental parameters in the cluster formation control of hypersonic vehicles, a centralized training and distributed execution intelligent cluster control method based on reinforcement learning is proposed. A hypersonic vehicle cluster kinematics model is established, and the observation space and action space are designed by taking the cooperative engagement and formation maintenance as primary objectives, and the overload, communication and obstacles as the constraints. A reward function is designed by taking into account the relationships between the relative positions and speeds of vehicles, vehicle and target, and vehicle and obstacle. The cluster formation control of hypersonic vehicles is achieved by continuously adjusting the weight of the reward function and training the hypersonic vehicles. The evaluation indicators are established to assess the performance of the algorithm, and a large number of simulation analysis in disturbed random environments are made. The results show that the proposed intelligent control method can be used to still complete the formation control in a test environment with increasing the random positions of obstacles, providing a new solution for formation control in high-speed and complex flight environments.

Key words: hypersonic vehicle, reinforcement learning, formation control, collaborative obstacle avoidance

中图分类号:

V249

胡砚洋, 何凡, 白成超. 高超声速飞行器末制导段协同避障决策方法[J]. 兵工学报, 2024, 45(9): 3147-3160.

HU Yanyang, HE Fan, BAI Chengchao. Cooperative Obstacle Avoidance Decision Method for the Terminal Guidance Phase of Hypersonic Vehicles[J]. Acta Armamentarii, 2024, 45(9): 3147-3160.

图/表 14

图1 高超声速飞行器学习交互框架

Fig.1 Hypersonic vehicle learning interaction framework

图2 障碍区域

Fig.2 Obstacle area

图3 策略神经网络架构

Fig.3 Policy neural network architecture

图4 值函数网络架构

Fig.4 Value function network architecture

图5 训练环境

Fig.5 Training environment

表1 SAC算法参数

Table 1 SAC algorithm parameters

参数	数值		参数	数值
γ	0.99		ρ	0.01
α	0.2		\|B\|	256

表2 观测空间参数

Table 2 Observation space parameters

参数	数值	参数	数值
K_R/km	4	K_v/(km·s^-1)	2.0
r_obs/km	15.0	K_o/km	60.0
K_TR/km	200.0	K_Tv/(km·s^-1)	2.0

表3 奖励函数参数

Table 3 Reward function parameters

参数	数值	参数	数值
k_fKeepV	-0.01	v_fKeepV/(km·s^-1)	2.0
k_fKeepP	-0.01	R_fKeepP/km	4.0
k_obsAvoidS	-0.07	k_obsDangerS	-0.7
k_obsAvoidG	-0.5	k_toTgtV	0.05
k_toTgtDone	1.0	d_obsSafe/km	60.0
h_safe/km	10.0	d_range/km	10.0

图6 训练过程奖励

Fig.6 Rewards in the process of training

表4 仿真参数设置

Table 4 Simulation parameter settings

参数	数值		参数	数值
高度h/km	30.0		速度v/(km·s^-1)	2.0
通信数量n_com	4		飞行器间距/km	10

图7 训练环境飞行结果

Fig.7 Fight results in training environment

图8 复杂环境飞行结果

Fig.8 Flight results in complex environment

图9 单次飞行性能指标曲线

Fig.9 Single flight performance index curve

图10 动作偏差下性能指标

Fig.10 Performance index under action deviation

参考文献 37

[1]	SZIROCZAK D, SMITH H. A review of design issues specific to hypersonic flight vehicles[J]. Progress in Aerospace Sciences, 2016, 84:1-28.
[2]	刘薇, 龚海华. 国外高超声速飞行器发展历程综述[J]. 飞航导弹, 2020(3):20-27,59.
	LIU W, GONG H H. A review of the development history of foreign hypersonic aircraft[J]. Aerodynamic Missile Journal, 2020(3):20-27,59. (in Chinese)
[3]	XU H J, MIRMIRANI M D, IOANNOU P A. Adaptive sliding mode control design for a hypersonic flight vehicle[J]. Journal of Guidance, Control, and Dynamics, 2004, 27(5): 829-838.
[4]	JEON I S, LEE J I, TAHK M J. Impact-time-control guidance law for anti-ship missiles[J]. IEEE Transactions on Control Systems Technology, 2006, 14(2):260-266.
[5]	ZENG J, DOU L H, XIN B. Cooperative salvo attack using guidance law of multiple missiles[J]. Journal of Advanced Computational Intelligence and Intelligent Informatics, 2015, 19(2):301-306.
[6]	VADDI S S, ALFRIEND K T, VADALI S R, et al. Formation establishment and reconfiguration using impulsive control[J]. Journal of Guidance, Control, and Dynamics, 2005, 28(2):262-268.
[7]	XIA Y Q, NA X T, SUN Z Q, et al. Formation control and collision avoidance for multi-agent systems based on position estimation[J]. ISA Transactions, 2016, 61:287-296. doi: S0019-0578(15)00314-6 pmid: 26786907
[8]	XU M L, WU Y P, YE Y D, et al. Collective crowd formation transform with mutual information-based runtime feedback: collective crowd formation transform with mutual information-based runtime feedback[J]. Computer Graphics Forum, 2015, 34(1):60-73.
[9]	YU J L, DONG X W, LI Q D, et al. Cooperative guidance strategy for multiple hypersonic gliding vehicles system[J]. Chinese Journal of Aeronautics, 2020, 33(3):990-1005.
[10]	TANG B, XI J X, LIU T Y, et al. Finite-time cooperative guidance law for multiple hypersonic vehicles in dive phase[C]//Proceedings of 2020 Chinese Automation Congress.Shanghai, China:IEEE, 2020:5137-5142.
[11]	ZHANG Y, WANG X, TANG S J. A globally fixed-time solution of distributed formation control for multiple hypersonic gliding vehicles[J]. Aerospace Science and Technology, 2020, 98:105643.
[12]	李文, 尚腾, 姚寅伟, 等. 速度时变情况下多飞行器时间协同制导方法研究[J]. 兵工学报, 2020, 41(6):1096-1110. doi: 10.3969/j.issn.1000-1093.2020.06.006
	LI W, SHANG T, YAO Y W, et al. Research on time-cooperative guidance of multiple flight vehicles with time-varying velocity[J]. Acta Armamentarii, 2020, 41(6):1096-1110. (in Chinese) doi: 10.3969/j.issn.1000-1093.2020.06.006
[13]	CHEN Q, WAN J, AI J L. L1 adaptive controller design for hypersonic formation flight[J]. Science China Technological Sciences, 2016, 59(10):1597-1608.
[14]	JIA S W, WANG X, LI F G, et al. Distributed analytical formation control and cooperative guidance for gliding vehicles[J]. International Journal of Aerospace Engineering, 2020, 2020:1-14.
[15]	安凯, 郭振云, 黄伟, 等. 低/高速飞行器系统编队协同控制方法研究进展[J]. 航空兵器, 2022, 29(5):53-65.
	AN K, GUO Z Y, HUANG W, et al. Research rrogress of formation-cooperative control methods for low-speed and high-speed vehicle systems[J]. Aero Weaponry, 2022, 29(5):53-65. (in Chinese)
[16]	MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540):529-533.
[17]	JIANG R, YE D, XIAO Y, et al. Orbital interception pursuit strategy for random evasion using deep reinforcement learning[J]. Space: Science & Technology, 2023, 3: 0086.
[18]	张晚晴, 余文斌, 李静琳, 等. 基于纵程解析解的飞行器智能横程机动再入协同制导[J]. 兵工学报, 2021, 42(7):1400-1411.
	ZHANG W Q, YU W B, LI J L, et al. Cooperative reentry guidance for intelligent lateral maneuver of hypersonic vehicle based on downrange analytical solution[J]. Acta Armamentarii, 2021, 42(7):1400-1411. (in Chinese) doi: 10.3969/j.issn.1000-1093.2021.07.007
[19]	SUN Y J, ZHANG R, LIANG W, B et al. Multi-agent cooperative search based on reinforcement learning[C]//Proceedings of the 2020 3rd International Conference on Unmanned Systems.New York,NY, US: IEEE, 2020: 891-896.
[20]	XIA Z Y, DU J, WANG J J, et al. Multi-agent reinforcement learning aided intelligent UAV swarm for target tracking[J]. IEEE Transactions on Vehicular Technology, 2022, 71(1):931-945.
[21]	SHI Y, SONG J S, HUA Y Z, et al. Leader-follower formation control for fixed-wing UAVs using deep reinforcement learning[C]//Proceedings of the 2022 41st Chinese Control Conference. Hefei, China: IEEE, 2022:3456-3461.
[22]	YU Z Q, LI J X, XU Y W, et al. Reinforcement learning-based fractional-order adaptive fault-tolerant formation control of networked fixed-wing UAVs with prescribed performance[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023:1-15. DOI:10.1109/TNNLS.2023.3281403.
[23]	LIU H, MENG Q Y, PENG F C, et al. Heterogeneous formation control of multiple UAVs with limited-input leader via reinforcement learning[J]. Neurocomputing, 2020, 412:63-71.
[24]	HAFEZ A T, GIVIGI S N, YOUSEFI S. Unmanned aerial vehicles formation using learning based model predictive control[J]. Asian Journal of Control, 2018, 20(3):1014-1026.
[25]	ZHAO W W, CHU H R, MIAO X K, et al. Research on the multiagent joint proximal policy optimization algorithm controlling cooperative fixed-wing UAV obstacle avoidance[J]. Sensors, 2020, 20(16): 4546.
[26]	XU D, CHEN G. Autonomous and cooperative control of UAV cluster with multi-agent reinforcement learning[J]. The Aeronautical Journal, 2022, 126(1300): 932-951.
[27]	PANG R. Multi-UAV formation maneuvering control based on Q-Learning fuzzy controller[C]//Proceedings of the 2010 2nd International Conference on Advanced Computer Control. Shenyang, China: IEEE, 2010: 252-257.
[28]	BAYERLEIN H, THEILE M, CACCAMO M, et al. Multi-UAV path planning for wireless data harvesting with deep reinforcement learning[J]. IEEE Open Journal of the Communications Society, 2021, 2:1171-1187.
[29]	ZHANG J D, YANG Q M, SHI G Q, et al. UAV cooperative air combat maneuver decision based on multi-agent reinforcement learning[J]. Journal of Systems Engineering and Electronics, 2021, 32(6):1421-1438. doi: 10.23919/JSEE.2021.000121
[30]	尹依伊, 王晓芳, 周健. 基于Q学习的多无人机协同航迹规划方法[J]. 兵工学报, 2023, 44(2):484-495. doi: 10.12382/bgxb.2021.0606
	YIN Y Y, WANG X F, ZHOU J. Q-learning-based multi-uav cooperative path planning method[J]. Acta Armamentarii, 2023, 44(2): 484-495. (in Chinese) doi: 10.12382/bgxb.2021.0606
[31]	韩煜, 宋韬, 郑多, 等. 基于冲突触发避碰机制的无人飞行器集群协同制导技术[J]. 兵工学报, 2023, 44(7): 1881-1895. doi: 10.12382/bgxb.2022.0152
	HAN Y, SONG T, ZHENG D, et al. Unmanned aerial vehicle cluster cooperative guidance technology based on conflict trigger mechanism[J]. Acta Armamentarii, 2023, 44(7): 1881-1895. (in Chinese) doi: 10.12382/bgxb.2022.0152
[32]	XU D, GUO Y X, YU Z Y, et al. PPO-exp: keeping fixed-wing UAV formation with deep reinforcement learning[J]. Drones, 2023, 7(1): 28.
[33]	WANG J N, XIN M. Integrated optimal formation control of multiple unmanned aerial vehicles[C]//Proceedings of AIAA Guidance, Navigation, and Control Conference. Minneapolis, MN, US:AIAA, 2012, 21(5):1731-1744.
[34]	LEE G T, KIM C O. Autonomous control of combat unmanned aerial vehicles to evade surface-to-air missiles using deep reinforcement learning[J]. IEEE Access, 2020, 8: 226724-226736.
[35]	相晓嘉, 闫超. 基于深度强化学习的固定翼无人机编队协调控制方法[J]. 航空学报, 2020, 42(4):524009.
	XIANG X J, YAN C. Coordination control method for fixed-wing UAV formation through deep reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica, 2020, 42(4): 524009. (in Chinese)
[36]	张卫东. 滑翔式高超声速飞行器再入轨迹规划与姿态控制[D]. 哈尔滨: 哈尔滨工业大学, 2017.
	ZHANG W D. Reentry trajectory planning and attitude control for hypersonic glide vehicles[D]. Harbin:Harbin Institute of Technology, 2017. (in Chinese)
[37]	HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//Proceedings of International Conference on Machine Learning. San Diego,CA, US: Journal of Machine Learning Research, 2018.