Cooperative Obstacle Avoidance Decision Method for the Terminal Guidance Phase of Hypersonic Vehicles

doi:10.12382/bgxb.2023.0831

Abstract

Abstract:

In addressing the issues of cluster control caused by the rapid flight speeds of aerial vehicles and the drastic changes in environmental parameters in the cluster formation control of hypersonic vehicles, a centralized training and distributed execution intelligent cluster control method based on reinforcement learning is proposed. A hypersonic vehicle cluster kinematics model is established, and the observation space and action space are designed by taking the cooperative engagement and formation maintenance as primary objectives, and the overload, communication and obstacles as the constraints. A reward function is designed by taking into account the relationships between the relative positions and speeds of vehicles, vehicle and target, and vehicle and obstacle. The cluster formation control of hypersonic vehicles is achieved by continuously adjusting the weight of the reward function and training the hypersonic vehicles. The evaluation indicators are established to assess the performance of the algorithm, and a large number of simulation analysis in disturbed random environments are made. The results show that the proposed intelligent control method can be used to still complete the formation control in a test environment with increasing the random positions of obstacles, providing a new solution for formation control in high-speed and complex flight environments.

Key words: hypersonic vehicle, reinforcement learning, formation control, collaborative obstacle avoidance

CLC Number:

V249

HU Yanyang, HE Fan, BAI Chengchao. Cooperative Obstacle Avoidance Decision Method for the Terminal Guidance Phase of Hypersonic Vehicles[J]. Acta Armamentarii, 2024, 45(9): 3147-3160.

Figures/Tables 14

Fig.1 Hypersonic vehicle learning interaction framework

Fig.2 Obstacle area

Fig.3 Policy neural network architecture

Fig.4 Value function network architecture

Fig.5 Training environment

Table 1 SAC algorithm parameters

参数	数值		参数	数值
γ	0.99		ρ	0.01
α	0.2		\|B\|	256

Table 2 Observation space parameters

参数	数值	参数	数值
K_R/km	4	K_v/(km·s^-1)	2.0
r_obs/km	15.0	K_o/km	60.0
K_TR/km	200.0	K_Tv/(km·s^-1)	2.0

Table 3 Reward function parameters

参数	数值	参数	数值
k_fKeepV	-0.01	v_fKeepV/(km·s^-1)	2.0
k_fKeepP	-0.01	R_fKeepP/km	4.0
k_obsAvoidS	-0.07	k_obsDangerS	-0.7
k_obsAvoidG	-0.5	k_toTgtV	0.05
k_toTgtDone	1.0	d_obsSafe/km	60.0
h_safe/km	10.0	d_range/km	10.0

Fig.6 Rewards in the process of training

Table 4 Simulation parameter settings

参数	数值		参数	数值
高度h/km	30.0		速度v/(km·s^-1)	2.0
通信数量n_com	4		飞行器间距/km	10

Fig.7 Fight results in training environment

Fig.8 Flight results in complex environment

Fig.9 Single flight performance index curve

Fig.10 Performance index under action deviation

References 37

[1]	SZIROCZAK D, SMITH H. A review of design issues specific to hypersonic flight vehicles[J]. Progress in Aerospace Sciences, 2016, 84:1-28.
[2]	刘薇, 龚海华. 国外高超声速飞行器发展历程综述[J]. 飞航导弹, 2020(3):20-27,59.
	LIU W, GONG H H. A review of the development history of foreign hypersonic aircraft[J]. Aerodynamic Missile Journal, 2020(3):20-27,59. (in Chinese)
[3]	XU H J, MIRMIRANI M D, IOANNOU P A. Adaptive sliding mode control design for a hypersonic flight vehicle[J]. Journal of Guidance, Control, and Dynamics, 2004, 27(5): 829-838.
[4]	JEON I S, LEE J I, TAHK M J. Impact-time-control guidance law for anti-ship missiles[J]. IEEE Transactions on Control Systems Technology, 2006, 14(2):260-266.
[5]	ZENG J, DOU L H, XIN B. Cooperative salvo attack using guidance law of multiple missiles[J]. Journal of Advanced Computational Intelligence and Intelligent Informatics, 2015, 19(2):301-306.
[6]	VADDI S S, ALFRIEND K T, VADALI S R, et al. Formation establishment and reconfiguration using impulsive control[J]. Journal of Guidance, Control, and Dynamics, 2005, 28(2):262-268.
[7]	XIA Y Q, NA X T, SUN Z Q, et al. Formation control and collision avoidance for multi-agent systems based on position estimation[J]. ISA Transactions, 2016, 61:287-296. doi: S0019-0578(15)00314-6 pmid: 26786907
[8]	XU M L, WU Y P, YE Y D, et al. Collective crowd formation transform with mutual information-based runtime feedback: collective crowd formation transform with mutual information-based runtime feedback[J]. Computer Graphics Forum, 2015, 34(1):60-73.
[9]	YU J L, DONG X W, LI Q D, et al. Cooperative guidance strategy for multiple hypersonic gliding vehicles system[J]. Chinese Journal of Aeronautics, 2020, 33(3):990-1005.
[10]	TANG B, XI J X, LIU T Y, et al. Finite-time cooperative guidance law for multiple hypersonic vehicles in dive phase[C]//Proceedings of 2020 Chinese Automation Congress.Shanghai, China:IEEE, 2020:5137-5142.
[11]	ZHANG Y, WANG X, TANG S J. A globally fixed-time solution of distributed formation control for multiple hypersonic gliding vehicles[J]. Aerospace Science and Technology, 2020, 98:105643.
[12]	李文, 尚腾, 姚寅伟, 等. 速度时变情况下多飞行器时间协同制导方法研究[J]. 兵工学报, 2020, 41(6):1096-1110. doi: 10.3969/j.issn.1000-1093.2020.06.006
	LI W, SHANG T, YAO Y W, et al. Research on time-cooperative guidance of multiple flight vehicles with time-varying velocity[J]. Acta Armamentarii, 2020, 41(6):1096-1110. (in Chinese) doi: 10.3969/j.issn.1000-1093.2020.06.006
[13]	CHEN Q, WAN J, AI J L. L1 adaptive controller design for hypersonic formation flight[J]. Science China Technological Sciences, 2016, 59(10):1597-1608.
[14]	JIA S W, WANG X, LI F G, et al. Distributed analytical formation control and cooperative guidance for gliding vehicles[J]. International Journal of Aerospace Engineering, 2020, 2020:1-14.
[15]	安凯, 郭振云, 黄伟, 等. 低/高速飞行器系统编队协同控制方法研究进展[J]. 航空兵器, 2022, 29(5):53-65.
	AN K, GUO Z Y, HUANG W, et al. Research rrogress of formation-cooperative control methods for low-speed and high-speed vehicle systems[J]. Aero Weaponry, 2022, 29(5):53-65. (in Chinese)
[16]	MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540):529-533.
[17]	JIANG R, YE D, XIAO Y, et al. Orbital interception pursuit strategy for random evasion using deep reinforcement learning[J]. Space: Science & Technology, 2023, 3: 0086.
[18]	张晚晴, 余文斌, 李静琳, 等. 基于纵程解析解的飞行器智能横程机动再入协同制导[J]. 兵工学报, 2021, 42(7):1400-1411.
	ZHANG W Q, YU W B, LI J L, et al. Cooperative reentry guidance for intelligent lateral maneuver of hypersonic vehicle based on downrange analytical solution[J]. Acta Armamentarii, 2021, 42(7):1400-1411. (in Chinese) doi: 10.3969/j.issn.1000-1093.2021.07.007
[19]	SUN Y J, ZHANG R, LIANG W, B et al. Multi-agent cooperative search based on reinforcement learning[C]//Proceedings of the 2020 3rd International Conference on Unmanned Systems.New York,NY, US: IEEE, 2020: 891-896.
[20]	XIA Z Y, DU J, WANG J J, et al. Multi-agent reinforcement learning aided intelligent UAV swarm for target tracking[J]. IEEE Transactions on Vehicular Technology, 2022, 71(1):931-945.
[21]	SHI Y, SONG J S, HUA Y Z, et al. Leader-follower formation control for fixed-wing UAVs using deep reinforcement learning[C]//Proceedings of the 2022 41st Chinese Control Conference. Hefei, China: IEEE, 2022:3456-3461.
[22]	YU Z Q, LI J X, XU Y W, et al. Reinforcement learning-based fractional-order adaptive fault-tolerant formation control of networked fixed-wing UAVs with prescribed performance[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023:1-15. DOI:10.1109/TNNLS.2023.3281403.
[23]	LIU H, MENG Q Y, PENG F C, et al. Heterogeneous formation control of multiple UAVs with limited-input leader via reinforcement learning[J]. Neurocomputing, 2020, 412:63-71.
[24]	HAFEZ A T, GIVIGI S N, YOUSEFI S. Unmanned aerial vehicles formation using learning based model predictive control[J]. Asian Journal of Control, 2018, 20(3):1014-1026.
[25]	ZHAO W W, CHU H R, MIAO X K, et al. Research on the multiagent joint proximal policy optimization algorithm controlling cooperative fixed-wing UAV obstacle avoidance[J]. Sensors, 2020, 20(16): 4546.
[26]	XU D, CHEN G. Autonomous and cooperative control of UAV cluster with multi-agent reinforcement learning[J]. The Aeronautical Journal, 2022, 126(1300): 932-951.
[27]	PANG R. Multi-UAV formation maneuvering control based on Q-Learning fuzzy controller[C]//Proceedings of the 2010 2nd International Conference on Advanced Computer Control. Shenyang, China: IEEE, 2010: 252-257.
[28]	BAYERLEIN H, THEILE M, CACCAMO M, et al. Multi-UAV path planning for wireless data harvesting with deep reinforcement learning[J]. IEEE Open Journal of the Communications Society, 2021, 2:1171-1187.
[29]	ZHANG J D, YANG Q M, SHI G Q, et al. UAV cooperative air combat maneuver decision based on multi-agent reinforcement learning[J]. Journal of Systems Engineering and Electronics, 2021, 32(6):1421-1438. doi: 10.23919/JSEE.2021.000121
[30]	尹依伊, 王晓芳, 周健. 基于Q学习的多无人机协同航迹规划方法[J]. 兵工学报, 2023, 44(2):484-495. doi: 10.12382/bgxb.2021.0606
	YIN Y Y, WANG X F, ZHOU J. Q-learning-based multi-uav cooperative path planning method[J]. Acta Armamentarii, 2023, 44(2): 484-495. (in Chinese) doi: 10.12382/bgxb.2021.0606
[31]	韩煜, 宋韬, 郑多, 等. 基于冲突触发避碰机制的无人飞行器集群协同制导技术[J]. 兵工学报, 2023, 44(7): 1881-1895. doi: 10.12382/bgxb.2022.0152
	HAN Y, SONG T, ZHENG D, et al. Unmanned aerial vehicle cluster cooperative guidance technology based on conflict trigger mechanism[J]. Acta Armamentarii, 2023, 44(7): 1881-1895. (in Chinese) doi: 10.12382/bgxb.2022.0152
[32]	XU D, GUO Y X, YU Z Y, et al. PPO-exp: keeping fixed-wing UAV formation with deep reinforcement learning[J]. Drones, 2023, 7(1): 28.
[33]	WANG J N, XIN M. Integrated optimal formation control of multiple unmanned aerial vehicles[C]//Proceedings of AIAA Guidance, Navigation, and Control Conference. Minneapolis, MN, US:AIAA, 2012, 21(5):1731-1744.
[34]	LEE G T, KIM C O. Autonomous control of combat unmanned aerial vehicles to evade surface-to-air missiles using deep reinforcement learning[J]. IEEE Access, 2020, 8: 226724-226736.
[35]	相晓嘉, 闫超. 基于深度强化学习的固定翼无人机编队协调控制方法[J]. 航空学报, 2020, 42(4):524009.
	XIANG X J, YAN C. Coordination control method for fixed-wing UAV formation through deep reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica, 2020, 42(4): 524009. (in Chinese)
[36]	张卫东. 滑翔式高超声速飞行器再入轨迹规划与姿态控制[D]. 哈尔滨: 哈尔滨工业大学, 2017.
	ZHANG W D. Reentry trajectory planning and attitude control for hypersonic glide vehicles[D]. Harbin:Harbin Institute of Technology, 2017. (in Chinese)
[37]	HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//Proceedings of International Conference on Machine Learning. San Diego,CA, US: Journal of Machine Learning Research, 2018.