Multi-target Assignment and Intelligent Decision Based on Reinforcement Learning

doi:10.3969/j.issn.1000-1093.2021.09.025

Abstract

Abstract: A reinforcement learning-based swarm intelligent decision-making method of cooperative multi-target attack under high-dynamic situation is proposed. The composite evaluation criteria of attack performance is established, including the evaluation of attack superiority based on relative motion information and the threat evaluation based on the inherent information of target. To evaluate the attack-defence effectiveness, a cost-effectiveness ratio index is designed by combining attack performance, penetration probability and attack cost together. In addition, a multi-target decision-making architecture based on reinforcement learning is constructed, and an action space with allocation vectors as basic elements and a state space based on quantified performance indicators are designed. Q-Learning is employed to make intelligent decisions on cooperative attack plans, including missile selection and target assignment. The simulated results show that reinforcement learning can achieve multi-target online decision-making with the optimal offensive and defensive effectiveness, and its computational efficiency has more obvious advantages than that of particle swarm optimizer.

Key words: targetassignment, cooperativeattack, attack-defenseeffectiveness, intelligentdecision, reinforcementlearning

CLC Number:

TJ761.1⁺4

ZHU Jianwen, ZHAO Changjian, LI Xiaoping, BAO Weimin. Multi-target Assignment and Intelligent Decision Based on Reinforcement Learning[J]. Acta Armamentarii, 2021, 42(9): 2040-2048.

References

［1］任章，郭栋，董希旺. 飞行器集群协同制导控制方法及应用研究［J］.导航定位与授时，2019, 6(5): 1-9.
REN Z, GUO D, DONG X W. Research on the cooperative gui-dance and control method and application for aerial vehicle swarm systems［J］. Navigation Position & Timing, 2019, 6(5): 1-9. (in Chinese)
［2］ BOGDANOWICZ Z R, TOLANO A, PATEL K, et al. Optimization of weapon-target pairings based on kill probabilities［J］. IEEE Transactions on Cybernetics, 2013, 43(6): 1835-1844.
［3］卢森堂.导弹自主编队协同制导控制技术［M］. 北京：国防工业出版社，2015: 88-96.
LU S T. Cooperative guidance & control of missiles autonomous formation［M］. Beijing: National Defense Industry Press, 2015: 88-96. (in Chinese)
［4］刘树衎, 王航宇, 卢发兴. 多枚反舰导弹协同攻击在线目标分配［J］. 指挥控制与仿真, 2016, 38(1): 38-40,52.
LIU S K, WANG H Y, LU F X. Online target assignment for cooperative attack of anti-ship of multiple missiles［J］. Command Control & Simulation, 2016, 38(1): 38-40,52. (in Chinese)
［5］ ZHAO M, ZHAO L L, SU X H, et al. Improved discrete mapping differential evolution for multi-unmanned aerial vehicles cooperative multi-targets assignment under unified model［J］. International Journal of Machine Learning & Cybernetics, 2017, 8(3): 765- 780.
［6］ DING Y F, YANG L Q, HOU J Y, et al. Multi-target collaborative combat decision-making by improved particle swarm optimizer［J］. Transactions of Nanjing University of Aeronautics and Astronautics, 2018, 35(1): 181-187.
［7］ SUN J J, LIU C S. Finite-horizon differential games for missile-target interception system using adaptive dynamic programming with input constraints［J］. International Journal of System Science, 2018, 49(2): 264-283.
［8］吴蔚楠. 多无人飞行器分布式任务规划技术研究［D］. 哈尔滨：哈尔滨工业大学，2018: 20-32.
WU W N. Research on distributed mission planning for multiple unmanned aerial vehicles ［D］. Harbin: Harbin Institute of Technology，2018: 20-32. (in Chinese)
［9］ CHEN W N, ZHANG J, CHUNG H S H, et al. A novel-based particle swarm optimization model for discrete optimization problems［J］. IEEE Transactions on Evolutionary Computation, 2010, 14(2): 278-300.
［10］费爱国，张陆游，刘刚，等. 基于粒子群拍卖混合算法的空空导弹制导权移交技术［J］. 宇航学报，2013, 34(3): 340-346.
FEI A G, ZHANG L Y, LIU G, et al. The technique for air-to-air missile guidance superiority handover based on particle swarm auction hybrid algorithm［J］. Journal of Astronautics, 2013, 34(3): 340-346. (in Chinese)
［11］ PRASHANT B, FARUK K, NAVDEEP S. Reinforcement learning based obstacle avoidance for autonomous underwater vehicle［J］. Journal of Marine Science and Application, 2019, 18(2): 228-238.
［12］ JUNELL J J, VAN KAMPENY E J, VISSER C D, et al. Reinforcement learning applied to a quadrotor guidance law in autonomous flight ［C］∥Proceedings of AIAA Guidance, Navigation, and Control Conference. Kissimmee, FL, US: AIAA, 2015.
［13］ GAUDET B, FURFARO R. Missile homing-phase guidance law design using reinforcement learning ［C］∥Proceedings of AIAA Guidance, Navigation, and Control Conference. Minneapolis, MN, US: AIAA, 2012.
［14］ GAUDET B, FURFARO R, LINARES R. Reinforcement learning for angle-only intercept guidance of maneuvering targets［C］∥Proceedings of AIAA SciTech Forum. Orlando, FL, US: AIAA, 2020.
［15］张秦浩，敖百强，张秦雪. Q-learning强化学习制导律［J］. 系统工程与电子技术, 2020, 42(2): 414-419.
ZHANG Q H, AO B Q, ZHANG Q X. Reinforcement learning guidance law of Q-learning［J］. Systems Engineering and Electronics, 2020, 42(2): 414-419. (in Chinese)
［16］刘冰雁, 叶雄兵, 岳智宏, 等. 基于多组并行深度Q网络的连续空间追逃博弈算法［J］. 兵工学报, 2021, 42(3): 663-672.
LIU B Y, YE X B, YUE Z H, et al. Continuous space pursuit-evasion game algorithm based on multi-group deep Q-network［J］. Acta Armamentarii, 2021, 42(3): 663-672. (in Chinese)



下2篇留版

[1]	WEI Ning, WANG Guan. Application of Reinforcement Learning in Decision-Making Management of Intelligent Unmanned System [J]. Acta Armamentarii, 2022, 43(S2): 164-169.
[2]	LI Li， LI Xuguang， GUO Kaijie， SHI Chao， CHEN Zhaowen. Simulation of Ground-air Cooperative Combat Based on Reinforcement Learning in Localization Environment [J]. Acta Armamentarii, 2022, 43(S1): 74-81.
[3]	WEI Lianzhen, GONG Jianwei, CHEN Huiyan, LI Zirui, GONG Cheng. Tracking and Aiming Adaptive Control for Unmanned Combat Ground Vehicle on the Move Based on Reinforcement LearningCompensation [J]. Acta Armamentarii, 2022, 43(8): 1947-1955.
[4]	MA Ye， FAN Wenhui， CHANG Tianqing. Optimization Method of Unmanned Swarm Defensive Combat Scheme Based on Intelligent Algorithm [J]. Acta Armamentarii, 2022, 43(6): 1415-1425.
[5]	ZHI Hongxin, ZHAO Peng, LI Zhong, PENG Xiangxin, LU Xuyang, WANG Chen. A Weapon-target Assignment in Air-defense Operations Based on Shooting Probability Constraint [J]. Acta Armamentarii, 2022, 43(4): 952-959.
[6]	LI Qingbo, LI Fang, DONG Ruixing, FAN Ruishan, XIE Wenlong. Navigation Ratio Design of Proportional Navigation Law Using Reinforcement Learning [J]. Acta Armamentarii, 2022, 43(12): 3040-3047.
[7]	WANG Yan, ZHAO Wenfei, TENG Kenan, ZHOU Lu, SHAN Xin. DWTA Model for Air Defense in Critical Area at Sea under Uncertain Factors [J]. Acta Armamentarii, 2022, 43(11): 2885-2896.
[8]	NIE Junfeng, CHEN Xingjun, SU Qi. Modeling and Optimization of Weapon-target Assignment for Group Targets Defense Based on NSGA-Ⅲ Algorithm [J]. Acta Armamentarii, 2021, 42(8): 1771-1779.
[9]	CHEN Zhongyuan, WEI Wenshu, CHEN Wanchun. Reinforcement Learning-based Intelligent Guidance Law for Cooperative Attack of Multiple Missiles [J]. Acta Armamentarii, 2021, 42(8): 1638-1647.
[10]	ZHANG Wanqing, YU Wenbin, LI Jinglin, CHEN Wanchun. Cooperative Reentry Guidance for Intelligent Lateral Maneuver of Hypersonic Vehicle Based on Downrange Analytical Solution [J]. Acta Armamentarii, 2021, 42(7): 1400-1411.
[11]	ZHANG Jin， GUO Hao， CHEN Tong. Weapon-target Assignment Based on Adaptable Hungarian Algorithm [J]. Acta Armamentarii, 2021, 42(6): 1339-1344.
[12]	GAO Ang， DONG Zhiming， YE Hongbing， SONG Jinghua， GUO Qisheng. Loitering Munition Penetration Control Decision Based on Deep Reinforcement Learning [J]. Acta Armamentarii, 2021, 42(5): 1101-1110.
[13]	SHI Zhenqing, LIANG Xiaolong, ZHANG Jiaqiang, LIU Liu, FAN Xiangyu. Study of Optimal Spatial Configuration of Aircraft Swarm Based on Cooperative Attack Zone [J]. Acta Armamentarii, 2019, 40(4): 788-798.
[14]	KONG Depeng, CHANG Tianqing, HAO Na, ZHANG Lei, GUO Libin. Confrontation-based Cooperative Fire Strike Decision-making Method of Assault Weapons and Support Weapons [J]. Acta Armamentarii, 2019, 40(3): 629-640.
[15]	SUN Haiwen, XIE Xiaofang, SUN Tao, PANG Wei. Improved Cuckoo Search Algorithm for Solving Antiaircraft Weapon-target Optimal Assignment Model [J]. Acta Armamentarii, 2019, 40(1): 189-197.