基于强化学习的集群多目标分配与智能决策方法

doi:10.3969/j.issn.1000-1093.2021.09.025

摘要/Abstract

摘要： 为提升高动态协同攻击条件下的攻防效能，研究基于强化学习的集群多目标智能分配与决策方法。建立综合攻击性能评估准则，包括基于相对运动信息的攻击优势度评估以及基于目标固有信息的威胁度评估。综合攻击性能、突防概率以及攻击消耗，设计攻防效费比性能指标。构建基于强化学习的多目标决策架构，设计以分配向量为基本元素的动作空间，以及基于量化性能指标的状态空间，利用Q-Learning方法对协同攻击方案，包括导弹选取以及分配形式进行智能决策。仿真结果表明，强化学习能够实现攻防效能最优的多目标在线决策，其计算效率相对于粒子群优化算法具有更明显的优势。

关键词: 目标分配, 协同攻击, 攻防效能, 智能决策, 强化学习

Abstract: A reinforcement learning-based swarm intelligent decision-making method of cooperative multi-target attack under high-dynamic situation is proposed. The composite evaluation criteria of attack performance is established, including the evaluation of attack superiority based on relative motion information and the threat evaluation based on the inherent information of target. To evaluate the attack-defence effectiveness, a cost-effectiveness ratio index is designed by combining attack performance, penetration probability and attack cost together. In addition, a multi-target decision-making architecture based on reinforcement learning is constructed, and an action space with allocation vectors as basic elements and a state space based on quantified performance indicators are designed. Q-Learning is employed to make intelligent decisions on cooperative attack plans, including missile selection and target assignment. The simulated results show that reinforcement learning can achieve multi-target online decision-making with the optimal offensive and defensive effectiveness, and its computational efficiency has more obvious advantages than that of particle swarm optimizer.

Key words: targetassignment, cooperativeattack, attack-defenseeffectiveness, intelligentdecision, reinforcementlearning

中图分类号:

TJ761.1⁺4

朱建文，赵长见，李小平，包为民. 基于强化学习的集群多目标分配与智能决策方法[J]. 兵工学报, 2021, 42(9): 2040-2048.

ZHU Jianwen, ZHAO Changjian, LI Xiaoping, BAO Weimin. Multi-target Assignment and Intelligent Decision Based on Reinforcement Learning[J]. Acta Armamentarii, 2021, 42(9): 2040-2048.

参考文献

［1］任章，郭栋，董希旺. 飞行器集群协同制导控制方法及应用研究［J］.导航定位与授时，2019, 6(5): 1-9.
REN Z, GUO D, DONG X W. Research on the cooperative gui-dance and control method and application for aerial vehicle swarm systems［J］. Navigation Position & Timing, 2019, 6(5): 1-9. (in Chinese)
［2］ BOGDANOWICZ Z R, TOLANO A, PATEL K, et al. Optimization of weapon-target pairings based on kill probabilities［J］. IEEE Transactions on Cybernetics, 2013, 43(6): 1835-1844.
［3］卢森堂.导弹自主编队协同制导控制技术［M］. 北京：国防工业出版社，2015: 88-96.
LU S T. Cooperative guidance & control of missiles autonomous formation［M］. Beijing: National Defense Industry Press, 2015: 88-96. (in Chinese)
［4］刘树衎, 王航宇, 卢发兴. 多枚反舰导弹协同攻击在线目标分配［J］. 指挥控制与仿真, 2016, 38(1): 38-40,52.
LIU S K, WANG H Y, LU F X. Online target assignment for cooperative attack of anti-ship of multiple missiles［J］. Command Control & Simulation, 2016, 38(1): 38-40,52. (in Chinese)
［5］ ZHAO M, ZHAO L L, SU X H, et al. Improved discrete mapping differential evolution for multi-unmanned aerial vehicles cooperative multi-targets assignment under unified model［J］. International Journal of Machine Learning & Cybernetics, 2017, 8(3): 765- 780.
［6］ DING Y F, YANG L Q, HOU J Y, et al. Multi-target collaborative combat decision-making by improved particle swarm optimizer［J］. Transactions of Nanjing University of Aeronautics and Astronautics, 2018, 35(1): 181-187.
［7］ SUN J J, LIU C S. Finite-horizon differential games for missile-target interception system using adaptive dynamic programming with input constraints［J］. International Journal of System Science, 2018, 49(2): 264-283.
［8］吴蔚楠. 多无人飞行器分布式任务规划技术研究［D］. 哈尔滨：哈尔滨工业大学，2018: 20-32.
WU W N. Research on distributed mission planning for multiple unmanned aerial vehicles ［D］. Harbin: Harbin Institute of Technology，2018: 20-32. (in Chinese)
［9］ CHEN W N, ZHANG J, CHUNG H S H, et al. A novel-based particle swarm optimization model for discrete optimization problems［J］. IEEE Transactions on Evolutionary Computation, 2010, 14(2): 278-300.
［10］费爱国，张陆游，刘刚，等. 基于粒子群拍卖混合算法的空空导弹制导权移交技术［J］. 宇航学报，2013, 34(3): 340-346.
FEI A G, ZHANG L Y, LIU G, et al. The technique for air-to-air missile guidance superiority handover based on particle swarm auction hybrid algorithm［J］. Journal of Astronautics, 2013, 34(3): 340-346. (in Chinese)
［11］ PRASHANT B, FARUK K, NAVDEEP S. Reinforcement learning based obstacle avoidance for autonomous underwater vehicle［J］. Journal of Marine Science and Application, 2019, 18(2): 228-238.
［12］ JUNELL J J, VAN KAMPENY E J, VISSER C D, et al. Reinforcement learning applied to a quadrotor guidance law in autonomous flight ［C］∥Proceedings of AIAA Guidance, Navigation, and Control Conference. Kissimmee, FL, US: AIAA, 2015.
［13］ GAUDET B, FURFARO R. Missile homing-phase guidance law design using reinforcement learning ［C］∥Proceedings of AIAA Guidance, Navigation, and Control Conference. Minneapolis, MN, US: AIAA, 2012.
［14］ GAUDET B, FURFARO R, LINARES R. Reinforcement learning for angle-only intercept guidance of maneuvering targets［C］∥Proceedings of AIAA SciTech Forum. Orlando, FL, US: AIAA, 2020.
［15］张秦浩，敖百强，张秦雪. Q-learning强化学习制导律［J］. 系统工程与电子技术, 2020, 42(2): 414-419.
ZHANG Q H, AO B Q, ZHANG Q X. Reinforcement learning guidance law of Q-learning［J］. Systems Engineering and Electronics, 2020, 42(2): 414-419. (in Chinese)
［16］刘冰雁, 叶雄兵, 岳智宏, 等. 基于多组并行深度Q网络的连续空间追逃博弈算法［J］. 兵工学报, 2021, 42(3): 663-672.
LIU B Y, YE X B, YUE Z H, et al. Continuous space pursuit-evasion game algorithm based on multi-group deep Q-network［J］. Acta Armamentarii, 2021, 42(3): 663-672. (in Chinese)



下2篇留版

[1]	李理，李旭光，郭凯杰，史超，陈昭文. 国产化环境下基于强化学习的地空协同作战仿真[J]. 兵工学报, 2022, 43(S1): 74-81.
[2]	马也，范文慧，常天庆. 基于智能算法的无人集群防御作战方案优化方法[J]. 兵工学报, 2022, 43(6): 1415-1425.
[3]	陈中原，韦文书，陈万春. 基于强化学习的多发导弹协同攻击智能制导律[J]. 兵工学报, 2021, 42(8): 1638-1647.
[4]	张晚晴，余文斌，李静琳，陈万春. 基于纵程解析解的飞行器智能横程机动再入协同制导[J]. 兵工学报, 2021, 42(7): 1400-1411.
[5]	张进，郭浩，陈统. 基于可适应匈牙利算法的武器-目标分配问题[J]. 兵工学报, 2021, 42(6): 1339-1344.
[6]	高昂，董志明，叶红兵，宋敬华，郭齐胜. 基于深度强化学习的巡飞弹突防控制决策[J]. 兵工学报, 2021, 42(5): 1101-1110.
[7]	耿天立，高昂，王琦，段渭军，胡延苏. 基于深度确定性策略梯度学习的无线反向散射数据卸载优化[J]. 兵工学报, 2021, 42(12): 2655-2663.
[8]	史振庆，梁晓龙，张佳强，刘流，范翔宇. 基于协同攻击区的航空集群最优空间构型研究[J]. 兵工学报, 2019, 40(4): 788-798.
[9]	孔德鹏，常天庆，郝娜，张雷，郭理彬. 基于对抗的突击武器与支援武器协同火力打击决策方法[J]. 兵工学报, 2019, 40(3): 629-640.
[10]	冷鹏飞，徐朝阳. 一种深度强化学习的雷达辐射源个体识别方法[J]. 兵工学报, 2018, 39(12): 2420-2426.
[11]	夏维，刘新学，范阳涛，元锋刚. 基于改进型多目标粒子群优化算法的武器-目标分配[J]. 兵工学报, 2016, 37(11): 2085-2093.
[12]	王然辉，王超. 面向对地打击武器-目标分配问题的遗传算法变量取值控制技术[J]. 兵工学报, 2016, 37(10): 1889-1895.
[13]	余戌曈，吴玲，卢发兴. 舰艇摇摆下射界动态变化对武器目标分配的影响[J]. 兵工学报, 2015, 36(9): 1819-1824.
[14]	唐传林，黄长强，杜海文，黄汉桥，丁达理，罗畅. 无人作战飞行器编队协同攻击轨迹规划研究[J]. 兵工学报, 2014, 35(4): 523-530.
[15]	杨飞，王青，侯砚泽. 基于整数域改进粒子群优化算法的多平台武器目标分配[J]. 兵工学报, 2011, 32(7): 906-912.