国产化环境下基于强化学习的地空协同作战仿真

doi:10.12382/bgxb.2022.A005

摘要/Abstract

摘要： 以未来战场无人地空协同作战为需求牵引，面对军事领域实战场景匮乏、训练数据不足的实际问题，聚焦仿真环境下的深度强化学习方法，实现地空协同作战仿真中多智能体决策模型。在飞腾CPU和昆仑K200硬件平台与麒麟V10操作系统环境下搭建虚拟仿真环境，设置仿真环境状态表征、各智能体动作空间及奖励机制，构建基于深度确定性策略梯度算法的多智能体模型（MADDPG），通过仿真实验验证采用MADDPG算法能够使奖励值在地空协同作战仿真场景中逐渐收敛，从而证明该模型应用于地空协同作战的决策有效性。

关键词: 地空协同作战, 强化学习, 深度确定性策略梯度算法, 多智能体模型, 国产化环境

Abstract: For the actual problems of lack of actual combat scenes and insufficient training data in the military field， the deep reinforcement learning method is used to realize a multi-agent decision-making model for the unmanned ground and air cooperative combat simulation. A virtual simulation environment is built using Phytium CPU and Kunlun K200 NPU as hardware platform and Kylin V10 operating system as software environment. The simulation environment state representation，agents’ action space and rewards mechanism are set， and a multi-agent decision-making model based on the deep deterministic policy gradient (MADDPG) algorithm is established. Simulation experiments verified that MADDPG algorithm can make reward value converge gradually in ground-air cooperative combat simulation scenarios， thus proving the effectiveness of MADDPG algorithm in the simulation scene of the ground-air cooperative combat.

Key words: ground-aircooperativecombat, reinforcementlearning, deepdeterministicpolicygradientalgorithm, multi-agentmodel, localizationenvironment

中图分类号:

E919
TP181

李理，李旭光，郭凯杰，史超，陈昭文. 国产化环境下基于强化学习的地空协同作战仿真[J]. 兵工学报, 2022, 43(S1): 74-81.

LI Li， LI Xuguang， GUO Kaijie， SHI Chao， CHEN Zhaowen. Simulation of Ground-air Cooperative Combat Based on Reinforcement Learning in Localization Environment[J]. Acta Armamentarii, 2022, 43(S1): 74-81.

参考文献

［1］孟红，朱森.地面无人系统的发展及未来趋势[J].兵工学报，2014，35(增刊1):1-7.
MENG H，ZHU S.The development and future trends of unmanned ground systems[J]. Acta Armamentarii，2014，35(S1):1-7.（in Chinese）
[2］张宇，张琰，邱绵浩，等.地空无人平台协同作战应用研究[J].火力与指挥控制，2021，46(5):1-5，11.
ZHANG Y，ZHANG Y，QIU M H，et al. Research on the ground-air unmanned platform cooperative combat application[J].Fire Control & Command Control，2021，46(5):1-5，11.(in Chinese)
[3］李航，刘代金，刘禹.军事智能博弈对抗系统设计框架研究[J].火力与指挥控制，2020，45(9):116-121.
LI H，LIU D J，LIU Y.Architecture design research of military intelligent wargame system[J].Fire Control & Command Control，2020，45(9):116-121.(in Chinese)
［4］ MNIH V，KAVUKCUOGLU K，SILVER D，et al. Playing Atari with deep reinforcement learning［J/OL］. Computer Science， 2013.arXiv preprint arXiv:1312.5602.
［5］ LILLICRAP T P，HUNT J J，PRITZEL A，et al. Continuous control with deep reinforcement learning［J］. Computer Science，2016，8(6): A187.
［6］ SILVER D，HUANG A，MADDISON C J，et al. Mastering the game of Go with deep neural networks and tree search［J］. Nature，2016，529:484-489.
［7］ HERNANDEZ-LEAL P，KARTAL B，TAYLOR M E. Is multiagent deep reinforcement learning the answer or the question? a brief survey［J/OL］.Computer Science，2019.rXiv:1810.05587v3.
［8］ TAMPUU A，MATIISEN T，KODELJA D，et al. Multiagent cooperation and competition with deep reinforcement learning［J］.PLoS ONE，2017，12(4):e0172395.
［9］ LOWE R，WU Y，TAMAR A，et al.Multi-agent actor-critic for mixed cooperative competitive environments［C］∥Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook，NY，US:Curran Associates Inc.，2017:6382-6393.
［10］ HANSEN S. Using deep Q-learning to control optimization hyperparameters［J/OL］.2016. arXiv preprint arXiv:1602.04062.
［11］ AKKAYA I，ANDRYCHOWICZ M，CHOCIEJ M，et al.Solving rubik's cube with a robot hand［J］.2019.arXiv preprint arXiv:1910.07113.
［12］ VINYALS O，BABUSCHKIN I，CZARNECKI W M，et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning［J］.Nature，2019，575:350-354.
［13］ KENDALL A，HAWKE J，JANZ D，et al.Learning to drive in a day［C］∥Proceedings of the IEEE International Conference on Robotics and Automation.Montreal，QC，Canada: IEEE，2019:8248-8254.
[14］陶九阳，吴琳，胡晓峰.AlphaGo技术原理分析及人工智能军事应用展望[J].指挥与控制学报，2016，2(2):114-120.
TAO J Y，WU L，HU X F. Principle analysis on AlphaGO and perspective in military application of artifical intelligence[J]. Journal of Command and Control，2016，2(2):114-120.(in Chinese)
[15］马悦，吴琳，许霄，等.智能化作战任务规划需求分析[J].指挥控制与仿真，2021，43(4):61-67.
MA Y，WU L，XU X，et al.Requirement analysis of intelligent operation task planning[J].Command Control & Simulation，2021，43(4):61-67.(in Chinese)
[16］徐志雄，曹雷，陈希亮，等.基于Double BP神经网络的分层强化学习方法[J/OL].解放军理工大学学报(自然科学版):1-6. [2021-07-12]. http:∥kns.cnki.net/kcms/detail/32.1430.N.20170417.1611.008.html.
XU Z X，CAO L，CHEN X L，et al.Hierarchical reinforcement learning method based on double BP neural network[J/OL].Journal of PLA University of Science and Technology (Natural Science Edition):1-6.[2021-07-12]. http:∥kns.cnki.net/kcms/detail/32.1430.N.20170417.1611.008.html.（in Chinese）
[17］徐志雄，曹雷，陈希亮.基于强化学习的无人坦克对战仿真研究[J].计算机工程与应用，2018，54(8):166-171.
XU Z X，CAO L，CHEN X L. Research on unmanned tank battle simulation based on reinforcement learning[J]. Computer Engineering and Applications，2018，54(8):166-171.(in Chinese)
[18］卢锐轩，孙莹，杨奇，等.基于人工智能技术的智能自博弈平台研究[J].战术导弹技术，2019(2):47-52，98.
LU R X，SUN Y，YANG Q，et al. Research on intelligent self-game platform based on artificial intelligence technology[J].Tactical Missile Technology，2019(2):47-52，98.(in Chinese)
[19］黄晓冬，苑海涛，毕敬，等.基于DQN的海战场舰船路径规划及仿真[J].系统仿真学报，2021，33(10):2440-2448.
HUANG X D，YUAN H T，BI J，et al. DQN-based path planning method and simulation for submarine and warship in naval battlefield[J].Journal of System Simulation，2021，33(10):2440-2448.(in Chinese)

[1]	马也，范文慧，常天庆. 基于智能算法的无人集群防御作战方案优化方法[J]. 兵工学报, 2022, 43(6): 1415-1425.
[2]	朱建文，赵长见，李小平，包为民. 基于强化学习的集群多目标分配与智能决策方法[J]. 兵工学报, 2021, 42(9): 2040-2048.
[3]	陈中原，韦文书，陈万春. 基于强化学习的多发导弹协同攻击智能制导律[J]. 兵工学报, 2021, 42(8): 1638-1647.
[4]	张晚晴，余文斌，李静琳，陈万春. 基于纵程解析解的飞行器智能横程机动再入协同制导[J]. 兵工学报, 2021, 42(7): 1400-1411.
[5]	高昂，董志明，叶红兵，宋敬华，郭齐胜. 基于深度强化学习的巡飞弹突防控制决策[J]. 兵工学报, 2021, 42(5): 1101-1110.
[6]	耿天立，高昂，王琦，段渭军，胡延苏. 基于深度确定性策略梯度学习的无线反向散射数据卸载优化[J]. 兵工学报, 2021, 42(12): 2655-2663.
[7]	冷鹏飞，徐朝阳. 一种深度强化学习的雷达辐射源个体识别方法[J]. 兵工学报, 2018, 39(12): 2420-2426.
[8]	杨萍，毕义明，孙淑玲. 具有自主决策能力的机动单元智能体研究[J]. 兵工学报, 2007, 28(11): 1363-1366.