A Hierarchical Multi-Agent Collaborative Decision-making Method Based on the Actor-critic Framework

doi:10.12382/bgxb.2023.0862

Abstract

Abstract:

A hierarchical multi-agent collaborative decision-making method based on the actor-critic (AC) frameworkis proposed to address the issues of improper task allocation and weak decision consistency in the collaborative decision-making of multiple agents in complex operational environments. The proposed method divides the decision-making process into different levels and utilizes the AC framework to facilitate information exchange and decision coordination among the agents, thereby enhancing thedecision efficiency and combat effectiveness. At the higher level, the top-level agents formulate thetask decisions by decomposing and assigning overall tasks to the lower-level agents. At the lower level, the lower-level agents make action decisions based on subtasks and provide feedback to the higher level. Experimental results demonstrate that the proposed method performs well in various operational simulation scenarios, showcasing its potential to enhance themilitary operational collaborative decision-making capability.

Key words: deep reinforcement learning, hierarchical multi-agent, information sharing, intelligent war-gaming simulation

CLC Number:

TP18
E91

FU Yanfang, LEI Kailin, WEI Jianing, CAO Zijian, YANG Bo, WANG Wei, SUN Zelong, LI Qinjie. A Hierarchical Multi-Agent Collaborative Decision-making Method Based on the Actor-critic Framework[J]. Acta Armamentarii, 2024, 45(10): 3385-3396.

Figures/Tables 17

References 32

[1]	常晓飞, 蒋邓怀, 姬晓闯, 等. 无人作战系统仿真发展综述[J]. 无人系统技术, 2021, 4(6): 28-36.
	CHANG X F, JIANG D H, JI X C, et al. Overview of unmanned combat system simulation development[J]. Unmanned Systems Technology, 2021, 4(6): 28-36. (in Chinese)
[2]	牛轶峰, 肖湘江, 柯冠岩. 无人机集群作战概念及关键技术分析[J]. 国防科技, 2013, 34(5): 37-43.
	NIU Y F, XIAO X J, KE G Y. Concept and key technology analysis of unmanned aerial vehicle swarm combat[J]. National Defense Science & Technology, 2013, 34(5): 37-43. (in Chinese)
[3]	刘彬. 多无人艇协同编队控制理论及实验研究[D]. 武汉: 华中科技大学, 2021.
	LIU B. Theory and experimental research on cooperative control of multi-unmanned surface vehicles[D]. Wuhan: Huazhong University of Science and Technology, 2021. (in Chinese)
[4]	裴国旭, 杜晓明, 薛昭, 等. 多智能体系统在军事仿真领域的应用现状[J]. 飞航导弹, 2017(2): 46-49, 73.
	PEI G X, DU X M, XUE Z, et al. Current application status of multi-agent systems in the field of military simulation[J]. Missile and Space Vehicle, 2017(2): 46-49, 73. (in Chinese)
[5]	张峰, 李明强, 唐思琦, 等. 多智能体协同决策方法研究[J]. 中国电子科学研究院学报, 2022, 17(9): 905-910.
	ZHANG F, LI M Q, TANG S Q, et al. Research on multi-agent collaborative decision-making methods[J]. Journal of the China Academy of Electronics and Information Technology, 2022, 17(9): 905-910. (in Chinese)
[6]	周文卿, 朱纪洪, 匡敏驰, 等. 基于预知博弈树的多无人机群智协同空战算法[J]. 中国科学:技术科学, 2023, 53(2): 187-199.
	ZHOU W Q, ZHU J H, KUANG M C, et al. Multi-unmanned aerial vehicle swarm intelligent collaborative air combat algorithm based on foreknowledge game tree[J]. Science China Technological Sciences, 2023, 53(2): 187-199. (in Chinese)
[7]	文超, 董文瀚, 解武杰, 等. 基于回访机制的无人机集群分布式协同区域搜索方法[J]. 航空学报, 2023, 44(11): 253-270.
	WEN C, DONG W H, XIE W J, et al. Distributed collaborative area search method for unmanned aerial vehicle swarm based on return mechanism[J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(11): 253-270. (in Chinese)
[8]	李子涵. 基于强化学习的无人机集群对抗仿真研究[D]. 西安: 西安工业大学, 2023.
	LI Z H. Research on unmanned aerial vehicle swarm adversarial simulation based on reinforcement learning[D]. Xi’an: Xi’an Technological University, 2023. (in Chinese)
[9]	李静晨, 史豪斌, 黄国胜. 基于自注意力机制和策略映射重组的多智能体强化学习算法[J]. 计算机学报, 2022, 45(9): 1842-1858.
	LI J C, SHI H B, HUANG G S. Multi-agent reinforcement learning algorithm based on self-attention mechanism and strategy mapping rearrangement[J]. Chinese Journal of Computers, 2022, 45(9): 1842-1858. (in Chinese)
[10]	赵立阳, 常天庆, 褚凯轩, 等. 完全合作类多智能体深度强化学习综述[J]. 计算机工程与应用, 2023, 59(12): 14-27. doi: 10.3778/j.issn.1002-8331.2209-0186
	ZHAO L Y, CHANG T Q, CHU K X, et al. A comprehensive review of fully cooperative multi-agent deep reinforcement learning[J]. Computer Engineering and Applications, 2023, 59(12): 14-27. (in Chinese)
[11]	夏家伟, 刘志坤, 朱旭芳, 等. 基于多智能体强化学习的无人艇集群集结方法[J]. 北京航空航天大学学报, 2023, 49(12): 3365-3376.
	XIA J W, LIU Z K, ZHU X F, et al. Method of unmanned boat swarm formation based on multi-agent reinforcement learning[J]. Journal of Beijing University of Aeronautics and Astronautics, 2023, 49(12): 3365-3376. (in Chinese)
[12]	YANG Y D, LUO R, LI M N, et al. Mean field multi-agent reinforcement learning[C]// Proceedings of the 35th International conference on machine learning. Stockholm, Sweden: PMLR, 2018, 80: 5571-5580.
[13]	陈佳黎. 面向动作类游戏仿真的多层深度强化学习研究[D]. 成都: 电子科技大学, 2020.
	CHEN J L. Research on multi-layer deep reinforcement learning for action-based game simulation[D]. Chengdu: University of Electronic Science and Technology of China, 2020. (in Chinese)
[14]	赖俊, 魏竞毅, 陈希亮. 分层强化学习综述[J]. 计算机工程与应用, 2021, 57(3): 72-79. doi: 10.3778/j.issn.1002-8331.2010-0038
	LAI J, WEI J Y, CHEN X L. A review of hierarchical reinforcement learning[J]. Computer Engineering and Applications, 2021, 57(3): 72-79. (in Chinese) doi: 10.3778/j.issn.1002-8331.2010-0038
[15]	王善锐. 基于目标分层的多智能体强化学习协作算法研究[D]. 北京: 北京交通大学, 2023.
	WANG S R. Research on multi-agent reinforcement learning collaboration algorithm based on target hierarchies[D]. Beijing: Beijing Jiaotong University, 2023. (in Chinese)
[16]	YANG X T, JI Z, WU J, et al. Hierarchical reinforcement learning with universal policies for multistep robotic manipulation[J]. IEEE Transactions on Neural Networks, 2022, 33(9): 4727-4741.
[17]	刘冰雁, 叶雄兵, 岳智宏, 等. 基于多组并行深度Q网络的连续空间追逃博弈算法[J]. 兵工学报, 2021, 42(3): 663-672. doi: 10.3969/j.issn.1000-1093.2021.03.024
	LIU B Y, YE X B, YUE Z H, et al. Continuous space pursuit-evasion game algorithm based on multiple parallel deepQ-networks[J]. Acta Armamentarii, 2021, 42(3): 663-672. (in Chinese)
[18]	于博文, 吕明, 张捷. 基于分层强化学习的联合作战仿真作战决策算法[J]. 火力与指挥控制, 2021, 46(10): 140-146.
	YU B W, LYU M, ZHANG J. Joint combat simulation decision-making algorithm based on hierarchical reinforcement learning[J]. Firepower and Command Control, 2021, 46(10): 140-146. (in Chinese)
[19]	BACON P L, HARB J, PRECUP D. The option-critic architecture[C]// Proceedings of the 31th AAAI conference on artificial intelligence. San Francisco, CA, US:AAAI, 2017: 1623.
[20]	张建东, 王鼎涵, 杨啟明, 等. 基于分层强化学习的无人机空战多维决策[J]. 兵工学报, 2023, 44(6): 1547-1563. doi: 10.12382/bgxb.2022.0711
	ZHANG J, WANG D H, YANG Q M, et al. Multi-dimensional decision-making for UAV air combat based on hierarchical reinforcement learning[J]. Acta Armamentarii, 2023, 44(6): 1547-1563. (in Chinese) doi: 10.12382/bgxb.2022.0711
[21]	邢云燕. 美军军事决策过程[J]. 国防科技, 2018, 39(1): 76-80.
	XING Y Y. The U.S. Military decision-making process[J]. National Defense Science and Technology, 2018, 39(1): 76-80. (in Chinese)
[22]	尹奇跃, 赵美静, 倪晚成, 等. 兵棋推演的智能决策技术与挑战[J]. 自动化学报, 2023, 49(5): 913-928.
	YIN Q Y, ZHAO M J, NI W C, et al. Intelligent decision-making techniques and challenges in military chess simulation[J]. Acta Automatica Sinica, 2023, 49(5): 913-928. (in Chinese)
[23]	SANTIAGO O, GABRIEL S, ALBERTO U, et al. A survey of real-time strategy game AI research and competition in StarCraft[J]. IEEE Transactions on Computational Intelligence and AI in Games, 2013, 5(4): 293-311.
[24]	文东日. 深度强化学习在军事领域的应用研究[J]. 军事运筹与评估, 2022, 37(3): 75-80.
	WEN D R. Research on the application of deep reinforcement learning in the military field[J]. Military Operations Research and Assessment, 2022, 37(3): 75-80. (in Chinese)
[25]	李航, 刘代金, 刘禹. 军事智能博弈对抗系统设计框架研究[J]. 火力与指挥控制, 2020, 45(9): 116-121.
	LI H, LIU D J, LIU Y. Research on the design framework of military intelligent game adversarial system[J]. Firepower and Command Control, 2020, 45(9): 116-121. (in Chinese)
[26]	MNIH V, BADIA A P, MIRZA M, et al. Asynchronous methods for deep reinforcement learning[C]// Proceedings of the 33rd International Conference on Machine Learning. New York, NY, US:PMLR, 2016, 48: 1928-1937.
[27]	HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]// Proceedings of the 35th International Conference on Machine Learning. Stockholm, Sweden:PMLR, 2018: 1861-1870.
[28]	FUJIMOTO S, HOOF H, MEGER D. Addressing function approximation error in actor-critic methods[C]// Proceedings of the 35th International Conference on Machine Learning. Stockholm, Sweden:PMLR, 2018: 1587-1596.
[29]	陈彩辉, 姜汉龙. 任务空间概念模型(CMMS)研究[J]. 计算机仿真, 2005(9): 80-84.
	CHEN C H, JIANG H L. Research on the conceptual model of task space (CMMS)[J]. Computer Simulation, 2005(9): 80-84. (in Chinese)
[30]	张博, 康凤举, 苏冰. 一种面向论证仿真的舰艇作战系统任务空间概念模型[J]. 兵工学报, 2015, 36(增刊2): 112-117.
	ZHANG B, KANG F J, SU B. A conceptual model of task space for naval combat system simulation with argumentation[J]. Acta Armamentarii, 2015, 36(S2): 112-117. (in Chinese)
[31]	NACHUM O, GU S X, LEE H, et al. Data-efficient hierarchical reinforcement learning[C]// Proceedings of the 32nd Annual Conferenceon Neural Information Processing Systems. Red Hook, NY, US: Curran Associates Inc., 2018.
[32]	LOWE R, WU Y, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[C]// Proceedings of the 31st Advances in Annual Conference on Neural Information Processing Systems. Long Beach, CA, US: NeurIPS, 2017.

状态类型	状态信息	One-hot编码
	对空拦截	[0,0,0,1]
TT_i	对陆打击	[0,0,1,0]
	空战巡逻	[0,1,0,0]
	反地面战巡逻	[1,0,0,0]
TTM_i	时刻T₁	[0,1]
	时刻T₂	[1,0]
TS_i	未启动	[0,1]
	启动	[1,0]
CT_i	导弹驱逐舰	[0,1]
	轰炸机	[1,0]

状态类型	状态信息	One-hot编码
	对空拦截	[0,0,0,1]
TT_i	对陆打击	[0,0,1,0]
	空战巡逻	[0,1,0,0]
	反地面战巡逻	[1,0,0,0]
TTM_i	时刻T₁	[0,1]
	时刻T₂	[1,0]
TS_i	未启动	[0,1]
	启动	[1,0]
CT_i	导弹驱逐舰	[0,1]
	轰炸机	[1,0]

层级	意义	奖赏值
HL_r	子任务决策正确子任务决策错误	600 -600
LL_r	毁伤次要目标接近目标特定区域	100 50
	自身毁伤超出特定范围	-100 -400

层级	意义	奖赏值
HL_r	子任务决策正确子任务决策错误	600 -600
LL_r	毁伤次要目标接近目标特定区域	100 50
	自身毁伤超出特定范围	-100 -400

推演方	单元名称	单元数量	单元武器
红方	MQ-1C型无人机	1	4×反坦克导弹
	MQ-1B型无人机	2	2×空对空针刺飞弹
蓝方	坦克排(4×T-72主战坦克)	1	80×12.7mm/50高射机炮
	MQ-1B型无人机	2	2×空对空针刺飞弹