李超 , 王瑞星 , 黄建忠 , 等 . 稀疏奖励下基于强化学习的无人集群自主决策与智能协同 [J ] . 兵工学报 , 2023 , 44 ( 6 ): 1537 - 1546 . DOI: 10.12382/bgxb.2022.0177 http://doi.org/10.12382/bgxb.2022.0177 无人集群将深刻地塑造战争样式,为提升无人集群自主决策算法能力,对异构无人集群攻防对抗自主决策方法进行研究。对无人集群对抗模型设计进行总体概述,并对无人集群攻防对抗场景进行模型设计;针对无人集群自主决策采用强化学习技术广泛存在的稀疏奖励问题,提出基于局部回报重塑的奖励机制设定方法;在此基础上叠加优先经验回放,有效地改善稀疏奖励问题;通过程序仿真和演示系统设计,验证该方法的优越性。该方法的研究将加速基于强化学习技术的无人集群自主决策算法网络收敛过程,对无人集群自主决策算法研究具有重要意义。

LI C , WANG R X , HUANG J Z , et al . Autonomous decision-making and intelligent collaboration of UAV swarms based on reinforcement learning with sparse rewards [J ] . Acta Armamentarii , 2023 , 44 ( 6 ): 1537 - 1546 . (in Chinese) DOI: 10.12382/bgxb.2022.0177 http://doi.org/10.12382/bgxb.2022.0177 UAV swarms will profoundly shape the pattern of warfare. In order to improve the autonomous decision-making algorithm capability of UAV swarms, the autonomous decision-making method for heterogeneous UAV swarm attack-defense confrontation scenarios is studied. An overview of the design of the UAV swarm confrontation model and the model design of the UAV swarm attack-defense confrontation scenario are carried out. To solve the sparse reward problem which widely exists in the reinforcement learning technology in the autonomous decision-making of the UAV swarm, a reward mechanism setting method based on local reward reshaping is proposed. And then, the prioritized experience replay is superimposed, which effectively improves the sparse reward problem. Finally, the superiority of this method is verified by simulation and demonstration system design. This study will accelerate the network convergence process of the autonomous decision-making algorithm for UAV swarms based on reinforcement learning technology, which is of great significance to the research on autonomous decision-making algorithms of UAV swarms.

吕震华 , 高亢 . 美国无人集群城市作战应用发展综述 [J ] . 中国电子科学研究院学报 , 2020 , 15 ( 8 ): 738 - 745 .

LÜ Z H , GAO K . Review of the development of drone swarm urban combat applications in the USA [J ] . Journal of China Academy of Electronics and Information Technology , 2020 , 15 ( 8 ): 738 - 745 . (in Chinese)

胡鹏林 , 赵春晖 , 胡劲文 , 等 . 拒止环境无人机集群协同感知与自主控制 [C ] // 第40届中国控制会议论文集(15) . 上海 : CNKI , 2021 : 728 - 733 .

HU P L , ZHAO C H , HU J W , et al . Cooperative sensing and autonomous control of UAV swarm in denied environmentt [C ] // Proceedings of the 40th Chinese Control Conference . Shanghai, China : CNKI , 2021 : 728 - 733 . (in Chinese)

孙立健 , 周鋆 , 朱承 , 等 . 马赛克战兵力设计下的边缘指挥与控制组织结构 [J ] . 指挥与控制学报 , 2022 , 8 ( 2 ): 141 - 149 .

SUN L J , ZHOU Y , ZHU C , et al . Organizational structure of edge C2 under force design of mosaic warfare [J ] . Journal of Command And Control , 2022 , 8 ( 2 ): 141 - 149 . (in Chinese)

DARIO S , ALESSANDRO V , PEKKA K , et al . Mobile-edge computing architecture: the role of MEC in the internet of things [J ] . IEEE Consumer Electronics Magazine , 2016 , 5 ( 4 ): 84 - 91 . DOI: 10.1109/MCE.2016.2590118 http://doi.org/10.1109/MCE.2016.2590118 https://ieeexplore.ieee.org/document/7574435/ https://ieeexplore.ieee.org/document/7574435/

陈霄 , 王潋 , 刘巍 , 等 . 美军机动边缘信息服务能力现状概述 [J ] . 电光与控制 , 2021 , 28 ( 7 ): 62 - 67 .

CHEN X , WANG L , LIU W , et al . Overview of the status quo of U.S. military mobile edge information service capability [J ] . Electronics Optics and Control , 2021 , 28 ( 7 ): 62 - 67 . (in Chinese)

陈霄 , 刘巍 , 夏淋淋 , 等 . 边缘计算军事应用需求及作战运用构想 [J ] . 火力与指挥控制 , 2021 , 46 ( 8 ): 1 - 4 .

CHEN X , LIU W , XIA L L , et al . Military application requirements and operational conception of edge computing [J ] . Fire Control & Command Control , 2021 , 46 ( 8 ): 1 - 4 . (in Chinese)

薛建强 , 史彦军 , 李波 . 面向无人集群的边缘计算技术综述 [J ] . 兵工学报 , 2023 , 44 ( 9 ): 2546 - 2555 . DOI: 10.12382/bgxb.2022.1209 http://doi.org/10.12382/bgxb.2022.1209 未来智能化战争无人集群作战中,计算力的云边端供给成为重要模式,其边缘计算技术作为关键使能技术,能解决作战边缘任务执行实时性差、带宽受限、数据安全等问题。阐明无人集群背景下边缘计算的概念和技术内涵,给出一个面向战术边缘的云-边-端分布式系统框架,以实现无人集群作战的信息互联互通、战场局部和全局态势感知、群智能决策和协同控制;对该框架涉及的关键技术,包括边缘计算框架、边云协同、计算卸载、边缘指挥控制等关键技术展开综述;展望和总结了无人集群作战下的边缘计算技术,为未来智能化战争边缘战术提供了参考。

XUE J Q , SHI Y J , LI B . Overview of edge computing technology for unmanned cluster [J ] . Acta Armamentarii , 2023 , 44 ( 9 ): 2546 - 2555 . (in Chinese)

王万斌 . 面向战术智能终端任务的移动边缘计算卸载策略研究 [D ] . 成都 : 电子科技大学 , 2022 .

WANG W B . Research on mobile edge computing offloading strategy for tactical intelligent terminal task [D ] . Chengdu : University of Electronic Science and Technology of China , 2022 . (in Chinese)

ZHOU J J , SU Z , XU Q C , et al . Cooperative content offloading scheme in air-ocean integrated networks [J ] . Peer-to-Peer Networking and Applications , 2021 , 14 ( 5 ): 3388 - 3404 . DOI: 10.1007/s12083-021-01160-z http://doi.org/10.1007/s12083-021-01160-z

LI K , NI W , YUAN X , et al . Deep-graph-based reinforcement learning for joint cruise control and task offloading for aerial edge internet of things(EdgeIoT) [J ] . IEEE Internet of Things Journal , 2022 , 9 ( 21 ): 21676 - 21686 . DOI: 10.1109/JIOT.2022.3182119 http://doi.org/10.1109/JIOT.2022.3182119 https://ieeexplore.ieee.org/document/9793853/ https://ieeexplore.ieee.org/document/9793853/

缪家辉 , 郑镐 , 谢正昊 , 等 . 数字孪生辅助UAV网络计算卸载优化 [J/OL ] . 北京邮电大学学报 , 2022 , 45 ( 6 ): 133 - 139 . DOI: 10.13190/j.jbupt.2022-181 https://dx.doi.org/10.13190/j.jbupt.2022-181 .

MIAO J H , ZHENG H , XIE Z H , et al . Offloading optimization in digital twin-aided UAV networks [J/OL ] . Journal of Beijing University of Posts and Telecommunications , 2022 , 45 ( 6 ): 133 - 139 . DOI: 10.13190/j.jbupt.2022-181 https://dx.doi.org/10.13190/j.jbupt.2022-181 . (in Chinese)

ZHOU S H , FEI S H , FENG Y Z . Deep reinforcement learning based UAV-assisted maritime network computation offloading strategy [C ] // Proceedings of 2022 IEEE/CIC International Conference on Communications in China . Foshan, China : IEEE , 2022 : 890 - 895 .

RAZA S , LIU W , AHMED M , et al . An efficient task offloading scheme in vehicular edge computing [J ] . Journal of Cloud Computing , 2020 , 9 : 1 - 14 .

赵晓焱 , 韩威 , 张俊娜 , 等 . 基于异步深度强化学习的车联网协作卸载策略 [J/OL ] . 计算机应用 . 2023 : 1 - 11 ,DOI: 10.11772/j.issn.1001-9081.2023050788 https://dx.doi.org/10.11772/j.issn.1001-9081.2023050788 .

ZHAO X Y , HAN W , ZHANG J N , et al . Collaborative offloading mechanism in internet of vehicles based on asynchronous deep reinforcement learning [J ] . Journal of Computer Applications , 2023 : 1 - 11 .doi: 10.11772/j.issn.1001-9081.2023050788 https://dx.doi.org/10.11772/j.issn.1001-9081.2023050788 . (in Chinese)

TAN K G , FENG L , DÁN G , et al . Decentralized convex optimization for joint task offloading and resource allocation of vehicular edge computing systems [J ] . IEEE Transactions on Vehicular Technology , 2022 , 71 ( 12 ): 13226 - 13241 . DOI: 10.1109/TVT.2022.3197627 http://doi.org/10.1109/TVT.2022.3197627 https://ieeexplore.ieee.org/document/9852713/ https://ieeexplore.ieee.org/document/9852713/

刘晓宇 , 许驰 , 曾鹏 , 等 . 面向异构工业任务高并发计算卸载的深度强化学习算法 [J ] . 计算机学报 , 2021 , 44 ( 12 ): 2367 - 2381 .

LIU X Y , XU C , ZENG P , et al . Deep reinforcement learning-based high concurrent computing offloading for heterogeneous industrial tasks [J ] . Chinese Journal of Computers , 2021 , 44 ( 12 ): 2367 - 2381 . (in Chinese)

ARDI T , TAMBET M , DORIAN K , et al . Multiagent cooperation and competition with deep reinforcement learning [J ] . PloS One , 2017 , 12 ( 4 ): e0172395 . DOI: 10.1371/journal.pone.0172395 http://doi.org/10.1371/journal.pone.0172395 https://dx.plos.org/10.1371/journal.pone.0172395 https://dx.plos.org/10.1371/journal.pone.0172395

PETER S , GUY L , AUDRUNAS G , et al . Value-decomposition networks for cooperative multi-agent learning:arXiv:1706.05296 [R ] . Ithaca,NY,US:Cornell University , 2017 :1706.05296.

TABISH R , MIKAYEL S , CHRISTIAN S D W , et al . Monotonic value function factorisation for deep multi-agent reinforcement learning [J ] . The Journal of Machine Learning Research , 2020 , 21 ( 1 ): 7234 - 7284 .

LOWE R , WU Y , TAMAR A , et al . Multi-agent actor-critic for mixed cooperative-competitive environments: arXiv:1706.02275 [R ] . Ithaca,NY,US:Cornell University , 2017 :706.02275.

ZHU S C , GUI L , ZHAO D M , et al . Learning-based computation offloading approaches in UAVs-assisted edge computing [J ] . IEEE Transactions on Vehicular Technology , 2021 , 70 ( 1 ): 928 - 944 . DOI: 10.1109/TVT.25 http://doi.org/10.1109/TVT.25 https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=25 https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=25

HE H , REN T , QIU Y , et al . Multi-agent computation offloading in UAV assisted MEC via deep reinforcement learning [C ] // Proceedings of Smart Computing and Communication. New York, NY , US : Springer , 2022 : 416 - 426 .

LI Y L , LIANG L , FU J L , et al . Multiagent reinforcementlearning for task offloading of space/aerial-assisted edgecomputing [J ] . Security and Communication Networks , 2022 , 2022 : 4193365 .17.

CHENG Z P , LIANG M H , CHEN N , et al . Deep reinforcement learning-based joint task and energy offloading in UAV-aided 6G intelligent edge networks [J ] . Computer Communications , 2022 , 192 : 234 - 244 . DOI: 10.1016/j.comcom.2022.06.017 http://doi.org/10.1016/j.comcom.2022.06.017 https://linkinghub.elsevier.com/retrieve/pii/S0140366422002195 https://linkinghub.elsevier.com/retrieve/pii/S0140366422002195

苏维亚 , 徐飞 , 王森 . 基于改进MADDPG的UAV轨迹和计算卸载联合优化算法 [J/OL ] . 计算机系统应用 , 2023 , 32 ( 11 ). DOI: 10.15888/j.cnki.csa.009277 https://dx.doi.org/10.15888/j.cnki.csa.009277 .

SU W Y , XU F , WANG S . Joint optimization algorithm for UAV trajectory and computational offloading based on improved MADDPG [J ] . Computer Systems & Applications , 2023 , 32 ( 11 ). DOI: 10.15888/j.cnki.csa.009277 https://dx.doi.org/10.15888/j.cnki.csa.009277 . (in Chinese)

李慧 . 基于强化学习的无人机用户自适应边缘计算卸载策略研究 [D ] . 深圳 : 哈尔滨工业大学(深圳) , 2021 .

LI H . Research on UAV user adaptive edge computing offloading based on reinforcement learning [D ] . Shenzhen : Harbin Institute of Technology, Shenzhen , 2021 . (in Chinese)

XUE J B , WU Q Q , ZHANG H J . Cost optimization of UAV-MEC network calculation offloading: a multi-agent reinforcement learning method [J ] . Ad Hoc Networks , 2022 , 136 : 102981 . DOI: 10.1016/j.adhoc.2022.102981 http://doi.org/10.1016/j.adhoc.2022.102981 https://linkinghub.elsevier.com/retrieve/pii/S1570870522001548 https://linkinghub.elsevier.com/retrieve/pii/S1570870522001548

DAI Z J , ZHANG Y , ZHANG W C , et al . A multi-agent collaborative environment learning method for UAV deployment and resource allocation [J ] . IEEE Transactions on Signal and Information Processing over Networks , 2022 , 8 : 120 - 130 . DOI: 10.1109/TSIPN.2022.3150911 http://doi.org/10.1109/TSIPN.2022.3150911 https://ieeexplore.ieee.org/document/9712375/ https://ieeexplore.ieee.org/document/9712375/

ZHANG H B , LIU X Y , BIAN X , et al . A resource allocation scheme for real-time energy-aware offloading in vehicular networks with MEC [J ] . Wireless Communications and Mobile Computing , 2022 , 2022 ( 10 ): 1 - 17 .

栗志 . 基于MEC的计算卸载及资源分配算法研究 [D ] . 南京 : 南京邮电大学 , 2021 .

LI Z . Research on computing offloading and resource allocation algorithm based on MEC [D ] . Nanjing : Nanjing University of Posts and Telecommunications , 2021 . (in Chinese)

ZHU B T , BEDEER E , NGUYEN H H , et al . UAV trajectory planning in wireless sensor networksfor energy consumption minimization by deep reinforcement learning [J ] . IEEE Transactions on Vehicular Technology , 2021 , 70 ( 9 ): 9540 - 9554 . DOI: 10.1109/TVT.2021.3102161 http://doi.org/10.1109/TVT.2021.3102161 https://ieeexplore.ieee.org/document/9507262/ https://ieeexplore.ieee.org/document/9507262/

杨力 , 马伟东 , 郭江宇 , 等 . 陆战场中的计算卸载和资源分配 [J ] . 火力与指挥控制 , 2023 , 48 ( 4 ): 17 - 23 ,31.

YANG L , MA W D , GUO J Y , et al . Computation offloading and resource allocation in the land battlefield [J ] . Fire Control & Command Control , 2023 , 48 ( 4 ): 17 - 23 ,31. (in Chinese)