TAN C S , MOHD-MOKHTAR R , ARSHAD M R . A comprehensive review of coverage path planning in robotics using classical and heuristic algorithms [J ] . IEEE Access , 2021 , 9 : 119310 - 119342 . DOI: 10.1109/ACCESS.2021.3108177 http://doi.org/10.1109/ACCESS.2021.3108177 https://ieeexplore.ieee.org/document/9523743/ https://ieeexplore.ieee.org/document/9523743/

李波 , 杨志鹏 , 贾卓然 , 等 . 一种无监督学习型神经网络的无人机全区域侦察路径规划 [J ] . 西北工业大学学报 , 2021 , 39 ( 1 ): 77 - 84 .

LI B , YANG Z P , JIA Z R , et al. An unsupervised learning neural network for planning UAV full-area reconnaissance path [J ] . Journal of Northwestern Polytechnical University , 2021 , 39 ( 1 ): 77 - 84 . (in Chinese) DOI: 10.1051/jnwpu/20213910077 http://doi.org/10.1051/jnwpu/20213910077 https://www.jnwpu.org/10.1051/jnwpu/20213910077 https://www.jnwpu.org/10.1051/jnwpu/20213910077 To plan a UAV's full-area reconnaissance path under uncertain information conditions, an unsupervised learning neural network based on the genetic algorithm is proposed. Firstly, the environment model, the UAV model and evaluation indexes are presented, and the neural network model for planning the UAV's full-area reconnaissance path is established. Because it is difficult to obtain the training samples for planning the UAV's full-area reconnaissance path, the genetic algorithm is used to optimize the unsupervised learning neural network parameters. Compared with the traditional methods, the evaluation indexes constructed in this paper do not need to specify UAV maneuver rules. The offline learning method proposed in the paper has excellent transfer performances. The simulation results show that the UAV based on the unsupervised learning neural network can plan effective full-area reconnaissance paths in the unknown environments and complete full-area reconnaissance missions.

吴文超 , 黄长强 , 宋磊 , 等 . 不确定环境下的多无人机协同搜索航路规划 [J ] . 兵工学报 , 2011 , 32 ( 11 ): 1337 - 1342 . 为解决多架无人机(UAV)在不确定环境中搜索目标的问题，根据多UAV协同搜索的基本原则，建立了多UAV协同搜索的环境模型和UAV运动模型，提出了一种满足UAV机动限制和适应数据通讯延迟的协同路径决策算法。根据先验知识将环境分为未知环境，已知环境和禁飞区，设计了搜索回报函数，引导UAV对未知环境进行搜索，对已知环境进行规避；提出了禁飞区回避决策，实现了UAV对禁飞区的完全回避。最后进行了仿真实验，并与2种非协同搜索算法的仿真结果进行了比较，结果表明，本文提出的协同搜索算法的有效性。

WU W C , HUANG C Q , SONG L , et al. Cooperative search and path planning of multi-unmanned air vehicles in uncertain environment [J ] . Acta Armamentarii , 2011 , 32 ( 11 ): 1337 - 1342 . (in Chinese)

SANTOS L C , SANTOS F N , PIRES E J S , et al. Path planning for ground robots in agriculture: A short review [C ] // Proceedings of the 2020 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC). Ponta Delgada , Portugal : IEEE , 2020 : 61 - 66 .

GHODKE V , MADAKE J . Navigational path-planning for all-terrain autonomous agricultural robot [J ] . arXiv preprint, [2021-09-05] . https://doi.org/10.48550/arXiv.2109.02015 https://doi.org/10.48550/arXiv.2109.02015 . https://doi.org/10.48550/arXiv.2109.02015 https://doi.org/10.48550/arXiv.2109.02015

DE CARVALHO R N , VIDAL H A , VIEIRA P , et al. Complete coverage path planning and guidance for cleaning robots [C ] // Proceeding of the IEEE International Symposium on Industrial Electronics.Kyoto , Japan : IEEE , 1997 : 677 - 682 .

HASAN K M , REZA K J . Path planning algorithm development for autonomous vacuum cleaner robots [C ] // Proceedings of the 2014 International Conference on Informatics, Electronics & Vision.Dhaka , Bangladesh : IEEE , 2014 : 1 - 6 .

ROTTMANN N , DENZ R , BRUDER R , et al. A probabilistic approach for complete coverage path planning with low-cost systems [C ] // Proceedings of the 2021 European Conference on Mobile Robots.Bonn , Germany : IEEE , 2021 : 1 - 8 .

CHANG S J , DAN B J . Free movimg pattern’s online spanning tree coverage algorithm [C ] // Proceedings of the 2006 SICE-ICASE International Joint Conference.Busan , Korea : IEEE , 2006 : 2935 - 2938 .

LUO H C , LIN H F , ZHU T , et al. Complete coverage path planning of UUV for marine mine countermeasure using grid division and spanning tree [C ] // Proceedings of the 2019 Chinese Control and Decision Conference. Nanchang , China : IEEE , 2019 : 5016 - 5021 .

DOGRU S , MARQUES L . A * -based solution to the coverage path planning problem [C ] // Proceedings of the Iberian Robotics Conference 2017.Seville, Spain:Springer International Publishing , 2017 : 240 - 248 .

李御驰 , 闫军涛 , 宋志华 , 等 . 基于遗传算法的无人机监视覆盖航路规划算法研究 [J ] . 计算机科学与应用 , 2019 , 9 ( 6 ): 1208 - 1215 .

LI Y C , YAN J T , SONG Z H , et al. Research on algorithm of UAV monitoring coverage path planning based on genetic algorithm [J ] . Computer Science and Application , 2019 , 9 ( 6 ): 1208 - 1215 . (in Chinese) DOI: 10.12677/CSA.2019.96135 http://doi.org/10.12677/CSA.2019.96135 https://www.HansPub.org/journal/doi.aspx?DOI=10.12677/CSA.2019.96135 https://www.HansPub.org/journal/doi.aspx?DOI=10.12677/CSA.2019.96135

PIARDI L , LIMA J , PEREIRA A I , et al. Coverage path planning optimization based on Q-learning algorithm [C ] // Proceedings of the 16th International Conference of Numerical Analysis and Applied Mathematics. Rhodes , Greece : AIP Conference Proceedings , 2019, 2133 : 220002 .

HEYDARI J , SAHA O , GANAPATHY V . Reinforcement learning-based coverage path planning with implicit cellular decomposition [J ] . arXiv preprint, [2021-10-18] . https://doi.org/10.48550/arXiv.2110.09018 https://doi.org/10.48550/arXiv.2110.09018 . https://doi.org/10.48550/arXiv.2110.09018 https://doi.org/10.48550/arXiv.2110.09018

BIALAS J , DOLLER M . Coverage path planning for unmanned aerial vehicles in complex 3D environments with deep reinforcement learning [C ] // Proceedings of the 2022 IEEE International Conference on Robotics and Biomimetics.Xishuangbanna , China : IEEE , 2022 : 1080 - 1085 .

张伟 , 王乃新 , 魏世琳 , 等 . 水下无人潜航器集群发展现状及关键技术综述 [J ] . 哈尔滨工程大学学报 , 2020 , 41 ( 2 ): 289 - 297 .

ZHANG W , WANG N X , WEI S L , et al. Overview of unmanned underwater vehicle swarm development status and key technologies [J ] . Journal of Harbin Engineering University , 2020 , 41 ( 2 ): 289 - 297 . (in Chinese)

罗志远 , 刘小峰 , 陈俊风 , 等 . 一种基于分步遗传算法的多无人清洁车区域覆盖路径规划方法 [J ] . 电子测量与仪器学报 , 2020 , 34 ( 8 ): 43 - 50 .

LUO Z Y , LIU X F , CHEN J F , et al. Method of area coverage path planning of multi-unmanned cleaning vehicles based on step by step genetic algorithm [J ] . Journal of Electronic Measurement and Instrumentation , 2020 , 34 ( 8 ): 43 - 50 . (in Chinese)

SANNA G , GODIO S , GUGLIERI G . Neural network based algorithm for multi-UAV coverage path planning [C ] // Proceedings of the 2021 International Conference on Unmanned Aircraft Systems (ICUAS). Athens , Greece : IEEE , 2021 : 1210 - 1217 .

LI W H , ZHAO T , DIAN S Y . Multirobot coverage path planning based on deep Q-network in unknown environment [J ] . Journal of Robotics , 2022 ( 2 ): 1 - 15 . DOI: 10.1155/2022/6825902 https://dx.doi.org/10.1155/2022/6825902 .

SUNEHAG P , LEVER G , GRUSLYS A , et al. Value-decomposition networks for cooperative multi-agent learning [J ] . arXiv preprint, [2017-06-16] . https://doi.org/10.48550/arXiv.1706.05296 https://doi.org/10.48550/arXiv.1706.05296 . https://doi.org/10.48550/arXiv.1706.05296 https://doi.org/10.48550/arXiv.1706.05296

王雪松 , 王荣荣 , 程玉虎 . 安全强化学习综述 [J ] . 自动化学报 , 2023 , 49 ( 9 ): 1813 - 1835 .

WANG X S , WANG R R , CHENG Y H . Safe reinforcement learning: a survey [J ] . Acta Automatica Sinica , 2023 , 49 ( 9 ): 1813 - 1835 . (in Chinese)

MATIGNON L , LAURENT G J , LE FORT-PIAT N . Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems [J ] . The Knowledge Engineering Review , 2012 , 27 ( 1 ): 1 - 31 . DOI: 10.1017/S0269888912000057 http://doi.org/10.1017/S0269888912000057 https://www.cambridge.org/core/product/identifier/S0269888912000057/type/journal_article https://www.cambridge.org/core/product/identifier/S0269888912000057/type/journal_article In the framework of fully cooperative multi-agent systems, independent (non-communicative) agents that learn by reinforcement must overcome several difficulties to manage to coordinate. This paper identifies several challenges responsible for the non-coordination of independent agents: Pareto-selection, non-stationarity, stochasticity, alter-exploration and shadowed equilibria. A selection of multi-agent domains is classified according to those challenges: matrix games, Boutilier's coordination game, predators pursuit domains and a special multi-state game. Moreover, the performance of a range of algorithms for independent reinforcement learners is evaluated empirically. Those algorithms are Q-learning variants: decentralized Q-learning, distributed Q-learning, hysteretic Q-learning, recursive frequency maximum Q-value and win-or-learn fast policy hill climbing. An overview of the learning algorithms’ strengths and weaknesses against each challenge concludes the paper and can serve as a basis for choosing the appropriate algorithm for a new domain. Furthermore, the distilled challenges may assist in the design of new learning algorithms that overcome these problems and achieve higher performance in multi-agent applications.

ZAREMBA W , SUTSKEVER I , VINYALS O . Recurrent neural network regularization [J ] . arXiv preprint, [2015-02-19] . https://doi.org/10.48550/arXiv.1409.2329 https://doi.org/10.48550/arXiv.1409.2329 . https://doi.org/10.48550/arXiv.1409.2329 https://doi.org/10.48550/arXiv.1409.2329

HU J L , WELLMAN M P . Nash Q-learning for general sum stochasticgames [J ] . Journal of Machine Learning Research , 2003 , 4 : 1039 - 1069 .