基于改进深度Q网络的机器人持续监测路径规划

doi:10.12382/bgxb.2023.0227

摘要/Abstract

摘要：

持续监测问题指的是通过规划移动机器人在路网中的巡逻路线,从而对路网环境实施长期监测,以实现保障环境安全的目的。环境中的待监测点通常受到最大允许监测周期(重访周期)的限制,并且最优的监测路径不应具有固定的周期,否则监测过程容易被恶意入侵者针对性地破坏。针对上述问题,提出一种基于改进深度Q网络(Deep Q Networks, DQN)的机器人监测路径规划算法。改进DQN的决策方法,使机器人获得一条监测频率高、安全性好(防止被智能入侵的能力)、非固定周期的监测路径。仿真实验结果表明:所提算法可以高效地覆盖所有待监测节点;与传统的DQN算法相比,该算法不会使监测陷入周期性的循环路径之中,增强了系统的抗入侵能力。

关键词: 持续监测, 强化学习, 路径规划, 移动机器人

Abstract:

Persistent monitoring refers to the long-term monitoring of road network environment by planning the patrol route of mobile robots in the road network, so as to achieve the purpose of ensuring environmental safety. The sites to be monitored in the environment are usually limited by the maximum allowable monitoring period (revisit period). A fixed monitoring period should not be set for an optimal monitoring path, otherwise, the monitoring process is easy to be destroyed by malicious intruders. To solve the above problems, a robot monitoring path planning algorithm based on improved Deep Q Networks (DQN) is proposed, the decision-making method of DQN is improved, and a monitoring path with high monitoring frequency, good security (ability to prevent intelligent intrusion) and non-fixed period is planned for robot. Simulated and experimental results show that the proposed algorithm can efficiently cover all nodes to be monitored. Compared with the traditional DQN algorithm, the proposed algorithm does not make the monitoring fall into the cyclic path, and enhances the anti-intrusion ability of the persistent monitoring system.

Key words: persistent monitoring, reinforcement learning, path planning, mobile robot

中图分类号:

TP24

王霄龙, 陈洋, 胡棉, 李旭东. 基于改进深度Q网络的机器人持续监测路径规划[J]. 兵工学报, 2024, 45(6): 1813-1823.

WANG Xiaolong, CHEN Yang, HU Mian, LI Xudong. Robot Path Planning for Persistent Monitoring Based on Improved Deep Q Networks[J]. Acta Armamentarii, 2024, 45(6): 1813-1823.

图/表 13

图1 路网结构

Fig.1 Structure of road network

图2 决策方法

Fig.2 Decision-making methods

图3 数据编码与网络结构

Fig.3 Data coding and network structure

图4 传统DQN算法轨迹

Fig.4 Trajectory of traditional DQN algorithm

图5 改进DQN算法轨迹

Fig.5 Trajectory of the improved DQN algorithm

表1 3种算法的数据对比

Table 1 Data comparison of three algorithms

算法	平均剩余寿命/min	监测延迟熵
改进DQN算法	689.2	44.52
传统DQN算法	699	14.29
TSP算法	693.0	11.75

图6 平均剩余寿命对比

Fig.6 Comparison of average remaining life

图7 最小剩余寿命对比

Fig.7 Comparison of minimum remaining life

图8 复杂路网

Fig.8 Complex road network

表2 复杂路网的数据对比

Table 2 Data comparison of complex road network

决策周期	传统DQN算法			改进DQN算法		监测延迟熵上界
决策周期	平均剩余寿命/min		监测延迟熵	平均剩余寿命/min	监测延迟熵	监测延迟熵上界
2000	1807.2	104.8		1666.2	111.0	133.6
4000	1806.5	93.2		1682.2	124.7	156.6
6000	1806.2	85.0		1682.5	128.9	169.8
8000	1806.1	79.6		1682.5	134.6	179.4
10000	1806.0	76.0		1681.2	133.2	186.7
12000	1805.9	73.5		1678.9	141.9	192.8
14000	1805.9	71.6		1681.2	141.4	197.9
16000	1805.9	70.0		1679.4	141.9	202.3
18000	1805.9	68.8		1680.6	139.7	206.2
20000	1805.9	67.8		1680.6	143.2	209.7

图9 重访周期散点图

Fig.9 Scatter plot of revisit period

图10 实物试验场景

Fig.10 Experimental scene

图11 实验的真实轨迹

Fig.11 Real trajectories of experiment

参考文献 21

[1]	PINTO S C, ANDERSSON S B, HENDRICKX J M, et al. Optimal periodic multi-agent persistent monitoring of a finite set of targets with uncertain states[C]∥Proceedings of 2020 American Control Conference. Denver, CO, US:IEEE,2020:5207-5212
[2]	HARI S, RATHINAM S, DARBHA S, et al. Optimal UAV route planning for persistent monitoring missions[J]. IEEE Transactions on Robotics, 2021, 37(2):550-566.
[3]	MAINI P, YU K, SUJIT P B, et al. Persistent monitoring with refueling on a terrain using a team of aerial and ground robots[C]∥Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems. Madrid, Spain:IEEE, 2018:8493-8498.
[4]	MAINI P, TOKEKAR P, SUJIT P B. Visibility-based persistent monitoring of piecewise linear features on a terrain using multiple aerial and ground robots[J]. IEEE Transactions on Automation Science and Engineering, 2021, 18(4):1692-1704.
[5]	LI Q, GAMA F, RIBEIRO A, et al. Graph neural networks for decentralized path planning[C]∥Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems. Las Vegas, NV, US: IEEE, 2020: 1901-1903.
[6]	TOLSTAYA E, GAMA F, PAULOS J, et al. Learning decentralized controllers for robot swarms with graph neural networks[C]∥Proceedings of Conference on Robot Learning, Osaka, Japan: Proceedings of Machine Learning Research, 2019:521-531.
[7]	LEW T, SHARMA A, HARRISON J, et al. Safe active dynamics learning and control: a sequential exploration-exploitation framework[J]. IEEE Transactions on Robotics, 2022, 38(5): 2888-2907.
[8]	ZHANG Z, WANG X H, ZHANG Q R, et al. Multi-robot cooperative pursuit via potential field-enhanced reinforcement learning[C]∥Proceedings of 2022 International Conference on Robotics and Automation. Philadelphia, PA, US: IEEE, 2022: 8808-8814.
[9]	ASARKAYA A S, AKSARAY D, AND YAZICIOĞLU Y, et al. Temporal-logic-constrained hybrid reinforcement learning to perform optimal aerial monitoring with delivery drones[C]∥Proceedings of 2021 International Conference on Unmanned Aircraft Systems. Athens, Greece: IEEE, 2021: 285-294.
[10]	MD KABA, UZUNBAS M G, LIM S N. A reinforcement learning approach to the view planning problem[C]∥Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, US: IEEE, 2017:5094-5102.
[11]	LIWEI A N, GUANG-HONG, et al. Opacity enforcement for confidential robust control in linear cyber-physical systems[J]. IEEE Transactions on Automatic Control, 2019, 65(3):1234-1241.
[12]	WELIKALA S, CASSANDRAS C G. Greedy initialization for distributed persistent monitoring in network systems[J]. Automatica, 2021, 134:109943.
[13]	WELIKALA S, CASSANDRAS C G. Asymptotic analysis for greedy initialization of threshold-based distributed optimization of persistent monitoring on graphs[J]. IFAC-PapersOnLine, 2020, 53(2):3433-3438.
[14]	ZHOU N, CASSANDRAS C G, YU X, et al. Optimal threshold-based distributed control policies for persistent monitoring on graphs[C]∥Proceedings of 2019 American Control Conference. Philadelphia, PA, US: IEEE, 2019: 2030-2035.
[15]	PINTO S C, ANDERSSON S B, HENDRICKX J M, et al. Multiagent persistent monitoring of targets with uncertain states[J]. IEEE Transactions on Automatic Control, 2022, 67(8): 3997-4012.
[16]	ZHOU N, CASSANDRAS C G, YU X, et al. The price of decentralization: event-driven optimization for multi-agent persistent monitoring tasks[J]. IEEE Transactions on Control of Network Systems, 2021: 8(2):976-986.
[17]	WANG Y W, WEI Y W, LIU X K, et al. Optimal persistent monitoring using second-order agents with physical constraints[J]. IEEE Transactions on Automatic Control, 2019:64(8): 3239-3252.
[18]	REZAZADEH N, KIA S S. A sub-modular receding horizon solution for mobile multi-agent persistent monitoring[J]. Automatica, 2021, 127(1): 109460.
[19]	李凯文, 张涛, 王锐, 等. 基于深度强化学习的组合优化研究进展[J]. 自动化学报, 2021, 47(11): 2521-2537
	LI K W, ZHANG T, WANG R, et al. Research progress on combinatorial optimization based on deep reinforcement learning[J]. Acta Automatica Sinica, 2021, 47(11): 2521-2537. (in Chinese)
[20]	SHU Y, CHEN Y, HU M, et al. UAV Path planning based on simultaneous optimization of monitoring frequency and security[C]∥Proceedings of the 2022 34rd Chinese Control and Decision Conference. Hefei, China: IEEE, 2022: 3808-3814.
[21]	MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518: 529-533.