A Mobile Robot Path Planning Algorithm Based on Ant Colony Optimization Guide Deep Q-Networks

doi:10.12382/bgxb.2025.0134

Abstract

Abstract:

To address the issues of slow convergence and poor path planning associated with the Deep Q-Network (DQN) algorithm for mobile robot path planning in large-scale complex unknown environments,a path planning algorithm combining Ant Colony Optimization (ACO) and DQN,termed ACOG-DQN,is proposed.Initially,the pheromone mechanism of ACO is introduced to facilitate the selection of potential paths with the goal of reaching the destination,thereby reducing the number of ineffective environmental explorations and determining the optimal path.Concurrently,the previous path selection experiences are filtered using a threshold to form a sample set for training the Q-network,which is then utilized to determine the optimal path for the mobile robot in the current environment.Finally,a path selection mechanism is designed where the weight of the Q-network’s optimal path increases over time,using the optimal paths determined by ACO and the Q-network,as well as those determined by random exploration,as candidates.This mechanism selects the current action,aiming to achieve a path that is ultimately decided entirely by the Q-network.Simulation and physical experiments conducted in three different complex environments demonstrate that the proposed ACOG-DQN algorithm exhibits superior performance in terms of convergence speed,path quality,and algorithm stability compared to the DQN algorithm,thereby validating the effectiveness of the proposed algorithm.

Key words: mobile robot, path planning, Deep Q-network, ant colony optimization algorithm, reinforcement learning, algorithm optimization

CLC Number:

TP183

LI Hailiang, LI Zonggang, NING Xiaogang, DU Yajiang. A Mobile Robot Path Planning Algorithm Based on Ant Colony Optimization Guide Deep Q-Networks[J]. Acta Armamentarii, 2025, 46(11): 250134-.

Figures/Tables 21

Fig.1 Grid map

Fig.2 The architecture of the DQN

Fig.3 The pheromone model of the ACO algorithm

Fig.4 Action selection

Fig.5 The variation process of the exploration rate

Fig.6 Pseudocode of the Action Selection Function

Fig.7 The architecture of the ACOG-DQN algorithm

Table 1 Hyperparameter settings

超参数名称	数值
经验池	100000
折扣因子	0.9
ε衰减系数	0.9998
最初ε值	0.8
最小ε值	0
初始样本数量	256
网络更新频率	200
学习率	0.1
经验筛选比例	1/3

Fig.8 Path comparison in a small-scale environment of different algorithms

Fig.9 Path comparison in a big-scale environment of different algorithms

Fig.10 Path comparison in a special corridor terrain of different algorithms

Table 2 Comparison of path lengths in different algorithms

算法	环境类型
算法	8×8	16×16	16×16回廊
DQN	10	19	未收敛
ACOG-DQN	10	15	25
AG-DQN	10	17	27
APF-DQN	10	17	29

Fig.11 Comparison chart of loss value changes during training of the ACOG-DQN algorithm and three other algorithms in small-scale static map

Fig.12 Comparison chart of loss value changes during training of the ACOG-DQN algorithm and three other algorithms in small-scale dynamic map

Fig.13 Comparison chart of loss value changes during training of the ACOG-DQN algorithm and three other algorithms in big-scale static map

Fig.14 Comparison chart of loss value changes during training of the ACOG-DQN algorithm and three other algorithms in big-scale dynamic map

Fig.15 Comparison chart of loss value changes during training of the ACOG-DQN algorithm and three other algorithms in specialized corridor map

Fig.16 Comparison of the convergence speeds of four algorithms in different environment maps

Fig.17 Hardware and software conditions

Fig.18 Presents the outcomes of global path planning within rviz

Fig.19 The keyframe fusion map of the robot movement process

References 24

[1]	LIU L X, WANG X, YANG X, et al. Path planning techniques for mobile robots:review and prospect[J]. Expert Systems with Applications, 2023, 227:120254. doi: 10.1016/j.eswa.2023.120254 URL
[2]	FENG Z, ZHOU L, HONG Q S. DBVS-APF-RRT:a global path planning algorithm with ultra-high speed generation of initial paths and high optimal path quality[J]. Expert Systems with Application, 2024, 249:123571. doi: 10.1016/j.eswa.2024.123571 URL
[3]	TOMÁS LOZANO-PÉREZ, WESLEY M A. An algorithm for planning collision free paths among polyhedral obstacles[J]. Communications of the ACM, 1979, 22(10):560-570. doi: 10.1145/359156.359164 URL
[4]	XIE Y, CHENG W S. AGV path planning based on smoothing A^* algorithm[J]. International Journal of Software Engineering & Applications, 2015, 6(5):1-8.
[5]	栾添添, 王皓, 孙明晓, 等. 基于动态变采样区域RRT的无人车路径规划[J]. 控制与决策, 2023, 38(6):1721-1729.
	LUAN T T, WANG H, SUN M X, et al. Path planning of unmanned vehicle based on dynamic variable sampling area RRT[J]. Control and Decision, 2023, 38(6):1721-1729. (in Chinese)
[6]	赵鹏程, 宋保维, 毛昭勇, 等. 基于改进的复合自适应遗传算法的UUV水下回收路径规划[J]. 兵工学报, 2022, 43(10):2598-2608.
	ZHAO P C, SONG B W, MAO Z Y, et al. Path planning for UUV underwater recovery based on improved composite adaptive genetic algorithm[J]. Acta Armamentarii, 2022, 43(10):2598-2608. (in Chinese) doi: 10.12382/bgxb.2021.0474
[7]	WANG Y H Z. Ant colony optimization for traveling salesman problem based on parameters optimization[J]. Applied Soft Computing, 2021, 107(000):107439. doi: 10.1016/j.asoc.2021.107439 URL
[8]	潘云伟, 李敏, 曾祥光, 等. 基于人工势场和改进强化学习的自主式水下潜航器避障和航迹规划[J]. 兵工学报, 2025, 46(4):72-83.
	PAN Y W, LI M, ZENG X G, et al. AUV obstacle avoidance and path planning based on artificial potential field and improved reinforcement learning[J]. Acta Armamentarii, 2025, 46(4):72-83. (in Chinese)
[9]	XI M, YANG J C, WEN J B, et al. Comprehensive ocean information-enabled AUV motion planning based on reinforcement learning[J]. Remote Sens, 2023, 15:3077. doi: 10.3390/rs15123077 URL
[10]	VAN OTTERLO M, WIERING M, WIERING M A, et al. Markov decision processes:concepts and algorithms[M]. Beilin,Germany: Springer, 2012.
[11]	杨尚志, 张刚, 陈跃华, 等. 基于改进分层DQN算法的智能体路径规划[J]. 计算机应用与软件, 2024, 41(5):226-232,239.
	YANG S Z, ZHANG G, CHEN Y H, et al. Path planning for agent based on improved layered DQN algorithms[J]. Computer Applications and Software, 2024, 41(5):226-232,239. (in Chinese)
[12]	倪培龙, 毛鹏军, 王宁, 等. 基于改进A-DDQN算法的机器人路径规划[J]. 系统仿真学报, 2025, 37(9):2420-2430. doi: 10.16182/j.issn1004731x.joss.24-0369
	NI P L, MAO P J, WANG N, et al. Robot path planning based on improved A-DDQN algorithm[J]. Journal of System Simulation, 2025, 37(9):2420-2430. (in Chinese) doi: 10.16182/j.issn1004731x.joss.24-0369
[13]	KONG F C, WANG Q, GAO S, et al. B-APFDQN:a UAV path planning algorithm based on deep Q-network and artificial potential field[J]. IEEE Access, 2023, 11:44051-44064. doi: 10.1109/ACCESS.2023.3273164 URL
[14]	田箫源, 董秀成. 基于改进DQN的移动机器人避障路径规划[J]. 中国惯性技术学报, 2024, 32(4):406-416.
	TIAN X Y, DONG X C. Obstacle avoidance path planning of mobile robot based on improved DQN[J]. Journal of Chinese Inertial Technology, 2024, 32(4):406-416. (in Chinese)
[15]	李宗刚, 韩森, 陈引娟, 等. 基于角度搜索和深度Q网络的移动机器人路径规划算法[J]. 兵工学报, 2025, 46(2):32-46.
	LI Z G, HAN S, CHEN Y J, et al. Mobile robots path planning algorithm based on angle searching and Deep Q-Network[J]. Acta Armamentarii, 2025, 46(2):32-46. (in Chinese)
[16]	GU Y W, ZHU Z T, LÜ J D, et al. DM-DQN:dueling munchausen deep Q network for robot path planning[J]. Complex & Intelligent Systems, 2022, 9(4):4287-4300.
[17]	王小康, 冀杰, 刘洋, 等. 基于改进Q学习算法的无人物流配送车路径规划[J]. 系统仿真学报, 2024, 36(5):1211-1221. doi: 10.16182/j.issn1004731x.joss.23-0051
	WANG X K, JI J, LIU Y, et al. Path planning of unmanned delivery vehicle based on improved Q-learning algorithm[J]. Journal of System Simulation, 2024, 36(5):1211-1221. (in Chinese) doi: 10.16182/j.issn1004731x.joss.23-0051
[18]	王霄龙, 陈洋, 胡棉, 等. 基于改进深度Q网络的机器人持续监测路径规划[J]. 兵工学报, 2024, 45(6):1813-1823. doi: 10.12382/bgxb.2023.0227
	WANG X L, CHEN Y, HU M, et al. Robot path planning for persistent monitoring based on improved Deep Q networks[J]. Acta Armamentarii, 2024, 45(6):1813-1823. (in Chinese) doi: 10.12382/bgxb.2023.0227
[19]	黄岩松, 姚锡凡, 景轩, 等. 基于深度Q网络的多起点多终点AGV路径规划[J]. 计算机集成制造系统, 2023, 29(8):2550-2562.
	HUANG Y S, YAO X F, JING X, et al. DQN-based AGV path planning for situations with multi-starts and multi-targets[J]. Computer Integrated Manufacturing Systems, 2023, 29(8):2550-2562. (in Chinese)
[20]	夏雨奇, 黄炎焱, 陈恰. 基于深度Q网络的无人车侦察路径规划[J]. 系统工程与电子技术, 2024, 46(9):3070-3081. doi: 10.12305/j.issn.1001-506X.2024.09.19
	XIA Y Q, HUANG Y Y, CHEN Q. Path planning for unmanned vehicle reconnaissance based on deep Q-network[J]. Systems Engineering and Electronics, 2024, 46(9):3070-3081. (in Chinese) doi: 10.12305/j.issn.1001-506X.2024.09.19
[21]	周治国, 余思雨, 于家宝, 等. 面向无人艇的T-DQN智能避障算法研究[J]. 自动化学报, 2023, 49(8):1645-1655.
	ZHOU Z G, YU S Y, YU J B, et al. Research on T-DQN intelligent obstacle avoidance algorithm of unmanned surface vehicle[J]. Acta Automatica Sinica, 2023, 49(8):1645-1655. (in Chinese)
[22]	田晓航, 霍鑫, 周典乐, 等. 基于蚁群信息素辅助的Q学习路径规划算法[J]. 控制与决策, 2023, 38(12):3345-3353.
	TIAN X H, HUO X, ZHOU D L, et al. Ant colony pheromone aided Q-learning path planning algorithm[J]. Control and Decision, 2023, 38(12):3345-3353. (in Chinese)
[23]	LUO L, ZHAO N, ZHU Y, et al. A guiding DQN algorithm for automated guided vehicle pathfinding problem of robotic mobile fulfillment systems[J]. Computers & Industrial Engineering, 2023, 178:109112. doi: 10.1016/j.cie.2023.109112 URL
[24]	YAN L, CHANG S, WANG X X, et al. A dual-stage coverage path planning method for bathymetric survey using an AUV in graph-based SLAM framework considering positioning uncertainty[J]. Ocean Engineering, 2024, 312(Part3):21.