Multi-UAV Sequential Capture Algorithm for Area Defense

doi:10.12382/bgxb.2024.0343

Abstract

Abstract:

For the interception issue of multiple intruders in area defense missions,a multi-UAV sequential capture algorithm is proposed by taking into account the temporal relationship between pursuit tasks and the overall interception effectiveness.The temporal and spatial rewards are constructed based on the long-term planning benefits and short-term execution effects of the tasks,which serve as the optimization objectives for task allocation and execution,respectively,and the dynamic and real-time solutions are achieved for complex game-theoretical problems.A reachability-set-based approach is used to describe the advantage levels of both attackers and defenders,and a deep Q-network is introduced to estimate the temporal rewards for tasks and then guide task allocation.The single attacker pursuit-evasion game problem is solved based on the spatial reward of task,and an optimal control strategy is presented for task execution in a continuous action space.Simulated results show that the peoposed algorithm optimizes the temporal and spatial rewards to facilitate the effective cooperation among multiple UAVs,enhances the capture success rate of the defenders,and has an increased scalability.

Key words: multi-unmanned aerial vehicle, temporal and spatial rewards, sequential capture, sequential task allocation, deep Q-network

CLC Number:

V279

HE Ziqi, LI Bochen, WANG Chenggang, SONG Lei. Multi-UAV Sequential Capture Algorithm for Area Defense[J]. Acta Armamentarii, 2025, 46(4): 240343-.

Figures/Tables 20

Fig.1 Schematic diagram of a problem scenario

Fig.2 Schematic diagram of algorithm framework

Fig.3 The attacker’s reachable set for R d j=0

Fig.4 The attacker’s reachable set for R d j≠0

Fig.5 Spatial reward of pursuit mission

Fig.6 All possible optimal capture points

Fig.7 Optimal capture point solving algorithm

Fig.8 Schematic diagram of DQN

Fig.9 Schematic diagram of neural net

Fig.10 Flowchart of sequential capture algorithm

Table 1 Training parameters of the algorithm

参数	数值
γ	0.98
M	12800
N_BatchSize	64
η	0.0003
N_maxEp	40000
N_maxStep	120
ε_max	0.95
Δε	0.001
N_Prestep	4000

Fig.11 Average reward curves of training

Fig.12 The trajectories of defenders

Table 2 Test results of the average number of interception failures for different algorithms

算法	平均拦截失败数量
DQN算法	1.15
DDQN算法	1.13
Dueling DQN算法	1.13
基于最优匹配的任务规划算法	1.51
基于改进粒子群的任务规划算法	2.96

Table 3 Attacker’s initial coordinates

编号	坐标/km
1	(-3.00,2.00)
2	(-2.75,2.25)
3	(-2.50,2.50)
4	(-2.25,2.75)
5	(1.00,2.50)
6	(-3.50,-4.25)
7	(-3.25,-4.25)
8	(-3.00,-4.25)
9	(-2.75,-4.25)
10	(2.00,-4.00)

Table 4 Defender’s initial coordinates

编号	坐标/km
1	(-1.25,-1.50)
2	(-1.00,-1.50)

Fig.13 The trajectories of UAVs obtained by task planning algorithm based on optimal matching

Fig.14 The trajectories of UAVs obtained by sequential capture algorithm based on temporal and spatial rewards

Fig.15 Average reward curves under different numbers of attackers

Table 5 Average number of failed interceptions under different numbers of attackers

场景	平均拦截失败数量
2对5	0.26
2对10	1.15
2对15	2.36

References 26

[1]	孔国杰, 冯时, 于会龙, 等. 无人集群系统协同运动规划技术综述[J]. 兵工学报, 2023, 44(1):11-26. doi: 10.12382/bgxb.2022.0930
	KONG G J, FENG S, YU H L, et al. A review on cooperative motion planning of unmanned vehicles[J]. Acta Armamentarii, 2023, 44(1):11-26. (in Chinese) doi: 10.12382/bgxb.2022.0930
[2]	SHISHIKA D, PAULOS J, KUMAR V. Cooperative team strategies for multi-player perimeter-defense games[J]. IEEE Robotics and Automation Letters, 2020, 5(2):2738-2745.
[3]	李军, 陈士超. 无人机蜂群关键技术发展综述[J]. 兵工学报, 2023, 44(9):2533-2545. doi: 10.12382/bgxb.2023.0514
	LI J, CHEN S C. Overview of key technology and its development of drone swarm[J]. Acta Armamentarii, 2023, 44(9):2533-2545. (in Chinese) doi: 10.12382/bgxb.2023.0514
[4]	ISAACS R. Differential games:a mathematical theory with applications to warfare and pursuit,control and optimization[M]. New York,NY,US: Dover Publications, 1999.
[5]	FISAC J F, SASTRY S S. The pursuit-evasion-defense differential game in dynamic constrained environments[C]// Proceedings of the 2015 54th IEEE Conference on Decision and Control.Washington,DC, US: IEEE, 2015:4549-4556.
[6]	GARCIA E, FUCHS Z E, MILUTINOVIC D, et al. A geometric approach for the cooperative two-pursuer one-evader differential game[J]. IFAC-Papers On Line, 2017, 50(1):15209-15214.
[7]	YAN R, DUAN X M, SHI Z Y, et al. Matching-based capture strategies for 3D heterogeneous multiplayer reach-avoid differential games[J]. Automatica, 2022,140:110207.
[8]	SHISHIKA D, KUMAR V. Local-game decomposition for multiplayer perimeter-defense problem[C]// Proceedings of the 2018 IEEE Conference on Decision and Control.Washington,DC, US: IEEE, 2018:2093-2100.
[9]	CHEN M, ZHOU Z Y, TOMLIN C J. Multiplayer reach-avoid games via pairwise outcomes[J]. IEEE Transactions on Automatic Control, 2017, 62(3):1451-1457.
[10]	王磊, 徐超, 李淼, 等. 多飞行器协同任务分配的改进粒子群优化算法[J]. 兵工学报, 2023, 44(8):2224-2232. doi: 10.12382/bgxb.2022.0968
	WANG L, XU C, LI M, et al. Improved particle swarm optimization algorithm for cooperative task assignment of multiple vehicles[J]. Acta Armamentarii, 2023, 44(8):2224-2232. (in Chinese) doi: 10.12382/bgxb.2022.0968
[11]	VELHAL S, SUNDARAM S, SUNDARARAJAN N. A decentralized multirobot spatiotemporal multitask assignment approach for perimeter defense[J]. IEEE Transactions on Robotics, 2022, 38(5):3085-3096.
[12]	BAKOLAS E, TSIOTRAS P. Optimal pursuit of moving targets using dynamic Voronoi diagrams[C]// Proceedings of the 49th IEEE Conference on Decision and Control.Atlanta,GA, US: IEEE, 2010:7431-7436.
[13]	PAN S, HUANG H M, DING J, et al.Pursuit, evasion and defense in the plane[C]// Proceedings of the 2012 American Control Conference.Montreal,QC, Canada:IEEE,2012:4167-4173.
[14]	PIERSON A, WANG Z J, SCHWAGER M. Intercepting rogue robots:an algorithm for capturing mul-tiple evaders with multiple pursuers[J]. IEEE Robotics and Automation Letters, 2017, 2(2):530-537.
[15]	GARCIA E, CASBEER D W, VON MOLL A, et al. Multiple pursuer multiple evader differential games[J]. IEEE Transactions on Automatic Control, 2021, 66(5):2345-2350.
[16]	WANG Y D, DONG L, SUN C Y. Cooperative control for multi-player pursuit-evasion games with reinforcement learning[J]. Neurocomputing, 2020,412:101-114.
[17]	AL-TAMIMI A, LEWIS F L, ABU-KHALAF M. Model-free Q-learning designs for linear discrete-time zero-sum games with application to H-infinity control[J]. Automatica, 2007, 43(3):473-481.
[18]	刘冰雁, 叶雄兵, 高勇, 等. 基于分支深度强化学习的非合作目标追逃博弈策略求解[J]. 航空学报, 2020, 41(10):324040. doi: 10.7527/S1000-6893.2020.24040
	LIU B Y, YE X B, GAO Y, et al. Strategy solution of non-cooperative target pursuit-evasion game based on branching deep reinforcement learning[J]. Acta Argonautica et Astronautica Sinica, 2020, 41(10):324040. (in Chinese)
[19]	SILVEIRA J, CABRAL K, RABBATH C A, et al. Deep reinforcement learning solution of reach-avoid games with superior evader in the context of unmanned aerial systems[C]// Proceedings of the 2023 International Conference on Unmanned Aircraft Systems.Warsaw, Poland:IEEE, 2023:911-918.
[20]	RASLAN H, SCHWARTZ H, GIVIGI S. A learning invader for the “Guarding a Territory” game[J]. Journal of Intelligent & Robotic Systems, 2016, 83(1):55-70.
[21]	PAULOS J, CHEN S W, SHISHIKA D, et al. Decentralization of multiagent policies by learning what to communicate[C]// Proceedings of the 2019 International Conference on Robotics and Automation.Washington,DC, US: IEEE, 2019:7990-7996.
[22]	LIU J W, OUYANG Z K, YANG J H, et al. Coordinated defense allocation in reach-avoid scenarios with efficient online optimization:arXiv:2304.00234[R/OL]. Ithaca,NY, US: Cornell University, 2023(2023-06-03). https://arxiv.org/abs/2304.00234.
[23]	张嘉玲, 姜永超, 李博宸, 等. 面向反水雷的海空协同时间最优雷区覆盖规划方法[J]. 指挥与控制学报, 2023, 9(4):469-477.
	ZHANG J L, JIANG Y C, LI B C, et al. Time-optimal minefield coverage planning method for anti-mine-oriented sea-air coordination[J]. Journal of Command and Control, 2023, 9(4):469-477. (in Chinese)
[24]	ZHOU Z Y, ZHANG W, DING J, et al. Cooperative pursuit with voronoi partitions[J]. Automatica, 2016,72:64-72.
[25]	DOROTHY M, MAITY D, SHISHIKA D, et al. One apollonius circle is enough for many pursuit-evasion games[J]. Automatica, 2024,163:111587.
[26]	MACHARET D G, CHEN A K, SHISHIKA D, et al. Adaptive partitioning for coordinated multi-agent perimeter defense[C]// Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems.Washington,DC, US: IEEE, 2020:7971-7977.