针对区域防御的多无人机序列捕捉算法

doi:10.12382/bgxb.2024.0343

摘要/Abstract

摘要：

针对区域防御任务中多个入侵者的拦截问题,考虑追捕任务间时序关系与总体拦截效能,提出一种多无人机序列捕捉算法。基于任务的长期规划收益与短期执行效果构建任务的时序收益与空间收益,分别作为任务分配和任务执行的优化目标,实现复杂博弈问题的动态实时求解。基于可达集方法描述攻防双方优势程度并构建任务时序收益,引入深度Q网络对其进行估计进而引导任务分配;基于任务空间收益求解单攻击者追逃博弈问题,给出连续动作空间任务执行的最优控制策略。仿真结果表明,所提算法通过优化任务时空收益能够实现多无人机间的有效合作,提升防御方的捕获成功率,并具有较强的可扩展性。

关键词: 多无人机, 时空任务收益, 序列捕捉, 时序任务分配, 深度Q网络

Abstract:

For the interception issue of multiple intruders in area defense missions,a multi-UAV sequential capture algorithm is proposed by taking into account the temporal relationship between pursuit tasks and the overall interception effectiveness.The temporal and spatial rewards are constructed based on the long-term planning benefits and short-term execution effects of the tasks,which serve as the optimization objectives for task allocation and execution,respectively,and the dynamic and real-time solutions are achieved for complex game-theoretical problems.A reachability-set-based approach is used to describe the advantage levels of both attackers and defenders,and a deep Q-network is introduced to estimate the temporal rewards for tasks and then guide task allocation.The single attacker pursuit-evasion game problem is solved based on the spatial reward of task,and an optimal control strategy is presented for task execution in a continuous action space.Simulated results show that the peoposed algorithm optimizes the temporal and spatial rewards to facilitate the effective cooperation among multiple UAVs,enhances the capture success rate of the defenders,and has an increased scalability.

Key words: multi-unmanned aerial vehicle, temporal and spatial rewards, sequential capture, sequential task allocation, deep Q-network

中图分类号:

V279

何子琦, 李博宸, 王成罡, 宋磊. 针对区域防御的多无人机序列捕捉算法[J]. 兵工学报, 2025, 46(4): 240343-.

HE Ziqi, LI Bochen, WANG Chenggang, SONG Lei. Multi-UAV Sequential Capture Algorithm for Area Defense[J]. Acta Armamentarii, 2025, 46(4): 240343-.

图/表 20

图1 问题场景示意图

Fig.1 Schematic diagram of a problem scenario

图2 算法框架示意图

Fig.2 Schematic diagram of algorithm framework

图3 R d j=0时攻击者的可达集

Fig.3 The attacker’s reachable set for R d j=0

图4 R d j≠0时攻击者的可达集

Fig.4 The attacker’s reachable set for R d j≠0

图5 追捕任务的空间收益

Fig.5 Spatial reward of pursuit mission

图6 所有可能的最优捕获点

Fig.6 All possible optimal capture points

图7 最优捕获点求解算法

Fig.7 Optimal capture point solving algorithm

图8 DQN算法结构示意图

Fig.8 Schematic diagram of DQN

图9 神经网络结构示意图

Fig.9 Schematic diagram of neural net

图10 序列捕捉算法流程

Fig.10 Flowchart of sequential capture algorithm

表1 算法训练参数

Table 1 Training parameters of the algorithm

参数	数值
γ	0.98
M	12800
N_BatchSize	64
η	0.0003
N_maxEp	40000
N_maxStep	120
ε_max	0.95
Δε	0.001
N_Prestep	4000

图11 训练平均奖励

Fig.11 Average reward curves of training

图12 防守无人机运动轨迹

Fig.12 The trajectories of defenders

表2 不同算法的平均拦截失败数量测试结果

Table 2 Test results of the average number of interception failures for different algorithms

算法	平均拦截失败数量
DQN算法	1.15
DDQN算法	1.13
Dueling DQN算法	1.13
基于最优匹配的任务规划算法	1.51
基于改进粒子群的任务规划算法	2.96

表3 攻击方无人机初始坐标

Table 3 Attacker’s initial coordinates

编号	坐标/km
1	(-3.00,2.00)
2	(-2.75,2.25)
3	(-2.50,2.50)
4	(-2.25,2.75)
5	(1.00,2.50)
6	(-3.50,-4.25)
7	(-3.25,-4.25)
8	(-3.00,-4.25)
9	(-2.75,-4.25)
10	(2.00,-4.00)

表4 防守方无人机初始坐标

Table 4 Defender’s initial coordinates

编号	坐标/km
1	(-1.25,-1.50)
2	(-1.00,-1.50)

图13 基于最优匹配的任务规划算法下无人机运动轨迹

Fig.13 The trajectories of UAVs obtained by task planning algorithm based on optimal matching

图14 基于时空收益的序列捕捉算法下无人机运动轨迹

Fig.14 The trajectories of UAVs obtained by sequential capture algorithm based on temporal and spatial rewards

图15 不同攻击方数量条件下的平均奖励

Fig.15 Average reward curves under different numbers of attackers

表5 攻击方数量不同条件下的平均拦截失败数量

Table 5 Average number of failed interceptions under different numbers of attackers

场景	平均拦截失败数量
2对5	0.26
2对10	1.15
2对15	2.36

参考文献 26

[1]	孔国杰, 冯时, 于会龙, 等. 无人集群系统协同运动规划技术综述[J]. 兵工学报, 2023, 44(1):11-26. doi: 10.12382/bgxb.2022.0930
	KONG G J, FENG S, YU H L, et al. A review on cooperative motion planning of unmanned vehicles[J]. Acta Armamentarii, 2023, 44(1):11-26. (in Chinese) doi: 10.12382/bgxb.2022.0930
[2]	SHISHIKA D, PAULOS J, KUMAR V. Cooperative team strategies for multi-player perimeter-defense games[J]. IEEE Robotics and Automation Letters, 2020, 5(2):2738-2745.
[3]	李军, 陈士超. 无人机蜂群关键技术发展综述[J]. 兵工学报, 2023, 44(9):2533-2545. doi: 10.12382/bgxb.2023.0514
	LI J, CHEN S C. Overview of key technology and its development of drone swarm[J]. Acta Armamentarii, 2023, 44(9):2533-2545. (in Chinese) doi: 10.12382/bgxb.2023.0514
[4]	ISAACS R. Differential games:a mathematical theory with applications to warfare and pursuit,control and optimization[M]. New York,NY,US: Dover Publications, 1999.
[5]	FISAC J F, SASTRY S S. The pursuit-evasion-defense differential game in dynamic constrained environments[C]// Proceedings of the 2015 54th IEEE Conference on Decision and Control.Washington,DC, US: IEEE, 2015:4549-4556.
[6]	GARCIA E, FUCHS Z E, MILUTINOVIC D, et al. A geometric approach for the cooperative two-pursuer one-evader differential game[J]. IFAC-Papers On Line, 2017, 50(1):15209-15214.
[7]	YAN R, DUAN X M, SHI Z Y, et al. Matching-based capture strategies for 3D heterogeneous multiplayer reach-avoid differential games[J]. Automatica, 2022,140:110207.
[8]	SHISHIKA D, KUMAR V. Local-game decomposition for multiplayer perimeter-defense problem[C]// Proceedings of the 2018 IEEE Conference on Decision and Control.Washington,DC, US: IEEE, 2018:2093-2100.
[9]	CHEN M, ZHOU Z Y, TOMLIN C J. Multiplayer reach-avoid games via pairwise outcomes[J]. IEEE Transactions on Automatic Control, 2017, 62(3):1451-1457.
[10]	王磊, 徐超, 李淼, 等. 多飞行器协同任务分配的改进粒子群优化算法[J]. 兵工学报, 2023, 44(8):2224-2232. doi: 10.12382/bgxb.2022.0968
	WANG L, XU C, LI M, et al. Improved particle swarm optimization algorithm for cooperative task assignment of multiple vehicles[J]. Acta Armamentarii, 2023, 44(8):2224-2232. (in Chinese) doi: 10.12382/bgxb.2022.0968
[11]	VELHAL S, SUNDARAM S, SUNDARARAJAN N. A decentralized multirobot spatiotemporal multitask assignment approach for perimeter defense[J]. IEEE Transactions on Robotics, 2022, 38(5):3085-3096.
[12]	BAKOLAS E, TSIOTRAS P. Optimal pursuit of moving targets using dynamic Voronoi diagrams[C]// Proceedings of the 49th IEEE Conference on Decision and Control.Atlanta,GA, US: IEEE, 2010:7431-7436.
[13]	PAN S, HUANG H M, DING J, et al.Pursuit, evasion and defense in the plane[C]// Proceedings of the 2012 American Control Conference.Montreal,QC, Canada:IEEE,2012:4167-4173.
[14]	PIERSON A, WANG Z J, SCHWAGER M. Intercepting rogue robots:an algorithm for capturing mul-tiple evaders with multiple pursuers[J]. IEEE Robotics and Automation Letters, 2017, 2(2):530-537.
[15]	GARCIA E, CASBEER D W, VON MOLL A, et al. Multiple pursuer multiple evader differential games[J]. IEEE Transactions on Automatic Control, 2021, 66(5):2345-2350.
[16]	WANG Y D, DONG L, SUN C Y. Cooperative control for multi-player pursuit-evasion games with reinforcement learning[J]. Neurocomputing, 2020,412:101-114.
[17]	AL-TAMIMI A, LEWIS F L, ABU-KHALAF M. Model-free Q-learning designs for linear discrete-time zero-sum games with application to H-infinity control[J]. Automatica, 2007, 43(3):473-481.
[18]	刘冰雁, 叶雄兵, 高勇, 等. 基于分支深度强化学习的非合作目标追逃博弈策略求解[J]. 航空学报, 2020, 41(10):324040. doi: 10.7527/S1000-6893.2020.24040
	LIU B Y, YE X B, GAO Y, et al. Strategy solution of non-cooperative target pursuit-evasion game based on branching deep reinforcement learning[J]. Acta Argonautica et Astronautica Sinica, 2020, 41(10):324040. (in Chinese)
[19]	SILVEIRA J, CABRAL K, RABBATH C A, et al. Deep reinforcement learning solution of reach-avoid games with superior evader in the context of unmanned aerial systems[C]// Proceedings of the 2023 International Conference on Unmanned Aircraft Systems.Warsaw, Poland:IEEE, 2023:911-918.
[20]	RASLAN H, SCHWARTZ H, GIVIGI S. A learning invader for the “Guarding a Territory” game[J]. Journal of Intelligent & Robotic Systems, 2016, 83(1):55-70.
[21]	PAULOS J, CHEN S W, SHISHIKA D, et al. Decentralization of multiagent policies by learning what to communicate[C]// Proceedings of the 2019 International Conference on Robotics and Automation.Washington,DC, US: IEEE, 2019:7990-7996.
[22]	LIU J W, OUYANG Z K, YANG J H, et al. Coordinated defense allocation in reach-avoid scenarios with efficient online optimization:arXiv:2304.00234[R/OL]. Ithaca,NY, US: Cornell University, 2023(2023-06-03). https://arxiv.org/abs/2304.00234.
[23]	张嘉玲, 姜永超, 李博宸, 等. 面向反水雷的海空协同时间最优雷区覆盖规划方法[J]. 指挥与控制学报, 2023, 9(4):469-477.
	ZHANG J L, JIANG Y C, LI B C, et al. Time-optimal minefield coverage planning method for anti-mine-oriented sea-air coordination[J]. Journal of Command and Control, 2023, 9(4):469-477. (in Chinese)
[24]	ZHOU Z Y, ZHANG W, DING J, et al. Cooperative pursuit with voronoi partitions[J]. Automatica, 2016,72:64-72.
[25]	DOROTHY M, MAITY D, SHISHIKA D, et al. One apollonius circle is enough for many pursuit-evasion games[J]. Automatica, 2024,163:111587.
[26]	MACHARET D G, CHEN A K, SHISHIKA D, et al. Adaptive partitioning for coordinated multi-agent perimeter defense[C]// Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems.Washington,DC, US: IEEE, 2020:7971-7977.