基于深度强化学习的落角和视场角约束制导律

doi:10.12382/bgxb.2024.0435

摘要/Abstract

摘要：

为满足日益复杂的作战需求,提升微型制导弹药在近距离下的制导性能,基于深度强化学习(Deep Reinforcement Learning,DRL)提出一种考虑视场角极限的落角约束制导律。推导导弹相对移动目标的落角误差估计公式,以落角误差和视角为状态量并构造分段奖励函数,将制导问题建模为时间离散的马尔科夫决策过程。通过偏置比例导引获得所需制导指令,并由DRL的策略网络输出其偏置项,通过近端策略优化算法对网络进行训练,得到最优制导策略,实现在无弹目距离信息下对视角和落角的约束。在不同视场角限制、期望落角、目标速度、初始位置和导弹速度下进行数值模拟和蒙特卡洛仿真,并对导弹在不同速度下的捕获区域进行对比分析。研究结果表明,所提制导律在不同初始条件下均能保持良好的制导性能,在近距离打击中相比现有制导律具有更大的捕获区域,在干扰作用下具有更小的落角误差分布,从而验证了该制导律的有效性与优越性。

关键词: 制导律, 落角约束, 深度强化学习, 有限视场角, 微型制导弹药

Abstract:

To meet the increasingly complex operational demands and improve the guidance performance of micro-guided munitions at close range,a impact angle constraint guidance law based on deep reinforcement learning (DRL) is proposed by considering the field-of-view (FOV) limitations.The estimation formula for the impact angle error of missile relative to a moving target is derived,the impact angle error and view angle are used as state variables,and a piecewise reward function is constructed.The guidance problem is modeled as a time-discrete Markov decision process (MDP).The required guidance commands are obtained through biased proportional navigation,and a bias term is output by the DRL policy network.The network is trained using the proximal policy optimization (PPO) algorithm,resulting in an optimal guidance strategy that ensures the constraints on both view and impact angles without information about missile-target distance.Numerical and Monte Carlo simulations under different initial conditions and the comparative analyses of capture regions of missiles at various speeds are conducted.The results show that the proposed guidance law maintains excellent guidance performance under various initial conditions,provides a larger capture region in close-range engagements compared to existing guidance laws,and demonstrates a smaller impact angle error distribution under interference,thus validating its effectiveness and superiority.

Key words: guidance law, impact angle constraint, deep reinforcement learning, limited field of view angle, micro-guided munition

中图分类号:

TJ765.3

先苏杰, 王康, 曾鑫, 宋杰, 吴志林. 基于深度强化学习的落角和视场角约束制导律[J]. 兵工学报, 2025, 46(4): 240435-.

XIAN Sujie, WANG Kang, ZENG Xin, SONG Jie, WU Zhilin. An Impact Angle and Field of View Constraints Guidance Law Based on Deep Reinforcement Learning[J]. Acta Armamentarii, 2025, 46(4): 240435-.

图/表 17

图1 交战几何

Fig.1 Engagement geometry

图2 vT估计的仿真结果

Fig.2 The estimated result of vT

图3 制导律训练算法的结构示意图

Fig.3 Structural diagram of guidance law training algorithm

图4 视场角约束下基于PPO的落角控制制导律训练算法

Fig.4 PPO-based guidance law training algorithm for impact angle control with FOV angle constraints

表1 神经网络架构

Table 1 Neural network architecture

神经网络	结构	激活函数
Actor	(2,64)	ReLU
	(64,128)	ReLU
	(128,64)	ReLU
	(64,1),(64,1)	Tanh,Sigmoid
Critic	(2,64)	ReLU
	(64,256)	ReLU
	(256,64)	ReLU
	(64,32)	ReLU
	(32,16)	ReLU
	(16,1)	Linner

表2 参数设置

Table 2 Parameter settings

参数	取值
N	3
v_M/(m·s^-1)	100
v_T/(m·s^-1)	10
a^bn/(m· $s - 2$ )	30
$e n θ$	0.5e_θ₀
$σ n M$ /(°)	10
q₀/(°)	0
σ_M0/(°)	15
θ_M0/(°)	15
n	32
α_ϑ	0.00001
α_φ	0.0001
k₁	0.25
γ	0.95
λ_GAE	0.95
ε	0.2
K	10
dt/s	0.01
Δt/s	0.02
训练回合总数	2000

表2 参数设置

Table 2 Parameter settings

参数	取值
N	3
v_M/(m·s^-1)	100
v_T/(m·s^-1)	10
a^bn/(m· $s - 2$ )	30
$e n θ$	0.5e_θ₀
$σ n M$ /(°)	10
q₀/(°)	0
σ_M0/(°)	15
θ_M0/(°)	15
n	32
α_ϑ	0.00001
α_φ	0.0001
k₁	0.25
γ	0.95
λ_GAE	0.95
ε	0.2
K	10
dt/s	0.01
Δt/s	0.02
训练回合总数	2000

图5 不同σlim下的数值仿真

Fig.5 Numerically simulated results for variousσlim

表3 不同σlim下的仿真结果

Table 3 Simulated results under differentσlimconditions

σ_lim/(°)	σ_sat/(°)	终端θ_M/(°)	最大σ_M/°	$∫ 0 t f a M$ dt
15	14.0	-29.98	15.00	78.52
30	28.5	-29.84	28.18	120.34
45	43.5	-30.00	41.86	163.13
无限制		-30.45	61.01	174.77

表3 不同σlim下的仿真结果

Table 3 Simulated results under differentσlimconditions

σ_lim/(°)	σ_sat/(°)	终端θ_M/(°)	最大σ_M/°	$∫ 0 t f a M$ dt
15	14.0	-29.98	15.00	78.52
30	28.5	-29.84	28.18	120.34
45	43.5	-30.00	41.86	163.13
无限制		-30.45	61.01	174.77

图6 不同θexp下的数值仿真

Fig.6 Numerically simulated results for various θexp

表4 不同θexp下的仿真结果

Table 4 Simulation results under different θexp

θ_exp/(°)	终端θ_M/(°)	最大σ_M/(°)	$∫ 0 t f a M$ dt
-30	-29.84	28.18	120.34
-45	-44.87	28.42	147.76
-60	-59.91	28.60	174.48
-75	-74.53	28.69	200.26

表4 不同θexp下的仿真结果

Table 4 Simulation results under different θexp

θ_exp/(°)	终端θ_M/(°)	最大σ_M/(°)	$∫ 0 t f a M$ dt
-30	-29.84	28.18	120.34
-45	-44.87	28.42	147.76
-60	-59.91	28.60	174.48
-75	-74.53	28.69	200.26

图7 不同vT下的数值仿真

Fig.7 Numerically simulated results for variousvT

图8 RLIACG、IACCG与D-RLIACG的捕获区域

Fig.8 The capture regions of RLIACG,IACCG and D-RLIACG

表5 不同vM下RLIACG与D-RLIACG、 IACCG的捕获率

Table 5 Capture rates of RLIACG,D-RLIACG and IACCG for variousvM %

v_M/(m·s^-1)	RLIACG	IACCG	D-RLIACG
100	88.89	68.15	65.56
150	71.85	38.15	57.78
200	47.04	21.11	39.63

图9 RLIACG与IACCG的蒙特卡洛仿真对比

Fig.9 Comparison of Monte Carlo simulations of RLIACG and IACCG

表6 RLIACG与IACCG、D-RLIACG的误差分布

Table 6 Error distributions of RLIACG,IACCG and D-RLIACG

统计量	脱靶量/m			落角误差/(°)
统计量	RLIACG	IACCG	D-RLIACG	RLIACG	IACCG	D-RLIACG
均值	0.24	0.26	1.15	0.22	0.91	0.47
标准差	0.19	0.19	0.26	0.14	0.76	0.44
最大值	0.75	0.90	1.90	0.77	5.91	4.30

表7 动力学模型与控制器的相关参数

Table 7 Related parameters of dynamic models and controllers

参数	数值
M_α/s^-2	-423.6
M_φ/s^-1	-0.07
M_δ/s^-2	331
Z_α/(m·s^-2·rad^-1)	-705
Z_δ/(m·s^-2·rad^-1)	-165
K_A/(rad·s·m^-1)	4.5×10^-4
K_DC	21.05
K_R/s	0.34
ω_i/(rad·s^-1)	16.36
C_x0	0.18
$C x α 2$ /(rad^-2)	7.65
S_m/m²	0.013
τ_δ/s	0.04

表7 动力学模型与控制器的相关参数

Table 7 Related parameters of dynamic models and controllers

参数	数值
M_α/s^-2	-423.6
M_φ/s^-1	-0.07
M_δ/s^-2	331
Z_α/(m·s^-2·rad^-1)	-705
Z_δ/(m·s^-2·rad^-1)	-165
K_A/(rad·s·m^-1)	4.5×10^-4
K_DC	21.05
K_R/s	0.34
ω_i/(rad·s^-1)	16.36
C_x0	0.18
$C x α 2$ /(rad^-2)	7.65
S_m/m²	0.013
τ_δ/s	0.04

图10 动力学仿真结果

Fig.10 Dynamic simulated results

参考文献 27

[1]	黄景帅, 张洪波, 汤国建, 等. 拦截大气层内机动目标的自适应积分滑模制导律[J]. 宇航学报, 2019, 40(1):51-60.
	HUANG J S, ZHANG H B, TANG G J, et al. Adaptive integral sliding-mode guidance law for intercepting endo-atmospheric maneuvering targets[J]. Journal of Astronautics, 2019, 40(1):51-60. (in Chinese)
[2]	SHASHI R K, SACHIT R, DEBASISH G. Nonsingular terminal sliding mode guidance with impact angle constraints[J]. Journal of Guidance,Control,and Dynamics, 2014, 37(4):1114-1130.
[3]	盛永智, 甘佳豪, 张成新. 弹道可调的落角约束分数阶滑模制导律设计[J]. 航空学报, 2023, 44(7):177-190.
	SHENG Y Z, GAN J H, ZHANG C X. Fractional order sliding mode guidance law design with trajectory adjustable and terminal angular constraint[J]. Acta Aeronauticaet Astronautica Sinica, 2023, 44(7):177-190. (in Chinese)
[4]	KORAY S, OEZGOEREN M. Control of impact angle using biased proportional navigation[C]// Proceedings of AIAA Guidance,Navigation,and Control Conference.Boston, US:AIAA,2013:5348-5362.
[5]	马帅, 王旭刚, 王中原, 等. 带初始前置角和末端攻击角约束的偏置比例导引律设计以及剩余飞行时间估计[J]. 兵工学报, 2019, 40(1):68-78. doi: 10.3969/j.issn.1000-1093.2019.01.009
	MA S, WANG X G, WANG Z Y, et al. BPNG law with arbitrary initial lead angle and terminal impact angle constraint and time-to-go estimation[J]. Acta Armamentarii, 2019, 40(1):68-78. (in Chinese) doi: 10.3969/j.issn.1000-1093.2019.01.009
[6]	王晓海, 孟秀云, 周峰, 等. 基于偏置比例导引的落角约束滑模制导律[J]. 系统工程与电子技术, 2021, 43(5):1295-1302. doi: 10.12305/j.issn.1001-506X.2021.05.17
	WANG X H, MENG X Y, ZHOU F, et al. Sliding mode guidance law with impact angle constraint based on bias proportional navigation[J]. Systems Engineering and Electronics, 2021, 43(5):1295-1302. (in Chinese) doi: 10.12305/j.issn.1001-506X.2021.05.17
[7]	严鹏辉, 刘刚, 缪前树. 基于落角约束的偏置比例导引律的研究[J]. 现代防御技术, 2021, 49(6):43-48,55. doi: 10.3969/j.issn.1009-086x.2021.06.008
	YAN P H, LIU G, MIAO Q S. Research on bias proportional navigation guidance law based on terminal impact angle constraint[J]. Modern Defence Technology, 2021, 49(6):43-48,55. (in Chinese)
[8]	KIM J H, PARK S S, PARK K K, et al. Quaternion based three-dimensional impact angle control guidance law[J]. IEEE Transactions on Aerospace and Electronic Systems, 2021, 57(4):2311-2323.
[9]	黄嘉, 常思江, 陈琦, 等. 不依赖剩余飞行时间的数据驱动攻击时间控制导引律[J]. 兵工学报, 2023, 44(8):2299-2309. doi: 10.12382/bgxb.2022.0324
	HUANG J, CHANG S J, CHEN Q, et al. Data-driven-based impact time control guidance law independent of time-to-go[J]. Acta Armamentarii, 2023, 44(8):2299-2309. (in Chinese) doi: 10.12382/bgxb.2022.0324
[10]	黄嘉, 常思江. 基于数据驱动的攻击时间和攻击角度控制导引律[J]. 系统工程与电子技术, 2022, 44(10):3213-3220. doi: 10.12305/j.issn.1001-506X.2022.10.26
	HUANG J, CHANG S J. Data-driven method based impact time and impact angle control guidance law[J]. Systems Engineering and Electronics, 2022, 44(10):3213-3220. (in Chinese) doi: 10.12305/j.issn.1001-506X.2022.10.26
[11]	YANG H, WU H J, BAI X B, et al. A new united proportional navigation guidance for impact angle constraint without measurement distance between vehicle and target[J/OL]. International Journal of Aerospace Engineering, 2022(2022-10-30). https://doi.org/10.1155/2022/6495746.
[12]	PARK B G, KWON H H, KIM Y H, et al. Composite guidance scheme for impact angle control against a non maneuvering moving target[J]. Journal of Guidance,Control,and Dynamics, 2016, 39(5):1132-1139.
[13]	CHO D, KIM S H. Direct impact angle control guidance for passive homing missiles[J]. Proceedings of the Institution of Mechanical Engineers,Part G:Journal of Aerospace Engineering, 2019, 234(15):2139-2152.
[14]	CHO N, KIM J, LEE S, et al. Generalized analysis of biased proportional navigation guidance with fractional power error feedback[J]. Journal of Guidance,Control,and Dynamics, 2022, 45(9):1598-1613.
[15]	王江, 刘经纬, 崔晓曦, 等. 有限视场下的攻击时间和角度多约束制导律[J]. 北京理工大学学报, 2024, 44(1):18-27.
	WANG J, LIU J W, CUI X X, et al. Multi-Constraint guidance law for impact time and angle with limited field-of-view[J]. Journal of Beijing Institute of Technology, 2024, 44(1):18-27. (in Chinese)
[16]	FAN J F, WANG G, CHEN Y, et al. Impact-angle-constraint look angle guidance for miniature missile[C]// Proceedings of the 2016 16th International Conference on Control,Automation and Systems.Gyeongju, South Korea: IEEE,2016:1005-1009.
[17]	李彤, 孟志鹏, 吕良, 等. 微型导弹纵向扰动抑制控制系统设计[J]. 国防科技大学学报, 2021, 43(1):7-15.
	LI T, MENG Z P, LYU L, et al. Mini-missile longitudinal control system design with disturbance rejection[J]. Journal of National University of Defense Technology, 2021, 43(1):7-15. (in Chinese)
[18]	TIAN J Y, XIONG N, ZHANG S F, et al. Integrated guidance and control for missile with narrow field-of-view strapdown seeker[J]. ISA Transactions, 2020, 10(6):124-137.
[19]	TIAN J Y, ZHANG S F, YANG H B. Enhanced extended state observer based control for missile acceleration autopilot[J]. ISA Transactions, 2020, 9(6):143-154.
[20]	梁晨, 王卫红, 赖超. 带攻击角度约束的深度强化元学习制导律[J]. 宇航学报, 2021, 42(5):611-620.
	LIANG C, WANG W H, LAI C. Deep reinforcement meta-learning guidance with impact angle constraint[J]. Journal of Astronautics, 2021, 42(5):611-620. (in Chinese)
[21]	HONG D, KIM M, PARK S. Study on reinforcement learning-based missile guidance law[J]. Applied Sciences. 2020, 10(18):6567.
[22]	HE S M, SHIN H S, TSOURDOS A. Computational missile guidance:a deep reinforcement learning approach[J]. Journal of Aerospace Information Systems, 2021, 18(8):571-582.
[23]	刘子超, 王江, 何绍溟, 等. 基于预测校正的落角约束计算制导方法[J]. 航空学报, 2022, 43(8):515-530.
	LIU Z C, WANG J, HE S M, et al. A computational guidance algorithm for impact angle control based on predictor-corrector concept[J]. Acta Aeronautica et Astronautica Sinica, 2022, 43(8):515-530. (in Chinese)
[24]	李博皓, 安旭曼, 杨晓飞, 等. 攻击角度约束下的分布式强化学习制导方法[J]. 宇航学报, 2022, 43(8):1061-1069.
	LI B H, AN X M, YANG X F, et al. A distributed reinforcement learning guidance method under impact angle constraints[J]. Journal of Astronautics, 2022, 43(8):1061-1069. (in Chinese)
[25]	LEE S M, LEE Y J, KIM Y D, et al. Impact angle control guidance considering seeker’s field-of-view limit based on reinforcement learning[J]. Journal of Guidance,Control,and Dynamics, 2023, 46(11):2168-2182.
[26]	郑成辰. 基于深度强化学习的约束末制导律研究[D]. 成都: 四川大学,2023:9.
	ZHENG C C. Research on constrained terminal guidance law based on deep reinforcement learning[D]. Chengdu: Sichuan University,2023:9. (in Chinese)
[27]	田嘉懿. 低成本全捷联微型导弹制导控制技术研究[D]. 长沙: 国防科学技术大学,2019:124.
	TIAN J Y. Research on guidance and control technologies for cost-effective all-strapdown miniature missiles[D]. Changsha: National University of Defense Technology,2019:124. (in Chinese)