An Impact Angle and Field of View Constraints Guidance Law Based on Deep Reinforcement Learning

doi:10.12382/bgxb.2024.0435

Abstract

Abstract:

To meet the increasingly complex operational demands and improve the guidance performance of micro-guided munitions at close range,a impact angle constraint guidance law based on deep reinforcement learning (DRL) is proposed by considering the field-of-view (FOV) limitations.The estimation formula for the impact angle error of missile relative to a moving target is derived,the impact angle error and view angle are used as state variables,and a piecewise reward function is constructed.The guidance problem is modeled as a time-discrete Markov decision process (MDP).The required guidance commands are obtained through biased proportional navigation,and a bias term is output by the DRL policy network.The network is trained using the proximal policy optimization (PPO) algorithm,resulting in an optimal guidance strategy that ensures the constraints on both view and impact angles without information about missile-target distance.Numerical and Monte Carlo simulations under different initial conditions and the comparative analyses of capture regions of missiles at various speeds are conducted.The results show that the proposed guidance law maintains excellent guidance performance under various initial conditions,provides a larger capture region in close-range engagements compared to existing guidance laws,and demonstrates a smaller impact angle error distribution under interference,thus validating its effectiveness and superiority.

Key words: guidance law, impact angle constraint, deep reinforcement learning, limited field of view angle, micro-guided munition

CLC Number:

TJ765.3

XIAN Sujie, WANG Kang, ZENG Xin, SONG Jie, WU Zhilin. An Impact Angle and Field of View Constraints Guidance Law Based on Deep Reinforcement Learning[J]. Acta Armamentarii, 2025, 46(4): 240435-.

Figures/Tables 17

Fig.1 Engagement geometry

Fig.2 The estimated result of vT

Fig.3 Structural diagram of guidance law training algorithm

Fig.4 PPO-based guidance law training algorithm for impact angle control with FOV angle constraints

Table 1 Neural network architecture

神经网络	结构	激活函数
Actor	(2,64)	ReLU
	(64,128)	ReLU
	(128,64)	ReLU
	(64,1),(64,1)	Tanh,Sigmoid
Critic	(2,64)	ReLU
	(64,256)	ReLU
	(256,64)	ReLU
	(64,32)	ReLU
	(32,16)	ReLU
	(16,1)	Linner

Table 2 Parameter settings

参数	取值
N	3
v_M/(m·s^-1)	100
v_T/(m·s^-1)	10
a^bn/(m· $s - 2$ )	30
$e n θ$	0.5e_θ₀
$σ n M$ /(°)	10
q₀/(°)	0
σ_M0/(°)	15
θ_M0/(°)	15
n	32
α_ϑ	0.00001
α_φ	0.0001
k₁	0.25
γ	0.95
λ_GAE	0.95
ε	0.2
K	10
dt/s	0.01
Δt/s	0.02
训练回合总数	2000

Table 2 Parameter settings

参数	取值
N	3
v_M/(m·s^-1)	100
v_T/(m·s^-1)	10
a^bn/(m· $s - 2$ )	30
$e n θ$	0.5e_θ₀
$σ n M$ /(°)	10
q₀/(°)	0
σ_M0/(°)	15
θ_M0/(°)	15
n	32
α_ϑ	0.00001
α_φ	0.0001
k₁	0.25
γ	0.95
λ_GAE	0.95
ε	0.2
K	10
dt/s	0.01
Δt/s	0.02
训练回合总数	2000

Fig.5 Numerically simulated results for variousσlim

Table 3 Simulated results under differentσlimconditions

σ_lim/(°)	σ_sat/(°)	终端θ_M/(°)	最大σ_M/°	$∫ 0 t f a M$ dt
15	14.0	-29.98	15.00	78.52
30	28.5	-29.84	28.18	120.34
45	43.5	-30.00	41.86	163.13
无限制		-30.45	61.01	174.77

Table 3 Simulated results under differentσlimconditions

σ_lim/(°)	σ_sat/(°)	终端θ_M/(°)	最大σ_M/°	$∫ 0 t f a M$ dt
15	14.0	-29.98	15.00	78.52
30	28.5	-29.84	28.18	120.34
45	43.5	-30.00	41.86	163.13
无限制		-30.45	61.01	174.77

Fig.6 Numerically simulated results for various θexp

Table 4 Simulation results under different θexp

θ_exp/(°)	终端θ_M/(°)	最大σ_M/(°)	$∫ 0 t f a M$ dt
-30	-29.84	28.18	120.34
-45	-44.87	28.42	147.76
-60	-59.91	28.60	174.48
-75	-74.53	28.69	200.26

Table 4 Simulation results under different θexp

θ_exp/(°)	终端θ_M/(°)	最大σ_M/(°)	$∫ 0 t f a M$ dt
-30	-29.84	28.18	120.34
-45	-44.87	28.42	147.76
-60	-59.91	28.60	174.48
-75	-74.53	28.69	200.26

Fig.7 Numerically simulated results for variousvT

Fig.8 The capture regions of RLIACG,IACCG and D-RLIACG

Table 5 Capture rates of RLIACG,D-RLIACG and IACCG for variousvM %

v_M/(m·s^-1)	RLIACG	IACCG	D-RLIACG
100	88.89	68.15	65.56
150	71.85	38.15	57.78
200	47.04	21.11	39.63

Fig.9 Comparison of Monte Carlo simulations of RLIACG and IACCG

Table 6 Error distributions of RLIACG,IACCG and D-RLIACG

统计量	脱靶量/m			落角误差/(°)
统计量	RLIACG	IACCG	D-RLIACG	RLIACG	IACCG	D-RLIACG
均值	0.24	0.26	1.15	0.22	0.91	0.47
标准差	0.19	0.19	0.26	0.14	0.76	0.44
最大值	0.75	0.90	1.90	0.77	5.91	4.30

Table 7 Related parameters of dynamic models and controllers

参数	数值
M_α/s^-2	-423.6
M_φ/s^-1	-0.07
M_δ/s^-2	331
Z_α/(m·s^-2·rad^-1)	-705
Z_δ/(m·s^-2·rad^-1)	-165
K_A/(rad·s·m^-1)	4.5×10^-4
K_DC	21.05
K_R/s	0.34
ω_i/(rad·s^-1)	16.36
C_x0	0.18
$C x α 2$ /(rad^-2)	7.65
S_m/m²	0.013
τ_δ/s	0.04

Table 7 Related parameters of dynamic models and controllers

参数	数值
M_α/s^-2	-423.6
M_φ/s^-1	-0.07
M_δ/s^-2	331
Z_α/(m·s^-2·rad^-1)	-705
Z_δ/(m·s^-2·rad^-1)	-165
K_A/(rad·s·m^-1)	4.5×10^-4
K_DC	21.05
K_R/s	0.34
ω_i/(rad·s^-1)	16.36
C_x0	0.18
$C x α 2$ /(rad^-2)	7.65
S_m/m²	0.013
τ_δ/s	0.04

Fig.10 Dynamic simulated results

References 27

[1]	黄景帅, 张洪波, 汤国建, 等. 拦截大气层内机动目标的自适应积分滑模制导律[J]. 宇航学报, 2019, 40(1):51-60.
	HUANG J S, ZHANG H B, TANG G J, et al. Adaptive integral sliding-mode guidance law for intercepting endo-atmospheric maneuvering targets[J]. Journal of Astronautics, 2019, 40(1):51-60. (in Chinese)
[2]	SHASHI R K, SACHIT R, DEBASISH G. Nonsingular terminal sliding mode guidance with impact angle constraints[J]. Journal of Guidance,Control,and Dynamics, 2014, 37(4):1114-1130.
[3]	盛永智, 甘佳豪, 张成新. 弹道可调的落角约束分数阶滑模制导律设计[J]. 航空学报, 2023, 44(7):177-190.
	SHENG Y Z, GAN J H, ZHANG C X. Fractional order sliding mode guidance law design with trajectory adjustable and terminal angular constraint[J]. Acta Aeronauticaet Astronautica Sinica, 2023, 44(7):177-190. (in Chinese)
[4]	KORAY S, OEZGOEREN M. Control of impact angle using biased proportional navigation[C]// Proceedings of AIAA Guidance,Navigation,and Control Conference.Boston, US:AIAA,2013:5348-5362.
[5]	马帅, 王旭刚, 王中原, 等. 带初始前置角和末端攻击角约束的偏置比例导引律设计以及剩余飞行时间估计[J]. 兵工学报, 2019, 40(1):68-78.
	MA S, WANG X G, WANG Z Y, et al. BPNG law with arbitrary initial lead angle and terminal impact angle constraint and time-to-go estimation[J]. Acta Armamentarii, 2019, 40(1):68-78. (in Chinese)
[6]	王晓海, 孟秀云, 周峰, 等. 基于偏置比例导引的落角约束滑模制导律[J]. 系统工程与电子技术, 2021, 43(5):1295-1302.
	WANG X H, MENG X Y, ZHOU F, et al. Sliding mode guidance law with impact angle constraint based on bias proportional navigation[J]. Systems Engineering and Electronics, 2021, 43(5):1295-1302. (in Chinese)
[7]	严鹏辉, 刘刚, 缪前树. 基于落角约束的偏置比例导引律的研究[J]. 现代防御技术, 2021, 49(6):43-48,55.
	YAN P H, LIU G, MIAO Q S. Research on bias proportional navigation guidance law based on terminal impact angle constraint[J]. Modern Defence Technology, 2021, 49(6):43-48,55. (in Chinese)
[8]	KIM J H, PARK S S, PARK K K, et al. Quaternion based three-dimensional impact angle control guidance law[J]. IEEE Transactions on Aerospace and Electronic Systems, 2021, 57(4):2311-2323.
[9]	黄嘉, 常思江, 陈琦, 等. 不依赖剩余飞行时间的数据驱动攻击时间控制导引律[J]. 兵工学报, 2023, 44(8):2299-2309.
	HUANG J, CHANG S J, CHEN Q, et al. Data-driven-based impact time control guidance law independent of time-to-go[J]. Acta Armamentarii, 2023, 44(8):2299-2309. (in Chinese)
[10]	黄嘉, 常思江. 基于数据驱动的攻击时间和攻击角度控制导引律[J]. 系统工程与电子技术, 2022, 44(10):3213-3220.
	HUANG J, CHANG S J. Data-driven method based impact time and impact angle control guidance law[J]. Systems Engineering and Electronics, 2022, 44(10):3213-3220. (in Chinese)
[11]	YANG H, WU H J, BAI X B, et al. A new united proportional navigation guidance for impact angle constraint without measurement distance between vehicle and target[J/OL]. International Journal of Aerospace Engineering, 2022(2022-10-30). https://doi.org/10.1155/2022/6495746.
[12]	PARK B G, KWON H H, KIM Y H, et al. Composite guidance scheme for impact angle control against a non maneuvering moving target[J]. Journal of Guidance,Control,and Dynamics, 2016, 39(5):1132-1139.
[13]	CHO D, KIM S H. Direct impact angle control guidance for passive homing missiles[J]. Proceedings of the Institution of Mechanical Engineers,Part G:Journal of Aerospace Engineering, 2019, 234(15):2139-2152.
[14]	CHO N, KIM J, LEE S, et al. Generalized analysis of biased proportional navigation guidance with fractional power error feedback[J]. Journal of Guidance,Control,and Dynamics, 2022, 45(9):1598-1613.
[15]	王江, 刘经纬, 崔晓曦, 等. 有限视场下的攻击时间和角度多约束制导律[J]. 北京理工大学学报, 2024, 44(1):18-27.
	WANG J, LIU J W, CUI X X, et al. Multi-Constraint guidance law for impact time and angle with limited field-of-view[J]. Journal of Beijing Institute of Technology, 2024, 44(1):18-27. (in Chinese)
[16]	FAN J F, WANG G, CHEN Y, et al. Impact-angle-constraint look angle guidance for miniature missile[C]// Proceedings of the 2016 16th International Conference on Control,Automation and Systems.Gyeongju, South Korea: IEEE,2016:1005-1009.
[17]	李彤, 孟志鹏, 吕良, 等. 微型导弹纵向扰动抑制控制系统设计[J]. 国防科技大学学报, 2021, 43(1):7-15.
	LI T, MENG Z P, LYU L, et al. Mini-missile longitudinal control system design with disturbance rejection[J]. Journal of National University of Defense Technology, 2021, 43(1):7-15. (in Chinese)
[18]	TIAN J Y, XIONG N, ZHANG S F, et al. Integrated guidance and control for missile with narrow field-of-view strapdown seeker[J]. ISA Transactions, 2020, 10(6):124-137.
[19]	TIAN J Y, ZHANG S F, YANG H B. Enhanced extended state observer based control for missile acceleration autopilot[J]. ISA Transactions, 2020, 9(6):143-154.
[20]	梁晨, 王卫红, 赖超. 带攻击角度约束的深度强化元学习制导律[J]. 宇航学报, 2021, 42(5):611-620.
	LIANG C, WANG W H, LAI C. Deep reinforcement meta-learning guidance with impact angle constraint[J]. Journal of Astronautics, 2021, 42(5):611-620. (in Chinese)
[21]	HONG D, KIM M, PARK S. Study on reinforcement learning-based missile guidance law[J]. Applied Sciences. 2020, 10(18):6567.
[22]	HE S M, SHIN H S, TSOURDOS A. Computational missile guidance:a deep reinforcement learning approach[J]. Journal of Aerospace Information Systems, 2021, 18(8):571-582.
[23]	刘子超, 王江, 何绍溟, 等. 基于预测校正的落角约束计算制导方法[J]. 航空学报, 2022, 43(8):515-530.
	LIU Z C, WANG J, HE S M, et al. A computational guidance algorithm for impact angle control based on predictor-corrector concept[J]. Acta Aeronautica et Astronautica Sinica, 2022, 43(8):515-530. (in Chinese)
[24]	李博皓, 安旭曼, 杨晓飞, 等. 攻击角度约束下的分布式强化学习制导方法[J]. 宇航学报, 2022, 43(8):1061-1069.
	LI B H, AN X M, YANG X F, et al. A distributed reinforcement learning guidance method under impact angle constraints[J]. Journal of Astronautics, 2022, 43(8):1061-1069. (in Chinese)
[25]	LEE S M, LEE Y J, KIM Y D, et al. Impact angle control guidance considering seeker’s field-of-view limit based on reinforcement learning[J]. Journal of Guidance,Control,and Dynamics, 2023, 46(11):2168-2182.
[26]	郑成辰. 基于深度强化学习的约束末制导律研究[D]. 成都: 四川大学,2023:9.
	ZHENG C C. Research on constrained terminal guidance law based on deep reinforcement learning[D]. Chengdu: Sichuan University,2023:9. (in Chinese)
[27]	田嘉懿. 低成本全捷联微型导弹制导控制技术研究[D]. 长沙: 国防科学技术大学,2019:124.
	TIAN J Y. Research on guidance and control technologies for cost-effective all-strapdown miniature missiles[D]. Changsha: National University of Defense Technology,2019:124. (in Chinese)