基于强化学习的对空雷达抗干扰波形设计

doi:10.12382/bgxb.2022.0069

摘要/Abstract

摘要：

被探测目标的电子战能力严重影响对空雷达的检测和识别性能。针对对空雷达抗干扰问题,提出基于强化学习的抗干扰波形设计方法。从博弈角度利用强化学习方法建立雷达和目标干扰间动态对抗模型,计算博弈各方状态、动作价值,利用策略迭代法生成最优策略,基于相位迭代法合成时域波形。仿真结果表明:使用新方法设计的雷达发射信号时,与线性调频信号、捷变频信号相比,信干噪比分别提高了6.39dB和1.12dB;信号总功率为11W时,与捷变频信号相比,目标检测概率提升了5%,可在低功率实现抗干扰的同时提升雷达信号抗截获性能。

关键词: 对空雷达, 波形设计, 强化学习, 抗干扰

Abstract:

The electronic warfare capability of the detected target seriously affects the detection and identification performance of the ground-based air surveillance radar. To address the anti-jamming problem of the air surveillance radar, an anti-jamming waveform design method based on reinforcement learning is proposed. From the perspective of game theory, the reinforcement learning method is used to establish a dynamic confrontation model between radar and target jamming, calculate the state and action value of all parties in the game. The strategy iteration method is employed to generate the optimal strategy, and the time domain waveform is synthesized based on the phase iteration method. The simulation results shows that: when using the radar transmitted signal designed by the proposed method, compared with linear frequency-modulated signals and frequency-agile signals, the signal-to-interference-noise ratio is increased by 6.39dB and 1.12dB respectively; when the total signal power is 11W, compared with the frequency-agile signal, the target detection probability is increased by 5%, meaning that the designed signal can improve the anti-interception performance of radar signals while achieving anti-jamming at low power.

Key words: ground-based air surveillance radar, waveform design, reinforcement learning, anti-jamming

郑泽新, 李伟, 邹鲲, 李艳福. 基于强化学习的对空雷达抗干扰波形设计[J]. 兵工学报, 2023, 44(5): 1422-1430.

ZHENG Zexin, LI Wei, ZOU Kun, LI Yanfu. Anti-jamming Waveform Design of Ground-based Air Surveillance Radar Based on Reinforcement Learning[J]. Acta Armamentarii, 2023, 44(5): 1422-1430.

图/表 11

图1 雷达探测场景

Fig.1 Radar detection scene

图2 信号模型

Fig.2 Signal model

图3 马尔可夫决策过程示意图

Fig.3 Schematic diagram of Markovdecision process

表1 强化学习模型参数设置

Table 1 Parameter setting of reinforcement learning model

马尔可夫决策模型	参数设置
状态空间	S={7位6进制数}
动作空间	A={7位6进制数}
状态转移概率	P(s,a)=1/(6⁷-1)
γ	0.9

表2 奖励函数结构

Table 2 Reward function structure

SINR/dB	<0	0~10	10~20	20~30	30~40	>40
Reward	-30	5	10	15	20	30

表3 仿真参数

Table 3 Simulation parameters

雷达系统	仿真参数设置
工作波段	X波段
中心频率/GHz	9.5
信号带宽/MHz	140
子频带带宽/MHz	20
目标飞机速度/(m·s^-1)	250
$σ h 2$ (f)	{1, 0.2, 1.2, 6, 19, 1, 5}
噪声	1

表3 仿真参数

Table 3 Simulation parameters

雷达系统	仿真参数设置
工作波段	X波段
中心频率/GHz	9.5
信号带宽/MHz	140
子频带带宽/MHz	20
目标飞机速度/(m·s^-1)	250
$σ h 2$ (f)	{1, 0.2, 1.2, 6, 19, 1, 5}
噪声	1

图4 雷达和干扰博弈过程一

Fig.4 Game process one of radar and jamming

图5 雷达和干扰博弈过程二

Fig.5 Game process two of radar and jamming

图6 雷达策略

Fig.6 Radar strategy

图7 干扰策略

Fig.7 Jamming strategy

图8 最优波形策略和传统雷达信号比较

Fig.8 Comparison of optimal waveform strategy and traditional radar signal

参考文献 32

[1]	方文, 全英汇, 沙明辉, 等. 捷变频联合波形熵的密集假目标干扰抑制算法[J]. 系统工程与电子技术, 2021, 43(6):1506-1514.
	FANG W, QUAN Y H, SHA M H, et al. Dense false targets jamming suppression algorithm based on frequency agility and waveform entropy[J]. Systems Engineering and Electronic Technology, 2021, 43(6): 1506-1514. (in Chinese)
[2]	全英汇, 方文, 沙明辉, 等. 频率捷变雷达波形对抗技术现状与展望[J]. 系统工程与电子技术, 2021, 43(11):3126-3136.
	QUAN Y H, FANG W, SHA M H, et al. Present situation and prospects of frequency agility radar waveform countermeasures[J]. System Engineering and Electronic Technology, 2021, 43 (11): 3126-3136. (in Chinese)
[3]	晏艺翡, 苏军海, 李英军. 雷达低截获波形抗主瓣干扰技术研究与试验[J]. 火控雷达技术, 2021, 50(2):31-35.
	YAN Y F, SU J H, LI Y J. The research and experiment of anti-interference ability on radar with low-acquisition wave[J]. Fire Control Radar Technology, 2021, 50(2):31-35. (in Chinese)
[4]	THORNTON C, BUEHRER R M, MARTONE A. constrained contextual bandit learning for adaptive radar waveform selection[J]. IEEE Transactions on Aerospace and Electronic Systems, 2022, 58(2):1133-1148. doi: 10.1109/TAES.2021.3109110 URL
[5]	夏栋, 张凯旋, 丁友宝, 等. 基于相位编码波形捷变和CFAR技术抗同频干扰[J]. 系统工程与电子技术, 2022, 44(4):1210-1219.
	XIA D, ZHANG K X, DING Y B, et al. Anti-co-frequency interference based on phase-encoded waveform agility and CFAR technology[J]. Systems Engineering and Electronic Technology, 2022, 44(4):1210-1219. (in Chinese)
[6]	HU H T, LUI R H, ZHANG J Y. An improved radar anti-jamming method[C]∥Proceedings of the 2018 IEEE 3rd International Conference on Signal and Image Processing.Shenzhen, China:IEEE, 2018:439-443.
[7]	WANG H Y, YANG X K, LI Y. Radar anti-retransmitted jamming technology based on agility waveforms[C]∥Proceedings of the 2019 IEEE International Conference on Signal, Information and Data Processing.Chongqing, China:IEEE, 2019:1-6.
[8]	彭博, 陈齐乐, 李锐. 基于变分正则化参数估计的伪码调相引信重构式干扰方法[J]. 兵工学报, doi: 10.12382/bgxb.2022.0572. doi: 10.12382/bgxb.2022.0572
	PENG B, CHEN Q L, LI R. Parameters estimation of binary phase-coded modulation radar waveform based on non-convex total variation regularization[J]. Acta Armamentarii, doi: 10.12382/bgxb.2022.0572. (in Chinese) doi: 10.12382/bgxb.2022.0572
[9]	HE B, SU H T. Joint power allocation and beamforming between a multistatic radar and jammer based on game theory[C]∥Proceedings of the 2019 IEEE 11st International Conference on Communication Software and Networks. Chongqing, China:IEEE, 2019:337-341.
[10]	YE F, ZHAO T, LI Y B, et al. Anti-interference strategy selection method based on the minimum loss criterion[C] ∥Proceedings of the 2020 IEEE USNC-CNC-URSI North American Radio Science Meeting.Montreal, QC, Canada:IEEE, 2020:79-80.
[11]	CHEN H W, TA S Y, SUN B. Cooperative game approach to power allocation for target tracking in distributed MIMO radar sensor networks[J]. IEEE Sensors Journal, 2015, 15(10): 5423-5432. doi: 10.1109/JSEN.2015.2431261 URL
[12]	SHI C G, WEI Q, WANG F, et al. Cooperative LPI performance optimization for multistatic radar system: a stackelberg game[C]∥Proceedings of the 2019 International Applied Computational Electromagnetics Society Symposium-China.Nanjing, China:IEEE, 2019:1-2.
[13]	WANG D, LI K M, ZHANG Q, et al. A cooperative task allocation game for Multi-target imaging in radar networks[J]. IEEE Sensors Journal, 2021, 21(6):7541-7550. doi: 10.1109/JSEN.7361 URL
[14]	李伟, 王泓霖, 郑家毅, 等. 博弈条件下雷达波形设计策略研究[J]. 电子与信息学报, 2019, 41(11):2654-2660.
	LI W, WANG H L, ZHENG J Y, et al. Research on radar waveform design strategy under game conditions[J]. Journal of Electronics and Information, 2019, 41(11): 2654-2660. (in Chinese)
[15]	WANG H L, LI W, WANG H, et al. Radar waveform strategy based on game theory[J]. Radio Engineering, 2019, 28(4):757-764.
[16]	甘奕夫, 李伟, 赵俊龙. 干扰条件下基于Bayesian博弈的认知制导雷达波形设计[J]. 空军工程大学学报(自然科学版), 2021, 22(2):91-98.
	GAN Y F, LI W, ZHAO J L. A cognitive guidance radar waveform design based on bayesian game under conditions of jamming[J]. Journal of Air Force Engineering University(Natural Science Edition), 2021, 22(2):91-98. (in Chinese)
[17]	XING Q, ZHU W G, JIA X. Research on method of intelligent radar confrontation based on reinforcement learning[C]∥Proceedings of the 2017 2nd IEEE International Conference on Computational Intelligence and Applications. Beijing, China: IEEE, 2017:471-475.
[18]	XING Q, ZHU W G, JIA X. Intelligent countermeasure design of radar working-modes unknown[C]∥Proceedings of the 2017 IEEE International Conference on Signal Processing, Communications and Computing. Xiamen,China:IEEE, 2017:1-5.
[19]	SELVI E, BUEHRER R M, MARTON A, et al. On the use of markov decision processes in cognitive radar: an application to target tracking[C]∥Proceedings of the 2018 IEEE Radar Conference. Oklahoma City,OK,US:IEEE, 2018:0537-0542.
[20]	KANG L J, JIU B, LIU H W, et al. Reinforcement learning based anti-jamming frequency hopping strategies design for cognitive radar[C]∥Proceedings of the 2018 IEEE International Conference on Signal Processing, Communications and Computing. Qingdao,China:IEEE, 2018:1-5.
[21]	LI K, JIU B, LIU H W. Deep Q-network based anti-jamming strategy design for frequency agile radar[C]∥Proceedings of the 2019 International Radar Conference.Toulon, France:IEEE, 2019:1-5.
[22]	WANG L, FORTUNATI S, GRECO M S, et al. Reinforcement learning-based waveform optimization for MIMO multi-target detection[C]∥Proceedings of the 2018 52nd Asilomar Conference on Signals, Systems, and Computers.Pacific Grove, CA, US:IEEE, 2018:1329-1333.
[23]	CAO X, ZHENG Z, AN D. Adaptive waveform selection algorithm based on reinforcement learning for cognitive radar[C]∥Proceedings of the 2019 IEEE 2nd International Conference on Automation,Electronics and Electrical Engineering.Shenyang, China:IEEE, 2019:208-213.
[24]	WANG Q, QIAO Y M, GAO L R. A cognitive radar waveform optimization approach based on deep reinforcement learning[C]∥Proceedings of the 2019 IEEE International Conference on Signal, Information and Data Processing.Chongqing, China:IEEE, 2019:1-6.
[25]	AK S, BRÜGGENWIRTH S. Avoiding jammers: a reinforcement learning approach[C] ∥Proceedings of the 2020 IEEE International Radar Conference.Washington, DC, US: IEEE, 2020:321-326.
[26]	SHI Y C, JIU B, YAN J K, et al. Data-driven simultaneous multibeam power allocation: when multiple targets tracking meets deep reinforcement learning[J]. IEEE Systems Journal, 2021, 15(1):1264-1274. doi: 10.1109/JSYST.2020.2984774 URL
[27]	AILIYAW, YIW, YUANY. Reinforcement learning-based joint adaptive frequency hopping and pulse-width allocation for radar anti-jamming[C]∥Proceedings of the 2020 IEEE Radar Conference.Florence, Italy:IEEE, 2020:1-6.
[28]	ROMERO R A, BAE J, GOODMAN N A. Theory and application of SNR and mutual information matched illumination waveforms[J]. IEEE Transactions on Aerospace and Electronic Systems, 2011, 47(2): 912-927. doi: 10.1109/TAES.2011.5751234 URL
[29]	STEVEN M K. 统计信号处理基础: 估计与检测理论[M].罗鹏飞,张文明,刘忠,等,译. 北京: 电子工业出版社, 2014: 425-445.
	STEVEN M K. Fundamentals of statistical signal processing: estimation and detection theory[M]. LUO P F, ZHANG W M, LIU Z, et al. translated. Beijing: Publishing House of Electronics Industry, 2014:425-445. (in Chinese)
[30]	SUTTON R S, BARTO A G. Reinforcement learning: an introduction[M]. Cambridge,MA, US: MIT Press, 1998:67-98.
[31]	黎湘, 范梅梅. 认知雷达及其关键技术研究进展[J]. 电子学报, 2012, 40(9):1863-1870. doi: 10.3969/j.issn.0372-2112.2012.09.025
	LI X, FAN M M. Research progress of cognitive radar and its key technologies[J]. Acta Electronica Sinica, 2012, 40(9):1863-1870. (in Chinese)
[32]	JACKSON L, KAY S, VANKAYALAPATI N. Iterative method for nonlinear FM synthesis of radar signals[J]. IEEE Transactions on Aerospace and Electronic Systems, 2010, 46(2):910-917. doi: 10.1109/TAES.2010.5461666 URL