基于可见光与红外图像融合的装甲目标检测算法

doi:10.12382/bgxb.2023.0401

摘要/Abstract

摘要：

针对基于可见光图像的装甲目标检测算法易受地面复杂环境干扰的问题,提出一种基于可见光与红外图像融合的装甲目标检测算法,通过卷积神经网络自适应融合可见光和红外图像特征,提高对地面复杂环境下装甲目标的检测精度。针对装甲目标检测任务,通过实拍方式构建一个在复杂地面环境下的可见光-红外装甲(Visible-Thermal Armored Vehicle,VTAV)目标图像数据集;基于经典的单阶无锚框检测模型,设计前端特征融合结构、中端特征融合结构和后端特征融合结构;在VTAV数据集上对比不同融合结构和不同融合方式间的检测性能差异。实验结果表明,后端特征融合结构性能最佳,与基于可见光图像的装甲目标检测算法相比,mAP@0.5∶0.95提升2.6%,表明基于可见光与红外图像融合的装甲目标检测算法能够有效提升地面复杂环境下装甲目标的检测精度。

关键词: 装甲目标, 目标检测, 可见光图像, 红外图像, 融合

Abstract:

The armored vehicle detection algorithm based on visible images is easily interfered by the complex ground environment. An armored vehicle detection algorithm based on fusion of visible and infrared images is proposed. The features of visible image and infrared image are adaptively fused by a convolutional neural network, which improves the detection accuracy of armored vehicle in complex ground environment. A visible-thermal armored vehicle (VTAV) dataset is constructed through on-site photography for the detection task of armored vehicle in complex ground environment. Based on the classic one-stage anchor-free detection algorithm, three fusion structures called early feature fusion, middle feature fusion and late feature fusion, are designed, and two different fusion methods are proposed. The detection performances of different fusion structures and fusion methods are compared on the VTAV dataset. The experimental results show that the peroformce of late feature fusion structure is the best, and compared to the armored vehicle detection algorithm based on visible image, mAP@0.5∶0.95 is increased by 2.6 %. The armored vehicle detection algorithm based on fusion of visible and infrared images has been proven to effectively improve the detection accuracy in complex ground environment.

Key words: armored vehicle, target detection, visible image, infrared image, fusion

中图分类号:

TP391.4

常天庆, 张杰, 赵立阳, 韩斌, 张雷. 基于可见光与红外图像融合的装甲目标检测算法[J]. 兵工学报, 2024, 45(7): 2085-2096.

CHANG Tianqing, ZHANG Jie, ZHAO Liyang, HAN Bin, ZHANG Lei. Research on Armored Vehicle Detection Algorithm Based on Visible and Infrared Image Fusion[J]. Acta Armamentarii, 2024, 45(7): 2085-2096.

图/表 17

图1 VTAV数据集部分图像对示例

Fig.1 Example of partial image pairs of VTAV dataset

表1 VATV数据集信息

Table 1 Information of VTAV dataset

数据集名称	图像对数量	装甲目标数量
VTAV数据集	4977	10244
训练集	3000	5784
验证集	977	2209
测试集	1000	2251

图2 不同数据集目标尺度分布图

Fig.2 Target scale distribution chart of different datasets

图3 不同数据集目标尺度散点图

Fig.3 Scatter plots of target scales for different datasets

图4 FCOS目标检测模型结构

Fig.4 Structure of FCOS target detection model

图5 边框回归示意图

Fig.5 Schematic diagram of bounding box regression

图6 前端特征融合检测结构

Fig.6 Structure of early feature fusion

图7 中端特征融合检测结构

Fig.7 Structure of middle feature fusion

图8 后端特征融合检测结构

Fig.8 Structure of late feature fusion

图9 通道拼接融合示意图

Fig.9 Fusion diagram of concatenating the channels

图10 可见光-红外图像拼接融合示意图

Fig.10 Fusion diagram of concatenating the visible and infrared images

表2 不同卷积核尺寸的融合效果

Table 2 Fusion effects of different convolutional kernel sizes

融合检测结构	融合级别	卷积核尺寸		mAP@0.5/ %	mAP@0.5∶ 0.95/%
融合检测结构	融合级别	1×1	3×3	mAP@0.5/ %	mAP@0.5∶ 0.95/%
EFF	像素级	√		50.4	19.8
EFF	像素级		√	62.2	28.6
MFF	特征级	√		78.9	45.2
MFF	特征级		√	78.9	45.8
LFF	特征级	√		78.8	45.2
LFF	特征级		√	79.3	46.1

表3 BN层和Relu激活层的融合效果

Table 3 Fusion effects of BN layer and Relu activation layer

融合检测结构	融合级别	BN+Relu	mAP@0.5/ %	mAP@0.5∶ 0.95/%
EFF	像素级		78.6	43.6
EFF	像素级	√	62.2	28.6
MFF	特征级		77.2	43.1
MFF	特征级	√	78.9	45.8
LFF	特征级		78.1	44.1
LFF	特征级	√	79.3	46.1

表4 预训练权重对前端融合检测结构的影响

Table 4 The influence of pre-trained weights on the structure of early feature fusion

BN+Relu	预训练权重	mAP@0.5/%	mAP@0.5∶0.95/%
		67.8	31.0
√		68.2	31.3
	√	78.6	43.6
√	√	62.2	28.6

表5 不同检测模型实验结果

Table 5 Experimental results of different detection models

检测结构	融合方式			mAP@0.5/%	mAP@0.5∶0.95/%	mAP@0.5∶0.95精度差	FPS/(帧·s)
检测结构	Add	Concatenation	RGBT	mAP@0.5/%	mAP@0.5∶0.95/%	mAP@0.5∶0.95精度差	FPS/(帧·s)
FCOS_T				61.6	24.2	-19.3	29.8
FCOS_RGB				78.5	43.5	0	29.8
EFF	√			76.5	39.6	-3.9	28.8
EFF		√		78.7	43.6	+0.1	28.6
EFF			√	72.0	39.6	-3.9	28.7
MFF	√			78.9	44.5	+1	19.5
MFF		√		78.9	45.8	+2.3	13
LFF	√			79.2	45.8	+2.3	18.9
LFF		√		79.3	46.1	+2.6	18

表6 不同检测算法的检测结果

Table 6 Detection results of different detection algorithms

表7 拓展实验结果

Table 7 Results of extended experiments

目标检测算法	输入图像	mAP@0.5/%	mAP@0.5∶0.95/%
YOLOv5_n	RGB	75.9	44.4
YOLOv5_n	RGB+T	77.2(+1.3)	45.9(+1.5)
YOLOv5_s	RGB	78.4	48.6
YOLOv5_s	RGB+T	81.0(+2.6)	50.5(+1.9)
YOLOv7_tiny	RGB	78.5	45.7
YOLOv7_tiny	RGB+T	79(+0.5)	46.5(+0.8)
YOLOv8_n	RGB	77.2	46.8
YOLOv8_n_	RGB+T	78.8(+1.6)	47.7(+0.9)

参考文献 24

[1]	刘科. 认知技术在战场态势感知中的应用[J]. 指挥信息系统与技术, 2021, 12(3):13-18.
	LIU K. Application of cognition technology in battlefield situation awareness[J]. Command Information System and Technology, 2021, 12(3):13-18. (in Chinese)
[2]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6):84-90.
[3]	REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, US:IEEE, 2016:779-788.
[4]	REDMON J, FARHADI A. YOLOv3:an incremental improvement:arXiv:1804.02767[R]. Ithaca,NY, US: Cornell University, 2018:1804.02767.
[5]	GE Z, LIU S T, WANG F, et al. YOLOx:exceeding yolo series in 2021:arXiv:2107.08430[R]. Ithaca,NY, US: Cornell University, 2021:2107.08430.
[6]	WANG C Y, BOCHKOVSKIY A, LIAO H Y M. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver, Canada: IEEE, 2023: 7464-7475.
[7]	REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6):1137-1149. doi: 10.1109/TPAMI.2016.2577031 pmid: 27295650
[8]	ZOU Z X, CHEN K Y, SHI Z W, et al. Object detection in 20 years: a survey[J]. Proceedings of the IEEE, 2023, 111(3):257-276.
[9]	王全东, 常天庆, 张雷, 等. 基于深度学习算法的坦克装甲目标自动检测与跟踪系统[J]. 系统工程与电子技术, 2018, 40(9):2143-2156.
	WANG Q D, CHANG T Q, ZHANG L, et al. Automatic detection and tracking system of tank armored targets based on deep learning algorithm[J]. System Engineering and Electronics, 2018, 40(9):2143-2156. (in Chinese)
[10]	于博文, 吕明. 改进的YOLOv3算法及其在军事目标检测中的应用[J]. 兵工学报, 2022, 43(2):345-354. doi: 10.3969/j.issn.1000-1093.2022.02.012
	YU B W, LÜ M. Improved YOLOv3 algotithm and its application in military target detection[J]. Acta Armamentarii, 2022, 43(2):345-354. (in Chinese)
[11]	宋晓茹, 刘康, 高嵩, 等. 复杂战场环境下改进YOLOv5军事目标识别算法研究[J]. 兵工学报, 2024, 45(3):934-947. doi: 10.12382/bgxb.2022.0736
	SONG X R, LIU K, GAO S, et al. Research on improving YOLOv5 military target recognition algorithm in complex battlefield environment[J]. Acta Armamentarii, 2024, 45(3):934-947. (in Chinese)
[12]	盛大俊, 张强. 基于边缘感知的深度神经网络红外装甲目标检测[J]. 红外技术, 2021, 43(8):784-791.
	SHENG D J, ZHANG Q. Infrared armored target detection based on edge-perception in deep neural network[J]. Infrared Technology, 2021, 43(8):784-791. (in Chinese)
[13]	LI C L, LIANG X Y, LU Y J, et al. RGB-T object tracking:Benchmark and baseline[J]. Pattern Recognition, 2019, 96:106977.
[14]	PENG P R, XU T F, HUANG B, et al. HAFNet: Hierarchical attentive fusion network for multispectral pedestrian detection[J]. Remote Sensing, 2023, 15(8):2041.
[15]	别倩, 王晓, 徐新, 等. 红外—可见光跨模态的行人检测综述[J]. 中国图象图形学报, 2023, 28(5): 1287-1307.
	BIE Q, WANG X, XU X, et al. 2023. Visible-infrared cross-modal pedestrian detection:a sum-mary[J]. Journal of Image and Graphics, 2023, 28(5):1287-1307. (in Chinese)
[16]	HWANG S, PARK J, KIM N, et al. Multispectral pedestrian detection: Benchmark dataset and baseline[C]// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, US: IEEE, 2015: 1037-1045.
[17]	LIU J J, ZHANG S T, WANG S, et al. Multispectral deep neural networks for pedestrian detection[C]//Proceedings of the British Machine Vision Conference.York, UK: BMVA Press, 2016:71-73.
[18]	SUN Y M, CAO B, ZHU P F, et al. Drone-based RGB-infrared cross-modality vehicle detection via uncertainty-aware learning[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(10): 6700-6713.
[19]	CHEN Y T, SHI J H, YE Z, et al. Multimodal object detection via probabilistic ensembling[C]//Proceedings of the Computer Vision-ECCV 2022: 17th European Conference. Tel Aviv, Israel: Springer, 2022: 139-158.
[20]	ZHENG Y, IZZAT I H, ZIAEE S. GFD-SSD: gated fusion double SSD for multispectral pedestrian detection:arXiv:1903.06999[R]. Ithaca,NY, US: Cornell University, 2019:1903.06999.
[21]	FANG Q Y, WANG Z K. Cross-modality attentive feature fusion for object detection in multispectral remote sensing imagery[J]. Pattern Recognition, 2022, 130: 108786.
[22]	WOLPERT A, TEUTSCH M, SARFRAZ M S, et al. Anchor-free small-scale multispectral pedestrian detection[C]//Proceedings of the 31st British Machine Vision Conference 2020. Virtual Event, UK: BMVA Press, 2020.
[23]	TIAN Z, SHEN C H, CHEN H, et al. FCOS:fully convolutional one-stage object detection[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Seoul, Korea: IEEE, 2019: 9626-9636.
[24]	CHEN K, WANG J Q, PANG J M, et al. MMDetection: Open mmlab detection toolbox and benchmark:arXiv:1906.07155[R]. Ithaca,NY, US: Cornell University, 2019:1906.07155.