Research on Armored Vehicle Detection Algorithm Based on Visible and Infrared Image Fusion

doi:10.12382/bgxb.2023.0401

Abstract

Abstract:

The armored vehicle detection algorithm based on visible images is easily interfered by the complex ground environment. An armored vehicle detection algorithm based on fusion of visible and infrared images is proposed. The features of visible image and infrared image are adaptively fused by a convolutional neural network, which improves the detection accuracy of armored vehicle in complex ground environment. A visible-thermal armored vehicle (VTAV) dataset is constructed through on-site photography for the detection task of armored vehicle in complex ground environment. Based on the classic one-stage anchor-free detection algorithm, three fusion structures called early feature fusion, middle feature fusion and late feature fusion, are designed, and two different fusion methods are proposed. The detection performances of different fusion structures and fusion methods are compared on the VTAV dataset. The experimental results show that the peroformce of late feature fusion structure is the best, and compared to the armored vehicle detection algorithm based on visible image, mAP@0.5∶0.95 is increased by 2.6 %. The armored vehicle detection algorithm based on fusion of visible and infrared images has been proven to effectively improve the detection accuracy in complex ground environment.

Key words: armored vehicle, target detection, visible image, infrared image, fusion

CLC Number:

TP391.4

CHANG Tianqing, ZHANG Jie, ZHAO Liyang, HAN Bin, ZHANG Lei. Research on Armored Vehicle Detection Algorithm Based on Visible and Infrared Image Fusion[J]. Acta Armamentarii, 2024, 45(7): 2085-2096.

Figures/Tables 17

References 24

[1]	刘科. 认知技术在战场态势感知中的应用[J]. 指挥信息系统与技术, 2021, 12(3):13-18.
	LIU K. Application of cognition technology in battlefield situation awareness[J]. Command Information System and Technology, 2021, 12(3):13-18. (in Chinese)
[2]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6):84-90.
[3]	REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, US:IEEE, 2016:779-788.
[4]	REDMON J, FARHADI A. YOLOv3:an incremental improvement:arXiv:1804.02767[R]. Ithaca,NY, US: Cornell University, 2018:1804.02767.
[5]	GE Z, LIU S T, WANG F, et al. YOLOx:exceeding yolo series in 2021:arXiv:2107.08430[R]. Ithaca,NY, US: Cornell University, 2021:2107.08430.
[6]	WANG C Y, BOCHKOVSKIY A, LIAO H Y M. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver, Canada: IEEE, 2023: 7464-7475.
[7]	REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6):1137-1149. doi: 10.1109/TPAMI.2016.2577031 pmid: 27295650
[8]	ZOU Z X, CHEN K Y, SHI Z W, et al. Object detection in 20 years: a survey[J]. Proceedings of the IEEE, 2023, 111(3):257-276.
[9]	王全东, 常天庆, 张雷, 等. 基于深度学习算法的坦克装甲目标自动检测与跟踪系统[J]. 系统工程与电子技术, 2018, 40(9):2143-2156.
	WANG Q D, CHANG T Q, ZHANG L, et al. Automatic detection and tracking system of tank armored targets based on deep learning algorithm[J]. System Engineering and Electronics, 2018, 40(9):2143-2156. (in Chinese)
[10]	于博文, 吕明. 改进的YOLOv3算法及其在军事目标检测中的应用[J]. 兵工学报, 2022, 43(2):345-354. doi: 10.3969/j.issn.1000-1093.2022.02.012
	YU B W, LÜ M. Improved YOLOv3 algotithm and its application in military target detection[J]. Acta Armamentarii, 2022, 43(2):345-354. (in Chinese)
[11]	宋晓茹, 刘康, 高嵩, 等. 复杂战场环境下改进YOLOv5军事目标识别算法研究[J]. 兵工学报, 2024, 45(3):934-947. doi: 10.12382/bgxb.2022.0736
	SONG X R, LIU K, GAO S, et al. Research on improving YOLOv5 military target recognition algorithm in complex battlefield environment[J]. Acta Armamentarii, 2024, 45(3):934-947. (in Chinese)
[12]	盛大俊, 张强. 基于边缘感知的深度神经网络红外装甲目标检测[J]. 红外技术, 2021, 43(8):784-791.
	SHENG D J, ZHANG Q. Infrared armored target detection based on edge-perception in deep neural network[J]. Infrared Technology, 2021, 43(8):784-791. (in Chinese)
[13]	LI C L, LIANG X Y, LU Y J, et al. RGB-T object tracking:Benchmark and baseline[J]. Pattern Recognition, 2019, 96:106977.
[14]	PENG P R, XU T F, HUANG B, et al. HAFNet: Hierarchical attentive fusion network for multispectral pedestrian detection[J]. Remote Sensing, 2023, 15(8):2041.
[15]	别倩, 王晓, 徐新, 等. 红外—可见光跨模态的行人检测综述[J]. 中国图象图形学报, 2023, 28(5): 1287-1307.
	BIE Q, WANG X, XU X, et al. 2023. Visible-infrared cross-modal pedestrian detection:a sum-mary[J]. Journal of Image and Graphics, 2023, 28(5):1287-1307. (in Chinese)
[16]	HWANG S, PARK J, KIM N, et al. Multispectral pedestrian detection: Benchmark dataset and baseline[C]// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, US: IEEE, 2015: 1037-1045.
[17]	LIU J J, ZHANG S T, WANG S, et al. Multispectral deep neural networks for pedestrian detection[C]//Proceedings of the British Machine Vision Conference.York, UK: BMVA Press, 2016:71-73.
[18]	SUN Y M, CAO B, ZHU P F, et al. Drone-based RGB-infrared cross-modality vehicle detection via uncertainty-aware learning[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(10): 6700-6713.
[19]	CHEN Y T, SHI J H, YE Z, et al. Multimodal object detection via probabilistic ensembling[C]//Proceedings of the Computer Vision-ECCV 2022: 17th European Conference. Tel Aviv, Israel: Springer, 2022: 139-158.
[20]	ZHENG Y, IZZAT I H, ZIAEE S. GFD-SSD: gated fusion double SSD for multispectral pedestrian detection:arXiv:1903.06999[R]. Ithaca,NY, US: Cornell University, 2019:1903.06999.
[21]	FANG Q Y, WANG Z K. Cross-modality attentive feature fusion for object detection in multispectral remote sensing imagery[J]. Pattern Recognition, 2022, 130: 108786.
[22]	WOLPERT A, TEUTSCH M, SARFRAZ M S, et al. Anchor-free small-scale multispectral pedestrian detection[C]//Proceedings of the 31st British Machine Vision Conference 2020. Virtual Event, UK: BMVA Press, 2020.
[23]	TIAN Z, SHEN C H, CHEN H, et al. FCOS:fully convolutional one-stage object detection[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Seoul, Korea: IEEE, 2019: 9626-9636.
[24]	CHEN K, WANG J Q, PANG J M, et al. MMDetection: Open mmlab detection toolbox and benchmark:arXiv:1906.07155[R]. Ithaca,NY, US: Cornell University, 2019:1906.07155.

数据集名称	图像对数量	装甲目标数量
VTAV数据集	4977	10244
训练集	3000	5784
验证集	977	2209
测试集	1000	2251

数据集名称	图像对数量	装甲目标数量
VTAV数据集	4977	10244
训练集	3000	5784
验证集	977	2209
测试集	1000	2251

融合检测结构	融合级别	卷积核尺寸		mAP@0.5/ %	mAP@0.5∶ 0.95/%
融合检测结构	融合级别	1×1	3×3	mAP@0.5/ %	mAP@0.5∶ 0.95/%
EFF	像素级	√		50.4	19.8
EFF	像素级		√	62.2	28.6
MFF	特征级	√		78.9	45.2
MFF	特征级		√	78.9	45.8
LFF	特征级	√		78.8	45.2
LFF	特征级		√	79.3	46.1

融合检测结构	融合级别	卷积核尺寸		mAP@0.5/ %	mAP@0.5∶ 0.95/%
融合检测结构	融合级别	1×1	3×3	mAP@0.5/ %	mAP@0.5∶ 0.95/%
EFF	像素级	√		50.4	19.8
EFF	像素级		√	62.2	28.6
MFF	特征级	√		78.9	45.2
MFF	特征级		√	78.9	45.8
LFF	特征级	√		78.8	45.2
LFF	特征级		√	79.3	46.1

融合检测结构	融合级别	BN+Relu	mAP@0.5/ %	mAP@0.5∶ 0.95/%
EFF	像素级		78.6	43.6
EFF	像素级	√	62.2	28.6
MFF	特征级		77.2	43.1
MFF	特征级	√	78.9	45.8
LFF	特征级		78.1	44.1
LFF	特征级	√	79.3	46.1