基于红外相机和毫米波雷达融合的烟雾遮挡无人驾驶车辆目标检测与跟踪

doi:10.12382/bgxb.2022.0602

摘要/Abstract

摘要：

战场环境下无人驾驶车辆的感知系统易受烟雾、扬尘等天气的影响,对关键目标的检测与跟踪能力大大下降,造成目标漏检、目标误检、目标丢失等严重后果。针对该问题,开发毫米波雷达和红外相机融合系统,采用目标级融合方式建立简洁有效的融合规则,提炼和组合各传感器的优势信息,最终输出稳定的目标感知结果。对毫米波雷达的目标进行有效性检验和提取,并提出改进的基于密度的含噪声空间聚类应用算法,以减少毫米波雷达噪音干扰。以YOLOv4网络为基础,引入MobileNetv2主干网络,在网络训练过程中运用迁移学习方法,同时对红外数据样本进行扩充,解决了红外图像训练样本少的问题。试验结果表明,相较于仅基于红外相机的算法,融合检测算法在烟雾环境下的精度显著提升,且算法实时性高,实现了烟雾环境下毫米波雷达与红外相机融合的目标检测与跟踪,提高了无人驾驶车辆目标检测与跟踪系统的抗烟雾干扰能力。

关键词: 无人驾驶车辆, 烟雾遮挡, 红外相机, 毫米波雷达, 目标检测, 目标跟踪, 改进YOLOv4网络

Abstract:

In the battlefield environment, the perception system of unmanned vehicle is susceptible to the influence of weather such as smoke and dust. The ability to detect and track key objects is greatly reduced under harsh weather conditions, resulting in serious consequences, such as object miss-detection, object misdetection and object missing. To address this problem, a fusion system of MMW radar and infrared camera is developed. The object-level fusion method is adopted to establish simple and effective fusion rules, extract and combine the dominant information from each sensor, and finally output stable objective perception results. The objects of MMW radar are checked and extracted. And an improved DBSCAN clustering algorithm is proposed to reduce the noise of MMW radar. The MobileNetv2 backbone network is introduced based on the YOLOv4 network. In the process of network training, the transfer learning method is used to expand the infrared data samples, which solves the problem of fewer training samples of infrared images. The experimental results show that the fusion algorithm has significantly better accuracy and high real-time performance in the smoke environment compared with the algorithm based on infrared camera only, which realizes the object detection and tracking of the fusion of MMW radar and infrared camera in the smoke environment, and improves the anti-interference ability of the object detection and tracking system of unmanned vehicles.

Key words: unmanned vehicle, smoke obscuring, infrared camera, MMW radar, object detection, object tracking, improved YOLOv4 algorithm

中图分类号:

TG156

熊光明, 罗震, 孙冬, 陶俊峰, 唐泽月, 吴超. 基于红外相机和毫米波雷达融合的烟雾遮挡无人驾驶车辆目标检测与跟踪[J]. 兵工学报, 2024, 45(3): 893-906.

XIONG Guangming, LUO Zhen, SUN Dong, TAO Junfeng, TANG Zeyue, WU Chao. Object Detection and Tracking for Unmanned Vehicles Based on Fusion of Infrared Camera and MMW Radar in Smoke-obscured Environment[J]. Acta Armamentarii, 2024, 45(3): 893-906.

图/表 15

图1 毫米波雷达与红外相机融合系统框架

Fig.1 Fusion system framework of MMW radar and infrared camera

图2 有效目标初选后的毫米波雷达二维点云图

Fig.2 2D point clouds of MMW radar after preliminary selection of effective objects

图3 自适应DBSCAN算法聚类后的毫米波雷达二维点云图

Fig.3 2D point clouds of MMW radar after clustering with adaptive DBSCAN algorithm

图4 改进YOLOv4网络结构

Fig.4 Structure of the improved YOLOv4 network

图5 部分采集红外图像数据

Fig.5 Partially collected infrared image data

图6 红外图像检测算法架构

Fig.6 Framework of infrared image detection algorithm

图7 红外图像变换效果

Fig.7 Infrared image transformation

图8 坐标系定义

Fig.8 Definition of coordinate system

表1 3种标定方法的标定误差对比

Table 1 Comparison of calibration errors of three calibration methods

算法	标定数据1	标定数据2	标定数据3	平均
直接法	42.1	40.5	41.5	41.4
最小二乘法	12.5	12.3	12.9	12.6
LM优化法	4.8	4.5	4.6	4.6

图9 多传感器目标融合流程

Fig.9 Flow chart of multi-sensor object fusion

图10 检测结果可视化

Fig.10 Visualization of detected results

表2 无人平台的传感器配置

Table 2 Sensor configuration of unmanned platform

传感器	型号	安装位置	作用
可见光相机	GT 1290/C	车前	目标检测
红外相机	Xsafe-II M系列	车前	烟雾环境下目标检测
毫米波雷达	行易道AMRR200	车前	烟雾环境下目标检测

表3 红外目标检测算法效果量化对比

Table 3 Quantitative comparison of infrared object detection algorithms

算法与网络	类别	P_Precision/%	P_Recall/%	F1	AP/%	mAP/%	FPS
YOLOv4-Tiny网络	car	84.01	84.64	0.84	89.03	82.31	63
	person	79.63	69.59	0.74	75.58
SSD-MobileNetv2算法	car	78.11	74.06	0.76	76.95	70.00	59
	person	75.32	60.57	0.67	63.06
改进YOLOv4网络	car	93.16	81.64	0.87	92.31	85.71	78
	person	89.09	58.88	0.72	79.11

图11 红外检测结果与真值

Fig.11 Detected result of infrared image and ground truth

表4 目标跟踪算法效果量化对比

Table 4 Quantitative comparison of object tracking algorithms

算法	环境	N^FP↓	N^FN↓	N^IDSW↓	ACC_MOTA↑/%
融合算法	非烟尘	30	264	24	71.00
仅基于红外相机的算法	非烟尘	72	336	48	61.40
融合算法	烟尘	33	215	20	70.80
仅基于红外相机的算法	烟尘	60	276	39	61.20

参考文献 25

[1]	陈慧岩, 张玉. 军用地面无人机动平台技术发展综述[J]. 兵工学报, 2014, 35(10):1696-1706. doi: 10.3969/j.issn.1000-1093.2014.10.026
	CHEN H Y, ZHANG Y. An overview of research on military unmanned ground vehicles[J]. Acta Armamentarii, 2014, 35(10): 1696-1706. (in Chinese)
[2]	杜佳, 宋春林. 一种改进的毫米波雷达多目标检测算法[J]. 通信技术, 2015, 48(7):808-813.
	DU J, SONG C L. A modified millimeter-wave radar multi-target detection algorithm[J]. Communications Technology, 2015, 48(7):808-813. (in Chinese)
[3]	WÖHLER C, SCHUMANN O, HAHN M, et al. Comparison of random forest and long short-term memory network performances in classification tasks using radar[C]//Proceedings of Sensor Data Fusion:Trends, Solutions, Applications. Bonn,Germany: IEEE, 2017.
[4]	HE Y Z, DENG B Y, WANG H J, et al. Infrared machine vision and infrared thermography with deep learning:a review[J]. Infrared Physics & Technology, 2021, 116(2):103754.
[5]	HUANG L X, LIANG Y, MENG L W, et al. Potential of visible and near-infrared hyperspectral imaging for detection of diaphania pyloalis larvae and damage on mulberry leaves[J]. Sensors, 2018, 18(7):2077. doi: 10.3390/s18072077 URL
[6]	DENG Z P, HAO S, ZHOU S L, et al. Toward fast and accurate vehicle detection in aerial images using coupled region-based convolutional neural networks[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2017, 99:1-13.
[7]	LIU F, HAN P L, WANG Y, et al. Super resolution reconstruction of infrared images based on classified dictionary learning[J]. Infrared Physics & Technology, 2018, 90:146-155.
[8]	KRISTO M, IVASIC K M, POBAR M. Thermal object detection in difficult weather conditions using YOLO[J]. IEEE Access, 2020, 99:1-1.
[9]	REDMON J, FARHADI A. An incremental improvement: arXiv:1804.02767v1[R/OL]. Ithaca,NY,US: Cornell University, (2018-04-08) [2020-08-06]. https://arxiv.org/abs/1804.02767.
[10]	梁杰, 李磊, 任君, 等. 基于深度学习的红外图像遮挡干扰检测方法[J]. 兵工学报, 2019, 40(7):1401-1410. doi: 10.3969/j.issn.1000-1093.2019.07.009
	LIANG J, LI L, REN J, et al. Infrared image occlusion interference detection method based on deep learning[J]. Acta Armamentarii, 2019, 40(7):1401-1410. (in Chinese) doi: 10.3969/j.issn.1000-1093.2019.07.009
[11]	GHASEMIAN N, AKHOONDZADEH M. Introducing two Random Forest based methods for cloud detection in remote sensing images[J]. Advances in Space Research, 2018, 62(2):288-303. doi: 10.1016/j.asr.2018.04.030 URL
[12]	JHA H, LODHI V, CHAKRAVARTY D. Object detection and identification using vision and radar data fusion system for ground-based navigation[C]// Proceedings of the 2019 6th International Conference on Signal Processing and Integrated Networks. Noida,India:IEEE, 2019: 590-593.
[13]	XIN B, TAN B, XU Z J, et al. A new method of target detection based on autonomous radar and camera data fusion[C]//Proceedings of Intelligent and Connected Vehicles Symposium. US: SAE, 2017. DOI:10.4271/2017-01-1977.
[14]	JHA H, LODHI V, CHAKRAVARTY D. Object detection and identification using vision and radar data fusion system for ground-based navigation[C]// Proceedings of the 2019 6th International Conference on Signal Processing and Integrated Networks. Noida,India:IEEE, 2019.
[15]	EVEEINGHAM M, ESLAMI S, GOOL L V, et al. The pascal visual object classes challenge: a retrospective[J]. International Journal of Computer Vision, 2015, 111(1):98-136. doi: 10.1007/s11263-014-0733-5 URL
[16]	GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? the KITTI vision benchmark suite[C]//Proceeding of IEEE Conference on Computer Vision & Pattern Recognition. Providence, RI, US: IEEE, 2012:3354-3361.
[17]	CAESAR H, VARUN B, LANG A H, et al. nuScenes:a multimodal dataset for autonomous driving[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle, WA,US:IEEE, 2020:11618-11628.
[18]	BOCHKOVSKIY A, WANG C Y, LIAO H. YOLOv4:optimal speed and accuracy of object detection: arXiv:2004.10934[R/OL]. Ithaca, NY, US: Cornell University, (2020-04-23) [2020-09-05]. https://arxiv.org/abs/2004.10934.
[19]	WANG C Y, LIAO H Y M, WU Y H, et al. CSPNet:a new backbone that can enhance learning capability of CNN[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Washington,D.C.,US: IEEE, 2020: 1571-1580.
[20]	SANDLER M, HOWARD A, ZHU M, et al. MobileNetV2: inverted residuals and linear bottlenecks[C]// Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. Salt Lake City, UT,US:IEEE, 2018:4510-4520.
[21]	孙冬. 基于红外相机和毫米雷达融合的目标检测与跟踪研究[D]. 北京: 北京理工大学, 2022.
	SUN D. Research on target detection and tracking based on fusion of infrared camera and millimeter radar[D]. Beijing: Beijing Institute of Technology, 2022. (in Chinese)
[22]	FLIR Thermal Dataset[EB/OL].(2021-05-10) [2022-05-18] https://www.flir.eu/oem/adas/adas-dataset-form/.
[23]	张海鸣, 龚建伟, 陈建松, 等. 非结构化环境下无人驾驶车辆跟驰方法[J]. 北京理工大学学报, 2019, 39(11):1126-1132.
	ZHANG H M, GONG J W, CHEN J S, et al. Study of autonomous vehicle following method in unstructed environment[J]. Transactions of Beijing Institute of Technology, 2019, 39(11): 1126-1132. (in Chinese)
[24]	TAO W, ZHENG N N, XIN J M, et al. Integrating millimeter wave radar with a monocular vision sensor for on-road obstacle detection Applications[J]. Sensors, 2011, 11(9):8992-9008. doi: 10.3390/s110908992 pmid: 22164117
[25]	MILAN A, LEAL-TAIXE L, REID I, et al. MOT16:a benchmark for multi-object tracking:arXiv:1603.00831[R/OL]. Ithaca,NY,US: Cornell University,(2016-05-02)[2020-06-07]. https://arxiv.org/abs/1603.00831v1.