基于灰度补偿和特征融合的沙尘图像修复方法

doi:10.12382/bgxb.2022.0510

摘要/Abstract

摘要：

沙尘颗粒对光的散射和吸收作用导致了沙尘天气下获取的可见光图像对比度低、颜色偏移严重,影响无人机、精确制导弹药对目标的识别与跟踪性能。由于场景结构复杂、参数估计困难等因素,现有的沙尘图像修复方法不能有效地提取图像中语义分量,导致修复后的图像颜色不真实、细节模糊。为此,提出基于灰度补偿的图像预处理模块和特征融合网络组成的沙尘图像修复框架。图像预处理模块对输入的沙尘图像进行灰度分布补偿来恢复场景中潜在的信息,并派生出颜色均衡和轮廓清晰的两个图像;融合网络对不同派生输入图像进行高维特征提取和融合,并恢复出高质量的图像。研究结果表明,所提出的方法修复图像取得了较高的指标统计值和良好的视觉效果,能有效提高沙尘图像的目标检测精度和分割准确率。

关键词: 灰度补偿, 特征融合, 派生输入, 图像修复, 卷积神经网络, 目标识别

Abstract:

Scattering and absorption of light by dust particles often result in sand-dust images with low contrast and significant color deviations, which can hinder the tracking performance of unmanned aerial vehicles and precision-guided ammunition for target recognition and tracking. Due to the complexity of scene structures, difficult parameter estimation and other factors, the existing sand-dust image restoration methods cannot effectively extract semantic components from images, resulting in unreal colors and blurred details of restored images. To address these issues, a two-stage sand dust image restoration framework consisting of gray compensation-based image pre-processing and feature fusion networks is proposed. The image pre-processing module compensates the gray distribution of the input sand images to recover latent scene information, producing two images with balanced color and clear contours. The fusion network then extracts and fuses high-dimensional features from different derived input images and restores high-quality images. The results show that high index statistics and good visual effects are obtained in the restored images, effectively improving detection and segmentation accuracy of sand-dust images.

Key words: gray compensation, feature fusion, derived input, image restoration, convolutional neural network, target recognition

中图分类号:

TP391.413

丁伯圣, 张睿恒, 徐立新, 陈慧敏. 基于灰度补偿和特征融合的沙尘图像修复方法[J]. 兵工学报, 2023, 44(10): 3115-3126.

DING Bosheng, ZHANG Ruiheng, XU Lixin, CHEN Huiming. Sand-dust Image Restoration Using Gray Compensation and Feature Fusion[J]. Acta Armamentarii, 2023, 44(10): 3115-3126.

图/表 19

图1 沙尘图像和对应清晰图像的灰度直方图统计

Fig.1 Gray histogram statistics of sand-dust image and clear image

图2 沙尘图像修复效果图

Fig.2 Restored sand-dust image

图3 沙尘图像修复框图

Fig.3 Sand-dust image restoration frameworks

图4 灰度补偿前后的图像示例

Fig.4 Images before and after gray compensation

图5 特征转换后的派生图像

Fig.5 Derived images after feature transformation

图6 特征融合网络框架

Fig.6 Feature fusion network frameworks

表1 编码-解码网络参数设置

Table 1 Encoding-decoding network parameter settings

层	输入	输出	卷积核尺寸	滑动步长	填充圈数
卷积层1	6	64	7	1
下卷积2	64	128	3	2	1
下卷积3	128	256	3	2	1
残差1~9	256	256	3	1
上卷积4	256	128	3	1/2	1
上卷积5	128	64	3	1/2	1

表2 鉴别器网络参数设置

Table 2 Discriminator network parameter settings

层	输入	输出	卷积核尺寸	滑动步长	填充圈数
卷积1	3	64	4	2	1
卷积2	64	128	4	2	1
卷积3	128	256	4	2	1
卷积4	256	512	4	1	1
卷积5	512	1	4	1	1

表3 不同方法在TestA测试集上的定量评估

Table 3 Quantitative evaluation of different methods on TestA dataset

参数	沙尘图像	CCH^[5]	STME^[10]	DCP^[8]	VRB^[7]	Refine-Net^[22]	Cycle-GAN^[13]	CAL^[20]	YUV^[21]	本文方法
SSIM	0.5557	0.5934	0.513	0.5897	0.5953	0.613	0.5981	0.5885	0.615	0.7433
PSNR	10.9907	12.1304	11.805	11.2573	12.9054	16.855	13.5896	12.356	15.64	20.1885

表4 不同方法在TestB测试集上的定量评估

Table 4 Quantitative evaluation of different methods on TestB dataset

参数	沙尘图像	CCH^[5]	STME^[10]	DCP^[8]	VRB^[7]	Refine-Net^[22]	Cycle-GAN^[13]	CAL^[20]	YUV^[21]	本文方法
NIQE	5.8695	6.2191	5.1063	5.6189	5.0804	5.141	6.0423	6.056	5.053	5.019
e		0.5683	0.753	1.099	0.3543	0.645	0.6381	0.554	0.736	1.3871
$r ¯$		0.6343	1.44	1.141	1.0826	1.387	1.5056	0.731	1.38	2.079

表4 不同方法在TestB测试集上的定量评估

Table 4 Quantitative evaluation of different methods on TestB dataset

参数	沙尘图像	CCH^[5]	STME^[10]	DCP^[8]	VRB^[7]	Refine-Net^[22]	Cycle-GAN^[13]	CAL^[20]	YUV^[21]	本文方法
NIQE	5.8695	6.2191	5.1063	5.6189	5.0804	5.141	6.0423	6.056	5.053	5.019
e		0.5683	0.753	1.099	0.3543	0.645	0.6381	0.554	0.736	1.3871
$r ¯$		0.6343	1.44	1.141	1.0826	1.387	1.5056	0.731	1.38	2.079

表5 不同方法在TESTA上的修复视觉效果

Table 5 Visual effects of image restoration by different methods on TESTA

表6 不同方法在TESTB上的修复视觉效果

Table 6 Visual effects of image restoration by different methods on TESTB

表7 预处理消融实验结果

Table 7 Pre-processing ablation test results

表8 派生输入的影响分析

Table 8 Impact analysis of derived inputs

输入图像	I_CB	I_CE	I_CB+I_CE
PSNR	18.45	17.25	20.1885
SSIM	0.492	0.625	0.7433

表9 不同损失函数的定量分析

Table 9 Quantitative analysis of different loss functions

损失函数	L_A	L_A+L_C	L_A+L_P	L_A+L_P+L_C
PSNR	9.4868	19.2175	15.701	20.1885
SSIM	0.0979	0.7195	0.673	0.7433

表10 不同损失函数训练的可视觉效果比较

Table 10 Comparison of visual effects after training with different loss functions

表11 沙尘图像及修复图像的目标检测结果比较

Table 11 Comparison of object detection results using sand-dust and restored images

表12 沙尘图像及修复图像的目标分割效果比较

Table 12 Comparison of target segmentation effects using sand-dust and restored images

表13 图像修复平均运行时间

Table 13 Average running time of image restoration methods

序号	方法	运行平台	运行时间/s
1	DCP	数字仿真软件(CPU)	2.266
2	CCH	数字仿真软件(CPU)	1.280
3	Refine-Net	Pytorch软件(GPU)	0.415
4	Cycle-GAN	TensorFlow软件(GPU)	0.533
5	本文方法	Pytorch软件(GPU)	0.124

参考文献 24

[1]	吴青青, 许廷发, 闫辉, 等. 复杂背景下的颜色分离背景差分目标检测方法[J]. 兵工学报, 2013, 34(4): 501-506. doi: 10. 3969/ j. issn. 1000-1093. 2013. 04. 019
	WU Q Q, XU T F, YAN H, et al. An improved color separation method for object detection in complex background[J]. Acta Armamentarii, 2013, 34(4): 501-506. (in Chinese)
[2]	马月红, 孔梦瑶. 基于改进快速区域卷积网络的目标检测轻量化算法[J]. 兵工学报, 2021, 42(12): 2664-2674. doi: 10.3969/j.issn.1000-1093.2021.12.014
	MA Y H, KONG M Y. A lightweight target detection algorithm based on the improved faster-RCNN[J]. Acta Armamentarii, 2021, 42(12): 2664-2674. (in Chinese) doi: 10.3969/j.issn.1000-1093.2021.12.014
[3]	XU G, WANG X T, XU X G. Single image enhancement in sandstorm weather via tensor least square[J]. IEEE/CAA Journal of Automatica Sinica, 2020, 7(6): 1649-1661. doi: 10.1109/JAS.2020.1003423 URL
[4]	CHENG Y Q, JIA Z H, LAI H C, et al. A fast sand-dust image enhancement algorithm by blue channel compensation and guided image filtering[J]. IEEE Access, 2020, 8(5): 196690-196699. doi: 10.1109/Access.6287639 URL
[5]	PARK T H, EOM I K. Sand-dust image enhancement using successive color balance with coincident chromatic histogram[J]. IEEE Access, 2021, 9(7): 19749-19760. doi: 10.1109/Access.6287639 URL
[6]	AL-AMEEN Z. Visibility enhancement for images captured in dusty weather via tuned tri-threshold fuzzy intensification operators[J]. International Journal of Intelligent Systems Technologies & Applications, 2016, 8(8): 10-17.
[7]	YANG Y, ZHANG C, LIU L L, et al. Visibility restoration of single image captured in dust and haze weather conditions[J]. Multidimensional Systems and Signal Processing, 2020, 31(2): 619-633. doi: 10.1007/s11045-019-00678-z
[8]	HE K M, SUN J, TANG X O. Single image haze removal using dark channel prior[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(12): 2341-2353. doi: 10.1109/TPAMI.2010.168 pmid: 20820075
[9]	SHI Z H, FENG Y N, ZHAO M H, et al. Let you see in sand dust weather:a method based on halo-reduced dark channel prior dehazing for sand-dust image enhancement[J]. IEEE Access, 2019, 7: 116722-116733. doi: 10.1109/Access.6287639 URL
[10]	KIM S E, PARKT H, EOM I K. Fast single image dehazing using saturation based transmission map estimation[J]. IEEE Transactions on Image Processing, 2019, 29: 1985-1998. doi: 10.1109/TIP.83 URL
[11]	于博文, 吕明. 改进的YOLOv3算法及其在军事目标检测中的应用[J]. 兵工学报, 2022, 43(2): 345-354. doi: 10.3969/j.issn.1000-1093.2022.02.012
	YU B W, LÜ M. Improved YOLOv3 algorithm and its application in military target detection[J]. Acta Armamentarii, 2022, 43(2): 345-354. (in Chinese) doi: 10.3969/j.issn.1000-1093.2022.02.012
[12]	薛灵芝, 曾向阳, 杨爽. 基于生成对抗网络的水声目标识别算法[J]. 兵工学报, 2021, 42(11): 2444-2452. doi: 10.3969/j.issn.1000-1093.2021.11.018
	XUE L Z, ZENG X Y, YANG S. Underwater acoustic target recognition algorithm based on generative adversarial networks[J]. Acta Armamentarii, 2021, 42(11): 2444-2452. (in Chinese) doi: 10.3969/j.issn.1000-1093.2021.11.018
[13]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 2242-2251.
[14]	REN W Q, MA L, ZHANG J W, et al. Gated fusion network for single image dehazing[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, US: IEEE, 2018.
[15]	ANCUTI C O, ANCUTI C. Single image dehazing by multi-scale fusion[J]. IEEE Transactions on Image Processing, 2013, 22(8): 3271-3282. pmid: 23674449
[16]	JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution[C]// Proceedings of European Conference on Computer Vision.Cham, Switzerland: Springer, 2016: 694-711.
[17]	FABBRI C, ISLAM M J, SATTAR J. Enhancing underwater imagery using generative adversarial networks[C]// Proceedings of 2018 IEEE International Conference on Robotics and Automation. Brisbane, QLD, Australia: IEEE, 2018: 7159-7165.
[18]	MITTAL A, SOUNDARAJAN R, BOVIK A C. Making a “completely blind” image quality analyzer[J]. IEEE Signal Processing Letters, 2012, 20(3): 209-212. doi: 10.1109/LSP.2012.2227726 URL
[19]	HAUTIERE N, TAREL J P, AUBERTD, et al. Blind contrast enhancement assessment by gradient ratioing at visible edges[J]. Image Analysis & Stereology, 2008, 27(2): 87-95.
[20]	SHI Z H, FENG Y N, ZHAO M H, et al. Normalised gamma transformation-based contrast-limited adaptive histogram equalisation with colour correction for sand-dust image enhancement[J]. IET Image Processing, 2020, 14(4): 747-756. doi: 10.1049/ipr2.v14.4 URL
[21]	GAO G X, LAI H C, LIU Y Q, et al. Sandstorm image enhancement based on YUV space[J]. Optik-International Journal for Light and Electron Optics, 2021, 226: 165659. doi: 10.1016/j.ijleo.2020.165659 URL
[22]	ZHAO S Y, ZHANG L, SHEN Y, et al. RefineDNet:a weakly supervised refinement framework for single image dehazing[J]. IEEE Transactions on Image Processing, 2021, 30: 3391-3404. doi: 10.1109/TIP.83 URL
[23]	REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN:towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2017, 39(6): 1137-1149.
[24]	HE K M, GKIOXARI G, DOLL P, et al. Mask R-CNN[C]// Proceedings of the 2017 IEEE international conference on computer vision. Venice, Italy:IEEE, 2017: 2961-2969.