Sand-dust Image Restoration Using Gray Compensation and Feature Fusion

doi:10.12382/bgxb.2022.0510

Abstract

Abstract:

Scattering and absorption of light by dust particles often result in sand-dust images with low contrast and significant color deviations, which can hinder the tracking performance of unmanned aerial vehicles and precision-guided ammunition for target recognition and tracking. Due to the complexity of scene structures, difficult parameter estimation and other factors, the existing sand-dust image restoration methods cannot effectively extract semantic components from images, resulting in unreal colors and blurred details of restored images. To address these issues, a two-stage sand dust image restoration framework consisting of gray compensation-based image pre-processing and feature fusion networks is proposed. The image pre-processing module compensates the gray distribution of the input sand images to recover latent scene information, producing two images with balanced color and clear contours. The fusion network then extracts and fuses high-dimensional features from different derived input images and restores high-quality images. The results show that high index statistics and good visual effects are obtained in the restored images, effectively improving detection and segmentation accuracy of sand-dust images.

Key words: gray compensation, feature fusion, derived input, image restoration, convolutional neural network, target recognition

CLC Number:

TP391.413

DING Bosheng, ZHANG Ruiheng, XU Lixin, CHEN Huiming. Sand-dust Image Restoration Using Gray Compensation and Feature Fusion[J]. Acta Armamentarii, 2023, 44(10): 3115-3126.

Figures/Tables 19

Fig.1 Gray histogram statistics of sand-dust image and clear image

Fig.2 Restored sand-dust image

Fig.3 Sand-dust image restoration frameworks

Fig.4 Images before and after gray compensation

Fig.5 Derived images after feature transformation

Fig.6 Feature fusion network frameworks

Table 1 Encoding-decoding network parameter settings

层	输入	输出	卷积核尺寸	滑动步长	填充圈数
卷积层1	6	64	7	1
下卷积2	64	128	3	2	1
下卷积3	128	256	3	2	1
残差1~9	256	256	3	1
上卷积4	256	128	3	1/2	1
上卷积5	128	64	3	1/2	1

Table 2 Discriminator network parameter settings

层	输入	输出	卷积核尺寸	滑动步长	填充圈数
卷积1	3	64	4	2	1
卷积2	64	128	4	2	1
卷积3	128	256	4	2	1
卷积4	256	512	4	1	1
卷积5	512	1	4	1	1

Table 3 Quantitative evaluation of different methods on TestA dataset

参数	沙尘图像	CCH^[5]	STME^[10]	DCP^[8]	VRB^[7]	Refine-Net^[22]	Cycle-GAN^[13]	CAL^[20]	YUV^[21]	本文方法
SSIM	0.5557	0.5934	0.513	0.5897	0.5953	0.613	0.5981	0.5885	0.615	0.7433
PSNR	10.9907	12.1304	11.805	11.2573	12.9054	16.855	13.5896	12.356	15.64	20.1885

Table 4 Quantitative evaluation of different methods on TestB dataset

参数	沙尘图像	CCH^[5]	STME^[10]	DCP^[8]	VRB^[7]	Refine-Net^[22]	Cycle-GAN^[13]	CAL^[20]	YUV^[21]	本文方法
NIQE	5.8695	6.2191	5.1063	5.6189	5.0804	5.141	6.0423	6.056	5.053	5.019
e		0.5683	0.753	1.099	0.3543	0.645	0.6381	0.554	0.736	1.3871
$r ¯$		0.6343	1.44	1.141	1.0826	1.387	1.5056	0.731	1.38	2.079

Table 4 Quantitative evaluation of different methods on TestB dataset

参数	沙尘图像	CCH^[5]	STME^[10]	DCP^[8]	VRB^[7]	Refine-Net^[22]	Cycle-GAN^[13]	CAL^[20]	YUV^[21]	本文方法
NIQE	5.8695	6.2191	5.1063	5.6189	5.0804	5.141	6.0423	6.056	5.053	5.019
e		0.5683	0.753	1.099	0.3543	0.645	0.6381	0.554	0.736	1.3871
$r ¯$		0.6343	1.44	1.141	1.0826	1.387	1.5056	0.731	1.38	2.079

Table 5 Visual effects of image restoration by different methods on TESTA

Table 6 Visual effects of image restoration by different methods on TESTB

Table 7 Pre-processing ablation test results

Table 8 Impact analysis of derived inputs

输入图像	I_CB	I_CE	I_CB+I_CE
PSNR	18.45	17.25	20.1885
SSIM	0.492	0.625	0.7433

Table 9 Quantitative analysis of different loss functions

损失函数	L_A	L_A+L_C	L_A+L_P	L_A+L_P+L_C
PSNR	9.4868	19.2175	15.701	20.1885
SSIM	0.0979	0.7195	0.673	0.7433

Table 10 Comparison of visual effects after training with different loss functions

Table 11 Comparison of object detection results using sand-dust and restored images

Table 12 Comparison of target segmentation effects using sand-dust and restored images

Table 13 Average running time of image restoration methods

序号	方法	运行平台	运行时间/s
1	DCP	数字仿真软件(CPU)	2.266
2	CCH	数字仿真软件(CPU)	1.280
3	Refine-Net	Pytorch软件(GPU)	0.415
4	Cycle-GAN	TensorFlow软件(GPU)	0.533
5	本文方法	Pytorch软件(GPU)	0.124

References 24

[1]	吴青青, 许廷发, 闫辉, 等. 复杂背景下的颜色分离背景差分目标检测方法[J]. 兵工学报, 2013, 34(4): 501-506. doi: 10. 3969/ j. issn. 1000-1093. 2013. 04. 019
	WU Q Q, XU T F, YAN H, et al. An improved color separation method for object detection in complex background[J]. Acta Armamentarii, 2013, 34(4): 501-506. (in Chinese)
[2]	马月红, 孔梦瑶. 基于改进快速区域卷积网络的目标检测轻量化算法[J]. 兵工学报, 2021, 42(12): 2664-2674. doi: 10.3969/j.issn.1000-1093.2021.12.014
	MA Y H, KONG M Y. A lightweight target detection algorithm based on the improved faster-RCNN[J]. Acta Armamentarii, 2021, 42(12): 2664-2674. (in Chinese) doi: 10.3969/j.issn.1000-1093.2021.12.014
[3]	XU G, WANG X T, XU X G. Single image enhancement in sandstorm weather via tensor least square[J]. IEEE/CAA Journal of Automatica Sinica, 2020, 7(6): 1649-1661. doi: 10.1109/JAS.2020.1003423 URL
[4]	CHENG Y Q, JIA Z H, LAI H C, et al. A fast sand-dust image enhancement algorithm by blue channel compensation and guided image filtering[J]. IEEE Access, 2020, 8(5): 196690-196699. doi: 10.1109/Access.6287639 URL
[5]	PARK T H, EOM I K. Sand-dust image enhancement using successive color balance with coincident chromatic histogram[J]. IEEE Access, 2021, 9(7): 19749-19760. doi: 10.1109/Access.6287639 URL
[6]	AL-AMEEN Z. Visibility enhancement for images captured in dusty weather via tuned tri-threshold fuzzy intensification operators[J]. International Journal of Intelligent Systems Technologies & Applications, 2016, 8(8): 10-17.
[7]	YANG Y, ZHANG C, LIU L L, et al. Visibility restoration of single image captured in dust and haze weather conditions[J]. Multidimensional Systems and Signal Processing, 2020, 31(2): 619-633. doi: 10.1007/s11045-019-00678-z
[8]	HE K M, SUN J, TANG X O. Single image haze removal using dark channel prior[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(12): 2341-2353. doi: 10.1109/TPAMI.2010.168 pmid: 20820075
[9]	SHI Z H, FENG Y N, ZHAO M H, et al. Let you see in sand dust weather:a method based on halo-reduced dark channel prior dehazing for sand-dust image enhancement[J]. IEEE Access, 2019, 7: 116722-116733. doi: 10.1109/Access.6287639 URL
[10]	KIM S E, PARKT H, EOM I K. Fast single image dehazing using saturation based transmission map estimation[J]. IEEE Transactions on Image Processing, 2019, 29: 1985-1998. doi: 10.1109/TIP.83 URL
[11]	于博文, 吕明. 改进的YOLOv3算法及其在军事目标检测中的应用[J]. 兵工学报, 2022, 43(2): 345-354. doi: 10.3969/j.issn.1000-1093.2022.02.012
	YU B W, LÜ M. Improved YOLOv3 algorithm and its application in military target detection[J]. Acta Armamentarii, 2022, 43(2): 345-354. (in Chinese) doi: 10.3969/j.issn.1000-1093.2022.02.012
[12]	薛灵芝, 曾向阳, 杨爽. 基于生成对抗网络的水声目标识别算法[J]. 兵工学报, 2021, 42(11): 2444-2452. doi: 10.3969/j.issn.1000-1093.2021.11.018
	XUE L Z, ZENG X Y, YANG S. Underwater acoustic target recognition algorithm based on generative adversarial networks[J]. Acta Armamentarii, 2021, 42(11): 2444-2452. (in Chinese) doi: 10.3969/j.issn.1000-1093.2021.11.018
[13]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 2242-2251.
[14]	REN W Q, MA L, ZHANG J W, et al. Gated fusion network for single image dehazing[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, US: IEEE, 2018.
[15]	ANCUTI C O, ANCUTI C. Single image dehazing by multi-scale fusion[J]. IEEE Transactions on Image Processing, 2013, 22(8): 3271-3282. pmid: 23674449
[16]	JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution[C]// Proceedings of European Conference on Computer Vision.Cham, Switzerland: Springer, 2016: 694-711.
[17]	FABBRI C, ISLAM M J, SATTAR J. Enhancing underwater imagery using generative adversarial networks[C]// Proceedings of 2018 IEEE International Conference on Robotics and Automation. Brisbane, QLD, Australia: IEEE, 2018: 7159-7165.
[18]	MITTAL A, SOUNDARAJAN R, BOVIK A C. Making a “completely blind” image quality analyzer[J]. IEEE Signal Processing Letters, 2012, 20(3): 209-212. doi: 10.1109/LSP.2012.2227726 URL
[19]	HAUTIERE N, TAREL J P, AUBERTD, et al. Blind contrast enhancement assessment by gradient ratioing at visible edges[J]. Image Analysis & Stereology, 2008, 27(2): 87-95.
[20]	SHI Z H, FENG Y N, ZHAO M H, et al. Normalised gamma transformation-based contrast-limited adaptive histogram equalisation with colour correction for sand-dust image enhancement[J]. IET Image Processing, 2020, 14(4): 747-756. doi: 10.1049/ipr2.v14.4 URL
[21]	GAO G X, LAI H C, LIU Y Q, et al. Sandstorm image enhancement based on YUV space[J]. Optik-International Journal for Light and Electron Optics, 2021, 226: 165659. doi: 10.1016/j.ijleo.2020.165659 URL
[22]	ZHAO S Y, ZHANG L, SHEN Y, et al. RefineDNet:a weakly supervised refinement framework for single image dehazing[J]. IEEE Transactions on Image Processing, 2021, 30: 3391-3404. doi: 10.1109/TIP.83 URL
[23]	REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN:towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2017, 39(6): 1137-1149.
[24]	HE K M, GKIOXARI G, DOLL P, et al. Mask R-CNN[C]// Proceedings of the 2017 IEEE international conference on computer vision. Venice, Italy:IEEE, 2017: 2961-2969.