An Algorithm of Battlefield Image Desmoking Based on Semantic Guidance and Contrastive Learning

doi:10.12382/bgxb.2022.0675

Abstract

Abstract:

Smoke, as the most common product of combat in modern warfare, reduces the visibility of combat scenarios inevitably, which in turn affects the performance of downstream military intelligence systems. Therefore, it is very important to restore the smoke-containing images. Existing algorithms usually ignore both the high-level semantic information in the image,and the degraded image itself can provide valuable supervision information for improving the smoke removal ability of network. Accordingly, a semantic-guidance and contrastive learning-based generative adversarial network (SCLGAN) is proposed to remove smoke from battlefield images. Specifically, semantic information is regarded as guidance to help the network better recover the structural and color information of images incorporating the high-level semantic features in low-level visual tasks. The contrastive learning paradigm is used to adopt clear image and smoke-containing image as positive and negative samples, and the contrastive regularization ensures that the restored image is pulled in closer to the clear image and pushed far away from the smoke-containing image. In addition, a smoke-containing battlefield dataset is first constructed to simulate the real smoke-containing battlefield scene, which promotes the development of related research. Experiments demonstrate that, compared with the existing smoke removal algorithms, the proposed algorithm can surpass the previous state-of-the-art methods in both quantitative and qualitative assessment.

Key words: military intelligence, image desmoking, generative adversarial network, semantic guidance, contrastive learning, attentive mechanism

CLC Number:

TP393.413

XIONG Jiamei, WANG Yongzhen, YAN Xuefeng, WEI Mingqiang. An Algorithm of Battlefield Image Desmoking Based on Semantic Guidance and Contrastive Learning[J]. Acta Armamentarii, 2024, 45(2): 671-683.

Figures/Tables 16

Fig.1 Examples of SBSI dataset

Table 1 Quantitative evaluation of the quality on SBSI dataset

数据集	迁移后的AuthESI↓	NIQE↓
HSTS	4.9600	4.962
真实战场含烟数据集	4.3442	3.452
筛选前的合成数据集	5.0339	3.746
SBSI	4.5824	3.503

Fig.2 Overview of SCLGAN for image desmoking

Fig.3 The architecture of generator

Fig.4 The architecture of discriminator

Table 2 Desmoking results of different algorithms on SBSI dataset

Table 3 Quantitative evaluation of different desmoking algorithms on SBSI dataset

去烟算法	PSNR/dB↑	SSIM↑	CIEDE2000↓
DCP	14.93	0.7796	13.3102
DSR-Net	20.19	0.8666	7.8871
AOD-Net	21.08	0.8787	6.8954
AttentiveGAN	21.54	0.8897	6.8333
LD-Net	20.94	0.9018	6.8645
GSR	24.45	0.9194	5.5566
PSD	22.47	0.9178	6.0656
YOLY	19.42	0.8502	9.7336
本文算法	30.14	0.9513	4.9969

Fig.5 Result visualization of main steps of SCLGAN algorithm

Table 4 Desmoking effects of different algorithms on the real-world smok-containing images

Table 5 Quantitative evaluation of different desmoking algorithms on the real-world battlefield smok-containing dataset

去雾算法	NIQE↓	σ↓	$r ¯$ ↑
DCP	3.443	0.0003	1.1893
DSR-Net	3.678	0.0003	1.2593
AOD-Net	3.590	0.0016	1.1766
AttentiveGAN	5.211	0.0088	1.3001
LDNet	3.675	0.0004	1.1182
GSR	5.409	0.0003	1.2732
PSD	3.580	0.0003	1.2565
YOLY	3.511	0.0110	1.8054
本文算法	3.422	0.0001	1.3793

Table 5 Quantitative evaluation of different desmoking algorithms on the real-world battlefield smok-containing dataset

去雾算法	NIQE↓	σ↓	$r ¯$ ↑
DCP	3.443	0.0003	1.1893
DSR-Net	3.678	0.0003	1.2593
AOD-Net	3.590	0.0016	1.1766
AttentiveGAN	5.211	0.0088	1.3001
LDNet	3.675	0.0004	1.1182
GSR	5.409	0.0003	1.2732
PSD	3.580	0.0003	1.2565
YOLY	3.511	0.0110	1.8054
本文算法	3.422	0.0001	1.3793

Fig.6 Desmoking results of battlefield images with thick smoke by SCLGAN algorithm

Table 6 Ablation result for SCLGAN with semantic guidance and contrastive learning

模型	基础模型	V1模型	V2模型	V3模型
感知对比学习损失	×	√	×	√
语义引导	×	×	√	√
指标	基础模型	V1模型	V2模型	V3模型
PSNR/dB↑	29.06	29.91	30.04	30.14
SSIM↑	0.928	0.951	0.944	0.951
CIEDE2000↓	5.942	6.089	5.064	4.997

Table 7 Ablation results of attention mechanism module

模型	PSNR/dB↑	SSIM↑	CIEDE2000↓
CA	27.15	0.930	6.440
FA	30.14	0.951	4.997
CBAM	29.38	0.951	5.365
FA+残差块-FA结构	29.40	0.949	6.883
FA+残差块-CA结构	29.59	0.950	5.384

Table 8 Ablation result of each loss

损失函数	C1	C2	C3	C4	C5	C6	C7
L1损失	√	×	×	√	√	×	√
MS-SSIM损失	×	√	×	√	×	√	√
对比正则损失	×	×	√	×	√	√	√
指标	C1	C2	C3	C4	C5	C6	C7
PSNR/dB↑	27.67	22.41	11.33	27.39	28.02	28.40	30.14
SSIM↑	0.909	0.465	0.085	0.899	0.867	0.909	0.951
CIEDE2000↓	5.772	9.915	29.104	5.985	6.958	8.444	4.997

Table 9 Ablation result for SCLGAN with semantic guidance and contrastive learning on real-world battlefield smok-cintaining dataset

模型	基础模型	V1模型	V2模型	V3模型
感知对比学习损失	×	√	×	√
语义引导	×	×	√	√
指标	基础模型	V1模型	V2模型	V3模型
NIQE↓	3.045	3.041	3.026	3.014
σ↓	0.0002	0.0003	0.0002	0.0001
$r ¯$ ↑	1.1020	1.1204	1.2323	1.1691

Table 9 Ablation result for SCLGAN with semantic guidance and contrastive learning on real-world battlefield smok-cintaining dataset

模型	基础模型	V1模型	V2模型	V3模型
感知对比学习损失	×	√	×	√
语义引导	×	×	√	√
指标	基础模型	V1模型	V2模型	V3模型
NIQE↓	3.045	3.041	3.026	3.014
σ↓	0.0002	0.0003	0.0002	0.0001
$r ¯$ ↑	1.1020	1.1204	1.2323	1.1691

Table 10 Average running time of different algorithms

去烟算法	平均运行时间/s
DCP	1.0389
DSR-Net	0.0692
AOD-Net	0.0685
AttentiveGAN	0.1431
LD-Net	0.0954
GSR	0.1647
PSD	0.5389
YOLY	41.5037
本文算法	0.0553

References 34

[1]	李力, 金伟其, 黄有为, 等. 一种低对比度可见光超视距成像的实时图像增强方法[J]. 兵工学报, 2010, 31(2): 242-247.
	LI L, JIN W Q, HUANG Y W, et al. Real-time image enhancement processing methods of low contrast visible light over-the-horizon imaging[J]. Acta Armamentarii, 2010, 31(2): 242-247. (in Chinese)
[2]	LI S, WANG S Y, ZHANG D, et al. Real-time smoke removal for the surveillance images under fire scenario[J]. Signal, Image and Video Processing, 2019, 13(5): 1037-1043. doi: 10.1007/s11760-019-01442-3
[3]	李庆忠, 刘清. 基于小波变换的低照度图像自适应增强算法[J]. 中国激光, 2015, 42(2): 272-278.
	LI Q Z, LIU Q. Adaptive enhancement algorithm for low illumination images based on wavelet transform[J]. Chinese Journal of Lasers, 2015, 42(2): 272-278. (in Chinese)
[4]	李春明, 姜雨彤, 宋海平, 等. 基于光学厚度代理模型的雾浓度估计及图像去雾[J]. 兵工学报, 2019, 40(7):1425-1433. doi: 10.3969/j.issn.1000-1093.2019.07.012
	LI C M, JIANG Y T, SONG H P, et al. Research on optical depth surrogate model-based method for estimating fog density and removing fog effect from images[J]. Acta Armamentarii, 2019, 40(7) : 1425-1433. (in Chinese) doi: 10.3969/j.issn.1000-1093.2019.07.012
[5]	姜雨彤, 宋海平, 王光辉. 基于质量评价最优的无人机航拍图像去雾方法[J]. 兵工学报, 2022, 43(1):148-158. doi: 10.3969/j.issn.1000-1093.2022.01.016
	JIANG Y T, SONG H P, WANG G H. Image dehazing based on the optimum of UAV aerial image quality evaluation[J]. Acta Armamentarii, 2022, 43(1): 148-158. (in Chinese) doi: 10.3969/j.issn.1000-1093.2022.01.016
[6]	HE K M, SUN J, TANG X O. Single image haze removal using dark channel prior[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 33(12): 2341-2353. doi: 10.1109/TPAMI.2010.168 URL
[7]	LI B Y, PENG X L, WANG Z Y, et al. AOD-Net: all-in-one dehazing network[C]//Proceedings of the IEEE international conference on computer vision. Venice, Italy:IEEE, 2017: 4770-4778.
[8]	BOLKAR S, WANG C C, CHEIKH F A, et al. Deep smoke removal from minimally invasive surgery videos[C]//Proceedings of IEEE International Conference on Image Processing. Athens, Greece: IEEE, 2018: 3403-3407.
[9]	MALAV R, KIM A, SAHOO S R, et al. DHSGAN: an end to end dehazing network for fog and smoke[C]//Proceedings of the 14th Asian Conference on Computer Vision. Perth, Australia:Springer, 2018: 593-608.
[10]	SIDOROV O, WANG C C, CHEIKH F A. Generative smoke removal: arXiv:1902.00311[R/OL]. Iathaca, NY, US: Cornell University, 2019(2019-12-04). https://doi.org/10.48550/arXiv.1902.00311.
[11]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal, Canada: MIT Press, 2014, 2:2672-2680.
[12]	HUANG Y F, CHEN X, XU L, et al. Single image desmoking via attentive generative adversarial network for smoke detection process[J]. Fire Technology, 2021, 57(6): 3021-3040. doi: 10.1007/s10694-021-01096-z
[13]	MCCARTNEY E J. Optics of the atmosphere: scattering by molecules and particles[M]. New York, NY, US:Wiley, 1976.
[14]	CHEN T, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations[C]//Proceedings of the 37th International conference on machine learning. Vienna, Austria:JMLR, 2020: 1597-1607.
[15]	WU H Y, QU Y Y, LIN S H, et al. Contrastive learning for compact single image dehazing[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, TN, US:IEEE, 2021:10551-10560.
[16]	QIN X, WANG Z L, BAI Y C, et al. FFA-Net: feature fusion attention network for single image dehazing[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 11908-11915. doi: 10.1609/aaai.v34i07.6865 URL
[17]	CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(4): 834-848. doi: 10.1109/TPAMI.2017.2699184 URL
[18]	ZHANG N, ZHANG L, CHENG Z X. Towards simulating foggy and hazy images and evaluating their authenticity[C]// Proceedings of the 24th International Conference on Neural Information Processing. Guangzhou, China: Springer, 2017, 10636: 405-415.
[19]	MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a “completely blind” image quality analyzer[J]. IEEE Signal processing letters, 2012, 20(3): 209-212. doi: 10.1109/LSP.2012.2227726 URL
[20]	LI B Y, REN W Q, FU D P, et al. Benchmarking single-image dehazing and beyond[J]. IEEE Transactions on Image Processing, 2018, 28(1): 492-505. doi: 10.1109/TIP.83 URL
[21]	WANG Y Z, YAN X F, GUAN D H, et al. Cycle-SNSPGAN: towards real-world image dehazing via cycle spectral normalized soft likelihood estimation patch GAN[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(11):20368-20382. doi: 10.1109/TITS.2022.3170328 URL
[22]	CHEN L C, ZHU Y K, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of the European conference on computer vision. Munich, Germany:Springer, 2018, 11211: 833-851.
[23]	HOWARD A G, ZHU M, CHEN B, et al. MobileNets: efficient convolutional neural networks for mobile vision applications: arXiv:1704.04861[R]. Iathaca, NY, US: Cornell University, 2017(2017-04-17). https://doi.org/10.48550/arXiv.1704.04861.
[24]	WANG Y B, LIN S H, QU Y Y, et al. Towards compact single image super-resolution via contrastive self-distillation[C]// Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Montreal, Canada: International Joint Conferences on Artificial Intelligence Organization, 2021:1122-1128.
[25]	JOHNSON J, ALAHI A, FEI-FEI L. Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of European conference on computer vision. Amsterdam, The Netherlands: Springer, 2016, 9906: 694-711.
[26]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition: arXiv:1409.1556[R/OL]. Iathaca, NY, US: Cornell University, 2015(2015-04-10). https://doi.org/10.48550/arXiv.1409.1556.
[27]	WANG Z, SIMONCELLI E P, BOVIK A C. Multiscale structural similarity for image quality assessment[C]//Proceedings of the Thrity-Seventh Asilomar Conference on Signals, Systems & Computers. Pacific Grove, CA, US:IEEE, 2003: 1398-1402.
[28]	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE transactions on image processing, 2004, 13(4): 600-612. doi: 10.1109/tip.2003.819861 pmid: 15376593
[29]	SHARMA G, WU W C, DALAL E N. The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations[J]. Color Research & Application, 2005, 30(1): 21-30.
[30]	ULLAH H, MUHAMMAD K, IRFAN M, et al. Light-DehazeNet: a novel lightweight CNN architecture for single image dehazing[J]. IEEE Transactions on Image Processing, 2021, 30: 8968-8982. doi: 10.1109/TIP.2021.3116790 URL
[31]	CHEN Z Y, WANG Y C, YANG Y, et al. PSD: principled synthetic-to-real dehazing guided by physical priors[C]//Proceedings of the 2021 IEEE/CVF conference on computer vision and pattern recognition. Nashville, TN, US:IEEE, 2021: 7176-7185.
[32]	LI B Y, GOU Y B, GU S H, et al. You only look yourself: Unsupervised and untrained single image dehazing neural network[J]. International Journal of Computer Vision, 2021, 129(5): 1754-1767. doi: 10.1007/s11263-021-01431-5
[33]	CHEN Z Y, JIANG T T, TIAN Y H. Quality assessment for comparing image enhancement algorithms[C]//Proceedings of 2014 IEEE conference on computer vision and pattern recognition. Columbus, OH, US:IEEE, 2014: 3003-3010.
[34]	WOO S, PARK J, LEE J Y, et al. Cbam: Convolutional block attention module[C]//Proceedings of the European conference on computer vision. Munich, Germany:Springer, 2018, 11211: 3-19.