一种基于语义引导和对比学习的战场图像去烟算法

doi:10.12382/bgxb.2022.0675

摘要/Abstract

摘要：

烟雾作为现代战争中最常见的作战产物,不可避免地会降低作战场景的可视性,进而影响下游军事智能系统的性能,因此对含烟图像进行复原处理非常重要,现有算法通常忽略图像中高阶的语义信息和降质图像本身都可以为提高网络去烟能力提供有价值的监督信息。为此,提出一种基于语义引导和对比学习的生成对抗网络来去除战场图像中的烟雾。通过在低阶视觉任务中融入高阶语义特征,将语义信息作为引导帮助网络更好地恢复图像的结构和色彩信息;利用对比学习范式将清晰和含烟图像构建为正、负样本,并采用对比约束使去烟后的图像与清晰图像接近,并远离含烟图像。此外,为模拟真实的战场含烟场景,首次构建一套含烟战场数据集,推进了相关研究的发展。实验结果表明,与现有图像去烟算法相比,所提方法在定量和定性指标上均达到了先进水平。

关键词: 军事智能, 图像去烟, 生成对抗网络, 语义引导, 对比学习, 注意力机制

Abstract:

Smoke, as the most common product of combat in modern warfare, reduces the visibility of combat scenarios inevitably, which in turn affects the performance of downstream military intelligence systems. Therefore, it is very important to restore the smoke-containing images. Existing algorithms usually ignore both the high-level semantic information in the image,and the degraded image itself can provide valuable supervision information for improving the smoke removal ability of network. Accordingly, a semantic-guidance and contrastive learning-based generative adversarial network (SCLGAN) is proposed to remove smoke from battlefield images. Specifically, semantic information is regarded as guidance to help the network better recover the structural and color information of images incorporating the high-level semantic features in low-level visual tasks. The contrastive learning paradigm is used to adopt clear image and smoke-containing image as positive and negative samples, and the contrastive regularization ensures that the restored image is pulled in closer to the clear image and pushed far away from the smoke-containing image. In addition, a smoke-containing battlefield dataset is first constructed to simulate the real smoke-containing battlefield scene, which promotes the development of related research. Experiments demonstrate that, compared with the existing smoke removal algorithms, the proposed algorithm can surpass the previous state-of-the-art methods in both quantitative and qualitative assessment.

Key words: military intelligence, image desmoking, generative adversarial network, semantic guidance, contrastive learning, attentive mechanism

中图分类号:

TP393.413

熊佳梅, 王永振, 燕雪峰, 魏明强. 一种基于语义引导和对比学习的战场图像去烟算法[J]. 兵工学报, 2024, 45(2): 671-683.

XIONG Jiamei, WANG Yongzhen, YAN Xuefeng, WEI Mingqiang. An Algorithm of Battlefield Image Desmoking Based on Semantic Guidance and Contrastive Learning[J]. Acta Armamentarii, 2024, 45(2): 671-683.

图/表 16

图1 合成战场含烟图像数据集示例

Fig.1 Examples of SBSI dataset

表1 SBSI数据集质量定量评估

Table 1 Quantitative evaluation of the quality on SBSI dataset

数据集	迁移后的AuthESI↓	NIQE↓
HSTS	4.9600	4.962
真实战场含烟数据集	4.3442	3.452
筛选前的合成数据集	5.0339	3.746
SBSI	4.5824	3.503

图2 SCLGAN图像去烟算法整体结构

Fig.2 Overview of SCLGAN for image desmoking

图3 生成器结构

Fig.3 The architecture of generator

图4 判别器结构

Fig.4 The architecture of discriminator

表2 不同算法在SBSI数据集上的去烟结果比较

Table 2 Desmoking results of different algorithms on SBSI dataset

表3 不同去烟算法在SBSI数据集上的定量比较

Table 3 Quantitative evaluation of different desmoking algorithms on SBSI dataset

去烟算法	PSNR/dB↑	SSIM↑	CIEDE2000↓
DCP	14.93	0.7796	13.3102
DSR-Net	20.19	0.8666	7.8871
AOD-Net	21.08	0.8787	6.8954
AttentiveGAN	21.54	0.8897	6.8333
LD-Net	20.94	0.9018	6.8645
GSR	24.45	0.9194	5.5566
PSD	22.47	0.9178	6.0656
YOLY	19.42	0.8502	9.7336
本文算法	30.14	0.9513	4.9969

图5 SCLGAN 算法主要步骤结果

Fig.5 Result visualization of main steps of SCLGAN algorithm

表4 不同算法在真实战场含烟图像上的去烟效果比较

Table 4 Desmoking effects of different algorithms on the real-world smok-containing images

表5 不同去烟算法在真实战场含烟图像上的定量评估

Table 5 Quantitative evaluation of different desmoking algorithms on the real-world battlefield smok-containing dataset

去雾算法	NIQE↓	σ↓	$r ¯$ ↑
DCP	3.443	0.0003	1.1893
DSR-Net	3.678	0.0003	1.2593
AOD-Net	3.590	0.0016	1.1766
AttentiveGAN	5.211	0.0088	1.3001
LDNet	3.675	0.0004	1.1182
GSR	5.409	0.0003	1.2732
PSD	3.580	0.0003	1.2565
YOLY	3.511	0.0110	1.8054
本文算法	3.422	0.0001	1.3793

表5 不同去烟算法在真实战场含烟图像上的定量评估

Table 5 Quantitative evaluation of different desmoking algorithms on the real-world battlefield smok-containing dataset

去雾算法	NIQE↓	σ↓	$r ¯$ ↑
DCP	3.443	0.0003	1.1893
DSR-Net	3.678	0.0003	1.2593
AOD-Net	3.590	0.0016	1.1766
AttentiveGAN	5.211	0.0088	1.3001
LDNet	3.675	0.0004	1.1182
GSR	5.409	0.0003	1.2732
PSD	3.580	0.0003	1.2565
YOLY	3.511	0.0110	1.8054
本文算法	3.422	0.0001	1.3793

图6 SCLGAN 算法含浓烟的战场图像去烟结果

Fig.6 Desmoking results of battlefield images with thick smoke by SCLGAN algorithm

表6 语义引导模块和对比学习模块的消融实验结果

Table 6 Ablation result for SCLGAN with semantic guidance and contrastive learning

模型	基础模型	V1模型	V2模型	V3模型
感知对比学习损失	×	√	×	√
语义引导	×	×	√	√
指标	基础模型	V1模型	V2模型	V3模型
PSNR/dB↑	29.06	29.91	30.04	30.14
SSIM↑	0.928	0.951	0.944	0.951
CIEDE2000↓	5.942	6.089	5.064	4.997

表7 注意力机制模块的消融实验结果

Table 7 Ablation results of attention mechanism module

模型	PSNR/dB↑	SSIM↑	CIEDE2000↓
CA	27.15	0.930	6.440
FA	30.14	0.951	4.997
CBAM	29.38	0.951	5.365
FA+残差块-FA结构	29.40	0.949	6.883
FA+残差块-CA结构	29.59	0.950	5.384

表8 各损失函数分量的消融实验结果

Table 8 Ablation result of each loss

损失函数	C1	C2	C3	C4	C5	C6	C7
L1损失	√	×	×	√	√	×	√
MS-SSIM损失	×	√	×	√	×	√	√
对比正则损失	×	×	√	×	√	√	√
指标	C1	C2	C3	C4	C5	C6	C7
PSNR/dB↑	27.67	22.41	11.33	27.39	28.02	28.40	30.14
SSIM↑	0.909	0.465	0.085	0.899	0.867	0.909	0.951
CIEDE2000↓	5.772	9.915	29.104	5.985	6.958	8.444	4.997

表9 在真实战场含烟数据集上语义引导模块和对比学习模块的消融实验结果

Table 9 Ablation result for SCLGAN with semantic guidance and contrastive learning on real-world battlefield smok-cintaining dataset

模型	基础模型	V1模型	V2模型	V3模型
感知对比学习损失	×	√	×	√
语义引导	×	×	√	√
指标	基础模型	V1模型	V2模型	V3模型
NIQE↓	3.045	3.041	3.026	3.014
σ↓	0.0002	0.0003	0.0002	0.0001
$r ¯$ ↑	1.1020	1.1204	1.2323	1.1691

表9 在真实战场含烟数据集上语义引导模块和对比学习模块的消融实验结果

Table 9 Ablation result for SCLGAN with semantic guidance and contrastive learning on real-world battlefield smok-cintaining dataset

模型	基础模型	V1模型	V2模型	V3模型
感知对比学习损失	×	√	×	√
语义引导	×	×	√	√
指标	基础模型	V1模型	V2模型	V3模型
NIQE↓	3.045	3.041	3.026	3.014
σ↓	0.0002	0.0003	0.0002	0.0001
$r ¯$ ↑	1.1020	1.1204	1.2323	1.1691

表10 不同算法的平均运行时间

Table 10 Average running time of different algorithms

去烟算法	平均运行时间/s
DCP	1.0389
DSR-Net	0.0692
AOD-Net	0.0685
AttentiveGAN	0.1431
LD-Net	0.0954
GSR	0.1647
PSD	0.5389
YOLY	41.5037
本文算法	0.0553

参考文献 34

[1]	李力, 金伟其, 黄有为, 等. 一种低对比度可见光超视距成像的实时图像增强方法[J]. 兵工学报, 2010, 31(2): 242-247.
	LI L, JIN W Q, HUANG Y W, et al. Real-time image enhancement processing methods of low contrast visible light over-the-horizon imaging[J]. Acta Armamentarii, 2010, 31(2): 242-247. (in Chinese)
[2]	LI S, WANG S Y, ZHANG D, et al. Real-time smoke removal for the surveillance images under fire scenario[J]. Signal, Image and Video Processing, 2019, 13(5): 1037-1043. doi: 10.1007/s11760-019-01442-3
[3]	李庆忠, 刘清. 基于小波变换的低照度图像自适应增强算法[J]. 中国激光, 2015, 42(2): 272-278.
	LI Q Z, LIU Q. Adaptive enhancement algorithm for low illumination images based on wavelet transform[J]. Chinese Journal of Lasers, 2015, 42(2): 272-278. (in Chinese)
[4]	李春明, 姜雨彤, 宋海平, 等. 基于光学厚度代理模型的雾浓度估计及图像去雾[J]. 兵工学报, 2019, 40(7):1425-1433. doi: 10.3969/j.issn.1000-1093.2019.07.012
	LI C M, JIANG Y T, SONG H P, et al. Research on optical depth surrogate model-based method for estimating fog density and removing fog effect from images[J]. Acta Armamentarii, 2019, 40(7) : 1425-1433. (in Chinese) doi: 10.3969/j.issn.1000-1093.2019.07.012
[5]	姜雨彤, 宋海平, 王光辉. 基于质量评价最优的无人机航拍图像去雾方法[J]. 兵工学报, 2022, 43(1):148-158. doi: 10.3969/j.issn.1000-1093.2022.01.016
	JIANG Y T, SONG H P, WANG G H. Image dehazing based on the optimum of UAV aerial image quality evaluation[J]. Acta Armamentarii, 2022, 43(1): 148-158. (in Chinese) doi: 10.3969/j.issn.1000-1093.2022.01.016
[6]	HE K M, SUN J, TANG X O. Single image haze removal using dark channel prior[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 33(12): 2341-2353. doi: 10.1109/TPAMI.2010.168 URL
[7]	LI B Y, PENG X L, WANG Z Y, et al. AOD-Net: all-in-one dehazing network[C]//Proceedings of the IEEE international conference on computer vision. Venice, Italy:IEEE, 2017: 4770-4778.
[8]	BOLKAR S, WANG C C, CHEIKH F A, et al. Deep smoke removal from minimally invasive surgery videos[C]//Proceedings of IEEE International Conference on Image Processing. Athens, Greece: IEEE, 2018: 3403-3407.
[9]	MALAV R, KIM A, SAHOO S R, et al. DHSGAN: an end to end dehazing network for fog and smoke[C]//Proceedings of the 14th Asian Conference on Computer Vision. Perth, Australia:Springer, 2018: 593-608.
[10]	SIDOROV O, WANG C C, CHEIKH F A. Generative smoke removal: arXiv:1902.00311[R/OL]. Iathaca, NY, US: Cornell University, 2019(2019-12-04). https://doi.org/10.48550/arXiv.1902.00311.
[11]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal, Canada: MIT Press, 2014, 2:2672-2680.
[12]	HUANG Y F, CHEN X, XU L, et al. Single image desmoking via attentive generative adversarial network for smoke detection process[J]. Fire Technology, 2021, 57(6): 3021-3040. doi: 10.1007/s10694-021-01096-z
[13]	MCCARTNEY E J. Optics of the atmosphere: scattering by molecules and particles[M]. New York, NY, US:Wiley, 1976.
[14]	CHEN T, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations[C]//Proceedings of the 37th International conference on machine learning. Vienna, Austria:JMLR, 2020: 1597-1607.
[15]	WU H Y, QU Y Y, LIN S H, et al. Contrastive learning for compact single image dehazing[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, TN, US:IEEE, 2021:10551-10560.
[16]	QIN X, WANG Z L, BAI Y C, et al. FFA-Net: feature fusion attention network for single image dehazing[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 11908-11915. doi: 10.1609/aaai.v34i07.6865 URL
[17]	CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(4): 834-848. doi: 10.1109/TPAMI.2017.2699184 URL
[18]	ZHANG N, ZHANG L, CHENG Z X. Towards simulating foggy and hazy images and evaluating their authenticity[C]// Proceedings of the 24th International Conference on Neural Information Processing. Guangzhou, China: Springer, 2017, 10636: 405-415.
[19]	MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a “completely blind” image quality analyzer[J]. IEEE Signal processing letters, 2012, 20(3): 209-212. doi: 10.1109/LSP.2012.2227726 URL
[20]	LI B Y, REN W Q, FU D P, et al. Benchmarking single-image dehazing and beyond[J]. IEEE Transactions on Image Processing, 2018, 28(1): 492-505. doi: 10.1109/TIP.83 URL
[21]	WANG Y Z, YAN X F, GUAN D H, et al. Cycle-SNSPGAN: towards real-world image dehazing via cycle spectral normalized soft likelihood estimation patch GAN[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(11):20368-20382. doi: 10.1109/TITS.2022.3170328 URL
[22]	CHEN L C, ZHU Y K, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of the European conference on computer vision. Munich, Germany:Springer, 2018, 11211: 833-851.
[23]	HOWARD A G, ZHU M, CHEN B, et al. MobileNets: efficient convolutional neural networks for mobile vision applications: arXiv:1704.04861[R]. Iathaca, NY, US: Cornell University, 2017(2017-04-17). https://doi.org/10.48550/arXiv.1704.04861.
[24]	WANG Y B, LIN S H, QU Y Y, et al. Towards compact single image super-resolution via contrastive self-distillation[C]// Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Montreal, Canada: International Joint Conferences on Artificial Intelligence Organization, 2021:1122-1128.
[25]	JOHNSON J, ALAHI A, FEI-FEI L. Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of European conference on computer vision. Amsterdam, The Netherlands: Springer, 2016, 9906: 694-711.
[26]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition: arXiv:1409.1556[R/OL]. Iathaca, NY, US: Cornell University, 2015(2015-04-10). https://doi.org/10.48550/arXiv.1409.1556.
[27]	WANG Z, SIMONCELLI E P, BOVIK A C. Multiscale structural similarity for image quality assessment[C]//Proceedings of the Thrity-Seventh Asilomar Conference on Signals, Systems & Computers. Pacific Grove, CA, US:IEEE, 2003: 1398-1402.
[28]	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE transactions on image processing, 2004, 13(4): 600-612. doi: 10.1109/tip.2003.819861 pmid: 15376593
[29]	SHARMA G, WU W C, DALAL E N. The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations[J]. Color Research & Application, 2005, 30(1): 21-30.
[30]	ULLAH H, MUHAMMAD K, IRFAN M, et al. Light-DehazeNet: a novel lightweight CNN architecture for single image dehazing[J]. IEEE Transactions on Image Processing, 2021, 30: 8968-8982. doi: 10.1109/TIP.2021.3116790 URL
[31]	CHEN Z Y, WANG Y C, YANG Y, et al. PSD: principled synthetic-to-real dehazing guided by physical priors[C]//Proceedings of the 2021 IEEE/CVF conference on computer vision and pattern recognition. Nashville, TN, US:IEEE, 2021: 7176-7185.
[32]	LI B Y, GOU Y B, GU S H, et al. You only look yourself: Unsupervised and untrained single image dehazing neural network[J]. International Journal of Computer Vision, 2021, 129(5): 1754-1767. doi: 10.1007/s11263-021-01431-5
[33]	CHEN Z Y, JIANG T T, TIAN Y H. Quality assessment for comparing image enhancement algorithms[C]//Proceedings of 2014 IEEE conference on computer vision and pattern recognition. Columbus, OH, US:IEEE, 2014: 3003-3010.
[34]	WOO S, PARK J, LEE J Y, et al. Cbam: Convolutional block attention module[C]//Proceedings of the European conference on computer vision. Munich, Germany:Springer, 2018, 11211: 3-19.