面向真实战场环境的Transformer-CNN多特征聚合图像去雾算法

doi:10.12382/bgxb.2022.0957

摘要/Abstract

摘要：

军事智能系统的发展对现代战争的作战方式和制胜机理产生重大影响,然而这些系统容易受到雾霾等天气的影响导致获取的图像出现模糊、退化等问题,给后续识别、追踪等作战任务带来挑战,因此对战场含雾图像进行复原非常重要。鉴于获取同一场景下的含雾、清晰图像对难度极大,现有网络大都采用合成数据进行训练;但真实雾图和合成雾图之间的间隙,会导致在合成数据下训练的模型在真实场景中泛化性差。为此,提出一种面向真实战场环境的自注意力模型-卷积神经网络(Transformer-Convolutional Neural Network,Transformer-CNN)多特征聚合图像去雾算法。采用半监督框架,利用合成和真实战场含雾图像训练网络,使模型能够更好地应对真实含雾场景。采用双分支特征聚合架构,将CNN分支提取的局部特征和Transformer分支学习的全局特征进行聚合,以进一步提高模型去雾能力。为模拟真实战场含雾场景,构建了一套含雾战场图像数据集。实验结果表明,与8种最先进的图像去雾算法相比,所提算法在合成数据和真实图像上均表现良好。

关键词: 军事智能, 图像去雾, 半监督网络, Transformer-CNN, 特征聚合

Abstract:

The development of military intelligence systems has a great influence on the fighting mode and winning mechanism of modern war. However, these systems are easily affected by haze and other bad weather conditions, resulting in blurred and degraded images, which brings challenges to the subsequent combat missions such as identification and tracking. Therefore, it is essential to restore the haze-free images on the battlefield. Since it is hard to capture the paired clean/hazy images, most existing methods adopt synthetic data for training. However, the gap between the real and synthetic hazy images will lead to the poor generalization of a model trained on synthetic data in the real world. To this end, a Transformer-CNN-based multi-feature aggregation algorithm is proposed for real battlefield image dehazing. This network adopts a semi-supervised framework to train the model with synthetic and real hazy images so that the model can better deal with the real hazy images. The algorithm applies a two-branch feature aggregation architecture to aggregate the local features extracted by CNN branch and the global features extracted by the Transformer branch to further improve the dehazing ability of the model. Moreover, a hazy battlefield image dataset is constructed to simulate the real battlefield hazy conditions. The experimental results show that, compared with 8 state-of-the-art image dehazing algorithms, the proposed algorithm performs well on both synthetic data and real images.

Key words: military intelligence, image dehazing, semi-supervised network, Transformer-CNN, feature aggregation

中图分类号:

TP393.413

王永振, 童鸣, 燕雪峰, 魏明强. 面向真实战场环境的Transformer-CNN多特征聚合图像去雾算法[J]. 兵工学报, 2024, 45(4): 1070-1081.

WANG Yongzhen, TONG Ming, YAN Xuefeng, WEI Mingqiang. Transformer-CNN-based Multi-feature Aggregation Algorithm for Real Battlefield Image Dehazing[J]. Acta Armamentarii, 2024, 45(4): 1070-1081.

图/表 15

图1 半监督去雾网络总体框架

Fig.1 Overview of semi-supervised dehazing network

图2 Transformer-CNN特征聚合网络

Fig.2 Transformer-CNN-based multi-feature aggregation network

图3 Dehazeformer网络模块

Fig.3 Dehazeformernet module

表1 Dehazeformer网络模块的详细结构参数

Table 1 Detailed structural parameters of Dehazeformer net module

参数	数值
块的数量	[8,8,8,4,4]
字向量维度	[24,48,96,48,24]
MLP缩放比例	[2,4,4,2,2]
多头自注意力比重	[1/4,1/2,3/4,0,0]
注意力机制头个数	[2,4,6,1,1]
卷积类型	深度可分离卷积

图4 判别器结构

Fig.4 The architecture of Discriminator

表2 合成数据集示例

Table 2 Samples of synthetic images

表3 战场含雾数据集质量定量评估比较

Table 3 Quantitative evaluation and comparison of the quality of battlefield hazy datasets

数据集	AuthESI↓	NIQE↓	FADE↑
OTS	2.7348	3.4295	$1.9090_$
SOTS	$3.1546_$	3.8320	1.8489
HSTS	3.4900	$4.3590_$	1.6106
本文数据集	3.1736	3.2790	1.9490

表3 战场含雾数据集质量定量评估比较

Table 3 Quantitative evaluation and comparison of the quality of battlefield hazy datasets

数据集	AuthESI↓	NIQE↓	FADE↑
OTS	2.7348	3.4295	$1.9090_$
SOTS	$3.1546_$	3.8320	1.8489
HSTS	3.4900	$4.3590_$	1.6106
本文数据集	3.1736	3.2790	1.9490

表4 不同去雾算法在合成战场含雾数据集上的定量比较

Table 4 Quantitative comparison of different dehazing algorithms on synthetic battlefield hazy datasets

去雾算法	PSNR/dB↑	SSIM↑	CIEDE2000↓
DCP算法	17.0833	0.8667	11.3986
AOD-Net算法	23.1935	0.9094	6.0887
GCA-Net算法	21.2835	0.9162	7.6218
LD-Net算法	24.5415	0.9222	5.7080
SGID-PFF算法	26.1124	0.8946	4.5316
Semi_dehazing算法	24.4518	0.9307	5.0534
YOLY算法	19.6918	0.8136	8.7296
PSD算法	18.0533	0.7902	10.5273
本文算法	31.7481	0.9663	2.5933

表5 不同算法在合成战场含雾数据集上的去雾结果比较

Table 5 Dehazing results of different algorithms on the synthetic battlefield images dataset

表6 不同算法在真实战场含雾图片上的去雾效果比较

Table 6 Dehazing effects of different algorithms on the real-world hazy images

表7 不同去雾算法在真实战场含雾数据集上的定量比较

Table 7 Quantitative comparison of different dehazing algorithms on real battlefield hazy datasets

去雾算法	BRISQUE↓	σ↓	HCC↑
DCP	29.3997	0.0068	0.0213
AOD-Net	33.2429	0.0719	0.1205
GCA-Net	28.8034	0.0228	0.2518
LD-Net	30.6247	0.0585	0.1830
SGID-PFF	44.6840	0.3524	0.0333
Semi_dehazing	$28.6624_$	0.0279	0.3514
YOLY	39.0062	0.0632	0.2159
PSD	31.8752	0.0310	$0.4302_$
本文算法	28.6583	$0.0084_$	0.5940

表7 不同去雾算法在真实战场含雾数据集上的定量比较

Table 7 Quantitative comparison of different dehazing algorithms on real battlefield hazy datasets

去雾算法	BRISQUE↓	σ↓	HCC↑
DCP	29.3997	0.0068	0.0213
AOD-Net	33.2429	0.0719	0.1205
GCA-Net	28.8034	0.0228	0.2518
LD-Net	30.6247	0.0585	0.1830
SGID-PFF	44.6840	0.3524	0.0333
Semi_dehazing	$28.6624_$	0.0279	0.3514
YOLY	39.0062	0.0632	0.2159
PSD	31.8752	0.0310	$0.4302_$
本文算法	28.6583	$0.0084_$	0.5940

表8 深度分支网络模块和残差学习模块的消融实验

Table 8 Ablation experiments of deep estimation network module and residual learning module

模块	Base模型	V₁模型	V₂模型	V₃模型
MixDehazeBlock	×	√	×	√
CNN分支网络	×	×	√	√
指标	Base模型	V₁模型	V₂模型	V₃模型
PSNR/dB↑	28.11	30.78	30.87	31.75
SSIM↑	0.945	0.962	0.960	0.966
CIEDE2000↓	3.332	2.846	2.900	2.593

表9 各损失函数分量的消融实验

Table 9 Ablation result of each loss

损失函数	C₁	C₂	C₃	C₄	C₅	C₆
L1 loss	√	√	√	√	√	√
GAN loss	×	√	√	√	√	√
color loss	×	×	√	√	√	√
Identity loss	×	×	×	√	√	√
DCLoss	×	×	×	×	√	√
TVLoss	×	×	×	×	×	√
指标	C₁	C₂	C₃	C₄	C₅	C₆
PSNR/dB↑	29.75	29.91	31.22	31.08	31.68	31.75
SSIM↑	0.961	0.963	0.962	0.963	0.966	0.966
CIEDE2000↓	3.231	3.144	2.822	2.838	2.649	2.593

表10 不同数量真实战场图像的消融实验

Table 10 Ablation experiments of different numbers of real battlefield images

真实含雾图像数量	合成含雾数据集		真实战场含雾数据集
真实含雾图像数量	PSNR	SSIM	BRISQUE	σ
0	31.746	0.9643	29.9008	0.0128
300	31.902	0.9638	$28.6868_$	0.0107
450	30.535	$0.9647_$	28.7711	$0.0087_$
900	$31.748_$	0.9663	28.6583	0.0084

表10 不同数量真实战场图像的消融实验

Table 10 Ablation experiments of different numbers of real battlefield images

真实含雾图像数量	合成含雾数据集		真实战场含雾数据集
真实含雾图像数量	PSNR	SSIM	BRISQUE	σ
0	31.746	0.9643	29.9008	0.0128
300	31.902	0.9638	$28.6868_$	0.0107
450	30.535	$0.9647_$	28.7711	$0.0087_$
900	$31.748_$	0.9663	28.6583	0.0084

表11 不同去雾算法运行时间比较

Table 11 Comparison of running time of different dehazing algorithms

去雾算法	计算平台	平均运行时间/s
DCP算法	CPU	1.1720
AOD-Net算法	GPU	0.1225
GCA-Net算法	GPU	0.1607
Semi_dehazing算法	GPU	0.7018
YOLY算法	GPU	40.5547
PSD算法	GPU	0.4212
本文算法	GPU	0.3219

参考文献 31

[1]	李力, 金伟其, 黄有为, 等. 一种低对比度可见光超视距成像的实时图像增强方法[J]. 兵工学报, 2010, 31(2): 242-247.
	LI L, JIN W Q, HUANG Y W, et al. Real-time image enhancement processing methods of low contrast visible light over-the-horizon imaging[J]. Acta Armamentarii, 2010, 31(2): 242-247. (in Chinese)
[2]	HE K M, SUN J, TANG X. Single image haze removal using dark channel prior[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 33(12): 2341-2353.
[3]	刘春辉, 齐越, 丁文锐. 基于大气光鲁棒估计的无人机图像去雾方法[J]. 北京航空航天大学学报, 2017, 43(6): 1105-1111.
	LIU C H, QI Y, DING W R. A haze removal method for unmanned aerial vehicle images based on robust estimation of atmospheric light[J]. Journal of Beijing University of Aeronautics and Astronautics, 2017, 43(6): 1105-1111. (in Chinese)
[4]	李春明, 姜雨彤, 宋海平, 等. 基于光学厚度代理模型的雾浓度估计及图像去雾[J]. 兵工学报, 2019, 40(7): 1425-1433. doi: 10.3969/j.issn.1000-1093.2019.07.012
	LI C M, JIANG Y T, SONG H P, et al. Research on optical depth surrogate model-based method for estimating fog density and removing fog effect from images[J]. Acta Armamentarii, 2019, 40(7): 1425-1433. (in Chinese) doi: 10.3969/j.issn.1000-1093.2019.07.012
[5]	杨勇, 邱根莹, 黄淑英, 等. 基于改进大气散射模型的单幅图像去雾方法[J]. 北京航空航天大学学报, 2022, 48(8): 1364-1375.
	YANG Y, QIU G Y, HUANG S Y, et al. Single image dehazing method based on improved atmospheric scattering model[J]. Journal of Beijing University of Aeronautics and Astronautics, 2022, 48(8): 1364-1375. (in Chinese)
[6]	姜雨彤, 宋海平, 王光辉. 基于质量评价最优的无人机航拍图像去雾方法[J]. 兵工学报, 2022, 43(1): 148-158. doi: 10.3969/j.issn.1000-1093.2022.01.016
	JIANG Y T, SONG H P, WANG G H. Image dehazing based on the optimum of UAV aerial image quality evaluation[J]. Acta Armamentarii, 2022, 43(1): 148-158. (in Chinese) doi: 10.3969/j.issn.1000-1093.2022.01.016
[7]	LI B Y, PENG X L, WANG Z Y, et al. Aod-net: All-in-one dehazing network[C]// Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 4770-4778.
[8]	CHEN D D, HE M M, FAN Q N, et al. Gated context aggregation network for image dehazing and deraining[C]// Proceedings of the IEEE Winter Conference on Applications of Computer Vision. Waikoloa Village, HI, US: IEEE, 2019: 1375-1383.
[9]	QIN X, WANG Z L, BAI Y C, et al. FFA-Net: feature fusion attention network for single image dehazing[C]// Proceedings of the AAAI Conference on Artificial Intelligence. New York, NY, US: AAAI, 2020, 34(7): 11908-11915.
[10]	BAI H R, PAN J S, XIANG X G, et al. Self-guided image dehazing using progressive feature fusion[J]. IEEE Transactions on Image Processing, 2022, 31: 1217-1229. doi: 10.1109/TIP.2022.3140609 pmid: 35015639
[11]	YOO J, KIM T, LEE S, et al. Rich CNN-transformer feature aggregation networks for super-resolution: arXiv:2203.07682[R/OL]. Ithaca, NY, US: Cornell University, 2022(2022-3-15). https://doi.org/10.48550/arXiv:2203.07682.
[12]	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision.Montreal, QC, Canada:IEEE, 2021: 10012-10022.
[13]	JIN C, WANG T, LI X B, et al. A transformer generative adversarial network for multi‐track music generation[J]. CAAI Transactions on Intelligence Technology, 2022, 7(3): 369-380.
[14]	SONG Y D, HE Z Q, QIAN H, et al. Vision transformers for single image dehazing:arXiv:2204.03883[R/OL]. Ithaca, NY, US: Cornell University, 2022(2017-4-8). https://doi.org/10.48550/arXiv:2204.03883.
[15]	LI L, DONG Y L, REN W Q, et al. Semi-supervised image dehazing[J]. IEEE Transactions on Image Processing, 2019, 29: 2766-2779.
[16]	WANG Y Z, YAN X F, GUAN D H, et al. Cycle-SNSPGAN: Towards real-world image dehazing via cycle spectral normalized soft likelihood estimation patch GAN[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(11):20368-20382.
[17]	IGNATOV A, KOBYSHEV N, TIMOFTE R, et al. Dslr-quality photos on mobile devices with deep convolutional networks[C]// Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 3277-3285.
[18]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 2223-2232.
[19]	NARASIMHAN S, NAYAR S. Vision and the atmosphere[J]. International Journal of Computer Vision, 2002, 48(3): 233-254.
[20]	ZHANG N, ZHANG L, CHENG Z X. Towards simulating foggy and hazy images and evaluating their authenticity[C]// Proceedings of the International Conference on Neural Information Processing Systems. Long Beach, CA, US: IEEE, 2017: 405-415.
[21]	MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a “completely blind” image quality analyzer[J]. IEEE Signal Processing Letters, 2012, 20(3): 209-212.
[22]	CHOI L K, YOU J, BOVIK A C. Referenceless prediction of perceptual fog density and perceptual image defogging[J]. IEEE Transactions on Image Processing, 2015, 24(11): 3888-3901. doi: 10.1109/TIP.2015.2456502 pmid: 26186784
[23]	LI B Y, REN W Q, FU D P, et al. Benchmarking single-image dehazing and beyond[J]. IEEE Transactions on Image Processing, 2018, 28(1): 492-505.
[24]	ULLAH H, MUHAMMAD K, IRFAN M, et al. Light-DehazeNet: a novel lightweight CNN architecture for single image dehazing[J]. IEEE Transactions on Image Processing, 2021, 30: 8968-8982.
[25]	LI B Y, GOU Y B, GU S H, et al. You only look yourself: unsupervised and untrained single image dehazing neural network[J]. International Journal of Computer Vision, 2021, 129(5): 1754-1767.
[26]	CHEN Z Y, WANG Y C, YANG Y, et al. PSD: principled synthetic-to-real dehazing guided by physical priors[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, QC, Canada: IEEE, 2021: 7180-7189.
[27]	WANY Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. doi: 10.1109/tip.2003.819861 pmid: 15376593
[28]	SHARMA G, WU W C, DALAL E N. The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations[J]. Color Research & Application, 2005, 30(1): 21-30.
[29]	MITTAL A, MOORTHY A K, BOVIK A C. No-reference image quality assessment in the spatial domain[J]. IEEE Transactions on image processing, 2012, 21(12): 4695-4708. doi: 10.1109/TIP.2012.2214050 pmid: 22910118
[30]	CHEN Z Y, JIANG T T, TIAN Y H. Quality assessment for comparing image enhancement algorithms[C]// Proceedings of the IEEE International Conference on Computer Vision.Columbus, OH, US:IEEE, 2014: 3003-3010.
[31]	XU Y, WEN J, FEI L K, et al. Review of video and image defogging algorithms and related studies on image restoration and enhancement[J]. IEEE Access, 2015, 4: 165-188.