Transformer-CNN-based Multi-feature Aggregation Algorithm for Real Battlefield Image Dehazing

doi:10.12382/bgxb.2022.0957

Abstract

Abstract:

The development of military intelligence systems has a great influence on the fighting mode and winning mechanism of modern war. However, these systems are easily affected by haze and other bad weather conditions, resulting in blurred and degraded images, which brings challenges to the subsequent combat missions such as identification and tracking. Therefore, it is essential to restore the haze-free images on the battlefield. Since it is hard to capture the paired clean/hazy images, most existing methods adopt synthetic data for training. However, the gap between the real and synthetic hazy images will lead to the poor generalization of a model trained on synthetic data in the real world. To this end, a Transformer-CNN-based multi-feature aggregation algorithm is proposed for real battlefield image dehazing. This network adopts a semi-supervised framework to train the model with synthetic and real hazy images so that the model can better deal with the real hazy images. The algorithm applies a two-branch feature aggregation architecture to aggregate the local features extracted by CNN branch and the global features extracted by the Transformer branch to further improve the dehazing ability of the model. Moreover, a hazy battlefield image dataset is constructed to simulate the real battlefield hazy conditions. The experimental results show that, compared with 8 state-of-the-art image dehazing algorithms, the proposed algorithm performs well on both synthetic data and real images.

Key words: military intelligence, image dehazing, semi-supervised network, Transformer-CNN, feature aggregation

CLC Number:

TP393.413

WANG Yongzhen, TONG Ming, YAN Xuefeng, WEI Mingqiang. Transformer-CNN-based Multi-feature Aggregation Algorithm for Real Battlefield Image Dehazing[J]. Acta Armamentarii, 2024, 45(4): 1070-1081.

Figures/Tables 15

Fig.1 Overview of semi-supervised dehazing network

Fig.2 Transformer-CNN-based multi-feature aggregation network

Fig.3 Dehazeformernet module

Table 1 Detailed structural parameters of Dehazeformer net module

参数	数值
块的数量	[8,8,8,4,4]
字向量维度	[24,48,96,48,24]
MLP缩放比例	[2,4,4,2,2]
多头自注意力比重	[1/4,1/2,3/4,0,0]
注意力机制头个数	[2,4,6,1,1]
卷积类型	深度可分离卷积

Fig.4 The architecture of Discriminator

Table 2 Samples of synthetic images

Table 3 Quantitative evaluation and comparison of the quality of battlefield hazy datasets

数据集	AuthESI↓	NIQE↓	FADE↑
OTS	2.7348	3.4295	$1.9090_$
SOTS	$3.1546_$	3.8320	1.8489
HSTS	3.4900	$4.3590_$	1.6106
本文数据集	3.1736	3.2790	1.9490

Table 3 Quantitative evaluation and comparison of the quality of battlefield hazy datasets

数据集	AuthESI↓	NIQE↓	FADE↑
OTS	2.7348	3.4295	$1.9090_$
SOTS	$3.1546_$	3.8320	1.8489
HSTS	3.4900	$4.3590_$	1.6106
本文数据集	3.1736	3.2790	1.9490

Table 4 Quantitative comparison of different dehazing algorithms on synthetic battlefield hazy datasets

去雾算法	PSNR/dB↑	SSIM↑	CIEDE2000↓
DCP算法	17.0833	0.8667	11.3986
AOD-Net算法	23.1935	0.9094	6.0887
GCA-Net算法	21.2835	0.9162	7.6218
LD-Net算法	24.5415	0.9222	5.7080
SGID-PFF算法	26.1124	0.8946	4.5316
Semi_dehazing算法	24.4518	0.9307	5.0534
YOLY算法	19.6918	0.8136	8.7296
PSD算法	18.0533	0.7902	10.5273
本文算法	31.7481	0.9663	2.5933

Table 5 Dehazing results of different algorithms on the synthetic battlefield images dataset

Table 6 Dehazing effects of different algorithms on the real-world hazy images

Table 7 Quantitative comparison of different dehazing algorithms on real battlefield hazy datasets

去雾算法	BRISQUE↓	σ↓	HCC↑
DCP	29.3997	0.0068	0.0213
AOD-Net	33.2429	0.0719	0.1205
GCA-Net	28.8034	0.0228	0.2518
LD-Net	30.6247	0.0585	0.1830
SGID-PFF	44.6840	0.3524	0.0333
Semi_dehazing	$28.6624_$	0.0279	0.3514
YOLY	39.0062	0.0632	0.2159
PSD	31.8752	0.0310	$0.4302_$
本文算法	28.6583	$0.0084_$	0.5940

Table 7 Quantitative comparison of different dehazing algorithms on real battlefield hazy datasets

去雾算法	BRISQUE↓	σ↓	HCC↑
DCP	29.3997	0.0068	0.0213
AOD-Net	33.2429	0.0719	0.1205
GCA-Net	28.8034	0.0228	0.2518
LD-Net	30.6247	0.0585	0.1830
SGID-PFF	44.6840	0.3524	0.0333
Semi_dehazing	$28.6624_$	0.0279	0.3514
YOLY	39.0062	0.0632	0.2159
PSD	31.8752	0.0310	$0.4302_$
本文算法	28.6583	$0.0084_$	0.5940

Table 8 Ablation experiments of deep estimation network module and residual learning module

模块	Base模型	V₁模型	V₂模型	V₃模型
MixDehazeBlock	×	√	×	√
CNN分支网络	×	×	√	√
指标	Base模型	V₁模型	V₂模型	V₃模型
PSNR/dB↑	28.11	30.78	30.87	31.75
SSIM↑	0.945	0.962	0.960	0.966
CIEDE2000↓	3.332	2.846	2.900	2.593

Table 9 Ablation result of each loss

损失函数	C₁	C₂	C₃	C₄	C₅	C₆
L1 loss	√	√	√	√	√	√
GAN loss	×	√	√	√	√	√
color loss	×	×	√	√	√	√
Identity loss	×	×	×	√	√	√
DCLoss	×	×	×	×	√	√
TVLoss	×	×	×	×	×	√
指标	C₁	C₂	C₃	C₄	C₅	C₆
PSNR/dB↑	29.75	29.91	31.22	31.08	31.68	31.75
SSIM↑	0.961	0.963	0.962	0.963	0.966	0.966
CIEDE2000↓	3.231	3.144	2.822	2.838	2.649	2.593

Table 10 Ablation experiments of different numbers of real battlefield images

真实含雾图像数量	合成含雾数据集		真实战场含雾数据集
真实含雾图像数量	PSNR	SSIM	BRISQUE	σ
0	31.746	0.9643	29.9008	0.0128
300	31.902	0.9638	$28.6868_$	0.0107
450	30.535	$0.9647_$	28.7711	$0.0087_$
900	$31.748_$	0.9663	28.6583	0.0084

Table 10 Ablation experiments of different numbers of real battlefield images

真实含雾图像数量	合成含雾数据集		真实战场含雾数据集
真实含雾图像数量	PSNR	SSIM	BRISQUE	σ
0	31.746	0.9643	29.9008	0.0128
300	31.902	0.9638	$28.6868_$	0.0107
450	30.535	$0.9647_$	28.7711	$0.0087_$
900	$31.748_$	0.9663	28.6583	0.0084

Table 11 Comparison of running time of different dehazing algorithms

去雾算法	计算平台	平均运行时间/s
DCP算法	CPU	1.1720
AOD-Net算法	GPU	0.1225
GCA-Net算法	GPU	0.1607
Semi_dehazing算法	GPU	0.7018
YOLY算法	GPU	40.5547
PSD算法	GPU	0.4212
本文算法	GPU	0.3219

References 31

[1]	李力, 金伟其, 黄有为, 等. 一种低对比度可见光超视距成像的实时图像增强方法[J]. 兵工学报, 2010, 31(2): 242-247.
	LI L, JIN W Q, HUANG Y W, et al. Real-time image enhancement processing methods of low contrast visible light over-the-horizon imaging[J]. Acta Armamentarii, 2010, 31(2): 242-247. (in Chinese)
[2]	HE K M, SUN J, TANG X. Single image haze removal using dark channel prior[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 33(12): 2341-2353.
[3]	刘春辉, 齐越, 丁文锐. 基于大气光鲁棒估计的无人机图像去雾方法[J]. 北京航空航天大学学报, 2017, 43(6): 1105-1111.
	LIU C H, QI Y, DING W R. A haze removal method for unmanned aerial vehicle images based on robust estimation of atmospheric light[J]. Journal of Beijing University of Aeronautics and Astronautics, 2017, 43(6): 1105-1111. (in Chinese)
[4]	李春明, 姜雨彤, 宋海平, 等. 基于光学厚度代理模型的雾浓度估计及图像去雾[J]. 兵工学报, 2019, 40(7): 1425-1433. doi: 10.3969/j.issn.1000-1093.2019.07.012
	LI C M, JIANG Y T, SONG H P, et al. Research on optical depth surrogate model-based method for estimating fog density and removing fog effect from images[J]. Acta Armamentarii, 2019, 40(7): 1425-1433. (in Chinese) doi: 10.3969/j.issn.1000-1093.2019.07.012
[5]	杨勇, 邱根莹, 黄淑英, 等. 基于改进大气散射模型的单幅图像去雾方法[J]. 北京航空航天大学学报, 2022, 48(8): 1364-1375.
	YANG Y, QIU G Y, HUANG S Y, et al. Single image dehazing method based on improved atmospheric scattering model[J]. Journal of Beijing University of Aeronautics and Astronautics, 2022, 48(8): 1364-1375. (in Chinese)
[6]	姜雨彤, 宋海平, 王光辉. 基于质量评价最优的无人机航拍图像去雾方法[J]. 兵工学报, 2022, 43(1): 148-158. doi: 10.3969/j.issn.1000-1093.2022.01.016
	JIANG Y T, SONG H P, WANG G H. Image dehazing based on the optimum of UAV aerial image quality evaluation[J]. Acta Armamentarii, 2022, 43(1): 148-158. (in Chinese) doi: 10.3969/j.issn.1000-1093.2022.01.016
[7]	LI B Y, PENG X L, WANG Z Y, et al. Aod-net: All-in-one dehazing network[C]// Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 4770-4778.
[8]	CHEN D D, HE M M, FAN Q N, et al. Gated context aggregation network for image dehazing and deraining[C]// Proceedings of the IEEE Winter Conference on Applications of Computer Vision. Waikoloa Village, HI, US: IEEE, 2019: 1375-1383.
[9]	QIN X, WANG Z L, BAI Y C, et al. FFA-Net: feature fusion attention network for single image dehazing[C]// Proceedings of the AAAI Conference on Artificial Intelligence. New York, NY, US: AAAI, 2020, 34(7): 11908-11915.
[10]	BAI H R, PAN J S, XIANG X G, et al. Self-guided image dehazing using progressive feature fusion[J]. IEEE Transactions on Image Processing, 2022, 31: 1217-1229. doi: 10.1109/TIP.2022.3140609 pmid: 35015639
[11]	YOO J, KIM T, LEE S, et al. Rich CNN-transformer feature aggregation networks for super-resolution: arXiv:2203.07682[R/OL]. Ithaca, NY, US: Cornell University, 2022(2022-3-15). https://doi.org/10.48550/arXiv:2203.07682.
[12]	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision.Montreal, QC, Canada:IEEE, 2021: 10012-10022.
[13]	JIN C, WANG T, LI X B, et al. A transformer generative adversarial network for multi‐track music generation[J]. CAAI Transactions on Intelligence Technology, 2022, 7(3): 369-380.
[14]	SONG Y D, HE Z Q, QIAN H, et al. Vision transformers for single image dehazing:arXiv:2204.03883[R/OL]. Ithaca, NY, US: Cornell University, 2022(2017-4-8). https://doi.org/10.48550/arXiv:2204.03883.
[15]	LI L, DONG Y L, REN W Q, et al. Semi-supervised image dehazing[J]. IEEE Transactions on Image Processing, 2019, 29: 2766-2779.
[16]	WANG Y Z, YAN X F, GUAN D H, et al. Cycle-SNSPGAN: Towards real-world image dehazing via cycle spectral normalized soft likelihood estimation patch GAN[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(11):20368-20382.
[17]	IGNATOV A, KOBYSHEV N, TIMOFTE R, et al. Dslr-quality photos on mobile devices with deep convolutional networks[C]// Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 3277-3285.
[18]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 2223-2232.
[19]	NARASIMHAN S, NAYAR S. Vision and the atmosphere[J]. International Journal of Computer Vision, 2002, 48(3): 233-254.
[20]	ZHANG N, ZHANG L, CHENG Z X. Towards simulating foggy and hazy images and evaluating their authenticity[C]// Proceedings of the International Conference on Neural Information Processing Systems. Long Beach, CA, US: IEEE, 2017: 405-415.
[21]	MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a “completely blind” image quality analyzer[J]. IEEE Signal Processing Letters, 2012, 20(3): 209-212.
[22]	CHOI L K, YOU J, BOVIK A C. Referenceless prediction of perceptual fog density and perceptual image defogging[J]. IEEE Transactions on Image Processing, 2015, 24(11): 3888-3901. doi: 10.1109/TIP.2015.2456502 pmid: 26186784
[23]	LI B Y, REN W Q, FU D P, et al. Benchmarking single-image dehazing and beyond[J]. IEEE Transactions on Image Processing, 2018, 28(1): 492-505.
[24]	ULLAH H, MUHAMMAD K, IRFAN M, et al. Light-DehazeNet: a novel lightweight CNN architecture for single image dehazing[J]. IEEE Transactions on Image Processing, 2021, 30: 8968-8982.
[25]	LI B Y, GOU Y B, GU S H, et al. You only look yourself: unsupervised and untrained single image dehazing neural network[J]. International Journal of Computer Vision, 2021, 129(5): 1754-1767.
[26]	CHEN Z Y, WANG Y C, YANG Y, et al. PSD: principled synthetic-to-real dehazing guided by physical priors[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, QC, Canada: IEEE, 2021: 7180-7189.
[27]	WANY Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. doi: 10.1109/tip.2003.819861 pmid: 15376593
[28]	SHARMA G, WU W C, DALAL E N. The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations[J]. Color Research & Application, 2005, 30(1): 21-30.
[29]	MITTAL A, MOORTHY A K, BOVIK A C. No-reference image quality assessment in the spatial domain[J]. IEEE Transactions on image processing, 2012, 21(12): 4695-4708. doi: 10.1109/TIP.2012.2214050 pmid: 22910118
[30]	CHEN Z Y, JIANG T T, TIAN Y H. Quality assessment for comparing image enhancement algorithms[C]// Proceedings of the IEEE International Conference on Computer Vision.Columbus, OH, US:IEEE, 2014: 3003-3010.
[31]	XU Y, WEN J, FEI L K, et al. Review of video and image defogging algorithms and related studies on image restoration and enhancement[J]. IEEE Access, 2015, 4: 165-188.