基于多判别器双流生成对抗网络的红外与可见光图像融合

doi:10.12382/bgxb.2023.0130

摘要/Abstract

摘要：

针对现有的红外与可见光图像融合方法对源图像信息保留不充分的问题,改进了一种基于多判别器双流生成对抗网络(Generative Adversarial Network,GAN)的红外与可见光图像融合算法。改进的GAN融合框架包含一个生成器与四个判别器,将差分图像作为辅助信息,从而进一步提升网络的融合性能。算法中的差分图像不仅被作为源图像的附加信息,引导生成器关注不同模态图像的独特信息,还被用作真实数据分布,协助差分判别器与生成器进行对抗训练。所改进网络模型中,生成器采用双编码器-单解码器结构,其中编码器旨在提取不同模态特征,主要采用密集连接结构,并结合注意力模块;解码器旨在根据联结高维特征重构出融合图像。判别器对输入图像是来自真实图像还是融合图像进行判断,并根据判断的结果对生成器进行约束优化。实验结果表明,相较于对比算法,所提算法在主观判读和客观指标评价两方面均取得了更优异的融合效果。

关键词: 图像融合, 红外图像, 差分图像, 生成对抗网络, 注意力机制

Abstract:

To address the problem of insufficiently retaining the source information during the process of fusing the infrared and visible images based on the existing methods, an algorithm for fusing the infrared and visible images using dual-stream generative adversarial networks (GAN) with multiple discriminators is improved. The improved GAN-based fusion framework consists of one generator and four discriminators, and utilizes the differential image as the auxiliary information to further improve the performance of the fusion network. The differential imagein the algorithm is not only used as the auxiliary information of source image to guide the generator to focus on the unique information of different modal images, but also used as the real data distribution to assist the differential discriminator in competitively training with the generator. In the improved network model, the generator adopts a dual encoder-single decoder structure, where the encoder aims to extract the features from different modalities mainly via a densely connected structure combined with an attention module, and the decoder is used to reconstruct the fused image based on the concatenated high-dimensional features. The discriminator evaluates whether the input image is the real image or the fusion image, and constrainedly optimizes the generator based on the evaluated results. Experimental results show that, compared with the other algorithms, the improved algorithm achieves better fusion results both in the subjective assessments and in the objective effects evaluated by the quantitative metrics.

Key words: image fusion, infrared image, differential image, generative adversarial network, attention mechanism

中图分类号:

TP391.4

武凌霄, 康家银, 姬云翔, 马寒雁. 基于多判别器双流生成对抗网络的红外与可见光图像融合[J]. 兵工学报, 2024, 45(6): 1799-1812.

WU Lingxiao, KANG Jiayin, JI Yunxiang, MA Hanyan. Infrared and Visible Image Fusion Using Dual-stream Generative Adversarial Network with Multiple Discriminators[J]. Acta Armamentarii, 2024, 45(6): 1799-1812.

图/表 20

图1 所改进算法的整体融合框架

Fig.1 Overall fusion framework of improved algorithm

图2 差分图像

Fig.2 Differential image

图3 生成器网络结构

Fig.3 Network structure of generator

图4 判别器网络结构

Fig.4 Network structure of discriminator

表1 不同算法的融合结果

Table 1 Fusion results of different algorithms

表2 不同算法在融合15组图像时取得的评价指标的平均值

Table 2 Average values of evaluation indexes regarding 15 pairs of images fused by different fusion algorithms

融合算法	IE	SF	SD	AG	CC	SCD
FusionGAN算法	6.4572	0.0222	8.2101	2.1793	0.4616	1.3952
NestFuse算法	6.9955	0.0346	9.1960	$3.4594_$	0.4929	1.6955
PMGI算法	$7.0088_$	0.0313	9.6518	3.2915	$0.5473_$	$1.7774_$
GANMcC算法	6.6649	0.0221	9.0181	2.3158	0.5441	1.6904
YDTR算法	6.4901	0.0280	8.9153	2.6654	0.5241	1.5652
本文算法	7.1902	$0.0341_$	$9.5277_$	3.5405	0.5504	1.9146

表2 不同算法在融合15组图像时取得的评价指标的平均值

Table 2 Average values of evaluation indexes regarding 15 pairs of images fused by different fusion algorithms

融合算法	IE	SF	SD	AG	CC	SCD
FusionGAN算法	6.4572	0.0222	8.2101	2.1793	0.4616	1.3952
NestFuse算法	6.9955	0.0346	9.1960	$3.4594_$	0.4929	1.6955
PMGI算法	$7.0088_$	0.0313	9.6518	3.2915	$0.5473_$	$1.7774_$
GANMcC算法	6.6649	0.0221	9.0181	2.3158	0.5441	1.6904
YDTR算法	6.4901	0.0280	8.9153	2.6654	0.5241	1.5652
本文算法	7.1902	$0.0341_$	$9.5277_$	3.5405	0.5504	1.9146

图5 不同算法在融合15组图像时取得的评价指标

Fig.5 Evaluation indexes regarding 15 pairs of images fused by different fusion algorithms

表3 不同算法在融合15组图像时平均运行时间

Table 3 Average running time for 15 pairs of images fused by different fusion algorithms

融合算法	FusionGAN算法	NestFuse算法	PMGI算法	GANMcC算法	YDTR算法	本文算法
平均运行时间/s	1.43	0.77	1.11	1.92	1.16	1.60

图6 对比算法的网络结构整体框架

Fig.6 Overall framework of network structure of comparison algorithm

图7 对比算法的生成器网络结构

Fig.7 Network structure of generator of comparison algorithm

图8 不同算法的融合结果

Fig.8 Fusion results of different algorithms

表4 不同算法在融合15组图像时取得的评价指标的平均值

Table 4 Average values of evaluation indexes regarding 15 pairs of images fused by different fusion algorithms

融合算法	IE	SF	SD	AG	CC	SCD
Non-dif算法	6.8284	0.0342	8.0800	3.3031	0.4828	1.6727
本文算法	7.1901	0.0340	9.5277	3.5405	0.5503	1.9146

图9 对比算法的网络结构整体框架

Fig.9 Overall framework of network structure of comparison algorithm

图10 对比算法的生成器网络结构

Fig.10 Network structure of generator of comparison algorithm

图11 不同算法的融合结果

Fig.11 Fusion results of different algorithms

表5 不同算法在融合15组图像时取得的评价指标的平均值

Table 5 Average values of evaluation indexes regarding 15 pairs of images fused by different fusion algorithms

融合算法	IE	SF	SD	AG	CC	SCD
ABS-dif算法	7.0124	0.0343	9.3386	3.5319	0.5398	1.8120
本文算法	7.1901	0.0340	9.5277	3.5405	0.5503	1.9146

图12 对比算法的网络结构整体框架

Fig.12 Overall framework of network structure of comparison algorithm

图13 对比算法的生成器网络结构

Fig.13 Network structure of generator of comparison algorithm

图14 不同算法的融合结果

Fig.14 Fusion results of different algorithms

表6 不同算法在融合15组图像时取得的评价指标的平均值

Table 6 Average values of evaluation indexes regarding 15 pairs of images fused by different fusion algorithms

融合算法	IE	SF	SD	AG	CC	SCD
Single-IR算法	$7.1922_$	0.0303	$9.5213_$	3.3886	0.5618	$1.8971_$
Single-VIS算法	7.2560	0.0382	9.1212	3.8123	0.4324	1.6286
本文算法	7.1901	$0.0340_$	9.5277	$3.5405_$	$0.5503_$	1.9146

表6 不同算法在融合15组图像时取得的评价指标的平均值

Table 6 Average values of evaluation indexes regarding 15 pairs of images fused by different fusion algorithms

融合算法	IE	SF	SD	AG	CC	SCD
Single-IR算法	$7.1922_$	0.0303	$9.5213_$	3.3886	0.5618	$1.8971_$
Single-VIS算法	7.2560	0.0382	9.1212	3.8123	0.4324	1.6286
本文算法	7.1901	$0.0340_$	9.5277	$3.5405_$	$0.5503_$	1.9146

参考文献 23

[1]	MA J Y, MA Y, LI C. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019, 45: 153-178. doi: 10.1016/j.inffus.2018.02.004
[2]	丁贵鹏, 陶钢, 李英超, 等. 基于非下采样轮廓波变换与引导滤波器的红外及可见光图像融合[J]. 兵工学报, 2021, 42(9): 1911-1922.
	DING G P, TAO G, LI Y C, et al. Infrared and visible images fusion based on non-subsampled contourlet transform and guided filter[J]. Acta Armamentarii, 2021, 42(9): 1911-1922. (in Chinese) doi: 10.3969/j.issn.1000-1093.2021.09.012
[3]	ZHANG S, HUANG F Y, LIU B Q, et al. A multi-modal image fusion framework based on guided filter and sparse representation[J]. Optics and Lasers in Engineering, 2021, 137: 106354.
[4]	ZHANG H, XU H, TIAN X, et al. Image fusion meets deep learning: a survey and perspective[J]. Information Fusion, 2021, 76: 323-336.
[5]	LI H, WU X J. DenseFuse: a fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2019, 28(5): 2614-2623.
[6]	LI H, WU X J, DURRANI T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656.
[7]	ZHANG H, XU H, XIAO Y, et al. Rethinking the image fusion: a fast unified image fusion network based on proportional maintenance of gradient and intensity[C]∥ Proceedings of the AAAI Conference on Artificial Intelligence. New York, US: AAAI Press, 2020, 34: 12797-12804.
[8]	TANG W, HE F Z, LIU Y. YDTR: infrared and visible image fusion via Y-shape dynamic transformer[J]. IEEE Transactions on Multimedia (Early Access), 2022, 25:5413-5428..
[9]	MA J Y, YU W, LIANG P W, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26.
[10]	MA J Y, XU H, JIANG J J, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995.
[11]	MA J Y, ZHANG H, SHAO Z F, et al. GANMcC: a generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 5005014.
[12]	SONG A Y, DUAN H X, PEI H D, et al. Triple-discriminator generative adversarial network for infrared and visible image fusion[J]. Neurocomputing, 2022, 483: 183-194.
[13]	康家银, 武凌霄, 张文娟, 等. 基于多判别器生成对抗网络的红外与可见光图像融合方法: 202211405079.7[P]. 2023-01-13.
	KANGJY, WU L X, ZHANG W J, et al. Infrared and visible image fusion using generative adversarial network with multiple discriminators: 202211405079.7[P]. 2023-01-13. (in Chinese)
[14]	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[C]∥Proceedings of International Conference on Neural Information Processing Systems. Cambridge, US: MIT Press, 2014: 2672-2680.
[15]	HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, US: IEEE, 2017: 4700-4708.
[16]	LUO P, ZHANG R, REN J, et al. Switchable normalization for learning-to-normalize deep representation[J]. IEEE transactions on pattern analysis and machine intelligence, 2019, 43(2): 712-728.
[17]	WOO S, PARK J, LEE J Y, et al. CBAM: Convolutional block attention module[C]∥Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer, 2018,11211:3-19.
[18]	ROBERTS J W, VAN AARDT J A, AHMED F B. Assessment of image fusion procedures using entropy, image quality, and multispectral classification[J]. Journal of Applied Remote Sensing, 2008, 2(1): 023522.
[19]	SHAH P, MERCHANT S N, DESAI U B. Multifocus and multispectral image fusion based on pixel significance using multiresolution decomposition[J]. Signal, Image and Video Processing, 2013, 7: 95-109.
[20]	JIN X, JIANG Q, YAO S W, et al. Infrared and visual image fusion method based on discrete cosine transform and local spatial frequency in discrete stationary wavelet transform domain[J]. Infrared Physics & Technology, 2018, 88: 1-12.
[21]	CUI G M, FENG H J, XU Z H, et al. Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition[J]. Optics Communications, 2015, 341: 199-209.
[22]	DESHMUKH M, BHOSALE U. Image fusion and image quality assessment of fused images[J]. International Journal of Image Processing, 2010, 4(5): 484-508.
[23]	ASLANTAS V, BENDES E. A new image quality metric for image fusion: The sum of the correlations of differences[J]. AEU-International Journal of Electronics and Communications, 2015, 69(12): 1890-1896.