Infrared and Visible Image Fusion Using Dual-stream Generative Adversarial Network with Multiple Discriminators

doi:10.12382/bgxb.2023.0130

Abstract

Abstract:

To address the problem of insufficiently retaining the source information during the process of fusing the infrared and visible images based on the existing methods, an algorithm for fusing the infrared and visible images using dual-stream generative adversarial networks (GAN) with multiple discriminators is improved. The improved GAN-based fusion framework consists of one generator and four discriminators, and utilizes the differential image as the auxiliary information to further improve the performance of the fusion network. The differential imagein the algorithm is not only used as the auxiliary information of source image to guide the generator to focus on the unique information of different modal images, but also used as the real data distribution to assist the differential discriminator in competitively training with the generator. In the improved network model, the generator adopts a dual encoder-single decoder structure, where the encoder aims to extract the features from different modalities mainly via a densely connected structure combined with an attention module, and the decoder is used to reconstruct the fused image based on the concatenated high-dimensional features. The discriminator evaluates whether the input image is the real image or the fusion image, and constrainedly optimizes the generator based on the evaluated results. Experimental results show that, compared with the other algorithms, the improved algorithm achieves better fusion results both in the subjective assessments and in the objective effects evaluated by the quantitative metrics.

Key words: image fusion, infrared image, differential image, generative adversarial network, attention mechanism

CLC Number:

TP391.4

WU Lingxiao, KANG Jiayin, JI Yunxiang, MA Hanyan. Infrared and Visible Image Fusion Using Dual-stream Generative Adversarial Network with Multiple Discriminators[J]. Acta Armamentarii, 2024, 45(6): 1799-1812.

Figures/Tables 20

Fig.1 Overall fusion framework of improved algorithm

Fig.2 Differential image

Fig.3 Network structure of generator

Fig.4 Network structure of discriminator

Table 1 Fusion results of different algorithms

Table 2 Average values of evaluation indexes regarding 15 pairs of images fused by different fusion algorithms

融合算法	IE	SF	SD	AG	CC	SCD
FusionGAN算法	6.4572	0.0222	8.2101	2.1793	0.4616	1.3952
NestFuse算法	6.9955	0.0346	9.1960	$3.4594_$	0.4929	1.6955
PMGI算法	$7.0088_$	0.0313	9.6518	3.2915	$0.5473_$	$1.7774_$
GANMcC算法	6.6649	0.0221	9.0181	2.3158	0.5441	1.6904
YDTR算法	6.4901	0.0280	8.9153	2.6654	0.5241	1.5652
本文算法	7.1902	$0.0341_$	$9.5277_$	3.5405	0.5504	1.9146

Table 2 Average values of evaluation indexes regarding 15 pairs of images fused by different fusion algorithms

融合算法	IE	SF	SD	AG	CC	SCD
FusionGAN算法	6.4572	0.0222	8.2101	2.1793	0.4616	1.3952
NestFuse算法	6.9955	0.0346	9.1960	$3.4594_$	0.4929	1.6955
PMGI算法	$7.0088_$	0.0313	9.6518	3.2915	$0.5473_$	$1.7774_$
GANMcC算法	6.6649	0.0221	9.0181	2.3158	0.5441	1.6904
YDTR算法	6.4901	0.0280	8.9153	2.6654	0.5241	1.5652
本文算法	7.1902	$0.0341_$	$9.5277_$	3.5405	0.5504	1.9146

Fig.5 Evaluation indexes regarding 15 pairs of images fused by different fusion algorithms

Table 3 Average running time for 15 pairs of images fused by different fusion algorithms

融合算法	FusionGAN算法	NestFuse算法	PMGI算法	GANMcC算法	YDTR算法	本文算法
平均运行时间/s	1.43	0.77	1.11	1.92	1.16	1.60

Fig.6 Overall framework of network structure of comparison algorithm

Fig.7 Network structure of generator of comparison algorithm

Fig.8 Fusion results of different algorithms

Table 4 Average values of evaluation indexes regarding 15 pairs of images fused by different fusion algorithms

融合算法	IE	SF	SD	AG	CC	SCD
Non-dif算法	6.8284	0.0342	8.0800	3.3031	0.4828	1.6727
本文算法	7.1901	0.0340	9.5277	3.5405	0.5503	1.9146

Fig.9 Overall framework of network structure of comparison algorithm

Fig.10 Network structure of generator of comparison algorithm

Fig.11 Fusion results of different algorithms

Table 5 Average values of evaluation indexes regarding 15 pairs of images fused by different fusion algorithms

融合算法	IE	SF	SD	AG	CC	SCD
ABS-dif算法	7.0124	0.0343	9.3386	3.5319	0.5398	1.8120
本文算法	7.1901	0.0340	9.5277	3.5405	0.5503	1.9146

Fig.12 Overall framework of network structure of comparison algorithm

Fig.13 Network structure of generator of comparison algorithm

Fig.14 Fusion results of different algorithms

Table 6 Average values of evaluation indexes regarding 15 pairs of images fused by different fusion algorithms

融合算法	IE	SF	SD	AG	CC	SCD
Single-IR算法	$7.1922_$	0.0303	$9.5213_$	3.3886	0.5618	$1.8971_$
Single-VIS算法	7.2560	0.0382	9.1212	3.8123	0.4324	1.6286
本文算法	7.1901	$0.0340_$	9.5277	$3.5405_$	$0.5503_$	1.9146

Table 6 Average values of evaluation indexes regarding 15 pairs of images fused by different fusion algorithms

融合算法	IE	SF	SD	AG	CC	SCD
Single-IR算法	$7.1922_$	0.0303	$9.5213_$	3.3886	0.5618	$1.8971_$
Single-VIS算法	7.2560	0.0382	9.1212	3.8123	0.4324	1.6286
本文算法	7.1901	$0.0340_$	9.5277	$3.5405_$	$0.5503_$	1.9146

References 23

[1]	MA J Y, MA Y, LI C. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019, 45: 153-178. doi: 10.1016/j.inffus.2018.02.004
[2]	丁贵鹏, 陶钢, 李英超, 等. 基于非下采样轮廓波变换与引导滤波器的红外及可见光图像融合[J]. 兵工学报, 2021, 42(9): 1911-1922.
	DING G P, TAO G, LI Y C, et al. Infrared and visible images fusion based on non-subsampled contourlet transform and guided filter[J]. Acta Armamentarii, 2021, 42(9): 1911-1922. (in Chinese) doi: 10.3969/j.issn.1000-1093.2021.09.012
[3]	ZHANG S, HUANG F Y, LIU B Q, et al. A multi-modal image fusion framework based on guided filter and sparse representation[J]. Optics and Lasers in Engineering, 2021, 137: 106354.
[4]	ZHANG H, XU H, TIAN X, et al. Image fusion meets deep learning: a survey and perspective[J]. Information Fusion, 2021, 76: 323-336.
[5]	LI H, WU X J. DenseFuse: a fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2019, 28(5): 2614-2623.
[6]	LI H, WU X J, DURRANI T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656.
[7]	ZHANG H, XU H, XIAO Y, et al. Rethinking the image fusion: a fast unified image fusion network based on proportional maintenance of gradient and intensity[C]∥ Proceedings of the AAAI Conference on Artificial Intelligence. New York, US: AAAI Press, 2020, 34: 12797-12804.
[8]	TANG W, HE F Z, LIU Y. YDTR: infrared and visible image fusion via Y-shape dynamic transformer[J]. IEEE Transactions on Multimedia (Early Access), 2022, 25:5413-5428..
[9]	MA J Y, YU W, LIANG P W, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26.
[10]	MA J Y, XU H, JIANG J J, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995.
[11]	MA J Y, ZHANG H, SHAO Z F, et al. GANMcC: a generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 5005014.
[12]	SONG A Y, DUAN H X, PEI H D, et al. Triple-discriminator generative adversarial network for infrared and visible image fusion[J]. Neurocomputing, 2022, 483: 183-194.
[13]	康家银, 武凌霄, 张文娟, 等. 基于多判别器生成对抗网络的红外与可见光图像融合方法: 202211405079.7[P]. 2023-01-13.
	KANGJY, WU L X, ZHANG W J, et al. Infrared and visible image fusion using generative adversarial network with multiple discriminators: 202211405079.7[P]. 2023-01-13. (in Chinese)
[14]	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[C]∥Proceedings of International Conference on Neural Information Processing Systems. Cambridge, US: MIT Press, 2014: 2672-2680.
[15]	HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, US: IEEE, 2017: 4700-4708.
[16]	LUO P, ZHANG R, REN J, et al. Switchable normalization for learning-to-normalize deep representation[J]. IEEE transactions on pattern analysis and machine intelligence, 2019, 43(2): 712-728.
[17]	WOO S, PARK J, LEE J Y, et al. CBAM: Convolutional block attention module[C]∥Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer, 2018,11211:3-19.
[18]	ROBERTS J W, VAN AARDT J A, AHMED F B. Assessment of image fusion procedures using entropy, image quality, and multispectral classification[J]. Journal of Applied Remote Sensing, 2008, 2(1): 023522.
[19]	SHAH P, MERCHANT S N, DESAI U B. Multifocus and multispectral image fusion based on pixel significance using multiresolution decomposition[J]. Signal, Image and Video Processing, 2013, 7: 95-109.
[20]	JIN X, JIANG Q, YAO S W, et al. Infrared and visual image fusion method based on discrete cosine transform and local spatial frequency in discrete stationary wavelet transform domain[J]. Infrared Physics & Technology, 2018, 88: 1-12.
[21]	CUI G M, FENG H J, XU Z H, et al. Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition[J]. Optics Communications, 2015, 341: 199-209.
[22]	DESHMUKH M, BHOSALE U. Image fusion and image quality assessment of fused images[J]. International Journal of Image Processing, 2010, 4(5): 484-508.
[23]	ASLANTAS V, BENDES E. A new image quality metric for image fusion: The sum of the correlations of differences[J]. AEU-International Journal of Electronics and Communications, 2015, 69(12): 1890-1896.