关键词: 图像融合, 红外图像, 生成对抗网络, Mamba

Abstract: Most of the existing methods for infrared and visible image fusion use Transformer to extract global features from images. However, the Transformer adopting self-attention mechanism needs to calculate the correlation between each pair of inputs, leading a sharp increase in computational complexity while processing long sequences. In contrast, Mamba is a lightweight model that can effectively model long-distance dependencies. To this end, this paper proposes a method for infrared and visible image fusion based on Mamba-empowered generative adversarial network with triple branches. The proposed method consists of a generator and two discriminators. Specifically, in the generator network, three independent and cooperative branches for feature extraction and fusion are designed, namely, the infrared branch, the visible branch, and the intermediate feature fusion branch are designed. More particularly, the infrared and visible branches extract global features respectively from the infrared and visible images through shallow feature extraction modules and Mamba blocks. Meanwhile, the global features extracted from the infrared branch and visible branch are hierarchically integrated into the convolutional modules of the intermediate fusion branch, thereby achieving fully interacting and merging between the local and global features. Furthermore, generating a fusion image with salient target and clear texture via competitively training the generator against two discriminators (infrared discriminator and visible discriminator). The extensively experimental results on public datasets show that the proposed method outperforms other state-of-the-art methods in both qualitative visual effects and quantitative objective metrics.

Key words: image fusion, infrared image, generative adversarial network, Mamba