MJ-GAN: Generative Adversarial Network with Multi-Grained Feature Extraction and Joint Attention Fusion for Infrared and Visible Image Fusion
The challenging issues in infrared and visible image fusion (IVIF) are extracting and fusing as much useful information as possible contained in the source images, namely, the rich textures in visible images and the significant contrast in infrared images. Existing fusion methods cannot address this...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-07-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/14/6322 |
| _version_ | 1797587535505915904 |
|---|---|
| author | Danqing Yang Xiaorui Wang Naibo Zhu Shuang Li Na Hou |
| author_facet | Danqing Yang Xiaorui Wang Naibo Zhu Shuang Li Na Hou |
| author_sort | Danqing Yang |
| collection | DOAJ |
| description | The challenging issues in infrared and visible image fusion (IVIF) are extracting and fusing as much useful information as possible contained in the source images, namely, the rich textures in visible images and the significant contrast in infrared images. Existing fusion methods cannot address this problem well due to the handcrafted fusion operations and the extraction of features only from a single scale. In this work, we solve the problems of insufficient information extraction and fusion from another perspective to overcome the difficulties in lacking textures and unhighlighted targets in fused images. We propose a multi-scale feature extraction (MFE) and joint attention fusion (JAF) based end-to-end method using a generative adversarial network (MJ-GAN) framework for the aim of IVIF. The MFE modules are embedded in the two-stream structure-based generator in a densely connected manner to comprehensively extract multi-grained deep features from the source image pairs and reuse them during reconstruction. Moreover, an improved self-attention structure is introduced into the MFEs to enhance the pertinence among multi-grained features. The merging procedure for salient and important features is conducted via the JAF network in a feature recalibration manner, which also produces the fused image in a reasonable manner. Eventually, we can reconstruct a primary fused image with the major infrared radiometric information and a small amount of visible texture information via a single decoder network. The dual discriminator with strong discriminative power can add more texture and contrast information to the final fused image. Extensive experiments on four publicly available datasets show that the proposed method ultimately achieves phenomenal performance in both visual quality and quantitative assessment compared with nine leading algorithms. |
| first_indexed | 2024-03-11T00:41:12Z |
| format | Article |
| id | doaj.art-96959f6dc032492da169dd8450a32190 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-11T00:41:12Z |
| publishDate | 2023-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-96959f6dc032492da169dd8450a321902023-11-18T21:16:07ZengMDPI AGSensors1424-82202023-07-012314632210.3390/s23146322MJ-GAN: Generative Adversarial Network with Multi-Grained Feature Extraction and Joint Attention Fusion for Infrared and Visible Image FusionDanqing Yang0Xiaorui Wang1Naibo Zhu2Shuang Li3Na Hou4School of Optoelectronic Engineering, Xidian University, Xi’an 710071, ChinaSchool of Optoelectronic Engineering, Xidian University, Xi’an 710071, ChinaResearch Institute of System Engineering, PLA Academy of Military Science, Beijing 100091, ChinaResearch Institute of System Engineering, PLA Academy of Military Science, Beijing 100091, ChinaResearch Institute of System Engineering, PLA Academy of Military Science, Beijing 100091, ChinaThe challenging issues in infrared and visible image fusion (IVIF) are extracting and fusing as much useful information as possible contained in the source images, namely, the rich textures in visible images and the significant contrast in infrared images. Existing fusion methods cannot address this problem well due to the handcrafted fusion operations and the extraction of features only from a single scale. In this work, we solve the problems of insufficient information extraction and fusion from another perspective to overcome the difficulties in lacking textures and unhighlighted targets in fused images. We propose a multi-scale feature extraction (MFE) and joint attention fusion (JAF) based end-to-end method using a generative adversarial network (MJ-GAN) framework for the aim of IVIF. The MFE modules are embedded in the two-stream structure-based generator in a densely connected manner to comprehensively extract multi-grained deep features from the source image pairs and reuse them during reconstruction. Moreover, an improved self-attention structure is introduced into the MFEs to enhance the pertinence among multi-grained features. The merging procedure for salient and important features is conducted via the JAF network in a feature recalibration manner, which also produces the fused image in a reasonable manner. Eventually, we can reconstruct a primary fused image with the major infrared radiometric information and a small amount of visible texture information via a single decoder network. The dual discriminator with strong discriminative power can add more texture and contrast information to the final fused image. Extensive experiments on four publicly available datasets show that the proposed method ultimately achieves phenomenal performance in both visual quality and quantitative assessment compared with nine leading algorithms.https://www.mdpi.com/1424-8220/23/14/6322infrared and visible image fusion (IVIF)multi-scale feature extraction (MFE)joint attention fusion (JAF)generative adversarial network (GAN)self-attention mechanism (SAM) |
| spellingShingle | Danqing Yang Xiaorui Wang Naibo Zhu Shuang Li Na Hou MJ-GAN: Generative Adversarial Network with Multi-Grained Feature Extraction and Joint Attention Fusion for Infrared and Visible Image Fusion Sensors infrared and visible image fusion (IVIF) multi-scale feature extraction (MFE) joint attention fusion (JAF) generative adversarial network (GAN) self-attention mechanism (SAM) |
| title | MJ-GAN: Generative Adversarial Network with Multi-Grained Feature Extraction and Joint Attention Fusion for Infrared and Visible Image Fusion |
| title_full | MJ-GAN: Generative Adversarial Network with Multi-Grained Feature Extraction and Joint Attention Fusion for Infrared and Visible Image Fusion |
| title_fullStr | MJ-GAN: Generative Adversarial Network with Multi-Grained Feature Extraction and Joint Attention Fusion for Infrared and Visible Image Fusion |
| title_full_unstemmed | MJ-GAN: Generative Adversarial Network with Multi-Grained Feature Extraction and Joint Attention Fusion for Infrared and Visible Image Fusion |
| title_short | MJ-GAN: Generative Adversarial Network with Multi-Grained Feature Extraction and Joint Attention Fusion for Infrared and Visible Image Fusion |
| title_sort | mj gan generative adversarial network with multi grained feature extraction and joint attention fusion for infrared and visible image fusion |
| topic | infrared and visible image fusion (IVIF) multi-scale feature extraction (MFE) joint attention fusion (JAF) generative adversarial network (GAN) self-attention mechanism (SAM) |
| url | https://www.mdpi.com/1424-8220/23/14/6322 |
| work_keys_str_mv | AT danqingyang mjgangenerativeadversarialnetworkwithmultigrainedfeatureextractionandjointattentionfusionforinfraredandvisibleimagefusion AT xiaoruiwang mjgangenerativeadversarialnetworkwithmultigrainedfeatureextractionandjointattentionfusionforinfraredandvisibleimagefusion AT naibozhu mjgangenerativeadversarialnetworkwithmultigrainedfeatureextractionandjointattentionfusionforinfraredandvisibleimagefusion AT shuangli mjgangenerativeadversarialnetworkwithmultigrainedfeatureextractionandjointattentionfusionforinfraredandvisibleimagefusion AT nahou mjgangenerativeadversarialnetworkwithmultigrainedfeatureextractionandjointattentionfusionforinfraredandvisibleimagefusion |