Hyperspectral and multispectral remote sensing image fusion using SwinGAN with joint adaptive spatial-spectral gradient loss function
Hyperspectral remote sensing image (HSI) fusion with multispectral remote sensing images (MSI) improves data resolution. However, current fusion algorithms focus on local information and overlook long-range dependencies. The parameter of network tuning prioritizes global optimization, neglecting spa...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Taylor & Francis Group
2023-12-01
|
| Series: | International Journal of Digital Earth |
| Subjects: | |
| Online Access: | http://dx.doi.org/10.1080/17538947.2023.2253206 |
| _version_ | 1827811032615092224 |
|---|---|
| author | Chunyu Zhu Shangqi Deng Jiaxin Li Ying Zhang Liwei Gong Liangbo Gao Na Ta Shengbo Chen Qiong Wu |
| author_facet | Chunyu Zhu Shangqi Deng Jiaxin Li Ying Zhang Liwei Gong Liangbo Gao Na Ta Shengbo Chen Qiong Wu |
| author_sort | Chunyu Zhu |
| collection | DOAJ |
| description | Hyperspectral remote sensing image (HSI) fusion with multispectral remote sensing images (MSI) improves data resolution. However, current fusion algorithms focus on local information and overlook long-range dependencies. The parameter of network tuning prioritizes global optimization, neglecting spatial and spectral constraints, and limiting spatial and spectral reconstruction capabilities. This study introduces SwinGAN, a fusion network combining Swin Transformer, CNN, and GAN architectures. SwinGAN's generator employs a detail injection framework to separately extract HSI and MSI features, fusing them to generate spatial residuals. These residuals are injected into the supersampled HSI to produce the final image, while a pure CNN architecture acts as the discriminator, enhancing the fusion quality. Additionally, we introduce a new adaptive loss function that improves image fusion accuracy. The loss function uses L1 loss as the content loss, and spatial and spectral gradient loss functions are introduced to improve the spatial representation and spectral fidelity of the fused images. Our experimental results on several datasets demonstrate that SwinGAN outperforms current popular algorithms in both spatial and spectral reconstruction capabilities. The ablation experiments also demonstrate the rationality of the various components of the proposed loss function. |
| first_indexed | 2024-03-11T22:59:45Z |
| format | Article |
| id | doaj.art-909e443645044d7faeff29a85244f675 |
| institution | Directory Open Access Journal |
| issn | 1753-8947 1753-8955 |
| language | English |
| last_indexed | 2024-03-11T22:59:45Z |
| publishDate | 2023-12-01 |
| publisher | Taylor & Francis Group |
| record_format | Article |
| series | International Journal of Digital Earth |
| spelling | doaj.art-909e443645044d7faeff29a85244f6752023-09-21T15:09:04ZengTaylor & Francis GroupInternational Journal of Digital Earth1753-89471753-89552023-12-011613580360010.1080/17538947.2023.22532062253206Hyperspectral and multispectral remote sensing image fusion using SwinGAN with joint adaptive spatial-spectral gradient loss functionChunyu Zhu0Shangqi Deng1Jiaxin Li2Ying Zhang3Liwei Gong4Liangbo Gao5Na Ta6Shengbo Chen7Qiong Wu8Jilin UniversityUniversity of Electronic Science and Technology of ChinaChinese Academy of SciencesJilin UniversityJilin UniversityJilin UniversityJilin UniversityJilin UniversityJilin UniversityHyperspectral remote sensing image (HSI) fusion with multispectral remote sensing images (MSI) improves data resolution. However, current fusion algorithms focus on local information and overlook long-range dependencies. The parameter of network tuning prioritizes global optimization, neglecting spatial and spectral constraints, and limiting spatial and spectral reconstruction capabilities. This study introduces SwinGAN, a fusion network combining Swin Transformer, CNN, and GAN architectures. SwinGAN's generator employs a detail injection framework to separately extract HSI and MSI features, fusing them to generate spatial residuals. These residuals are injected into the supersampled HSI to produce the final image, while a pure CNN architecture acts as the discriminator, enhancing the fusion quality. Additionally, we introduce a new adaptive loss function that improves image fusion accuracy. The loss function uses L1 loss as the content loss, and spatial and spectral gradient loss functions are introduced to improve the spatial representation and spectral fidelity of the fused images. Our experimental results on several datasets demonstrate that SwinGAN outperforms current popular algorithms in both spatial and spectral reconstruction capabilities. The ablation experiments also demonstrate the rationality of the various components of the proposed loss function.http://dx.doi.org/10.1080/17538947.2023.2253206swinganhsimsiimage fusionspatial gradient lossspectral gradient loss |
| spellingShingle | Chunyu Zhu Shangqi Deng Jiaxin Li Ying Zhang Liwei Gong Liangbo Gao Na Ta Shengbo Chen Qiong Wu Hyperspectral and multispectral remote sensing image fusion using SwinGAN with joint adaptive spatial-spectral gradient loss function International Journal of Digital Earth swingan hsi msi image fusion spatial gradient loss spectral gradient loss |
| title | Hyperspectral and multispectral remote sensing image fusion using SwinGAN with joint adaptive spatial-spectral gradient loss function |
| title_full | Hyperspectral and multispectral remote sensing image fusion using SwinGAN with joint adaptive spatial-spectral gradient loss function |
| title_fullStr | Hyperspectral and multispectral remote sensing image fusion using SwinGAN with joint adaptive spatial-spectral gradient loss function |
| title_full_unstemmed | Hyperspectral and multispectral remote sensing image fusion using SwinGAN with joint adaptive spatial-spectral gradient loss function |
| title_short | Hyperspectral and multispectral remote sensing image fusion using SwinGAN with joint adaptive spatial-spectral gradient loss function |
| title_sort | hyperspectral and multispectral remote sensing image fusion using swingan with joint adaptive spatial spectral gradient loss function |
| topic | swingan hsi msi image fusion spatial gradient loss spectral gradient loss |
| url | http://dx.doi.org/10.1080/17538947.2023.2253206 |
| work_keys_str_mv | AT chunyuzhu hyperspectralandmultispectralremotesensingimagefusionusingswinganwithjointadaptivespatialspectralgradientlossfunction AT shangqideng hyperspectralandmultispectralremotesensingimagefusionusingswinganwithjointadaptivespatialspectralgradientlossfunction AT jiaxinli hyperspectralandmultispectralremotesensingimagefusionusingswinganwithjointadaptivespatialspectralgradientlossfunction AT yingzhang hyperspectralandmultispectralremotesensingimagefusionusingswinganwithjointadaptivespatialspectralgradientlossfunction AT liweigong hyperspectralandmultispectralremotesensingimagefusionusingswinganwithjointadaptivespatialspectralgradientlossfunction AT liangbogao hyperspectralandmultispectralremotesensingimagefusionusingswinganwithjointadaptivespatialspectralgradientlossfunction AT nata hyperspectralandmultispectralremotesensingimagefusionusingswinganwithjointadaptivespatialspectralgradientlossfunction AT shengbochen hyperspectralandmultispectralremotesensingimagefusionusingswinganwithjointadaptivespatialspectralgradientlossfunction AT qiongwu hyperspectralandmultispectralremotesensingimagefusionusingswinganwithjointadaptivespatialspectralgradientlossfunction |