Physics-Based GAN With Iterative Refinement Unit for Hyperspectral and Multispectral Image Fusion
Hyperspectral image (HSI) fusion can effectively improve the spatial resolution of HSIs by integrating high-resolution multispectral images (MSIs). Considering the spatial and spectral degradation relationship between a fused image and input images, a physics-based GAN is proposed to fuse HSI and MS...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9435191/ |
| _version_ | 1818619377019256832 |
|---|---|
| author | Jiajun Xiao Jie Li Qiangqiang Yuan Menghui Jiang Liangpei Zhang |
| author_facet | Jiajun Xiao Jie Li Qiangqiang Yuan Menghui Jiang Liangpei Zhang |
| author_sort | Jiajun Xiao |
| collection | DOAJ |
| description | Hyperspectral image (HSI) fusion can effectively improve the spatial resolution of HSIs by integrating high-resolution multispectral images (MSIs). Considering the spatial and spectral degradation relationship between a fused image and input images, a physics-based GAN is proposed to fuse HSI and MSI. A physical model estimating degradation of image is introduced in the generator and in the discriminators. For the generator, a set of recursive modules including a physical degradation model and a multiscale residual channel attention fusion module integrate the spectral-spatial difference information between input images and estimated degradation images to restore the details of the fused image. Subsequently, the residual spatial attention fusion module is used to combine the results of all recursions to obtain the final reconstructed result. As for the discriminators, three networks with the final fused image, estimated LR HSI and estimated MSI as inputs share the same architecture. Finally, the loss function that contains adversarial losses and L1 losses of the fused image and estimated degradation images is used to optimize network parameters. The experimental results demonstrate that the proposed method outperforms state-of-the-art methods. |
| first_indexed | 2024-12-16T17:36:31Z |
| format | Article |
| id | doaj.art-1a45fd5b14354d00a4106ed56a3c81e8 |
| institution | Directory Open Access Journal |
| issn | 2151-1535 |
| language | English |
| last_indexed | 2024-12-16T17:36:31Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| spelling | doaj.art-1a45fd5b14354d00a4106ed56a3c81e82022-12-21T22:22:45ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing2151-15352021-01-01146827684110.1109/JSTARS.2021.30757279435191Physics-Based GAN With Iterative Refinement Unit for Hyperspectral and Multispectral Image FusionJiajun Xiao0Jie Li1https://orcid.org/0000-0002-4063-9381Qiangqiang Yuan2https://orcid.org/0000-0001-7140-2224Menghui Jiang3Liangpei Zhang4https://orcid.org/0000-0001-6890-3650School of Geodesy and Geomatics, Wuhan University, Wuhan, ChinaSchool of Geodesy and Geomatics, Wuhan University, Wuhan, ChinaSchool of Geodesy and Geomatics, Wuhan University, Wuhan, ChinaSchool of Resource and Environmental Sciences, Wuhan University, Wuhan, ChinaState Key Laboratory of Information Engineering in Surveying, Mapping, and Remote Sensing, Wuhan University, Wuhan, ChinaHyperspectral image (HSI) fusion can effectively improve the spatial resolution of HSIs by integrating high-resolution multispectral images (MSIs). Considering the spatial and spectral degradation relationship between a fused image and input images, a physics-based GAN is proposed to fuse HSI and MSI. A physical model estimating degradation of image is introduced in the generator and in the discriminators. For the generator, a set of recursive modules including a physical degradation model and a multiscale residual channel attention fusion module integrate the spectral-spatial difference information between input images and estimated degradation images to restore the details of the fused image. Subsequently, the residual spatial attention fusion module is used to combine the results of all recursions to obtain the final reconstructed result. As for the discriminators, three networks with the final fused image, estimated LR HSI and estimated MSI as inputs share the same architecture. Finally, the loss function that contains adversarial losses and L1 losses of the fused image and estimated degradation images is used to optimize network parameters. The experimental results demonstrate that the proposed method outperforms state-of-the-art methods.https://ieeexplore.ieee.org/document/9435191/Attention moduleGANhyperspectral image (HSI)physics model |
| spellingShingle | Jiajun Xiao Jie Li Qiangqiang Yuan Menghui Jiang Liangpei Zhang Physics-Based GAN With Iterative Refinement Unit for Hyperspectral and Multispectral Image Fusion IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Attention module GAN hyperspectral image (HSI) physics model |
| title | Physics-Based GAN With Iterative Refinement Unit for Hyperspectral and Multispectral Image Fusion |
| title_full | Physics-Based GAN With Iterative Refinement Unit for Hyperspectral and Multispectral Image Fusion |
| title_fullStr | Physics-Based GAN With Iterative Refinement Unit for Hyperspectral and Multispectral Image Fusion |
| title_full_unstemmed | Physics-Based GAN With Iterative Refinement Unit for Hyperspectral and Multispectral Image Fusion |
| title_short | Physics-Based GAN With Iterative Refinement Unit for Hyperspectral and Multispectral Image Fusion |
| title_sort | physics based gan with iterative refinement unit for hyperspectral and multispectral image fusion |
| topic | Attention module GAN hyperspectral image (HSI) physics model |
| url | https://ieeexplore.ieee.org/document/9435191/ |
| work_keys_str_mv | AT jiajunxiao physicsbasedganwithiterativerefinementunitforhyperspectralandmultispectralimagefusion AT jieli physicsbasedganwithiterativerefinementunitforhyperspectralandmultispectralimagefusion AT qiangqiangyuan physicsbasedganwithiterativerefinementunitforhyperspectralandmultispectralimagefusion AT menghuijiang physicsbasedganwithiterativerefinementunitforhyperspectralandmultispectralimagefusion AT liangpeizhang physicsbasedganwithiterativerefinementunitforhyperspectralandmultispectralimagefusion |