NMF-DuNet: Nonnegative Matrix Factorization Inspired Deep Unrolling Networks for Hyperspectral and Multispectral Image Fusion
The fusion of high-resolution multispectral image (HrMSI) and low-resolution hyperspectral image (LrHSI) has been acknowledged as a promising method for generating a high-resolution hyperspectral image (HrHSI), which is also termed to be an essential part for precise recognition and cataloguing of t...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9822395/ |
| _version_ | 1828396688041050112 |
|---|---|
| author | Abdolraheem Khader Jingxiang Yang Liang Xiao |
| author_facet | Abdolraheem Khader Jingxiang Yang Liang Xiao |
| author_sort | Abdolraheem Khader |
| collection | DOAJ |
| description | The fusion of high-resolution multispectral image (HrMSI) and low-resolution hyperspectral image (LrHSI) has been acknowledged as a promising method for generating a high-resolution hyperspectral image (HrHSI), which is also termed to be an essential part for precise recognition and cataloguing of the underlying materials. In order to improve the fusion of the LrHSI and HrMSI performance, in this article, we propose a novel nonnegative matrix factorization inspired deep unrolling networks (NMF-DuNet) for fusing LrHSI and HrMSI. For this aim, initially, a variational fusion model regularized by nonnegative sparse prior is proposed and then is solved through the gradient descent optimization method and unrolled towards the deep network. The nonnegative coefficient matrices and orthogonal of the proposed transform coefficients constraints are both incorporated into the proposed method. Moreover, the fusion of HrMSI and LrHSI heavily depends on an imaging model that explains the degeneracy of HSI in the spectral and spatial regions. Practically, the imaging model is often unknown. The degradation model is represented implicitly via a proposed network, and both the degradation model and sparse priors are jointly optimized through the training process of the proposed network. Instead of being hand-crafted, all the parameters of NMF-DuNet are learned end-to-end. Compared to the previous state-of-the-art model-based and learning-based fusion approaches, the hardware-friendly proposed NMF-DuNet outperforms both the model-based and learning-based fusion approaches and requires a far smaller number of trainable parameters and storage space while preserving the real-time performance. |
| first_indexed | 2024-12-10T08:34:21Z |
| format | Article |
| id | doaj.art-25b37e15341f4129a84588f77892f5e3 |
| institution | Directory Open Access Journal |
| issn | 2151-1535 |
| language | English |
| last_indexed | 2024-12-10T08:34:21Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| spelling | doaj.art-25b37e15341f4129a84588f77892f5e32022-12-22T01:56:01ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing2151-15352022-01-01155704572010.1109/JSTARS.2022.31895519822395NMF-DuNet: Nonnegative Matrix Factorization Inspired Deep Unrolling Networks for Hyperspectral and Multispectral Image FusionAbdolraheem Khader0https://orcid.org/0000-0002-1164-3103Jingxiang Yang1Liang Xiao2https://orcid.org/0000-0003-0178-9384School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, ChinaSchool of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, ChinaSchool of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, ChinaThe fusion of high-resolution multispectral image (HrMSI) and low-resolution hyperspectral image (LrHSI) has been acknowledged as a promising method for generating a high-resolution hyperspectral image (HrHSI), which is also termed to be an essential part for precise recognition and cataloguing of the underlying materials. In order to improve the fusion of the LrHSI and HrMSI performance, in this article, we propose a novel nonnegative matrix factorization inspired deep unrolling networks (NMF-DuNet) for fusing LrHSI and HrMSI. For this aim, initially, a variational fusion model regularized by nonnegative sparse prior is proposed and then is solved through the gradient descent optimization method and unrolled towards the deep network. The nonnegative coefficient matrices and orthogonal of the proposed transform coefficients constraints are both incorporated into the proposed method. Moreover, the fusion of HrMSI and LrHSI heavily depends on an imaging model that explains the degeneracy of HSI in the spectral and spatial regions. Practically, the imaging model is often unknown. The degradation model is represented implicitly via a proposed network, and both the degradation model and sparse priors are jointly optimized through the training process of the proposed network. Instead of being hand-crafted, all the parameters of NMF-DuNet are learned end-to-end. Compared to the previous state-of-the-art model-based and learning-based fusion approaches, the hardware-friendly proposed NMF-DuNet outperforms both the model-based and learning-based fusion approaches and requires a far smaller number of trainable parameters and storage space while preserving the real-time performance.https://ieeexplore.ieee.org/document/9822395/Deep learninghyperspectral image superresolutionhyperspectral imagingimages fusionsparse coding (SC) |
| spellingShingle | Abdolraheem Khader Jingxiang Yang Liang Xiao NMF-DuNet: Nonnegative Matrix Factorization Inspired Deep Unrolling Networks for Hyperspectral and Multispectral Image Fusion IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Deep learning hyperspectral image superresolution hyperspectral imaging images fusion sparse coding (SC) |
| title | NMF-DuNet: Nonnegative Matrix Factorization Inspired Deep Unrolling Networks for Hyperspectral and Multispectral Image Fusion |
| title_full | NMF-DuNet: Nonnegative Matrix Factorization Inspired Deep Unrolling Networks for Hyperspectral and Multispectral Image Fusion |
| title_fullStr | NMF-DuNet: Nonnegative Matrix Factorization Inspired Deep Unrolling Networks for Hyperspectral and Multispectral Image Fusion |
| title_full_unstemmed | NMF-DuNet: Nonnegative Matrix Factorization Inspired Deep Unrolling Networks for Hyperspectral and Multispectral Image Fusion |
| title_short | NMF-DuNet: Nonnegative Matrix Factorization Inspired Deep Unrolling Networks for Hyperspectral and Multispectral Image Fusion |
| title_sort | nmf dunet nonnegative matrix factorization inspired deep unrolling networks for hyperspectral and multispectral image fusion |
| topic | Deep learning hyperspectral image superresolution hyperspectral imaging images fusion sparse coding (SC) |
| url | https://ieeexplore.ieee.org/document/9822395/ |
| work_keys_str_mv | AT abdolraheemkhader nmfdunetnonnegativematrixfactorizationinspireddeepunrollingnetworksforhyperspectralandmultispectralimagefusion AT jingxiangyang nmfdunetnonnegativematrixfactorizationinspireddeepunrollingnetworksforhyperspectralandmultispectralimagefusion AT liangxiao nmfdunetnonnegativematrixfactorizationinspireddeepunrollingnetworksforhyperspectralandmultispectralimagefusion |