Landsat-8 to Sentinel-2 Satellite Imagery Super-Resolution-Based Multiscale Dilated Transformer Generative Adversarial Networks
Image super-resolution (SR) techniques can improve the spatial resolution of remote sensing images to provide more feature details and information, which is important for a wide range of remote sensing applications, including land use/cover classification (LUCC). Convolutional neural networks (CNNs)...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-11-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/15/22/5272 |
| _version_ | 1797457886900649984 |
|---|---|
| author | Chunyang Wang Xian Zhang Wei Yang Gaige Wang Zongze Zhao Xuan Liu Bibo Lu |
| author_facet | Chunyang Wang Xian Zhang Wei Yang Gaige Wang Zongze Zhao Xuan Liu Bibo Lu |
| author_sort | Chunyang Wang |
| collection | DOAJ |
| description | Image super-resolution (SR) techniques can improve the spatial resolution of remote sensing images to provide more feature details and information, which is important for a wide range of remote sensing applications, including land use/cover classification (LUCC). Convolutional neural networks (CNNs) have achieved impressive results in the field of image SR, but the inherent localization of convolution limits the performance of CNN-based SR models. Therefore, we propose a new method, namely, the dilated Transformer generative adversarial network (DTGAN) for the SR of multispectral remote sensing images. DTGAN combines the local focus of CNNs with the global perspective of Transformers to better capture both local and global features in remote sensing images. We introduce dilated convolutions into the self-attention computation of Transformers to control the network’s focus on different scales of image features. This enhancement improves the network’s ability to reconstruct details at various scales in the images. SR imagery provides richer surface information and reduces ambiguity for the LUCC task, thereby enhancing the accuracy of LUCC. Our work comprises two main stages: remote sensing image SR and LUCC. In the SR stage, we conducted comprehensive experiments on Landsat-8 (L8) and Sentinel-2 (S2) remote sensing datasets. The results indicate that DTGAN generates super-resolution (SR) images with minimal computation. Additionally, it outperforms other methods in terms of the spectral angle mapper (SAM) and learned perceptual image patch similarity (LPIPS) metrics, as well as visual quality. In the LUCC stage, DTGAN was used to generate SR images of areas outside the training samples, and then the SR imagery was used in the LUCC task. The results indicated a significant improvement in the accuracy of LUCC based on SR imagery compared to low-resolution (LR) LUCC maps. Specifically, there were enhancements of 0.130 in precision, 0.178 in recall, and 0.157 in the F1-score. |
| first_indexed | 2024-03-09T16:29:13Z |
| format | Article |
| id | doaj.art-30cc987b689246298f78f0b7059cd4d2 |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-09T16:29:13Z |
| publishDate | 2023-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-30cc987b689246298f78f0b7059cd4d22023-11-24T15:04:08ZengMDPI AGRemote Sensing2072-42922023-11-011522527210.3390/rs15225272Landsat-8 to Sentinel-2 Satellite Imagery Super-Resolution-Based Multiscale Dilated Transformer Generative Adversarial NetworksChunyang Wang0Xian Zhang1Wei Yang2Gaige Wang3Zongze Zhao4Xuan Liu5Bibo Lu6School of Computer Science and Technology, Henan Polytechnic University, Jiaozuo 454000, ChinaSchool of Surveying and Land Information Engineering, Henan Polytechnic University, Jiaozuo 454000, ChinaCenter for Environmental Remote Sensing, Chiba University, Chiba 2638522, JapanSchool of Computer Science and Technology, Ocean University of China, Qingdao 266100, ChinaSchool of Surveying and Land Information Engineering, Henan Polytechnic University, Jiaozuo 454000, ChinaSchool of Surveying and Land Information Engineering, Henan Polytechnic University, Jiaozuo 454000, ChinaSchool of Computer Science and Technology, Henan Polytechnic University, Jiaozuo 454000, ChinaImage super-resolution (SR) techniques can improve the spatial resolution of remote sensing images to provide more feature details and information, which is important for a wide range of remote sensing applications, including land use/cover classification (LUCC). Convolutional neural networks (CNNs) have achieved impressive results in the field of image SR, but the inherent localization of convolution limits the performance of CNN-based SR models. Therefore, we propose a new method, namely, the dilated Transformer generative adversarial network (DTGAN) for the SR of multispectral remote sensing images. DTGAN combines the local focus of CNNs with the global perspective of Transformers to better capture both local and global features in remote sensing images. We introduce dilated convolutions into the self-attention computation of Transformers to control the network’s focus on different scales of image features. This enhancement improves the network’s ability to reconstruct details at various scales in the images. SR imagery provides richer surface information and reduces ambiguity for the LUCC task, thereby enhancing the accuracy of LUCC. Our work comprises two main stages: remote sensing image SR and LUCC. In the SR stage, we conducted comprehensive experiments on Landsat-8 (L8) and Sentinel-2 (S2) remote sensing datasets. The results indicate that DTGAN generates super-resolution (SR) images with minimal computation. Additionally, it outperforms other methods in terms of the spectral angle mapper (SAM) and learned perceptual image patch similarity (LPIPS) metrics, as well as visual quality. In the LUCC stage, DTGAN was used to generate SR images of areas outside the training samples, and then the SR imagery was used in the LUCC task. The results indicated a significant improvement in the accuracy of LUCC based on SR imagery compared to low-resolution (LR) LUCC maps. Specifically, there were enhancements of 0.130 in precision, 0.178 in recall, and 0.157 in the F1-score.https://www.mdpi.com/2072-4292/15/22/5272super-resolutionland use/cover classificationdeep learningremote sensingTransformergenerative adversarial network |
| spellingShingle | Chunyang Wang Xian Zhang Wei Yang Gaige Wang Zongze Zhao Xuan Liu Bibo Lu Landsat-8 to Sentinel-2 Satellite Imagery Super-Resolution-Based Multiscale Dilated Transformer Generative Adversarial Networks Remote Sensing super-resolution land use/cover classification deep learning remote sensing Transformer generative adversarial network |
| title | Landsat-8 to Sentinel-2 Satellite Imagery Super-Resolution-Based Multiscale Dilated Transformer Generative Adversarial Networks |
| title_full | Landsat-8 to Sentinel-2 Satellite Imagery Super-Resolution-Based Multiscale Dilated Transformer Generative Adversarial Networks |
| title_fullStr | Landsat-8 to Sentinel-2 Satellite Imagery Super-Resolution-Based Multiscale Dilated Transformer Generative Adversarial Networks |
| title_full_unstemmed | Landsat-8 to Sentinel-2 Satellite Imagery Super-Resolution-Based Multiscale Dilated Transformer Generative Adversarial Networks |
| title_short | Landsat-8 to Sentinel-2 Satellite Imagery Super-Resolution-Based Multiscale Dilated Transformer Generative Adversarial Networks |
| title_sort | landsat 8 to sentinel 2 satellite imagery super resolution based multiscale dilated transformer generative adversarial networks |
| topic | super-resolution land use/cover classification deep learning remote sensing Transformer generative adversarial network |
| url | https://www.mdpi.com/2072-4292/15/22/5272 |
| work_keys_str_mv | AT chunyangwang landsat8tosentinel2satelliteimagerysuperresolutionbasedmultiscaledilatedtransformergenerativeadversarialnetworks AT xianzhang landsat8tosentinel2satelliteimagerysuperresolutionbasedmultiscaledilatedtransformergenerativeadversarialnetworks AT weiyang landsat8tosentinel2satelliteimagerysuperresolutionbasedmultiscaledilatedtransformergenerativeadversarialnetworks AT gaigewang landsat8tosentinel2satelliteimagerysuperresolutionbasedmultiscaledilatedtransformergenerativeadversarialnetworks AT zongzezhao landsat8tosentinel2satelliteimagerysuperresolutionbasedmultiscaledilatedtransformergenerativeadversarialnetworks AT xuanliu landsat8tosentinel2satelliteimagerysuperresolutionbasedmultiscaledilatedtransformergenerativeadversarialnetworks AT bibolu landsat8tosentinel2satelliteimagerysuperresolutionbasedmultiscaledilatedtransformergenerativeadversarialnetworks |