Deep Learning-Based Single Image Super-Resolution: An Investigation for Dense Scene Reconstruction with UAS Photogrammetry
The deep convolutional neural network (DCNN) has recently been applied to the highly challenging and ill-posed problem of single image super-resolution (SISR), which aims to predict high-resolution (HR) images from their corresponding low-resolution (LR) images. In many remote sensing (RS) applicati...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-05-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/12/11/1757 |
| _version_ | 1797566702804795392 |
|---|---|
| author | Mohammad Pashaei Michael J. Starek Hamid Kamangir Jacob Berryhill |
| author_facet | Mohammad Pashaei Michael J. Starek Hamid Kamangir Jacob Berryhill |
| author_sort | Mohammad Pashaei |
| collection | DOAJ |
| description | The deep convolutional neural network (DCNN) has recently been applied to the highly challenging and ill-posed problem of single image super-resolution (SISR), which aims to predict high-resolution (HR) images from their corresponding low-resolution (LR) images. In many remote sensing (RS) applications, spatial resolution of the aerial or satellite imagery has a great impact on the accuracy and reliability of information extracted from the images. In this study, the potential of a DCNN-based SISR model, called enhanced super-resolution generative adversarial network (ESRGAN), to predict the spatial information degraded or lost in a hyper-spatial resolution unmanned aircraft system (UAS) RGB image set is investigated. ESRGAN model is trained over a limited number of original HR (50 out of 450 total images) and virtually-generated LR UAS images by downsampling the original HR images using a bicubic kernel with a factor <inline-formula> <math display="inline"> <semantics> <mrow> <mo>×</mo> <mn>4</mn> </mrow> </semantics> </math> </inline-formula>. Quantitative and qualitative assessments of super-resolved images using standard image quality measures (IQMs) confirm that the DCNN-based SISR approach can be successfully applied on LR UAS imagery for spatial resolution enhancement. The performance of DCNN-based SISR approach for the UAS image set closely approximates performances reported on standard SISR image sets with mean peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) index values of around 28 dB and 0.85 dB, respectively. Furthermore, by exploiting the rigorous Structure-from-Motion (SfM) photogrammetry procedure, an accurate task-based IQM for evaluating the quality of the super-resolved images is carried out. Results verify that the interior and exterior imaging geometry, which are extremely important for extracting highly accurate spatial information from UAS imagery in photogrammetric applications, can be accurately retrieved from a super-resolved image set. The number of corresponding keypoints and dense points generated from the SfM photogrammetry process are about 6 and 17 times more than those extracted from the corresponding LR image set, respectively. |
| first_indexed | 2024-03-10T19:30:13Z |
| format | Article |
| id | doaj.art-fd1d6380bbb64b938f4c7a9d3101da0d |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-10T19:30:13Z |
| publishDate | 2020-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-fd1d6380bbb64b938f4c7a9d3101da0d2023-11-20T02:12:37ZengMDPI AGRemote Sensing2072-42922020-05-011211175710.3390/rs12111757Deep Learning-Based Single Image Super-Resolution: An Investigation for Dense Scene Reconstruction with UAS PhotogrammetryMohammad Pashaei0Michael J. Starek1Hamid Kamangir2Jacob Berryhill3Department of Computing Sciences, Texas A&M University-Corpus Christi, Corpus Christi, TX 78412, USADepartment of Computing Sciences, Texas A&M University-Corpus Christi, Corpus Christi, TX 78412, USAConrad Blucher Institute for Surveying and Science, Texas A&M University-Corpus Christi, Corpus Christi, TX 78412, USAConrad Blucher Institute for Surveying and Science, Texas A&M University-Corpus Christi, Corpus Christi, TX 78412, USAThe deep convolutional neural network (DCNN) has recently been applied to the highly challenging and ill-posed problem of single image super-resolution (SISR), which aims to predict high-resolution (HR) images from their corresponding low-resolution (LR) images. In many remote sensing (RS) applications, spatial resolution of the aerial or satellite imagery has a great impact on the accuracy and reliability of information extracted from the images. In this study, the potential of a DCNN-based SISR model, called enhanced super-resolution generative adversarial network (ESRGAN), to predict the spatial information degraded or lost in a hyper-spatial resolution unmanned aircraft system (UAS) RGB image set is investigated. ESRGAN model is trained over a limited number of original HR (50 out of 450 total images) and virtually-generated LR UAS images by downsampling the original HR images using a bicubic kernel with a factor <inline-formula> <math display="inline"> <semantics> <mrow> <mo>×</mo> <mn>4</mn> </mrow> </semantics> </math> </inline-formula>. Quantitative and qualitative assessments of super-resolved images using standard image quality measures (IQMs) confirm that the DCNN-based SISR approach can be successfully applied on LR UAS imagery for spatial resolution enhancement. The performance of DCNN-based SISR approach for the UAS image set closely approximates performances reported on standard SISR image sets with mean peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) index values of around 28 dB and 0.85 dB, respectively. Furthermore, by exploiting the rigorous Structure-from-Motion (SfM) photogrammetry procedure, an accurate task-based IQM for evaluating the quality of the super-resolved images is carried out. Results verify that the interior and exterior imaging geometry, which are extremely important for extracting highly accurate spatial information from UAS imagery in photogrammetric applications, can be accurately retrieved from a super-resolved image set. The number of corresponding keypoints and dense points generated from the SfM photogrammetry process are about 6 and 17 times more than those extracted from the corresponding LR image set, respectively.https://www.mdpi.com/2072-4292/12/11/1757unmanned aircraft system (UAS)deep learningsuper-resolution (SR)convolutional neural network (CNN)generative adversarial network (GAN)structure-from-motion |
| spellingShingle | Mohammad Pashaei Michael J. Starek Hamid Kamangir Jacob Berryhill Deep Learning-Based Single Image Super-Resolution: An Investigation for Dense Scene Reconstruction with UAS Photogrammetry Remote Sensing unmanned aircraft system (UAS) deep learning super-resolution (SR) convolutional neural network (CNN) generative adversarial network (GAN) structure-from-motion |
| title | Deep Learning-Based Single Image Super-Resolution: An Investigation for Dense Scene Reconstruction with UAS Photogrammetry |
| title_full | Deep Learning-Based Single Image Super-Resolution: An Investigation for Dense Scene Reconstruction with UAS Photogrammetry |
| title_fullStr | Deep Learning-Based Single Image Super-Resolution: An Investigation for Dense Scene Reconstruction with UAS Photogrammetry |
| title_full_unstemmed | Deep Learning-Based Single Image Super-Resolution: An Investigation for Dense Scene Reconstruction with UAS Photogrammetry |
| title_short | Deep Learning-Based Single Image Super-Resolution: An Investigation for Dense Scene Reconstruction with UAS Photogrammetry |
| title_sort | deep learning based single image super resolution an investigation for dense scene reconstruction with uas photogrammetry |
| topic | unmanned aircraft system (UAS) deep learning super-resolution (SR) convolutional neural network (CNN) generative adversarial network (GAN) structure-from-motion |
| url | https://www.mdpi.com/2072-4292/12/11/1757 |
| work_keys_str_mv | AT mohammadpashaei deeplearningbasedsingleimagesuperresolutionaninvestigationfordensescenereconstructionwithuasphotogrammetry AT michaeljstarek deeplearningbasedsingleimagesuperresolutionaninvestigationfordensescenereconstructionwithuasphotogrammetry AT hamidkamangir deeplearningbasedsingleimagesuperresolutionaninvestigationfordensescenereconstructionwithuasphotogrammetry AT jacobberryhill deeplearningbasedsingleimagesuperresolutionaninvestigationfordensescenereconstructionwithuasphotogrammetry |