Gated Convolutional Networks for Cloud Removal From Bi-Temporal Remote Sensing Images
Pixels of clouds and cloud shadows in a remote sensing image impact image quality, image interpretation, and subsequent applications. In this paper, we propose a novel cloud removal method based on deep learning that automatically reconstructs the invalid pixels with the auxiliary information from m...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-10-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/12/20/3427 |
| _version_ | 1827704021617475584 |
|---|---|
| author | Peiyu Dai Shunping Ji Yongjun Zhang |
| author_facet | Peiyu Dai Shunping Ji Yongjun Zhang |
| author_sort | Peiyu Dai |
| collection | DOAJ |
| description | Pixels of clouds and cloud shadows in a remote sensing image impact image quality, image interpretation, and subsequent applications. In this paper, we propose a novel cloud removal method based on deep learning that automatically reconstructs the invalid pixels with the auxiliary information from multi-temporal images. Our method’s innovation lies in its feature extraction and loss functions, which reside in a novel gated convolutional network (GCN) instead of a series of common convolutions. It takes the current cloudy image, a recent cloudless image, and the mask of clouds as input, without any requirements of external training samples, to realize a self-training process with clean pixels in the bi-temporal images as natural training samples. In our feature extraction, gated convolutional layers, for the first time, are introduced to discriminate cloudy pixels from clean pixels, which make up for a common convolution layer’s lack of the ability to discriminate. Our multi-level constrained joint loss function, which consists of an image-level loss, a feature-level loss, and a total variation loss, can achieve local and global consistency both in shallow and deep levels of features. The total variation loss is introduced into the deep-learning-based cloud removal task for the first time to eliminate the color and texture discontinuity around cloud outlines needing repair. On the WHU cloud dataset with diverse land cover scenes and different imaging conditions, our experimental results demonstrated that our method consistently reconstructed the cloud and cloud shadow pixels in various remote sensing images and outperformed several mainstream deep-learning-based methods and a conventional method for every indicator by a large margin. |
| first_indexed | 2024-03-10T15:30:55Z |
| format | Article |
| id | doaj.art-cdf6e549c2e94def95cea90ac1138e45 |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-10T15:30:55Z |
| publishDate | 2020-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-cdf6e549c2e94def95cea90ac1138e452023-11-20T17:38:23ZengMDPI AGRemote Sensing2072-42922020-10-011220342710.3390/rs12203427Gated Convolutional Networks for Cloud Removal From Bi-Temporal Remote Sensing ImagesPeiyu Dai0Shunping Ji1Yongjun Zhang2School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, ChinaSchool of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, ChinaSchool of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, ChinaPixels of clouds and cloud shadows in a remote sensing image impact image quality, image interpretation, and subsequent applications. In this paper, we propose a novel cloud removal method based on deep learning that automatically reconstructs the invalid pixels with the auxiliary information from multi-temporal images. Our method’s innovation lies in its feature extraction and loss functions, which reside in a novel gated convolutional network (GCN) instead of a series of common convolutions. It takes the current cloudy image, a recent cloudless image, and the mask of clouds as input, without any requirements of external training samples, to realize a self-training process with clean pixels in the bi-temporal images as natural training samples. In our feature extraction, gated convolutional layers, for the first time, are introduced to discriminate cloudy pixels from clean pixels, which make up for a common convolution layer’s lack of the ability to discriminate. Our multi-level constrained joint loss function, which consists of an image-level loss, a feature-level loss, and a total variation loss, can achieve local and global consistency both in shallow and deep levels of features. The total variation loss is introduced into the deep-learning-based cloud removal task for the first time to eliminate the color and texture discontinuity around cloud outlines needing repair. On the WHU cloud dataset with diverse land cover scenes and different imaging conditions, our experimental results demonstrated that our method consistently reconstructed the cloud and cloud shadow pixels in various remote sensing images and outperformed several mainstream deep-learning-based methods and a conventional method for every indicator by a large margin.https://www.mdpi.com/2072-4292/12/20/3427cloud removalgated convolutionmulti-temporal remote sensing imagesjoint loss function |
| spellingShingle | Peiyu Dai Shunping Ji Yongjun Zhang Gated Convolutional Networks for Cloud Removal From Bi-Temporal Remote Sensing Images Remote Sensing cloud removal gated convolution multi-temporal remote sensing images joint loss function |
| title | Gated Convolutional Networks for Cloud Removal From Bi-Temporal Remote Sensing Images |
| title_full | Gated Convolutional Networks for Cloud Removal From Bi-Temporal Remote Sensing Images |
| title_fullStr | Gated Convolutional Networks for Cloud Removal From Bi-Temporal Remote Sensing Images |
| title_full_unstemmed | Gated Convolutional Networks for Cloud Removal From Bi-Temporal Remote Sensing Images |
| title_short | Gated Convolutional Networks for Cloud Removal From Bi-Temporal Remote Sensing Images |
| title_sort | gated convolutional networks for cloud removal from bi temporal remote sensing images |
| topic | cloud removal gated convolution multi-temporal remote sensing images joint loss function |
| url | https://www.mdpi.com/2072-4292/12/20/3427 |
| work_keys_str_mv | AT peiyudai gatedconvolutionalnetworksforcloudremovalfrombitemporalremotesensingimages AT shunpingji gatedconvolutionalnetworksforcloudremovalfrombitemporalremotesensingimages AT yongjunzhang gatedconvolutionalnetworksforcloudremovalfrombitemporalremotesensingimages |