Self-Supervised Pretraining of Transformers for Satellite Image Time Series Classification
Satellite image time series (SITS) classification is a major research topic in remote sensing and is relevant for a wide range of applications. Deep learning approaches have been commonly employed for the SITS classification and have provided state-of-the-art performance. However, deep learning meth...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9252123/ |
| _version_ | 1818893251813310464 |
|---|---|
| author | Yuan Yuan Lei Lin |
| author_facet | Yuan Yuan Lei Lin |
| author_sort | Yuan Yuan |
| collection | DOAJ |
| description | Satellite image time series (SITS) classification is a major research topic in remote sensing and is relevant for a wide range of applications. Deep learning approaches have been commonly employed for the SITS classification and have provided state-of-the-art performance. However, deep learning methods suffer from overfitting when labeled data are scarce. To address this problem, we propose a novel self-supervised pretraining scheme to initialize a transformer-based network by utilizing large-scale unlabeled data. In detail, the model is asked to predict randomly contaminated observations given an entire time series of a pixel. The main idea of our proposal is to leverage the inherent temporal structure of satellite time series to learn general-purpose spectral-temporal representations related to land cover semantics. Once pretraining is completed, the pretrained network can be further adapted to various SITS classification tasks by fine-tuning all the model parameters on small-scale task-related labeled data. In this way, the general knowledge and representations about SITS can be transferred to a label-scarce task, thereby improving the generalization performance of the model as well as reducing the risk of overfitting. Comprehensive experiments have been carried out on three benchmark datasets over large study areas. Experimental results demonstrate the effectiveness of the proposed pretraining scheme, leading to substantial improvements in classification accuracy using transformer, 1-D convolutional neural network, and bidirectional long short-term memory network. The code and the pretrained model will be available at https://github.com/linlei1214/SITS-BERT upon publication. |
| first_indexed | 2024-12-19T18:09:38Z |
| format | Article |
| id | doaj.art-4fb58140e0d948d39692eecad8be09a1 |
| institution | Directory Open Access Journal |
| issn | 2151-1535 |
| language | English |
| last_indexed | 2024-12-19T18:09:38Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| spelling | doaj.art-4fb58140e0d948d39692eecad8be09a12022-12-21T20:11:21ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing2151-15352021-01-011447448710.1109/JSTARS.2020.30366029252123Self-Supervised Pretraining of Transformers for Satellite Image Time Series ClassificationYuan Yuan0https://orcid.org/0000-0003-1860-3275Lei Lin1https://orcid.org/0000-0002-7012-4901School of Geographic and Biologic Information, Nanjing University of Posts and Telecommunications, Nanjing, ChinaBeijing Qihoo Technology Company Ltd., Beijing, ChinaSatellite image time series (SITS) classification is a major research topic in remote sensing and is relevant for a wide range of applications. Deep learning approaches have been commonly employed for the SITS classification and have provided state-of-the-art performance. However, deep learning methods suffer from overfitting when labeled data are scarce. To address this problem, we propose a novel self-supervised pretraining scheme to initialize a transformer-based network by utilizing large-scale unlabeled data. In detail, the model is asked to predict randomly contaminated observations given an entire time series of a pixel. The main idea of our proposal is to leverage the inherent temporal structure of satellite time series to learn general-purpose spectral-temporal representations related to land cover semantics. Once pretraining is completed, the pretrained network can be further adapted to various SITS classification tasks by fine-tuning all the model parameters on small-scale task-related labeled data. In this way, the general knowledge and representations about SITS can be transferred to a label-scarce task, thereby improving the generalization performance of the model as well as reducing the risk of overfitting. Comprehensive experiments have been carried out on three benchmark datasets over large study areas. Experimental results demonstrate the effectiveness of the proposed pretraining scheme, leading to substantial improvements in classification accuracy using transformer, 1-D convolutional neural network, and bidirectional long short-term memory network. The code and the pretrained model will be available at https://github.com/linlei1214/SITS-BERT upon publication.https://ieeexplore.ieee.org/document/9252123/Bidirectional encoder representations from Transformers (BERT)classificationsatellite image time series (SITS)self-supervised learningtransfer learningunsupervised pretraining |
| spellingShingle | Yuan Yuan Lei Lin Self-Supervised Pretraining of Transformers for Satellite Image Time Series Classification IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Bidirectional encoder representations from Transformers (BERT) classification satellite image time series (SITS) self-supervised learning transfer learning unsupervised pretraining |
| title | Self-Supervised Pretraining of Transformers for Satellite Image Time Series Classification |
| title_full | Self-Supervised Pretraining of Transformers for Satellite Image Time Series Classification |
| title_fullStr | Self-Supervised Pretraining of Transformers for Satellite Image Time Series Classification |
| title_full_unstemmed | Self-Supervised Pretraining of Transformers for Satellite Image Time Series Classification |
| title_short | Self-Supervised Pretraining of Transformers for Satellite Image Time Series Classification |
| title_sort | self supervised pretraining of transformers for satellite image time series classification |
| topic | Bidirectional encoder representations from Transformers (BERT) classification satellite image time series (SITS) self-supervised learning transfer learning unsupervised pretraining |
| url | https://ieeexplore.ieee.org/document/9252123/ |
| work_keys_str_mv | AT yuanyuan selfsupervisedpretrainingoftransformersforsatelliteimagetimeseriesclassification AT leilin selfsupervisedpretrainingoftransformersforsatelliteimagetimeseriesclassification |