MSFusion: Multistage for Remote Sensing Image Spatiotemporal Fusion Based on Texture Transformer and Convolutional Neural Network
Due to the limitations of current technology and budget, a single satellite sensor cannot obtain high spatiotemporal resolution remote sensing images. Therefore, remote sensing image spatio-temporal fusion technology is considered as an effective solution and has attracted extensive attention. In th...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9786792/ |
| _version_ | 1811332169846489088 |
|---|---|
| author | Guangqi Yang Yurong Qian Hui Liu Bochuan Tang Ranran Qi Yi Lu Jun Geng |
| author_facet | Guangqi Yang Yurong Qian Hui Liu Bochuan Tang Ranran Qi Yi Lu Jun Geng |
| author_sort | Guangqi Yang |
| collection | DOAJ |
| description | Due to the limitations of current technology and budget, a single satellite sensor cannot obtain high spatiotemporal resolution remote sensing images. Therefore, remote sensing image spatio-temporal fusion technology is considered as an effective solution and has attracted extensive attention. In the field of deep learning, due to the fixed size of the perception field of a convolutional neural network, it is impossible to model the correlation of global features, and the features extracted only through convolution operation lack the ability to capture long-distance features. At the same time, complex fusion methods cannot better integrate temporal and spatial features. In order to solve these problems, we propose a multistage remote sensing image spatio-temporal fusion model based on texture transformer and convolutional neural network. The model combines the advantages of transformer and convolutional network, uses a lightweight convolution network to extract spatial features and temporal discrepancy features, uses transformer to learn global temporal correlation, and finally, fuses temporal features with spatial features. In order to make full use of the features obtained in different stages, we design a cross-stage adaptive fusion module CSAFM. The module adopts the self-attention mechanism to adaptively integrate the features of different scales while considering the temporal and spatial characteristics. To test the robustness of the model, the experiments are carried out on three datasets of CIA, LGC, and DX. Compared with five typical spatio-temporal fusion algorithms, we obtain excellent results, which prove the superiority of MSFusion model. |
| first_indexed | 2024-04-13T16:32:45Z |
| format | Article |
| id | doaj.art-b1d1e7abf9f74dd8a283c3c95d92c399 |
| institution | Directory Open Access Journal |
| issn | 2151-1535 |
| language | English |
| last_indexed | 2024-04-13T16:32:45Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| spelling | doaj.art-b1d1e7abf9f74dd8a283c3c95d92c3992022-12-22T02:39:32ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing2151-15352022-01-01154653466610.1109/JSTARS.2022.31794159786792MSFusion: Multistage for Remote Sensing Image Spatiotemporal Fusion Based on Texture Transformer and Convolutional Neural NetworkGuangqi Yang0https://orcid.org/0000-0001-5588-864XYurong Qian1Hui Liu2https://orcid.org/0000-0002-7868-8314Bochuan Tang3Ranran Qi4Yi Lu5Jun Geng6School of Software, Key Laboratory of signal detection and processing in Xinjiang Uygur Autonomous Region, Xinjiang University, Urumqi, ChinaSchool of Software, Key Laboratory of signal detection and processing in Xinjiang Uygur Autonomous Region, Xinjiang University, Urumqi, ChinaKey Laboratory of Software Engineering, Urumqi, ChinaSchool of Software, Key Laboratory of signal detection and processing in Xinjiang Uygur Autonomous Region, Xinjiang University, Urumqi, ChinaSchool of Software, Key Laboratory of signal detection and processing in Xinjiang Uygur Autonomous Region, Xinjiang University, Urumqi, ChinaSchool of Software, Key Laboratory of signal detection and processing in Xinjiang Uygur Autonomous Region, Xinjiang University, Urumqi, ChinaSchool of Software, Xinjiang University, Urumqi, ChinaDue to the limitations of current technology and budget, a single satellite sensor cannot obtain high spatiotemporal resolution remote sensing images. Therefore, remote sensing image spatio-temporal fusion technology is considered as an effective solution and has attracted extensive attention. In the field of deep learning, due to the fixed size of the perception field of a convolutional neural network, it is impossible to model the correlation of global features, and the features extracted only through convolution operation lack the ability to capture long-distance features. At the same time, complex fusion methods cannot better integrate temporal and spatial features. In order to solve these problems, we propose a multistage remote sensing image spatio-temporal fusion model based on texture transformer and convolutional neural network. The model combines the advantages of transformer and convolutional network, uses a lightweight convolution network to extract spatial features and temporal discrepancy features, uses transformer to learn global temporal correlation, and finally, fuses temporal features with spatial features. In order to make full use of the features obtained in different stages, we design a cross-stage adaptive fusion module CSAFM. The module adopts the self-attention mechanism to adaptively integrate the features of different scales while considering the temporal and spatial characteristics. To test the robustness of the model, the experiments are carried out on three datasets of CIA, LGC, and DX. Compared with five typical spatio-temporal fusion algorithms, we obtain excellent results, which prove the superiority of MSFusion model.https://ieeexplore.ieee.org/document/9786792/Multistage feature fusionmultitemporal remote sensing dataremote sensingself-attentionspatiotemporal fusiontransformer |
| spellingShingle | Guangqi Yang Yurong Qian Hui Liu Bochuan Tang Ranran Qi Yi Lu Jun Geng MSFusion: Multistage for Remote Sensing Image Spatiotemporal Fusion Based on Texture Transformer and Convolutional Neural Network IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Multistage feature fusion multitemporal remote sensing data remote sensing self-attention spatiotemporal fusion transformer |
| title | MSFusion: Multistage for Remote Sensing Image Spatiotemporal Fusion Based on Texture Transformer and Convolutional Neural Network |
| title_full | MSFusion: Multistage for Remote Sensing Image Spatiotemporal Fusion Based on Texture Transformer and Convolutional Neural Network |
| title_fullStr | MSFusion: Multistage for Remote Sensing Image Spatiotemporal Fusion Based on Texture Transformer and Convolutional Neural Network |
| title_full_unstemmed | MSFusion: Multistage for Remote Sensing Image Spatiotemporal Fusion Based on Texture Transformer and Convolutional Neural Network |
| title_short | MSFusion: Multistage for Remote Sensing Image Spatiotemporal Fusion Based on Texture Transformer and Convolutional Neural Network |
| title_sort | msfusion multistage for remote sensing image spatiotemporal fusion based on texture transformer and convolutional neural network |
| topic | Multistage feature fusion multitemporal remote sensing data remote sensing self-attention spatiotemporal fusion transformer |
| url | https://ieeexplore.ieee.org/document/9786792/ |
| work_keys_str_mv | AT guangqiyang msfusionmultistageforremotesensingimagespatiotemporalfusionbasedontexturetransformerandconvolutionalneuralnetwork AT yurongqian msfusionmultistageforremotesensingimagespatiotemporalfusionbasedontexturetransformerandconvolutionalneuralnetwork AT huiliu msfusionmultistageforremotesensingimagespatiotemporalfusionbasedontexturetransformerandconvolutionalneuralnetwork AT bochuantang msfusionmultistageforremotesensingimagespatiotemporalfusionbasedontexturetransformerandconvolutionalneuralnetwork AT ranranqi msfusionmultistageforremotesensingimagespatiotemporalfusionbasedontexturetransformerandconvolutionalneuralnetwork AT yilu msfusionmultistageforremotesensingimagespatiotemporalfusionbasedontexturetransformerandconvolutionalneuralnetwork AT jungeng msfusionmultistageforremotesensingimagespatiotemporalfusionbasedontexturetransformerandconvolutionalneuralnetwork |