Two-Stream Swin Transformer with Differentiable Sobel Operator for Remote Sensing Image Classification

Remote sensing (RS) image classification has attracted much attention recently and is widely used in various fields. Different to natural images, the RS image scenes consist of complex backgrounds and various stochastically arranged objects, thus making it difficult for networks to focus on the targ...

Full description

Bibliographic Details
Main Authors: Siyuan Hao, Bin Wu, Kun Zhao, Yuanxin Ye, Wei Wang
Format: Article
Language:English
Published: MDPI AG 2022-03-01
Series:Remote Sensing
Subjects:
Online Access:https://www.mdpi.com/2072-4292/14/6/1507
_version_ 1797442545299488768
author Siyuan Hao
Bin Wu
Kun Zhao
Yuanxin Ye
Wei Wang
author_facet Siyuan Hao
Bin Wu
Kun Zhao
Yuanxin Ye
Wei Wang
author_sort Siyuan Hao
collection DOAJ
description Remote sensing (RS) image classification has attracted much attention recently and is widely used in various fields. Different to natural images, the RS image scenes consist of complex backgrounds and various stochastically arranged objects, thus making it difficult for networks to focus on the target objects in the scene. However, conventional classification methods do not have any special treatment for remote sensing images. In this paper, we propose a two-stream swin transformer network (TSTNet) to address these issues. TSTNet consists of two streams (i.e., original stream and edge stream) which use both the deep features of the original images and the ones from the edges to make predictions. The swin transformer is used as the backbone of each stream given its good performance. In addition, a differentiable edge Sobel operator module (DESOM) is included in the edge stream which can learn the parameters of Sobel operator adaptively and provide more robust edge information that can suppress background noise. Experimental results on three publicly available remote sensing datasets show that our TSTNet achieves superior performance over the state-of-the-art (SOTA) methods.
first_indexed 2024-03-09T12:44:28Z
format Article
id doaj.art-03de899bb81d4802a9381dc2413f18be
institution Directory Open Access Journal
issn 2072-4292
language English
last_indexed 2024-03-09T12:44:28Z
publishDate 2022-03-01
publisher MDPI AG
record_format Article
series Remote Sensing
spelling doaj.art-03de899bb81d4802a9381dc2413f18be2023-11-30T22:14:09ZengMDPI AGRemote Sensing2072-42922022-03-01146150710.3390/rs14061507Two-Stream Swin Transformer with Differentiable Sobel Operator for Remote Sensing Image ClassificationSiyuan Hao0Bin Wu1Kun Zhao2Yuanxin Ye3Wei Wang4The College of Communication and Electronic Engineering, Qingdao University of Technology, Qingdao 266520, ChinaThe College of Communication and Electronic Engineering, Qingdao University of Technology, Qingdao 266520, ChinaThe College of Communication and Electronic Engineering, Qingdao University of Technology, Qingdao 266520, ChinaThe Faculty of Geosciences and Environmental Engineering, Southwest Jiaotong University, Chengdu 610031, ChinaThe Department of Information Engineering and Computer Science, University of Trento, Via Sommarive, 38123 Trento, ItalyRemote sensing (RS) image classification has attracted much attention recently and is widely used in various fields. Different to natural images, the RS image scenes consist of complex backgrounds and various stochastically arranged objects, thus making it difficult for networks to focus on the target objects in the scene. However, conventional classification methods do not have any special treatment for remote sensing images. In this paper, we propose a two-stream swin transformer network (TSTNet) to address these issues. TSTNet consists of two streams (i.e., original stream and edge stream) which use both the deep features of the original images and the ones from the edges to make predictions. The swin transformer is used as the backbone of each stream given its good performance. In addition, a differentiable edge Sobel operator module (DESOM) is included in the edge stream which can learn the parameters of Sobel operator adaptively and provide more robust edge information that can suppress background noise. Experimental results on three publicly available remote sensing datasets show that our TSTNet achieves superior performance over the state-of-the-art (SOTA) methods.https://www.mdpi.com/2072-4292/14/6/1507remote sensingscene classificationdeep learningswin transformerfeature fusionedge detection
spellingShingle Siyuan Hao
Bin Wu
Kun Zhao
Yuanxin Ye
Wei Wang
Two-Stream Swin Transformer with Differentiable Sobel Operator for Remote Sensing Image Classification
Remote Sensing
remote sensing
scene classification
deep learning
swin transformer
feature fusion
edge detection
title Two-Stream Swin Transformer with Differentiable Sobel Operator for Remote Sensing Image Classification
title_full Two-Stream Swin Transformer with Differentiable Sobel Operator for Remote Sensing Image Classification
title_fullStr Two-Stream Swin Transformer with Differentiable Sobel Operator for Remote Sensing Image Classification
title_full_unstemmed Two-Stream Swin Transformer with Differentiable Sobel Operator for Remote Sensing Image Classification
title_short Two-Stream Swin Transformer with Differentiable Sobel Operator for Remote Sensing Image Classification
title_sort two stream swin transformer with differentiable sobel operator for remote sensing image classification
topic remote sensing
scene classification
deep learning
swin transformer
feature fusion
edge detection
url https://www.mdpi.com/2072-4292/14/6/1507
work_keys_str_mv AT siyuanhao twostreamswintransformerwithdifferentiablesobeloperatorforremotesensingimageclassification
AT binwu twostreamswintransformerwithdifferentiablesobeloperatorforremotesensingimageclassification
AT kunzhao twostreamswintransformerwithdifferentiablesobeloperatorforremotesensingimageclassification
AT yuanxinye twostreamswintransformerwithdifferentiablesobeloperatorforremotesensingimageclassification
AT weiwang twostreamswintransformerwithdifferentiablesobeloperatorforremotesensingimageclassification