SS-TMNet: Spatial–Spectral Transformer Network with Multi-Scale Convolution for Hyperspectral Image Classification
Hyperspectral image (HSI) classification is a significant foundation for remote sensing image analysis, widely used in biology, aerospace, and other applications. Convolution neural networks (CNNs) and attention mechanisms have shown outstanding ability in HSI classification and have been widely stu...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-02-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/15/5/1206 |
| _version_ | 1797614419835879424 |
|---|---|
| author | Xiaohui Huang Yunfei Zhou Xiaofei Yang Xianhong Zhu Ke Wang |
| author_facet | Xiaohui Huang Yunfei Zhou Xiaofei Yang Xianhong Zhu Ke Wang |
| author_sort | Xiaohui Huang |
| collection | DOAJ |
| description | Hyperspectral image (HSI) classification is a significant foundation for remote sensing image analysis, widely used in biology, aerospace, and other applications. Convolution neural networks (CNNs) and attention mechanisms have shown outstanding ability in HSI classification and have been widely studied in recent years. However, the existing CNN-based and attention mechanism-based methods cannot fully use spatial–spectral information, which is not conducive to further improving HSI classification accuracy. This paper proposes a new spatial–spectral Transformer network with multi-scale convolution (SS-TMNet), which can effectively extract local and global spatial–spectral information. SS-TMNet includes two key modules, i.e., multi-scale 3D convolution projection module (MSCP) and spatial–spectral attention module (SSAM). The MSCP uses multi-scale 3D convolutions with different depths to extract the fused spatial–spectral features. The spatial–spectral attention module includes three branches: height spatial attention, width spatial attention, and spectral attention, which can extract the fusion information of spatial and spectral features. The proposed SS-TMNet was tested on three widely used HSI datasets: Pavia University, IndianPines, and Houston2013. The experimental results show that the proposed SS-TMNet is superior to the existing methods. |
| first_indexed | 2024-03-11T07:12:08Z |
| format | Article |
| id | doaj.art-1e601a081cd2490e941aea1a85021057 |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-11T07:12:08Z |
| publishDate | 2023-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-1e601a081cd2490e941aea1a850210572023-11-17T08:29:51ZengMDPI AGRemote Sensing2072-42922023-02-01155120610.3390/rs15051206SS-TMNet: Spatial–Spectral Transformer Network with Multi-Scale Convolution for Hyperspectral Image ClassificationXiaohui Huang0Yunfei Zhou1Xiaofei Yang2Xianhong Zhu3Ke Wang4School of Information Engineering, East China Jiaotong University, Nanchang 330013, ChinaSchool of Information Engineering, East China Jiaotong University, Nanchang 330013, ChinaThe Department of Computer and Information Science, University of Macau, Macau 519000, ChinaSchool of Information Engineering, East China Jiaotong University, Nanchang 330013, ChinaSchool of Computer Science, Shenyang Aerospace University, Shenyang 110136, ChinaHyperspectral image (HSI) classification is a significant foundation for remote sensing image analysis, widely used in biology, aerospace, and other applications. Convolution neural networks (CNNs) and attention mechanisms have shown outstanding ability in HSI classification and have been widely studied in recent years. However, the existing CNN-based and attention mechanism-based methods cannot fully use spatial–spectral information, which is not conducive to further improving HSI classification accuracy. This paper proposes a new spatial–spectral Transformer network with multi-scale convolution (SS-TMNet), which can effectively extract local and global spatial–spectral information. SS-TMNet includes two key modules, i.e., multi-scale 3D convolution projection module (MSCP) and spatial–spectral attention module (SSAM). The MSCP uses multi-scale 3D convolutions with different depths to extract the fused spatial–spectral features. The spatial–spectral attention module includes three branches: height spatial attention, width spatial attention, and spectral attention, which can extract the fusion information of spatial and spectral features. The proposed SS-TMNet was tested on three widely used HSI datasets: Pavia University, IndianPines, and Houston2013. The experimental results show that the proposed SS-TMNet is superior to the existing methods.https://www.mdpi.com/2072-4292/15/5/1206multi-scale 3D convolutionconvolution neural network (CNN)attention mechanismhyperspectral image (HSI) classification |
| spellingShingle | Xiaohui Huang Yunfei Zhou Xiaofei Yang Xianhong Zhu Ke Wang SS-TMNet: Spatial–Spectral Transformer Network with Multi-Scale Convolution for Hyperspectral Image Classification Remote Sensing multi-scale 3D convolution convolution neural network (CNN) attention mechanism hyperspectral image (HSI) classification |
| title | SS-TMNet: Spatial–Spectral Transformer Network with Multi-Scale Convolution for Hyperspectral Image Classification |
| title_full | SS-TMNet: Spatial–Spectral Transformer Network with Multi-Scale Convolution for Hyperspectral Image Classification |
| title_fullStr | SS-TMNet: Spatial–Spectral Transformer Network with Multi-Scale Convolution for Hyperspectral Image Classification |
| title_full_unstemmed | SS-TMNet: Spatial–Spectral Transformer Network with Multi-Scale Convolution for Hyperspectral Image Classification |
| title_short | SS-TMNet: Spatial–Spectral Transformer Network with Multi-Scale Convolution for Hyperspectral Image Classification |
| title_sort | ss tmnet spatial spectral transformer network with multi scale convolution for hyperspectral image classification |
| topic | multi-scale 3D convolution convolution neural network (CNN) attention mechanism hyperspectral image (HSI) classification |
| url | https://www.mdpi.com/2072-4292/15/5/1206 |
| work_keys_str_mv | AT xiaohuihuang sstmnetspatialspectraltransformernetworkwithmultiscaleconvolutionforhyperspectralimageclassification AT yunfeizhou sstmnetspatialspectraltransformernetworkwithmultiscaleconvolutionforhyperspectralimageclassification AT xiaofeiyang sstmnetspatialspectraltransformernetworkwithmultiscaleconvolutionforhyperspectralimageclassification AT xianhongzhu sstmnetspatialspectraltransformernetworkwithmultiscaleconvolutionforhyperspectralimageclassification AT kewang sstmnetspatialspectraltransformernetworkwithmultiscaleconvolutionforhyperspectralimageclassification |