Joint Classification of Hyperspectral and LiDAR Data Using Binary-Tree Transformer Network
The joint utilization of multi-source data is of great significance in geospatial observation applications, such as urban planning, disaster assessment, and military applications. However, this approach is confronted with challenges including inconsistent data structures, irrelevant physical propert...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-05-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/15/11/2706 |
| _version_ | 1827739281801609216 |
|---|---|
| author | Huacui Song Yuanwei Yang Xianjun Gao Maqun Zhang Shaohua Li Bo Liu Yanjun Wang Yuan Kou |
| author_facet | Huacui Song Yuanwei Yang Xianjun Gao Maqun Zhang Shaohua Li Bo Liu Yanjun Wang Yuan Kou |
| author_sort | Huacui Song |
| collection | DOAJ |
| description | The joint utilization of multi-source data is of great significance in geospatial observation applications, such as urban planning, disaster assessment, and military applications. However, this approach is confronted with challenges including inconsistent data structures, irrelevant physical properties, scarce training data, insufficient utilization of information and an imperfect feature fusion method. Therefore, this paper proposes a novel binary-tree Transformer network (BTRF-Net), which is used to fuse heterogeneous information and utilize complementarity among multi-source remote sensing data to enhance the joint classification performance of hyperspectral image (HSI) and light detection and ranging (LiDAR) data. Firstly, a hyperspectral network (HSI-Net) is employed to extract spectral and spatial features of hyperspectral images, while the elevation information of LiDAR data is extracted using the LiDAR network (LiDAR-Net). Secondly, a multi-source transformer complementor (MSTC) is designed that utilizes the complementarity and cooperation among multi-modal feature information in remote sensing images to better capture their correlation. The multi-head complementarity attention mechanism (MHCA) within this complementor can effectively capture global features and local texture information of images, hence achieving full feature fusion. Then, to fully obtain feature information of multi-source remote sensing images, this paper designs a complete binary tree structure, binary feature search tree (BFST), which fuses multi-modal features at different network levels to obtain multiple image features with stronger representation abilities, effectively enhancing the stability and robustness of the network. Finally, several groups of experiments are designed to compare and analyze the proposed BTRF-Net with traditional methods and several advanced deep learning networks using two datasets: Houston and Trento. The results show that the proposed network outperforms other state-of-the-art methods even with small training samples. |
| first_indexed | 2024-03-11T02:58:34Z |
| format | Article |
| id | doaj.art-6a739566bd4f46cbb78f8198ed90ff1a |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-11T02:58:34Z |
| publishDate | 2023-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-6a739566bd4f46cbb78f8198ed90ff1a2023-11-18T08:27:35ZengMDPI AGRemote Sensing2072-42922023-05-011511270610.3390/rs15112706Joint Classification of Hyperspectral and LiDAR Data Using Binary-Tree Transformer NetworkHuacui Song0Yuanwei Yang1Xianjun Gao2Maqun Zhang3Shaohua Li4Bo Liu5Yanjun Wang6Yuan Kou7School of Geosciences, Yangtze University, Wuhan 430100, ChinaSchool of Geosciences, Yangtze University, Wuhan 430100, ChinaSchool of Geosciences, Yangtze University, Wuhan 430100, ChinaCollege of Information Science and Engineering, Ocean University of China, Qingdao 266100, ChinaSchool of Geosciences, Yangtze University, Wuhan 430100, ChinaKey Laboratory of Mine Environmental Monitoring and Improving around Poyang Lake of Ministry of Natural Resources, East China University of Technology, Nanchang 330013, ChinaHunan Provincial Key Laboratory of Geo-Information Engineering in Surveying, Mapping and Remote Sensing, Hunan University of Science and Technology, Xiangtan 411201, ChinaThe First Surveying and Mapping Institute of Hunan Province, Changsha 410000, ChinaThe joint utilization of multi-source data is of great significance in geospatial observation applications, such as urban planning, disaster assessment, and military applications. However, this approach is confronted with challenges including inconsistent data structures, irrelevant physical properties, scarce training data, insufficient utilization of information and an imperfect feature fusion method. Therefore, this paper proposes a novel binary-tree Transformer network (BTRF-Net), which is used to fuse heterogeneous information and utilize complementarity among multi-source remote sensing data to enhance the joint classification performance of hyperspectral image (HSI) and light detection and ranging (LiDAR) data. Firstly, a hyperspectral network (HSI-Net) is employed to extract spectral and spatial features of hyperspectral images, while the elevation information of LiDAR data is extracted using the LiDAR network (LiDAR-Net). Secondly, a multi-source transformer complementor (MSTC) is designed that utilizes the complementarity and cooperation among multi-modal feature information in remote sensing images to better capture their correlation. The multi-head complementarity attention mechanism (MHCA) within this complementor can effectively capture global features and local texture information of images, hence achieving full feature fusion. Then, to fully obtain feature information of multi-source remote sensing images, this paper designs a complete binary tree structure, binary feature search tree (BFST), which fuses multi-modal features at different network levels to obtain multiple image features with stronger representation abilities, effectively enhancing the stability and robustness of the network. Finally, several groups of experiments are designed to compare and analyze the proposed BTRF-Net with traditional methods and several advanced deep learning networks using two datasets: Houston and Trento. The results show that the proposed network outperforms other state-of-the-art methods even with small training samples.https://www.mdpi.com/2072-4292/15/11/2706HSILiDARtransformercomplementortree |
| spellingShingle | Huacui Song Yuanwei Yang Xianjun Gao Maqun Zhang Shaohua Li Bo Liu Yanjun Wang Yuan Kou Joint Classification of Hyperspectral and LiDAR Data Using Binary-Tree Transformer Network Remote Sensing HSI LiDAR transformer complementor tree |
| title | Joint Classification of Hyperspectral and LiDAR Data Using Binary-Tree Transformer Network |
| title_full | Joint Classification of Hyperspectral and LiDAR Data Using Binary-Tree Transformer Network |
| title_fullStr | Joint Classification of Hyperspectral and LiDAR Data Using Binary-Tree Transformer Network |
| title_full_unstemmed | Joint Classification of Hyperspectral and LiDAR Data Using Binary-Tree Transformer Network |
| title_short | Joint Classification of Hyperspectral and LiDAR Data Using Binary-Tree Transformer Network |
| title_sort | joint classification of hyperspectral and lidar data using binary tree transformer network |
| topic | HSI LiDAR transformer complementor tree |
| url | https://www.mdpi.com/2072-4292/15/11/2706 |
| work_keys_str_mv | AT huacuisong jointclassificationofhyperspectralandlidardatausingbinarytreetransformernetwork AT yuanweiyang jointclassificationofhyperspectralandlidardatausingbinarytreetransformernetwork AT xianjungao jointclassificationofhyperspectralandlidardatausingbinarytreetransformernetwork AT maqunzhang jointclassificationofhyperspectralandlidardatausingbinarytreetransformernetwork AT shaohuali jointclassificationofhyperspectralandlidardatausingbinarytreetransformernetwork AT boliu jointclassificationofhyperspectralandlidardatausingbinarytreetransformernetwork AT yanjunwang jointclassificationofhyperspectralandlidardatausingbinarytreetransformernetwork AT yuankou jointclassificationofhyperspectralandlidardatausingbinarytreetransformernetwork |