RANet: A Reliability-Guided Aggregation Network for Hyperspectral and RGB Fusion Tracking
Object tracking based on RGB images may fail when the color of the tracked object is similar to that of the background. Hyperspectral images with rich spectral features can provide more information for RGB-based trackers. However, there is no fusion tracking algorithm based on hyperspectral and RGB...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-06-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/14/12/2765 |
| _version_ | 1827657019596734464 |
|---|---|
| author | Chunhui Zhao Hongjiao Liu Nan Su Lu Wang Yiming Yan |
| author_facet | Chunhui Zhao Hongjiao Liu Nan Su Lu Wang Yiming Yan |
| author_sort | Chunhui Zhao |
| collection | DOAJ |
| description | Object tracking based on RGB images may fail when the color of the tracked object is similar to that of the background. Hyperspectral images with rich spectral features can provide more information for RGB-based trackers. However, there is no fusion tracking algorithm based on hyperspectral and RGB images. In this paper, we propose a reliability-guided aggregation network (RANet) for hyperspectral and RGB tracking, which guides the combination of hyperspectral information and RGB information through modality reliability to improve tracking performance. Specifically, a dual branch based on the Transformer Tracking (TransT) structure is constructed to obtain the information of hyperspectral and RGB modalities. Then, a classification response aggregation module is designed to combine the different modality information by fusing the response predicted through the classification head. Finally, the reliability of different modalities is also considered in the aggregation module to guide the aggregation of the different modality information. Massive experimental results on the public dataset composed of hyperspectral and RGB image sequences show that the performance of the tracker based on our fusion method is better than that of the corresponding single-modality tracker, which fully proves the effectiveness of the fusion method. Among them, the RANet tracker based on the TransT tracker achieves the best performance accuracy of 0.709, indicating the effectiveness and superiority of the RANet tracker. |
| first_indexed | 2024-03-09T22:37:26Z |
| format | Article |
| id | doaj.art-1f8f029cf93b48a6a5713db493ddb277 |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-09T22:37:26Z |
| publishDate | 2022-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-1f8f029cf93b48a6a5713db493ddb2772023-11-23T18:46:27ZengMDPI AGRemote Sensing2072-42922022-06-011412276510.3390/rs14122765RANet: A Reliability-Guided Aggregation Network for Hyperspectral and RGB Fusion TrackingChunhui Zhao0Hongjiao Liu1Nan Su2Lu Wang3Yiming Yan4College of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, ChinaCollege of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, ChinaCollege of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, ChinaCollege of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, ChinaCollege of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, ChinaObject tracking based on RGB images may fail when the color of the tracked object is similar to that of the background. Hyperspectral images with rich spectral features can provide more information for RGB-based trackers. However, there is no fusion tracking algorithm based on hyperspectral and RGB images. In this paper, we propose a reliability-guided aggregation network (RANet) for hyperspectral and RGB tracking, which guides the combination of hyperspectral information and RGB information through modality reliability to improve tracking performance. Specifically, a dual branch based on the Transformer Tracking (TransT) structure is constructed to obtain the information of hyperspectral and RGB modalities. Then, a classification response aggregation module is designed to combine the different modality information by fusing the response predicted through the classification head. Finally, the reliability of different modalities is also considered in the aggregation module to guide the aggregation of the different modality information. Massive experimental results on the public dataset composed of hyperspectral and RGB image sequences show that the performance of the tracker based on our fusion method is better than that of the corresponding single-modality tracker, which fully proves the effectiveness of the fusion method. Among them, the RANet tracker based on the TransT tracker achieves the best performance accuracy of 0.709, indicating the effectiveness and superiority of the RANet tracker.https://www.mdpi.com/2072-4292/14/12/2765fusion trackinghyperspectral imagetransformerdeep learning |
| spellingShingle | Chunhui Zhao Hongjiao Liu Nan Su Lu Wang Yiming Yan RANet: A Reliability-Guided Aggregation Network for Hyperspectral and RGB Fusion Tracking Remote Sensing fusion tracking hyperspectral image transformer deep learning |
| title | RANet: A Reliability-Guided Aggregation Network for Hyperspectral and RGB Fusion Tracking |
| title_full | RANet: A Reliability-Guided Aggregation Network for Hyperspectral and RGB Fusion Tracking |
| title_fullStr | RANet: A Reliability-Guided Aggregation Network for Hyperspectral and RGB Fusion Tracking |
| title_full_unstemmed | RANet: A Reliability-Guided Aggregation Network for Hyperspectral and RGB Fusion Tracking |
| title_short | RANet: A Reliability-Guided Aggregation Network for Hyperspectral and RGB Fusion Tracking |
| title_sort | ranet a reliability guided aggregation network for hyperspectral and rgb fusion tracking |
| topic | fusion tracking hyperspectral image transformer deep learning |
| url | https://www.mdpi.com/2072-4292/14/12/2765 |
| work_keys_str_mv | AT chunhuizhao ranetareliabilityguidedaggregationnetworkforhyperspectralandrgbfusiontracking AT hongjiaoliu ranetareliabilityguidedaggregationnetworkforhyperspectralandrgbfusiontracking AT nansu ranetareliabilityguidedaggregationnetworkforhyperspectralandrgbfusiontracking AT luwang ranetareliabilityguidedaggregationnetworkforhyperspectralandrgbfusiontracking AT yimingyan ranetareliabilityguidedaggregationnetworkforhyperspectralandrgbfusiontracking |