Real-Time Infrared Small Target Detection With Nonlocal Spatial-Temporal Feature Fusion
Infrared small target detection is a challenging task in which many researchers have made lots of achievements. While the performance of single-frame (SF) detection is still limited due to the lack of usage of multiframe (MF) continuous information, many spatial-temporal detection methods have been...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10480567/ |
| _version_ | 1797201323614011392 |
|---|---|
| author | Hai Xu Sheng Zhong Tianxu Zhang Xu Zou |
| author_facet | Hai Xu Sheng Zhong Tianxu Zhang Xu Zou |
| author_sort | Hai Xu |
| collection | DOAJ |
| description | Infrared small target detection is a challenging task in which many researchers have made lots of achievements. While the performance of single-frame (SF) detection is still limited due to the lack of usage of multiframe (MF) continuous information, many spatial-temporal detection methods have been developed. However, most algorithms need to register the image or feed a set of images as the input. Inputting a batch of group images usually leads to large computations, which heavily affects their real-time processing capability in resource-limited machines. To tackle the problem, we propose a nonlocal multiframe network (NLMF-Net) with only a few additional computations (no more than 0.01 GFLOPs) compared to the SF baseline while achieving significant performance improvements. The NLMF-Net correlates features from grid cells with high confidence between current and past frames. While most background grid cells are removed after the SF processing, the MF feature fusion only focuses on a few potential target grid cells, resulting in high computation efficiency. The proposed vector length similarity module enlarges the difference between different grid cells and the non max similarity suppression further suppresses the backgrounds during the fusion, promoting the MF performance. The NLMF-Net can be readily deployed on Jetson Nano at a speed of 20 FPS for 288 × 384 image processing or Mi Pad 2 with a speed over 35 FPS for 128 × 128 part image processing. Extensive experiments show that our proposed method achieves state-of-the-art performance on three datasets while maintaining high efficiency in a real-time processing manner. |
| first_indexed | 2024-04-24T07:45:43Z |
| format | Article |
| id | doaj.art-66d66ace9ba648c892e1a97bdc685095 |
| institution | Directory Open Access Journal |
| issn | 1939-1404 2151-1535 |
| language | English |
| last_indexed | 2024-04-24T07:45:43Z |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| spelling | doaj.art-66d66ace9ba648c892e1a97bdc6850952024-04-18T23:00:17ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing1939-14042151-15352024-01-01177888790210.1109/JSTARS.2024.338238910480567Real-Time Infrared Small Target Detection With Nonlocal Spatial-Temporal Feature FusionHai Xu0https://orcid.org/0000-0002-3674-180XSheng Zhong1https://orcid.org/0000-0003-2865-8202Tianxu Zhang2https://orcid.org/0009-0003-1071-4146Xu Zou3https://orcid.org/0000-0002-0251-7404National Key Laboratory of Multispectral Information Intelligent Processing Technology/ School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, ChinaNational Key Laboratory of Multispectral Information Intelligent Processing Technology/ School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, ChinaNational Key Laboratory of Multispectral Information Intelligent Processing Technology/ School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, ChinaNational Key Laboratory of Multispectral Information Intelligent Processing Technology/ School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, ChinaInfrared small target detection is a challenging task in which many researchers have made lots of achievements. While the performance of single-frame (SF) detection is still limited due to the lack of usage of multiframe (MF) continuous information, many spatial-temporal detection methods have been developed. However, most algorithms need to register the image or feed a set of images as the input. Inputting a batch of group images usually leads to large computations, which heavily affects their real-time processing capability in resource-limited machines. To tackle the problem, we propose a nonlocal multiframe network (NLMF-Net) with only a few additional computations (no more than 0.01 GFLOPs) compared to the SF baseline while achieving significant performance improvements. The NLMF-Net correlates features from grid cells with high confidence between current and past frames. While most background grid cells are removed after the SF processing, the MF feature fusion only focuses on a few potential target grid cells, resulting in high computation efficiency. The proposed vector length similarity module enlarges the difference between different grid cells and the non max similarity suppression further suppresses the backgrounds during the fusion, promoting the MF performance. The NLMF-Net can be readily deployed on Jetson Nano at a speed of 20 FPS for 288 × 384 image processing or Mi Pad 2 with a speed over 35 FPS for 128 × 128 part image processing. Extensive experiments show that our proposed method achieves state-of-the-art performance on three datasets while maintaining high efficiency in a real-time processing manner.https://ieeexplore.ieee.org/document/10480567/Convolution neural network (CNN)infrared small target (IST) detectionreal-time processingspatial-temporal (ST) fusion |
| spellingShingle | Hai Xu Sheng Zhong Tianxu Zhang Xu Zou Real-Time Infrared Small Target Detection With Nonlocal Spatial-Temporal Feature Fusion IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Convolution neural network (CNN) infrared small target (IST) detection real-time processing spatial-temporal (ST) fusion |
| title | Real-Time Infrared Small Target Detection With Nonlocal Spatial-Temporal Feature Fusion |
| title_full | Real-Time Infrared Small Target Detection With Nonlocal Spatial-Temporal Feature Fusion |
| title_fullStr | Real-Time Infrared Small Target Detection With Nonlocal Spatial-Temporal Feature Fusion |
| title_full_unstemmed | Real-Time Infrared Small Target Detection With Nonlocal Spatial-Temporal Feature Fusion |
| title_short | Real-Time Infrared Small Target Detection With Nonlocal Spatial-Temporal Feature Fusion |
| title_sort | real time infrared small target detection with nonlocal spatial temporal feature fusion |
| topic | Convolution neural network (CNN) infrared small target (IST) detection real-time processing spatial-temporal (ST) fusion |
| url | https://ieeexplore.ieee.org/document/10480567/ |
| work_keys_str_mv | AT haixu realtimeinfraredsmalltargetdetectionwithnonlocalspatialtemporalfeaturefusion AT shengzhong realtimeinfraredsmalltargetdetectionwithnonlocalspatialtemporalfeaturefusion AT tianxuzhang realtimeinfraredsmalltargetdetectionwithnonlocalspatialtemporalfeaturefusion AT xuzou realtimeinfraredsmalltargetdetectionwithnonlocalspatialtemporalfeaturefusion |