Visible-Thermal Image Object Detection via the Combination of Illumination Conditions and Temperature Information
Object detection plays an important role in autonomous driving, disaster rescue, robot navigation, intelligent video surveillance, and many other fields. Nonetheless, visible images are poor under weak illumination conditions, and thermal infrared images are noisy and have low resolution. Consequent...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-09-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/13/18/3656 |
| _version_ | 1797517342690770944 |
|---|---|
| author | Hang Zhou Min Sun Xiang Ren Xiuyuan Wang |
| author_facet | Hang Zhou Min Sun Xiang Ren Xiuyuan Wang |
| author_sort | Hang Zhou |
| collection | DOAJ |
| description | Object detection plays an important role in autonomous driving, disaster rescue, robot navigation, intelligent video surveillance, and many other fields. Nonetheless, visible images are poor under weak illumination conditions, and thermal infrared images are noisy and have low resolution. Consequently, neither of these two data sources yields satisfactory results when used alone. While some scholars have combined visible and thermal images for object detection, most did not consider the illumination conditions and the different contributions of diverse data sources to the results. In addition, few studies have made use of the temperature characteristics of thermal images. Therefore, in the present study, visible and thermal images are utilized as the dataset, and RetinaNet is used as the baseline to fuse features from different data sources for object detection. Moreover, a dynamic weight fusion method, which is based on channel attention according to different illumination conditions, is used in the fusion component, and the channel attention and a priori temperature mask (CAPTM) module is proposed; the CAPTM can be applied to a deep learning network as a priori knowledge and maximizes the advantage of temperature information from thermal images. The main innovations of the present research include the following: (1) the consideration of different illumination conditions and the use of different fusion parameters for different conditions in the feature fusion of visible and thermal images; (2) the dynamic fusion of different data sources in the feature fusion of visible and thermal images; (3) the use of temperature information as a priori knowledge (CAPTM) in feature extraction. To a certain extent, the proposed methods improve the accuracy of object detection at night or under other weak illumination conditions and with a single data source. Compared with the state-of-the-art (SOTA) method, the proposed method is found to achieve superior detection accuracy with an overall mean average precision (mAP) improvement of 0.69%, including an AP improvement of 2.55% for the detection of the Person category. The results demonstrate the effectiveness of the research methods for object detection, especially temperature information-rich object detection. |
| first_indexed | 2024-03-10T07:15:20Z |
| format | Article |
| id | doaj.art-52970a0ca8a944ef9f9ebc7fa4f8a663 |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-10T07:15:20Z |
| publishDate | 2021-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-52970a0ca8a944ef9f9ebc7fa4f8a6632023-11-22T15:06:22ZengMDPI AGRemote Sensing2072-42922021-09-011318365610.3390/rs13183656Visible-Thermal Image Object Detection via the Combination of Illumination Conditions and Temperature InformationHang Zhou0Min Sun1Xiang Ren2Xiuyuan Wang3Institute of Remote Sensing and GIS, Peking University, Beijing 100871, ChinaInstitute of Remote Sensing and GIS, Peking University, Beijing 100871, ChinaInstitute of Remote Sensing and GIS, Peking University, Beijing 100871, ChinaInstitute of Remote Sensing and GIS, Peking University, Beijing 100871, ChinaObject detection plays an important role in autonomous driving, disaster rescue, robot navigation, intelligent video surveillance, and many other fields. Nonetheless, visible images are poor under weak illumination conditions, and thermal infrared images are noisy and have low resolution. Consequently, neither of these two data sources yields satisfactory results when used alone. While some scholars have combined visible and thermal images for object detection, most did not consider the illumination conditions and the different contributions of diverse data sources to the results. In addition, few studies have made use of the temperature characteristics of thermal images. Therefore, in the present study, visible and thermal images are utilized as the dataset, and RetinaNet is used as the baseline to fuse features from different data sources for object detection. Moreover, a dynamic weight fusion method, which is based on channel attention according to different illumination conditions, is used in the fusion component, and the channel attention and a priori temperature mask (CAPTM) module is proposed; the CAPTM can be applied to a deep learning network as a priori knowledge and maximizes the advantage of temperature information from thermal images. The main innovations of the present research include the following: (1) the consideration of different illumination conditions and the use of different fusion parameters for different conditions in the feature fusion of visible and thermal images; (2) the dynamic fusion of different data sources in the feature fusion of visible and thermal images; (3) the use of temperature information as a priori knowledge (CAPTM) in feature extraction. To a certain extent, the proposed methods improve the accuracy of object detection at night or under other weak illumination conditions and with a single data source. Compared with the state-of-the-art (SOTA) method, the proposed method is found to achieve superior detection accuracy with an overall mean average precision (mAP) improvement of 0.69%, including an AP improvement of 2.55% for the detection of the Person category. The results demonstrate the effectiveness of the research methods for object detection, especially temperature information-rich object detection.https://www.mdpi.com/2072-4292/13/18/3656object detectionmulti-spectral fusionvisible and thermal imagesRetinaNetillumination conditionsdynamic weight fusion |
| spellingShingle | Hang Zhou Min Sun Xiang Ren Xiuyuan Wang Visible-Thermal Image Object Detection via the Combination of Illumination Conditions and Temperature Information Remote Sensing object detection multi-spectral fusion visible and thermal images RetinaNet illumination conditions dynamic weight fusion |
| title | Visible-Thermal Image Object Detection via the Combination of Illumination Conditions and Temperature Information |
| title_full | Visible-Thermal Image Object Detection via the Combination of Illumination Conditions and Temperature Information |
| title_fullStr | Visible-Thermal Image Object Detection via the Combination of Illumination Conditions and Temperature Information |
| title_full_unstemmed | Visible-Thermal Image Object Detection via the Combination of Illumination Conditions and Temperature Information |
| title_short | Visible-Thermal Image Object Detection via the Combination of Illumination Conditions and Temperature Information |
| title_sort | visible thermal image object detection via the combination of illumination conditions and temperature information |
| topic | object detection multi-spectral fusion visible and thermal images RetinaNet illumination conditions dynamic weight fusion |
| url | https://www.mdpi.com/2072-4292/13/18/3656 |
| work_keys_str_mv | AT hangzhou visiblethermalimageobjectdetectionviathecombinationofilluminationconditionsandtemperatureinformation AT minsun visiblethermalimageobjectdetectionviathecombinationofilluminationconditionsandtemperatureinformation AT xiangren visiblethermalimageobjectdetectionviathecombinationofilluminationconditionsandtemperatureinformation AT xiuyuanwang visiblethermalimageobjectdetectionviathecombinationofilluminationconditionsandtemperatureinformation |