A Deep Learning-Enhanced Multi-Modal Sensing Platform for Robust Human Object Detection and Tracking in Challenging Environments
In modern security situations, tracking multiple human objects in real-time within challenging urban environments is a critical capability for enhancing situational awareness, minimizing response time, and increasing overall operational effectiveness. Tracking multiple entities enables informed deci...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-08-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/12/16/3423 |
| _version_ | 1797584923170701312 |
|---|---|
| author | Peng Cheng Zinan Xiong Yajie Bao Ping Zhuang Yunqi Zhang Erik Blasch Genshe Chen |
| author_facet | Peng Cheng Zinan Xiong Yajie Bao Ping Zhuang Yunqi Zhang Erik Blasch Genshe Chen |
| author_sort | Peng Cheng |
| collection | DOAJ |
| description | In modern security situations, tracking multiple human objects in real-time within challenging urban environments is a critical capability for enhancing situational awareness, minimizing response time, and increasing overall operational effectiveness. Tracking multiple entities enables informed decision-making, risk mitigation, and the safeguarding of civil-military operations to ensure safety and mission success. This paper presents a multi-modal electro-optical/infrared (EO/IR) and radio frequency (RF) fused sensing (MEIRFS) platform for real-time human object detection, recognition, classification, and tracking in challenging environments. By utilizing different sensors in a complementary manner, the robustness of the sensing system is enhanced, enabling reliable detection and recognition results across various situations. Specifically designed radar tags and thermal tags can be used to discriminate between friendly and non-friendly objects. The system incorporates deep learning-based image fusion and human object recognition and tracking (HORT) algorithms to ensure accurate situation assessment. After integrating into an all-terrain robot, multiple ground tests were conducted to verify the consistency of the HORT in various environments. The MEIRFS sensor system has been designed to meet the Size, Weight, Power, and Cost (SWaP-C) requirements for installation on autonomous ground and aerial vehicles. |
| first_indexed | 2024-03-10T23:59:36Z |
| format | Article |
| id | doaj.art-cbdaf629ab5049c19f719a0e6080fdc9 |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-10T23:59:36Z |
| publishDate | 2023-08-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-cbdaf629ab5049c19f719a0e6080fdc92023-11-19T00:53:24ZengMDPI AGElectronics2079-92922023-08-011216342310.3390/electronics12163423A Deep Learning-Enhanced Multi-Modal Sensing Platform for Robust Human Object Detection and Tracking in Challenging EnvironmentsPeng Cheng0Zinan Xiong1Yajie Bao2Ping Zhuang3Yunqi Zhang4Erik Blasch5Genshe Chen6Intelligent Fusion Technology, Inc., Germantown, MD 20874, USAIntelligent Fusion Technology, Inc., Germantown, MD 20874, USAIntelligent Fusion Technology, Inc., Germantown, MD 20874, USAIntelligent Fusion Technology, Inc., Germantown, MD 20874, USAIntelligent Fusion Technology, Inc., Germantown, MD 20874, USAMOVEJ Analytics, Dayton, OH 45324, USAIntelligent Fusion Technology, Inc., Germantown, MD 20874, USAIn modern security situations, tracking multiple human objects in real-time within challenging urban environments is a critical capability for enhancing situational awareness, minimizing response time, and increasing overall operational effectiveness. Tracking multiple entities enables informed decision-making, risk mitigation, and the safeguarding of civil-military operations to ensure safety and mission success. This paper presents a multi-modal electro-optical/infrared (EO/IR) and radio frequency (RF) fused sensing (MEIRFS) platform for real-time human object detection, recognition, classification, and tracking in challenging environments. By utilizing different sensors in a complementary manner, the robustness of the sensing system is enhanced, enabling reliable detection and recognition results across various situations. Specifically designed radar tags and thermal tags can be used to discriminate between friendly and non-friendly objects. The system incorporates deep learning-based image fusion and human object recognition and tracking (HORT) algorithms to ensure accurate situation assessment. After integrating into an all-terrain robot, multiple ground tests were conducted to verify the consistency of the HORT in various environments. The MEIRFS sensor system has been designed to meet the Size, Weight, Power, and Cost (SWaP-C) requirements for installation on autonomous ground and aerial vehicles.https://www.mdpi.com/2079-9292/12/16/3423human object recognition and trackingmulti-modal sensingEO/IRradarmobile platformdeep learning |
| spellingShingle | Peng Cheng Zinan Xiong Yajie Bao Ping Zhuang Yunqi Zhang Erik Blasch Genshe Chen A Deep Learning-Enhanced Multi-Modal Sensing Platform for Robust Human Object Detection and Tracking in Challenging Environments Electronics human object recognition and tracking multi-modal sensing EO/IR radar mobile platform deep learning |
| title | A Deep Learning-Enhanced Multi-Modal Sensing Platform for Robust Human Object Detection and Tracking in Challenging Environments |
| title_full | A Deep Learning-Enhanced Multi-Modal Sensing Platform for Robust Human Object Detection and Tracking in Challenging Environments |
| title_fullStr | A Deep Learning-Enhanced Multi-Modal Sensing Platform for Robust Human Object Detection and Tracking in Challenging Environments |
| title_full_unstemmed | A Deep Learning-Enhanced Multi-Modal Sensing Platform for Robust Human Object Detection and Tracking in Challenging Environments |
| title_short | A Deep Learning-Enhanced Multi-Modal Sensing Platform for Robust Human Object Detection and Tracking in Challenging Environments |
| title_sort | deep learning enhanced multi modal sensing platform for robust human object detection and tracking in challenging environments |
| topic | human object recognition and tracking multi-modal sensing EO/IR radar mobile platform deep learning |
| url | https://www.mdpi.com/2079-9292/12/16/3423 |
| work_keys_str_mv | AT pengcheng adeeplearningenhancedmultimodalsensingplatformforrobusthumanobjectdetectionandtrackinginchallengingenvironments AT zinanxiong adeeplearningenhancedmultimodalsensingplatformforrobusthumanobjectdetectionandtrackinginchallengingenvironments AT yajiebao adeeplearningenhancedmultimodalsensingplatformforrobusthumanobjectdetectionandtrackinginchallengingenvironments AT pingzhuang adeeplearningenhancedmultimodalsensingplatformforrobusthumanobjectdetectionandtrackinginchallengingenvironments AT yunqizhang adeeplearningenhancedmultimodalsensingplatformforrobusthumanobjectdetectionandtrackinginchallengingenvironments AT erikblasch adeeplearningenhancedmultimodalsensingplatformforrobusthumanobjectdetectionandtrackinginchallengingenvironments AT genshechen adeeplearningenhancedmultimodalsensingplatformforrobusthumanobjectdetectionandtrackinginchallengingenvironments AT pengcheng deeplearningenhancedmultimodalsensingplatformforrobusthumanobjectdetectionandtrackinginchallengingenvironments AT zinanxiong deeplearningenhancedmultimodalsensingplatformforrobusthumanobjectdetectionandtrackinginchallengingenvironments AT yajiebao deeplearningenhancedmultimodalsensingplatformforrobusthumanobjectdetectionandtrackinginchallengingenvironments AT pingzhuang deeplearningenhancedmultimodalsensingplatformforrobusthumanobjectdetectionandtrackinginchallengingenvironments AT yunqizhang deeplearningenhancedmultimodalsensingplatformforrobusthumanobjectdetectionandtrackinginchallengingenvironments AT erikblasch deeplearningenhancedmultimodalsensingplatformforrobusthumanobjectdetectionandtrackinginchallengingenvironments AT genshechen deeplearningenhancedmultimodalsensingplatformforrobusthumanobjectdetectionandtrackinginchallengingenvironments |