A hybrid neuromorphic object tracking and classification framework for real-time systems
Deep learning inference that needs to largely take place on the "edge" is a highly computational and memory intensive workload, making it intractable for low-power, embedded platforms such as mobile nodes and remote security applications. To address this challenge, this article proposes a...
| Main Authors: | , , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/170575 |
| _version_ | 1811687972437753856 |
|---|---|
| author | Ussa, Andres Rajen, Chockalingam Senthil Pulluri, Tarun Singla, Deepak Acharya, Jyotibdha Chuanrong, Gideon Fu Basu, Arindam Ramesh, Bharath |
| author2 | School of Electrical and Electronic Engineering |
| author_facet | School of Electrical and Electronic Engineering Ussa, Andres Rajen, Chockalingam Senthil Pulluri, Tarun Singla, Deepak Acharya, Jyotibdha Chuanrong, Gideon Fu Basu, Arindam Ramesh, Bharath |
| author_sort | Ussa, Andres |
| collection | NTU |
| description | Deep learning inference that needs to largely take place on the "edge" is a highly computational and memory intensive workload, making it intractable for low-power, embedded platforms such as mobile nodes and remote security applications. To address this challenge, this article proposes a real-time, hybrid neuromorphic framework for object tracking and classification using event-based cameras that possess desirable properties such as low-power consumption (5-14 mW) and high dynamic range (120 dB). Nonetheless, unlike traditional approaches of using event-by-event processing, this work uses a mixed frame and event approach to get energy savings with high performance. Using a frame-based region proposal method based on the density of foreground events, a hardware-friendly object tracking scheme is implemented using the apparent object velocity while tackling occlusion scenarios. The frame-based object track input is converted back to spikes for TrueNorth (TN) classification via the energy-efficient deep network (EEDN) pipeline. Using originally collected datasets, we train the TN model on the hardware track outputs, instead of using ground truth object locations as commonly done, and demonstrate the ability of our system to handle practical surveillance scenarios. As an alternative tracker paradigm, we also propose a continuous-time tracker with C ++ implementation where each event is processed individually, which better exploits the low latency and asynchronous nature of neuromorphic vision sensors. Subsequently, we extensively compare the proposed methodologies to state-of-the-art event-based and frame-based methods for object tracking and classification, and demonstrate the use case of our neuromorphic approach for real-time and embedded applications without sacrificing performance. Finally, we also showcase the efficacy of the proposed neuromorphic system to a standard RGB camera setup when simultaneously evaluated over several hours of traffic recordings. |
| first_indexed | 2024-10-01T05:24:48Z |
| format | Journal Article |
| id | ntu-10356/170575 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T05:24:48Z |
| publishDate | 2023 |
| record_format | dspace |
| spelling | ntu-10356/1705752023-09-19T08:14:08Z A hybrid neuromorphic object tracking and classification framework for real-time systems Ussa, Andres Rajen, Chockalingam Senthil Pulluri, Tarun Singla, Deepak Acharya, Jyotibdha Chuanrong, Gideon Fu Basu, Arindam Ramesh, Bharath School of Electrical and Electronic Engineering Engineering::Electrical and electronic engineering Neuromorphics Proposals Deep learning inference that needs to largely take place on the "edge" is a highly computational and memory intensive workload, making it intractable for low-power, embedded platforms such as mobile nodes and remote security applications. To address this challenge, this article proposes a real-time, hybrid neuromorphic framework for object tracking and classification using event-based cameras that possess desirable properties such as low-power consumption (5-14 mW) and high dynamic range (120 dB). Nonetheless, unlike traditional approaches of using event-by-event processing, this work uses a mixed frame and event approach to get energy savings with high performance. Using a frame-based region proposal method based on the density of foreground events, a hardware-friendly object tracking scheme is implemented using the apparent object velocity while tackling occlusion scenarios. The frame-based object track input is converted back to spikes for TrueNorth (TN) classification via the energy-efficient deep network (EEDN) pipeline. Using originally collected datasets, we train the TN model on the hardware track outputs, instead of using ground truth object locations as commonly done, and demonstrate the ability of our system to handle practical surveillance scenarios. As an alternative tracker paradigm, we also propose a continuous-time tracker with C ++ implementation where each event is processed individually, which better exploits the low latency and asynchronous nature of neuromorphic vision sensors. Subsequently, we extensively compare the proposed methodologies to state-of-the-art event-based and frame-based methods for object tracking and classification, and demonstrate the use case of our neuromorphic approach for real-time and embedded applications without sacrificing performance. Finally, we also showcase the efficacy of the proposed neuromorphic system to a standard RGB camera setup when simultaneously evaluated over several hours of traffic recordings. 2023-09-19T08:14:08Z 2023-09-19T08:14:08Z 2023 Journal Article Ussa, A., Rajen, C. S., Pulluri, T., Singla, D., Acharya, J., Chuanrong, G. F., Basu, A. & Ramesh, B. (2023). A hybrid neuromorphic object tracking and classification framework for real-time systems. IEEE Transactions On Neural Networks and Learning Systems. https://dx.doi.org/10.1109/TNNLS.2023.3243679 2162-237X https://hdl.handle.net/10356/170575 10.1109/TNNLS.2023.3243679 37027553 2-s2.0-85149413489 en IEEE Transactions on Neural Networks and Learning Systems © 2023 IEEE. All rights reserved. |
| spellingShingle | Engineering::Electrical and electronic engineering Neuromorphics Proposals Ussa, Andres Rajen, Chockalingam Senthil Pulluri, Tarun Singla, Deepak Acharya, Jyotibdha Chuanrong, Gideon Fu Basu, Arindam Ramesh, Bharath A hybrid neuromorphic object tracking and classification framework for real-time systems |
| title | A hybrid neuromorphic object tracking and classification framework for real-time systems |
| title_full | A hybrid neuromorphic object tracking and classification framework for real-time systems |
| title_fullStr | A hybrid neuromorphic object tracking and classification framework for real-time systems |
| title_full_unstemmed | A hybrid neuromorphic object tracking and classification framework for real-time systems |
| title_short | A hybrid neuromorphic object tracking and classification framework for real-time systems |
| title_sort | hybrid neuromorphic object tracking and classification framework for real time systems |
| topic | Engineering::Electrical and electronic engineering Neuromorphics Proposals |
| url | https://hdl.handle.net/10356/170575 |
| work_keys_str_mv | AT ussaandres ahybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT rajenchockalingamsenthil ahybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT pulluritarun ahybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT singladeepak ahybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT acharyajyotibdha ahybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT chuanronggideonfu ahybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT basuarindam ahybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT rameshbharath ahybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT ussaandres hybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT rajenchockalingamsenthil hybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT pulluritarun hybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT singladeepak hybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT acharyajyotibdha hybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT chuanronggideonfu hybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT basuarindam hybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems AT rameshbharath hybridneuromorphicobjecttrackingandclassificationframeworkforrealtimesystems |