Making Sense of Neuromorphic Event Data for Human Action Recognition
Neuromorphic vision sensors provide low power sensing and capture salient spatial-temporal events. The majority of the existing neuromorphic sensing work focus on object detection. However, since they only record the events, they provide an efficient signal domain for privacy aware surveillance task...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9446050/ |
| _version_ | 1818899919088386048 |
|---|---|
| author | Salah Al-Obaidi Hiba Al-Khafaji Charith Abhayaratne |
| author_facet | Salah Al-Obaidi Hiba Al-Khafaji Charith Abhayaratne |
| author_sort | Salah Al-Obaidi |
| collection | DOAJ |
| description | Neuromorphic vision sensors provide low power sensing and capture salient spatial-temporal events. The majority of the existing neuromorphic sensing work focus on object detection. However, since they only record the events, they provide an efficient signal domain for privacy aware surveillance tasks. This paper explores how the neuromorphic vision sensor data streams can be analysed for human action recognition, which is a challenging application. The proposed method is based on handcrafted features. It consists of a pre-processing step for removing the noisy events followed by the extraction of handcrafted local and global feature vectors corresponding to the underlying human action. The local features are extracted considering a set of high-order descriptive statistics from the spatio-temporal events in a time window slice, while the global features are extracted by considering the frequencies of occurrences of the temporal event sequences. Then, low complexity classifiers, such as, support vector machines (SVM) and K-Nearest Neighbours (KNNs), are trained using these feature vectors. The proposed method evaluation uses three groups of datasets: Emulator-based, re-recording-based and native NVS-based. The proposed method has outperformed the existing methods in terms of human action recognition accuracy rates by 0.54%, 19.3%, and 25.61% for E-KTH, E-UCF11 and E-HMDB51 datasets, respectively. This paper also reports results for three further datasets: E-UCF50, R-UCF50, and N-Actions, which are reported for the first time for human action recognition on neuromorphic vision sensor domain. |
| first_indexed | 2024-12-19T19:55:36Z |
| format | Article |
| id | doaj.art-26a3f2dc17dd409e9550ff7e5d065665 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-19T19:55:36Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-26a3f2dc17dd409e9550ff7e5d0656652022-12-21T20:07:48ZengIEEEIEEE Access2169-35362021-01-019826868270010.1109/ACCESS.2021.30857089446050Making Sense of Neuromorphic Event Data for Human Action RecognitionSalah Al-Obaidi0https://orcid.org/0000-0003-4521-7193Hiba Al-Khafaji1https://orcid.org/0000-0002-4977-9284Charith Abhayaratne2https://orcid.org/0000-0002-2799-7395Department of Electronic and Electrical Engineering, The University of Sheffield, Sheffield, U.K.Department of Electronic and Electrical Engineering, The University of Sheffield, Sheffield, U.K.Department of Electronic and Electrical Engineering, The University of Sheffield, Sheffield, U.K.Neuromorphic vision sensors provide low power sensing and capture salient spatial-temporal events. The majority of the existing neuromorphic sensing work focus on object detection. However, since they only record the events, they provide an efficient signal domain for privacy aware surveillance tasks. This paper explores how the neuromorphic vision sensor data streams can be analysed for human action recognition, which is a challenging application. The proposed method is based on handcrafted features. It consists of a pre-processing step for removing the noisy events followed by the extraction of handcrafted local and global feature vectors corresponding to the underlying human action. The local features are extracted considering a set of high-order descriptive statistics from the spatio-temporal events in a time window slice, while the global features are extracted by considering the frequencies of occurrences of the temporal event sequences. Then, low complexity classifiers, such as, support vector machines (SVM) and K-Nearest Neighbours (KNNs), are trained using these feature vectors. The proposed method evaluation uses three groups of datasets: Emulator-based, re-recording-based and native NVS-based. The proposed method has outperformed the existing methods in terms of human action recognition accuracy rates by 0.54%, 19.3%, and 25.61% for E-KTH, E-UCF11 and E-HMDB51 datasets, respectively. This paper also reports results for three further datasets: E-UCF50, R-UCF50, and N-Actions, which are reported for the first time for human action recognition on neuromorphic vision sensor domain.https://ieeexplore.ieee.org/document/9446050/Neuromorphic vision sensing (NVS)event camerasdynamic vision sensing (DVS)human action recognition (HAR)local featuresglobal features |
| spellingShingle | Salah Al-Obaidi Hiba Al-Khafaji Charith Abhayaratne Making Sense of Neuromorphic Event Data for Human Action Recognition IEEE Access Neuromorphic vision sensing (NVS) event cameras dynamic vision sensing (DVS) human action recognition (HAR) local features global features |
| title | Making Sense of Neuromorphic Event Data for Human Action Recognition |
| title_full | Making Sense of Neuromorphic Event Data for Human Action Recognition |
| title_fullStr | Making Sense of Neuromorphic Event Data for Human Action Recognition |
| title_full_unstemmed | Making Sense of Neuromorphic Event Data for Human Action Recognition |
| title_short | Making Sense of Neuromorphic Event Data for Human Action Recognition |
| title_sort | making sense of neuromorphic event data for human action recognition |
| topic | Neuromorphic vision sensing (NVS) event cameras dynamic vision sensing (DVS) human action recognition (HAR) local features global features |
| url | https://ieeexplore.ieee.org/document/9446050/ |
| work_keys_str_mv | AT salahalobaidi makingsenseofneuromorphiceventdataforhumanactionrecognition AT hibaalkhafaji makingsenseofneuromorphiceventdataforhumanactionrecognition AT charithabhayaratne makingsenseofneuromorphiceventdataforhumanactionrecognition |