Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural Networks
Sign language recognition has been utilized in human–machine interactions, improving the lives of people with speech impairments or who rely on nonverbal instructions. Thanks to its higher temporal resolution, less visual redundancy information and lower energy consumption, the use of an event camer...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-02-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/12/4/786 |
| _version_ | 1797621339506343936 |
|---|---|
| author | Xuena Chen Li Su Jinxiu Zhao Keni Qiu Na Jiang Guang Zhai |
| author_facet | Xuena Chen Li Su Jinxiu Zhao Keni Qiu Na Jiang Guang Zhai |
| author_sort | Xuena Chen |
| collection | DOAJ |
| description | Sign language recognition has been utilized in human–machine interactions, improving the lives of people with speech impairments or who rely on nonverbal instructions. Thanks to its higher temporal resolution, less visual redundancy information and lower energy consumption, the use of an event camera with a new dynamic vision sensor (DVS) shows promise with regard to sign language recognition with robot perception and intelligent control. Although previous work has focused on event camera-based, simple gesture datasets, such as DVS128Gesture, event camera gesture datasets inspired by sign language are critical, which poses a great impediment to the development of event camera-based sign language recognition. An effective method to extract spatio-temporal features from event data is significantly desired. Firstly, the event-based sign language gesture datasets are proposed and the data have two sources: traditional sign language videos to event stream (DVS_Sign_v2e) and DAVIS346 (DVS_Sign). In the present dataset, data are divided into five classification, verbs, quantifiers, position, things and people, adapting to actual scenarios where robots provide instruction or assistance. Sign language classification is demonstrated in spike neuron networks with a spatio-temporal back-propagation training method, leading to the best recognition accuracy of 77%. This work paves the way for the combination of event camera-based sign language gesture recognition and robotic perception for the future intelligent systems. |
| first_indexed | 2024-03-11T08:54:34Z |
| format | Article |
| id | doaj.art-075c36a02f07445aa3ce753346711ef6 |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-11T08:54:34Z |
| publishDate | 2023-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-075c36a02f07445aa3ce753346711ef62023-11-16T20:10:01ZengMDPI AGElectronics2079-92922023-02-0112478610.3390/electronics12040786Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural NetworksXuena Chen0Li Su1Jinxiu Zhao2Keni Qiu3Na Jiang4Guang Zhai5School of Information Engineering, Capital Normal University, Beijing 100048, ChinaSchool of Information Engineering, Capital Normal University, Beijing 100048, ChinaSchool of Information Engineering, Capital Normal University, Beijing 100048, ChinaSchool of Information Engineering, Capital Normal University, Beijing 100048, ChinaSchool of Information Engineering, Capital Normal University, Beijing 100048, ChinaSchool of Aerospace Engineering, Beijing Institute of Technology, Beijing 100081, ChinaSign language recognition has been utilized in human–machine interactions, improving the lives of people with speech impairments or who rely on nonverbal instructions. Thanks to its higher temporal resolution, less visual redundancy information and lower energy consumption, the use of an event camera with a new dynamic vision sensor (DVS) shows promise with regard to sign language recognition with robot perception and intelligent control. Although previous work has focused on event camera-based, simple gesture datasets, such as DVS128Gesture, event camera gesture datasets inspired by sign language are critical, which poses a great impediment to the development of event camera-based sign language recognition. An effective method to extract spatio-temporal features from event data is significantly desired. Firstly, the event-based sign language gesture datasets are proposed and the data have two sources: traditional sign language videos to event stream (DVS_Sign_v2e) and DAVIS346 (DVS_Sign). In the present dataset, data are divided into five classification, verbs, quantifiers, position, things and people, adapting to actual scenarios where robots provide instruction or assistance. Sign language classification is demonstrated in spike neuron networks with a spatio-temporal back-propagation training method, leading to the best recognition accuracy of 77%. This work paves the way for the combination of event camera-based sign language gesture recognition and robotic perception for the future intelligent systems.https://www.mdpi.com/2079-9292/12/4/786event cameraspiking neural networkDVS-sign languagesign language recognitionintelligent system |
| spellingShingle | Xuena Chen Li Su Jinxiu Zhao Keni Qiu Na Jiang Guang Zhai Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural Networks Electronics event camera spiking neural network DVS-sign language sign language recognition intelligent system |
| title | Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural Networks |
| title_full | Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural Networks |
| title_fullStr | Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural Networks |
| title_full_unstemmed | Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural Networks |
| title_short | Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural Networks |
| title_sort | sign language gesture recognition and classification based on event camera with spiking neural networks |
| topic | event camera spiking neural network DVS-sign language sign language recognition intelligent system |
| url | https://www.mdpi.com/2079-9292/12/4/786 |
| work_keys_str_mv | AT xuenachen signlanguagegesturerecognitionandclassificationbasedoneventcamerawithspikingneuralnetworks AT lisu signlanguagegesturerecognitionandclassificationbasedoneventcamerawithspikingneuralnetworks AT jinxiuzhao signlanguagegesturerecognitionandclassificationbasedoneventcamerawithspikingneuralnetworks AT keniqiu signlanguagegesturerecognitionandclassificationbasedoneventcamerawithspikingneuralnetworks AT najiang signlanguagegesturerecognitionandclassificationbasedoneventcamerawithspikingneuralnetworks AT guangzhai signlanguagegesturerecognitionandclassificationbasedoneventcamerawithspikingneuralnetworks |