KFSENet: A Key Frame-Based Skeleton Feature Estimation and Action Recognition Network for Improved Robot Vision with Face and Emotion Recognition
In this paper, we propose an integrated approach to robot vision: a key frame-based skeleton feature estimation and action recognition network (KFSENet) that incorporates action recognition with face and emotion recognition to enable social robots to engage in more personal interactions. Instead of...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-05-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/12/11/5455 |
| _version_ | 1797494221322584064 |
|---|---|
| author | Dinh-Son Le Hai-Hong Phan Ha Huy Hung Van-An Tran The-Hung Nguyen Dinh-Quan Nguyen |
| author_facet | Dinh-Son Le Hai-Hong Phan Ha Huy Hung Van-An Tran The-Hung Nguyen Dinh-Quan Nguyen |
| author_sort | Dinh-Son Le |
| collection | DOAJ |
| description | In this paper, we propose an integrated approach to robot vision: a key frame-based skeleton feature estimation and action recognition network (KFSENet) that incorporates action recognition with face and emotion recognition to enable social robots to engage in more personal interactions. Instead of extracting the human skeleton features from the entire video, we propose a key frame-based approach for their extraction using pose estimation models. We select the key frames using the gradient of a proposed total motion metric that is computed using dense optical flow. We use the extracted human skeleton features from the selected key frames to train a deep neural network (i.e., the double-feature double-motion network (DDNet)) for action recognition. The proposed KFSENet utilizes a simpler model to learn and differentiate between the different action classes, is computationally simpler and yields better action recognition performance when compared with existing methods. The use of key frames allows the proposed method to eliminate unnecessary and redundant information, which improves its classification accuracy and decreases its computational cost. The proposed method is tested on both publicly available standard benchmark datasets and self-collected datasets. The performance of the proposed method is compared to existing state-of-the-art methods. Our results indicate that the proposed method yields better performance compared with existing methods. Moreover, our proposed framework integrates face and emotion recognition to enable social robots to engage in more personal interaction with humans. |
| first_indexed | 2024-03-10T01:31:14Z |
| format | Article |
| id | doaj.art-fbe4662c22524291b6b47482df7a7023 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-10T01:31:14Z |
| publishDate | 2022-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-fbe4662c22524291b6b47482df7a70232023-11-23T13:41:59ZengMDPI AGApplied Sciences2076-34172022-05-011211545510.3390/app12115455KFSENet: A Key Frame-Based Skeleton Feature Estimation and Action Recognition Network for Improved Robot Vision with Face and Emotion RecognitionDinh-Son Le0Hai-Hong Phan1Ha Huy Hung2Van-An Tran3The-Hung Nguyen4Dinh-Quan Nguyen5Faculty of Information Technology, Le Quy Don Technical University, 236 Hoang Quoc Viet, Bac Tu Liem, Ha Noi 11900, VietnamFaculty of Information Technology, Le Quy Don Technical University, 236 Hoang Quoc Viet, Bac Tu Liem, Ha Noi 11900, VietnamFaculty of Aerospace Engineering, Le Quy Don Technical University, 236 Hoang Quoc Viet, Bac Tu Liem, Ha Noi 11900, VietnamFaculty of Information Technology, Le Quy Don Technical University, 236 Hoang Quoc Viet, Bac Tu Liem, Ha Noi 11900, VietnamFaculty of Technical Management, Le Quy Don Technical University, 236 Hoang Quoc Viet, Bac Tu Liem, Ha Noi 11900, VietnamFaculty of Aerospace Engineering, Le Quy Don Technical University, 236 Hoang Quoc Viet, Bac Tu Liem, Ha Noi 11900, VietnamIn this paper, we propose an integrated approach to robot vision: a key frame-based skeleton feature estimation and action recognition network (KFSENet) that incorporates action recognition with face and emotion recognition to enable social robots to engage in more personal interactions. Instead of extracting the human skeleton features from the entire video, we propose a key frame-based approach for their extraction using pose estimation models. We select the key frames using the gradient of a proposed total motion metric that is computed using dense optical flow. We use the extracted human skeleton features from the selected key frames to train a deep neural network (i.e., the double-feature double-motion network (DDNet)) for action recognition. The proposed KFSENet utilizes a simpler model to learn and differentiate between the different action classes, is computationally simpler and yields better action recognition performance when compared with existing methods. The use of key frames allows the proposed method to eliminate unnecessary and redundant information, which improves its classification accuracy and decreases its computational cost. The proposed method is tested on both publicly available standard benchmark datasets and self-collected datasets. The performance of the proposed method is compared to existing state-of-the-art methods. Our results indicate that the proposed method yields better performance compared with existing methods. Moreover, our proposed framework integrates face and emotion recognition to enable social robots to engage in more personal interaction with humans.https://www.mdpi.com/2076-3417/12/11/5455robotsvisionkey framedeep neural networkoptical flowface recognition |
| spellingShingle | Dinh-Son Le Hai-Hong Phan Ha Huy Hung Van-An Tran The-Hung Nguyen Dinh-Quan Nguyen KFSENet: A Key Frame-Based Skeleton Feature Estimation and Action Recognition Network for Improved Robot Vision with Face and Emotion Recognition Applied Sciences robots vision key frame deep neural network optical flow face recognition |
| title | KFSENet: A Key Frame-Based Skeleton Feature Estimation and Action Recognition Network for Improved Robot Vision with Face and Emotion Recognition |
| title_full | KFSENet: A Key Frame-Based Skeleton Feature Estimation and Action Recognition Network for Improved Robot Vision with Face and Emotion Recognition |
| title_fullStr | KFSENet: A Key Frame-Based Skeleton Feature Estimation and Action Recognition Network for Improved Robot Vision with Face and Emotion Recognition |
| title_full_unstemmed | KFSENet: A Key Frame-Based Skeleton Feature Estimation and Action Recognition Network for Improved Robot Vision with Face and Emotion Recognition |
| title_short | KFSENet: A Key Frame-Based Skeleton Feature Estimation and Action Recognition Network for Improved Robot Vision with Face and Emotion Recognition |
| title_sort | kfsenet a key frame based skeleton feature estimation and action recognition network for improved robot vision with face and emotion recognition |
| topic | robots vision key frame deep neural network optical flow face recognition |
| url | https://www.mdpi.com/2076-3417/12/11/5455 |
| work_keys_str_mv | AT dinhsonle kfsenetakeyframebasedskeletonfeatureestimationandactionrecognitionnetworkforimprovedrobotvisionwithfaceandemotionrecognition AT haihongphan kfsenetakeyframebasedskeletonfeatureestimationandactionrecognitionnetworkforimprovedrobotvisionwithfaceandemotionrecognition AT hahuyhung kfsenetakeyframebasedskeletonfeatureestimationandactionrecognitionnetworkforimprovedrobotvisionwithfaceandemotionrecognition AT vanantran kfsenetakeyframebasedskeletonfeatureestimationandactionrecognitionnetworkforimprovedrobotvisionwithfaceandemotionrecognition AT thehungnguyen kfsenetakeyframebasedskeletonfeatureestimationandactionrecognitionnetworkforimprovedrobotvisionwithfaceandemotionrecognition AT dinhquannguyen kfsenetakeyframebasedskeletonfeatureestimationandactionrecognitionnetworkforimprovedrobotvisionwithfaceandemotionrecognition |