Face Recognition and Tracking Framework for Human–Robot Interaction
Recently, face recognition became a key element in social cognition which is used in various applications including human–robot interaction (HRI), pedestrian identification, and surveillance systems. Deep convolutional neural networks (CNNs) have achieved notable progress in recognizing faces. Howev...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-05-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/12/11/5568 |
| _version_ | 1827665551942483968 |
|---|---|
| author | Aly Khalifa Ahmed A. Abdelrahman Dominykas Strazdas Jan Hintz Thorsten Hempel Ayoub Al-Hamadi |
| author_facet | Aly Khalifa Ahmed A. Abdelrahman Dominykas Strazdas Jan Hintz Thorsten Hempel Ayoub Al-Hamadi |
| author_sort | Aly Khalifa |
| collection | DOAJ |
| description | Recently, face recognition became a key element in social cognition which is used in various applications including human–robot interaction (HRI), pedestrian identification, and surveillance systems. Deep convolutional neural networks (CNNs) have achieved notable progress in recognizing faces. However, achieving accurate and real-time face recognition is still a challenging problem, especially in unconstrained environments due to occlusion, lighting conditions, and the diversity in head poses. In this paper, we present a robust face recognition and tracking framework in unconstrained settings. We developed our framework based on lightweight CNNs for all face recognition stages, including face detection, alignment and feature extraction, to achieve higher accuracies in these challenging circumstances while maintaining the real-time capabilities required for HRI systems. To maintain the accuracy, a single-shot multi-level face localization in the wild (RetinaFace) is utilized for face detection, and additive angular margin loss (ArcFace) is employed for recognition. For further enhancement, we introduce a face tracking algorithm that combines the information from tracked faces with the recognized identity to use in the further frames. This tracking algorithm improves the overall processing time and accuracy. The proposed system performance is tested in real-time experiments applied in an HRI study. Our proposed framework achieves real-time capabilities with an average of 99%, 95%, and 97% precision, recall, and F-score respectively. In addition, we implemented our system as a modular ROS package that makes it straightforward for integration in different real-world HRI systems. |
| first_indexed | 2024-03-10T01:29:37Z |
| format | Article |
| id | doaj.art-5d8772a6ad7446c89007588322569747 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-10T01:29:37Z |
| publishDate | 2022-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-5d8772a6ad7446c890075883225697472023-11-23T13:43:59ZengMDPI AGApplied Sciences2076-34172022-05-011211556810.3390/app12115568Face Recognition and Tracking Framework for Human–Robot InteractionAly Khalifa0Ahmed A. Abdelrahman1Dominykas Strazdas2Jan Hintz3Thorsten Hempel4Ayoub Al-Hamadi5Neuro-Information Technology, Otto-von-Guericke-University Magdeburg, 39106 Magdeburg, GermanyNeuro-Information Technology, Otto-von-Guericke-University Magdeburg, 39106 Magdeburg, GermanyNeuro-Information Technology, Otto-von-Guericke-University Magdeburg, 39106 Magdeburg, GermanyNeuro-Information Technology, Otto-von-Guericke-University Magdeburg, 39106 Magdeburg, GermanyNeuro-Information Technology, Otto-von-Guericke-University Magdeburg, 39106 Magdeburg, GermanyNeuro-Information Technology, Otto-von-Guericke-University Magdeburg, 39106 Magdeburg, GermanyRecently, face recognition became a key element in social cognition which is used in various applications including human–robot interaction (HRI), pedestrian identification, and surveillance systems. Deep convolutional neural networks (CNNs) have achieved notable progress in recognizing faces. However, achieving accurate and real-time face recognition is still a challenging problem, especially in unconstrained environments due to occlusion, lighting conditions, and the diversity in head poses. In this paper, we present a robust face recognition and tracking framework in unconstrained settings. We developed our framework based on lightweight CNNs for all face recognition stages, including face detection, alignment and feature extraction, to achieve higher accuracies in these challenging circumstances while maintaining the real-time capabilities required for HRI systems. To maintain the accuracy, a single-shot multi-level face localization in the wild (RetinaFace) is utilized for face detection, and additive angular margin loss (ArcFace) is employed for recognition. For further enhancement, we introduce a face tracking algorithm that combines the information from tracked faces with the recognized identity to use in the further frames. This tracking algorithm improves the overall processing time and accuracy. The proposed system performance is tested in real-time experiments applied in an HRI study. Our proposed framework achieves real-time capabilities with an average of 99%, 95%, and 97% precision, recall, and F-score respectively. In addition, we implemented our system as a modular ROS package that makes it straightforward for integration in different real-world HRI systems.https://www.mdpi.com/2076-3417/12/11/5568face recognitionface trackingface detectionface alignmentperson identificationhuman–robot interaction |
| spellingShingle | Aly Khalifa Ahmed A. Abdelrahman Dominykas Strazdas Jan Hintz Thorsten Hempel Ayoub Al-Hamadi Face Recognition and Tracking Framework for Human–Robot Interaction Applied Sciences face recognition face tracking face detection face alignment person identification human–robot interaction |
| title | Face Recognition and Tracking Framework for Human–Robot Interaction |
| title_full | Face Recognition and Tracking Framework for Human–Robot Interaction |
| title_fullStr | Face Recognition and Tracking Framework for Human–Robot Interaction |
| title_full_unstemmed | Face Recognition and Tracking Framework for Human–Robot Interaction |
| title_short | Face Recognition and Tracking Framework for Human–Robot Interaction |
| title_sort | face recognition and tracking framework for human robot interaction |
| topic | face recognition face tracking face detection face alignment person identification human–robot interaction |
| url | https://www.mdpi.com/2076-3417/12/11/5568 |
| work_keys_str_mv | AT alykhalifa facerecognitionandtrackingframeworkforhumanrobotinteraction AT ahmedaabdelrahman facerecognitionandtrackingframeworkforhumanrobotinteraction AT dominykasstrazdas facerecognitionandtrackingframeworkforhumanrobotinteraction AT janhintz facerecognitionandtrackingframeworkforhumanrobotinteraction AT thorstenhempel facerecognitionandtrackingframeworkforhumanrobotinteraction AT ayoubalhamadi facerecognitionandtrackingframeworkforhumanrobotinteraction |