Development of Real-Time Landmark-Based Emotion Recognition CNN for Masked Faces
Owing to the availability of a wide range of emotion recognition applications in our lives, such as for mental status calculation, the demand for high-performance emotion recognition approaches remains uncertain. Nevertheless, the wearing of facial masks has been indispensable during the COVID-19 pa...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-11-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/22/22/8704 |
| _version_ | 1827643750098141184 |
|---|---|
| author | Akhmedov Farkhod Akmalbek Bobomirzaevich Abdusalomov Mukhriddin Mukhiddinov Young-Im Cho |
| author_facet | Akhmedov Farkhod Akmalbek Bobomirzaevich Abdusalomov Mukhriddin Mukhiddinov Young-Im Cho |
| author_sort | Akhmedov Farkhod |
| collection | DOAJ |
| description | Owing to the availability of a wide range of emotion recognition applications in our lives, such as for mental status calculation, the demand for high-performance emotion recognition approaches remains uncertain. Nevertheless, the wearing of facial masks has been indispensable during the COVID-19 pandemic. In this study, we propose a graph-based emotion recognition method that adopts landmarks on the upper part of the face. Based on the proposed approach, several pre-processing steps were applied. After pre-processing, facial expression features need to be extracted from facial key points. The main steps of emotion recognition on masked faces include face detection by using Haar–Cascade, landmark implementation through a media-pipe face mesh model, and model training on seven emotional classes. The FER-2013 dataset was used for model training. An emotion detection model was developed for non-masked faces. Thereafter, landmarks were applied to the upper part of the face. After the detection of faces and landmark locations were extracted, we captured coordinates of emotional class landmarks and exported to a comma-separated values (csv) file. After that, model weights were transferred to the emotional classes. Finally, a landmark-based emotion recognition model for the upper facial parts was tested both on images and in real time using a web camera application. The results showed that the proposed model achieved an overall accuracy of 91.2% for seven emotional classes in the case of an image application. Image based emotion detection of the proposed model accuracy showed relatively higher results than the real-time emotion detection. |
| first_indexed | 2024-03-09T18:01:27Z |
| format | Article |
| id | doaj.art-d20aae27761f4b0b9fa247821f5bd80f |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T18:01:27Z |
| publishDate | 2022-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-d20aae27761f4b0b9fa247821f5bd80f2023-11-24T09:54:29ZengMDPI AGSensors1424-82202022-11-012222870410.3390/s22228704Development of Real-Time Landmark-Based Emotion Recognition CNN for Masked FacesAkhmedov Farkhod0Akmalbek Bobomirzaevich Abdusalomov1Mukhriddin Mukhiddinov2Young-Im Cho3Department Computer Engineering, Gachon University, Sujeong-gu, Seongnam-si 461-701, KoreaDepartment Computer Engineering, Gachon University, Sujeong-gu, Seongnam-si 461-701, KoreaDepartment Computer Engineering, Gachon University, Sujeong-gu, Seongnam-si 461-701, KoreaDepartment Computer Engineering, Gachon University, Sujeong-gu, Seongnam-si 461-701, KoreaOwing to the availability of a wide range of emotion recognition applications in our lives, such as for mental status calculation, the demand for high-performance emotion recognition approaches remains uncertain. Nevertheless, the wearing of facial masks has been indispensable during the COVID-19 pandemic. In this study, we propose a graph-based emotion recognition method that adopts landmarks on the upper part of the face. Based on the proposed approach, several pre-processing steps were applied. After pre-processing, facial expression features need to be extracted from facial key points. The main steps of emotion recognition on masked faces include face detection by using Haar–Cascade, landmark implementation through a media-pipe face mesh model, and model training on seven emotional classes. The FER-2013 dataset was used for model training. An emotion detection model was developed for non-masked faces. Thereafter, landmarks were applied to the upper part of the face. After the detection of faces and landmark locations were extracted, we captured coordinates of emotional class landmarks and exported to a comma-separated values (csv) file. After that, model weights were transferred to the emotional classes. Finally, a landmark-based emotion recognition model for the upper facial parts was tested both on images and in real time using a web camera application. The results showed that the proposed model achieved an overall accuracy of 91.2% for seven emotional classes in the case of an image application. Image based emotion detection of the proposed model accuracy showed relatively higher results than the real-time emotion detection.https://www.mdpi.com/1424-8220/22/22/8704face detectionemotion recognitionfacial masklandmark vectors applicationfacial expression detection |
| spellingShingle | Akhmedov Farkhod Akmalbek Bobomirzaevich Abdusalomov Mukhriddin Mukhiddinov Young-Im Cho Development of Real-Time Landmark-Based Emotion Recognition CNN for Masked Faces Sensors face detection emotion recognition facial mask landmark vectors application facial expression detection |
| title | Development of Real-Time Landmark-Based Emotion Recognition CNN for Masked Faces |
| title_full | Development of Real-Time Landmark-Based Emotion Recognition CNN for Masked Faces |
| title_fullStr | Development of Real-Time Landmark-Based Emotion Recognition CNN for Masked Faces |
| title_full_unstemmed | Development of Real-Time Landmark-Based Emotion Recognition CNN for Masked Faces |
| title_short | Development of Real-Time Landmark-Based Emotion Recognition CNN for Masked Faces |
| title_sort | development of real time landmark based emotion recognition cnn for masked faces |
| topic | face detection emotion recognition facial mask landmark vectors application facial expression detection |
| url | https://www.mdpi.com/1424-8220/22/22/8704 |
| work_keys_str_mv | AT akhmedovfarkhod developmentofrealtimelandmarkbasedemotionrecognitioncnnformaskedfaces AT akmalbekbobomirzaevichabdusalomov developmentofrealtimelandmarkbasedemotionrecognitioncnnformaskedfaces AT mukhriddinmukhiddinov developmentofrealtimelandmarkbasedemotionrecognitioncnnformaskedfaces AT youngimcho developmentofrealtimelandmarkbasedemotionrecognitioncnnformaskedfaces |