Facial Expression Recognition Using Hybrid Features of Pixel and Geometry
Facial Expression Recognition (FER) has long been a challenging task in the field of computer vision. Most of the existing FER methods extract facial features on the basis of face pixels, ignoring the relative geometric position dependencies of facial landmark points. This article presents a hybrid...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9335586/ |
| _version_ | 1811210539871764480 |
|---|---|
| author | Chang Liu Kaoru Hirota Junjie Ma Zhiyang Jia Yaping Dai |
| author_facet | Chang Liu Kaoru Hirota Junjie Ma Zhiyang Jia Yaping Dai |
| author_sort | Chang Liu |
| collection | DOAJ |
| description | Facial Expression Recognition (FER) has long been a challenging task in the field of computer vision. Most of the existing FER methods extract facial features on the basis of face pixels, ignoring the relative geometric position dependencies of facial landmark points. This article presents a hybrid feature extraction network to enhance the discriminative power of emotional features. The proposed network consists of a Spatial Attention Convolutional Neural Network (SACNN) and a series of Long Short-term Memory networks with Attention mechanism (ALSTMs). The SACNN is employed to extract the expressional features from static face images and the ALSTMs is designed to explore the potentials of facial landmarks for expression recognition. A deep geometric feature descriptor is proposed to characterize the relative geometric position correlation of facial landmarks. The landmarks are divided into seven groups to extract deep geometric features, and the attention module in ALSTMs can adaptively estimate the importance of different landmark regions. By jointly combining SACNN and ALSTMs, the hybrid features are obtained for expression recognition. Experiments conducted on three public databases, FER2013, CK+, and JAFFE, demonstrate that the proposed method outperforms the previous methods, with the accuracies of 74.31%, 95.15%, and 98.57%, respectively. The preliminary results of Emotion Understanding Robot System (EURS) indicate that the proposed method has the potential to improve the performance of human-robot interaction. |
| first_indexed | 2024-04-12T04:56:20Z |
| format | Article |
| id | doaj.art-e151acaa912948fba054239c89e874df |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-12T04:56:20Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-e151acaa912948fba054239c89e874df2022-12-22T03:47:07ZengIEEEIEEE Access2169-35362021-01-019188761888910.1109/ACCESS.2021.30543329335586Facial Expression Recognition Using Hybrid Features of Pixel and GeometryChang Liu0https://orcid.org/0000-0001-5641-8084Kaoru Hirota1https://orcid.org/0000-0001-5347-6182Junjie Ma2https://orcid.org/0000-0002-8593-3074Zhiyang Jia3https://orcid.org/0000-0003-3248-8875Yaping Dai4https://orcid.org/0000-0001-8795-5333School of Automation, Beijing Institute of Technology, Beijing, ChinaSchool of Automation, Beijing Institute of Technology, Beijing, ChinaDepartment of Computer Science and Technology, Tsinghua University, Beijing, ChinaSchool of Automation, Beijing Institute of Technology, Beijing, ChinaSchool of Automation, Beijing Institute of Technology, Beijing, ChinaFacial Expression Recognition (FER) has long been a challenging task in the field of computer vision. Most of the existing FER methods extract facial features on the basis of face pixels, ignoring the relative geometric position dependencies of facial landmark points. This article presents a hybrid feature extraction network to enhance the discriminative power of emotional features. The proposed network consists of a Spatial Attention Convolutional Neural Network (SACNN) and a series of Long Short-term Memory networks with Attention mechanism (ALSTMs). The SACNN is employed to extract the expressional features from static face images and the ALSTMs is designed to explore the potentials of facial landmarks for expression recognition. A deep geometric feature descriptor is proposed to characterize the relative geometric position correlation of facial landmarks. The landmarks are divided into seven groups to extract deep geometric features, and the attention module in ALSTMs can adaptively estimate the importance of different landmark regions. By jointly combining SACNN and ALSTMs, the hybrid features are obtained for expression recognition. Experiments conducted on three public databases, FER2013, CK+, and JAFFE, demonstrate that the proposed method outperforms the previous methods, with the accuracies of 74.31%, 95.15%, and 98.57%, respectively. The preliminary results of Emotion Understanding Robot System (EURS) indicate that the proposed method has the potential to improve the performance of human-robot interaction.https://ieeexplore.ieee.org/document/9335586/Facial expression recognitionlong short-term memory networkrelative geometric position dependencyhybrid featureattention mechanism |
| spellingShingle | Chang Liu Kaoru Hirota Junjie Ma Zhiyang Jia Yaping Dai Facial Expression Recognition Using Hybrid Features of Pixel and Geometry IEEE Access Facial expression recognition long short-term memory network relative geometric position dependency hybrid feature attention mechanism |
| title | Facial Expression Recognition Using Hybrid Features of Pixel and Geometry |
| title_full | Facial Expression Recognition Using Hybrid Features of Pixel and Geometry |
| title_fullStr | Facial Expression Recognition Using Hybrid Features of Pixel and Geometry |
| title_full_unstemmed | Facial Expression Recognition Using Hybrid Features of Pixel and Geometry |
| title_short | Facial Expression Recognition Using Hybrid Features of Pixel and Geometry |
| title_sort | facial expression recognition using hybrid features of pixel and geometry |
| topic | Facial expression recognition long short-term memory network relative geometric position dependency hybrid feature attention mechanism |
| url | https://ieeexplore.ieee.org/document/9335586/ |
| work_keys_str_mv | AT changliu facialexpressionrecognitionusinghybridfeaturesofpixelandgeometry AT kaoruhirota facialexpressionrecognitionusinghybridfeaturesofpixelandgeometry AT junjiema facialexpressionrecognitionusinghybridfeaturesofpixelandgeometry AT zhiyangjia facialexpressionrecognitionusinghybridfeaturesofpixelandgeometry AT yapingdai facialexpressionrecognitionusinghybridfeaturesofpixelandgeometry |