Efficient Facial Expression Recognition Algorithm Based on Hierarchical Deep Neural Network Structure
With the continued development of artificial intelligence (AI) technology, research on interaction technology has become more popular. Facial expression recognition (FER) is an important type of visual information that can be used to understand a human's emotional situation. In particular, the...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2019-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/8673885/ |
| _version_ | 1818565974254682112 |
|---|---|
| author | Ji-Hae Kim Byung-Gyu Kim Partha Pratim Roy Da-Mi Jeong |
| author_facet | Ji-Hae Kim Byung-Gyu Kim Partha Pratim Roy Da-Mi Jeong |
| author_sort | Ji-Hae Kim |
| collection | DOAJ |
| description | With the continued development of artificial intelligence (AI) technology, research on interaction technology has become more popular. Facial expression recognition (FER) is an important type of visual information that can be used to understand a human's emotional situation. In particular, the importance of AI systems has recently increased due to advancements in research on AI systems applied to AI robots. In this paper, we propose a new scheme for FER system based on hierarchical deep learning. The feature extracted from the appearance feature-based network is fused with the geometric feature in a hierarchical structure. The appearance feature-based network extracts holistic features of the face using the preprocessed LBP image, whereas the geometric feature-based network learns the coordinate change of action units (AUs) landmark, which is a muscle that moves mainly when making facial expressions. The proposed method combines the result of the softmax function of two features by considering the error associated with the second highest emotion (Top-2) prediction result. In addition, we propose a technique to generate facial images with neutral emotion using the autoencoder technique. By this technique, we can extract the dynamic facial features between the neutral and emotional images without sequence data. We compare the proposed algorithm with the other recent algorithms for CK+ and JAFFE dataset, which are typically considered to be verified datasets in the facial expression recognition. The ten-fold cross validation results show 96.46% of accuracy in the CK+ dataset and 91.27% of accuracy in the JAFFE dataset. When comparing with other methods, the result of the proposed hierarchical deep network structure shows up to about 3% of the accuracy improvement and 1.3% of average improvement in CK+ dataset, respectively. In JAFFE datasets, up to about 7% of the accuracy is enhanced, and the average improvement is verified by about 1.5%. |
| first_indexed | 2024-12-14T01:47:45Z |
| format | Article |
| id | doaj.art-3ae96a9cdc58456490ebe4ee25ca1209 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-14T01:47:45Z |
| publishDate | 2019-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-3ae96a9cdc58456490ebe4ee25ca12092022-12-21T23:21:29ZengIEEEIEEE Access2169-35362019-01-017412734128510.1109/ACCESS.2019.29073278673885Efficient Facial Expression Recognition Algorithm Based on Hierarchical Deep Neural Network StructureJi-Hae Kim0Byung-Gyu Kim1https://orcid.org/0000-0001-6555-3464Partha Pratim Roy2Da-Mi Jeong3Department of IT Engineering, Sookmyung Women’s University, Seoul, South KoreaDepartment of IT Engineering, Sookmyung Women’s University, Seoul, South KoreaSchool of Computer Science and Engineering, IIT Roorkee, Roorkee, IndiaDepartment of IT Engineering, Sookmyung Women’s University, Seoul, South KoreaWith the continued development of artificial intelligence (AI) technology, research on interaction technology has become more popular. Facial expression recognition (FER) is an important type of visual information that can be used to understand a human's emotional situation. In particular, the importance of AI systems has recently increased due to advancements in research on AI systems applied to AI robots. In this paper, we propose a new scheme for FER system based on hierarchical deep learning. The feature extracted from the appearance feature-based network is fused with the geometric feature in a hierarchical structure. The appearance feature-based network extracts holistic features of the face using the preprocessed LBP image, whereas the geometric feature-based network learns the coordinate change of action units (AUs) landmark, which is a muscle that moves mainly when making facial expressions. The proposed method combines the result of the softmax function of two features by considering the error associated with the second highest emotion (Top-2) prediction result. In addition, we propose a technique to generate facial images with neutral emotion using the autoencoder technique. By this technique, we can extract the dynamic facial features between the neutral and emotional images without sequence data. We compare the proposed algorithm with the other recent algorithms for CK+ and JAFFE dataset, which are typically considered to be verified datasets in the facial expression recognition. The ten-fold cross validation results show 96.46% of accuracy in the CK+ dataset and 91.27% of accuracy in the JAFFE dataset. When comparing with other methods, the result of the proposed hierarchical deep network structure shows up to about 3% of the accuracy improvement and 1.3% of average improvement in CK+ dataset, respectively. In JAFFE datasets, up to about 7% of the accuracy is enhanced, and the average improvement is verified by about 1.5%.https://ieeexplore.ieee.org/document/8673885/Artificial intelligence (AI)facial expression recognition (FER)emotion recognitiondeep learningLBP featuregeometric feature |
| spellingShingle | Ji-Hae Kim Byung-Gyu Kim Partha Pratim Roy Da-Mi Jeong Efficient Facial Expression Recognition Algorithm Based on Hierarchical Deep Neural Network Structure IEEE Access Artificial intelligence (AI) facial expression recognition (FER) emotion recognition deep learning LBP feature geometric feature |
| title | Efficient Facial Expression Recognition Algorithm Based on Hierarchical Deep Neural Network Structure |
| title_full | Efficient Facial Expression Recognition Algorithm Based on Hierarchical Deep Neural Network Structure |
| title_fullStr | Efficient Facial Expression Recognition Algorithm Based on Hierarchical Deep Neural Network Structure |
| title_full_unstemmed | Efficient Facial Expression Recognition Algorithm Based on Hierarchical Deep Neural Network Structure |
| title_short | Efficient Facial Expression Recognition Algorithm Based on Hierarchical Deep Neural Network Structure |
| title_sort | efficient facial expression recognition algorithm based on hierarchical deep neural network structure |
| topic | Artificial intelligence (AI) facial expression recognition (FER) emotion recognition deep learning LBP feature geometric feature |
| url | https://ieeexplore.ieee.org/document/8673885/ |
| work_keys_str_mv | AT jihaekim efficientfacialexpressionrecognitionalgorithmbasedonhierarchicaldeepneuralnetworkstructure AT byunggyukim efficientfacialexpressionrecognitionalgorithmbasedonhierarchicaldeepneuralnetworkstructure AT parthapratimroy efficientfacialexpressionrecognitionalgorithmbasedonhierarchicaldeepneuralnetworkstructure AT damijeong efficientfacialexpressionrecognitionalgorithmbasedonhierarchicaldeepneuralnetworkstructure |