Micro-Expression-Based Emotion Recognition Using Waterfall Atrous Spatial Pyramid Pooling Networks
Understanding a person’s attitude or sentiment from their facial expressions has long been a straightforward task for humans. Numerous methods and techniques have been used to classify and interpret human emotions that are commonly communicated through facial expressions, with either macro- or micro...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-06-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/22/12/4634 |
| _version_ | 1797482372167368704 |
|---|---|
| author | Marzuraikah Mohd Stofa Mohd Asyraf Zulkifley Muhammad Ammirrul Atiqi Mohd Zainuri |
| author_facet | Marzuraikah Mohd Stofa Mohd Asyraf Zulkifley Muhammad Ammirrul Atiqi Mohd Zainuri |
| author_sort | Marzuraikah Mohd Stofa |
| collection | DOAJ |
| description | Understanding a person’s attitude or sentiment from their facial expressions has long been a straightforward task for humans. Numerous methods and techniques have been used to classify and interpret human emotions that are commonly communicated through facial expressions, with either macro- or micro-expressions. However, performing this task using computer-based techniques or algorithms has been proven to be extremely difficult, whereby it is a time-consuming task to annotate it manually. Compared to macro-expressions, micro-expressions manifest the real emotional cues of a human, which they try to suppress and hide. Different methods and algorithms for recognizing emotions using micro-expressions are examined in this research, and the results are presented in a comparative approach. The proposed technique is based on a multi-scale deep learning approach that aims to extract facial cues of various subjects under various conditions. Then, two popular multi-scale approaches are explored, Spatial Pyramid Pooling (SPP) and Atrous Spatial Pyramid Pooling (ASPP), which are then optimized to suit the purpose of emotion recognition using micro-expression cues. There are four new architectures introduced in this paper based on multi-layer multi-scale convolutional networks using both direct and waterfall network flows. The experimental results show that the ASPP module with waterfall network flow, which we coined as WASPP-Net, outperforms the state-of-the-art benchmark techniques with an accuracy of 80.5%. For future work, a high-resolution approach to multi-scale approaches can be explored to further improve the recognition performance. |
| first_indexed | 2024-03-09T22:31:22Z |
| format | Article |
| id | doaj.art-e4f943b924e9403a9c7f0a1954c83d01 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T22:31:22Z |
| publishDate | 2022-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-e4f943b924e9403a9c7f0a1954c83d012023-11-23T18:56:33ZengMDPI AGSensors1424-82202022-06-012212463410.3390/s22124634Micro-Expression-Based Emotion Recognition Using Waterfall Atrous Spatial Pyramid Pooling NetworksMarzuraikah Mohd Stofa0Mohd Asyraf Zulkifley1Muhammad Ammirrul Atiqi Mohd Zainuri2Department of Electrical, Electronic and Systems Engineering, Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia, Bangi 43600, Selangor, MalaysiaDepartment of Electrical, Electronic and Systems Engineering, Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia, Bangi 43600, Selangor, MalaysiaDepartment of Electrical, Electronic and Systems Engineering, Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia, Bangi 43600, Selangor, MalaysiaUnderstanding a person’s attitude or sentiment from their facial expressions has long been a straightforward task for humans. Numerous methods and techniques have been used to classify and interpret human emotions that are commonly communicated through facial expressions, with either macro- or micro-expressions. However, performing this task using computer-based techniques or algorithms has been proven to be extremely difficult, whereby it is a time-consuming task to annotate it manually. Compared to macro-expressions, micro-expressions manifest the real emotional cues of a human, which they try to suppress and hide. Different methods and algorithms for recognizing emotions using micro-expressions are examined in this research, and the results are presented in a comparative approach. The proposed technique is based on a multi-scale deep learning approach that aims to extract facial cues of various subjects under various conditions. Then, two popular multi-scale approaches are explored, Spatial Pyramid Pooling (SPP) and Atrous Spatial Pyramid Pooling (ASPP), which are then optimized to suit the purpose of emotion recognition using micro-expression cues. There are four new architectures introduced in this paper based on multi-layer multi-scale convolutional networks using both direct and waterfall network flows. The experimental results show that the ASPP module with waterfall network flow, which we coined as WASPP-Net, outperforms the state-of-the-art benchmark techniques with an accuracy of 80.5%. For future work, a high-resolution approach to multi-scale approaches can be explored to further improve the recognition performance.https://www.mdpi.com/1424-8220/22/12/4634deep learningconvolutional neural networksmicro-expression analysisemotion classification |
| spellingShingle | Marzuraikah Mohd Stofa Mohd Asyraf Zulkifley Muhammad Ammirrul Atiqi Mohd Zainuri Micro-Expression-Based Emotion Recognition Using Waterfall Atrous Spatial Pyramid Pooling Networks Sensors deep learning convolutional neural networks micro-expression analysis emotion classification |
| title | Micro-Expression-Based Emotion Recognition Using Waterfall Atrous Spatial Pyramid Pooling Networks |
| title_full | Micro-Expression-Based Emotion Recognition Using Waterfall Atrous Spatial Pyramid Pooling Networks |
| title_fullStr | Micro-Expression-Based Emotion Recognition Using Waterfall Atrous Spatial Pyramid Pooling Networks |
| title_full_unstemmed | Micro-Expression-Based Emotion Recognition Using Waterfall Atrous Spatial Pyramid Pooling Networks |
| title_short | Micro-Expression-Based Emotion Recognition Using Waterfall Atrous Spatial Pyramid Pooling Networks |
| title_sort | micro expression based emotion recognition using waterfall atrous spatial pyramid pooling networks |
| topic | deep learning convolutional neural networks micro-expression analysis emotion classification |
| url | https://www.mdpi.com/1424-8220/22/12/4634 |
| work_keys_str_mv | AT marzuraikahmohdstofa microexpressionbasedemotionrecognitionusingwaterfallatrousspatialpyramidpoolingnetworks AT mohdasyrafzulkifley microexpressionbasedemotionrecognitionusingwaterfallatrousspatialpyramidpoolingnetworks AT muhammadammirrulatiqimohdzainuri microexpressionbasedemotionrecognitionusingwaterfallatrousspatialpyramidpoolingnetworks |