Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition
Emotions are multimodal processes that play a crucial role in our everyday lives. Recognizing emotions is becoming more critical in a wide range of application domains such as healthcare, education, human-computer interaction, Virtual Reality, intelligent agents, entertainment, and more. Facial macr...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-06-01
|
| Series: | Frontiers in Psychology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fpsyg.2022.864047/full |
| _version_ | 1818216368834609152 |
|---|---|
| author | Nastaran Saffaryazdi Syed Talal Wasim Kuldeep Dileep Alireza Farrokhi Nia Suranga Nanayakkara Elizabeth Broadbent Mark Billinghurst |
| author_facet | Nastaran Saffaryazdi Syed Talal Wasim Kuldeep Dileep Alireza Farrokhi Nia Suranga Nanayakkara Elizabeth Broadbent Mark Billinghurst |
| author_sort | Nastaran Saffaryazdi |
| collection | DOAJ |
| description | Emotions are multimodal processes that play a crucial role in our everyday lives. Recognizing emotions is becoming more critical in a wide range of application domains such as healthcare, education, human-computer interaction, Virtual Reality, intelligent agents, entertainment, and more. Facial macro-expressions or intense facial expressions are the most common modalities in recognizing emotional states. However, since facial expressions can be voluntarily controlled, they may not accurately represent emotional states. Earlier studies have shown that facial micro-expressions are more reliable than facial macro-expressions for revealing emotions. They are subtle, involuntary movements responding to external stimuli that cannot be controlled. This paper proposes using facial micro-expressions combined with brain and physiological signals to more reliably detect underlying emotions. We describe our models for measuring arousal and valence levels from a combination of facial micro-expressions, Electroencephalography (EEG) signals, galvanic skin responses (GSR), and Photoplethysmography (PPG) signals. We then evaluate our model using the DEAP dataset and our own dataset based on a subject-independent approach. Lastly, we discuss our results, the limitations of our work, and how these limitations could be overcome. We also discuss future directions for using facial micro-expressions and physiological signals in emotion recognition. |
| first_indexed | 2024-12-12T06:50:52Z |
| format | Article |
| id | doaj.art-453449fab21d485cb75b2708e2f32c79 |
| institution | Directory Open Access Journal |
| issn | 1664-1078 |
| language | English |
| last_indexed | 2024-12-12T06:50:52Z |
| publishDate | 2022-06-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Psychology |
| spelling | doaj.art-453449fab21d485cb75b2708e2f32c792022-12-22T00:34:04ZengFrontiers Media S.A.Frontiers in Psychology1664-10782022-06-011310.3389/fpsyg.2022.864047864047Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion RecognitionNastaran Saffaryazdi0Syed Talal Wasim1Kuldeep Dileep2Alireza Farrokhi Nia3Suranga Nanayakkara4Elizabeth Broadbent5Mark Billinghurst6Empathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New ZealandEmpathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New ZealandEmpathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New ZealandEmpathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New ZealandAugmented Human Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New ZealandDepartment of Psychological Medicine, The University of Auckland, Auckland, New ZealandEmpathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New ZealandEmotions are multimodal processes that play a crucial role in our everyday lives. Recognizing emotions is becoming more critical in a wide range of application domains such as healthcare, education, human-computer interaction, Virtual Reality, intelligent agents, entertainment, and more. Facial macro-expressions or intense facial expressions are the most common modalities in recognizing emotional states. However, since facial expressions can be voluntarily controlled, they may not accurately represent emotional states. Earlier studies have shown that facial micro-expressions are more reliable than facial macro-expressions for revealing emotions. They are subtle, involuntary movements responding to external stimuli that cannot be controlled. This paper proposes using facial micro-expressions combined with brain and physiological signals to more reliably detect underlying emotions. We describe our models for measuring arousal and valence levels from a combination of facial micro-expressions, Electroencephalography (EEG) signals, galvanic skin responses (GSR), and Photoplethysmography (PPG) signals. We then evaluate our model using the DEAP dataset and our own dataset based on a subject-independent approach. Lastly, we discuss our results, the limitations of our work, and how these limitations could be overcome. We also discuss future directions for using facial micro-expressions and physiological signals in emotion recognition.https://www.frontiersin.org/articles/10.3389/fpsyg.2022.864047/fullemotion recognitionelectroencephalography (EEG)facial micro-expressionsphysiological signalsneural networksdecision fusion |
| spellingShingle | Nastaran Saffaryazdi Syed Talal Wasim Kuldeep Dileep Alireza Farrokhi Nia Suranga Nanayakkara Elizabeth Broadbent Mark Billinghurst Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition Frontiers in Psychology emotion recognition electroencephalography (EEG) facial micro-expressions physiological signals neural networks decision fusion |
| title | Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition |
| title_full | Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition |
| title_fullStr | Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition |
| title_full_unstemmed | Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition |
| title_short | Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition |
| title_sort | using facial micro expressions in combination with eeg and physiological signals for emotion recognition |
| topic | emotion recognition electroencephalography (EEG) facial micro-expressions physiological signals neural networks decision fusion |
| url | https://www.frontiersin.org/articles/10.3389/fpsyg.2022.864047/full |
| work_keys_str_mv | AT nastaransaffaryazdi usingfacialmicroexpressionsincombinationwitheegandphysiologicalsignalsforemotionrecognition AT syedtalalwasim usingfacialmicroexpressionsincombinationwitheegandphysiologicalsignalsforemotionrecognition AT kuldeepdileep usingfacialmicroexpressionsincombinationwitheegandphysiologicalsignalsforemotionrecognition AT alirezafarrokhinia usingfacialmicroexpressionsincombinationwitheegandphysiologicalsignalsforemotionrecognition AT surangananayakkara usingfacialmicroexpressionsincombinationwitheegandphysiologicalsignalsforemotionrecognition AT elizabethbroadbent usingfacialmicroexpressionsincombinationwitheegandphysiologicalsignalsforemotionrecognition AT markbillinghurst usingfacialmicroexpressionsincombinationwitheegandphysiologicalsignalsforemotionrecognition |