Analyzing the Direction of Emotional Influence in Nonverbal Dyadic Communication: A Facial-Expression Study
Identifying the direction of emotional influence in a dyadic dialogue is of increasing interest in the psychological sciences with applications in psychotherapy, analysis of political interactions, or interpersonal conflict behavior. Facial expressions are widely described as being automatic and thu...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9425581/ |
| _version_ | 1798001908017790976 |
|---|---|
| author | Maha Shadaydeh Lea Muller Dana Schneider Martin Thummel Thomas Kessler Joachim Denzler |
| author_facet | Maha Shadaydeh Lea Muller Dana Schneider Martin Thummel Thomas Kessler Joachim Denzler |
| author_sort | Maha Shadaydeh |
| collection | DOAJ |
| description | Identifying the direction of emotional influence in a dyadic dialogue is of increasing interest in the psychological sciences with applications in psychotherapy, analysis of political interactions, or interpersonal conflict behavior. Facial expressions are widely described as being automatic and thus hard to be overtly influenced. As such, they are a perfect measure for a better understanding of unintentional behavior cues about socio-emotional cognitive processes. With this view, this study is concerned with the analysis of the direction of emotional influence in dyadic dialogues based on facial expressions only. We exploit computer vision capabilities along with causal inference theory for quantitative verification of hypotheses on the direction of emotional influence, i.e., cause-effect relationships, in dyadic dialogues. We address two main issues. First, in a dyadic dialogue, emotional influence occurs over transient time intervals and with intensity and direction that are variant over time. To this end, we propose a relevant interval selection approach that we use prior to causal inference to identify those transient intervals where causal inference should be applied. Second, we propose to use fine-grained facial expressions that are present when strong distinct facial emotions are not visible. To specify the direction of influence, we apply the concept of Granger causality to the time-series of facial expressions over selected relevant intervals. We tested our approach on newly, experimentally obtained data. Based on quantitative verification of hypotheses on the direction of emotional influence, we were able to show that the proposed approach is promising to reveal the cause-effect pattern in various instructed interaction conditions. |
| first_indexed | 2024-04-11T11:43:45Z |
| format | Article |
| id | doaj.art-ba2cae7bd3634580bdaffe9679ed8c03 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-11T11:43:45Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-ba2cae7bd3634580bdaffe9679ed8c032022-12-22T04:25:43ZengIEEEIEEE Access2169-35362021-01-019737807379010.1109/ACCESS.2021.30781959425581Analyzing the Direction of Emotional Influence in Nonverbal Dyadic Communication: A Facial-Expression StudyMaha Shadaydeh0https://orcid.org/0000-0001-6455-2400Lea Muller1Dana Schneider2Martin Thummel3https://orcid.org/0000-0002-6993-6101Thomas Kessler4https://orcid.org/0000-0001-5318-2379Joachim Denzler5https://orcid.org/0000-0002-3193-3300Computer Vision Group, Friedrich Schiller University of Jena, Jena, GermanyMax Planck Institute for Intelligent Systems, Tübingen, GermanyDepartment of Social Psychology, Friedrich Schiller University of Jena, Jena, GermanyComputer Vision Group, Friedrich Schiller University of Jena, Jena, GermanyDepartment of Social Psychology, Friedrich Schiller University of Jena, Jena, GermanyComputer Vision Group, Friedrich Schiller University of Jena, Jena, GermanyIdentifying the direction of emotional influence in a dyadic dialogue is of increasing interest in the psychological sciences with applications in psychotherapy, analysis of political interactions, or interpersonal conflict behavior. Facial expressions are widely described as being automatic and thus hard to be overtly influenced. As such, they are a perfect measure for a better understanding of unintentional behavior cues about socio-emotional cognitive processes. With this view, this study is concerned with the analysis of the direction of emotional influence in dyadic dialogues based on facial expressions only. We exploit computer vision capabilities along with causal inference theory for quantitative verification of hypotheses on the direction of emotional influence, i.e., cause-effect relationships, in dyadic dialogues. We address two main issues. First, in a dyadic dialogue, emotional influence occurs over transient time intervals and with intensity and direction that are variant over time. To this end, we propose a relevant interval selection approach that we use prior to causal inference to identify those transient intervals where causal inference should be applied. Second, we propose to use fine-grained facial expressions that are present when strong distinct facial emotions are not visible. To specify the direction of influence, we apply the concept of Granger causality to the time-series of facial expressions over selected relevant intervals. We tested our approach on newly, experimentally obtained data. Based on quantitative verification of hypotheses on the direction of emotional influence, we were able to show that the proposed approach is promising to reveal the cause-effect pattern in various instructed interaction conditions.https://ieeexplore.ieee.org/document/9425581/Direction of emotional influencenonverbal human communicationdyadic interactionsfacial action unitsfine-grained facial expressiongranger causality |
| spellingShingle | Maha Shadaydeh Lea Muller Dana Schneider Martin Thummel Thomas Kessler Joachim Denzler Analyzing the Direction of Emotional Influence in Nonverbal Dyadic Communication: A Facial-Expression Study IEEE Access Direction of emotional influence nonverbal human communication dyadic interactions facial action units fine-grained facial expression granger causality |
| title | Analyzing the Direction of Emotional Influence in Nonverbal Dyadic Communication: A Facial-Expression Study |
| title_full | Analyzing the Direction of Emotional Influence in Nonverbal Dyadic Communication: A Facial-Expression Study |
| title_fullStr | Analyzing the Direction of Emotional Influence in Nonverbal Dyadic Communication: A Facial-Expression Study |
| title_full_unstemmed | Analyzing the Direction of Emotional Influence in Nonverbal Dyadic Communication: A Facial-Expression Study |
| title_short | Analyzing the Direction of Emotional Influence in Nonverbal Dyadic Communication: A Facial-Expression Study |
| title_sort | analyzing the direction of emotional influence in nonverbal dyadic communication a facial expression study |
| topic | Direction of emotional influence nonverbal human communication dyadic interactions facial action units fine-grained facial expression granger causality |
| url | https://ieeexplore.ieee.org/document/9425581/ |
| work_keys_str_mv | AT mahashadaydeh analyzingthedirectionofemotionalinfluenceinnonverbaldyadiccommunicationafacialexpressionstudy AT leamuller analyzingthedirectionofemotionalinfluenceinnonverbaldyadiccommunicationafacialexpressionstudy AT danaschneider analyzingthedirectionofemotionalinfluenceinnonverbaldyadiccommunicationafacialexpressionstudy AT martinthummel analyzingthedirectionofemotionalinfluenceinnonverbaldyadiccommunicationafacialexpressionstudy AT thomaskessler analyzingthedirectionofemotionalinfluenceinnonverbaldyadiccommunicationafacialexpressionstudy AT joachimdenzler analyzingthedirectionofemotionalinfluenceinnonverbaldyadiccommunicationafacialexpressionstudy |