The cross-race effect in automatic facial expression recognition violates measurement invariance
Emotion has been a subject undergoing intensive research in psychology and cognitive neuroscience over several decades. Recently, more and more studies of emotion have adopted automatic rather than manual methods of facial emotion recognition to analyze images or videos of human faces. Compared to m...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-12-01
|
| Series: | Frontiers in Psychology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1201145/full |
| _version_ | 1797374786191491072 |
|---|---|
| author | Yen-Ting Li Su-Ling Yeh Su-Ling Yeh Su-Ling Yeh Su-Ling Yeh Tsung-Ren Huang Tsung-Ren Huang Tsung-Ren Huang Tsung-Ren Huang |
| author_facet | Yen-Ting Li Su-Ling Yeh Su-Ling Yeh Su-Ling Yeh Su-Ling Yeh Tsung-Ren Huang Tsung-Ren Huang Tsung-Ren Huang Tsung-Ren Huang |
| author_sort | Yen-Ting Li |
| collection | DOAJ |
| description | Emotion has been a subject undergoing intensive research in psychology and cognitive neuroscience over several decades. Recently, more and more studies of emotion have adopted automatic rather than manual methods of facial emotion recognition to analyze images or videos of human faces. Compared to manual methods, these computer-vision-based, automatic methods can help objectively and rapidly analyze a large amount of data. These automatic methods have also been validated and believed to be accurate in their judgments. However, these automatic methods often rely on statistical learning models (e.g., deep neural networks), which are intrinsically inductive and thus suffer from problems of induction. Specifically, the models that were trained primarily on Western faces may not generalize well to accurately judge Eastern faces, which can then jeopardize the measurement invariance of emotions in cross-cultural studies. To demonstrate such a possibility, the present study carries out a cross-racial validation of two popular facial emotion recognition systems—FaceReader and DeepFace—using two Western and two Eastern face datasets. Although both systems could achieve overall high accuracies in the judgments of emotion category on the Western datasets, they performed relatively poorly on the Eastern datasets, especially in recognition of negative emotions. While these results caution the use of these automatic methods of emotion recognition on non-Western faces, the results also suggest that the measurements of happiness outputted by these automatic methods are accurate and invariant across races and hence can still be utilized for cross-cultural studies of positive psychology. |
| first_indexed | 2024-03-08T19:09:45Z |
| format | Article |
| id | doaj.art-835ce9d5252f48dd84e254aa1c32d99d |
| institution | Directory Open Access Journal |
| issn | 1664-1078 |
| language | English |
| last_indexed | 2024-03-08T19:09:45Z |
| publishDate | 2023-12-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Psychology |
| spelling | doaj.art-835ce9d5252f48dd84e254aa1c32d99d2023-12-27T17:20:24ZengFrontiers Media S.A.Frontiers in Psychology1664-10782023-12-011410.3389/fpsyg.2023.12011451201145The cross-race effect in automatic facial expression recognition violates measurement invarianceYen-Ting Li0Su-Ling Yeh1Su-Ling Yeh2Su-Ling Yeh3Su-Ling Yeh4Tsung-Ren Huang5Tsung-Ren Huang6Tsung-Ren Huang7Tsung-Ren Huang8Department of Psychology, National Taiwan University, Taipei City, TaiwanDepartment of Psychology, National Taiwan University, Taipei City, TaiwanGraduate Institute of Brain and Mind Sciences, National Taiwan University, Taipei City, TaiwanNeurobiology and Cognitive Science Center, National Taiwan University, Taipei City, TaiwanCenter for Artificial Intelligence and Advanced Robotics, National Taiwan University, Taipei City, TaiwanDepartment of Psychology, National Taiwan University, Taipei City, TaiwanGraduate Institute of Brain and Mind Sciences, National Taiwan University, Taipei City, TaiwanNeurobiology and Cognitive Science Center, National Taiwan University, Taipei City, TaiwanCenter for Artificial Intelligence and Advanced Robotics, National Taiwan University, Taipei City, TaiwanEmotion has been a subject undergoing intensive research in psychology and cognitive neuroscience over several decades. Recently, more and more studies of emotion have adopted automatic rather than manual methods of facial emotion recognition to analyze images or videos of human faces. Compared to manual methods, these computer-vision-based, automatic methods can help objectively and rapidly analyze a large amount of data. These automatic methods have also been validated and believed to be accurate in their judgments. However, these automatic methods often rely on statistical learning models (e.g., deep neural networks), which are intrinsically inductive and thus suffer from problems of induction. Specifically, the models that were trained primarily on Western faces may not generalize well to accurately judge Eastern faces, which can then jeopardize the measurement invariance of emotions in cross-cultural studies. To demonstrate such a possibility, the present study carries out a cross-racial validation of two popular facial emotion recognition systems—FaceReader and DeepFace—using two Western and two Eastern face datasets. Although both systems could achieve overall high accuracies in the judgments of emotion category on the Western datasets, they performed relatively poorly on the Eastern datasets, especially in recognition of negative emotions. While these results caution the use of these automatic methods of emotion recognition on non-Western faces, the results also suggest that the measurements of happiness outputted by these automatic methods are accurate and invariant across races and hence can still be utilized for cross-cultural studies of positive psychology.https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1201145/fullcross-cultural psychologycross-race effectemotion recognitionfacial expressioninductive biasmeasurement invariance |
| spellingShingle | Yen-Ting Li Su-Ling Yeh Su-Ling Yeh Su-Ling Yeh Su-Ling Yeh Tsung-Ren Huang Tsung-Ren Huang Tsung-Ren Huang Tsung-Ren Huang The cross-race effect in automatic facial expression recognition violates measurement invariance Frontiers in Psychology cross-cultural psychology cross-race effect emotion recognition facial expression inductive bias measurement invariance |
| title | The cross-race effect in automatic facial expression recognition violates measurement invariance |
| title_full | The cross-race effect in automatic facial expression recognition violates measurement invariance |
| title_fullStr | The cross-race effect in automatic facial expression recognition violates measurement invariance |
| title_full_unstemmed | The cross-race effect in automatic facial expression recognition violates measurement invariance |
| title_short | The cross-race effect in automatic facial expression recognition violates measurement invariance |
| title_sort | cross race effect in automatic facial expression recognition violates measurement invariance |
| topic | cross-cultural psychology cross-race effect emotion recognition facial expression inductive bias measurement invariance |
| url | https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1201145/full |
| work_keys_str_mv | AT yentingli thecrossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT sulingyeh thecrossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT sulingyeh thecrossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT sulingyeh thecrossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT sulingyeh thecrossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT tsungrenhuang thecrossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT tsungrenhuang thecrossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT tsungrenhuang thecrossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT tsungrenhuang thecrossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT yentingli crossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT sulingyeh crossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT sulingyeh crossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT sulingyeh crossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT sulingyeh crossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT tsungrenhuang crossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT tsungrenhuang crossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT tsungrenhuang crossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance AT tsungrenhuang crossraceeffectinautomaticfacialexpressionrecognitionviolatesmeasurementinvariance |