CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism
© 2018 IEEE. Many children on autism spectrum have atypical behavioral expressions of engagement compared to their neu-rotypical peers. In this paper, we investigate the performance of deep learning models in the task of automated engagement estimation from face images of children with autism. Speci...
| Main Authors: | , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Institute of Electrical and Electronics Engineers (IEEE)
2021
|
| Online Access: | https://hdl.handle.net/1721.1/137990 |
| _version_ | 1811075835648540672 |
|---|---|
| author | Rudovic, Ognjen Utsumi, Yuria Lee, Jaeryoung Hernandez, Javier Ferrer, Eduardo Castello Schuller, Bjorn Picard, Rosalind W. |
| author2 | Massachusetts Institute of Technology. Media Laboratory |
| author_facet | Massachusetts Institute of Technology. Media Laboratory Rudovic, Ognjen Utsumi, Yuria Lee, Jaeryoung Hernandez, Javier Ferrer, Eduardo Castello Schuller, Bjorn Picard, Rosalind W. |
| author_sort | Rudovic, Ognjen |
| collection | MIT |
| description | © 2018 IEEE. Many children on autism spectrum have atypical behavioral expressions of engagement compared to their neu-rotypical peers. In this paper, we investigate the performance of deep learning models in the task of automated engagement estimation from face images of children with autism. Specifically, we use the video data of 30 children with different cultural backgrounds (Asia vs. Europe) recorded during a single session of a robot-assisted autism therapy. We perform a thorough evaluation of the proposed deep architectures for the target task, including within- and across-culture evaluations, as well as when using the child-independent and child-dependent settings. We also introduce a novel deep learning model, named CultureNet, which efficiently leverages the multi-cultural data when performing the adaptation of the proposed deep architecture to the target culture and child. We show that due to the highly heterogeneous nature of the image data of children with autism, the child-independent models lead to overall poor estimation of target engagement levels. On the other hand, when a small amount of data of target children is used to enhance the model learning, the estimation performance on the held-out data from those children increases significantly. This is the first time that the effects of individual and cultural differences in children with autism have empirically been studied in the context of deep learning performed directly from face images. |
| first_indexed | 2024-09-23T10:12:30Z |
| format | Article |
| id | mit-1721.1/137990 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T10:12:30Z |
| publishDate | 2021 |
| publisher | Institute of Electrical and Electronics Engineers (IEEE) |
| record_format | dspace |
| spelling | mit-1721.1/1379902024-08-09T19:13:15Z CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism Rudovic, Ognjen Utsumi, Yuria Lee, Jaeryoung Hernandez, Javier Ferrer, Eduardo Castello Schuller, Bjorn Picard, Rosalind W. Massachusetts Institute of Technology. Media Laboratory Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science © 2018 IEEE. Many children on autism spectrum have atypical behavioral expressions of engagement compared to their neu-rotypical peers. In this paper, we investigate the performance of deep learning models in the task of automated engagement estimation from face images of children with autism. Specifically, we use the video data of 30 children with different cultural backgrounds (Asia vs. Europe) recorded during a single session of a robot-assisted autism therapy. We perform a thorough evaluation of the proposed deep architectures for the target task, including within- and across-culture evaluations, as well as when using the child-independent and child-dependent settings. We also introduce a novel deep learning model, named CultureNet, which efficiently leverages the multi-cultural data when performing the adaptation of the proposed deep architecture to the target culture and child. We show that due to the highly heterogeneous nature of the image data of children with autism, the child-independent models lead to overall poor estimation of target engagement levels. On the other hand, when a small amount of data of target children is used to enhance the model learning, the estimation performance on the held-out data from those children increases significantly. This is the first time that the effects of individual and cultural differences in children with autism have empirically been studied in the context of deep learning performed directly from face images. 2021-11-09T17:22:48Z 2021-11-09T17:22:48Z 2018-10 2019-07-31T18:43:24Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/137990 Rudovic, Ognjen, Utsumi, Yuria, Lee, Jaeryoung, Hernandez, Javier, Ferrer, Eduardo Castello et al. 2018. "CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism." en 10.1109/iros.2018.8594177 Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf Institute of Electrical and Electronics Engineers (IEEE) MIT web domain |
| spellingShingle | Rudovic, Ognjen Utsumi, Yuria Lee, Jaeryoung Hernandez, Javier Ferrer, Eduardo Castello Schuller, Bjorn Picard, Rosalind W. CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism |
| title | CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism |
| title_full | CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism |
| title_fullStr | CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism |
| title_full_unstemmed | CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism |
| title_short | CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism |
| title_sort | culturenet a deep learning approach for engagement intensity estimation from face images of children with autism |
| url | https://hdl.handle.net/1721.1/137990 |
| work_keys_str_mv | AT rudovicognjen culturenetadeeplearningapproachforengagementintensityestimationfromfaceimagesofchildrenwithautism AT utsumiyuria culturenetadeeplearningapproachforengagementintensityestimationfromfaceimagesofchildrenwithautism AT leejaeryoung culturenetadeeplearningapproachforengagementintensityestimationfromfaceimagesofchildrenwithautism AT hernandezjavier culturenetadeeplearningapproachforengagementintensityestimationfromfaceimagesofchildrenwithautism AT ferrereduardocastello culturenetadeeplearningapproachforengagementintensityestimationfromfaceimagesofchildrenwithautism AT schullerbjorn culturenetadeeplearningapproachforengagementintensityestimationfromfaceimagesofchildrenwithautism AT picardrosalindw culturenetadeeplearningapproachforengagementintensityestimationfromfaceimagesofchildrenwithautism |