Computer Vision-Based Assessment of Autistic Children: Analyzing Interactions, Emotions, Human Pose, and Life Skills
In this paper, the proposed work implements and tests the computer vision applications to perform the skill and emotion assessment of children with Autism Spectrum Disorder (ASD) by extracting various bio-behaviors, human activities, child-therapist interactions, and joint pose estimations from the...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10105862/ |
| _version_ | 1797822218512629760 |
|---|---|
| author | Varun Ganjigunte Prakash Manu Kohli Swati Kohli A. P. Prathosh Tanu Wadhera Diptanshu Das Debasis Panigrahi John Vijay Sagar Kommu |
| author_facet | Varun Ganjigunte Prakash Manu Kohli Swati Kohli A. P. Prathosh Tanu Wadhera Diptanshu Das Debasis Panigrahi John Vijay Sagar Kommu |
| author_sort | Varun Ganjigunte Prakash |
| collection | DOAJ |
| description | In this paper, the proposed work implements and tests the computer vision applications to perform the skill and emotion assessment of children with Autism Spectrum Disorder (ASD) by extracting various bio-behaviors, human activities, child-therapist interactions, and joint pose estimations from the recorded videos of interactive single- or two-person play-based intervention sessions. A comprehensive data set of 300 videos is amassed from ASD children engaged in social interaction, and three novel deep learning-based vision models are developed, which are explained as follows: (i) activity comprehension to analyze child-play partner interactions (activity comprehension model); (ii) an automatic joint attention recognition framework using head and hand pose; and (iii) emotion and facial expression recognition. The proposed models are also tested on children’s real-world, 68 unseen videos captured from the clinic, and public datasets. The activity comprehension model has an overall accuracy of 72.32%, the joint attention recognition models have an accuracy of 97% for follow eye gaze and 93.4% for hand pointing, and the facial expression recognition model has an overall accuracy of 95.1%. The proposed models could extract behaviors of interest, events of activities, emotions, and social skills from free-play and intervention session videos of long duration and provide temporal plots for session monitoring and assessment, thus empowering clinicians with insightful data useful in diagnosis, assessment, treatment formulation, and monitoring ASD children with limited supervision. |
| first_indexed | 2024-03-13T10:04:44Z |
| format | Article |
| id | doaj.art-cd1dc25d704b413084bec82568757ff8 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-13T10:04:44Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-cd1dc25d704b413084bec82568757ff82023-05-22T23:00:34ZengIEEEIEEE Access2169-35362023-01-0111479074792910.1109/ACCESS.2023.326902710105862Computer Vision-Based Assessment of Autistic Children: Analyzing Interactions, Emotions, Human Pose, and Life SkillsVarun Ganjigunte Prakash0https://orcid.org/0000-0002-9356-7035Manu Kohli1Swati Kohli2A. P. Prathosh3https://orcid.org/0000-0002-8699-5760Tanu Wadhera4https://orcid.org/0000-0002-7646-2303Diptanshu Das5https://orcid.org/0000-0002-7221-5022Debasis Panigrahi6John Vijay Sagar Kommu7https://orcid.org/0000-0001-9044-2344CogniAble, Gurugram, Haryana, IndiaCogniAble, Gurugram, Haryana, IndiaCogniAble, Gurugram, Haryana, IndiaDepartment of Electrical Communication Engineering, Signal Processing Building West, Indian Institute of Science, Bengaluru, IndiaDepartment of Electronics and Communication Engineering, Indian Institute of Information Technology Una (IIITU), Una, Himachal Pradesh, IndiaInstitute of NeuroDevelopment, Kolkata, West Bengal, IndiaJagannath Hospital, Bhubaneswar, Odisha, IndiaDepartment of Child and Adolescent Psychiatry, National Institute of Mental Health and Neurosciences (NIMHANS), Bengaluru, Karnataka, IndiaIn this paper, the proposed work implements and tests the computer vision applications to perform the skill and emotion assessment of children with Autism Spectrum Disorder (ASD) by extracting various bio-behaviors, human activities, child-therapist interactions, and joint pose estimations from the recorded videos of interactive single- or two-person play-based intervention sessions. A comprehensive data set of 300 videos is amassed from ASD children engaged in social interaction, and three novel deep learning-based vision models are developed, which are explained as follows: (i) activity comprehension to analyze child-play partner interactions (activity comprehension model); (ii) an automatic joint attention recognition framework using head and hand pose; and (iii) emotion and facial expression recognition. The proposed models are also tested on children’s real-world, 68 unseen videos captured from the clinic, and public datasets. The activity comprehension model has an overall accuracy of 72.32%, the joint attention recognition models have an accuracy of 97% for follow eye gaze and 93.4% for hand pointing, and the facial expression recognition model has an overall accuracy of 95.1%. The proposed models could extract behaviors of interest, events of activities, emotions, and social skills from free-play and intervention session videos of long duration and provide temporal plots for session monitoring and assessment, thus empowering clinicians with insightful data useful in diagnosis, assessment, treatment formulation, and monitoring ASD children with limited supervision.https://ieeexplore.ieee.org/document/10105862/Autism spectrum disorderactivity comprehensionfacial expressionsjoint attentionASD screeningapplied behavior analysis |
| spellingShingle | Varun Ganjigunte Prakash Manu Kohli Swati Kohli A. P. Prathosh Tanu Wadhera Diptanshu Das Debasis Panigrahi John Vijay Sagar Kommu Computer Vision-Based Assessment of Autistic Children: Analyzing Interactions, Emotions, Human Pose, and Life Skills IEEE Access Autism spectrum disorder activity comprehension facial expressions joint attention ASD screening applied behavior analysis |
| title | Computer Vision-Based Assessment of Autistic Children: Analyzing Interactions, Emotions, Human Pose, and Life Skills |
| title_full | Computer Vision-Based Assessment of Autistic Children: Analyzing Interactions, Emotions, Human Pose, and Life Skills |
| title_fullStr | Computer Vision-Based Assessment of Autistic Children: Analyzing Interactions, Emotions, Human Pose, and Life Skills |
| title_full_unstemmed | Computer Vision-Based Assessment of Autistic Children: Analyzing Interactions, Emotions, Human Pose, and Life Skills |
| title_short | Computer Vision-Based Assessment of Autistic Children: Analyzing Interactions, Emotions, Human Pose, and Life Skills |
| title_sort | computer vision based assessment of autistic children analyzing interactions emotions human pose and life skills |
| topic | Autism spectrum disorder activity comprehension facial expressions joint attention ASD screening applied behavior analysis |
| url | https://ieeexplore.ieee.org/document/10105862/ |
| work_keys_str_mv | AT varunganjigunteprakash computervisionbasedassessmentofautisticchildrenanalyzinginteractionsemotionshumanposeandlifeskills AT manukohli computervisionbasedassessmentofautisticchildrenanalyzinginteractionsemotionshumanposeandlifeskills AT swatikohli computervisionbasedassessmentofautisticchildrenanalyzinginteractionsemotionshumanposeandlifeskills AT apprathosh computervisionbasedassessmentofautisticchildrenanalyzinginteractionsemotionshumanposeandlifeskills AT tanuwadhera computervisionbasedassessmentofautisticchildrenanalyzinginteractionsemotionshumanposeandlifeskills AT diptanshudas computervisionbasedassessmentofautisticchildrenanalyzinginteractionsemotionshumanposeandlifeskills AT debasispanigrahi computervisionbasedassessmentofautisticchildrenanalyzinginteractionsemotionshumanposeandlifeskills AT johnvijaysagarkommu computervisionbasedassessmentofautisticchildrenanalyzinginteractionsemotionshumanposeandlifeskills |