Crowdsourced privacy-preserved feature tagging of short home videos for machine learning ASD detection
Abstract Standard medical diagnosis of mental health conditions requires licensed experts who are increasingly outnumbered by those at risk, limiting reach. We test the hypothesis that a trustworthy crowd of non-experts can efficiently annotate behavioral features needed for accurate machine learnin...
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2021-04-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-021-87059-4 |
| _version_ | 1818858338705735680 |
|---|---|
| author | Peter Washington Qandeel Tariq Emilie Leblanc Brianna Chrisman Kaitlyn Dunlap Aaron Kline Haik Kalantarian Yordan Penev Kelley Paskov Catalin Voss Nathaniel Stockham Maya Varma Arman Husic Jack Kent Nick Haber Terry Winograd Dennis P. Wall |
| author_facet | Peter Washington Qandeel Tariq Emilie Leblanc Brianna Chrisman Kaitlyn Dunlap Aaron Kline Haik Kalantarian Yordan Penev Kelley Paskov Catalin Voss Nathaniel Stockham Maya Varma Arman Husic Jack Kent Nick Haber Terry Winograd Dennis P. Wall |
| author_sort | Peter Washington |
| collection | DOAJ |
| description | Abstract Standard medical diagnosis of mental health conditions requires licensed experts who are increasingly outnumbered by those at risk, limiting reach. We test the hypothesis that a trustworthy crowd of non-experts can efficiently annotate behavioral features needed for accurate machine learning detection of the common childhood developmental disorder Autism Spectrum Disorder (ASD) for children under 8 years old. We implement a novel process for identifying and certifying a trustworthy distributed workforce for video feature extraction, selecting a workforce of 102 workers from a pool of 1,107. Two previously validated ASD logistic regression classifiers, evaluated against parent-reported diagnoses, were used to assess the accuracy of the trusted crowd’s ratings of unstructured home videos. A representative balanced sample (N = 50 videos) of videos were evaluated with and without face box and pitch shift privacy alterations, with AUROC and AUPRC scores > 0.98. With both privacy-preserving modifications, sensitivity is preserved (96.0%) while maintaining specificity (80.0%) and accuracy (88.0%) at levels comparable to prior classification methods without alterations. We find that machine learning classification from features extracted by a certified nonexpert crowd achieves high performance for ASD detection from natural home videos of the child at risk and maintains high sensitivity when privacy-preserving mechanisms are applied. These results suggest that privacy-safeguarded crowdsourced analysis of short home videos can help enable rapid and mobile machine-learning detection of developmental delays in children. |
| first_indexed | 2024-12-19T08:54:42Z |
| format | Article |
| id | doaj.art-7c1e977803b04ec7833a0f2e6600ab08 |
| institution | Directory Open Access Journal |
| issn | 2045-2322 |
| language | English |
| last_indexed | 2024-12-19T08:54:42Z |
| publishDate | 2021-04-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj.art-7c1e977803b04ec7833a0f2e6600ab082022-12-21T20:28:38ZengNature PortfolioScientific Reports2045-23222021-04-0111111110.1038/s41598-021-87059-4Crowdsourced privacy-preserved feature tagging of short home videos for machine learning ASD detectionPeter Washington0Qandeel Tariq1Emilie Leblanc2Brianna Chrisman3Kaitlyn Dunlap4Aaron Kline5Haik Kalantarian6Yordan Penev7Kelley Paskov8Catalin Voss9Nathaniel Stockham10Maya Varma11Arman Husic12Jack Kent13Nick Haber14Terry Winograd15Dennis P. Wall16Department of Bioengineering, Stanford UniversityResearch ScientistDepartment of Pediatrics (Systems Medicine), Stanford UniversityDepartment of Bioengineering, Stanford UniversityDepartment of Pediatrics (Systems Medicine), Stanford UniversityDepartment of Pediatrics (Systems Medicine), Stanford UniversityDepartment of Pediatrics (Systems Medicine), Stanford UniversityDepartment of Pediatrics (Systems Medicine), Stanford UniversityDepartment of Biomedical Data Science, Stanford UniversityDepartment of Computer Science, Stanford UniversityDepartment of Neuroscience, Stanford UniversityDepartment of Computer Science, Stanford UniversityDepartment of Pediatrics (Systems Medicine), Stanford UniversityDepartment of Pediatrics (Systems Medicine), Stanford UniversityGraduate School of Education, Stanford UniversityDepartment of Computer Science, Stanford UniversityDepartment of Pediatrics (Systems Medicine), Stanford UniversityAbstract Standard medical diagnosis of mental health conditions requires licensed experts who are increasingly outnumbered by those at risk, limiting reach. We test the hypothesis that a trustworthy crowd of non-experts can efficiently annotate behavioral features needed for accurate machine learning detection of the common childhood developmental disorder Autism Spectrum Disorder (ASD) for children under 8 years old. We implement a novel process for identifying and certifying a trustworthy distributed workforce for video feature extraction, selecting a workforce of 102 workers from a pool of 1,107. Two previously validated ASD logistic regression classifiers, evaluated against parent-reported diagnoses, were used to assess the accuracy of the trusted crowd’s ratings of unstructured home videos. A representative balanced sample (N = 50 videos) of videos were evaluated with and without face box and pitch shift privacy alterations, with AUROC and AUPRC scores > 0.98. With both privacy-preserving modifications, sensitivity is preserved (96.0%) while maintaining specificity (80.0%) and accuracy (88.0%) at levels comparable to prior classification methods without alterations. We find that machine learning classification from features extracted by a certified nonexpert crowd achieves high performance for ASD detection from natural home videos of the child at risk and maintains high sensitivity when privacy-preserving mechanisms are applied. These results suggest that privacy-safeguarded crowdsourced analysis of short home videos can help enable rapid and mobile machine-learning detection of developmental delays in children.https://doi.org/10.1038/s41598-021-87059-4 |
| spellingShingle | Peter Washington Qandeel Tariq Emilie Leblanc Brianna Chrisman Kaitlyn Dunlap Aaron Kline Haik Kalantarian Yordan Penev Kelley Paskov Catalin Voss Nathaniel Stockham Maya Varma Arman Husic Jack Kent Nick Haber Terry Winograd Dennis P. Wall Crowdsourced privacy-preserved feature tagging of short home videos for machine learning ASD detection Scientific Reports |
| title | Crowdsourced privacy-preserved feature tagging of short home videos for machine learning ASD detection |
| title_full | Crowdsourced privacy-preserved feature tagging of short home videos for machine learning ASD detection |
| title_fullStr | Crowdsourced privacy-preserved feature tagging of short home videos for machine learning ASD detection |
| title_full_unstemmed | Crowdsourced privacy-preserved feature tagging of short home videos for machine learning ASD detection |
| title_short | Crowdsourced privacy-preserved feature tagging of short home videos for machine learning ASD detection |
| title_sort | crowdsourced privacy preserved feature tagging of short home videos for machine learning asd detection |
| url | https://doi.org/10.1038/s41598-021-87059-4 |
| work_keys_str_mv | AT peterwashington crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT qandeeltariq crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT emilieleblanc crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT briannachrisman crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT kaitlyndunlap crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT aaronkline crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT haikkalantarian crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT yordanpenev crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT kelleypaskov crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT catalinvoss crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT nathanielstockham crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT mayavarma crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT armanhusic crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT jackkent crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT nickhaber crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT terrywinograd crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection AT dennispwall crowdsourcedprivacypreservedfeaturetaggingofshorthomevideosformachinelearningasddetection |