Evaluating Web-Based Automatic Transcription for Alzheimer Speech Data: Transcript Comparison and Machine Learning Analysis
BackgroundSpeech data for medical research can be collected noninvasively and in large volumes. Speech analysis has shown promise in diagnosing neurodegenerative disease. To effectively leverage speech data, transcription is important, as there is valuable information contain...
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2022-09-01
|
| Series: | JMIR Aging |
| Online Access: | https://aging.jmir.org/2022/3/e33460 |
| _version_ | 1827857958243926016 |
|---|---|
| author | Thomas Soroski Thiago da Cunha Vasco Sally Newton-Mason Saffrin Granby Caitlin Lewis Anuj Harisinghani Matteo Rizzo Cristina Conati Gabriel Murray Giuseppe Carenini Thalia S Field Hyeju Jang |
| author_facet | Thomas Soroski Thiago da Cunha Vasco Sally Newton-Mason Saffrin Granby Caitlin Lewis Anuj Harisinghani Matteo Rizzo Cristina Conati Gabriel Murray Giuseppe Carenini Thalia S Field Hyeju Jang |
| author_sort | Thomas Soroski |
| collection | DOAJ |
| description |
BackgroundSpeech data for medical research can be collected noninvasively and in large volumes. Speech analysis has shown promise in diagnosing neurodegenerative disease. To effectively leverage speech data, transcription is important, as there is valuable information contained in lexical content. Manual transcription, while highly accurate, limits the potential scalability and cost savings associated with language-based screening.
ObjectiveTo better understand the use of automatic transcription for classification of neurodegenerative disease, namely, Alzheimer disease (AD), mild cognitive impairment (MCI), or subjective memory complaints (SMC) versus healthy controls, we compared automatically generated transcripts against transcripts that went through manual correction.
MethodsWe recruited individuals from a memory clinic (“patients”) with a diagnosis of mild-to-moderate AD, (n=44, 30%), MCI (n=20, 13%), SMC (n=8, 5%), as well as healthy controls (n=77, 52%) living in the community. Participants were asked to describe a standardized picture, read a paragraph, and recall a pleasant life experience. We compared transcripts generated using Google speech-to-text software to manually verified transcripts by examining transcription confidence scores, transcription error rates, and machine learning classification accuracy. For the classification tasks, logistic regression, Gaussian naive Bayes, and random forests were used.
ResultsThe transcription software showed higher confidence scores (P<.001) and lower error rates (P>.05) for speech from healthy controls compared with patients. Classification models using human-verified transcripts significantly (P<.001) outperformed automatically generated transcript models for both spontaneous speech tasks. This comparison showed no difference in the reading task. Manually adding pauses to transcripts had no impact on classification performance. However, manually correcting both spontaneous speech tasks led to significantly higher performances in the machine learning models.
ConclusionsWe found that automatically transcribed speech data could be used to distinguish patients with a diagnosis of AD, MCI, or SMC from controls. We recommend a human verification step to improve the performance of automatic transcripts, especially for spontaneous tasks. Moreover, human verification can focus on correcting errors and adding punctuation to transcripts. However, manual addition of pauses is not needed, which can simplify the human verification step to more efficiently process large volumes of speech data. |
| first_indexed | 2024-03-12T12:48:58Z |
| format | Article |
| id | doaj.art-d8dcd6819cb84036ba99adf78894fa49 |
| institution | Directory Open Access Journal |
| issn | 2561-7605 |
| language | English |
| last_indexed | 2024-03-12T12:48:58Z |
| publishDate | 2022-09-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Aging |
| spelling | doaj.art-d8dcd6819cb84036ba99adf78894fa492023-08-28T23:06:55ZengJMIR PublicationsJMIR Aging2561-76052022-09-0153e3346010.2196/33460Evaluating Web-Based Automatic Transcription for Alzheimer Speech Data: Transcript Comparison and Machine Learning AnalysisThomas Soroskihttps://orcid.org/0000-0002-2847-0990Thiago da Cunha Vascohttps://orcid.org/0000-0002-6709-0456Sally Newton-Masonhttps://orcid.org/0000-0003-3390-8253Saffrin Granbyhttps://orcid.org/0000-0002-9995-6357Caitlin Lewishttps://orcid.org/0000-0002-5886-5002Anuj Harisinghanihttps://orcid.org/0000-0001-7206-4888Matteo Rizzohttps://orcid.org/0000-0002-9723-211XCristina Conatihttps://orcid.org/0000-0002-8434-9335Gabriel Murrayhttps://orcid.org/0000-0003-2233-6288Giuseppe Careninihttps://orcid.org/0000-0003-4310-0119Thalia S Fieldhttps://orcid.org/0000-0002-1176-0633Hyeju Janghttps://orcid.org/0000-0002-7652-1158 BackgroundSpeech data for medical research can be collected noninvasively and in large volumes. Speech analysis has shown promise in diagnosing neurodegenerative disease. To effectively leverage speech data, transcription is important, as there is valuable information contained in lexical content. Manual transcription, while highly accurate, limits the potential scalability and cost savings associated with language-based screening. ObjectiveTo better understand the use of automatic transcription for classification of neurodegenerative disease, namely, Alzheimer disease (AD), mild cognitive impairment (MCI), or subjective memory complaints (SMC) versus healthy controls, we compared automatically generated transcripts against transcripts that went through manual correction. MethodsWe recruited individuals from a memory clinic (“patients”) with a diagnosis of mild-to-moderate AD, (n=44, 30%), MCI (n=20, 13%), SMC (n=8, 5%), as well as healthy controls (n=77, 52%) living in the community. Participants were asked to describe a standardized picture, read a paragraph, and recall a pleasant life experience. We compared transcripts generated using Google speech-to-text software to manually verified transcripts by examining transcription confidence scores, transcription error rates, and machine learning classification accuracy. For the classification tasks, logistic regression, Gaussian naive Bayes, and random forests were used. ResultsThe transcription software showed higher confidence scores (P<.001) and lower error rates (P>.05) for speech from healthy controls compared with patients. Classification models using human-verified transcripts significantly (P<.001) outperformed automatically generated transcript models for both spontaneous speech tasks. This comparison showed no difference in the reading task. Manually adding pauses to transcripts had no impact on classification performance. However, manually correcting both spontaneous speech tasks led to significantly higher performances in the machine learning models. ConclusionsWe found that automatically transcribed speech data could be used to distinguish patients with a diagnosis of AD, MCI, or SMC from controls. We recommend a human verification step to improve the performance of automatic transcripts, especially for spontaneous tasks. Moreover, human verification can focus on correcting errors and adding punctuation to transcripts. However, manual addition of pauses is not needed, which can simplify the human verification step to more efficiently process large volumes of speech data.https://aging.jmir.org/2022/3/e33460 |
| spellingShingle | Thomas Soroski Thiago da Cunha Vasco Sally Newton-Mason Saffrin Granby Caitlin Lewis Anuj Harisinghani Matteo Rizzo Cristina Conati Gabriel Murray Giuseppe Carenini Thalia S Field Hyeju Jang Evaluating Web-Based Automatic Transcription for Alzheimer Speech Data: Transcript Comparison and Machine Learning Analysis JMIR Aging |
| title | Evaluating Web-Based Automatic Transcription for Alzheimer Speech Data: Transcript Comparison and Machine Learning Analysis |
| title_full | Evaluating Web-Based Automatic Transcription for Alzheimer Speech Data: Transcript Comparison and Machine Learning Analysis |
| title_fullStr | Evaluating Web-Based Automatic Transcription for Alzheimer Speech Data: Transcript Comparison and Machine Learning Analysis |
| title_full_unstemmed | Evaluating Web-Based Automatic Transcription for Alzheimer Speech Data: Transcript Comparison and Machine Learning Analysis |
| title_short | Evaluating Web-Based Automatic Transcription for Alzheimer Speech Data: Transcript Comparison and Machine Learning Analysis |
| title_sort | evaluating web based automatic transcription for alzheimer speech data transcript comparison and machine learning analysis |
| url | https://aging.jmir.org/2022/3/e33460 |
| work_keys_str_mv | AT thomassoroski evaluatingwebbasedautomatictranscriptionforalzheimerspeechdatatranscriptcomparisonandmachinelearninganalysis AT thiagodacunhavasco evaluatingwebbasedautomatictranscriptionforalzheimerspeechdatatranscriptcomparisonandmachinelearninganalysis AT sallynewtonmason evaluatingwebbasedautomatictranscriptionforalzheimerspeechdatatranscriptcomparisonandmachinelearninganalysis AT saffringranby evaluatingwebbasedautomatictranscriptionforalzheimerspeechdatatranscriptcomparisonandmachinelearninganalysis AT caitlinlewis evaluatingwebbasedautomatictranscriptionforalzheimerspeechdatatranscriptcomparisonandmachinelearninganalysis AT anujharisinghani evaluatingwebbasedautomatictranscriptionforalzheimerspeechdatatranscriptcomparisonandmachinelearninganalysis AT matteorizzo evaluatingwebbasedautomatictranscriptionforalzheimerspeechdatatranscriptcomparisonandmachinelearninganalysis AT cristinaconati evaluatingwebbasedautomatictranscriptionforalzheimerspeechdatatranscriptcomparisonandmachinelearninganalysis AT gabrielmurray evaluatingwebbasedautomatictranscriptionforalzheimerspeechdatatranscriptcomparisonandmachinelearninganalysis AT giuseppecarenini evaluatingwebbasedautomatictranscriptionforalzheimerspeechdatatranscriptcomparisonandmachinelearninganalysis AT thaliasfield evaluatingwebbasedautomatictranscriptionforalzheimerspeechdatatranscriptcomparisonandmachinelearninganalysis AT hyejujang evaluatingwebbasedautomatictranscriptionforalzheimerspeechdatatranscriptcomparisonandmachinelearninganalysis |