Using HIPAA (Health Insurance Portability and Accountability Act)–Compliant Transcription Services for Virtual Psychiatric Interviews: Pilot Comparison Study
BackgroundAutomatic speech recognition (ASR) technology is increasingly being used for transcription in clinical contexts. Although there are numerous transcription services using ASR, few studies have compared the word error rate (WER) between different transcription service...
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2023-10-01
|
| Series: | JMIR Mental Health |
| Online Access: | https://mental.jmir.org/2023/1/e48517 |
| _version_ | 1827778086517604352 |
|---|---|
| author | Salman Seyedi Emily Griner Lisette Corbin Zifan Jiang Kailey Roberts Luca Iacobelli Aaron Milloy Mina Boazak Ali Bahrami Rad Ahmed Abbasi Robert O Cotes Gari D Clifford |
| author_facet | Salman Seyedi Emily Griner Lisette Corbin Zifan Jiang Kailey Roberts Luca Iacobelli Aaron Milloy Mina Boazak Ali Bahrami Rad Ahmed Abbasi Robert O Cotes Gari D Clifford |
| author_sort | Salman Seyedi |
| collection | DOAJ |
| description |
BackgroundAutomatic speech recognition (ASR) technology is increasingly being used for transcription in clinical contexts. Although there are numerous transcription services using ASR, few studies have compared the word error rate (WER) between different transcription services among different diagnostic groups in a mental health setting. There has also been little research into the types of words ASR transcriptions mistakenly generate or omit.
ObjectiveThis study compared the WER of 3 ASR transcription services (Amazon Transcribe [Amazon.com, Inc], Zoom-Otter AI [Zoom Video Communications, Inc], and Whisper [OpenAI Inc]) in interviews across 2 different clinical categories (controls and participants experiencing a variety of mental health conditions). These ASR transcription services were also compared with a commercial human transcription service, Rev (Rev.Com, Inc). Words that were either included or excluded by the error in the transcripts were systematically analyzed by their Linguistic Inquiry and Word Count categories.
MethodsParticipants completed a 1-time research psychiatric interview, which was recorded on a secure server. Transcriptions created by the research team were used as the gold standard from which WER was calculated. The interviewees were categorized into either the control group (n=18) or the mental health condition group (n=47) using the Mini-International Neuropsychiatric Interview. The total sample included 65 participants. Brunner-Munzel tests were used for comparing independent sets, such as the diagnostic groupings, and Wilcoxon signed rank tests were used for correlated samples when comparing the total sample between different transcription services.
ResultsThere were significant differences between each ASR transcription service’s WER (P<.001). Amazon Transcribe’s output exhibited significantly lower WERs compared with the Zoom-Otter AI’s and Whisper’s ASR. ASR performances did not significantly differ across the 2 different clinical categories within each service (P>.05). A comparison between the human transcription service output from Rev and the best-performing ASR (Amazon Transcribe) demonstrated a significant difference (P<.001), with Rev having a slightly lower median WER (7.6%, IQR 5.4%-11.35 vs 8.9%, IQR 6.9%-11.6%). Heat maps and spider plots were used to visualize the most common errors in Linguistic Inquiry and Word Count categories, which were found to be within 3 overarching categories: Conversation, Cognition, and Function.
ConclusionsOverall, consistent with previous literature, our results suggest that the WER between manual and automated transcription services may be narrowing as ASR services advance. These advances, coupled with decreased cost and time in receiving transcriptions, may make ASR transcriptions a more viable option within health care settings. However, more research is required to determine if errors in specific types of words impact the analysis and usability of these transcriptions, particularly for specific applications and in a variety of populations in terms of clinical diagnosis, literacy level, accent, and cultural origin. |
| first_indexed | 2024-03-11T14:27:10Z |
| format | Article |
| id | doaj.art-6235d8e01f9a49558c2566fbf9c4091e |
| institution | Directory Open Access Journal |
| issn | 2368-7959 |
| language | English |
| last_indexed | 2024-03-11T14:27:10Z |
| publishDate | 2023-10-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Mental Health |
| spelling | doaj.art-6235d8e01f9a49558c2566fbf9c4091e2023-10-31T14:01:03ZengJMIR PublicationsJMIR Mental Health2368-79592023-10-0110e4851710.2196/48517Using HIPAA (Health Insurance Portability and Accountability Act)–Compliant Transcription Services for Virtual Psychiatric Interviews: Pilot Comparison StudySalman Seyedihttps://orcid.org/0000-0002-6017-7049Emily Grinerhttps://orcid.org/0000-0001-9161-8874Lisette Corbinhttps://orcid.org/0000-0001-9382-0750Zifan Jianghttps://orcid.org/0000-0002-3570-9461Kailey Robertshttps://orcid.org/0000-0002-6076-3654Luca Iacobellihttps://orcid.org/0000-0002-6514-4465Aaron Milloyhttps://orcid.org/0009-0000-9467-3326Mina Boazakhttps://orcid.org/0000-0002-3908-5887Ali Bahrami Radhttps://orcid.org/0000-0002-5654-4301Ahmed Abbasihttps://orcid.org/0000-0001-7698-7794Robert O Coteshttps://orcid.org/0000-0001-9903-8807Gari D Cliffordhttps://orcid.org/0000-0002-5709-201X BackgroundAutomatic speech recognition (ASR) technology is increasingly being used for transcription in clinical contexts. Although there are numerous transcription services using ASR, few studies have compared the word error rate (WER) between different transcription services among different diagnostic groups in a mental health setting. There has also been little research into the types of words ASR transcriptions mistakenly generate or omit. ObjectiveThis study compared the WER of 3 ASR transcription services (Amazon Transcribe [Amazon.com, Inc], Zoom-Otter AI [Zoom Video Communications, Inc], and Whisper [OpenAI Inc]) in interviews across 2 different clinical categories (controls and participants experiencing a variety of mental health conditions). These ASR transcription services were also compared with a commercial human transcription service, Rev (Rev.Com, Inc). Words that were either included or excluded by the error in the transcripts were systematically analyzed by their Linguistic Inquiry and Word Count categories. MethodsParticipants completed a 1-time research psychiatric interview, which was recorded on a secure server. Transcriptions created by the research team were used as the gold standard from which WER was calculated. The interviewees were categorized into either the control group (n=18) or the mental health condition group (n=47) using the Mini-International Neuropsychiatric Interview. The total sample included 65 participants. Brunner-Munzel tests were used for comparing independent sets, such as the diagnostic groupings, and Wilcoxon signed rank tests were used for correlated samples when comparing the total sample between different transcription services. ResultsThere were significant differences between each ASR transcription service’s WER (P<.001). Amazon Transcribe’s output exhibited significantly lower WERs compared with the Zoom-Otter AI’s and Whisper’s ASR. ASR performances did not significantly differ across the 2 different clinical categories within each service (P>.05). A comparison between the human transcription service output from Rev and the best-performing ASR (Amazon Transcribe) demonstrated a significant difference (P<.001), with Rev having a slightly lower median WER (7.6%, IQR 5.4%-11.35 vs 8.9%, IQR 6.9%-11.6%). Heat maps and spider plots were used to visualize the most common errors in Linguistic Inquiry and Word Count categories, which were found to be within 3 overarching categories: Conversation, Cognition, and Function. ConclusionsOverall, consistent with previous literature, our results suggest that the WER between manual and automated transcription services may be narrowing as ASR services advance. These advances, coupled with decreased cost and time in receiving transcriptions, may make ASR transcriptions a more viable option within health care settings. However, more research is required to determine if errors in specific types of words impact the analysis and usability of these transcriptions, particularly for specific applications and in a variety of populations in terms of clinical diagnosis, literacy level, accent, and cultural origin.https://mental.jmir.org/2023/1/e48517 |
| spellingShingle | Salman Seyedi Emily Griner Lisette Corbin Zifan Jiang Kailey Roberts Luca Iacobelli Aaron Milloy Mina Boazak Ali Bahrami Rad Ahmed Abbasi Robert O Cotes Gari D Clifford Using HIPAA (Health Insurance Portability and Accountability Act)–Compliant Transcription Services for Virtual Psychiatric Interviews: Pilot Comparison Study JMIR Mental Health |
| title | Using HIPAA (Health Insurance Portability and Accountability Act)–Compliant Transcription Services for Virtual Psychiatric Interviews: Pilot Comparison Study |
| title_full | Using HIPAA (Health Insurance Portability and Accountability Act)–Compliant Transcription Services for Virtual Psychiatric Interviews: Pilot Comparison Study |
| title_fullStr | Using HIPAA (Health Insurance Portability and Accountability Act)–Compliant Transcription Services for Virtual Psychiatric Interviews: Pilot Comparison Study |
| title_full_unstemmed | Using HIPAA (Health Insurance Portability and Accountability Act)–Compliant Transcription Services for Virtual Psychiatric Interviews: Pilot Comparison Study |
| title_short | Using HIPAA (Health Insurance Portability and Accountability Act)–Compliant Transcription Services for Virtual Psychiatric Interviews: Pilot Comparison Study |
| title_sort | using hipaa health insurance portability and accountability act compliant transcription services for virtual psychiatric interviews pilot comparison study |
| url | https://mental.jmir.org/2023/1/e48517 |
| work_keys_str_mv | AT salmanseyedi usinghipaahealthinsuranceportabilityandaccountabilityactcomplianttranscriptionservicesforvirtualpsychiatricinterviewspilotcomparisonstudy AT emilygriner usinghipaahealthinsuranceportabilityandaccountabilityactcomplianttranscriptionservicesforvirtualpsychiatricinterviewspilotcomparisonstudy AT lisettecorbin usinghipaahealthinsuranceportabilityandaccountabilityactcomplianttranscriptionservicesforvirtualpsychiatricinterviewspilotcomparisonstudy AT zifanjiang usinghipaahealthinsuranceportabilityandaccountabilityactcomplianttranscriptionservicesforvirtualpsychiatricinterviewspilotcomparisonstudy AT kaileyroberts usinghipaahealthinsuranceportabilityandaccountabilityactcomplianttranscriptionservicesforvirtualpsychiatricinterviewspilotcomparisonstudy AT lucaiacobelli usinghipaahealthinsuranceportabilityandaccountabilityactcomplianttranscriptionservicesforvirtualpsychiatricinterviewspilotcomparisonstudy AT aaronmilloy usinghipaahealthinsuranceportabilityandaccountabilityactcomplianttranscriptionservicesforvirtualpsychiatricinterviewspilotcomparisonstudy AT minaboazak usinghipaahealthinsuranceportabilityandaccountabilityactcomplianttranscriptionservicesforvirtualpsychiatricinterviewspilotcomparisonstudy AT alibahramirad usinghipaahealthinsuranceportabilityandaccountabilityactcomplianttranscriptionservicesforvirtualpsychiatricinterviewspilotcomparisonstudy AT ahmedabbasi usinghipaahealthinsuranceportabilityandaccountabilityactcomplianttranscriptionservicesforvirtualpsychiatricinterviewspilotcomparisonstudy AT robertocotes usinghipaahealthinsuranceportabilityandaccountabilityactcomplianttranscriptionservicesforvirtualpsychiatricinterviewspilotcomparisonstudy AT garidclifford usinghipaahealthinsuranceportabilityandaccountabilityactcomplianttranscriptionservicesforvirtualpsychiatricinterviewspilotcomparisonstudy |