Evaluating Conversational Agents for Mental Health: Scoping Review of Outcomes and Outcome Measurement Instruments
BackgroundRapid proliferation of mental health interventions delivered through conversational agents (CAs) calls for high-quality evidence to support their implementation and adoption. Selecting appropriate outcomes, instruments for measuring outcomes, and assessment methods...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2023-04-01
|
| Series: | Journal of Medical Internet Research |
| Online Access: | https://www.jmir.org/2023/1/e44548 |
| _version_ | 1797734183227883520 |
|---|---|
| author | Ahmad Ishqi Jabir Laura Martinengo Xiaowen Lin John Torous Mythily Subramaniam Lorainne Tudor Car |
| author_facet | Ahmad Ishqi Jabir Laura Martinengo Xiaowen Lin John Torous Mythily Subramaniam Lorainne Tudor Car |
| author_sort | Ahmad Ishqi Jabir |
| collection | DOAJ |
| description |
BackgroundRapid proliferation of mental health interventions delivered through conversational agents (CAs) calls for high-quality evidence to support their implementation and adoption. Selecting appropriate outcomes, instruments for measuring outcomes, and assessment methods are crucial for ensuring that interventions are evaluated effectively and with a high level of quality.
ObjectiveWe aimed to identify the types of outcomes, outcome measurement instruments, and assessment methods used to assess the clinical, user experience, and technical outcomes in studies that evaluated the effectiveness of CA interventions for mental health.
MethodsWe undertook a scoping review of the relevant literature to review the types of outcomes, outcome measurement instruments, and assessment methods in studies that evaluated the effectiveness of CA interventions for mental health. We performed a comprehensive search of electronic databases, including PubMed, Cochrane Central Register of Controlled Trials, Embase (Ovid), PsychINFO, and Web of Science, as well as Google Scholar and Google. We included experimental studies evaluating CA mental health interventions. The screening and data extraction were performed independently by 2 review authors in parallel. Descriptive and thematic analyses of the findings were performed.
ResultsWe included 32 studies that targeted the promotion of mental well-being (17/32, 53%) and the treatment and monitoring of mental health symptoms (21/32, 66%). The studies reported 203 outcome measurement instruments used to measure clinical outcomes (123/203, 60.6%), user experience outcomes (75/203, 36.9%), technical outcomes (2/203, 1.0%), and other outcomes (3/203, 1.5%). Most of the outcome measurement instruments were used in only 1 study (150/203, 73.9%) and were self-reported questionnaires (170/203, 83.7%), and most were delivered electronically via survey platforms (61/203, 30.0%). No validity evidence was cited for more than half of the outcome measurement instruments (107/203, 52.7%), which were largely created or adapted for the study in which they were used (95/107, 88.8%).
ConclusionsThe diversity of outcomes and the choice of outcome measurement instruments employed in studies on CAs for mental health point to the need for an established minimum core outcome set and greater use of validated instruments. Future studies should also capitalize on the affordances made available by CAs and smartphones to streamline the evaluation and reduce participants’ input burden inherent to self-reporting. |
| first_indexed | 2024-03-12T12:39:36Z |
| format | Article |
| id | doaj.art-a7646d7d2620417f905ff6d3f12c06b2 |
| institution | Directory Open Access Journal |
| issn | 1438-8871 |
| language | English |
| last_indexed | 2024-03-12T12:39:36Z |
| publishDate | 2023-04-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | Journal of Medical Internet Research |
| spelling | doaj.art-a7646d7d2620417f905ff6d3f12c06b22023-08-28T23:55:23ZengJMIR PublicationsJournal of Medical Internet Research1438-88712023-04-0125e4454810.2196/44548Evaluating Conversational Agents for Mental Health: Scoping Review of Outcomes and Outcome Measurement InstrumentsAhmad Ishqi Jabirhttps://orcid.org/0000-0001-6864-9867Laura Martinengohttps://orcid.org/0000-0003-3539-7207Xiaowen Linhttps://orcid.org/0000-0001-7497-026XJohn Toroushttps://orcid.org/0000-0002-5362-7937Mythily Subramaniamhttps://orcid.org/0000-0003-4530-1096Lorainne Tudor Carhttps://orcid.org/0000-0001-8414-7664 BackgroundRapid proliferation of mental health interventions delivered through conversational agents (CAs) calls for high-quality evidence to support their implementation and adoption. Selecting appropriate outcomes, instruments for measuring outcomes, and assessment methods are crucial for ensuring that interventions are evaluated effectively and with a high level of quality. ObjectiveWe aimed to identify the types of outcomes, outcome measurement instruments, and assessment methods used to assess the clinical, user experience, and technical outcomes in studies that evaluated the effectiveness of CA interventions for mental health. MethodsWe undertook a scoping review of the relevant literature to review the types of outcomes, outcome measurement instruments, and assessment methods in studies that evaluated the effectiveness of CA interventions for mental health. We performed a comprehensive search of electronic databases, including PubMed, Cochrane Central Register of Controlled Trials, Embase (Ovid), PsychINFO, and Web of Science, as well as Google Scholar and Google. We included experimental studies evaluating CA mental health interventions. The screening and data extraction were performed independently by 2 review authors in parallel. Descriptive and thematic analyses of the findings were performed. ResultsWe included 32 studies that targeted the promotion of mental well-being (17/32, 53%) and the treatment and monitoring of mental health symptoms (21/32, 66%). The studies reported 203 outcome measurement instruments used to measure clinical outcomes (123/203, 60.6%), user experience outcomes (75/203, 36.9%), technical outcomes (2/203, 1.0%), and other outcomes (3/203, 1.5%). Most of the outcome measurement instruments were used in only 1 study (150/203, 73.9%) and were self-reported questionnaires (170/203, 83.7%), and most were delivered electronically via survey platforms (61/203, 30.0%). No validity evidence was cited for more than half of the outcome measurement instruments (107/203, 52.7%), which were largely created or adapted for the study in which they were used (95/107, 88.8%). ConclusionsThe diversity of outcomes and the choice of outcome measurement instruments employed in studies on CAs for mental health point to the need for an established minimum core outcome set and greater use of validated instruments. Future studies should also capitalize on the affordances made available by CAs and smartphones to streamline the evaluation and reduce participants’ input burden inherent to self-reporting.https://www.jmir.org/2023/1/e44548 |
| spellingShingle | Ahmad Ishqi Jabir Laura Martinengo Xiaowen Lin John Torous Mythily Subramaniam Lorainne Tudor Car Evaluating Conversational Agents for Mental Health: Scoping Review of Outcomes and Outcome Measurement Instruments Journal of Medical Internet Research |
| title | Evaluating Conversational Agents for Mental Health: Scoping Review of Outcomes and Outcome Measurement Instruments |
| title_full | Evaluating Conversational Agents for Mental Health: Scoping Review of Outcomes and Outcome Measurement Instruments |
| title_fullStr | Evaluating Conversational Agents for Mental Health: Scoping Review of Outcomes and Outcome Measurement Instruments |
| title_full_unstemmed | Evaluating Conversational Agents for Mental Health: Scoping Review of Outcomes and Outcome Measurement Instruments |
| title_short | Evaluating Conversational Agents for Mental Health: Scoping Review of Outcomes and Outcome Measurement Instruments |
| title_sort | evaluating conversational agents for mental health scoping review of outcomes and outcome measurement instruments |
| url | https://www.jmir.org/2023/1/e44548 |
| work_keys_str_mv | AT ahmadishqijabir evaluatingconversationalagentsformentalhealthscopingreviewofoutcomesandoutcomemeasurementinstruments AT lauramartinengo evaluatingconversationalagentsformentalhealthscopingreviewofoutcomesandoutcomemeasurementinstruments AT xiaowenlin evaluatingconversationalagentsformentalhealthscopingreviewofoutcomesandoutcomemeasurementinstruments AT johntorous evaluatingconversationalagentsformentalhealthscopingreviewofoutcomesandoutcomemeasurementinstruments AT mythilysubramaniam evaluatingconversationalagentsformentalhealthscopingreviewofoutcomesandoutcomemeasurementinstruments AT lorainnetudorcar evaluatingconversationalagentsformentalhealthscopingreviewofoutcomesandoutcomemeasurementinstruments |