A review of the explainability and safety of conversational agents for mental health to identify avenues for improvement
Virtual Mental Health Assistants (VMHAs) continuously evolve to support the overloaded global healthcare system, which receives approximately 60 million primary care visits and 6 million emergency room visits annually. These systems, developed by clinical psychologists, psychiatrists, and AI researc...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-10-01
|
| Series: | Frontiers in Artificial Intelligence |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/frai.2023.1229805/full |
| _version_ | 1797661202227134464 |
|---|---|
| author | Surjodeep Sarkar Manas Gaur Lujie Karen Chen Muskan Garg Biplav Srivastava |
| author_facet | Surjodeep Sarkar Manas Gaur Lujie Karen Chen Muskan Garg Biplav Srivastava |
| author_sort | Surjodeep Sarkar |
| collection | DOAJ |
| description | Virtual Mental Health Assistants (VMHAs) continuously evolve to support the overloaded global healthcare system, which receives approximately 60 million primary care visits and 6 million emergency room visits annually. These systems, developed by clinical psychologists, psychiatrists, and AI researchers, are designed to aid in Cognitive Behavioral Therapy (CBT). The main focus of VMHAs is to provide relevant information to mental health professionals (MHPs) and engage in meaningful conversations to support individuals with mental health conditions. However, certain gaps prevent VMHAs from fully delivering on their promise during active communications. One of the gaps is their inability to explain their decisions to patients and MHPs, making conversations less trustworthy. Additionally, VMHAs can be vulnerable in providing unsafe responses to patient queries, further undermining their reliability. In this review, we assess the current state of VMHAs on the grounds of user-level explainability and safety, a set of desired properties for the broader adoption of VMHAs. This includes the examination of ChatGPT, a conversation agent developed on AI-driven models: GPT3.5 and GPT-4, that has been proposed for use in providing mental health services. By harnessing the collaborative and impactful contributions of AI, natural language processing, and the mental health professionals (MHPs) community, the review identifies opportunities for technological progress in VMHAs to ensure their capabilities include explainable and safe behaviors. It also emphasizes the importance of measures to guarantee that these advancements align with the promise of fostering trustworthy conversations. |
| first_indexed | 2024-03-11T18:41:55Z |
| format | Article |
| id | doaj.art-a06f9a6c7c884d6bb76ed716e9f1e2c7 |
| institution | Directory Open Access Journal |
| issn | 2624-8212 |
| language | English |
| last_indexed | 2024-03-11T18:41:55Z |
| publishDate | 2023-10-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Artificial Intelligence |
| spelling | doaj.art-a06f9a6c7c884d6bb76ed716e9f1e2c72023-10-12T12:46:18ZengFrontiers Media S.A.Frontiers in Artificial Intelligence2624-82122023-10-01610.3389/frai.2023.12298051229805A review of the explainability and safety of conversational agents for mental health to identify avenues for improvementSurjodeep Sarkar0Manas Gaur1Lujie Karen Chen2Muskan Garg3Biplav Srivastava4Department of Computer Science and Electrical Engineering, University of Maryland, Baltimore County, Baltimore, MD, United StatesDepartment of Computer Science and Electrical Engineering, University of Maryland, Baltimore County, Baltimore, MD, United StatesDepartment of Information Systems, University of Maryland, Baltimore County, Baltimore, MD, United StatesDepartment of AI & Informatics, Mayo Clinic, Rochester, MN, United StatesAI Institute, University of South Carolina, Columbia, SC, United StatesVirtual Mental Health Assistants (VMHAs) continuously evolve to support the overloaded global healthcare system, which receives approximately 60 million primary care visits and 6 million emergency room visits annually. These systems, developed by clinical psychologists, psychiatrists, and AI researchers, are designed to aid in Cognitive Behavioral Therapy (CBT). The main focus of VMHAs is to provide relevant information to mental health professionals (MHPs) and engage in meaningful conversations to support individuals with mental health conditions. However, certain gaps prevent VMHAs from fully delivering on their promise during active communications. One of the gaps is their inability to explain their decisions to patients and MHPs, making conversations less trustworthy. Additionally, VMHAs can be vulnerable in providing unsafe responses to patient queries, further undermining their reliability. In this review, we assess the current state of VMHAs on the grounds of user-level explainability and safety, a set of desired properties for the broader adoption of VMHAs. This includes the examination of ChatGPT, a conversation agent developed on AI-driven models: GPT3.5 and GPT-4, that has been proposed for use in providing mental health services. By harnessing the collaborative and impactful contributions of AI, natural language processing, and the mental health professionals (MHPs) community, the review identifies opportunities for technological progress in VMHAs to ensure their capabilities include explainable and safe behaviors. It also emphasizes the importance of measures to guarantee that these advancements align with the promise of fostering trustworthy conversations.https://www.frontiersin.org/articles/10.3389/frai.2023.1229805/fullexplainable AIsafetyconversational AIevaluation metricsknowledge-infused learningmental health |
| spellingShingle | Surjodeep Sarkar Manas Gaur Lujie Karen Chen Muskan Garg Biplav Srivastava A review of the explainability and safety of conversational agents for mental health to identify avenues for improvement Frontiers in Artificial Intelligence explainable AI safety conversational AI evaluation metrics knowledge-infused learning mental health |
| title | A review of the explainability and safety of conversational agents for mental health to identify avenues for improvement |
| title_full | A review of the explainability and safety of conversational agents for mental health to identify avenues for improvement |
| title_fullStr | A review of the explainability and safety of conversational agents for mental health to identify avenues for improvement |
| title_full_unstemmed | A review of the explainability and safety of conversational agents for mental health to identify avenues for improvement |
| title_short | A review of the explainability and safety of conversational agents for mental health to identify avenues for improvement |
| title_sort | review of the explainability and safety of conversational agents for mental health to identify avenues for improvement |
| topic | explainable AI safety conversational AI evaluation metrics knowledge-infused learning mental health |
| url | https://www.frontiersin.org/articles/10.3389/frai.2023.1229805/full |
| work_keys_str_mv | AT surjodeepsarkar areviewoftheexplainabilityandsafetyofconversationalagentsformentalhealthtoidentifyavenuesforimprovement AT manasgaur areviewoftheexplainabilityandsafetyofconversationalagentsformentalhealthtoidentifyavenuesforimprovement AT lujiekarenchen areviewoftheexplainabilityandsafetyofconversationalagentsformentalhealthtoidentifyavenuesforimprovement AT muskangarg areviewoftheexplainabilityandsafetyofconversationalagentsformentalhealthtoidentifyavenuesforimprovement AT biplavsrivastava areviewoftheexplainabilityandsafetyofconversationalagentsformentalhealthtoidentifyavenuesforimprovement AT surjodeepsarkar reviewoftheexplainabilityandsafetyofconversationalagentsformentalhealthtoidentifyavenuesforimprovement AT manasgaur reviewoftheexplainabilityandsafetyofconversationalagentsformentalhealthtoidentifyavenuesforimprovement AT lujiekarenchen reviewoftheexplainabilityandsafetyofconversationalagentsformentalhealthtoidentifyavenuesforimprovement AT muskangarg reviewoftheexplainabilityandsafetyofconversationalagentsformentalhealthtoidentifyavenuesforimprovement AT biplavsrivastava reviewoftheexplainabilityandsafetyofconversationalagentsformentalhealthtoidentifyavenuesforimprovement |