The Effects of a Health Care Chatbot’s Complexity and Persona on User Trust, Perceived Usability, and Effectiveness: Mixed Methods Study
BackgroundThe rising adoption of telehealth provides new opportunities for more effective and equitable health care information mediums. The ability of chatbots to provide a conversational, personal, and comprehendible avenue for learning about health care information make th...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2023-02-01
|
| Series: | JMIR Human Factors |
| Online Access: | https://humanfactors.jmir.org/2023/1/e41017 |
| _version_ | 1797734416705912832 |
|---|---|
| author | Joshua Biro Courtney Linder David Neyens |
| author_facet | Joshua Biro Courtney Linder David Neyens |
| author_sort | Joshua Biro |
| collection | DOAJ |
| description |
BackgroundThe rising adoption of telehealth provides new opportunities for more effective and equitable health care information mediums. The ability of chatbots to provide a conversational, personal, and comprehendible avenue for learning about health care information make them a promising tool for addressing health care inequity as health care trends continue toward web-based and remote processes. Although chatbots have been studied in the health care domain for their efficacy for smoking cessation, diet recommendation, and other assistive applications, few studies have examined how specific design characteristics influence the effectiveness of chatbots in providing health information.
ObjectiveOur objective was to investigate the influence of different design considerations on the effectiveness of an educational health care chatbot.
MethodsA 2×3 between-subjects study was performed with 2 independent variables: a chatbot’s complexity of responses (eg, technical or nontechnical language) and the presented qualifications of the chatbot’s persona (eg, doctor, nurse, or nursing student). Regression models were used to evaluate the impact of these variables on 3 outcome measures: effectiveness, usability, and trust. A qualitative transcript review was also done to review how participants engaged with the chatbot.
ResultsAnalysis of 71 participants found that participants who received technical language responses were significantly more likely to be in the high effectiveness group, which had higher improvements in test scores (odds ratio [OR] 2.73, 95% CI 1.05-7.41; P=.04). Participants with higher health literacy (OR 2.04, 95% CI 1.11-4.00, P=.03) were significantly more likely to trust the chatbot. The participants engaged with the chatbot in a variety of ways, with some taking a conversational approach and others treating the chatbot more like a search engine.
ConclusionsGiven their increasing popularity, it is vital that we consider how chatbots are designed and implemented. This study showed that factors such as chatbots’ persona and language complexity are two design considerations that influence the ability of chatbots to successfully provide health care information. |
| first_indexed | 2024-03-12T12:44:07Z |
| format | Article |
| id | doaj.art-13ce5ba3685f4840a382f3b44cd92a87 |
| institution | Directory Open Access Journal |
| issn | 2292-9495 |
| language | English |
| last_indexed | 2024-03-12T12:44:07Z |
| publishDate | 2023-02-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Human Factors |
| spelling | doaj.art-13ce5ba3685f4840a382f3b44cd92a872023-08-28T23:34:07ZengJMIR PublicationsJMIR Human Factors2292-94952023-02-0110e4101710.2196/41017The Effects of a Health Care Chatbot’s Complexity and Persona on User Trust, Perceived Usability, and Effectiveness: Mixed Methods StudyJoshua Birohttps://orcid.org/0000-0001-7362-4138Courtney Linderhttps://orcid.org/0000-0002-5810-7009David Neyenshttps://orcid.org/0000-0002-3443-518X BackgroundThe rising adoption of telehealth provides new opportunities for more effective and equitable health care information mediums. The ability of chatbots to provide a conversational, personal, and comprehendible avenue for learning about health care information make them a promising tool for addressing health care inequity as health care trends continue toward web-based and remote processes. Although chatbots have been studied in the health care domain for their efficacy for smoking cessation, diet recommendation, and other assistive applications, few studies have examined how specific design characteristics influence the effectiveness of chatbots in providing health information. ObjectiveOur objective was to investigate the influence of different design considerations on the effectiveness of an educational health care chatbot. MethodsA 2×3 between-subjects study was performed with 2 independent variables: a chatbot’s complexity of responses (eg, technical or nontechnical language) and the presented qualifications of the chatbot’s persona (eg, doctor, nurse, or nursing student). Regression models were used to evaluate the impact of these variables on 3 outcome measures: effectiveness, usability, and trust. A qualitative transcript review was also done to review how participants engaged with the chatbot. ResultsAnalysis of 71 participants found that participants who received technical language responses were significantly more likely to be in the high effectiveness group, which had higher improvements in test scores (odds ratio [OR] 2.73, 95% CI 1.05-7.41; P=.04). Participants with higher health literacy (OR 2.04, 95% CI 1.11-4.00, P=.03) were significantly more likely to trust the chatbot. The participants engaged with the chatbot in a variety of ways, with some taking a conversational approach and others treating the chatbot more like a search engine. ConclusionsGiven their increasing popularity, it is vital that we consider how chatbots are designed and implemented. This study showed that factors such as chatbots’ persona and language complexity are two design considerations that influence the ability of chatbots to successfully provide health care information.https://humanfactors.jmir.org/2023/1/e41017 |
| spellingShingle | Joshua Biro Courtney Linder David Neyens The Effects of a Health Care Chatbot’s Complexity and Persona on User Trust, Perceived Usability, and Effectiveness: Mixed Methods Study JMIR Human Factors |
| title | The Effects of a Health Care Chatbot’s Complexity and Persona on User Trust, Perceived Usability, and Effectiveness: Mixed Methods Study |
| title_full | The Effects of a Health Care Chatbot’s Complexity and Persona on User Trust, Perceived Usability, and Effectiveness: Mixed Methods Study |
| title_fullStr | The Effects of a Health Care Chatbot’s Complexity and Persona on User Trust, Perceived Usability, and Effectiveness: Mixed Methods Study |
| title_full_unstemmed | The Effects of a Health Care Chatbot’s Complexity and Persona on User Trust, Perceived Usability, and Effectiveness: Mixed Methods Study |
| title_short | The Effects of a Health Care Chatbot’s Complexity and Persona on User Trust, Perceived Usability, and Effectiveness: Mixed Methods Study |
| title_sort | effects of a health care chatbot s complexity and persona on user trust perceived usability and effectiveness mixed methods study |
| url | https://humanfactors.jmir.org/2023/1/e41017 |
| work_keys_str_mv | AT joshuabiro theeffectsofahealthcarechatbotscomplexityandpersonaonusertrustperceivedusabilityandeffectivenessmixedmethodsstudy AT courtneylinder theeffectsofahealthcarechatbotscomplexityandpersonaonusertrustperceivedusabilityandeffectivenessmixedmethodsstudy AT davidneyens theeffectsofahealthcarechatbotscomplexityandpersonaonusertrustperceivedusabilityandeffectivenessmixedmethodsstudy AT joshuabiro effectsofahealthcarechatbotscomplexityandpersonaonusertrustperceivedusabilityandeffectivenessmixedmethodsstudy AT courtneylinder effectsofahealthcarechatbotscomplexityandpersonaonusertrustperceivedusabilityandeffectivenessmixedmethodsstudy AT davidneyens effectsofahealthcarechatbotscomplexityandpersonaonusertrustperceivedusabilityandeffectivenessmixedmethodsstudy |