A chat about actinic keratosis: Examining capabilities and user experience of ChatGPT as a digital health technology in dermato‐oncology
Abstract Background The potential applications of artificial intelligence (AI) in dermatology are evolving rapidly. Chatbots are an emerging trend in healthcare that rely on large language models (LLMs) to generate answers to prompts from users. However, the factuality and user experience (UX) of su...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2024-03-01
|
| Series: | JEADV Clinical Practice |
| Subjects: | |
| Online Access: | https://doi.org/10.1002/jvc2.263 |
| _version_ | 1827337807756001280 |
|---|---|
| author | Heather C. Lent Vinzent K. Ortner Katrine E. Karmisholt Stine R. Wiegell Christoffer V. Nissen Silje H. Omland Maria R. Kamstrup Katrine Togsverd‐Bo Merete Haedersdal |
| author_facet | Heather C. Lent Vinzent K. Ortner Katrine E. Karmisholt Stine R. Wiegell Christoffer V. Nissen Silje H. Omland Maria R. Kamstrup Katrine Togsverd‐Bo Merete Haedersdal |
| author_sort | Heather C. Lent |
| collection | DOAJ |
| description | Abstract Background The potential applications of artificial intelligence (AI) in dermatology are evolving rapidly. Chatbots are an emerging trend in healthcare that rely on large language models (LLMs) to generate answers to prompts from users. However, the factuality and user experience (UX) of such chatbots remain to be evaluated in the context of dermato‐oncology. Objectives To examine the potential of Chat Generative Pretrained Transformer (ChatGPT) as a reliable source of information in the context of actinic keratosis (AK) and to evaluate clinicians' attitudes and UX with regard to the chatbot. Methods A set of 38 clinical questions were compiled and entered as natural language queries in separate, individual conversation threads in ChatGPT (OpenAI, default GPT 3.5). Questions pertain to patient education, diagnosis, and treatment. ChatGPT's responses were presented to a panel of 7 dermatologists for rating of factual accuracy, currency of information, and completeness of the response. Attitudes towards ChatGTP were explored qualitatively and quantitatively using a validated user experience questionnaire (UEQ). Results ChatGPT answered 12 questions (31.6%) with an accurate, current, and complete response. ChatGPT performed best for questions on patient education, including pathogenesis of AK and potential risk factors, but struggled with diagnosis and treatment. Major deficits were seen in grading AK, providing up‐to‐date treatment guidance, and asserting incorrect information with unwarranted confidence. Further, responses were considered verbose with an average word count of 198 (SD 55) and overly alarming of the risk of malignant transformation. Based on UEQ responses, the expert panel considered ChatGPT an attractive and efficient tool, scoring highest for speed of information retrieval, but deemed the chatbot inaccurate and verbose, scoring lowest for clarity. Conclusions While dermatologists rated ChatGPT high in UX, the underlying LLMs that enable such chatbots require further development to guarantee accuracy and concision required in a clinical setting. |
| first_indexed | 2024-03-07T19:02:22Z |
| format | Article |
| id | doaj.art-654cb69cf5a3481cbe6d8bcd5392e5f2 |
| institution | Directory Open Access Journal |
| issn | 2768-6566 |
| language | English |
| last_indexed | 2024-03-07T19:02:22Z |
| publishDate | 2024-03-01 |
| publisher | Wiley |
| record_format | Article |
| series | JEADV Clinical Practice |
| spelling | doaj.art-654cb69cf5a3481cbe6d8bcd5392e5f22024-03-01T11:39:22ZengWileyJEADV Clinical Practice2768-65662024-03-013125826510.1002/jvc2.263A chat about actinic keratosis: Examining capabilities and user experience of ChatGPT as a digital health technology in dermato‐oncologyHeather C. Lent0Vinzent K. Ortner1Katrine E. Karmisholt2Stine R. Wiegell3Christoffer V. Nissen4Silje H. Omland5Maria R. Kamstrup6Katrine Togsverd‐Bo7Merete Haedersdal8Department of Computer Science Aalborg University Copenhagen DenmarkDepartment of Dermatology Copenhagen University Hospital, Bispebjerg and Frederiksberg Copenhagen DenmarkDepartment of Dermatology Copenhagen University Hospital, Bispebjerg and Frederiksberg Copenhagen DenmarkDepartment of Dermatology Copenhagen University Hospital, Bispebjerg and Frederiksberg Copenhagen DenmarkDepartment of Dermatology Copenhagen University Hospital, Bispebjerg and Frederiksberg Copenhagen DenmarkDepartment of Computer Science Aalborg University Copenhagen DenmarkDepartment of Dermatology Copenhagen University Hospital, Bispebjerg and Frederiksberg Copenhagen DenmarkDepartment of Dermatology Copenhagen University Hospital, Bispebjerg and Frederiksberg Copenhagen DenmarkDepartment of Dermatology Copenhagen University Hospital, Bispebjerg and Frederiksberg Copenhagen DenmarkAbstract Background The potential applications of artificial intelligence (AI) in dermatology are evolving rapidly. Chatbots are an emerging trend in healthcare that rely on large language models (LLMs) to generate answers to prompts from users. However, the factuality and user experience (UX) of such chatbots remain to be evaluated in the context of dermato‐oncology. Objectives To examine the potential of Chat Generative Pretrained Transformer (ChatGPT) as a reliable source of information in the context of actinic keratosis (AK) and to evaluate clinicians' attitudes and UX with regard to the chatbot. Methods A set of 38 clinical questions were compiled and entered as natural language queries in separate, individual conversation threads in ChatGPT (OpenAI, default GPT 3.5). Questions pertain to patient education, diagnosis, and treatment. ChatGPT's responses were presented to a panel of 7 dermatologists for rating of factual accuracy, currency of information, and completeness of the response. Attitudes towards ChatGTP were explored qualitatively and quantitatively using a validated user experience questionnaire (UEQ). Results ChatGPT answered 12 questions (31.6%) with an accurate, current, and complete response. ChatGPT performed best for questions on patient education, including pathogenesis of AK and potential risk factors, but struggled with diagnosis and treatment. Major deficits were seen in grading AK, providing up‐to‐date treatment guidance, and asserting incorrect information with unwarranted confidence. Further, responses were considered verbose with an average word count of 198 (SD 55) and overly alarming of the risk of malignant transformation. Based on UEQ responses, the expert panel considered ChatGPT an attractive and efficient tool, scoring highest for speed of information retrieval, but deemed the chatbot inaccurate and verbose, scoring lowest for clarity. Conclusions While dermatologists rated ChatGPT high in UX, the underlying LLMs that enable such chatbots require further development to guarantee accuracy and concision required in a clinical setting.https://doi.org/10.1002/jvc2.263actinic keratosisChatGPTlarge language modelsnatural language processingskin canceruser experience |
| spellingShingle | Heather C. Lent Vinzent K. Ortner Katrine E. Karmisholt Stine R. Wiegell Christoffer V. Nissen Silje H. Omland Maria R. Kamstrup Katrine Togsverd‐Bo Merete Haedersdal A chat about actinic keratosis: Examining capabilities and user experience of ChatGPT as a digital health technology in dermato‐oncology JEADV Clinical Practice actinic keratosis ChatGPT large language models natural language processing skin cancer user experience |
| title | A chat about actinic keratosis: Examining capabilities and user experience of ChatGPT as a digital health technology in dermato‐oncology |
| title_full | A chat about actinic keratosis: Examining capabilities and user experience of ChatGPT as a digital health technology in dermato‐oncology |
| title_fullStr | A chat about actinic keratosis: Examining capabilities and user experience of ChatGPT as a digital health technology in dermato‐oncology |
| title_full_unstemmed | A chat about actinic keratosis: Examining capabilities and user experience of ChatGPT as a digital health technology in dermato‐oncology |
| title_short | A chat about actinic keratosis: Examining capabilities and user experience of ChatGPT as a digital health technology in dermato‐oncology |
| title_sort | chat about actinic keratosis examining capabilities and user experience of chatgpt as a digital health technology in dermato oncology |
| topic | actinic keratosis ChatGPT large language models natural language processing skin cancer user experience |
| url | https://doi.org/10.1002/jvc2.263 |
| work_keys_str_mv | AT heatherclent achataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT vinzentkortner achataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT katrineekarmisholt achataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT stinerwiegell achataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT christoffervnissen achataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT siljehomland achataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT mariarkamstrup achataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT katrinetogsverdbo achataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT meretehaedersdal achataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT heatherclent chataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT vinzentkortner chataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT katrineekarmisholt chataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT stinerwiegell chataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT christoffervnissen chataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT siljehomland chataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT mariarkamstrup chataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT katrinetogsverdbo chataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology AT meretehaedersdal chataboutactinickeratosisexaminingcapabilitiesanduserexperienceofchatgptasadigitalhealthtechnologyindermatooncology |