Exploring the potential of ChatGPT as an adjunct for generating diagnosis based on chief complaint and cone beam CT radiologic findings
Abstract Aim This study aimed to assess the performance of OpenAI’s ChatGPT in generating diagnosis based on chief complaint and cone beam computed tomography (CBCT) radiologic findings. Materials and methods 102 CBCT reports (48 with dental diseases (DD) and 54 with neoplastic/cystic diseases (N/CD...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2024-02-01
|
| Series: | BMC Medical Informatics and Decision Making |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12911-024-02445-y |
| _version_ | 1827327271728316416 |
|---|---|
| author | Yanni Hu Ziyang Hu Wenjing Liu Antian Gao Shanhui Wen Shu Liu Zitong Lin |
| author_facet | Yanni Hu Ziyang Hu Wenjing Liu Antian Gao Shanhui Wen Shu Liu Zitong Lin |
| author_sort | Yanni Hu |
| collection | DOAJ |
| description | Abstract Aim This study aimed to assess the performance of OpenAI’s ChatGPT in generating diagnosis based on chief complaint and cone beam computed tomography (CBCT) radiologic findings. Materials and methods 102 CBCT reports (48 with dental diseases (DD) and 54 with neoplastic/cystic diseases (N/CD)) were collected. ChatGPT was provided with chief complaint and CBCT radiologic findings. Diagnostic outputs from ChatGPT were scored based on five-point Likert scale. For diagnosis accuracy, the scoring was based on the accuracy of chief complaint related diagnosis and chief complaint unrelated diagnoses (1–5 points); for diagnosis completeness, the scoring was based on how many accurate diagnoses included in ChatGPT’s output for one case (1–5 points); for text quality, the scoring was based on how many text errors included in ChatGPT’s output for one case (1–5 points). For 54 N/CD cases, the consistence of the diagnosis generated by ChatGPT with pathological diagnosis was also calculated. The constitution of text errors in ChatGPT’s outputs was evaluated. Results After subjective ratings by expert reviewers on a five-point Likert scale, the final score of diagnosis accuracy, diagnosis completeness and text quality of ChatGPT was 3.7, 4.5 and 4.6 for the 102 cases. For diagnostic accuracy, it performed significantly better on N/CD (3.8/5) compared to DD (3.6/5). For 54 N/CD cases, 21(38.9%) cases have first diagnosis completely consistent with pathological diagnosis. No text errors were observed in 88.7% of all the 390 text items. Conclusion ChatGPT showed potential in generating radiographic diagnosis based on chief complaint and radiologic findings. However, the performance of ChatGPT varied with task complexity, necessitating professional oversight due to a certain error rate. |
| first_indexed | 2024-03-07T14:57:44Z |
| format | Article |
| id | doaj.art-b8dd61c5ab7444c9bd8907ed9290b42e |
| institution | Directory Open Access Journal |
| issn | 1472-6947 |
| language | English |
| last_indexed | 2024-03-07T14:57:44Z |
| publishDate | 2024-02-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Informatics and Decision Making |
| spelling | doaj.art-b8dd61c5ab7444c9bd8907ed9290b42e2024-03-05T19:19:53ZengBMCBMC Medical Informatics and Decision Making1472-69472024-02-012411910.1186/s12911-024-02445-yExploring the potential of ChatGPT as an adjunct for generating diagnosis based on chief complaint and cone beam CT radiologic findingsYanni Hu0Ziyang Hu1Wenjing Liu2Antian Gao3Shanhui Wen4Shu Liu5Zitong Lin6Department of Dentomaxillofacial Radiology, Nanjing Stomatological Hospital, Affiliated Hospital of Medical School, Institute of Stomatology, Nanjing UniversityDepartment of Dentomaxillofacial Radiology, Nanjing Stomatological Hospital, Affiliated Hospital of Medical School, Institute of Stomatology, Nanjing UniversityDepartment of Dentomaxillofacial Radiology, Nanjing Stomatological Hospital, Affiliated Hospital of Medical School, Institute of Stomatology, Nanjing UniversityDepartment of Dentomaxillofacial Radiology, Nanjing Stomatological Hospital, Affiliated Hospital of Medical School, Institute of Stomatology, Nanjing UniversityDepartment of Dentomaxillofacial Radiology, Nanjing Stomatological Hospital, Affiliated Hospital of Medical School, Institute of Stomatology, Nanjing UniversityDepartment of Dentomaxillofacial Radiology, Nanjing Stomatological Hospital, Affiliated Hospital of Medical School, Institute of Stomatology, Nanjing UniversityDepartment of Dentomaxillofacial Radiology, Nanjing Stomatological Hospital, Affiliated Hospital of Medical School, Institute of Stomatology, Nanjing UniversityAbstract Aim This study aimed to assess the performance of OpenAI’s ChatGPT in generating diagnosis based on chief complaint and cone beam computed tomography (CBCT) radiologic findings. Materials and methods 102 CBCT reports (48 with dental diseases (DD) and 54 with neoplastic/cystic diseases (N/CD)) were collected. ChatGPT was provided with chief complaint and CBCT radiologic findings. Diagnostic outputs from ChatGPT were scored based on five-point Likert scale. For diagnosis accuracy, the scoring was based on the accuracy of chief complaint related diagnosis and chief complaint unrelated diagnoses (1–5 points); for diagnosis completeness, the scoring was based on how many accurate diagnoses included in ChatGPT’s output for one case (1–5 points); for text quality, the scoring was based on how many text errors included in ChatGPT’s output for one case (1–5 points). For 54 N/CD cases, the consistence of the diagnosis generated by ChatGPT with pathological diagnosis was also calculated. The constitution of text errors in ChatGPT’s outputs was evaluated. Results After subjective ratings by expert reviewers on a five-point Likert scale, the final score of diagnosis accuracy, diagnosis completeness and text quality of ChatGPT was 3.7, 4.5 and 4.6 for the 102 cases. For diagnostic accuracy, it performed significantly better on N/CD (3.8/5) compared to DD (3.6/5). For 54 N/CD cases, 21(38.9%) cases have first diagnosis completely consistent with pathological diagnosis. No text errors were observed in 88.7% of all the 390 text items. Conclusion ChatGPT showed potential in generating radiographic diagnosis based on chief complaint and radiologic findings. However, the performance of ChatGPT varied with task complexity, necessitating professional oversight due to a certain error rate.https://doi.org/10.1186/s12911-024-02445-yLarge language modelCBCTDental DiseaseNeoplastic/cystic diseasesRadiologic findingRadiologic impression |
| spellingShingle | Yanni Hu Ziyang Hu Wenjing Liu Antian Gao Shanhui Wen Shu Liu Zitong Lin Exploring the potential of ChatGPT as an adjunct for generating diagnosis based on chief complaint and cone beam CT radiologic findings BMC Medical Informatics and Decision Making Large language model CBCT Dental Disease Neoplastic/cystic diseases Radiologic finding Radiologic impression |
| title | Exploring the potential of ChatGPT as an adjunct for generating diagnosis based on chief complaint and cone beam CT radiologic findings |
| title_full | Exploring the potential of ChatGPT as an adjunct for generating diagnosis based on chief complaint and cone beam CT radiologic findings |
| title_fullStr | Exploring the potential of ChatGPT as an adjunct for generating diagnosis based on chief complaint and cone beam CT radiologic findings |
| title_full_unstemmed | Exploring the potential of ChatGPT as an adjunct for generating diagnosis based on chief complaint and cone beam CT radiologic findings |
| title_short | Exploring the potential of ChatGPT as an adjunct for generating diagnosis based on chief complaint and cone beam CT radiologic findings |
| title_sort | exploring the potential of chatgpt as an adjunct for generating diagnosis based on chief complaint and cone beam ct radiologic findings |
| topic | Large language model CBCT Dental Disease Neoplastic/cystic diseases Radiologic finding Radiologic impression |
| url | https://doi.org/10.1186/s12911-024-02445-y |
| work_keys_str_mv | AT yannihu exploringthepotentialofchatgptasanadjunctforgeneratingdiagnosisbasedonchiefcomplaintandconebeamctradiologicfindings AT ziyanghu exploringthepotentialofchatgptasanadjunctforgeneratingdiagnosisbasedonchiefcomplaintandconebeamctradiologicfindings AT wenjingliu exploringthepotentialofchatgptasanadjunctforgeneratingdiagnosisbasedonchiefcomplaintandconebeamctradiologicfindings AT antiangao exploringthepotentialofchatgptasanadjunctforgeneratingdiagnosisbasedonchiefcomplaintandconebeamctradiologicfindings AT shanhuiwen exploringthepotentialofchatgptasanadjunctforgeneratingdiagnosisbasedonchiefcomplaintandconebeamctradiologicfindings AT shuliu exploringthepotentialofchatgptasanadjunctforgeneratingdiagnosisbasedonchiefcomplaintandconebeamctradiologicfindings AT zitonglin exploringthepotentialofchatgptasanadjunctforgeneratingdiagnosisbasedonchiefcomplaintandconebeamctradiologicfindings |