Assessment of Quality and Readability of Information Provided by ChatGPT in Relation to Anterior Cruciate Ligament Injury
The aim of our study was to evaluate the potential role of Artificial Intelligence tools like ChatGPT in patient education. To do this, we assessed both the quality and readability of information provided by ChatGPT 3.5 and 4 in relation to Anterior Cruciate Ligament (ACL) injury and treatment. Chat...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-01-01
|
| Series: | Journal of Personalized Medicine |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2075-4426/14/1/104 |
| _version_ | 1797343181275136000 |
|---|---|
| author | Stephen Fahy Stephan Oehme Danko Milinkovic Tobias Jung Benjamin Bartek |
| author_facet | Stephen Fahy Stephan Oehme Danko Milinkovic Tobias Jung Benjamin Bartek |
| author_sort | Stephen Fahy |
| collection | DOAJ |
| description | The aim of our study was to evaluate the potential role of Artificial Intelligence tools like ChatGPT in patient education. To do this, we assessed both the quality and readability of information provided by ChatGPT 3.5 and 4 in relation to Anterior Cruciate Ligament (ACL) injury and treatment. ChatGPT 3.5 and 4 were used to answer common patient queries relating to ACL injuries and treatment. The quality of the information was assessed using the DISCERN criteria. Readability was assessed with the use of seven readability formulae: the Flesch–Kincaid Reading Grade Level, the Flesch Reading Ease Score, the Raygor Estimate, the SMOG, the Fry, the FORCAST, and the Gunning Fog. The mean reading grade level (RGL) was compared with the recommended 8th-grade reading level, the mean RGL among adults in America. The perceived quality and mean RGL of answers given by both ChatGPT 3.5 and 4 was also compared. Both ChatGPT 3.5 and 4 yielded DISCERN scores suggesting “good” quality of information, with ChatGPT 4 slightly outperforming 3.5. However, readability levels for both versions significantly exceeded the average 8th-grade reading level for American patients. ChatGPT 3.5 had a mean RGL of 18.08, while the mean RGL of ChatGPT 4 was 17.9, exceeding the average American reading grade level by 10.08 grade levels and 9.09 grade levels, respectively. While ChatGPT can provide both reliable and good quality information on ACL injuries and treatment options, the readability of the content may limit its utility. Additionally, the consistent lack of source citation represents a significant area of concern for patients and clinicians alike. If AI is to play a role in patient education, it must reliably produce information which is accurate, easily comprehensible, and clearly sourced. |
| first_indexed | 2024-03-08T10:45:01Z |
| format | Article |
| id | doaj.art-1d96d11021a64cbb81a4de05072288b2 |
| institution | Directory Open Access Journal |
| issn | 2075-4426 |
| language | English |
| last_indexed | 2024-03-08T10:45:01Z |
| publishDate | 2024-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Journal of Personalized Medicine |
| spelling | doaj.art-1d96d11021a64cbb81a4de05072288b22024-01-26T17:19:35ZengMDPI AGJournal of Personalized Medicine2075-44262024-01-0114110410.3390/jpm14010104Assessment of Quality and Readability of Information Provided by ChatGPT in Relation to Anterior Cruciate Ligament InjuryStephen Fahy0Stephan Oehme1Danko Milinkovic2Tobias Jung3Benjamin Bartek4Centrum für Muskuloskeletale Chirurgie, Charité Universitätsmedizin, 10117 Berlin, GermanyCentrum für Muskuloskeletale Chirurgie, Charité Universitätsmedizin, 10117 Berlin, GermanyCentrum für Muskuloskeletale Chirurgie, Charité Universitätsmedizin, 10117 Berlin, GermanyCentrum für Muskuloskeletale Chirurgie, Charité Universitätsmedizin, 10117 Berlin, GermanyCentrum für Muskuloskeletale Chirurgie, Charité Universitätsmedizin, 10117 Berlin, GermanyThe aim of our study was to evaluate the potential role of Artificial Intelligence tools like ChatGPT in patient education. To do this, we assessed both the quality and readability of information provided by ChatGPT 3.5 and 4 in relation to Anterior Cruciate Ligament (ACL) injury and treatment. ChatGPT 3.5 and 4 were used to answer common patient queries relating to ACL injuries and treatment. The quality of the information was assessed using the DISCERN criteria. Readability was assessed with the use of seven readability formulae: the Flesch–Kincaid Reading Grade Level, the Flesch Reading Ease Score, the Raygor Estimate, the SMOG, the Fry, the FORCAST, and the Gunning Fog. The mean reading grade level (RGL) was compared with the recommended 8th-grade reading level, the mean RGL among adults in America. The perceived quality and mean RGL of answers given by both ChatGPT 3.5 and 4 was also compared. Both ChatGPT 3.5 and 4 yielded DISCERN scores suggesting “good” quality of information, with ChatGPT 4 slightly outperforming 3.5. However, readability levels for both versions significantly exceeded the average 8th-grade reading level for American patients. ChatGPT 3.5 had a mean RGL of 18.08, while the mean RGL of ChatGPT 4 was 17.9, exceeding the average American reading grade level by 10.08 grade levels and 9.09 grade levels, respectively. While ChatGPT can provide both reliable and good quality information on ACL injuries and treatment options, the readability of the content may limit its utility. Additionally, the consistent lack of source citation represents a significant area of concern for patients and clinicians alike. If AI is to play a role in patient education, it must reliably produce information which is accurate, easily comprehensible, and clearly sourced.https://www.mdpi.com/2075-4426/14/1/104anterior cruciate ligament (ACL)ACL reconstruction surgery (ACL-R)health literacyreadabilityChatGPTartificial intelligence (AI) |
| spellingShingle | Stephen Fahy Stephan Oehme Danko Milinkovic Tobias Jung Benjamin Bartek Assessment of Quality and Readability of Information Provided by ChatGPT in Relation to Anterior Cruciate Ligament Injury Journal of Personalized Medicine anterior cruciate ligament (ACL) ACL reconstruction surgery (ACL-R) health literacy readability ChatGPT artificial intelligence (AI) |
| title | Assessment of Quality and Readability of Information Provided by ChatGPT in Relation to Anterior Cruciate Ligament Injury |
| title_full | Assessment of Quality and Readability of Information Provided by ChatGPT in Relation to Anterior Cruciate Ligament Injury |
| title_fullStr | Assessment of Quality and Readability of Information Provided by ChatGPT in Relation to Anterior Cruciate Ligament Injury |
| title_full_unstemmed | Assessment of Quality and Readability of Information Provided by ChatGPT in Relation to Anterior Cruciate Ligament Injury |
| title_short | Assessment of Quality and Readability of Information Provided by ChatGPT in Relation to Anterior Cruciate Ligament Injury |
| title_sort | assessment of quality and readability of information provided by chatgpt in relation to anterior cruciate ligament injury |
| topic | anterior cruciate ligament (ACL) ACL reconstruction surgery (ACL-R) health literacy readability ChatGPT artificial intelligence (AI) |
| url | https://www.mdpi.com/2075-4426/14/1/104 |
| work_keys_str_mv | AT stephenfahy assessmentofqualityandreadabilityofinformationprovidedbychatgptinrelationtoanteriorcruciateligamentinjury AT stephanoehme assessmentofqualityandreadabilityofinformationprovidedbychatgptinrelationtoanteriorcruciateligamentinjury AT dankomilinkovic assessmentofqualityandreadabilityofinformationprovidedbychatgptinrelationtoanteriorcruciateligamentinjury AT tobiasjung assessmentofqualityandreadabilityofinformationprovidedbychatgptinrelationtoanteriorcruciateligamentinjury AT benjaminbartek assessmentofqualityandreadabilityofinformationprovidedbychatgptinrelationtoanteriorcruciateligamentinjury |