Assessment of the capacity of ChatGPT as a self-learning tool in medical pharmacology: a study using MCQs
Abstract Background ChatGPT is a large language model developed by OpenAI that exhibits a remarkable ability to simulate human speech. This investigation attempts to evaluate the potential of ChatGPT as a standalone self-learning tool, with specific attention on its efficacy in answering multiple-ch...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-11-01
|
| Series: | BMC Medical Education |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12909-023-04832-x |
| _version_ | 1797558970400899072 |
|---|---|
| author | Woong Choi |
| author_facet | Woong Choi |
| author_sort | Woong Choi |
| collection | DOAJ |
| description | Abstract Background ChatGPT is a large language model developed by OpenAI that exhibits a remarkable ability to simulate human speech. This investigation attempts to evaluate the potential of ChatGPT as a standalone self-learning tool, with specific attention on its efficacy in answering multiple-choice questions (MCQs) and providing credible rationale for its responses. Methods The study used 78 test items from the Korean Comprehensive Basic Medical Sciences Examination (K-CBMSE) for years 2019 to 2021. 78 test items translated from Korean to English with four lead-in prompts per item resulted in a total of 312 MCQs. The MCQs were submitted to ChatGPT and the responses were analyzed for correctness, consistency, and relevance. Results ChatGPT responded with an overall accuracy of 76.0%. Compared to its performance on recall and interpretation questions, the model performed poorly on problem-solving questions. ChatGPT offered correct rationales for 77.8% (182/234) of the responses, with errors primarily arising from faulty information and flawed reasoning. In terms of references, ChatGPT provided incorrect citations for 69.7% (191/274) of the responses. While the veracity of reference paragraphs could not be ascertained, 77.0% (47/61) were deemed pertinent and accurate with respect to the answer key. Conclusion The current version of ChatGPT has limitations in accurately answering MCQs and generating correct and relevant rationales, particularly when it comes to referencing. To avoid possible threats such as spreading inaccuracies and decreasing critical thinking skills, ChatGPT should be used with supervision. |
| first_indexed | 2024-03-10T17:37:52Z |
| format | Article |
| id | doaj.art-bd12b5719dbe4f8688ae7ff203529753 |
| institution | Directory Open Access Journal |
| issn | 1472-6920 |
| language | English |
| last_indexed | 2024-03-10T17:37:52Z |
| publishDate | 2023-11-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Education |
| spelling | doaj.art-bd12b5719dbe4f8688ae7ff2035297532023-11-20T09:47:29ZengBMCBMC Medical Education1472-69202023-11-012311810.1186/s12909-023-04832-xAssessment of the capacity of ChatGPT as a self-learning tool in medical pharmacology: a study using MCQsWoong Choi0Department of Pharmacology, College of Medicine, Chungbuk National UniversityAbstract Background ChatGPT is a large language model developed by OpenAI that exhibits a remarkable ability to simulate human speech. This investigation attempts to evaluate the potential of ChatGPT as a standalone self-learning tool, with specific attention on its efficacy in answering multiple-choice questions (MCQs) and providing credible rationale for its responses. Methods The study used 78 test items from the Korean Comprehensive Basic Medical Sciences Examination (K-CBMSE) for years 2019 to 2021. 78 test items translated from Korean to English with four lead-in prompts per item resulted in a total of 312 MCQs. The MCQs were submitted to ChatGPT and the responses were analyzed for correctness, consistency, and relevance. Results ChatGPT responded with an overall accuracy of 76.0%. Compared to its performance on recall and interpretation questions, the model performed poorly on problem-solving questions. ChatGPT offered correct rationales for 77.8% (182/234) of the responses, with errors primarily arising from faulty information and flawed reasoning. In terms of references, ChatGPT provided incorrect citations for 69.7% (191/274) of the responses. While the veracity of reference paragraphs could not be ascertained, 77.0% (47/61) were deemed pertinent and accurate with respect to the answer key. Conclusion The current version of ChatGPT has limitations in accurately answering MCQs and generating correct and relevant rationales, particularly when it comes to referencing. To avoid possible threats such as spreading inaccuracies and decreasing critical thinking skills, ChatGPT should be used with supervision.https://doi.org/10.1186/s12909-023-04832-xChatGPTLarge language modelSelf-directed learningPerformanceMultiple-choice questionsRationale |
| spellingShingle | Woong Choi Assessment of the capacity of ChatGPT as a self-learning tool in medical pharmacology: a study using MCQs BMC Medical Education ChatGPT Large language model Self-directed learning Performance Multiple-choice questions Rationale |
| title | Assessment of the capacity of ChatGPT as a self-learning tool in medical pharmacology: a study using MCQs |
| title_full | Assessment of the capacity of ChatGPT as a self-learning tool in medical pharmacology: a study using MCQs |
| title_fullStr | Assessment of the capacity of ChatGPT as a self-learning tool in medical pharmacology: a study using MCQs |
| title_full_unstemmed | Assessment of the capacity of ChatGPT as a self-learning tool in medical pharmacology: a study using MCQs |
| title_short | Assessment of the capacity of ChatGPT as a self-learning tool in medical pharmacology: a study using MCQs |
| title_sort | assessment of the capacity of chatgpt as a self learning tool in medical pharmacology a study using mcqs |
| topic | ChatGPT Large language model Self-directed learning Performance Multiple-choice questions Rationale |
| url | https://doi.org/10.1186/s12909-023-04832-x |
| work_keys_str_mv | AT woongchoi assessmentofthecapacityofchatgptasaselflearningtoolinmedicalpharmacologyastudyusingmcqs |