To Evaluate the Efficiency of ChatGPT in Medical Education: An Analysis of MCQ-Based Learning and Assessment
Objective: This study aimed to evaluate the potential of ChatGPT to help students for their assess ments via MCQ at different level of cognition by using different subjects of Internal medicine. Methods: This cross-sectional study was conducted in the Department of Internal Medicine in col labora...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
KARACHI MEDICAL AND DENTAL COLLEGE
2023-12-01
|
| Series: | Annals of Abbasi Shaheed Hospital and Karachi Medical & Dental College |
| Subjects: | |
| Online Access: | https://annals-ashkmdc.org/index.php/ashkmdc/article/view/759 |
| _version_ | 1797365449710632960 |
|---|---|
| author | Shirjeel Husain Zaid Ansari Azhar Hussain Sahar Abbasi Tahera Ayoob Rimsha Mujahid |
| author_facet | Shirjeel Husain Zaid Ansari Azhar Hussain Sahar Abbasi Tahera Ayoob Rimsha Mujahid |
| author_sort | Shirjeel Husain |
| collection | DOAJ |
| description |
Objective: This study aimed to evaluate the potential of ChatGPT to help students for their assess ments via MCQ at different level of cognition by using different subjects of Internal medicine.
Methods: This cross-sectional study was conducted in the Department of Internal Medicine in col laboration with post graduate medical education department from June 2023 to August 2023. An MCQ bank was established from three books of MCQ’s on subject of Internal Medicine. Total 1428 MCQ’s were followed for scrutiny and 307 MCQ’s were selected for the assigned task. The selected MCQ’s were manually entered one by one in a fresh Chat GPT session. The response was noted against the replies given in respective MCQ’s book and marked as correct, not correct or partially correct. MCQ’s were categorized as per chapters in Internal medicine and as per cognition level of MCQ’s i.e. C1, C2 and C3. Data was analyzed on SPSS version 21.00.
Results: Chat GPT replied with 199 correct replies while 98 were wrong and 10 were partially correct. Chat GPT scored 64% overall in all categories. At level of cognition, it solved C2 MCQ’s by 80 % but scored 69% and 54% in C1 and C3 categories respectively. Chat GPT replied with 80% accuracy for C2 level MCQ’s while results were low for C3 category at around 54%. C1 also had low percentage of correct answers standing close to 69.8%. Almost all subjects showed healthy responses around the mean except for endocrinology and hematology where responses are below 60% and 40% respectively.
Conclusion: This study findings suggest that ChatGPT is a useful tool for students and medical educationist with its current framework but a subtle approach should be inclined towards its role in future.
|
| first_indexed | 2024-03-08T16:49:08Z |
| format | Article |
| id | doaj.art-751ae612dc8f4964a58c525171ae5419 |
| institution | Directory Open Access Journal |
| issn | 1563-3241 |
| language | English |
| last_indexed | 2024-03-08T16:49:08Z |
| publishDate | 2023-12-01 |
| publisher | KARACHI MEDICAL AND DENTAL COLLEGE |
| record_format | Article |
| series | Annals of Abbasi Shaheed Hospital and Karachi Medical & Dental College |
| spelling | doaj.art-751ae612dc8f4964a58c525171ae54192024-01-05T05:18:26ZengKARACHI MEDICAL AND DENTAL COLLEGEAnnals of Abbasi Shaheed Hospital and Karachi Medical & Dental College1563-32412023-12-012804To Evaluate the Efficiency of ChatGPT in Medical Education: An Analysis of MCQ-Based Learning and AssessmentShirjeel Husain0Zaid Ansari1Azhar Hussain2Sahar Abbasi3Tahera Ayoob4Rimsha Mujahid5Department of Internal Medicine, Liaquat College of Medicine and DentistryDepartment of Internal Medicine, Liaquat College of Medicine and Dentistry Darul Sehat HospitalDepartment of Internal Medicine, Liaquat College of Medicine and Dentistry & Darul Sehat Hospital, Karachi.Directorate of Medical Education, College of Physicians & Surgeons PakistanDepartment of Oral Surgery, Liaquat College of Medicine and Dentistry & Qamar Dental Hospital, KarachiDepartment of Internal Medicine, Liaquat College of Medicine and Dentistry & Darul Sehat Hospital, Karachi. Objective: This study aimed to evaluate the potential of ChatGPT to help students for their assess ments via MCQ at different level of cognition by using different subjects of Internal medicine. Methods: This cross-sectional study was conducted in the Department of Internal Medicine in col laboration with post graduate medical education department from June 2023 to August 2023. An MCQ bank was established from three books of MCQ’s on subject of Internal Medicine. Total 1428 MCQ’s were followed for scrutiny and 307 MCQ’s were selected for the assigned task. The selected MCQ’s were manually entered one by one in a fresh Chat GPT session. The response was noted against the replies given in respective MCQ’s book and marked as correct, not correct or partially correct. MCQ’s were categorized as per chapters in Internal medicine and as per cognition level of MCQ’s i.e. C1, C2 and C3. Data was analyzed on SPSS version 21.00. Results: Chat GPT replied with 199 correct replies while 98 were wrong and 10 were partially correct. Chat GPT scored 64% overall in all categories. At level of cognition, it solved C2 MCQ’s by 80 % but scored 69% and 54% in C1 and C3 categories respectively. Chat GPT replied with 80% accuracy for C2 level MCQ’s while results were low for C3 category at around 54%. C1 also had low percentage of correct answers standing close to 69.8%. Almost all subjects showed healthy responses around the mean except for endocrinology and hematology where responses are below 60% and 40% respectively. Conclusion: This study findings suggest that ChatGPT is a useful tool for students and medical educationist with its current framework but a subtle approach should be inclined towards its role in future. https://annals-ashkmdc.org/index.php/ashkmdc/article/view/759Medical education |
| spellingShingle | Shirjeel Husain Zaid Ansari Azhar Hussain Sahar Abbasi Tahera Ayoob Rimsha Mujahid To Evaluate the Efficiency of ChatGPT in Medical Education: An Analysis of MCQ-Based Learning and Assessment Annals of Abbasi Shaheed Hospital and Karachi Medical & Dental College Medical education |
| title | To Evaluate the Efficiency of ChatGPT in Medical Education: An Analysis of MCQ-Based Learning and Assessment |
| title_full | To Evaluate the Efficiency of ChatGPT in Medical Education: An Analysis of MCQ-Based Learning and Assessment |
| title_fullStr | To Evaluate the Efficiency of ChatGPT in Medical Education: An Analysis of MCQ-Based Learning and Assessment |
| title_full_unstemmed | To Evaluate the Efficiency of ChatGPT in Medical Education: An Analysis of MCQ-Based Learning and Assessment |
| title_short | To Evaluate the Efficiency of ChatGPT in Medical Education: An Analysis of MCQ-Based Learning and Assessment |
| title_sort | to evaluate the efficiency of chatgpt in medical education an analysis of mcq based learning and assessment |
| topic | Medical education |
| url | https://annals-ashkmdc.org/index.php/ashkmdc/article/view/759 |
| work_keys_str_mv | AT shirjeelhusain toevaluatetheefficiencyofchatgptinmedicaleducationananalysisofmcqbasedlearningandassessment AT zaidansari toevaluatetheefficiencyofchatgptinmedicaleducationananalysisofmcqbasedlearningandassessment AT azharhussain toevaluatetheefficiencyofchatgptinmedicaleducationananalysisofmcqbasedlearningandassessment AT saharabbasi toevaluatetheefficiencyofchatgptinmedicaleducationananalysisofmcqbasedlearningandassessment AT taheraayoob toevaluatetheefficiencyofchatgptinmedicaleducationananalysisofmcqbasedlearningandassessment AT rimshamujahid toevaluatetheefficiencyofchatgptinmedicaleducationananalysisofmcqbasedlearningandassessment |