ChatGPT’s performance in German OB/GYN exams – paving the way for AI-enhanced medical education and clinical practice
BackgroundChat Generative Pre-Trained Transformer (ChatGPT) is an artificial learning and large language model tool developed by OpenAI in 2022. It utilizes deep learning algorithms to process natural language and generate responses, which renders it suitable for conversational interfaces. ChatGPT’s...
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-12-01
|
| Series: | Frontiers in Medicine |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fmed.2023.1296615/full |
| _version_ | 1827586042462470144 |
|---|---|
| author | Maximilian Riedel Katharina Kaefinger Antonia Stuehrenberg Viktoria Ritter Niklas Amann Anna Graf Florian Recker Evelyn Klein Marion Kiechle Fabian Riedel Bastian Meyer |
| author_facet | Maximilian Riedel Katharina Kaefinger Antonia Stuehrenberg Viktoria Ritter Niklas Amann Anna Graf Florian Recker Evelyn Klein Marion Kiechle Fabian Riedel Bastian Meyer |
| author_sort | Maximilian Riedel |
| collection | DOAJ |
| description | BackgroundChat Generative Pre-Trained Transformer (ChatGPT) is an artificial learning and large language model tool developed by OpenAI in 2022. It utilizes deep learning algorithms to process natural language and generate responses, which renders it suitable for conversational interfaces. ChatGPT’s potential to transform medical education and clinical practice is currently being explored, but its capabilities and limitations in this domain remain incompletely investigated. The present study aimed to assess ChatGPT’s performance in medical knowledge competency for problem assessment in obstetrics and gynecology (OB/GYN).MethodsTwo datasets were established for analysis: questions (1) from OB/GYN course exams at a German university hospital and (2) from the German medical state licensing exams. In order to assess ChatGPT’s performance, questions were entered into the chat interface, and responses were documented. A quantitative analysis compared ChatGPT’s accuracy with that of medical students for different levels of difficulty and types of questions. Additionally, a qualitative analysis assessed the quality of ChatGPT’s responses regarding ease of understanding, conciseness, accuracy, completeness, and relevance. Non-obvious insights generated by ChatGPT were evaluated, and a density index of insights was established in order to quantify the tool’s ability to provide students with relevant and concise medical knowledge.ResultsChatGPT demonstrated consistent and comparable performance across both datasets. It provided correct responses at a rate comparable with that of medical students, thereby indicating its ability to handle a diverse spectrum of questions ranging from general knowledge to complex clinical case presentations. The tool’s accuracy was partly affected by question difficulty in the medical state exam dataset. Our qualitative assessment revealed that ChatGPT provided mostly accurate, complete, and relevant answers. ChatGPT additionally provided many non-obvious insights, especially in correctly answered questions, which indicates its potential for enhancing autonomous medical learning.ConclusionChatGPT has promise as a supplementary tool in medical education and clinical practice. Its ability to provide accurate and insightful responses showcases its adaptability to complex clinical scenarios. As AI technologies continue to evolve, ChatGPT and similar tools may contribute to more efficient and personalized learning experiences and assistance for health care providers. |
| first_indexed | 2024-03-08T23:55:22Z |
| format | Article |
| id | doaj.art-099dd837406c4dbf8a0617e090d61839 |
| institution | Directory Open Access Journal |
| issn | 2296-858X |
| language | English |
| last_indexed | 2024-03-08T23:55:22Z |
| publishDate | 2023-12-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Medicine |
| spelling | doaj.art-099dd837406c4dbf8a0617e090d618392023-12-13T04:38:13ZengFrontiers Media S.A.Frontiers in Medicine2296-858X2023-12-011010.3389/fmed.2023.12966151296615ChatGPT’s performance in German OB/GYN exams – paving the way for AI-enhanced medical education and clinical practiceMaximilian Riedel0Katharina Kaefinger1Antonia Stuehrenberg2Viktoria Ritter3Niklas Amann4Anna Graf5Florian Recker6Evelyn Klein7Marion Kiechle8Fabian Riedel9Bastian Meyer10Department of Gynecology and Obstetrics, Klinikum Rechts der Isar, Technical University Munich (TU), Munich, GermanyDepartment of Gynecology and Obstetrics, Klinikum Rechts der Isar, Technical University Munich (TU), Munich, GermanyDepartment of Gynecology and Obstetrics, Klinikum Rechts der Isar, Technical University Munich (TU), Munich, GermanyDepartment of Gynecology and Obstetrics, Klinikum Rechts der Isar, Technical University Munich (TU), Munich, GermanyDepartment of Gynecology and Obstetrics, Friedrich–Alexander-University Erlangen–Nuremberg (FAU), Erlangen, GermanyDepartment of Gynecology and Obstetrics, Klinikum Rechts der Isar, Technical University Munich (TU), Munich, GermanyDepartment of Gynecology and Obstetrics, Bonn University Hospital, Bonn, GermanyDepartment of Gynecology and Obstetrics, Klinikum Rechts der Isar, Technical University Munich (TU), Munich, GermanyDepartment of Gynecology and Obstetrics, Klinikum Rechts der Isar, Technical University Munich (TU), Munich, GermanyDepartment of Gynecology and Obstetrics, Heidelberg University Hospital, Heidelberg, GermanyDepartment of Gynecology and Obstetrics, Klinikum Rechts der Isar, Technical University Munich (TU), Munich, GermanyBackgroundChat Generative Pre-Trained Transformer (ChatGPT) is an artificial learning and large language model tool developed by OpenAI in 2022. It utilizes deep learning algorithms to process natural language and generate responses, which renders it suitable for conversational interfaces. ChatGPT’s potential to transform medical education and clinical practice is currently being explored, but its capabilities and limitations in this domain remain incompletely investigated. The present study aimed to assess ChatGPT’s performance in medical knowledge competency for problem assessment in obstetrics and gynecology (OB/GYN).MethodsTwo datasets were established for analysis: questions (1) from OB/GYN course exams at a German university hospital and (2) from the German medical state licensing exams. In order to assess ChatGPT’s performance, questions were entered into the chat interface, and responses were documented. A quantitative analysis compared ChatGPT’s accuracy with that of medical students for different levels of difficulty and types of questions. Additionally, a qualitative analysis assessed the quality of ChatGPT’s responses regarding ease of understanding, conciseness, accuracy, completeness, and relevance. Non-obvious insights generated by ChatGPT were evaluated, and a density index of insights was established in order to quantify the tool’s ability to provide students with relevant and concise medical knowledge.ResultsChatGPT demonstrated consistent and comparable performance across both datasets. It provided correct responses at a rate comparable with that of medical students, thereby indicating its ability to handle a diverse spectrum of questions ranging from general knowledge to complex clinical case presentations. The tool’s accuracy was partly affected by question difficulty in the medical state exam dataset. Our qualitative assessment revealed that ChatGPT provided mostly accurate, complete, and relevant answers. ChatGPT additionally provided many non-obvious insights, especially in correctly answered questions, which indicates its potential for enhancing autonomous medical learning.ConclusionChatGPT has promise as a supplementary tool in medical education and clinical practice. Its ability to provide accurate and insightful responses showcases its adaptability to complex clinical scenarios. As AI technologies continue to evolve, ChatGPT and similar tools may contribute to more efficient and personalized learning experiences and assistance for health care providers.https://www.frontiersin.org/articles/10.3389/fmed.2023.1296615/fullartificial intelligenceChatGPTmedical educationmachine learningobstetrics and gynecologystudents |
| spellingShingle | Maximilian Riedel Katharina Kaefinger Antonia Stuehrenberg Viktoria Ritter Niklas Amann Anna Graf Florian Recker Evelyn Klein Marion Kiechle Fabian Riedel Bastian Meyer ChatGPT’s performance in German OB/GYN exams – paving the way for AI-enhanced medical education and clinical practice Frontiers in Medicine artificial intelligence ChatGPT medical education machine learning obstetrics and gynecology students |
| title | ChatGPT’s performance in German OB/GYN exams – paving the way for AI-enhanced medical education and clinical practice |
| title_full | ChatGPT’s performance in German OB/GYN exams – paving the way for AI-enhanced medical education and clinical practice |
| title_fullStr | ChatGPT’s performance in German OB/GYN exams – paving the way for AI-enhanced medical education and clinical practice |
| title_full_unstemmed | ChatGPT’s performance in German OB/GYN exams – paving the way for AI-enhanced medical education and clinical practice |
| title_short | ChatGPT’s performance in German OB/GYN exams – paving the way for AI-enhanced medical education and clinical practice |
| title_sort | chatgpt s performance in german ob gyn exams paving the way for ai enhanced medical education and clinical practice |
| topic | artificial intelligence ChatGPT medical education machine learning obstetrics and gynecology students |
| url | https://www.frontiersin.org/articles/10.3389/fmed.2023.1296615/full |
| work_keys_str_mv | AT maximilianriedel chatgptsperformanceingermanobgynexamspavingthewayforaienhancedmedicaleducationandclinicalpractice AT katharinakaefinger chatgptsperformanceingermanobgynexamspavingthewayforaienhancedmedicaleducationandclinicalpractice AT antoniastuehrenberg chatgptsperformanceingermanobgynexamspavingthewayforaienhancedmedicaleducationandclinicalpractice AT viktoriaritter chatgptsperformanceingermanobgynexamspavingthewayforaienhancedmedicaleducationandclinicalpractice AT niklasamann chatgptsperformanceingermanobgynexamspavingthewayforaienhancedmedicaleducationandclinicalpractice AT annagraf chatgptsperformanceingermanobgynexamspavingthewayforaienhancedmedicaleducationandclinicalpractice AT florianrecker chatgptsperformanceingermanobgynexamspavingthewayforaienhancedmedicaleducationandclinicalpractice AT evelynklein chatgptsperformanceingermanobgynexamspavingthewayforaienhancedmedicaleducationandclinicalpractice AT marionkiechle chatgptsperformanceingermanobgynexamspavingthewayforaienhancedmedicaleducationandclinicalpractice AT fabianriedel chatgptsperformanceingermanobgynexamspavingthewayforaienhancedmedicaleducationandclinicalpractice AT bastianmeyer chatgptsperformanceingermanobgynexamspavingthewayforaienhancedmedicaleducationandclinicalpractice |