Comparing ChatGPT’s ability to rate the degree of stereotypes and the consistency of stereotype attribution with those of medical students in New Zealand in developing a similarity rating test: a methodological study
Learning about one’s implicit bias is crucial for improving one’s cultural competency and thereby reducing health inequity. To evaluate bias among medical students following a previously developed cultural training program targeting New Zealand Māori, we developed a text-based, self-evaluation tool...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Korea Health Personnel Licensing Examination Institute
2023-06-01
|
| Series: | Journal of Educational Evaluation for Health Professions |
| Subjects: | |
| Online Access: | http://www.jeehp.org/upload/jeehp-20-17.pdf |
| _version_ | 1797787266080309248 |
|---|---|
| author | Chao-Cheng Lin Zaine Akuhata-Huntington Che-Wei Hsu |
| author_facet | Chao-Cheng Lin Zaine Akuhata-Huntington Che-Wei Hsu |
| author_sort | Chao-Cheng Lin |
| collection | DOAJ |
| description | Learning about one’s implicit bias is crucial for improving one’s cultural competency and thereby reducing health inequity. To evaluate bias among medical students following a previously developed cultural training program targeting New Zealand Māori, we developed a text-based, self-evaluation tool called the Similarity Rating Test (SRT). The development process of the SRT was resource-intensive, limiting its generalizability and applicability. Here, we explored the potential of ChatGPT, an automated chatbot, to assist in the development process of the SRT by comparing ChatGPT’s and students’ evaluations of the SRT. Despite results showing non-significant equivalence and difference between ChatGPT’s and students’ ratings, ChatGPT’s ratings were more consistent than students’ ratings. The consistency rate was higher for non-stereotypical than for stereotypical statements, regardless of rater type. Further studies are warranted to validate ChatGPT’s potential for assisting in SRT development for implementation in medical education and evaluation of ethnic stereotypes and related topics. |
| first_indexed | 2024-03-13T01:20:25Z |
| format | Article |
| id | doaj.art-83b26fed941643eb9a7dcce902d30cb3 |
| institution | Directory Open Access Journal |
| issn | 1975-5937 |
| language | English |
| last_indexed | 2024-03-13T01:20:25Z |
| publishDate | 2023-06-01 |
| publisher | Korea Health Personnel Licensing Examination Institute |
| record_format | Article |
| series | Journal of Educational Evaluation for Health Professions |
| spelling | doaj.art-83b26fed941643eb9a7dcce902d30cb32023-07-05T04:47:44ZengKorea Health Personnel Licensing Examination InstituteJournal of Educational Evaluation for Health Professions1975-59372023-06-012010.3352/jeehp.2023.20.17481Comparing ChatGPT’s ability to rate the degree of stereotypes and the consistency of stereotype attribution with those of medical students in New Zealand in developing a similarity rating test: a methodological studyChao-Cheng Lin0Zaine Akuhata-Huntington1Che-Wei Hsu2Department of Psychological Medicine, Dunedin School of Medicine, the University of Otago, Dunedin, New ZealandKōhatu Centre for Hauora Māori, Dunedin School of Medicine, the University of Otago, Dunedin, New ZealandDepartment of Psychological Medicine, Dunedin School of Medicine, the University of Otago, Dunedin, New ZealandLearning about one’s implicit bias is crucial for improving one’s cultural competency and thereby reducing health inequity. To evaluate bias among medical students following a previously developed cultural training program targeting New Zealand Māori, we developed a text-based, self-evaluation tool called the Similarity Rating Test (SRT). The development process of the SRT was resource-intensive, limiting its generalizability and applicability. Here, we explored the potential of ChatGPT, an automated chatbot, to assist in the development process of the SRT by comparing ChatGPT’s and students’ evaluations of the SRT. Despite results showing non-significant equivalence and difference between ChatGPT’s and students’ ratings, ChatGPT’s ratings were more consistent than students’ ratings. The consistency rate was higher for non-stereotypical than for stereotypical statements, regardless of rater type. Further studies are warranted to validate ChatGPT’s potential for assisting in SRT development for implementation in medical education and evaluation of ethnic stereotypes and related topics.http://www.jeehp.org/upload/jeehp-20-17.pdfartificial intelligencecultural competencyimplicit biasmedical educationnew zealand |
| spellingShingle | Chao-Cheng Lin Zaine Akuhata-Huntington Che-Wei Hsu Comparing ChatGPT’s ability to rate the degree of stereotypes and the consistency of stereotype attribution with those of medical students in New Zealand in developing a similarity rating test: a methodological study Journal of Educational Evaluation for Health Professions artificial intelligence cultural competency implicit bias medical education new zealand |
| title | Comparing ChatGPT’s ability to rate the degree of stereotypes and the consistency of stereotype attribution with those of medical students in New Zealand in developing a similarity rating test: a methodological study |
| title_full | Comparing ChatGPT’s ability to rate the degree of stereotypes and the consistency of stereotype attribution with those of medical students in New Zealand in developing a similarity rating test: a methodological study |
| title_fullStr | Comparing ChatGPT’s ability to rate the degree of stereotypes and the consistency of stereotype attribution with those of medical students in New Zealand in developing a similarity rating test: a methodological study |
| title_full_unstemmed | Comparing ChatGPT’s ability to rate the degree of stereotypes and the consistency of stereotype attribution with those of medical students in New Zealand in developing a similarity rating test: a methodological study |
| title_short | Comparing ChatGPT’s ability to rate the degree of stereotypes and the consistency of stereotype attribution with those of medical students in New Zealand in developing a similarity rating test: a methodological study |
| title_sort | comparing chatgpt s ability to rate the degree of stereotypes and the consistency of stereotype attribution with those of medical students in new zealand in developing a similarity rating test a methodological study |
| topic | artificial intelligence cultural competency implicit bias medical education new zealand |
| url | http://www.jeehp.org/upload/jeehp-20-17.pdf |
| work_keys_str_mv | AT chaochenglin comparingchatgptsabilitytoratethedegreeofstereotypesandtheconsistencyofstereotypeattributionwiththoseofmedicalstudentsinnewzealandindevelopingasimilarityratingtestamethodologicalstudy AT zaineakuhatahuntington comparingchatgptsabilitytoratethedegreeofstereotypesandtheconsistencyofstereotypeattributionwiththoseofmedicalstudentsinnewzealandindevelopingasimilarityratingtestamethodologicalstudy AT cheweihsu comparingchatgptsabilitytoratethedegreeofstereotypesandtheconsistencyofstereotypeattributionwiththoseofmedicalstudentsinnewzealandindevelopingasimilarityratingtestamethodologicalstudy |