Performance of ChatGPT on the test of understanding graphs in kinematics
The well-known artificial intelligence-based chatbot ChatGPT-4 has become able to process image data as input in October 2023. We investigated its performance on the test of understanding graphs in kinematics to inform the physics education community of the current potential of using ChatGPT in the...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
American Physical Society
2024-02-01
|
| Series: | Physical Review Physics Education Research |
| Online Access: | http://doi.org/10.1103/PhysRevPhysEducRes.20.010109 |
| _version_ | 1797298894093156352 |
|---|---|
| author | Giulia Polverini Bor Gregorcic |
| author_facet | Giulia Polverini Bor Gregorcic |
| author_sort | Giulia Polverini |
| collection | DOAJ |
| description | The well-known artificial intelligence-based chatbot ChatGPT-4 has become able to process image data as input in October 2023. We investigated its performance on the test of understanding graphs in kinematics to inform the physics education community of the current potential of using ChatGPT in the education process, particularly on tasks that involve graphical interpretation. We found that ChatGPT, on average, performed similarly to students taking a high-school level physics course, but with important differences in the distribution of the correctness of its responses, as well as in terms of the displayed “reasoning” and “visual” abilities. While ChatGPT was very successful at proposing productive strategies for solving the tasks on the test and expressed correct reasoning in most of its responses, it had difficulties correctly “seeing” graphs. We suggest that, based on its performance, caution and a critical approach are needed if one intends to use it in the role of a tutor, a model of a student, or a tool for assisting vision-impaired persons in the context of kinematics graphs. |
| first_indexed | 2024-03-07T22:42:43Z |
| format | Article |
| id | doaj.art-79e95ade20174870b38e1f4ceb4f8a16 |
| institution | Directory Open Access Journal |
| issn | 2469-9896 |
| language | English |
| last_indexed | 2024-03-07T22:42:43Z |
| publishDate | 2024-02-01 |
| publisher | American Physical Society |
| record_format | Article |
| series | Physical Review Physics Education Research |
| spelling | doaj.art-79e95ade20174870b38e1f4ceb4f8a162024-02-23T15:07:41ZengAmerican Physical SocietyPhysical Review Physics Education Research2469-98962024-02-0120101010910.1103/PhysRevPhysEducRes.20.010109Performance of ChatGPT on the test of understanding graphs in kinematicsGiulia PolveriniBor GregorcicThe well-known artificial intelligence-based chatbot ChatGPT-4 has become able to process image data as input in October 2023. We investigated its performance on the test of understanding graphs in kinematics to inform the physics education community of the current potential of using ChatGPT in the education process, particularly on tasks that involve graphical interpretation. We found that ChatGPT, on average, performed similarly to students taking a high-school level physics course, but with important differences in the distribution of the correctness of its responses, as well as in terms of the displayed “reasoning” and “visual” abilities. While ChatGPT was very successful at proposing productive strategies for solving the tasks on the test and expressed correct reasoning in most of its responses, it had difficulties correctly “seeing” graphs. We suggest that, based on its performance, caution and a critical approach are needed if one intends to use it in the role of a tutor, a model of a student, or a tool for assisting vision-impaired persons in the context of kinematics graphs.http://doi.org/10.1103/PhysRevPhysEducRes.20.010109 |
| spellingShingle | Giulia Polverini Bor Gregorcic Performance of ChatGPT on the test of understanding graphs in kinematics Physical Review Physics Education Research |
| title | Performance of ChatGPT on the test of understanding graphs in kinematics |
| title_full | Performance of ChatGPT on the test of understanding graphs in kinematics |
| title_fullStr | Performance of ChatGPT on the test of understanding graphs in kinematics |
| title_full_unstemmed | Performance of ChatGPT on the test of understanding graphs in kinematics |
| title_short | Performance of ChatGPT on the test of understanding graphs in kinematics |
| title_sort | performance of chatgpt on the test of understanding graphs in kinematics |
| url | http://doi.org/10.1103/PhysRevPhysEducRes.20.010109 |
| work_keys_str_mv | AT giuliapolverini performanceofchatgptonthetestofunderstandinggraphsinkinematics AT borgregorcic performanceofchatgptonthetestofunderstandinggraphsinkinematics |