Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving
The study explores the capabilities of OpenAI's ChatGPT in solving different types of physics problems. ChatGPT (with GPT-4) was queried to solve a total of 40 problems from a college-level engineering physics course. These problems ranged from well-specified problems, where all data required f...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2024-01-01
|
| Series: | Frontiers in Education |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/feduc.2023.1330486/full |
| _version_ | 1797352546626437120 |
|---|---|
| author | Karen D. Wang Eric Burkholder Carl Wieman Carl Wieman Shima Salehi Nick Haber |
| author_facet | Karen D. Wang Eric Burkholder Carl Wieman Carl Wieman Shima Salehi Nick Haber |
| author_sort | Karen D. Wang |
| collection | DOAJ |
| description | The study explores the capabilities of OpenAI's ChatGPT in solving different types of physics problems. ChatGPT (with GPT-4) was queried to solve a total of 40 problems from a college-level engineering physics course. These problems ranged from well-specified problems, where all data required for solving the problem was provided, to under-specified, real-world problems where not all necessary data were given. Our findings show that ChatGPT could successfully solve 62.5% of the well-specified problems, but its accuracy drops to 8.3% for under-specified problems. Analysis of the model's incorrect solutions revealed three distinct failure modes: (1) failure to construct accurate models of the physical world, (2) failure to make reasonable assumptions about missing data, and (3) calculation errors. The study offers implications for how to leverage LLM-augmented instructional materials to enhance STEM education. The insights also contribute to the broader discourse on AI's strengths and limitations, serving both educators aiming to leverage the technology and researchers investigating human-AI collaboration frameworks for problem-solving and decision-making. |
| first_indexed | 2024-03-08T13:18:21Z |
| format | Article |
| id | doaj.art-3d60c42411d54f2ca99918f2e4cc519e |
| institution | Directory Open Access Journal |
| issn | 2504-284X |
| language | English |
| last_indexed | 2024-03-08T13:18:21Z |
| publishDate | 2024-01-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Education |
| spelling | doaj.art-3d60c42411d54f2ca99918f2e4cc519e2024-01-18T04:24:23ZengFrontiers Media S.A.Frontiers in Education2504-284X2024-01-01810.3389/feduc.2023.13304861330486Examining the potential and pitfalls of ChatGPT in science and engineering problem-solvingKaren D. Wang0Eric Burkholder1Carl Wieman2Carl Wieman3Shima Salehi4Nick Haber5Graduate School of Education, Stanford University, Stanford, CA, United StatesDepartment of Physics, Auburn University, Auburn, AL, United StatesGraduate School of Education, Stanford University, Stanford, CA, United StatesDepartment of Physics, Stanford University, Stanford, CA, United StatesGraduate School of Education, Stanford University, Stanford, CA, United StatesGraduate School of Education, Stanford University, Stanford, CA, United StatesThe study explores the capabilities of OpenAI's ChatGPT in solving different types of physics problems. ChatGPT (with GPT-4) was queried to solve a total of 40 problems from a college-level engineering physics course. These problems ranged from well-specified problems, where all data required for solving the problem was provided, to under-specified, real-world problems where not all necessary data were given. Our findings show that ChatGPT could successfully solve 62.5% of the well-specified problems, but its accuracy drops to 8.3% for under-specified problems. Analysis of the model's incorrect solutions revealed three distinct failure modes: (1) failure to construct accurate models of the physical world, (2) failure to make reasonable assumptions about missing data, and (3) calculation errors. The study offers implications for how to leverage LLM-augmented instructional materials to enhance STEM education. The insights also contribute to the broader discourse on AI's strengths and limitations, serving both educators aiming to leverage the technology and researchers investigating human-AI collaboration frameworks for problem-solving and decision-making.https://www.frontiersin.org/articles/10.3389/feduc.2023.1330486/fullChatGPTGPT-4generative AI modelsproblem-solvingauthentic problemsSTEM education |
| spellingShingle | Karen D. Wang Eric Burkholder Carl Wieman Carl Wieman Shima Salehi Nick Haber Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving Frontiers in Education ChatGPT GPT-4 generative AI models problem-solving authentic problems STEM education |
| title | Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving |
| title_full | Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving |
| title_fullStr | Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving |
| title_full_unstemmed | Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving |
| title_short | Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving |
| title_sort | examining the potential and pitfalls of chatgpt in science and engineering problem solving |
| topic | ChatGPT GPT-4 generative AI models problem-solving authentic problems STEM education |
| url | https://www.frontiersin.org/articles/10.3389/feduc.2023.1330486/full |
| work_keys_str_mv | AT karendwang examiningthepotentialandpitfallsofchatgptinscienceandengineeringproblemsolving AT ericburkholder examiningthepotentialandpitfallsofchatgptinscienceandengineeringproblemsolving AT carlwieman examiningthepotentialandpitfallsofchatgptinscienceandengineeringproblemsolving AT carlwieman examiningthepotentialandpitfallsofchatgptinscienceandengineeringproblemsolving AT shimasalehi examiningthepotentialandpitfallsofchatgptinscienceandengineeringproblemsolving AT nickhaber examiningthepotentialandpitfallsofchatgptinscienceandengineeringproblemsolving |