Students’ use of large language models in engineering education: A case study on technology acceptance, perceptions, efficacy, and detection chances
The accessibility of advanced Artificial Intelligence-based tools, like ChatGPT, has made Large Language Models (LLMs) readily available to students. These LLMs can generate original written content to assist students in their academic assessments. With the rapid adoption of LLMs, exemplified by the...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-01-01
|
| Series: | Computers and Education: Artificial Intelligence |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666920X23000516 |
| _version_ | 1797383674419740672 |
|---|---|
| author | Margherita Bernabei Silvia Colabianchi Andrea Falegnami Francesco Costantino |
| author_facet | Margherita Bernabei Silvia Colabianchi Andrea Falegnami Francesco Costantino |
| author_sort | Margherita Bernabei |
| collection | DOAJ |
| description | The accessibility of advanced Artificial Intelligence-based tools, like ChatGPT, has made Large Language Models (LLMs) readily available to students. These LLMs can generate original written content to assist students in their academic assessments. With the rapid adoption of LLMs, exemplified by the popularity of OpenAI's ChatGPT, there is a growing need to explore their application in education. Few studies examine students' use of LLMs as learning tools. This paper focuses on the application of ChatGPT in engineering higher education through an in-depth case study. It investigates whether engineering students can generate high-quality university essays with LLMs assistance, whether existing LLMs identification systems can detect essays produced with LLMs, and how students perceive the usefulness and acceptance of LLMs in learning. The research adopts a deductive/inductive approach, combining conceptualization and empirical evidence analysis. The study involves mechanical and management engineering students, who compose essays using LLMs. The essay assessment showed good results, but some recommendations emerged for teachers and students. Thirteen LLMs detectors were tested without achieving satisfactory results, suggesting to avoid LLMs ban. In addition, students were administered a questionnaire based on constructs and items that follow the technology acceptance models available in the literature. The results contribute to qualitative evidence by highlighting possible future research and educational practices. |
| first_indexed | 2024-03-08T21:24:28Z |
| format | Article |
| id | doaj.art-aef5a8508e464472be2bb5cc3a24e60e |
| institution | Directory Open Access Journal |
| issn | 2666-920X |
| language | English |
| last_indexed | 2024-03-08T21:24:28Z |
| publishDate | 2023-01-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Computers and Education: Artificial Intelligence |
| spelling | doaj.art-aef5a8508e464472be2bb5cc3a24e60e2023-12-21T07:38:05ZengElsevierComputers and Education: Artificial Intelligence2666-920X2023-01-015100172Students’ use of large language models in engineering education: A case study on technology acceptance, perceptions, efficacy, and detection chancesMargherita Bernabei0Silvia Colabianchi1Andrea Falegnami2Francesco Costantino3Department of Mechanical and Aerospace Engineering, Sapienza University of Rome, Via Eudossiana 18, 00184, Rome, ItalyDepartment of Computer, Control, and Management Engineering Antonio Ruberti, Sapienza University of Rome, Via Ariosto 25, 00185, Rome, ItalyManagement Engineering Faculty, Uninettuno University, Corso Vittorio Emanuele II 39 00186, Rome, ItalyDepartment of Computer, Control, and Management Engineering Antonio Ruberti, Sapienza University of Rome, Via Ariosto 25, 00185, Rome, Italy; Corresponding author. Via Ariosto 25, 00185, Rome, Italy.The accessibility of advanced Artificial Intelligence-based tools, like ChatGPT, has made Large Language Models (LLMs) readily available to students. These LLMs can generate original written content to assist students in their academic assessments. With the rapid adoption of LLMs, exemplified by the popularity of OpenAI's ChatGPT, there is a growing need to explore their application in education. Few studies examine students' use of LLMs as learning tools. This paper focuses on the application of ChatGPT in engineering higher education through an in-depth case study. It investigates whether engineering students can generate high-quality university essays with LLMs assistance, whether existing LLMs identification systems can detect essays produced with LLMs, and how students perceive the usefulness and acceptance of LLMs in learning. The research adopts a deductive/inductive approach, combining conceptualization and empirical evidence analysis. The study involves mechanical and management engineering students, who compose essays using LLMs. The essay assessment showed good results, but some recommendations emerged for teachers and students. Thirteen LLMs detectors were tested without achieving satisfactory results, suggesting to avoid LLMs ban. In addition, students were administered a questionnaire based on constructs and items that follow the technology acceptance models available in the literature. The results contribute to qualitative evidence by highlighting possible future research and educational practices.http://www.sciencedirect.com/science/article/pii/S2666920X23000516LLMChatGPTHigher educationEssay generation |
| spellingShingle | Margherita Bernabei Silvia Colabianchi Andrea Falegnami Francesco Costantino Students’ use of large language models in engineering education: A case study on technology acceptance, perceptions, efficacy, and detection chances Computers and Education: Artificial Intelligence LLM ChatGPT Higher education Essay generation |
| title | Students’ use of large language models in engineering education: A case study on technology acceptance, perceptions, efficacy, and detection chances |
| title_full | Students’ use of large language models in engineering education: A case study on technology acceptance, perceptions, efficacy, and detection chances |
| title_fullStr | Students’ use of large language models in engineering education: A case study on technology acceptance, perceptions, efficacy, and detection chances |
| title_full_unstemmed | Students’ use of large language models in engineering education: A case study on technology acceptance, perceptions, efficacy, and detection chances |
| title_short | Students’ use of large language models in engineering education: A case study on technology acceptance, perceptions, efficacy, and detection chances |
| title_sort | students use of large language models in engineering education a case study on technology acceptance perceptions efficacy and detection chances |
| topic | LLM ChatGPT Higher education Essay generation |
| url | http://www.sciencedirect.com/science/article/pii/S2666920X23000516 |
| work_keys_str_mv | AT margheritabernabei studentsuseoflargelanguagemodelsinengineeringeducationacasestudyontechnologyacceptanceperceptionsefficacyanddetectionchances AT silviacolabianchi studentsuseoflargelanguagemodelsinengineeringeducationacasestudyontechnologyacceptanceperceptionsefficacyanddetectionchances AT andreafalegnami studentsuseoflargelanguagemodelsinengineeringeducationacasestudyontechnologyacceptanceperceptionsefficacyanddetectionchances AT francescocostantino studentsuseoflargelanguagemodelsinengineeringeducationacasestudyontechnologyacceptanceperceptionsefficacyanddetectionchances |