AI-generated feedback on writing: insights into efficacy and ENL student preference
Abstract The question of how generative AI tools, such as large language models and chatbots, can be leveraged ethically and effectively in education is ongoing. Given the critical role that writing plays in learning and assessment within educational institutions, it is of growing importance for edu...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2023-10-01
|
| Series: | International Journal of Educational Technology in Higher Education |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s41239-023-00425-2 |
| _version_ | 1797647212055887872 |
|---|---|
| author | Juan Escalante Austin Pack Alex Barrett |
| author_facet | Juan Escalante Austin Pack Alex Barrett |
| author_sort | Juan Escalante |
| collection | DOAJ |
| description | Abstract The question of how generative AI tools, such as large language models and chatbots, can be leveraged ethically and effectively in education is ongoing. Given the critical role that writing plays in learning and assessment within educational institutions, it is of growing importance for educators to make thoughtful and informed decisions as to how and in what capacity generative AI tools should be leveraged to assist in the development of students’ writing skills. This paper reports on two longitudinal studies. Study 1 examined learning outcomes of 48 university English as a new language (ENL) learners in a six-week long repeated measures quasi experimental design where the experimental group received writing feedback generated from ChatGPT (GPT-4) and the control group received feedback from their human tutor. Study 2 analyzed the perceptions of a different group of 43 ENLs who received feedback from both ChatGPT and their tutor. Results of study 1 showed no difference in learning outcomes between the two groups. Study 2 results revealed a near even split in preference for AI-generated or human-generated feedback, with clear advantages to both forms of feedback apparent from the data. The main implication of these studies is that the use of AI-generated feedback can likely be incorporated into ENL essay evaluation without affecting learning outcomes, although we recommend a blended approach that utilizes the strengths of both forms of feedback. The main contribution of this paper is in addressing generative AI as an automatic essay evaluator while incorporating learner perspectives. |
| first_indexed | 2024-03-11T15:14:01Z |
| format | Article |
| id | doaj.art-a43ee9464eb041d3a7fbbd2bcf0122aa |
| institution | Directory Open Access Journal |
| issn | 2365-9440 |
| language | English |
| last_indexed | 2024-03-11T15:14:01Z |
| publishDate | 2023-10-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | International Journal of Educational Technology in Higher Education |
| spelling | doaj.art-a43ee9464eb041d3a7fbbd2bcf0122aa2023-10-29T12:32:13ZengSpringerOpenInternational Journal of Educational Technology in Higher Education2365-94402023-10-0120112010.1186/s41239-023-00425-2AI-generated feedback on writing: insights into efficacy and ENL student preferenceJuan Escalante0Austin Pack1Alex Barrett2Faculty of Education and Social Work, Brigham Young University-HawaiiFaculty of Education and Social Work, Brigham Young University-HawaiiCollege of Education, Florida State UniversityAbstract The question of how generative AI tools, such as large language models and chatbots, can be leveraged ethically and effectively in education is ongoing. Given the critical role that writing plays in learning and assessment within educational institutions, it is of growing importance for educators to make thoughtful and informed decisions as to how and in what capacity generative AI tools should be leveraged to assist in the development of students’ writing skills. This paper reports on two longitudinal studies. Study 1 examined learning outcomes of 48 university English as a new language (ENL) learners in a six-week long repeated measures quasi experimental design where the experimental group received writing feedback generated from ChatGPT (GPT-4) and the control group received feedback from their human tutor. Study 2 analyzed the perceptions of a different group of 43 ENLs who received feedback from both ChatGPT and their tutor. Results of study 1 showed no difference in learning outcomes between the two groups. Study 2 results revealed a near even split in preference for AI-generated or human-generated feedback, with clear advantages to both forms of feedback apparent from the data. The main implication of these studies is that the use of AI-generated feedback can likely be incorporated into ENL essay evaluation without affecting learning outcomes, although we recommend a blended approach that utilizes the strengths of both forms of feedback. The main contribution of this paper is in addressing generative AI as an automatic essay evaluator while incorporating learner perspectives.https://doi.org/10.1186/s41239-023-00425-2Automated writing evaluationChatGPTArtificial intelligenceLanguage education |
| spellingShingle | Juan Escalante Austin Pack Alex Barrett AI-generated feedback on writing: insights into efficacy and ENL student preference International Journal of Educational Technology in Higher Education Automated writing evaluation ChatGPT Artificial intelligence Language education |
| title | AI-generated feedback on writing: insights into efficacy and ENL student preference |
| title_full | AI-generated feedback on writing: insights into efficacy and ENL student preference |
| title_fullStr | AI-generated feedback on writing: insights into efficacy and ENL student preference |
| title_full_unstemmed | AI-generated feedback on writing: insights into efficacy and ENL student preference |
| title_short | AI-generated feedback on writing: insights into efficacy and ENL student preference |
| title_sort | ai generated feedback on writing insights into efficacy and enl student preference |
| topic | Automated writing evaluation ChatGPT Artificial intelligence Language education |
| url | https://doi.org/10.1186/s41239-023-00425-2 |
| work_keys_str_mv | AT juanescalante aigeneratedfeedbackonwritinginsightsintoefficacyandenlstudentpreference AT austinpack aigeneratedfeedbackonwritinginsightsintoefficacyandenlstudentpreference AT alexbarrett aigeneratedfeedbackonwritinginsightsintoefficacyandenlstudentpreference |