Automated Assessment of Comprehension Strategies from Self-Explanations Using LLMs
Text comprehension is an essential skill in today’s information-rich world, and self-explanation practice helps students improve their understanding of complex texts. This study was centered on leveraging open-source Large Language Models (LLMs), specifically FLAN-T5, to automatically assess the com...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-10-01
|
| Series: | Information |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2078-2489/14/10/567 |
| _version_ | 1827720847071117312 |
|---|---|
| author | Bogdan Nicula Mihai Dascalu Tracy Arner Renu Balyan Danielle S. McNamara |
| author_facet | Bogdan Nicula Mihai Dascalu Tracy Arner Renu Balyan Danielle S. McNamara |
| author_sort | Bogdan Nicula |
| collection | DOAJ |
| description | Text comprehension is an essential skill in today’s information-rich world, and self-explanation practice helps students improve their understanding of complex texts. This study was centered on leveraging open-source Large Language Models (LLMs), specifically FLAN-T5, to automatically assess the comprehension strategies employed by readers while understanding Science, Technology, Engineering, and Mathematics (STEM) texts. The experiments relied on a corpus of three datasets (N = 11,833) with self-explanations annotated on 4 dimensions: 3 comprehension strategies (i.e., bridging, elaboration, and paraphrasing) and overall quality. Besides FLAN-T5, we also considered GPT3.5-turbo to establish a stronger baseline. Our experiments indicated that the performance improved with fine-tuning, having a larger LLM model, and providing examples via the prompt. Our best model considered a pretrained FLAN-T5 XXL model and obtained a weighted F1-score of 0.721, surpassing the 0.699 F1-score previously obtained using smaller models (i.e., RoBERTa). |
| first_indexed | 2024-03-10T21:11:07Z |
| format | Article |
| id | doaj.art-bfaeca1d096f4e6491985b5ae61a3bfc |
| institution | Directory Open Access Journal |
| issn | 2078-2489 |
| language | English |
| last_indexed | 2024-03-10T21:11:07Z |
| publishDate | 2023-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Information |
| spelling | doaj.art-bfaeca1d096f4e6491985b5ae61a3bfc2023-11-19T16:48:17ZengMDPI AGInformation2078-24892023-10-01141056710.3390/info14100567Automated Assessment of Comprehension Strategies from Self-Explanations Using LLMsBogdan Nicula0Mihai Dascalu1Tracy Arner2Renu Balyan3Danielle S. McNamara4Computer Science and Engineering Department, National University of Science and Technology POLITEHNICA of Bucharest, 313 Splaiul Independentei, 060042 Bucharest, RomaniaComputer Science and Engineering Department, National University of Science and Technology POLITEHNICA of Bucharest, 313 Splaiul Independentei, 060042 Bucharest, RomaniaDepartment of Psychology, Arizona State University, P.O. Box 871104, Tempe, AZ 85287, USAMath/CIS Department, SUNY Old Westbury, Old Westbury, NY 11568, USADepartment of Psychology, Arizona State University, P.O. Box 871104, Tempe, AZ 85287, USAText comprehension is an essential skill in today’s information-rich world, and self-explanation practice helps students improve their understanding of complex texts. This study was centered on leveraging open-source Large Language Models (LLMs), specifically FLAN-T5, to automatically assess the comprehension strategies employed by readers while understanding Science, Technology, Engineering, and Mathematics (STEM) texts. The experiments relied on a corpus of three datasets (N = 11,833) with self-explanations annotated on 4 dimensions: 3 comprehension strategies (i.e., bridging, elaboration, and paraphrasing) and overall quality. Besides FLAN-T5, we also considered GPT3.5-turbo to establish a stronger baseline. Our experiments indicated that the performance improved with fine-tuning, having a larger LLM model, and providing examples via the prompt. Our best model considered a pretrained FLAN-T5 XXL model and obtained a weighted F1-score of 0.721, surpassing the 0.699 F1-score previously obtained using smaller models (i.e., RoBERTa).https://www.mdpi.com/2078-2489/14/10/567language modelslarge language modelsself-explanationself-explanation strategies |
| spellingShingle | Bogdan Nicula Mihai Dascalu Tracy Arner Renu Balyan Danielle S. McNamara Automated Assessment of Comprehension Strategies from Self-Explanations Using LLMs Information language models large language models self-explanation self-explanation strategies |
| title | Automated Assessment of Comprehension Strategies from Self-Explanations Using LLMs |
| title_full | Automated Assessment of Comprehension Strategies from Self-Explanations Using LLMs |
| title_fullStr | Automated Assessment of Comprehension Strategies from Self-Explanations Using LLMs |
| title_full_unstemmed | Automated Assessment of Comprehension Strategies from Self-Explanations Using LLMs |
| title_short | Automated Assessment of Comprehension Strategies from Self-Explanations Using LLMs |
| title_sort | automated assessment of comprehension strategies from self explanations using llms |
| topic | language models large language models self-explanation self-explanation strategies |
| url | https://www.mdpi.com/2078-2489/14/10/567 |
| work_keys_str_mv | AT bogdannicula automatedassessmentofcomprehensionstrategiesfromselfexplanationsusingllms AT mihaidascalu automatedassessmentofcomprehensionstrategiesfromselfexplanationsusingllms AT tracyarner automatedassessmentofcomprehensionstrategiesfromselfexplanationsusingllms AT renubalyan automatedassessmentofcomprehensionstrategiesfromselfexplanationsusingllms AT daniellesmcnamara automatedassessmentofcomprehensionstrategiesfromselfexplanationsusingllms |