Crowdsourcing-based automated essay scoring framework
Automated Essay Scoring (AES) is a challenging topic in Natural Language Processing. Although deep learning models achieve remarkable performance for the AES task, they have two major problems. Most AES models are unable to handle the out-of-vocabulary (OOV) words. Besides, training AES models is...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis-Master by Research |
| Language: | English |
| Published: |
Nanyang Technological University
2023
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/164525 |
| _version_ | 1824455860722597888 |
|---|---|
| author | Bai, Huanyu |
| author2 | Hui Siu Cheung |
| author_facet | Hui Siu Cheung Bai, Huanyu |
| author_sort | Bai, Huanyu |
| collection | NTU |
| description | Automated Essay Scoring (AES) is a challenging topic in Natural Language Processing. Although deep learning models achieve remarkable performance for the AES task, they have two major problems. Most AES models are unable to handle the out-of-vocabulary (OOV) words. Besides, training AES models is costly in practice. This thesis aims to handle these two problems.
To tackle the OOV problem, this thesis proposes the Gated Character-aware Convolutional Neural Network (GCCNN) model. The GCCNN model incorporates character-level information into the AES model by using a vector gating mechanism to fuse the word-level and character-level information. The experimental results show that the proposed GCCNN model outperforms several strong baseline models. In addition, the qualitative analysis demonstrates the importance of character-level information for tackling the OOV problem for the AES task.
Moreover, this thesis proposes the Incremental Learning with Dynamic Exemplar Herding (ILDEH) approach to efficiently train the AES models. The ILDEH approach trains the AES model in the crowdsourcing environment. To effectively improve the AES performance, the ILDEH approach simultaneously tackles catastrophic forgetting and concept drift by the Linear Outlier Suppression loss and Dynamic Exemplar Herding algorithm. The experimental results show that the ILDEH approach outperforms all baseline approaches and significantly reduces the training time. Note that the ILDEH approach is model-agnostic and can also be applied to other classification tasks.
Lastly, this thesis proposes the Crowdsourcing-based Automated Essay Scoring (CAES) framework by integrating the GCCNN model and the ILDEH approach. The CAES framework collects graded essays online through crowdsourcing. The ILDEH approach is used to incrementally train the GCCNN models, which are used to provide instant essay assessment service. By using the CAES framework, one single CPU server is sufficient for most AES systems. |
| first_indexed | 2025-02-19T03:44:55Z |
| format | Thesis-Master by Research |
| id | ntu-10356/164525 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2025-02-19T03:44:55Z |
| publishDate | 2023 |
| publisher | Nanyang Technological University |
| record_format | dspace |
| spelling | ntu-10356/1645252023-02-01T03:20:56Z Crowdsourcing-based automated essay scoring framework Bai, Huanyu Hui Siu Cheung School of Computer Science and Engineering ASSCHUI@ntu.edu.sg Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence Automated Essay Scoring (AES) is a challenging topic in Natural Language Processing. Although deep learning models achieve remarkable performance for the AES task, they have two major problems. Most AES models are unable to handle the out-of-vocabulary (OOV) words. Besides, training AES models is costly in practice. This thesis aims to handle these two problems. To tackle the OOV problem, this thesis proposes the Gated Character-aware Convolutional Neural Network (GCCNN) model. The GCCNN model incorporates character-level information into the AES model by using a vector gating mechanism to fuse the word-level and character-level information. The experimental results show that the proposed GCCNN model outperforms several strong baseline models. In addition, the qualitative analysis demonstrates the importance of character-level information for tackling the OOV problem for the AES task. Moreover, this thesis proposes the Incremental Learning with Dynamic Exemplar Herding (ILDEH) approach to efficiently train the AES models. The ILDEH approach trains the AES model in the crowdsourcing environment. To effectively improve the AES performance, the ILDEH approach simultaneously tackles catastrophic forgetting and concept drift by the Linear Outlier Suppression loss and Dynamic Exemplar Herding algorithm. The experimental results show that the ILDEH approach outperforms all baseline approaches and significantly reduces the training time. Note that the ILDEH approach is model-agnostic and can also be applied to other classification tasks. Lastly, this thesis proposes the Crowdsourcing-based Automated Essay Scoring (CAES) framework by integrating the GCCNN model and the ILDEH approach. The CAES framework collects graded essays online through crowdsourcing. The ILDEH approach is used to incrementally train the GCCNN models, which are used to provide instant essay assessment service. By using the CAES framework, one single CPU server is sufficient for most AES systems. Master of Engineering 2023-01-31T05:10:54Z 2023-01-31T05:10:54Z 2022 Thesis-Master by Research Bai, H. (2022). Crowdsourcing-based automated essay scoring framework. Master's thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/164525 https://hdl.handle.net/10356/164525 10.32657/10356/164525 en This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0). application/pdf Nanyang Technological University |
| spellingShingle | Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence Bai, Huanyu Crowdsourcing-based automated essay scoring framework |
| title | Crowdsourcing-based automated essay scoring framework |
| title_full | Crowdsourcing-based automated essay scoring framework |
| title_fullStr | Crowdsourcing-based automated essay scoring framework |
| title_full_unstemmed | Crowdsourcing-based automated essay scoring framework |
| title_short | Crowdsourcing-based automated essay scoring framework |
| title_sort | crowdsourcing based automated essay scoring framework |
| topic | Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence |
| url | https://hdl.handle.net/10356/164525 |
| work_keys_str_mv | AT baihuanyu crowdsourcingbasedautomatedessayscoringframework |