Comparison of the Evaluation Metrics for Neural Grammatical Error Correction With Overcorrection
Grammar error correction (GEC) refers to the proper correction of grammatical errors in a given sentence. Important factors to consider in GEC are not only the grammatical correction of the sentence, but also the recognition of a correct sentence in which no changes are required. However, GEC approa...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2020-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9102992/ |
| _version_ | 1818616638699732992 |
|---|---|
| author | Chanjun Park Yeongwook Yang Chanhee Lee Heuiseok Lim |
| author_facet | Chanjun Park Yeongwook Yang Chanhee Lee Heuiseok Lim |
| author_sort | Chanjun Park |
| collection | DOAJ |
| description | Grammar error correction (GEC) refers to the proper correction of grammatical errors in a given sentence. Important factors to consider in GEC are not only the grammatical correction of the sentence, but also the recognition of a correct sentence in which no changes are required. However, GEC approaches in which deep learning recently started being used consider only the former aspect, which leads to overcorrection, whereby changes are made to a correct sentence unnecessarily. Because this bias is also reflected in performance metrics, conventional performance metrics consider only part of the important factors in GEC. This study proposes a new metric to consider both important aspects in GEC and to provide a new viewpoint for the GEC task. To the best of the authors knowledge, this study is the first to deal with comprehensively considering the correction performance and overcorrection problem in GEC. The experimental results demonstrate that the model performance ranking was reversed when evaluating the performance with the proposed metric compared to the General Language Understanding Evaluation benchmark [21], which only considers the correction performance. This indicates that the high performance of the correction does not result in less problems with the overcorrection and that the overcorrection problem should also be considered when evaluating the model performance. Moreover, we found that the copy mechanism [14] helps to alleviate the problem of overcorrection. |
| first_indexed | 2024-12-16T16:52:59Z |
| format | Article |
| id | doaj.art-8787a4a400744e489223cee7d7f8fb1b |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-16T16:52:59Z |
| publishDate | 2020-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-8787a4a400744e489223cee7d7f8fb1b2022-12-21T22:23:58ZengIEEEIEEE Access2169-35362020-01-01810626410627210.1109/ACCESS.2020.29981499102992Comparison of the Evaluation Metrics for Neural Grammatical Error Correction With OvercorrectionChanjun Park0Yeongwook Yang1https://orcid.org/0000-0003-3219-7250Chanhee Lee2https://orcid.org/0000-0002-9300-5251Heuiseok Lim3https://orcid.org/0000-0002-9269-1157Department of Computer Science and Engineering, Korea University, Seoul, South KoreaCenter for Educational Technology, Institute of Education, University of Tartu, Tartu, EstoniaDepartment of Computer Science and Engineering, Korea University, Seoul, South KoreaDepartment of Computer Science and Engineering, Korea University, Seoul, South KoreaGrammar error correction (GEC) refers to the proper correction of grammatical errors in a given sentence. Important factors to consider in GEC are not only the grammatical correction of the sentence, but also the recognition of a correct sentence in which no changes are required. However, GEC approaches in which deep learning recently started being used consider only the former aspect, which leads to overcorrection, whereby changes are made to a correct sentence unnecessarily. Because this bias is also reflected in performance metrics, conventional performance metrics consider only part of the important factors in GEC. This study proposes a new metric to consider both important aspects in GEC and to provide a new viewpoint for the GEC task. To the best of the authors knowledge, this study is the first to deal with comprehensively considering the correction performance and overcorrection problem in GEC. The experimental results demonstrate that the model performance ranking was reversed when evaluating the performance with the proposed metric compared to the General Language Understanding Evaluation benchmark [21], which only considers the correction performance. This indicates that the high performance of the correction does not result in less problems with the overcorrection and that the overcorrection problem should also be considered when evaluating the model performance. Moreover, we found that the copy mechanism [14] helps to alleviate the problem of overcorrection.https://ieeexplore.ieee.org/document/9102992/Grammar error correctionovercorrectionneural machine translationcopy mechanismmetric |
| spellingShingle | Chanjun Park Yeongwook Yang Chanhee Lee Heuiseok Lim Comparison of the Evaluation Metrics for Neural Grammatical Error Correction With Overcorrection IEEE Access Grammar error correction overcorrection neural machine translation copy mechanism metric |
| title | Comparison of the Evaluation Metrics for Neural Grammatical Error Correction With Overcorrection |
| title_full | Comparison of the Evaluation Metrics for Neural Grammatical Error Correction With Overcorrection |
| title_fullStr | Comparison of the Evaluation Metrics for Neural Grammatical Error Correction With Overcorrection |
| title_full_unstemmed | Comparison of the Evaluation Metrics for Neural Grammatical Error Correction With Overcorrection |
| title_short | Comparison of the Evaluation Metrics for Neural Grammatical Error Correction With Overcorrection |
| title_sort | comparison of the evaluation metrics for neural grammatical error correction with overcorrection |
| topic | Grammar error correction overcorrection neural machine translation copy mechanism metric |
| url | https://ieeexplore.ieee.org/document/9102992/ |
| work_keys_str_mv | AT chanjunpark comparisonoftheevaluationmetricsforneuralgrammaticalerrorcorrectionwithovercorrection AT yeongwookyang comparisonoftheevaluationmetricsforneuralgrammaticalerrorcorrectionwithovercorrection AT chanheelee comparisonoftheevaluationmetricsforneuralgrammaticalerrorcorrectionwithovercorrection AT heuiseoklim comparisonoftheevaluationmetricsforneuralgrammaticalerrorcorrectionwithovercorrection |