MeaningBERT: assessing meaning preservation between sentences
In the field of automatic text simplification, assessing whether or not the meaning of the original text has been preserved during simplification is of paramount importance. Metrics relying on n-gram overlap assessment may struggle to deal with simplifications which replace complex phrases with thei...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-09-01
|
| Series: | Frontiers in Artificial Intelligence |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/frai.2023.1223924/full |
| _version_ | 1797676783698444288 |
|---|---|
| author | David Beauchemin Horacio Saggion Richard Khoury |
| author_facet | David Beauchemin Horacio Saggion Richard Khoury |
| author_sort | David Beauchemin |
| collection | DOAJ |
| description | In the field of automatic text simplification, assessing whether or not the meaning of the original text has been preserved during simplification is of paramount importance. Metrics relying on n-gram overlap assessment may struggle to deal with simplifications which replace complex phrases with their simpler paraphrases. Current evaluation metrics for meaning preservation based on large language models (LLMs), such as BertScore in machine translation or QuestEval in summarization, have been proposed. However, none has a strong correlation with human judgment of meaning preservation. Moreover, such metrics have not been assessed in the context of text simplification research. In this study, we present a meta-evaluation of several metrics we apply to measure content similarity in text simplification. We also show that the metrics are unable to pass two trivial, inexpensive content preservation tests. Another contribution of this study is MeaningBERT (https://github.com/GRAAL-Research/MeaningBERT), a new trainable metric designed to assess meaning preservation between two sentences in text simplification, showing how it correlates with human judgment. To demonstrate its quality and versatility, we will also present a compilation of datasets used to assess meaning preservation and benchmark our study against a large selection of popular metrics. |
| first_indexed | 2024-03-11T22:35:21Z |
| format | Article |
| id | doaj.art-e939175f0d3d4afaa9dc78b709a92d70 |
| institution | Directory Open Access Journal |
| issn | 2624-8212 |
| language | English |
| last_indexed | 2024-03-11T22:35:21Z |
| publishDate | 2023-09-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Artificial Intelligence |
| spelling | doaj.art-e939175f0d3d4afaa9dc78b709a92d702023-09-22T19:08:47ZengFrontiers Media S.A.Frontiers in Artificial Intelligence2624-82122023-09-01610.3389/frai.2023.12239241223924MeaningBERT: assessing meaning preservation between sentencesDavid Beauchemin0Horacio Saggion1Richard Khoury2Group for Research in Artificial Intelligence of Laval University, Department of Computer Science and Software Engineering, Université Laval, Québec, QC, CanadaLarge Scale Text Understanding System Lab, Natural Language Processing Group, Department of Information and Communication Technologies, Universitat Pompeu Fabra, Barcelona, SpainGroup for Research in Artificial Intelligence of Laval University, Department of Computer Science and Software Engineering, Université Laval, Québec, QC, CanadaIn the field of automatic text simplification, assessing whether or not the meaning of the original text has been preserved during simplification is of paramount importance. Metrics relying on n-gram overlap assessment may struggle to deal with simplifications which replace complex phrases with their simpler paraphrases. Current evaluation metrics for meaning preservation based on large language models (LLMs), such as BertScore in machine translation or QuestEval in summarization, have been proposed. However, none has a strong correlation with human judgment of meaning preservation. Moreover, such metrics have not been assessed in the context of text simplification research. In this study, we present a meta-evaluation of several metrics we apply to measure content similarity in text simplification. We also show that the metrics are unable to pass two trivial, inexpensive content preservation tests. Another contribution of this study is MeaningBERT (https://github.com/GRAAL-Research/MeaningBERT), a new trainable metric designed to assess meaning preservation between two sentences in text simplification, showing how it correlates with human judgment. To demonstrate its quality and versatility, we will also present a compilation of datasets used to assess meaning preservation and benchmark our study against a large selection of popular metrics.https://www.frontiersin.org/articles/10.3389/frai.2023.1223924/fullevaluation of text simplification systemsmeaning preservationautomatic text simplificationlexical simplificationsyntactic simplificationfew-shot evaluation of text simplification systems |
| spellingShingle | David Beauchemin Horacio Saggion Richard Khoury MeaningBERT: assessing meaning preservation between sentences Frontiers in Artificial Intelligence evaluation of text simplification systems meaning preservation automatic text simplification lexical simplification syntactic simplification few-shot evaluation of text simplification systems |
| title | MeaningBERT: assessing meaning preservation between sentences |
| title_full | MeaningBERT: assessing meaning preservation between sentences |
| title_fullStr | MeaningBERT: assessing meaning preservation between sentences |
| title_full_unstemmed | MeaningBERT: assessing meaning preservation between sentences |
| title_short | MeaningBERT: assessing meaning preservation between sentences |
| title_sort | meaningbert assessing meaning preservation between sentences |
| topic | evaluation of text simplification systems meaning preservation automatic text simplification lexical simplification syntactic simplification few-shot evaluation of text simplification systems |
| url | https://www.frontiersin.org/articles/10.3389/frai.2023.1223924/full |
| work_keys_str_mv | AT davidbeauchemin meaningbertassessingmeaningpreservationbetweensentences AT horaciosaggion meaningbertassessingmeaningpreservationbetweensentences AT richardkhoury meaningbertassessingmeaningpreservationbetweensentences |