Discourse Structure in Machine Translation Evaluation
In this article, we explore the potential of using sentence-level discourse structure for machine translation evaluation. We first design discourse-aware similarity measures, which use all-subtree kernels to compare discourse parse trees in accordance with the Rhetorical Structure Theory (RST). Then...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The MIT Press
2017-09-01
|
| Series: | Computational Linguistics |
| Online Access: | http://dx.doi.org/10.1162/coli_a_00298 |
| _version_ | 1797795377494097920 |
|---|---|
| author | Shafiq Joty Francisco Guzmán Lluís Màrquez Preslav Nakov |
| author_facet | Shafiq Joty Francisco Guzmán Lluís Màrquez Preslav Nakov |
| author_sort | Shafiq Joty |
| collection | DOAJ |
| description | In this article, we explore the potential of using sentence-level discourse structure for machine translation evaluation. We first design discourse-aware similarity measures, which use all-subtree kernels to compare discourse parse trees in accordance with the Rhetorical Structure Theory (RST). Then, we show that a simple linear combination with these measures can help improve various existing machine translation evaluation metrics regarding correlation with human judgments both at the segment level and at the system level. This suggests that discourse information is complementary to the information used by many of the existing evaluation metrics, and thus it could be taken into account when developing richer evaluation metrics, such as the WMT-14 winning combined metric DiscoTK<jats:sub>party</jats:sub>. We also provide a detailed analysis of the relevance of various discourse elements and relations from the RST parse trees for machine
translation evaluation. In particular, we show that (i) all aspects of the RST tree are relevant, (ii) nuclearity is more useful than relation type, and (iii) the similarity of the translation RST tree to the reference RST tree is positively correlated with translation quality. |
| first_indexed | 2024-03-13T03:18:10Z |
| format | Article |
| id | doaj.art-6a2f0f06540341fc95a08ea72c3d2142 |
| institution | Directory Open Access Journal |
| issn | 1530-9312 |
| language | English |
| last_indexed | 2024-03-13T03:18:10Z |
| publishDate | 2017-09-01 |
| publisher | The MIT Press |
| record_format | Article |
| series | Computational Linguistics |
| spelling | doaj.art-6a2f0f06540341fc95a08ea72c3d21422023-06-25T14:50:05ZengThe MIT PressComputational Linguistics1530-93122017-09-0143410.1162/coli_a_00298Discourse Structure in Machine Translation EvaluationShafiq JotyFrancisco GuzmánLluís MàrquezPreslav NakovIn this article, we explore the potential of using sentence-level discourse structure for machine translation evaluation. We first design discourse-aware similarity measures, which use all-subtree kernels to compare discourse parse trees in accordance with the Rhetorical Structure Theory (RST). Then, we show that a simple linear combination with these measures can help improve various existing machine translation evaluation metrics regarding correlation with human judgments both at the segment level and at the system level. This suggests that discourse information is complementary to the information used by many of the existing evaluation metrics, and thus it could be taken into account when developing richer evaluation metrics, such as the WMT-14 winning combined metric DiscoTK<jats:sub>party</jats:sub>. We also provide a detailed analysis of the relevance of various discourse elements and relations from the RST parse trees for machine translation evaluation. In particular, we show that (i) all aspects of the RST tree are relevant, (ii) nuclearity is more useful than relation type, and (iii) the similarity of the translation RST tree to the reference RST tree is positively correlated with translation quality.http://dx.doi.org/10.1162/coli_a_00298 |
| spellingShingle | Shafiq Joty Francisco Guzmán Lluís Màrquez Preslav Nakov Discourse Structure in Machine Translation Evaluation Computational Linguistics |
| title | Discourse Structure in Machine Translation Evaluation |
| title_full | Discourse Structure in Machine Translation Evaluation |
| title_fullStr | Discourse Structure in Machine Translation Evaluation |
| title_full_unstemmed | Discourse Structure in Machine Translation Evaluation |
| title_short | Discourse Structure in Machine Translation Evaluation |
| title_sort | discourse structure in machine translation evaluation |
| url | http://dx.doi.org/10.1162/coli_a_00298 |
| work_keys_str_mv | AT shafiqjoty discoursestructureinmachinetranslationevaluation AT franciscoguzman discoursestructureinmachinetranslationevaluation AT lluismarquez discoursestructureinmachinetranslationevaluation AT preslavnakov discoursestructureinmachinetranslationevaluation |