Revisiting Negation in Neural Machine Translation
In this paper, we evaluate the translation of negation both automatically and manually, in English–German (EN–DE) and English– Chinese (EN–ZH). We show that the ability of neural machine translation (NMT) models to translate negation has improved with deeper and more advanced network...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The MIT Press
2021-01-01
|
| Series: | Transactions of the Association for Computational Linguistics |
| Online Access: | https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00395/106793/Revisiting-Negation-in-Neural-Machine-Translation |
| _version_ | 1818474966379659264 |
|---|---|
| author | Gongbo Tang Philipp Rönchen Rico Sennrich Joakim Nivre |
| author_facet | Gongbo Tang Philipp Rönchen Rico Sennrich Joakim Nivre |
| author_sort | Gongbo Tang |
| collection | DOAJ |
| description |
In this paper, we evaluate the translation of negation both automatically and manually, in English–German (EN–DE) and English– Chinese (EN–ZH). We show that the ability of neural machine translation (NMT) models to translate negation has improved with deeper and more advanced networks, although the performance varies between language pairs and translation directions. The accuracy of manual evaluation in EN→DE, DE→EN, EN→ZH, and ZH→EN is 95.7%, 94.8%, 93.4%, and 91.7%, respectively. In addition, we show that under-translation is the most significant error type in NMT, which contrasts with the more diverse error profile previously observed for statistical machine translation. To better understand the root of the under-translation of negation, we study the model’s information flow and training data. While our information flow analysis does not reveal any deficiencies that could be used to detect or fix the under-translation of negation, we find that negation is often rephrased during training, which could make it more difficult for the model to learn a reliable link between source and target negation. We finally conduct intrinsic analysis and extrinsic probing tasks on negation, showing that NMT models can distinguish negation and non-negation tokens very well and encode a lot of information about negation in hidden states but nevertheless leave room for improvement. |
| first_indexed | 2024-04-14T04:44:04Z |
| format | Article |
| id | doaj.art-e59dc6942e564cabb1e6de3489c18e5c |
| institution | Directory Open Access Journal |
| issn | 2307-387X |
| language | English |
| last_indexed | 2024-04-14T04:44:04Z |
| publishDate | 2021-01-01 |
| publisher | The MIT Press |
| record_format | Article |
| series | Transactions of the Association for Computational Linguistics |
| spelling | doaj.art-e59dc6942e564cabb1e6de3489c18e5c2022-12-22T02:11:32ZengThe MIT PressTransactions of the Association for Computational Linguistics2307-387X2021-01-01974075510.1162/tacl_a_00395Revisiting Negation in Neural Machine TranslationGongbo Tang0Philipp Rönchen1Rico Sennrich2Joakim Nivre3Department of Linguistics and Philology, Uppsala University, Sweden. gongbo.tang@lingfil.uu.seDepartment of Linguistics and Philology, Uppsala University, Sweden. philipp.rönchen@lingfil.uu.seDepartment of Computational Linguistics, University of Zurich, SwitzerlandDepartment of Linguistics and Philology, Uppsala University, Sweden. joakim.nivre@lingfil.uu.se In this paper, we evaluate the translation of negation both automatically and manually, in English–German (EN–DE) and English– Chinese (EN–ZH). We show that the ability of neural machine translation (NMT) models to translate negation has improved with deeper and more advanced networks, although the performance varies between language pairs and translation directions. The accuracy of manual evaluation in EN→DE, DE→EN, EN→ZH, and ZH→EN is 95.7%, 94.8%, 93.4%, and 91.7%, respectively. In addition, we show that under-translation is the most significant error type in NMT, which contrasts with the more diverse error profile previously observed for statistical machine translation. To better understand the root of the under-translation of negation, we study the model’s information flow and training data. While our information flow analysis does not reveal any deficiencies that could be used to detect or fix the under-translation of negation, we find that negation is often rephrased during training, which could make it more difficult for the model to learn a reliable link between source and target negation. We finally conduct intrinsic analysis and extrinsic probing tasks on negation, showing that NMT models can distinguish negation and non-negation tokens very well and encode a lot of information about negation in hidden states but nevertheless leave room for improvement.https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00395/106793/Revisiting-Negation-in-Neural-Machine-Translation |
| spellingShingle | Gongbo Tang Philipp Rönchen Rico Sennrich Joakim Nivre Revisiting Negation in Neural Machine Translation Transactions of the Association for Computational Linguistics |
| title | Revisiting Negation in Neural Machine Translation |
| title_full | Revisiting Negation in Neural Machine Translation |
| title_fullStr | Revisiting Negation in Neural Machine Translation |
| title_full_unstemmed | Revisiting Negation in Neural Machine Translation |
| title_short | Revisiting Negation in Neural Machine Translation |
| title_sort | revisiting negation in neural machine translation |
| url | https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00395/106793/Revisiting-Negation-in-Neural-Machine-Translation |
| work_keys_str_mv | AT gongbotang revisitingnegationinneuralmachinetranslation AT philippronchen revisitingnegationinneuralmachinetranslation AT ricosennrich revisitingnegationinneuralmachinetranslation AT joakimnivre revisitingnegationinneuralmachinetranslation |