A Smaller and Better Word Embedding for Neural Machine Translation
Word embeddings play an important role in Neural Machine Translation (NMT). However, it still has a series of problems such as ignoring the prior knowledge of the association between words, relying on specific task constraints passively in parameter training, and isolating individual embedding&#...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10107984/ |
| _version_ | 1797835801169494016 |
|---|---|
| author | Qi Chen |
| author_facet | Qi Chen |
| author_sort | Qi Chen |
| collection | DOAJ |
| description | Word embeddings play an important role in Neural Machine Translation (NMT). However, it still has a series of problems such as ignoring the prior knowledge of the association between words, relying on specific task constraints passively in parameter training, and isolating individual embedding’s learning process from one another. In this paper, we propose a new word embedding method to add the prior knowledge of the association between words to the training process, and at the same time to share the iterative training results among all word embeddings. This method is applicable to all mainstream NMT systems. In our experiments, it achieves an improvement of +0.9 BLEU points on the WMT’14 English<inline-formula> <tex-math notation="LaTeX">$\rightarrow $ </tex-math></inline-formula>German task. On the Global Voices v2018q4 Spanish<inline-formula> <tex-math notation="LaTeX">$\rightarrow $ </tex-math></inline-formula>Czech low-resource translation tasks, it leads to a more prominent performance improvement over the strong baselines (a +2.6 BLEU improvement on average). As another “bonus”, the new word embedding has far fewer parameters than the traditional word embedding, even as low as 15% of the parameters of the baselines. |
| first_indexed | 2024-04-09T14:58:16Z |
| format | Article |
| id | doaj.art-2146be6e26e541dd9123040f9ed58a3d |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-09T14:58:16Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-2146be6e26e541dd9123040f9ed58a3d2023-05-01T23:01:02ZengIEEEIEEE Access2169-35362023-01-0111407704077810.1109/ACCESS.2023.327017110107984A Smaller and Better Word Embedding for Neural Machine TranslationQi Chen0https://orcid.org/0000-0003-0796-8033School of Computer Science and Technology, Northeastern University, Shenyang, Liaoning, ChinaWord embeddings play an important role in Neural Machine Translation (NMT). However, it still has a series of problems such as ignoring the prior knowledge of the association between words, relying on specific task constraints passively in parameter training, and isolating individual embedding’s learning process from one another. In this paper, we propose a new word embedding method to add the prior knowledge of the association between words to the training process, and at the same time to share the iterative training results among all word embeddings. This method is applicable to all mainstream NMT systems. In our experiments, it achieves an improvement of +0.9 BLEU points on the WMT’14 English<inline-formula> <tex-math notation="LaTeX">$\rightarrow $ </tex-math></inline-formula>German task. On the Global Voices v2018q4 Spanish<inline-formula> <tex-math notation="LaTeX">$\rightarrow $ </tex-math></inline-formula>Czech low-resource translation tasks, it leads to a more prominent performance improvement over the strong baselines (a +2.6 BLEU improvement on average). As another “bonus”, the new word embedding has far fewer parameters than the traditional word embedding, even as low as 15% of the parameters of the baselines.https://ieeexplore.ieee.org/document/10107984/Neural networkneural machine translationword embedding |
| spellingShingle | Qi Chen A Smaller and Better Word Embedding for Neural Machine Translation IEEE Access Neural network neural machine translation word embedding |
| title | A Smaller and Better Word Embedding for Neural Machine Translation |
| title_full | A Smaller and Better Word Embedding for Neural Machine Translation |
| title_fullStr | A Smaller and Better Word Embedding for Neural Machine Translation |
| title_full_unstemmed | A Smaller and Better Word Embedding for Neural Machine Translation |
| title_short | A Smaller and Better Word Embedding for Neural Machine Translation |
| title_sort | smaller and better word embedding for neural machine translation |
| topic | Neural network neural machine translation word embedding |
| url | https://ieeexplore.ieee.org/document/10107984/ |
| work_keys_str_mv | AT qichen asmallerandbetterwordembeddingforneuralmachinetranslation AT qichen smallerandbetterwordembeddingforneuralmachinetranslation |