Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing
We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. While contextual embeddings have been shown to yield richer representations of meaning compared to their static counterparts, aligning them poses a challenge due to...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Association for Computational Linguistics
2020
|
| Online Access: | https://hdl.handle.net/1721.1/128715 |
| _version_ | 1811069512258158592 |
|---|---|
| author | Schuster, Tal Ram, Ori Barzilay, Regina Globerson, Amir |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Schuster, Tal Ram, Ori Barzilay, Regina Globerson, Amir |
| author_sort | Schuster, Tal |
| collection | MIT |
| description | We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. While contextual embeddings have been shown to yield richer representations of meaning compared to their static counterparts, aligning them poses a challenge due to their dynamic nature. To this end, we construct context-independent variants of the original monolingual spaces and utilize their mapping to derive an alignment for the context-dependent spaces. This mapping readily supports processing of a target language, improving transfer by context-aware embeddings. Our experimental results demonstrate the effectiveness of this approach for zero-shot and few-shot learning of dependency parsing. Specifically, our method consistently outperforms the previous state-of-the-art on 6 tested languages, yielding an improvement of 6.8 LAS points on average. |
| first_indexed | 2024-09-23T08:11:37Z |
| format | Article |
| id | mit-1721.1/128715 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T08:11:37Z |
| publishDate | 2020 |
| publisher | Association for Computational Linguistics |
| record_format | dspace |
| spelling | mit-1721.1/1287152022-09-23T11:31:40Z Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing Schuster, Tal Ram, Ori Barzilay, Regina Globerson, Amir Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. While contextual embeddings have been shown to yield richer representations of meaning compared to their static counterparts, aligning them poses a challenge due to their dynamic nature. To this end, we construct context-independent variants of the original monolingual spaces and utilize their mapping to derive an alignment for the context-dependent spaces. This mapping readily supports processing of a target language, improving transfer by context-aware embeddings. Our experimental results demonstrate the effectiveness of this approach for zero-shot and few-shot learning of dependency parsing. Specifically, our method consistently outperforms the previous state-of-the-art on 6 tested languages, yielding an improvement of 6.8 LAS points on average. Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA) (Contract FA8650-17-C- 9116) 2020-12-02T20:59:53Z 2020-12-02T20:59:53Z 2019 2019 2020-12-01T13:18:41Z Article http://purl.org/eprint/type/ConferencePaper 978-1-950737-13-0 https://hdl.handle.net/1721.1/128715 Schuster, Tal et al. "Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing." 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2019, Minneapolis, Minnesota, Association for Computational Linguistics, 2019. © 2019 Association for Computational Linguistics en http://dx.doi.org/10.18653/V1/N19-1162 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Creative Commons Attribution 4.0 International license https://creativecommons.org/licenses/by/4.0/ application/pdf Association for Computational Linguistics Association for Computational Linguistics |
| spellingShingle | Schuster, Tal Ram, Ori Barzilay, Regina Globerson, Amir Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing |
| title | Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing |
| title_full | Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing |
| title_fullStr | Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing |
| title_full_unstemmed | Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing |
| title_short | Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing |
| title_sort | cross lingual alignment of contextual word embeddings with applications to zero shot dependency parsing |
| url | https://hdl.handle.net/1721.1/128715 |
| work_keys_str_mv | AT schustertal crosslingualalignmentofcontextualwordembeddingswithapplicationstozeroshotdependencyparsing AT ramori crosslingualalignmentofcontextualwordembeddingswithapplicationstozeroshotdependencyparsing AT barzilayregina crosslingualalignmentofcontextualwordembeddingswithapplicationstozeroshotdependencyparsing AT globersonamir crosslingualalignmentofcontextualwordembeddingswithapplicationstozeroshotdependencyparsing |