Let’s Play <tt>Mono</tt>-<tt>Poly</tt>: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses
Pre-trained language models (LMs) encode rich information about linguistic structure but their knowledge about lexical polysemy remains unclear. We propose a novel experimental setup for analyzing this knowledge in LMs specifically trained for different languages (English, French, Sp...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The MIT Press
2021-01-01
|
| Series: | Transactions of the Association for Computational Linguistics |
| Online Access: | https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00400/106797/Let-s-Play-Mono-Poly-BERT-Can-Reveal-Words |
| _version_ | 1811299978703798272 |
|---|---|
| author | Aina Garí Soler Marianna Apidianaki |
| author_facet | Aina Garí Soler Marianna Apidianaki |
| author_sort | Aina Garí Soler |
| collection | DOAJ |
| description |
Pre-trained language models (LMs) encode rich information about linguistic structure but their knowledge about lexical polysemy remains unclear. We propose a novel experimental setup for analyzing this knowledge in LMs specifically trained for different languages (English, French, Spanish, and Greek) and in multilingual BERT. We perform our analysis on datasets carefully designed to reflect different sense distributions, and control for parameters that are highly correlated with polysemy such as frequency and grammatical category. We demonstrate that BERT-derived representations reflect words’ polysemy level and their partitionability into senses. Polysemy-related information is more clearly present in English BERT embeddings, but models in other languages also manage to establish relevant distinctions between words at different polysemy levels. Our results contribute to a better understanding of the knowledge encoded in contextualized representations and open up new avenues for multilingual lexical semantics research. |
| first_indexed | 2024-04-13T06:43:35Z |
| format | Article |
| id | doaj.art-2343eb75c94b4cf48fbd81a58394650e |
| institution | Directory Open Access Journal |
| issn | 2307-387X |
| language | English |
| last_indexed | 2024-04-13T06:43:35Z |
| publishDate | 2021-01-01 |
| publisher | The MIT Press |
| record_format | Article |
| series | Transactions of the Association for Computational Linguistics |
| spelling | doaj.art-2343eb75c94b4cf48fbd81a58394650e2022-12-22T02:57:39ZengThe MIT PressTransactions of the Association for Computational Linguistics2307-387X2021-01-01982584410.1162/tacl_a_00400Let’s Play <tt>Mono</tt>-<tt>Poly</tt>: BERT Can Reveal Words’ Polysemy Level and Partitionability into SensesAina Garí Soler0Marianna Apidianaki1Université Paris-Saclay CNRS, LISN 91400, Orsay, France. aina.gari@limsi.frDepartment of Digital Humanities University of Helsinki Helsinki, Finland. marianna.apidianaki@helsinki.fi Pre-trained language models (LMs) encode rich information about linguistic structure but their knowledge about lexical polysemy remains unclear. We propose a novel experimental setup for analyzing this knowledge in LMs specifically trained for different languages (English, French, Spanish, and Greek) and in multilingual BERT. We perform our analysis on datasets carefully designed to reflect different sense distributions, and control for parameters that are highly correlated with polysemy such as frequency and grammatical category. We demonstrate that BERT-derived representations reflect words’ polysemy level and their partitionability into senses. Polysemy-related information is more clearly present in English BERT embeddings, but models in other languages also manage to establish relevant distinctions between words at different polysemy levels. Our results contribute to a better understanding of the knowledge encoded in contextualized representations and open up new avenues for multilingual lexical semantics research.https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00400/106797/Let-s-Play-Mono-Poly-BERT-Can-Reveal-Words |
| spellingShingle | Aina Garí Soler Marianna Apidianaki Let’s Play <tt>Mono</tt>-<tt>Poly</tt>: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses Transactions of the Association for Computational Linguistics |
| title | Let’s Play <tt>Mono</tt>-<tt>Poly</tt>: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses |
| title_full | Let’s Play <tt>Mono</tt>-<tt>Poly</tt>: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses |
| title_fullStr | Let’s Play <tt>Mono</tt>-<tt>Poly</tt>: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses |
| title_full_unstemmed | Let’s Play <tt>Mono</tt>-<tt>Poly</tt>: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses |
| title_short | Let’s Play <tt>Mono</tt>-<tt>Poly</tt>: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses |
| title_sort | let s play tt mono tt tt poly tt bert can reveal words polysemy level and partitionability into senses |
| url | https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00400/106797/Let-s-Play-Mono-Poly-BERT-Can-Reveal-Words |
| work_keys_str_mv | AT ainagarisoler letsplayttmonottttpolyttbertcanrevealwordspolysemylevelandpartitionabilityintosenses AT mariannaapidianaki letsplayttmonottttpolyttbertcanrevealwordspolysemylevelandpartitionabilityintosenses |