Information Theory–based Compositional Distributional Semantics
In the context of text representation, Compositional Distributional Semantics models aim to fuse the Distributional Hypothesis and the Principle of Compositionality. Text embedding is based on co-ocurrence distributions and the representations are in turn combined by compositional functions taking i...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The MIT Press
2022-08-01
|
| Series: | Computational Linguistics |
| Online Access: | http://dx.doi.org/10.1162/coli_a_00454 |
| _version_ | 1797795426686992384 |
|---|---|
| author | Enrique Amigó Alejandro Ariza-Casabona Victor Fresno M. Antònia Martí |
| author_facet | Enrique Amigó Alejandro Ariza-Casabona Victor Fresno M. Antònia Martí |
| author_sort | Enrique Amigó |
| collection | DOAJ |
| description | In the context of text representation, Compositional Distributional Semantics models aim to fuse the Distributional Hypothesis and the Principle of Compositionality. Text embedding is based on co-ocurrence distributions and the representations are in turn combined by compositional functions taking into account the text structure. However, the theoretical basis of compositional functions is still an open issue. In this article we define and study the notion of Information Theory–based Compositional Distributional Semantics (ICDS): (i) We first establish formal properties for embedding, composition, and similarity functions based on Shannon’s Information Theory; (ii) we analyze the existing approaches under this prism, checking whether or not they comply with the established desirable properties; (iii) we propose two parameterizable composition and similarity functions that generalize traditional
approaches while fulfilling the formal properties; and finally (iv) we perform an empirical study on several textual similarity datasets that include sentences with a high and low lexical overlap, and on the similarity between words and their description. Our theoretical analysis and empirical results show that fulfilling formal properties affects positively the accuracy of text representation models in terms of correspondence (isometry) between the embedding and meaning spaces. |
| first_indexed | 2024-03-13T03:17:46Z |
| format | Article |
| id | doaj.art-0e807c0c39b847ac927c0de549aea901 |
| institution | Directory Open Access Journal |
| issn | 1530-9312 |
| language | English |
| last_indexed | 2024-03-13T03:17:46Z |
| publishDate | 2022-08-01 |
| publisher | The MIT Press |
| record_format | Article |
| series | Computational Linguistics |
| spelling | doaj.art-0e807c0c39b847ac927c0de549aea9012023-06-25T14:50:05ZengThe MIT PressComputational Linguistics1530-93122022-08-0148410.1162/coli_a_00454Information Theory–based Compositional Distributional SemanticsEnrique AmigóAlejandro Ariza-CasabonaVictor FresnoM. Antònia MartíIn the context of text representation, Compositional Distributional Semantics models aim to fuse the Distributional Hypothesis and the Principle of Compositionality. Text embedding is based on co-ocurrence distributions and the representations are in turn combined by compositional functions taking into account the text structure. However, the theoretical basis of compositional functions is still an open issue. In this article we define and study the notion of Information Theory–based Compositional Distributional Semantics (ICDS): (i) We first establish formal properties for embedding, composition, and similarity functions based on Shannon’s Information Theory; (ii) we analyze the existing approaches under this prism, checking whether or not they comply with the established desirable properties; (iii) we propose two parameterizable composition and similarity functions that generalize traditional approaches while fulfilling the formal properties; and finally (iv) we perform an empirical study on several textual similarity datasets that include sentences with a high and low lexical overlap, and on the similarity between words and their description. Our theoretical analysis and empirical results show that fulfilling formal properties affects positively the accuracy of text representation models in terms of correspondence (isometry) between the embedding and meaning spaces.http://dx.doi.org/10.1162/coli_a_00454 |
| spellingShingle | Enrique Amigó Alejandro Ariza-Casabona Victor Fresno M. Antònia Martí Information Theory–based Compositional Distributional Semantics Computational Linguistics |
| title | Information Theory–based Compositional Distributional Semantics |
| title_full | Information Theory–based Compositional Distributional Semantics |
| title_fullStr | Information Theory–based Compositional Distributional Semantics |
| title_full_unstemmed | Information Theory–based Compositional Distributional Semantics |
| title_short | Information Theory–based Compositional Distributional Semantics |
| title_sort | information theory based compositional distributional semantics |
| url | http://dx.doi.org/10.1162/coli_a_00454 |
| work_keys_str_mv | AT enriqueamigo informationtheorybasedcompositionaldistributionalsemantics AT alejandroarizacasabona informationtheorybasedcompositionaldistributionalsemantics AT victorfresno informationtheorybasedcompositionaldistributionalsemantics AT mantoniamarti informationtheorybasedcompositionaldistributionalsemantics |