Interactions Between Syntax and Semantics in Language Models
How do syntax and semantics interact in neural language models? Is the intuitive dichotomy between semantics and syntax useful as a mental model of their behavior? In this thesis, I systematically investigate how models handle the interactions between sentence corruptions of each kind. I develop a s...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2022
|
| Online Access: | https://hdl.handle.net/1721.1/144680 |
| _version_ | 1826200814039859200 |
|---|---|
| author | Bau, Anthony |
| author2 | Andreas, Jacob |
| author_facet | Andreas, Jacob Bau, Anthony |
| author_sort | Bau, Anthony |
| collection | MIT |
| description | How do syntax and semantics interact in neural language models? Is the intuitive dichotomy between semantics and syntax useful as a mental model of their behavior? In this thesis, I systematically investigate how models handle the interactions between sentence corruptions of each kind. I develop a setup for generating “syntactic” and “semantic” corruptions and combining them. Using this setup, I develop a general framework for understanding how classes of corruptions interact with each other, arguing that this can be understood in terms of a single “entanglement” parameter. This allows me to quantify entanglement between syntax and semantics in a variety of situations. The result is that semantics is more entangled with syntax than it is with itself, and that syntax is highly self-entangled. |
| first_indexed | 2024-09-23T11:41:54Z |
| format | Thesis |
| id | mit-1721.1/144680 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T11:41:54Z |
| publishDate | 2022 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1446802022-08-30T03:39:29Z Interactions Between Syntax and Semantics in Language Models Bau, Anthony Andreas, Jacob Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science How do syntax and semantics interact in neural language models? Is the intuitive dichotomy between semantics and syntax useful as a mental model of their behavior? In this thesis, I systematically investigate how models handle the interactions between sentence corruptions of each kind. I develop a setup for generating “syntactic” and “semantic” corruptions and combining them. Using this setup, I develop a general framework for understanding how classes of corruptions interact with each other, arguing that this can be understood in terms of a single “entanglement” parameter. This allows me to quantify entanglement between syntax and semantics in a variety of situations. The result is that semantics is more entangled with syntax than it is with itself, and that syntax is highly self-entangled. M.Eng. 2022-08-29T16:04:17Z 2022-08-29T16:04:17Z 2022-05 2022-06-02T20:24:16.955Z Thesis https://hdl.handle.net/1721.1/144680 In Copyright - Educational Use Permitted Copyright MIT http://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Bau, Anthony Interactions Between Syntax and Semantics in Language Models |
| title | Interactions Between Syntax and Semantics in Language Models |
| title_full | Interactions Between Syntax and Semantics in Language Models |
| title_fullStr | Interactions Between Syntax and Semantics in Language Models |
| title_full_unstemmed | Interactions Between Syntax and Semantics in Language Models |
| title_short | Interactions Between Syntax and Semantics in Language Models |
| title_sort | interactions between syntax and semantics in language models |
| url | https://hdl.handle.net/1721.1/144680 |
| work_keys_str_mv | AT bauanthony interactionsbetweensyntaxandsemanticsinlanguagemodels |