Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study
© 2019 Association for Computational Linguistics Neural language models have achieved state-of-the-art performances on many NLP tasks, and recently have been shown to learn a number of hierarchically-sensitive syntactic dependencies between individual words. However, equally important for language p...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Association for Computational Linguistics
2021
|
| Online Access: | https://hdl.handle.net/1721.1/137251 |
| _version_ | 1826206165452718080 |
|---|---|
| author | An, Aixiu Qian, Peng Wilcox, Ethan Levy, Roger |
| author2 | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences |
| author_facet | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences An, Aixiu Qian, Peng Wilcox, Ethan Levy, Roger |
| author_sort | An, Aixiu |
| collection | MIT |
| description | © 2019 Association for Computational Linguistics Neural language models have achieved state-of-the-art performances on many NLP tasks, and recently have been shown to learn a number of hierarchically-sensitive syntactic dependencies between individual words. However, equally important for language processing is the ability to combine words into phrasal constituents, and use constituent-level features to drive downstream expectations. Here we investigate neural models' ability to represent constituent-level features, using coordinated noun phrases as a case study. We assess whether different neural language models trained on English and French represent phrase-level number and gender features, and use those features to drive downstream expectations. Our results suggest that models use a linear combination of NP constituent number to drive CoordNP/verb number agreement. This behavior is highly regular and even sensitive to local syntactic context, however it differs crucially from observed human behavior. Models have less success with gender agreement. Models trained on large corpora perform best, and there is no obvious advantage for models trained using explicit syntactic supervision. |
| first_indexed | 2024-09-23T13:24:48Z |
| format | Article |
| id | mit-1721.1/137251 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T13:24:48Z |
| publishDate | 2021 |
| publisher | Association for Computational Linguistics |
| record_format | dspace |
| spelling | mit-1721.1/1372512023-02-03T21:45:03Z Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study An, Aixiu Qian, Peng Wilcox, Ethan Levy, Roger Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences © 2019 Association for Computational Linguistics Neural language models have achieved state-of-the-art performances on many NLP tasks, and recently have been shown to learn a number of hierarchically-sensitive syntactic dependencies between individual words. However, equally important for language processing is the ability to combine words into phrasal constituents, and use constituent-level features to drive downstream expectations. Here we investigate neural models' ability to represent constituent-level features, using coordinated noun phrases as a case study. We assess whether different neural language models trained on English and French represent phrase-level number and gender features, and use those features to drive downstream expectations. Our results suggest that models use a linear combination of NP constituent number to drive CoordNP/verb number agreement. This behavior is highly regular and even sensitive to local syntactic context, however it differs crucially from observed human behavior. Models have less success with gender agreement. Models trained on large corpora perform best, and there is no obvious advantage for models trained using explicit syntactic supervision. 2021-11-03T17:21:57Z 2021-11-03T17:21:57Z 2019 2021-04-12T18:52:39Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/137251 An, Aixiu, Qian, Peng, Wilcox, Ethan and Levy, Roger. 2019. "Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study." EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference. en 10.18653/v1/d19-1287 EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf Association for Computational Linguistics Association for Computational Linguistics |
| spellingShingle | An, Aixiu Qian, Peng Wilcox, Ethan Levy, Roger Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study |
| title | Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study |
| title_full | Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study |
| title_fullStr | Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study |
| title_full_unstemmed | Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study |
| title_short | Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study |
| title_sort | representation of constituents in neural language models coordination phrase as a case study |
| url | https://hdl.handle.net/1721.1/137251 |
| work_keys_str_mv | AT anaixiu representationofconstituentsinneurallanguagemodelscoordinationphraseasacasestudy AT qianpeng representationofconstituentsinneurallanguagemodelscoordinationphraseasacasestudy AT wilcoxethan representationofconstituentsinneurallanguagemodelscoordinationphraseasacasestudy AT levyroger representationofconstituentsinneurallanguagemodelscoordinationphraseasacasestudy |