SyntaxGym: An Online Platform for Targeted Evaluation of Language Models
Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models. However, this line of research requires an uncommon confluence of skills: both the theoretical knowledge needed to design controlled psycholinguistic experiments, and the technic...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Association for Computational Linguistics (ACL)
2021
|
| Online Access: | https://hdl.handle.net/1721.1/138281 |
| _version_ | 1811089384356708352 |
|---|---|
| author | Gauthier, Jon Hu, Jennifer Wilcox, Ethan Qian, Peng Levy, Roger |
| author_facet | Gauthier, Jon Hu, Jennifer Wilcox, Ethan Qian, Peng Levy, Roger |
| author_sort | Gauthier, Jon |
| collection | MIT |
| description | Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models. However, this
line of research requires an uncommon confluence of skills: both the theoretical knowledge needed to design controlled psycholinguistic experiments, and the technical proficiency needed to train and deploy large-scale
language models. We present SyntaxGym,
an online platform designed to make targeted
evaluations accessible to both experts in NLP
and linguistics, reproducible across computing environments, and standardized following the norms of psycholinguistic experimental design. This paper releases two tools of independent value for the computational linguistics community:
1. A website, syntaxgym.org, which
centralizes the process of targeted syntactic evaluation and provides easy tools for
analysis and visualization;
2. Two command-line tools, syntaxgym
and lm-zoo, which allow any user to
reproduce targeted syntactic evaluations
and general language model inference on
their own machine. |
| first_indexed | 2024-09-23T14:18:19Z |
| format | Article |
| id | mit-1721.1/138281 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T14:18:19Z |
| publishDate | 2021 |
| publisher | Association for Computational Linguistics (ACL) |
| record_format | dspace |
| spelling | mit-1721.1/1382812021-12-02T03:13:40Z SyntaxGym: An Online Platform for Targeted Evaluation of Language Models Gauthier, Jon Hu, Jennifer Wilcox, Ethan Qian, Peng Levy, Roger Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models. However, this line of research requires an uncommon confluence of skills: both the theoretical knowledge needed to design controlled psycholinguistic experiments, and the technical proficiency needed to train and deploy large-scale language models. We present SyntaxGym, an online platform designed to make targeted evaluations accessible to both experts in NLP and linguistics, reproducible across computing environments, and standardized following the norms of psycholinguistic experimental design. This paper releases two tools of independent value for the computational linguistics community: 1. A website, syntaxgym.org, which centralizes the process of targeted syntactic evaluation and provides easy tools for analysis and visualization; 2. Two command-line tools, syntaxgym and lm-zoo, which allow any user to reproduce targeted syntactic evaluations and general language model inference on their own machine. 2021-12-01T17:49:30Z 2021-12-01T17:49:30Z 2020 2021-12-01T17:47:34Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/138281 Gauthier, Jon, Hu, Jennifer, Wilcox, Ethan, Qian, Peng and Levy, Roger. 2020. "SyntaxGym: An Online Platform for Targeted Evaluation of Language Models." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. en 10.18653/V1/2020.ACL-DEMOS.10 Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations Creative Commons Attribution 4.0 International license https://creativecommons.org/licenses/by/4.0/ application/pdf Association for Computational Linguistics (ACL) Association for Computational Linguistics |
| spellingShingle | Gauthier, Jon Hu, Jennifer Wilcox, Ethan Qian, Peng Levy, Roger SyntaxGym: An Online Platform for Targeted Evaluation of Language Models |
| title | SyntaxGym: An Online Platform for Targeted Evaluation of Language Models |
| title_full | SyntaxGym: An Online Platform for Targeted Evaluation of Language Models |
| title_fullStr | SyntaxGym: An Online Platform for Targeted Evaluation of Language Models |
| title_full_unstemmed | SyntaxGym: An Online Platform for Targeted Evaluation of Language Models |
| title_short | SyntaxGym: An Online Platform for Targeted Evaluation of Language Models |
| title_sort | syntaxgym an online platform for targeted evaluation of language models |
| url | https://hdl.handle.net/1721.1/138281 |
| work_keys_str_mv | AT gauthierjon syntaxgymanonlineplatformfortargetedevaluationoflanguagemodels AT hujennifer syntaxgymanonlineplatformfortargetedevaluationoflanguagemodels AT wilcoxethan syntaxgymanonlineplatformfortargetedevaluationoflanguagemodels AT qianpeng syntaxgymanonlineplatformfortargetedevaluationoflanguagemodels AT levyroger syntaxgymanonlineplatformfortargetedevaluationoflanguagemodels |