| Summary: | Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models. However, this
line of research requires an uncommon confluence of skills: both the theoretical knowledge needed to design controlled psycholinguistic experiments, and the technical proficiency needed to train and deploy large-scale
language models. We present SyntaxGym,
an online platform designed to make targeted
evaluations accessible to both experts in NLP
and linguistics, reproducible across computing environments, and standardized following the norms of psycholinguistic experimental design. This paper releases two tools of independent value for the computational linguistics community:
1. A website, syntaxgym.org, which
centralizes the process of targeted syntactic evaluation and provides easy tools for
analysis and visualization;
2. Two command-line tools, syntaxgym
and lm-zoo, which allow any user to
reproduce targeted syntactic evaluations
and general language model inference on
their own machine.
|
|---|