Using mechanical turk to obtain and analyze English acceptability judgments
The prevalent method in theoretical syntax and semantics research involves obtaining a judgment of the acceptability of a sentence/meaning pair, typically by just the author of the paper, sometimes with feedback from colleagues. The weakness of the traditional non-quantitative single-sentence/single...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
John Wiley & Sons, Inc.
2012
|
| Online Access: | http://hdl.handle.net/1721.1/73969 https://orcid.org/0000-0002-5912-883X |
| _version_ | 1811089412916772864 |
|---|---|
| author | Gibson, Edward A. Piantadosi, Steven Thomas Fedorenko, Kristina |
| author2 | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences |
| author_facet | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences Gibson, Edward A. Piantadosi, Steven Thomas Fedorenko, Kristina |
| author_sort | Gibson, Edward A. |
| collection | MIT |
| description | The prevalent method in theoretical syntax and semantics research involves obtaining a judgment of the acceptability of a sentence/meaning pair, typically by just the author of the paper, sometimes with feedback from colleagues. The weakness of the traditional non-quantitative single-sentence/single-participant methodology, along with the existence of cognitive and social biases, has the unwanted effect that claims in the syntax and semantics literature cannot be trusted. Even if most of the judgments in an arbitrary syntax/semantics paper can be substantiated with rigorous quantitative experiments, the existence of a small set of judgments that do not conform to the authors’ intuitions can have a large effect on the potential theories. Whereas it is clearly desirable to quantitatively evaluate all syntactic and semantic hypotheses, it has been time-consuming in the past to find a large pool of naïve experimental participants for behavioral experiments. The advent of Amazon.com’s Mechanical Turk now makes this process very simple. Mechanical Turk is a marketplace interface that can be used for collecting behavioral data over the internet quickly and inexpensively. The cost of using an interface like Mechanical Turk is minimal, and the time that it takes for the results to be returned is very short. Many linguistic surveys can be completed within a day, at a cost of less than $50. In this paper, we provide detailed instructions for how to use our freely available software in order to (a) post-linguistic acceptability surveys to Mechanical Turk; and (b) extract and analyze the resulting data. |
| first_indexed | 2024-09-23T14:18:46Z |
| format | Article |
| id | mit-1721.1/73969 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T14:18:46Z |
| publishDate | 2012 |
| publisher | John Wiley & Sons, Inc. |
| record_format | dspace |
| spelling | mit-1721.1/739692022-09-29T08:37:07Z Using mechanical turk to obtain and analyze English acceptability judgments Gibson, Edward A. Piantadosi, Steven Thomas Fedorenko, Kristina Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences Gibson, Edward A. Gibson, Edward A. Piantadosi, Steven Thomas Fedorenko, Kristina The prevalent method in theoretical syntax and semantics research involves obtaining a judgment of the acceptability of a sentence/meaning pair, typically by just the author of the paper, sometimes with feedback from colleagues. The weakness of the traditional non-quantitative single-sentence/single-participant methodology, along with the existence of cognitive and social biases, has the unwanted effect that claims in the syntax and semantics literature cannot be trusted. Even if most of the judgments in an arbitrary syntax/semantics paper can be substantiated with rigorous quantitative experiments, the existence of a small set of judgments that do not conform to the authors’ intuitions can have a large effect on the potential theories. Whereas it is clearly desirable to quantitatively evaluate all syntactic and semantic hypotheses, it has been time-consuming in the past to find a large pool of naïve experimental participants for behavioral experiments. The advent of Amazon.com’s Mechanical Turk now makes this process very simple. Mechanical Turk is a marketplace interface that can be used for collecting behavioral data over the internet quickly and inexpensively. The cost of using an interface like Mechanical Turk is minimal, and the time that it takes for the results to be returned is very short. Many linguistic surveys can be completed within a day, at a cost of less than $50. In this paper, we provide detailed instructions for how to use our freely available software in order to (a) post-linguistic acceptability surveys to Mechanical Turk; and (b) extract and analyze the resulting data. 2012-10-15T15:59:01Z 2012-10-15T15:59:01Z 2011-08 2011-04 Article http://purl.org/eprint/type/JournalArticle 1749-818X http://hdl.handle.net/1721.1/73969 Gibson, Edward, Steve Piantadosi, and Kristina Fedorenko. “Using Mechanical Turk to Obtain and Analyze English Acceptability Judgments.” Language and Linguistics Compass 5.8 (2011): 509–524. Web. https://orcid.org/0000-0002-5912-883X en_US http://dx.doi.org/10.1111/j.1749-818X.2011.00295.x Language and Linguistics Compass Creative Commons Attribution-Noncommercial-Share Alike 3.0 http://creativecommons.org/licenses/by-nc-sa/3.0/ application/pdf John Wiley & Sons, Inc. Gibson via Courtney Crummett |
| spellingShingle | Gibson, Edward A. Piantadosi, Steven Thomas Fedorenko, Kristina Using mechanical turk to obtain and analyze English acceptability judgments |
| title | Using mechanical turk to obtain and analyze English acceptability judgments |
| title_full | Using mechanical turk to obtain and analyze English acceptability judgments |
| title_fullStr | Using mechanical turk to obtain and analyze English acceptability judgments |
| title_full_unstemmed | Using mechanical turk to obtain and analyze English acceptability judgments |
| title_short | Using mechanical turk to obtain and analyze English acceptability judgments |
| title_sort | using mechanical turk to obtain and analyze english acceptability judgments |
| url | http://hdl.handle.net/1721.1/73969 https://orcid.org/0000-0002-5912-883X |
| work_keys_str_mv | AT gibsonedwarda usingmechanicalturktoobtainandanalyzeenglishacceptabilityjudgments AT piantadosisteventhomas usingmechanicalturktoobtainandanalyzeenglishacceptabilityjudgments AT fedorenkokristina usingmechanicalturktoobtainandanalyzeenglishacceptabilityjudgments |