Assessing the Efficacy of a Participant-Vetting Procedure to Improve Data-Quality on Amazon’s Mechanical Turk
In recent years, Amazon’s Mechanical Turk (MTurk) has become a pivotal source for participant recruitment in many social-science fields. In the last several years, however, concerns about data quality have arisen. In response, CloudResearch developed an intensive pre-screening procedure to vet the f...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
PsychOpen GOLD/ Leibniz Institute for Psychology
2022-06-01
|
| Series: | Methodology |
| Subjects: | |
| Online Access: | https://meth.psychopen.eu/index.php/meth/article/view/8331 |
| _version_ | 1797964709951963136 |
|---|---|
| author | Emilio D. Rivera Benjamin M. Wilkowski Aaron J. Moss Cheskie Rosenzweig Leib Litman |
| author_facet | Emilio D. Rivera Benjamin M. Wilkowski Aaron J. Moss Cheskie Rosenzweig Leib Litman |
| author_sort | Emilio D. Rivera |
| collection | DOAJ |
| description | In recent years, Amazon’s Mechanical Turk (MTurk) has become a pivotal source for participant recruitment in many social-science fields. In the last several years, however, concerns about data quality have arisen. In response, CloudResearch developed an intensive pre-screening procedure to vet the full participant pool available on MTurk and exclude those providing low-quality data. To assess its efficacy, we compared three MTurk samples that completed identical measures: Sample 1 was collected prior to the pre-screening’s implementation. Sample 2 was collected shortly following its implementation, and Sample 3 was collected nearly a full-year after its implementation. Results indicated that the reliability and validity of scales improved with the implementation of this prescreening procedure, and that this was especially apparent with more recent versions. Thus, this prescreening procedure appears to be a valuable tool to help ensure the collection of high-quality data on MTurk. |
| first_indexed | 2024-04-11T01:48:11Z |
| format | Article |
| id | doaj.art-e394e22692af47c9bc2bcec9a3b4eec8 |
| institution | Directory Open Access Journal |
| issn | 1614-2241 |
| language | English |
| last_indexed | 2024-04-11T01:48:11Z |
| publishDate | 2022-06-01 |
| publisher | PsychOpen GOLD/ Leibniz Institute for Psychology |
| record_format | Article |
| series | Methodology |
| spelling | doaj.art-e394e22692af47c9bc2bcec9a3b4eec82023-01-03T07:30:06ZengPsychOpen GOLD/ Leibniz Institute for PsychologyMethodology1614-22412022-06-0118212614310.5964/meth.8331meth.8331Assessing the Efficacy of a Participant-Vetting Procedure to Improve Data-Quality on Amazon’s Mechanical TurkEmilio D. Rivera0Benjamin M. Wilkowski1Aaron J. Moss2Cheskie Rosenzweig3Leib Litman4Department of Psychology, University of Wyoming, Laramie, WY, USADepartment of Psychology, University of Wyoming, Laramie, WY, USAPrime Research Solutions, Queens, NY, USAPrime Research Solutions, Queens, NY, USAPrime Research Solutions, Queens, NY, USAIn recent years, Amazon’s Mechanical Turk (MTurk) has become a pivotal source for participant recruitment in many social-science fields. In the last several years, however, concerns about data quality have arisen. In response, CloudResearch developed an intensive pre-screening procedure to vet the full participant pool available on MTurk and exclude those providing low-quality data. To assess its efficacy, we compared three MTurk samples that completed identical measures: Sample 1 was collected prior to the pre-screening’s implementation. Sample 2 was collected shortly following its implementation, and Sample 3 was collected nearly a full-year after its implementation. Results indicated that the reliability and validity of scales improved with the implementation of this prescreening procedure, and that this was especially apparent with more recent versions. Thus, this prescreening procedure appears to be a valuable tool to help ensure the collection of high-quality data on MTurk.https://meth.psychopen.eu/index.php/meth/article/view/8331online data collectionscreening solutionsvetting proceduresdata quality |
| spellingShingle | Emilio D. Rivera Benjamin M. Wilkowski Aaron J. Moss Cheskie Rosenzweig Leib Litman Assessing the Efficacy of a Participant-Vetting Procedure to Improve Data-Quality on Amazon’s Mechanical Turk Methodology online data collection screening solutions vetting procedures data quality |
| title | Assessing the Efficacy of a Participant-Vetting Procedure to Improve Data-Quality on Amazon’s Mechanical Turk |
| title_full | Assessing the Efficacy of a Participant-Vetting Procedure to Improve Data-Quality on Amazon’s Mechanical Turk |
| title_fullStr | Assessing the Efficacy of a Participant-Vetting Procedure to Improve Data-Quality on Amazon’s Mechanical Turk |
| title_full_unstemmed | Assessing the Efficacy of a Participant-Vetting Procedure to Improve Data-Quality on Amazon’s Mechanical Turk |
| title_short | Assessing the Efficacy of a Participant-Vetting Procedure to Improve Data-Quality on Amazon’s Mechanical Turk |
| title_sort | assessing the efficacy of a participant vetting procedure to improve data quality on amazon s mechanical turk |
| topic | online data collection screening solutions vetting procedures data quality |
| url | https://meth.psychopen.eu/index.php/meth/article/view/8331 |
| work_keys_str_mv | AT emiliodrivera assessingtheefficacyofaparticipantvettingproceduretoimprovedataqualityonamazonsmechanicalturk AT benjaminmwilkowski assessingtheefficacyofaparticipantvettingproceduretoimprovedataqualityonamazonsmechanicalturk AT aaronjmoss assessingtheefficacyofaparticipantvettingproceduretoimprovedataqualityonamazonsmechanicalturk AT cheskierosenzweig assessingtheefficacyofaparticipantvettingproceduretoimprovedataqualityonamazonsmechanicalturk AT leiblitman assessingtheefficacyofaparticipantvettingproceduretoimprovedataqualityonamazonsmechanicalturk |