What Quality Control Mechanisms do We Need for High-Quality Crowd Work?
Crowd sourcing and human computation has slowly become a mainstay for many application areas that seek to leverage the crowd in the development of high quality datasets, annotations, and problem solving beyond the reach of current AI solutions. One of the major challenges to the domain is ensuring h...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9893786/ |
| _version_ | 1811206259723993088 |
|---|---|
| author | Margeret Hall Mohammad Farhad Afzali Markus Krause Simon Caton |
| author_facet | Margeret Hall Mohammad Farhad Afzali Markus Krause Simon Caton |
| author_sort | Margeret Hall |
| collection | DOAJ |
| description | Crowd sourcing and human computation has slowly become a mainstay for many application areas that seek to leverage the crowd in the development of high quality datasets, annotations, and problem solving beyond the reach of current AI solutions. One of the major challenges to the domain is ensuring high-quality and diligent work. In response, the literature has seen a large number of quality control mechanisms each voicing (sometimes domain-specific) benefits and advantages when deployed in largescale human computation projects. This creates a complex design space for practitioners: it is not always clear which mechanism(s) to use for maximal quality control. In this article, we argue that this decision is perhaps overinflated and that provided there is “some kind” of quality control that this obviously known to crowd workers this is sufficient for “high-quality” solutions. To evidence this, and provide a basis for discussion, we undertake two experiments where we explore the relationship between task design, task complexity, quality control and solution quality. We do this with tasks from natural language processing, and image recognition of varying complexity. We illustrate that minimal quality control is enough to repel constantly underperforming contributors and that this is constant across tasks of varying complexity and formats. Our key takeaway: quality control is necessary, but seemingly not how it is implemented. |
| first_indexed | 2024-04-12T03:44:43Z |
| format | Article |
| id | doaj.art-c888ebfe45ae45cd8fee2db92bd19ed7 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-12T03:44:43Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-c888ebfe45ae45cd8fee2db92bd19ed72022-12-22T03:49:11ZengIEEEIEEE Access2169-35362022-01-0110997099972310.1109/ACCESS.2022.32072929893786What Quality Control Mechanisms do We Need for High-Quality Crowd Work?Margeret Hall0https://orcid.org/0000-0003-1049-3040Mohammad Farhad Afzali1https://orcid.org/0000-0001-7457-9148Markus Krause2Simon Caton3https://orcid.org/0000-0001-9379-3879Department of Strategy and Innovation, Wirtschaftsuniversität Wien, Vienna, AustriaCollege of Information Science and Technology, University of Nebraska Omaha, Omaha, NE, USABrainworks.ai, Berkeley, CA, USASchool of Computer Science, University College Dublin, Dublin, IrelandCrowd sourcing and human computation has slowly become a mainstay for many application areas that seek to leverage the crowd in the development of high quality datasets, annotations, and problem solving beyond the reach of current AI solutions. One of the major challenges to the domain is ensuring high-quality and diligent work. In response, the literature has seen a large number of quality control mechanisms each voicing (sometimes domain-specific) benefits and advantages when deployed in largescale human computation projects. This creates a complex design space for practitioners: it is not always clear which mechanism(s) to use for maximal quality control. In this article, we argue that this decision is perhaps overinflated and that provided there is “some kind” of quality control that this obviously known to crowd workers this is sufficient for “high-quality” solutions. To evidence this, and provide a basis for discussion, we undertake two experiments where we explore the relationship between task design, task complexity, quality control and solution quality. We do this with tasks from natural language processing, and image recognition of varying complexity. We illustrate that minimal quality control is enough to repel constantly underperforming contributors and that this is constant across tasks of varying complexity and formats. Our key takeaway: quality control is necessary, but seemingly not how it is implemented.https://ieeexplore.ieee.org/document/9893786/Quality controlhuman computationnatural language processingimage recognitioncrowd work |
| spellingShingle | Margeret Hall Mohammad Farhad Afzali Markus Krause Simon Caton What Quality Control Mechanisms do We Need for High-Quality Crowd Work? IEEE Access Quality control human computation natural language processing image recognition crowd work |
| title | What Quality Control Mechanisms do We Need for High-Quality Crowd Work? |
| title_full | What Quality Control Mechanisms do We Need for High-Quality Crowd Work? |
| title_fullStr | What Quality Control Mechanisms do We Need for High-Quality Crowd Work? |
| title_full_unstemmed | What Quality Control Mechanisms do We Need for High-Quality Crowd Work? |
| title_short | What Quality Control Mechanisms do We Need for High-Quality Crowd Work? |
| title_sort | what quality control mechanisms do we need for high quality crowd work |
| topic | Quality control human computation natural language processing image recognition crowd work |
| url | https://ieeexplore.ieee.org/document/9893786/ |
| work_keys_str_mv | AT margerethall whatqualitycontrolmechanismsdoweneedforhighqualitycrowdwork AT mohammadfarhadafzali whatqualitycontrolmechanismsdoweneedforhighqualitycrowdwork AT markuskrause whatqualitycontrolmechanismsdoweneedforhighqualitycrowdwork AT simoncaton whatqualitycontrolmechanismsdoweneedforhighqualitycrowdwork |