Analytic Methods for Optimizing Realtime Crowdsourcing
Realtime crowdsourcing research has demonstrated that it is possible to recruit paid crowds within seconds by managing a small, fast-reacting worker pool. Realtime crowds enable crowd-powered systems that respond at interactive speeds: for example, cameras, robots and instant opinion polls. So far,...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
2017
|
| Online Access: | http://hdl.handle.net/1721.1/108085 https://orcid.org/0000-0002-0024-5847 https://orcid.org/0000-0002-0442-691X |
| _version_ | 1826207232597950464 |
|---|---|
| author | Brandt, Joel Karger, David R Bernstein, Michael Scott Miller, Robert C |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Brandt, Joel Karger, David R Bernstein, Michael Scott Miller, Robert C |
| author_sort | Brandt, Joel |
| collection | MIT |
| description | Realtime crowdsourcing research has demonstrated that it is possible to recruit paid crowds within seconds by managing a small, fast-reacting worker pool. Realtime crowds enable crowd-powered systems that respond at interactive speeds: for example, cameras, robots and instant opinion polls. So far, these techniques have mainly been proof-of-concept prototypes: research has not yet attempted to understand how they might work at large scale or optimize their cost/performance trade-offs. In this paper, we use queueing theory to analyze the retainer model for realtime crowdsourcing, in particular its expected wait time and cost to requesters. We provide an algorithm that allows requesters to minimize their cost subject to performance requirements. We then propose and analyze three techniques to improve performance: push notifications, shared retainer pools, and precruitment, which involves recalling retainer workers before a task actually arrives. An experimental validation finds that precruited workers begin a task 500 milliseconds after it is posted, delivering results below the one-second cognitive threshold for an end-user to stay in flow. |
| first_indexed | 2024-09-23T13:46:07Z |
| format | Article |
| id | mit-1721.1/108085 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T13:46:07Z |
| publishDate | 2017 |
| record_format | dspace |
| spelling | mit-1721.1/1080852022-10-01T17:02:14Z Analytic Methods for Optimizing Realtime Crowdsourcing Brandt, Joel Karger, David R Bernstein, Michael Scott Miller, Robert C Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Karger, David R. Karger, David R Bernstein, Michael Scott Miller, Robert C Realtime crowdsourcing research has demonstrated that it is possible to recruit paid crowds within seconds by managing a small, fast-reacting worker pool. Realtime crowds enable crowd-powered systems that respond at interactive speeds: for example, cameras, robots and instant opinion polls. So far, these techniques have mainly been proof-of-concept prototypes: research has not yet attempted to understand how they might work at large scale or optimize their cost/performance trade-offs. In this paper, we use queueing theory to analyze the retainer model for realtime crowdsourcing, in particular its expected wait time and cost to requesters. We provide an algorithm that allows requesters to minimize their cost subject to performance requirements. We then propose and analyze three techniques to improve performance: push notifications, shared retainer pools, and precruitment, which involves recalling retainer workers before a task actually arrives. An experimental validation finds that precruited workers begin a task 500 milliseconds after it is posted, delivering results below the one-second cognitive threshold for an end-user to stay in flow. 2017-04-12T20:16:57Z 2017-04-12T20:16:57Z 2012-04 Article http://purl.org/eprint/type/ConferencePaper http://hdl.handle.net/1721.1/108085 Bernstein, Michael S. et al. "Analytic Methods for Optimizing Realtime Crowdsourcing." Collective Intelligence Conference 2012, Cambridge, MA, USA, 18-20 April, 2012. https://orcid.org/0000-0002-0024-5847 https://orcid.org/0000-0002-0442-691X en_US https://arxiv.org/abs/1204.2995 Proceedings of the Collective Intelligence Conference, 2012 Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf Prof. Karger via Phoebe Ayres |
| spellingShingle | Brandt, Joel Karger, David R Bernstein, Michael Scott Miller, Robert C Analytic Methods for Optimizing Realtime Crowdsourcing |
| title | Analytic Methods for Optimizing Realtime Crowdsourcing |
| title_full | Analytic Methods for Optimizing Realtime Crowdsourcing |
| title_fullStr | Analytic Methods for Optimizing Realtime Crowdsourcing |
| title_full_unstemmed | Analytic Methods for Optimizing Realtime Crowdsourcing |
| title_short | Analytic Methods for Optimizing Realtime Crowdsourcing |
| title_sort | analytic methods for optimizing realtime crowdsourcing |
| url | http://hdl.handle.net/1721.1/108085 https://orcid.org/0000-0002-0024-5847 https://orcid.org/0000-0002-0442-691X |
| work_keys_str_mv | AT brandtjoel analyticmethodsforoptimizingrealtimecrowdsourcing AT kargerdavidr analyticmethodsforoptimizingrealtimecrowdsourcing AT bernsteinmichaelscott analyticmethodsforoptimizingrealtimecrowdsourcing AT millerrobertc analyticmethodsforoptimizingrealtimecrowdsourcing |