Iterative Learning for K-Approval Votes in Crowdsourcing Systems
Crowdsourcing systems have emerged as cornerstones to collect large amounts of qualified data in various human-powered problems with a relatively low budget. In eliciting the wisdom of crowds, many web-based crowdsourcing platforms have encouraged workers to select top-<i>K</i> alternati...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-01-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/11/2/630 |
| _version_ | 1827602399996411904 |
|---|---|
| author | Joonyoung Kim Donghyeon Lee Kyomin Jung |
| author_facet | Joonyoung Kim Donghyeon Lee Kyomin Jung |
| author_sort | Joonyoung Kim |
| collection | DOAJ |
| description | Crowdsourcing systems have emerged as cornerstones to collect large amounts of qualified data in various human-powered problems with a relatively low budget. In eliciting the wisdom of crowds, many web-based crowdsourcing platforms have encouraged workers to select top-<i>K</i> alternatives rather than just one choice, which is called “<i>K</i>-approval voting”. This kind of setting has the advantage of inducing workers to make fewer mistakes when they respond to target tasks. However, there is not much work on inferring the correct answer from crowd-sourced data via a <i>K</i>-approval voting. In this paper, we propose a novel and efficient iterative algorithm to infer correct answers for a <i>K</i>-approval voting, which can be directly applied to real-world crowdsourcing systems. We analyze the average performance of our algorithm, and prove the theoretical error bound that decays exponentially in terms of the quality of workers and the number of queries. Through extensive experiments including the mixed case with various types of tasks, we show that our algorithm outperforms Expectation and Maximization (EM) and existing baseline algorithms. |
| first_indexed | 2024-03-09T05:17:50Z |
| format | Article |
| id | doaj.art-21888cacd0ad465ebd97fe16454165af |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-09T05:17:50Z |
| publishDate | 2021-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-21888cacd0ad465ebd97fe16454165af2023-12-03T12:43:45ZengMDPI AGApplied Sciences2076-34172021-01-0111263010.3390/app11020630Iterative Learning for K-Approval Votes in Crowdsourcing SystemsJoonyoung Kim0Donghyeon Lee1Kyomin Jung2Department of Electrical and Computer Engineering, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul 08826, KoreaDepartment of Electrical and Computer Engineering, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul 08826, KoreaDepartment of Electrical and Computer Engineering, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul 08826, KoreaCrowdsourcing systems have emerged as cornerstones to collect large amounts of qualified data in various human-powered problems with a relatively low budget. In eliciting the wisdom of crowds, many web-based crowdsourcing platforms have encouraged workers to select top-<i>K</i> alternatives rather than just one choice, which is called “<i>K</i>-approval voting”. This kind of setting has the advantage of inducing workers to make fewer mistakes when they respond to target tasks. However, there is not much work on inferring the correct answer from crowd-sourced data via a <i>K</i>-approval voting. In this paper, we propose a novel and efficient iterative algorithm to infer correct answers for a <i>K</i>-approval voting, which can be directly applied to real-world crowdsourcing systems. We analyze the average performance of our algorithm, and prove the theoretical error bound that decays exponentially in terms of the quality of workers and the number of queries. Through extensive experiments including the mixed case with various types of tasks, we show that our algorithm outperforms Expectation and Maximization (EM) and existing baseline algorithms.https://www.mdpi.com/2076-3417/11/2/630crowdsourcinginference algorithmsstatistical learning |
| spellingShingle | Joonyoung Kim Donghyeon Lee Kyomin Jung Iterative Learning for K-Approval Votes in Crowdsourcing Systems Applied Sciences crowdsourcing inference algorithms statistical learning |
| title | Iterative Learning for K-Approval Votes in Crowdsourcing Systems |
| title_full | Iterative Learning for K-Approval Votes in Crowdsourcing Systems |
| title_fullStr | Iterative Learning for K-Approval Votes in Crowdsourcing Systems |
| title_full_unstemmed | Iterative Learning for K-Approval Votes in Crowdsourcing Systems |
| title_short | Iterative Learning for K-Approval Votes in Crowdsourcing Systems |
| title_sort | iterative learning for k approval votes in crowdsourcing systems |
| topic | crowdsourcing inference algorithms statistical learning |
| url | https://www.mdpi.com/2076-3417/11/2/630 |
| work_keys_str_mv | AT joonyoungkim iterativelearningforkapprovalvotesincrowdsourcingsystems AT donghyeonlee iterativelearningforkapprovalvotesincrowdsourcingsystems AT kyominjung iterativelearningforkapprovalvotesincrowdsourcingsystems |