Aggregating Reliable Submissions in Crowdsourcing Systems
Crowdsourcing is a cost-effective method that gathers crowd wisdom to solve machine-hard problems. In crowdsourcing systems, requesters post tasks for obtaining reliable solutions. Nevertheless, since workers have various expertise and knowledge background, they probably deliver low-quality and ambi...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9614193/ |
| _version_ | 1819127400908193792 |
|---|---|
| author | Ayswarya R. Kurup G. P. Sajeev J. Swaminathan |
| author_facet | Ayswarya R. Kurup G. P. Sajeev J. Swaminathan |
| author_sort | Ayswarya R. Kurup |
| collection | DOAJ |
| description | Crowdsourcing is a cost-effective method that gathers crowd wisdom to solve machine-hard problems. In crowdsourcing systems, requesters post tasks for obtaining reliable solutions. Nevertheless, since workers have various expertise and knowledge background, they probably deliver low-quality and ambiguous submissions. A task aggregation scheme is generally employed in crowdsourcing systems, to deal with this problem. Existing methods mainly focus on structured submissions and also do not consider the cost incurred for completing a task. We exploit features of submissions to improve the task aggregation for proposing a method which is applicable to both structured and unstructured tasks. Moreover, existing probabilistic methods for answer aggregation are sensitive to sparsity. Our approach uses a generative probabilistic model that incorporates similarity in answers along with worker and task features. Thereafter, we present a method for minimizing the cost of tasks, that eventually leverages the quality of answers. We conduct experiments on empirical data that demonstrates the effectiveness of our method compared to state-of-the-art approaches. |
| first_indexed | 2024-12-22T08:11:20Z |
| format | Article |
| id | doaj.art-51b6538ea0dd427c878fa4ba81d86588 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-22T08:11:20Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-51b6538ea0dd427c878fa4ba81d865882022-12-21T18:33:01ZengIEEEIEEE Access2169-35362021-01-01915305815307110.1109/ACCESS.2021.31279949614193Aggregating Reliable Submissions in Crowdsourcing SystemsAyswarya R. Kurup0https://orcid.org/0000-0003-3390-9360G. P. Sajeev1https://orcid.org/0000-0002-9422-6689J. Swaminathan2https://orcid.org/0000-0001-5646-3213Department of Computer Science and Engineering, Amrita School of Engineering, Amrita Vishwa Vidyapeetham, Amritapuri Campus, Kollam, IndiaDepartment of Computer Science and Engineering, Amrita School of Engineering, Amrita Vishwa Vidyapeetham, Amritapuri Campus, Kollam, IndiaDepartment of Computer Science and Engineering, Amrita School of Engineering, Amrita Vishwa Vidyapeetham, Amritapuri Campus, Kollam, IndiaCrowdsourcing is a cost-effective method that gathers crowd wisdom to solve machine-hard problems. In crowdsourcing systems, requesters post tasks for obtaining reliable solutions. Nevertheless, since workers have various expertise and knowledge background, they probably deliver low-quality and ambiguous submissions. A task aggregation scheme is generally employed in crowdsourcing systems, to deal with this problem. Existing methods mainly focus on structured submissions and also do not consider the cost incurred for completing a task. We exploit features of submissions to improve the task aggregation for proposing a method which is applicable to both structured and unstructured tasks. Moreover, existing probabilistic methods for answer aggregation are sensitive to sparsity. Our approach uses a generative probabilistic model that incorporates similarity in answers along with worker and task features. Thereafter, we present a method for minimizing the cost of tasks, that eventually leverages the quality of answers. We conduct experiments on empirical data that demonstrates the effectiveness of our method compared to state-of-the-art approaches.https://ieeexplore.ieee.org/document/9614193/Answer aggregationexpertness estimationprobabilistic modelquality controlcost minimization |
| spellingShingle | Ayswarya R. Kurup G. P. Sajeev J. Swaminathan Aggregating Reliable Submissions in Crowdsourcing Systems IEEE Access Answer aggregation expertness estimation probabilistic model quality control cost minimization |
| title | Aggregating Reliable Submissions in Crowdsourcing Systems |
| title_full | Aggregating Reliable Submissions in Crowdsourcing Systems |
| title_fullStr | Aggregating Reliable Submissions in Crowdsourcing Systems |
| title_full_unstemmed | Aggregating Reliable Submissions in Crowdsourcing Systems |
| title_short | Aggregating Reliable Submissions in Crowdsourcing Systems |
| title_sort | aggregating reliable submissions in crowdsourcing systems |
| topic | Answer aggregation expertness estimation probabilistic model quality control cost minimization |
| url | https://ieeexplore.ieee.org/document/9614193/ |
| work_keys_str_mv | AT ayswaryarkurup aggregatingreliablesubmissionsincrowdsourcingsystems AT gpsajeev aggregatingreliablesubmissionsincrowdsourcingsystems AT jswaminathan aggregatingreliablesubmissionsincrowdsourcingsystems |