Bayesian Nonparametric Crowdsourcing
Crowdsourcing has been proven to be an effective and efficient tool to annotate large data-sets. User annotations are often noisy, so methods to combine the annotations to produce reliable estimates of the ground truth are necessary. We claim that considering the existence of clusters of users in th...
| Main Authors: | , , , |
|---|---|
| Format: | Journal article |
| Published: |
Journal of Machine Learning Research
2015
|
| _version_ | 1797087679234441216 |
|---|---|
| author | Moreno, P Artes-Rodriguez, A Teh, Y Perez-Cruz, F |
| author_facet | Moreno, P Artes-Rodriguez, A Teh, Y Perez-Cruz, F |
| author_sort | Moreno, P |
| collection | OXFORD |
| description | Crowdsourcing has been proven to be an effective and efficient tool to annotate large data-sets. User annotations are often noisy, so methods to combine the annotations to produce reliable estimates of the ground truth are necessary. We claim that considering the existence of clusters of users in this combination step can improve the performance. This is especially important in early stages of crowdsourcing implementations, where the number of annotations is low. At this stage there is not enough information to accurately estimate the bias introduced by each annotator separately, so we have to resort to models that consider the statistical links among them. In addition, finding these clusters is interesting in itself as knowing the behavior of the pool of annotators allows implementing efficient active learning strategies. Based on this, we propose in this paper two new fully unsupervised models based on a Chinese restaurant process (CRP) prior and a hierarchical structure that allows inferring these groups jointly with the ground truth and the properties of the users. Efficient inference algorithms based on Gibbs sampling with auxiliary variables are proposed. Finally, we perform experiments, both on synthetic and real databases, to show the advantages of our models over state-of-the-art algorithms. |
| first_indexed | 2024-03-07T02:39:07Z |
| format | Journal article |
| id | oxford-uuid:a9d4ac8b-4cbf-4a6b-ad70-2c2a66ffd1d4 |
| institution | University of Oxford |
| last_indexed | 2024-03-07T02:39:07Z |
| publishDate | 2015 |
| publisher | Journal of Machine Learning Research |
| record_format | dspace |
| spelling | oxford-uuid:a9d4ac8b-4cbf-4a6b-ad70-2c2a66ffd1d42022-03-27T03:10:58ZBayesian Nonparametric CrowdsourcingJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:a9d4ac8b-4cbf-4a6b-ad70-2c2a66ffd1d4Symplectic Elements at OxfordJournal of Machine Learning Research2015Moreno, PArtes-Rodriguez, ATeh, YPerez-Cruz, FCrowdsourcing has been proven to be an effective and efficient tool to annotate large data-sets. User annotations are often noisy, so methods to combine the annotations to produce reliable estimates of the ground truth are necessary. We claim that considering the existence of clusters of users in this combination step can improve the performance. This is especially important in early stages of crowdsourcing implementations, where the number of annotations is low. At this stage there is not enough information to accurately estimate the bias introduced by each annotator separately, so we have to resort to models that consider the statistical links among them. In addition, finding these clusters is interesting in itself as knowing the behavior of the pool of annotators allows implementing efficient active learning strategies. Based on this, we propose in this paper two new fully unsupervised models based on a Chinese restaurant process (CRP) prior and a hierarchical structure that allows inferring these groups jointly with the ground truth and the properties of the users. Efficient inference algorithms based on Gibbs sampling with auxiliary variables are proposed. Finally, we perform experiments, both on synthetic and real databases, to show the advantages of our models over state-of-the-art algorithms. |
| spellingShingle | Moreno, P Artes-Rodriguez, A Teh, Y Perez-Cruz, F Bayesian Nonparametric Crowdsourcing |
| title | Bayesian Nonparametric Crowdsourcing |
| title_full | Bayesian Nonparametric Crowdsourcing |
| title_fullStr | Bayesian Nonparametric Crowdsourcing |
| title_full_unstemmed | Bayesian Nonparametric Crowdsourcing |
| title_short | Bayesian Nonparametric Crowdsourcing |
| title_sort | bayesian nonparametric crowdsourcing |
| work_keys_str_mv | AT morenop bayesiannonparametriccrowdsourcing AT artesrodrigueza bayesiannonparametriccrowdsourcing AT tehy bayesiannonparametriccrowdsourcing AT perezcruzf bayesiannonparametriccrowdsourcing |