Bayesian aggregation of categorical distributions with applications in crowdsourcing
A key problem in crowdsourcing is the aggregation of judgments of proportions. For example, workers might be presented with a news article or an image, and be asked to identify the proportion of each topic, sentiment, object, or colour present in it. These varying judgments then need to be aggregate...
| Main Authors: | , , , , |
|---|---|
| Format: | Conference item |
| Published: |
AAAI Press / International Joint Conferences on Artificial Intelligence
2017
|
| _version_ | 1797051202976874496 |
|---|---|
| author | Augustin, A Venanzi, M Hare, J Rogers, A Jennings, N |
| author_facet | Augustin, A Venanzi, M Hare, J Rogers, A Jennings, N |
| author_sort | Augustin, A |
| collection | OXFORD |
| description | A key problem in crowdsourcing is the aggregation of judgments of proportions. For example, workers might be presented with a news article or an image, and be asked to identify the proportion of each topic, sentiment, object, or colour present in it. These varying judgments then need to be aggregated to form a consensus view of the document’s contents. Often, however, these judgments can be skewed by workers who provide judgments randomly (i.e. they are spammers). Spammers make the cost of acquiring judgments more expensive and degrade the accuracy of the aggregation. For such cases, we provide a new Bayesian framework for aggregating these responses (expressed in the form of categorical distributions) that for the first time accounts for spammers. We elicit 796 judgments about proportions of objects and colours in images. Experimental results on three real-world datasets show comparable aggregation accuracy when 60% of the workers are spammers, as other state of the art approaches do when there are no spammers. |
| first_indexed | 2024-03-06T18:16:35Z |
| format | Conference item |
| id | oxford-uuid:04d242be-6364-47bc-bb57-41f62a448c61 |
| institution | University of Oxford |
| last_indexed | 2024-03-06T18:16:35Z |
| publishDate | 2017 |
| publisher | AAAI Press / International Joint Conferences on Artificial Intelligence |
| record_format | dspace |
| spelling | oxford-uuid:04d242be-6364-47bc-bb57-41f62a448c612022-03-26T08:53:51ZBayesian aggregation of categorical distributions with applications in crowdsourcingConference itemhttp://purl.org/coar/resource_type/c_5794uuid:04d242be-6364-47bc-bb57-41f62a448c61Symplectic Elements at OxfordAAAI Press / International Joint Conferences on Artificial Intelligence2017Augustin, AVenanzi, MHare, JRogers, AJennings, NA key problem in crowdsourcing is the aggregation of judgments of proportions. For example, workers might be presented with a news article or an image, and be asked to identify the proportion of each topic, sentiment, object, or colour present in it. These varying judgments then need to be aggregated to form a consensus view of the document’s contents. Often, however, these judgments can be skewed by workers who provide judgments randomly (i.e. they are spammers). Spammers make the cost of acquiring judgments more expensive and degrade the accuracy of the aggregation. For such cases, we provide a new Bayesian framework for aggregating these responses (expressed in the form of categorical distributions) that for the first time accounts for spammers. We elicit 796 judgments about proportions of objects and colours in images. Experimental results on three real-world datasets show comparable aggregation accuracy when 60% of the workers are spammers, as other state of the art approaches do when there are no spammers. |
| spellingShingle | Augustin, A Venanzi, M Hare, J Rogers, A Jennings, N Bayesian aggregation of categorical distributions with applications in crowdsourcing |
| title | Bayesian aggregation of categorical distributions with applications in crowdsourcing |
| title_full | Bayesian aggregation of categorical distributions with applications in crowdsourcing |
| title_fullStr | Bayesian aggregation of categorical distributions with applications in crowdsourcing |
| title_full_unstemmed | Bayesian aggregation of categorical distributions with applications in crowdsourcing |
| title_short | Bayesian aggregation of categorical distributions with applications in crowdsourcing |
| title_sort | bayesian aggregation of categorical distributions with applications in crowdsourcing |
| work_keys_str_mv | AT augustina bayesianaggregationofcategoricaldistributionswithapplicationsincrowdsourcing AT venanzim bayesianaggregationofcategoricaldistributionswithapplicationsincrowdsourcing AT harej bayesianaggregationofcategoricaldistributionswithapplicationsincrowdsourcing AT rogersa bayesianaggregationofcategoricaldistributionswithapplicationsincrowdsourcing AT jenningsn bayesianaggregationofcategoricaldistributionswithapplicationsincrowdsourcing |