Validating Bayesian truth serum in large-scale online human experiments
This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Bayesian truth serum (BTS) is an exciting new method for improvi...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Published: |
Public Library of Science
2018
|
| Online Access: | http://hdl.handle.net/1721.1/113265 https://orcid.org/0000-0001-9487-9359 |
| _version_ | 1826204274815664128 |
|---|---|
| author | Pickard, Galen Frank, Morgan Ryan Cebrian, Manuel Rahwan, Iyad |
| author2 | Massachusetts Institute of Technology. Media Laboratory |
| author_facet | Massachusetts Institute of Technology. Media Laboratory Pickard, Galen Frank, Morgan Ryan Cebrian, Manuel Rahwan, Iyad |
| author_sort | Pickard, Galen |
| collection | MIT |
| description | This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Bayesian truth serum (BTS) is an exciting new method for improving honesty and information quality in multiple-choice survey, but, despite the method's mathematical reliance on large sample sizes, existing literature about BTS only focuses on small experiments. Combined with the prevalence of online survey platforms, such as Amazon's Mechanical Turk, which facilitate surveys with hundreds or thousands of participants, BTS must be effective in large-scale experiments for BTS to become a readily accepted tool in real-world applications. We demonstrate that BTS quantifiably improves honesty in large-scale online surveys where the "honest" distribution of answers is known in expectation on aggregate. Furthermore, we explore a marketing application where "honest" answers cannot be known, but find that BTS treatment impacts the resulting distributions of answers. |
| first_indexed | 2024-09-23T12:51:42Z |
| format | Article |
| id | mit-1721.1/113265 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T12:51:42Z |
| publishDate | 2018 |
| publisher | Public Library of Science |
| record_format | dspace |
| spelling | mit-1721.1/1132652022-10-01T11:36:15Z Validating Bayesian truth serum in large-scale online human experiments Pickard, Galen Frank, Morgan Ryan Cebrian, Manuel Rahwan, Iyad Massachusetts Institute of Technology. Media Laboratory Frank, Morgan Ryan Cebrian, Manuel Rahwan, Iyad This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Bayesian truth serum (BTS) is an exciting new method for improving honesty and information quality in multiple-choice survey, but, despite the method's mathematical reliance on large sample sizes, existing literature about BTS only focuses on small experiments. Combined with the prevalence of online survey platforms, such as Amazon's Mechanical Turk, which facilitate surveys with hundreds or thousands of participants, BTS must be effective in large-scale experiments for BTS to become a readily accepted tool in real-world applications. We demonstrate that BTS quantifiably improves honesty in large-scale online surveys where the "honest" distribution of answers is known in expectation on aggregate. Furthermore, we explore a marketing application where "honest" answers cannot be known, but find that BTS treatment impacts the resulting distributions of answers. 2018-01-22T20:31:10Z 2018-01-22T20:31:10Z 2017-05 2016-11 2018-01-19T18:52:11Z Article http://purl.org/eprint/type/JournalArticle 1932-6203 http://hdl.handle.net/1721.1/113265 Frank, Morgan R. et al. “Validating Bayesian Truth Serum in Large-Scale Online Human Experiments.” Edited by Chuhsing Kate Hsiao. PLOS ONE 12, 5 (May 2017): e0177385 © 2017 Frank et al https://orcid.org/0000-0001-9487-9359 http://dx.doi.org/10.1371/JOURNAL.PONE.0177385 PLOS ONE Creative Commons Attribution 4.0 International License http://creativecommons.org/licenses/by/4.0 application/pdf Public Library of Science PLoS |
| spellingShingle | Pickard, Galen Frank, Morgan Ryan Cebrian, Manuel Rahwan, Iyad Validating Bayesian truth serum in large-scale online human experiments |
| title | Validating Bayesian truth serum in large-scale online human experiments |
| title_full | Validating Bayesian truth serum in large-scale online human experiments |
| title_fullStr | Validating Bayesian truth serum in large-scale online human experiments |
| title_full_unstemmed | Validating Bayesian truth serum in large-scale online human experiments |
| title_short | Validating Bayesian truth serum in large-scale online human experiments |
| title_sort | validating bayesian truth serum in large scale online human experiments |
| url | http://hdl.handle.net/1721.1/113265 https://orcid.org/0000-0001-9487-9359 |
| work_keys_str_mv | AT pickardgalen validatingbayesiantruthseruminlargescaleonlinehumanexperiments AT frankmorganryan validatingbayesiantruthseruminlargescaleonlinehumanexperiments AT cebrianmanuel validatingbayesiantruthseruminlargescaleonlinehumanexperiments AT rahwaniyad validatingbayesiantruthseruminlargescaleonlinehumanexperiments |