Comparing input interfaces to elicit belief distributions
This paper introduces a new software interface to elicit belief distributions of any shape: Click-and-Drag. The interface was tested against the state of the art in the experimental literature—a text-based interface and multiple sliders—and in the online forecasting industry—a distribution-manipulat...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Cambridge University Press
2023-01-01
|
| Series: | Judgment and Decision Making |
| Subjects: | |
| Online Access: | https://www.cambridge.org/core/product/identifier/S1930297523000219/type/journal_article |
| _version_ | 1797741355645009920 |
|---|---|
| author | Paolo Crosetto Thomas de Haan |
| author_facet | Paolo Crosetto Thomas de Haan |
| author_sort | Paolo Crosetto |
| collection | DOAJ |
| description | This paper introduces a new software interface to elicit belief distributions of any shape: Click-and-Drag. The interface was tested against the state of the art in the experimental literature—a text-based interface and multiple sliders—and in the online forecasting industry—a distribution-manipulation interface similar to the one used by the most popular crowd-forecasting website. By means of a pre-registered experiment on Amazon Mechanical Turk, quantitative data on the accuracy of reported beliefs in a series of induced-value scenarios varying by granularity, shape, and time constraints, as well as subjective data on user experience were collected. Click-and-Drag outperformed all other interfaces by accuracy and speed, and was self-reported as being more intuitive and less frustrating, confirming the pre-registered hypothesis. Aside of the pre-registered results, Click-and-Drag generated the least drop-out rate from the task, and scored best in a sentiment analysis of an open-ended general question. Further, the interface was used to collect homegrown predictions on temperature in New York City in 2022 and 2042. Click-and-Drag elicited distributions were smoother with less idiosyncratic spikes. Free and open source, ready to use oTree, Qualtrics and Limesurvey plugins for Click-and-Drag, and all other tested interfaces are available at https://beliefelicitation.github.io/. |
| first_indexed | 2024-03-12T14:25:32Z |
| format | Article |
| id | doaj.art-b00ab2b6da004945a78b327c4d44592f |
| institution | Directory Open Access Journal |
| issn | 1930-2975 |
| language | English |
| last_indexed | 2024-03-12T14:25:32Z |
| publishDate | 2023-01-01 |
| publisher | Cambridge University Press |
| record_format | Article |
| series | Judgment and Decision Making |
| spelling | doaj.art-b00ab2b6da004945a78b327c4d44592f2023-08-18T07:41:32ZengCambridge University PressJudgment and Decision Making1930-29752023-01-011810.1017/jdm.2023.21Comparing input interfaces to elicit belief distributionsPaolo Crosetto0https://orcid.org/0000-0002-9153-0159Thomas de Haan1https://orcid.org/0000-0001-8658-3232GAEL, Grenoble INP, CNRS, INRAE, Université Grenoble Alpes, Grenoble, FranceDepartment of Economics, University of Bergen, Bergen, NorwayThis paper introduces a new software interface to elicit belief distributions of any shape: Click-and-Drag. The interface was tested against the state of the art in the experimental literature—a text-based interface and multiple sliders—and in the online forecasting industry—a distribution-manipulation interface similar to the one used by the most popular crowd-forecasting website. By means of a pre-registered experiment on Amazon Mechanical Turk, quantitative data on the accuracy of reported beliefs in a series of induced-value scenarios varying by granularity, shape, and time constraints, as well as subjective data on user experience were collected. Click-and-Drag outperformed all other interfaces by accuracy and speed, and was self-reported as being more intuitive and less frustrating, confirming the pre-registered hypothesis. Aside of the pre-registered results, Click-and-Drag generated the least drop-out rate from the task, and scored best in a sentiment analysis of an open-ended general question. Further, the interface was used to collect homegrown predictions on temperature in New York City in 2022 and 2042. Click-and-Drag elicited distributions were smoother with less idiosyncratic spikes. Free and open source, ready to use oTree, Qualtrics and Limesurvey plugins for Click-and-Drag, and all other tested interfaces are available at https://beliefelicitation.github.io/.https://www.cambridge.org/core/product/identifier/S1930297523000219/type/journal_articlebelief elicitationforecastingscoring rulesinterfaces |
| spellingShingle | Paolo Crosetto Thomas de Haan Comparing input interfaces to elicit belief distributions Judgment and Decision Making belief elicitation forecasting scoring rules interfaces |
| title | Comparing input interfaces to elicit belief distributions |
| title_full | Comparing input interfaces to elicit belief distributions |
| title_fullStr | Comparing input interfaces to elicit belief distributions |
| title_full_unstemmed | Comparing input interfaces to elicit belief distributions |
| title_short | Comparing input interfaces to elicit belief distributions |
| title_sort | comparing input interfaces to elicit belief distributions |
| topic | belief elicitation forecasting scoring rules interfaces |
| url | https://www.cambridge.org/core/product/identifier/S1930297523000219/type/journal_article |
| work_keys_str_mv | AT paolocrosetto comparinginputinterfacestoelicitbeliefdistributions AT thomasdehaan comparinginputinterfacestoelicitbeliefdistributions |