Aggregating multiple probability intervals to improve calibration
We apply the principles of the “Wisdom of Crowds (WoC)” to improve the calibration of interval estimates. Previous research has documented the significant impact of the WoC on the accuracy of point estimates but only a few studies have examined its effectiveness in aggregating interval estimates. We...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Cambridge University Press
2015-03-01
|
| Series: | Judgment and Decision Making |
| Subjects: | |
| Online Access: | https://www.cambridge.org/core/product/identifier/S1930297500003910/type/journal_article |
| _version_ | 1827828192190136320 |
|---|---|
| author | Saemi Park David V. Budescu |
| author_facet | Saemi Park David V. Budescu |
| author_sort | Saemi Park |
| collection | DOAJ |
| description | We apply the principles of the “Wisdom of Crowds (WoC)” to improve the calibration of interval estimates. Previous research has documented the significant impact of the WoC on the accuracy of point estimates but only a few studies have examined its effectiveness in aggregating interval estimates. We demonstrate that collective probability intervals obtained by several heuristics can reduce the typical overconfidence of the individual estimates. We re-analyzed data from Glaser, Langer and Weber (2013) and from Soll and Klayman (2004) and applied four heuristics Averaging, Median, Enveloping, Probability averaging-suggested by Gaba, Tsetlin and Winkler (2014) and new heuristics, Averaging with trimming and Quartiles. We used the hit rate and the Mean Squared Error (MSE) to evaluate the quality of the methods. All methods reduced miscalibration to some degree, and Quartiles was the most beneficial securing accuracy and informativeness. |
| first_indexed | 2024-03-12T03:47:30Z |
| format | Article |
| id | doaj.art-68083682ba7f4b89808ada4435338327 |
| institution | Directory Open Access Journal |
| issn | 1930-2975 |
| language | English |
| last_indexed | 2024-03-12T03:47:30Z |
| publishDate | 2015-03-01 |
| publisher | Cambridge University Press |
| record_format | Article |
| series | Judgment and Decision Making |
| spelling | doaj.art-68083682ba7f4b89808ada44353383272023-09-03T12:43:18ZengCambridge University PressJudgment and Decision Making1930-29752015-03-011013014310.1017/S1930297500003910Aggregating multiple probability intervals to improve calibrationSaemi Park0David V. Budescu1Department of Psychology, Fordham UniversityCorresponding author: Department of Psychology, Fordham University, Dealy Hall, 411 East Fordham Road, Bronx, NY, 10458We apply the principles of the “Wisdom of Crowds (WoC)” to improve the calibration of interval estimates. Previous research has documented the significant impact of the WoC on the accuracy of point estimates but only a few studies have examined its effectiveness in aggregating interval estimates. We demonstrate that collective probability intervals obtained by several heuristics can reduce the typical overconfidence of the individual estimates. We re-analyzed data from Glaser, Langer and Weber (2013) and from Soll and Klayman (2004) and applied four heuristics Averaging, Median, Enveloping, Probability averaging-suggested by Gaba, Tsetlin and Winkler (2014) and new heuristics, Averaging with trimming and Quartiles. We used the hit rate and the Mean Squared Error (MSE) to evaluate the quality of the methods. All methods reduced miscalibration to some degree, and Quartiles was the most beneficial securing accuracy and informativeness.https://www.cambridge.org/core/product/identifier/S1930297500003910/type/journal_articleoverconfidencesubjective probabilityprobability intervalshit rateWisdom of Crowds |
| spellingShingle | Saemi Park David V. Budescu Aggregating multiple probability intervals to improve calibration Judgment and Decision Making overconfidence subjective probability probability intervals hit rate Wisdom of Crowds |
| title | Aggregating multiple probability intervals to improve calibration |
| title_full | Aggregating multiple probability intervals to improve calibration |
| title_fullStr | Aggregating multiple probability intervals to improve calibration |
| title_full_unstemmed | Aggregating multiple probability intervals to improve calibration |
| title_short | Aggregating multiple probability intervals to improve calibration |
| title_sort | aggregating multiple probability intervals to improve calibration |
| topic | overconfidence subjective probability probability intervals hit rate Wisdom of Crowds |
| url | https://www.cambridge.org/core/product/identifier/S1930297500003910/type/journal_article |
| work_keys_str_mv | AT saemipark aggregatingmultipleprobabilityintervalstoimprovecalibration AT davidvbudescu aggregatingmultipleprobabilityintervalstoimprovecalibration |