A model-based approach for the analysis of the calibration of probability judgments
The calibration of probability or confidence judgments concerns the association between the judgments and some estimate of the correct probabilities of events. Researchers rely on estimates using relative frequencies computed by aggregating data over observations. We show that this approach creates...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Cambridge University Press
2011-12-01
|
| Series: | Judgment and Decision Making |
| Subjects: | |
| Online Access: | https://www.cambridge.org/core/product/identifier/S1930297500004277/type/journal_article |
| _version_ | 1797701648071524352 |
|---|---|
| author | David V. Budescu Timothy R. Johnson Andreas Glöckner Benjamin E. Hilbig |
| author_facet | David V. Budescu Timothy R. Johnson Andreas Glöckner Benjamin E. Hilbig |
| author_sort | David V. Budescu |
| collection | DOAJ |
| description | The calibration of probability or confidence judgments concerns the association between the judgments and some estimate of the correct probabilities of events. Researchers rely on estimates using relative frequencies computed by aggregating data over observations. We show that this approach creates conceptual problems, and may result in the confounding of explanatory variables or unstable estimates. To circumvent these problems we propose using probability estimates obtained from statistical models—specifically mixed models for binary data—in the analysis of calibration. We illustrate this methodology by re-analyzing data from a published study and comparing the results from this approach to those based on relative frequencies. The model-based estimates avoid problems with confounding variables and provided more precise estimates, resulting in better inferences. |
| first_indexed | 2024-03-12T04:38:42Z |
| format | Article |
| id | doaj.art-18eb30015f934c0ebd6f95603502cf7c |
| institution | Directory Open Access Journal |
| issn | 1930-2975 |
| language | English |
| last_indexed | 2024-03-12T04:38:42Z |
| publishDate | 2011-12-01 |
| publisher | Cambridge University Press |
| record_format | Article |
| series | Judgment and Decision Making |
| spelling | doaj.art-18eb30015f934c0ebd6f95603502cf7c2023-09-03T09:46:15ZengCambridge University PressJudgment and Decision Making1930-29752011-12-01685786910.1017/S1930297500004277A model-based approach for the analysis of the calibration of probability judgmentsDavid V. Budescu0Timothy R. Johnson1Andreas GlöcknerBenjamin E. HilbigDepartment of Psychology, Fordham University, Dealy Hall, 411 East Fordham Road, Bronx, NY, 10458, USADepartment of Statistics, University of IdahoThe calibration of probability or confidence judgments concerns the association between the judgments and some estimate of the correct probabilities of events. Researchers rely on estimates using relative frequencies computed by aggregating data over observations. We show that this approach creates conceptual problems, and may result in the confounding of explanatory variables or unstable estimates. To circumvent these problems we propose using probability estimates obtained from statistical models—specifically mixed models for binary data—in the analysis of calibration. We illustrate this methodology by re-analyzing data from a published study and comparing the results from this approach to those based on relative frequencies. The model-based estimates avoid problems with confounding variables and provided more precise estimates, resulting in better inferences.https://www.cambridge.org/core/product/identifier/S1930297500004277/type/journal_articlecalibrationconfidence judgmentsmixed modelsmultilevel modelsoverconfidencesubjective probability |
| spellingShingle | David V. Budescu Timothy R. Johnson Andreas Glöckner Benjamin E. Hilbig A model-based approach for the analysis of the calibration of probability judgments Judgment and Decision Making calibration confidence judgments mixed models multilevel models overconfidence subjective probability |
| title | A model-based approach for the analysis of the calibration of probability judgments |
| title_full | A model-based approach for the analysis of the calibration of probability judgments |
| title_fullStr | A model-based approach for the analysis of the calibration of probability judgments |
| title_full_unstemmed | A model-based approach for the analysis of the calibration of probability judgments |
| title_short | A model-based approach for the analysis of the calibration of probability judgments |
| title_sort | model based approach for the analysis of the calibration of probability judgments |
| topic | calibration confidence judgments mixed models multilevel models overconfidence subjective probability |
| url | https://www.cambridge.org/core/product/identifier/S1930297500004277/type/journal_article |
| work_keys_str_mv | AT davidvbudescu amodelbasedapproachfortheanalysisofthecalibrationofprobabilityjudgments AT timothyrjohnson amodelbasedapproachfortheanalysisofthecalibrationofprobabilityjudgments AT andreasglockner amodelbasedapproachfortheanalysisofthecalibrationofprobabilityjudgments AT benjaminehilbig amodelbasedapproachfortheanalysisofthecalibrationofprobabilityjudgments AT davidvbudescu modelbasedapproachfortheanalysisofthecalibrationofprobabilityjudgments AT timothyrjohnson modelbasedapproachfortheanalysisofthecalibrationofprobabilityjudgments AT andreasglockner modelbasedapproachfortheanalysisofthecalibrationofprobabilityjudgments AT benjaminehilbig modelbasedapproachfortheanalysisofthecalibrationofprobabilityjudgments |