A model-based approach for the analysis of the calibration of probability judgments
The calibration of probability or confidence judgments concerns the association between the judgments and some estimate of the correct probabilities of events. Researchers rely on estimates using relative frequencies computed by aggregating data over observations. We show that this approach creates...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Cambridge University Press
2011-12-01
|
| Series: | Judgment and Decision Making |
| Subjects: | |
| Online Access: | http://journal.sjdm.org/11/m01/m01.pdf |
| _version_ | 1797717612540461056 |
|---|---|
| author | David V. Budescu Timothy R. Johnson |
| author_facet | David V. Budescu Timothy R. Johnson |
| author_sort | David V. Budescu |
| collection | DOAJ |
| description | The calibration of probability or confidence judgments concerns the association between the judgments and some estimate of the correct probabilities of events. Researchers rely on estimates using relative frequencies computed by aggregating data over observations. We show that this approach creates conceptual problems, and may result in the confounding of explanatory variables or unstable estimates. To circumvent these problems we propose using probability estimates obtained from statistical models---specifically mixed models for binary data---in the analysis of calibration. We illustrate this methodology by re-analyzing data from a published study and comparing the results from this approach to those based on relative frequencies. The model-based estimates avoid problems with confounding variables and provided more precise estimates, resulting in better inferences. |
| first_indexed | 2024-03-12T08:38:53Z |
| format | Article |
| id | doaj.art-8760c67a140f4197976b1a33174bbe51 |
| institution | Directory Open Access Journal |
| issn | 1930-2975 |
| language | English |
| last_indexed | 2024-03-12T08:38:53Z |
| publishDate | 2011-12-01 |
| publisher | Cambridge University Press |
| record_format | Article |
| series | Judgment and Decision Making |
| spelling | doaj.art-8760c67a140f4197976b1a33174bbe512023-09-02T17:01:27ZengCambridge University PressJudgment and Decision Making1930-29752011-12-0168857869A model-based approach for the analysis of the calibration of probability judgmentsDavid V. BudescuTimothy R. JohnsonThe calibration of probability or confidence judgments concerns the association between the judgments and some estimate of the correct probabilities of events. Researchers rely on estimates using relative frequencies computed by aggregating data over observations. We show that this approach creates conceptual problems, and may result in the confounding of explanatory variables or unstable estimates. To circumvent these problems we propose using probability estimates obtained from statistical models---specifically mixed models for binary data---in the analysis of calibration. We illustrate this methodology by re-analyzing data from a published study and comparing the results from this approach to those based on relative frequencies. The model-based estimates avoid problems with confounding variables and provided more precise estimates, resulting in better inferences.http://journal.sjdm.org/11/m01/m01.pdfcalibrationconfidence judgmentsmixed modelsmultilevel modelsoverconfidencesubjective probability.NAKeywords |
| spellingShingle | David V. Budescu Timothy R. Johnson A model-based approach for the analysis of the calibration of probability judgments Judgment and Decision Making calibration confidence judgments mixed models multilevel models overconfidence subjective probability.NAKeywords |
| title | A model-based approach for the analysis of the calibration of probability judgments |
| title_full | A model-based approach for the analysis of the calibration of probability judgments |
| title_fullStr | A model-based approach for the analysis of the calibration of probability judgments |
| title_full_unstemmed | A model-based approach for the analysis of the calibration of probability judgments |
| title_short | A model-based approach for the analysis of the calibration of probability judgments |
| title_sort | model based approach for the analysis of the calibration of probability judgments |
| topic | calibration confidence judgments mixed models multilevel models overconfidence subjective probability.NAKeywords |
| url | http://journal.sjdm.org/11/m01/m01.pdf |
| work_keys_str_mv | AT davidvbudescu amodelbasedapproachfortheanalysisofthecalibrationofprobabilityjudgments AT timothyrjohnson amodelbasedapproachfortheanalysisofthecalibrationofprobabilityjudgments AT davidvbudescu modelbasedapproachfortheanalysisofthecalibrationofprobabilityjudgments AT timothyrjohnson modelbasedapproachfortheanalysisofthecalibrationofprobabilityjudgments |