Forecasting forecaster accuracy: Contributions of past performance and individual differences
A growing body of research indicates that forecasting skill is a unique and stable trait: forecasters with a track record of high accuracy tend to maintain this record. But how does one identify skilled forecasters effectively? We address this question using data collected during two seasons of a lo...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Cambridge University Press
2021-03-01
|
| Series: | Judgment and Decision Making |
| Subjects: | |
| Online Access: | https://www.cambridge.org/core/product/identifier/S1930297500008597/type/journal_article |
| _version_ | 1797701678616543232 |
|---|---|
| author | Mark Himmelstein Pavel Atanasov David V. Budescu |
| author_facet | Mark Himmelstein Pavel Atanasov David V. Budescu |
| author_sort | Mark Himmelstein |
| collection | DOAJ |
| description | A growing body of research indicates that forecasting skill is a unique and stable trait: forecasters with a track record of high accuracy tend to maintain this record. But how does one identify skilled forecasters effectively? We address this question using data collected during two seasons of a longitudinal geopolitical forecasting tournament. Our first analysis, which compares psychometric traits assessed prior to forecasting, indicates intelligence consistently predicts accuracy. Next, using methods adapted from classical test theory and item response theory, we model latent forecasting skill based on the forecasters’ past accuracy, while accounting for the timing of their forecasts relative to question resolution. Our results suggest these methods perform better at assessing forecasting skill than simpler methods employed by many previous studies. By parsing the data at different time points during the competitions, we assess the relative importance of each information source over time. When past performance information is limited, psychometric traits are useful predictors of future performance, but, as more information becomes available, past performance becomes the stronger predictor of future accuracy. Finally, we demonstrate the predictive validity of these results on out-of-sample data, and their utility in producing performance weights for wisdom-of-crowds aggregations. |
| first_indexed | 2024-03-12T04:40:09Z |
| format | Article |
| id | doaj.art-8f3f09157bf64427b5db4b29dc0ae3dd |
| institution | Directory Open Access Journal |
| issn | 1930-2975 |
| language | English |
| last_indexed | 2024-03-12T04:40:09Z |
| publishDate | 2021-03-01 |
| publisher | Cambridge University Press |
| record_format | Article |
| series | Judgment and Decision Making |
| spelling | doaj.art-8f3f09157bf64427b5db4b29dc0ae3dd2023-09-03T09:45:57ZengCambridge University PressJudgment and Decision Making1930-29752021-03-011632336210.1017/S1930297500008597Forecasting forecaster accuracy: Contributions of past performance and individual differencesMark Himmelstein0https://orcid.org/0000-0001-8681-0482Pavel Atanasov1https://orcid.org/0000-0002-9963-7225David V. Budescu2https://orcid.org/0000-0001-9613-0317Department of Psychology, Fordham UniversityPytho LLCDepartment of Psychology, Fordham UniversityA growing body of research indicates that forecasting skill is a unique and stable trait: forecasters with a track record of high accuracy tend to maintain this record. But how does one identify skilled forecasters effectively? We address this question using data collected during two seasons of a longitudinal geopolitical forecasting tournament. Our first analysis, which compares psychometric traits assessed prior to forecasting, indicates intelligence consistently predicts accuracy. Next, using methods adapted from classical test theory and item response theory, we model latent forecasting skill based on the forecasters’ past accuracy, while accounting for the timing of their forecasts relative to question resolution. Our results suggest these methods perform better at assessing forecasting skill than simpler methods employed by many previous studies. By parsing the data at different time points during the competitions, we assess the relative importance of each information source over time. When past performance information is limited, psychometric traits are useful predictors of future performance, but, as more information becomes available, past performance becomes the stronger predictor of future accuracy. Finally, we demonstrate the predictive validity of these results on out-of-sample data, and their utility in producing performance weights for wisdom-of-crowds aggregations.https://www.cambridge.org/core/product/identifier/S1930297500008597/type/journal_articleforecastingindividual differencesitem response modelslongitudinal analysisskill assessmentwisdom-of-crowdshybrid forecasting competition |
| spellingShingle | Mark Himmelstein Pavel Atanasov David V. Budescu Forecasting forecaster accuracy: Contributions of past performance and individual differences Judgment and Decision Making forecasting individual differences item response models longitudinal analysis skill assessment wisdom-of-crowds hybrid forecasting competition |
| title | Forecasting forecaster accuracy: Contributions of past performance and individual differences |
| title_full | Forecasting forecaster accuracy: Contributions of past performance and individual differences |
| title_fullStr | Forecasting forecaster accuracy: Contributions of past performance and individual differences |
| title_full_unstemmed | Forecasting forecaster accuracy: Contributions of past performance and individual differences |
| title_short | Forecasting forecaster accuracy: Contributions of past performance and individual differences |
| title_sort | forecasting forecaster accuracy contributions of past performance and individual differences |
| topic | forecasting individual differences item response models longitudinal analysis skill assessment wisdom-of-crowds hybrid forecasting competition |
| url | https://www.cambridge.org/core/product/identifier/S1930297500008597/type/journal_article |
| work_keys_str_mv | AT markhimmelstein forecastingforecasteraccuracycontributionsofpastperformanceandindividualdifferences AT pavelatanasov forecastingforecasteraccuracycontributionsofpastperformanceandindividualdifferences AT davidvbudescu forecastingforecasteraccuracycontributionsofpastperformanceandindividualdifferences |