A response to Mandel’s (2019) commentary on Stastny and Lehner (2018)
Stastny and Lehner (2018) compared the accuracy of forecasts in an intelligence community prediction market to comparable forecasts in analysis reports prepared by groups of professional intelligence analysts. To obtain quantitative probabilities from the analysis reports experienced analysts were a...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Cambridge University Press
2019-09-01
|
| Series: | Judgment and Decision Making |
| Subjects: | |
| Online Access: | https://www.cambridge.org/core/product/identifier/S1930297500004915/type/journal_article |
| _version_ | 1797696024508104704 |
|---|---|
| author | Paul Lehner Bradley Stastny |
| author_facet | Paul Lehner Bradley Stastny |
| author_sort | Paul Lehner |
| collection | DOAJ |
| description | Stastny and Lehner (2018) compared the accuracy of forecasts in an intelligence community prediction market to comparable forecasts in analysis reports prepared by groups of professional intelligence analysts. To obtain quantitative probabilities from the analysis reports experienced analysts were asked to read the reports and state what probability they thought the reports implied for each forecast question. These were called imputed probabilities. Stastny and Lehner found that the prediction market was more accurate than the imputed probabilities and concluded that this was evidence that the prediction market was more accurate than the analysis reports. In a commentary, Mandel (2019) took exception to this interpretation. In a re-analysis of the data, Mandel found a very strong correlation between readers’ personal and imputed probabilities. From this Mandel builds a case that the imputed probabilities are little more than a reflection of the readers’ personal views; that they do not fairly reflect the contents of the analysis reports; and therefore, any accuracy results are spurious. This paper argues two points. First, the high correlation between imputed and personal probabilities was not evidence of substantial imputation bias. Rather it was the natural by-product of the fact that the imputed and personal probabilities were both forecasts of the same events. An additional analysis shows a much lower level of imputation bias that is consistent with the original results and interpretation. Second, the focus of Stastny and Lehner (2018) was on the reports as understood by readers. In this context, even if there was substantial imputation bias it would not invalidate accuracy results; it would instead provide a possible causal explanation of those results. |
| first_indexed | 2024-03-12T03:20:36Z |
| format | Article |
| id | doaj.art-e8855a90794444aa80362f1508435d81 |
| institution | Directory Open Access Journal |
| issn | 1930-2975 |
| language | English |
| last_indexed | 2024-03-12T03:20:36Z |
| publishDate | 2019-09-01 |
| publisher | Cambridge University Press |
| record_format | Article |
| series | Judgment and Decision Making |
| spelling | doaj.art-e8855a90794444aa80362f1508435d812023-09-03T14:02:45ZengCambridge University PressJudgment and Decision Making1930-29752019-09-011461761910.1017/S1930297500004915A response to Mandel’s (2019) commentary on Stastny and Lehner (2018)Paul Lehner0Bradley Stastny1The MITRE CorporationWork completed while at The MITRE Corporation, currently at GoogleStastny and Lehner (2018) compared the accuracy of forecasts in an intelligence community prediction market to comparable forecasts in analysis reports prepared by groups of professional intelligence analysts. To obtain quantitative probabilities from the analysis reports experienced analysts were asked to read the reports and state what probability they thought the reports implied for each forecast question. These were called imputed probabilities. Stastny and Lehner found that the prediction market was more accurate than the imputed probabilities and concluded that this was evidence that the prediction market was more accurate than the analysis reports. In a commentary, Mandel (2019) took exception to this interpretation. In a re-analysis of the data, Mandel found a very strong correlation between readers’ personal and imputed probabilities. From this Mandel builds a case that the imputed probabilities are little more than a reflection of the readers’ personal views; that they do not fairly reflect the contents of the analysis reports; and therefore, any accuracy results are spurious. This paper argues two points. First, the high correlation between imputed and personal probabilities was not evidence of substantial imputation bias. Rather it was the natural by-product of the fact that the imputed and personal probabilities were both forecasts of the same events. An additional analysis shows a much lower level of imputation bias that is consistent with the original results and interpretation. Second, the focus of Stastny and Lehner (2018) was on the reports as understood by readers. In this context, even if there was substantial imputation bias it would not invalidate accuracy results; it would instead provide a possible causal explanation of those results.https://www.cambridge.org/core/product/identifier/S1930297500004915/type/journal_articleforecastsprediction marketimputed probability |
| spellingShingle | Paul Lehner Bradley Stastny A response to Mandel’s (2019) commentary on Stastny and Lehner (2018) Judgment and Decision Making forecasts prediction market imputed probability |
| title | A response to Mandel’s (2019) commentary on Stastny and Lehner (2018) |
| title_full | A response to Mandel’s (2019) commentary on Stastny and Lehner (2018) |
| title_fullStr | A response to Mandel’s (2019) commentary on Stastny and Lehner (2018) |
| title_full_unstemmed | A response to Mandel’s (2019) commentary on Stastny and Lehner (2018) |
| title_short | A response to Mandel’s (2019) commentary on Stastny and Lehner (2018) |
| title_sort | response to mandel s 2019 commentary on stastny and lehner 2018 |
| topic | forecasts prediction market imputed probability |
| url | https://www.cambridge.org/core/product/identifier/S1930297500004915/type/journal_article |
| work_keys_str_mv | AT paullehner aresponsetomandels2019commentaryonstastnyandlehner2018 AT bradleystastny aresponsetomandels2019commentaryonstastnyandlehner2018 AT paullehner responsetomandels2019commentaryonstastnyandlehner2018 AT bradleystastny responsetomandels2019commentaryonstastnyandlehner2018 |