Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets with application to COVID eventsResearch in context
Summary: Background: The recent COVID-19 pandemic highlighted the challenges for traditional forecasting. Prediction markets are a promising way to generate collective forecasts and could potentially be enhanced if high-quality crowdsourced inputs were identified and preferentially weighted for lik...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-10-01
|
| Series: | EBioMedicine |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2352396423003493 |
| _version_ | 1827819965911138304 |
|---|---|
| author | Alexander Gruen Karl R. Mattingly Ellen Morwitch Frederik Bossaerts Manning Clifford Chad Nash John P.A. Ioannidis Anne-Louise Ponsonby |
| author_facet | Alexander Gruen Karl R. Mattingly Ellen Morwitch Frederik Bossaerts Manning Clifford Chad Nash John P.A. Ioannidis Anne-Louise Ponsonby |
| author_sort | Alexander Gruen |
| collection | DOAJ |
| description | Summary: Background: The recent COVID-19 pandemic highlighted the challenges for traditional forecasting. Prediction markets are a promising way to generate collective forecasts and could potentially be enhanced if high-quality crowdsourced inputs were identified and preferentially weighted for likely accuracy in real-time with machine learning. Methods: We aim to leverage human prediction markets with real-time machine weighting of likely higher accuracy trades to improve performance. The crowd sourced Almanis prediction market longitudinal platform (n = 1822) and Next Generation Social Science (NGS2) platform (n = 103) were utilised. Findings: A 43-feature model predicted accurate forecasters, those with top quintile relative Brier accuracy, with subsequent replication in two out-of-sample datasets (pboth <1 × 10−9). Trades graded by this model as having higher accuracy scores than others produced a greater AUC temporal gain in the overall market after vs before trade. Accuracy score-weighted forecasts had higher accuracy than market forecasts alone, particularly when the two systems disagreed by 5% or more for binary event prediction: the hybrid system demonstrating substantial % AUC gains of 13.2%, p = 1.35 × 10−14 and 13.8%, p = 0.003 in two out-of-sample datasets. When discordant, the hybrid model was correct for COVID-19 event occurrence 72.7% of the time vs 27.3% for market models, p = 0.007. This net classification benefit was replicated in the separate Almanis B dataset, p = 2.4 × 10−7. Interpretation: Real-time machine classification followed by weighting human trades according to likely accuracy improves collective forecasting performance. This could provide improved anticipation of and thus response to emerging risks. Funding: This work was supported by an AusIndustry R and D tax incentive program from the Department of Industry, Science, Energy and Resources, Australia, to SlowVoice Pty Ltd. (IR 2101990) and Fellowship (GNT 1110200) and Investigator grant (GNT 1197234) to A-L Ponsonby by the National Health and Medical Research Council of Australia. |
| first_indexed | 2024-03-12T01:21:32Z |
| format | Article |
| id | doaj.art-765e063c58d14d58926bc5310f12ffd3 |
| institution | Directory Open Access Journal |
| issn | 2352-3964 |
| language | English |
| last_indexed | 2024-03-12T01:21:32Z |
| publishDate | 2023-10-01 |
| publisher | Elsevier |
| record_format | Article |
| series | EBioMedicine |
| spelling | doaj.art-765e063c58d14d58926bc5310f12ffd32023-09-13T04:25:09ZengElsevierEBioMedicine2352-39642023-10-0196104783Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets with application to COVID eventsResearch in contextAlexander Gruen0Karl R. Mattingly1Ellen Morwitch2Frederik Bossaerts3Manning Clifford4Chad Nash5John P.A. Ioannidis6Anne-Louise Ponsonby7The Florey Institute of Neuroscience and Mental Health, Melbourne, AustraliaDysrupt Labs by SlowVoice, Melbourne, AustraliaThe Florey Institute of Neuroscience and Mental Health, Melbourne, AustraliaDysrupt Labs by SlowVoice, Melbourne, AustraliaBoston Consulting Group, Melbourne, AustraliaDysrupt Labs by SlowVoice, Melbourne, AustraliaStanford Prevention Research Center, Department of Medicine, and Departments of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, Stanford University, Meta-Research Innovation Center at Stanford, Stanford, CA, USAThe Florey Institute of Neuroscience and Mental Health, Melbourne, Australia; Murdoch Children's Research Institute, Royal Children's Hospital, Melbourne, Australia; Centre of Epidemiology and Biostatistics, School of Population and Global Health, University of Melbourne, Australia; Corresponding author. The Florey Institute of Neuroscience and Mental Health; Melbourne, Australia.Summary: Background: The recent COVID-19 pandemic highlighted the challenges for traditional forecasting. Prediction markets are a promising way to generate collective forecasts and could potentially be enhanced if high-quality crowdsourced inputs were identified and preferentially weighted for likely accuracy in real-time with machine learning. Methods: We aim to leverage human prediction markets with real-time machine weighting of likely higher accuracy trades to improve performance. The crowd sourced Almanis prediction market longitudinal platform (n = 1822) and Next Generation Social Science (NGS2) platform (n = 103) were utilised. Findings: A 43-feature model predicted accurate forecasters, those with top quintile relative Brier accuracy, with subsequent replication in two out-of-sample datasets (pboth <1 × 10−9). Trades graded by this model as having higher accuracy scores than others produced a greater AUC temporal gain in the overall market after vs before trade. Accuracy score-weighted forecasts had higher accuracy than market forecasts alone, particularly when the two systems disagreed by 5% or more for binary event prediction: the hybrid system demonstrating substantial % AUC gains of 13.2%, p = 1.35 × 10−14 and 13.8%, p = 0.003 in two out-of-sample datasets. When discordant, the hybrid model was correct for COVID-19 event occurrence 72.7% of the time vs 27.3% for market models, p = 0.007. This net classification benefit was replicated in the separate Almanis B dataset, p = 2.4 × 10−7. Interpretation: Real-time machine classification followed by weighting human trades according to likely accuracy improves collective forecasting performance. This could provide improved anticipation of and thus response to emerging risks. Funding: This work was supported by an AusIndustry R and D tax incentive program from the Department of Industry, Science, Energy and Resources, Australia, to SlowVoice Pty Ltd. (IR 2101990) and Fellowship (GNT 1110200) and Investigator grant (GNT 1197234) to A-L Ponsonby by the National Health and Medical Research Council of Australia.http://www.sciencedirect.com/science/article/pii/S2352396423003493Collective intelligenceForecast accuracyPrediction marketHuman-machine forecastCOVID |
| spellingShingle | Alexander Gruen Karl R. Mattingly Ellen Morwitch Frederik Bossaerts Manning Clifford Chad Nash John P.A. Ioannidis Anne-Louise Ponsonby Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets with application to COVID eventsResearch in context EBioMedicine Collective intelligence Forecast accuracy Prediction market Human-machine forecast COVID |
| title | Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets with application to COVID eventsResearch in context |
| title_full | Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets with application to COVID eventsResearch in context |
| title_fullStr | Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets with application to COVID eventsResearch in context |
| title_full_unstemmed | Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets with application to COVID eventsResearch in context |
| title_short | Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets with application to COVID eventsResearch in context |
| title_sort | machine learning augmentation reduces prediction error in collective forecasting development and validation across prediction markets with application to covid eventsresearch in context |
| topic | Collective intelligence Forecast accuracy Prediction market Human-machine forecast COVID |
| url | http://www.sciencedirect.com/science/article/pii/S2352396423003493 |
| work_keys_str_mv | AT alexandergruen machinelearningaugmentationreducespredictionerrorincollectiveforecastingdevelopmentandvalidationacrosspredictionmarketswithapplicationtocovideventsresearchincontext AT karlrmattingly machinelearningaugmentationreducespredictionerrorincollectiveforecastingdevelopmentandvalidationacrosspredictionmarketswithapplicationtocovideventsresearchincontext AT ellenmorwitch machinelearningaugmentationreducespredictionerrorincollectiveforecastingdevelopmentandvalidationacrosspredictionmarketswithapplicationtocovideventsresearchincontext AT frederikbossaerts machinelearningaugmentationreducespredictionerrorincollectiveforecastingdevelopmentandvalidationacrosspredictionmarketswithapplicationtocovideventsresearchincontext AT manningclifford machinelearningaugmentationreducespredictionerrorincollectiveforecastingdevelopmentandvalidationacrosspredictionmarketswithapplicationtocovideventsresearchincontext AT chadnash machinelearningaugmentationreducespredictionerrorincollectiveforecastingdevelopmentandvalidationacrosspredictionmarketswithapplicationtocovideventsresearchincontext AT johnpaioannidis machinelearningaugmentationreducespredictionerrorincollectiveforecastingdevelopmentandvalidationacrosspredictionmarketswithapplicationtocovideventsresearchincontext AT annelouiseponsonby machinelearningaugmentationreducespredictionerrorincollectiveforecastingdevelopmentandvalidationacrosspredictionmarketswithapplicationtocovideventsresearchincontext |