The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt
Infectious disease forecasting is gaining traction in the public health community; however, limited systematic comparisons of model performance exist. Here we present the results of a synthetic forecasting challenge inspired by the West African Ebola crisis in 2014–2015 and involving 16 internationa...
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2018-03-01
|
| Series: | Epidemics |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S1755436517301275 |
| _version_ | 1819084909246939136 |
|---|---|
| author | Cécile Viboud Kaiyuan Sun Robert Gaffey Marco Ajelli Laura Fumanelli Stefano Merler Qian Zhang Gerardo Chowell Lone Simonsen Alessandro Vespignani |
| author_facet | Cécile Viboud Kaiyuan Sun Robert Gaffey Marco Ajelli Laura Fumanelli Stefano Merler Qian Zhang Gerardo Chowell Lone Simonsen Alessandro Vespignani |
| author_sort | Cécile Viboud |
| collection | DOAJ |
| description | Infectious disease forecasting is gaining traction in the public health community; however, limited systematic comparisons of model performance exist. Here we present the results of a synthetic forecasting challenge inspired by the West African Ebola crisis in 2014–2015 and involving 16 international academic teams and US government agencies, and compare the predictive performance of 8 independent modeling approaches. Challenge participants were invited to predict 140 epidemiological targets across 5 different time points of 4 synthetic Ebola outbreaks, each involving different levels of interventions and “fog of war” in outbreak data made available for predictions. Prediction targets included 1–4 week-ahead case incidences, outbreak size, peak timing, and several natural history parameters. With respect to weekly case incidence targets, ensemble predictions based on a Bayesian average of the 8 participating models outperformed any individual model and did substantially better than a null auto-regressive model. There was no relationship between model complexity and prediction accuracy; however, the top performing models for short-term weekly incidence were reactive models with few parameters, fitted to a short and recent part of the outbreak. Individual model outputs and ensemble predictions improved with data accuracy and availability; by the second time point, just before the peak of the epidemic, estimates of final size were within 20% of the target. The 4th challenge scenario − mirroring an uncontrolled Ebola outbreak with substantial data reporting noise − was poorly predicted by all modeling teams. Overall, this synthetic forecasting challenge provided a deep understanding of model performance under controlled data and epidemiological conditions. We recommend such “peace time” forecasting challenges as key elements to improve coordination and inspire collaboration between modeling groups ahead of the next pandemic threat, and to assess model forecasting accuracy for a variety of known and hypothetical pathogens. Keywords: Ebola epidemic, Mathematical modeling, Forecasting challenge, Model comparison, Synthetic data, Prediction performance, Prediction horizon, Data accuracy |
| first_indexed | 2024-12-21T20:55:57Z |
| format | Article |
| id | doaj.art-4bf3bb8f78d246c0870d39b014016dcb |
| institution | Directory Open Access Journal |
| issn | 1755-4365 |
| language | English |
| last_indexed | 2024-12-21T20:55:57Z |
| publishDate | 2018-03-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Epidemics |
| spelling | doaj.art-4bf3bb8f78d246c0870d39b014016dcb2022-12-21T18:50:34ZengElsevierEpidemics1755-43652018-03-01221321The RAPIDD ebola forecasting challenge: Synthesis and lessons learntCécile Viboud0Kaiyuan Sun1Robert Gaffey2Marco Ajelli3Laura Fumanelli4Stefano Merler5Qian Zhang6Gerardo Chowell7Lone Simonsen8Alessandro Vespignani9Division of International Epidemiology and Population Studies, Fogarty International Center, National Institutes of Health, Bethesda, MD, USA; Corresponding author.Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA, USADivision of International Epidemiology and Population Studies, Fogarty International Center, National Institutes of Health, Bethesda, MD, USABruno Kessler Foundation, Trento, ItalyBruno Kessler Foundation, Trento, ItalyBruno Kessler Foundation, Trento, ItalyLaboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA, USADivision of International Epidemiology and Population Studies, Fogarty International Center, National Institutes of Health, Bethesda, MD, USA; School of Public Health, Georgia State University, Atlanta, GA, USADivision of International Epidemiology and Population Studies, Fogarty International Center, National Institutes of Health, Bethesda, MD, USA; Department of Global Health, George Washington University, Washington DC, USALaboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA, USA; Institute for Quantitative Social Sciences at Harvard University, Cambridge, MA, USA; Institute for Scientific Interchange Foundation, Turin, ItalyInfectious disease forecasting is gaining traction in the public health community; however, limited systematic comparisons of model performance exist. Here we present the results of a synthetic forecasting challenge inspired by the West African Ebola crisis in 2014–2015 and involving 16 international academic teams and US government agencies, and compare the predictive performance of 8 independent modeling approaches. Challenge participants were invited to predict 140 epidemiological targets across 5 different time points of 4 synthetic Ebola outbreaks, each involving different levels of interventions and “fog of war” in outbreak data made available for predictions. Prediction targets included 1–4 week-ahead case incidences, outbreak size, peak timing, and several natural history parameters. With respect to weekly case incidence targets, ensemble predictions based on a Bayesian average of the 8 participating models outperformed any individual model and did substantially better than a null auto-regressive model. There was no relationship between model complexity and prediction accuracy; however, the top performing models for short-term weekly incidence were reactive models with few parameters, fitted to a short and recent part of the outbreak. Individual model outputs and ensemble predictions improved with data accuracy and availability; by the second time point, just before the peak of the epidemic, estimates of final size were within 20% of the target. The 4th challenge scenario − mirroring an uncontrolled Ebola outbreak with substantial data reporting noise − was poorly predicted by all modeling teams. Overall, this synthetic forecasting challenge provided a deep understanding of model performance under controlled data and epidemiological conditions. We recommend such “peace time” forecasting challenges as key elements to improve coordination and inspire collaboration between modeling groups ahead of the next pandemic threat, and to assess model forecasting accuracy for a variety of known and hypothetical pathogens. Keywords: Ebola epidemic, Mathematical modeling, Forecasting challenge, Model comparison, Synthetic data, Prediction performance, Prediction horizon, Data accuracyhttp://www.sciencedirect.com/science/article/pii/S1755436517301275 |
| spellingShingle | Cécile Viboud Kaiyuan Sun Robert Gaffey Marco Ajelli Laura Fumanelli Stefano Merler Qian Zhang Gerardo Chowell Lone Simonsen Alessandro Vespignani The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt Epidemics |
| title | The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt |
| title_full | The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt |
| title_fullStr | The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt |
| title_full_unstemmed | The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt |
| title_short | The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt |
| title_sort | rapidd ebola forecasting challenge synthesis and lessons learnt |
| url | http://www.sciencedirect.com/science/article/pii/S1755436517301275 |
| work_keys_str_mv | AT cecileviboud therapiddebolaforecastingchallengesynthesisandlessonslearnt AT kaiyuansun therapiddebolaforecastingchallengesynthesisandlessonslearnt AT robertgaffey therapiddebolaforecastingchallengesynthesisandlessonslearnt AT marcoajelli therapiddebolaforecastingchallengesynthesisandlessonslearnt AT laurafumanelli therapiddebolaforecastingchallengesynthesisandlessonslearnt AT stefanomerler therapiddebolaforecastingchallengesynthesisandlessonslearnt AT qianzhang therapiddebolaforecastingchallengesynthesisandlessonslearnt AT gerardochowell therapiddebolaforecastingchallengesynthesisandlessonslearnt AT lonesimonsen therapiddebolaforecastingchallengesynthesisandlessonslearnt AT alessandrovespignani therapiddebolaforecastingchallengesynthesisandlessonslearnt AT cecileviboud rapiddebolaforecastingchallengesynthesisandlessonslearnt AT kaiyuansun rapiddebolaforecastingchallengesynthesisandlessonslearnt AT robertgaffey rapiddebolaforecastingchallengesynthesisandlessonslearnt AT marcoajelli rapiddebolaforecastingchallengesynthesisandlessonslearnt AT laurafumanelli rapiddebolaforecastingchallengesynthesisandlessonslearnt AT stefanomerler rapiddebolaforecastingchallengesynthesisandlessonslearnt AT qianzhang rapiddebolaforecastingchallengesynthesisandlessonslearnt AT gerardochowell rapiddebolaforecastingchallengesynthesisandlessonslearnt AT lonesimonsen rapiddebolaforecastingchallengesynthesisandlessonslearnt AT alessandrovespignani rapiddebolaforecastingchallengesynthesisandlessonslearnt |