Scoring epidemiological forecasts on transformed scales.
Forecast evaluation is essential for the development of predictive epidemic models and can inform their use for public health decision-making. Common scores to evaluate epidemiological forecasts are the Continuous Ranked Probability Score (CRPS) and the Weighted Interval Score (WIS), which can be se...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2023-08-01
|
| Series: | PLoS Computational Biology |
| Online Access: | https://doi.org/10.1371/journal.pcbi.1011393 |
| _version_ | 1827348641303494656 |
|---|---|
| author | Nikos I Bosse Sam Abbott Anne Cori Edwin van Leeuwen Johannes Bracher Sebastian Funk |
| author_facet | Nikos I Bosse Sam Abbott Anne Cori Edwin van Leeuwen Johannes Bracher Sebastian Funk |
| author_sort | Nikos I Bosse |
| collection | DOAJ |
| description | Forecast evaluation is essential for the development of predictive epidemic models and can inform their use for public health decision-making. Common scores to evaluate epidemiological forecasts are the Continuous Ranked Probability Score (CRPS) and the Weighted Interval Score (WIS), which can be seen as measures of the absolute distance between the forecast distribution and the observation. However, applying these scores directly to predicted and observed incidence counts may not be the most appropriate due to the exponential nature of epidemic processes and the varying magnitudes of observed values across space and time. In this paper, we argue that transforming counts before applying scores such as the CRPS or WIS can effectively mitigate these difficulties and yield epidemiologically meaningful and easily interpretable results. Using the CRPS on log-transformed values as an example, we list three attractive properties: Firstly, it can be interpreted as a probabilistic version of a relative error. Secondly, it reflects how well models predicted the time-varying epidemic growth rate. And lastly, using arguments on variance-stabilizing transformations, it can be shown that under the assumption of a quadratic mean-variance relationship, the logarithmic transformation leads to expected CRPS values which are independent of the order of magnitude of the predicted quantity. Applying a transformation of log(x + 1) to data and forecasts from the European COVID-19 Forecast Hub, we find that it changes model rankings regardless of stratification by forecast date, location or target types. Situations in which models missed the beginning of upward swings are more strongly emphasised while failing to predict a downturn following a peak is less severely penalised when scoring transformed forecasts as opposed to untransformed ones. We conclude that appropriate transformations, of which the natural logarithm is only one particularly attractive option, should be considered when assessing the performance of different models in the context of infectious disease incidence. |
| first_indexed | 2024-03-08T00:15:11Z |
| format | Article |
| id | doaj.art-7ea04384a7214855b5bb4e6b4036f84e |
| institution | Directory Open Access Journal |
| issn | 1553-734X 1553-7358 |
| language | English |
| last_indexed | 2024-03-08T00:15:11Z |
| publishDate | 2023-08-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS Computational Biology |
| spelling | doaj.art-7ea04384a7214855b5bb4e6b4036f84e2024-02-17T05:31:19ZengPublic Library of Science (PLoS)PLoS Computational Biology1553-734X1553-73582023-08-01198e101139310.1371/journal.pcbi.1011393Scoring epidemiological forecasts on transformed scales.Nikos I BosseSam AbbottAnne CoriEdwin van LeeuwenJohannes BracherSebastian FunkForecast evaluation is essential for the development of predictive epidemic models and can inform their use for public health decision-making. Common scores to evaluate epidemiological forecasts are the Continuous Ranked Probability Score (CRPS) and the Weighted Interval Score (WIS), which can be seen as measures of the absolute distance between the forecast distribution and the observation. However, applying these scores directly to predicted and observed incidence counts may not be the most appropriate due to the exponential nature of epidemic processes and the varying magnitudes of observed values across space and time. In this paper, we argue that transforming counts before applying scores such as the CRPS or WIS can effectively mitigate these difficulties and yield epidemiologically meaningful and easily interpretable results. Using the CRPS on log-transformed values as an example, we list three attractive properties: Firstly, it can be interpreted as a probabilistic version of a relative error. Secondly, it reflects how well models predicted the time-varying epidemic growth rate. And lastly, using arguments on variance-stabilizing transformations, it can be shown that under the assumption of a quadratic mean-variance relationship, the logarithmic transformation leads to expected CRPS values which are independent of the order of magnitude of the predicted quantity. Applying a transformation of log(x + 1) to data and forecasts from the European COVID-19 Forecast Hub, we find that it changes model rankings regardless of stratification by forecast date, location or target types. Situations in which models missed the beginning of upward swings are more strongly emphasised while failing to predict a downturn following a peak is less severely penalised when scoring transformed forecasts as opposed to untransformed ones. We conclude that appropriate transformations, of which the natural logarithm is only one particularly attractive option, should be considered when assessing the performance of different models in the context of infectious disease incidence.https://doi.org/10.1371/journal.pcbi.1011393 |
| spellingShingle | Nikos I Bosse Sam Abbott Anne Cori Edwin van Leeuwen Johannes Bracher Sebastian Funk Scoring epidemiological forecasts on transformed scales. PLoS Computational Biology |
| title | Scoring epidemiological forecasts on transformed scales. |
| title_full | Scoring epidemiological forecasts on transformed scales. |
| title_fullStr | Scoring epidemiological forecasts on transformed scales. |
| title_full_unstemmed | Scoring epidemiological forecasts on transformed scales. |
| title_short | Scoring epidemiological forecasts on transformed scales. |
| title_sort | scoring epidemiological forecasts on transformed scales |
| url | https://doi.org/10.1371/journal.pcbi.1011393 |
| work_keys_str_mv | AT nikosibosse scoringepidemiologicalforecastsontransformedscales AT samabbott scoringepidemiologicalforecastsontransformedscales AT annecori scoringepidemiologicalforecastsontransformedscales AT edwinvanleeuwen scoringepidemiologicalforecastsontransformedscales AT johannesbracher scoringepidemiologicalforecastsontransformedscales AT sebastianfunk scoringepidemiologicalforecastsontransformedscales |