Dynamic updating of clinical survival prediction models in a changing environment
<p><strong>Background:</strong> Over time, the performance of clinical prediction models may deteriorate due to changes in clinical management, data quality, disease risk and/or patient mix. Such prediction models must be updated in order to remain useful. In this study, we investi...
| Main Authors: | , , , , |
|---|---|
| Format: | Journal article |
| Language: | English |
| Published: |
BioMed Central
2023
|
| _version_ | 1824458779231518720 |
|---|---|
| author | Tanner, KT Keogh, RH Coupland, CAC Hippisley-Cox, J Diaz-Ordaz, K |
| author_facet | Tanner, KT Keogh, RH Coupland, CAC Hippisley-Cox, J Diaz-Ordaz, K |
| author_sort | Tanner, KT |
| collection | OXFORD |
| description | <p><strong>Background:</strong> Over time, the performance of clinical prediction models may deteriorate due to changes in clinical management, data quality, disease risk and/or patient mix. Such prediction models must be updated in order to remain useful. In this study, we investigate dynamic model updating of clinical survival prediction models. In contrast to discrete or one-time updating, dynamic updating refers to a repeated process for updating a prediction model with new data. We aim to extend previous research which focused largely on binary outcome prediction models by concentrating on time-to-event outcomes. We were motivated by the rapidly changing environment seen during the COVID-19 pandemic where mortality rates changed over time and new treatments and vaccines were introduced.</p>
<br>
<p><strong>Methods:</strong> We illustrate three methods for dynamic model updating: Bayesian dynamic updating, recalibration, and full refitting. We use a simulation study to compare performance in a range of scenarios including changing mortality rates, predictors with low prevalence and the introduction of a new treatment. Next, the updating strategies were applied to a model for predicting 70-day COVID-19-related mortality using patient data from QResearch, an electronic health records database from general practices in the UK.</p>
<br>
<p><strong>Results:</strong> In simulated scenarios with mortality rates changing over time, all updating methods resulted in better calibration than not updating. Moreover, dynamic updating outperformed ad hoc updating. In the simulation scenario with a new predictor and a small updating dataset, Bayesian updating improved the C-index over not updating and refitting. In the motivating example with a rare outcome, no single updating method offered the best performance.</p>
<br>
<p><strong>Conclusions:</strong> We found that a dynamic updating process outperformed one-time discrete updating in the simulations. Bayesian updating offered good performance overall, even in scenarios with new predictors and few events. Intercept recalibration was effective in scenarios with smaller sample size and changing baseline hazard. Refitting performance depended on sample size and produced abrupt changes in hazard ratio estimates between periods.</p> |
| first_indexed | 2025-02-19T04:31:19Z |
| format | Journal article |
| id | oxford-uuid:386ceec8-3aed-4042-987c-acdadb5198aa |
| institution | University of Oxford |
| language | English |
| last_indexed | 2025-02-19T04:31:19Z |
| publishDate | 2023 |
| publisher | BioMed Central |
| record_format | dspace |
| spelling | oxford-uuid:386ceec8-3aed-4042-987c-acdadb5198aa2025-01-10T15:08:49ZDynamic updating of clinical survival prediction models in a changing environmentJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:386ceec8-3aed-4042-987c-acdadb5198aaEnglishSymplectic ElementsBioMed Central2023Tanner, KTKeogh, RHCoupland, CACHippisley-Cox, JDiaz-Ordaz, K<p><strong>Background:</strong> Over time, the performance of clinical prediction models may deteriorate due to changes in clinical management, data quality, disease risk and/or patient mix. Such prediction models must be updated in order to remain useful. In this study, we investigate dynamic model updating of clinical survival prediction models. In contrast to discrete or one-time updating, dynamic updating refers to a repeated process for updating a prediction model with new data. We aim to extend previous research which focused largely on binary outcome prediction models by concentrating on time-to-event outcomes. We were motivated by the rapidly changing environment seen during the COVID-19 pandemic where mortality rates changed over time and new treatments and vaccines were introduced.</p> <br> <p><strong>Methods:</strong> We illustrate three methods for dynamic model updating: Bayesian dynamic updating, recalibration, and full refitting. We use a simulation study to compare performance in a range of scenarios including changing mortality rates, predictors with low prevalence and the introduction of a new treatment. Next, the updating strategies were applied to a model for predicting 70-day COVID-19-related mortality using patient data from QResearch, an electronic health records database from general practices in the UK.</p> <br> <p><strong>Results:</strong> In simulated scenarios with mortality rates changing over time, all updating methods resulted in better calibration than not updating. Moreover, dynamic updating outperformed ad hoc updating. In the simulation scenario with a new predictor and a small updating dataset, Bayesian updating improved the C-index over not updating and refitting. In the motivating example with a rare outcome, no single updating method offered the best performance.</p> <br> <p><strong>Conclusions:</strong> We found that a dynamic updating process outperformed one-time discrete updating in the simulations. Bayesian updating offered good performance overall, even in scenarios with new predictors and few events. Intercept recalibration was effective in scenarios with smaller sample size and changing baseline hazard. Refitting performance depended on sample size and produced abrupt changes in hazard ratio estimates between periods.</p> |
| spellingShingle | Tanner, KT Keogh, RH Coupland, CAC Hippisley-Cox, J Diaz-Ordaz, K Dynamic updating of clinical survival prediction models in a changing environment |
| title | Dynamic updating of clinical survival prediction models in a changing environment |
| title_full | Dynamic updating of clinical survival prediction models in a changing environment |
| title_fullStr | Dynamic updating of clinical survival prediction models in a changing environment |
| title_full_unstemmed | Dynamic updating of clinical survival prediction models in a changing environment |
| title_short | Dynamic updating of clinical survival prediction models in a changing environment |
| title_sort | dynamic updating of clinical survival prediction models in a changing environment |
| work_keys_str_mv | AT tannerkt dynamicupdatingofclinicalsurvivalpredictionmodelsinachangingenvironment AT keoghrh dynamicupdatingofclinicalsurvivalpredictionmodelsinachangingenvironment AT couplandcac dynamicupdatingofclinicalsurvivalpredictionmodelsinachangingenvironment AT hippisleycoxj dynamicupdatingofclinicalsurvivalpredictionmodelsinachangingenvironment AT diazordazk dynamicupdatingofclinicalsurvivalpredictionmodelsinachangingenvironment |