Dynamic updating of clinical survival prediction models in a changing environment
Abstract Background Over time, the performance of clinical prediction models may deteriorate due to changes in clinical management, data quality, disease risk and/or patient mix. Such prediction models must be updated in order to remain useful. In this study, we investigate dynamic model updating of...
| Asıl Yazarlar: | , , , , |
|---|---|
| Materyal Türü: | Makale |
| Dil: | English |
| Baskı/Yayın Bilgisi: |
BMC
2023-12-01
|
| Seri Bilgileri: | Diagnostic and Prognostic Research |
| Konular: | |
| Online Erişim: | https://doi.org/10.1186/s41512-023-00163-z |
| _version_ | 1827581287541506048 |
|---|---|
| author | Kamaryn T. Tanner Ruth H. Keogh Carol A. C. Coupland Julia Hippisley-Cox Karla Diaz-Ordaz |
| author_facet | Kamaryn T. Tanner Ruth H. Keogh Carol A. C. Coupland Julia Hippisley-Cox Karla Diaz-Ordaz |
| author_sort | Kamaryn T. Tanner |
| collection | DOAJ |
| description | Abstract Background Over time, the performance of clinical prediction models may deteriorate due to changes in clinical management, data quality, disease risk and/or patient mix. Such prediction models must be updated in order to remain useful. In this study, we investigate dynamic model updating of clinical survival prediction models. In contrast to discrete or one-time updating, dynamic updating refers to a repeated process for updating a prediction model with new data. We aim to extend previous research which focused largely on binary outcome prediction models by concentrating on time-to-event outcomes. We were motivated by the rapidly changing environment seen during the COVID-19 pandemic where mortality rates changed over time and new treatments and vaccines were introduced. Methods We illustrate three methods for dynamic model updating: Bayesian dynamic updating, recalibration, and full refitting. We use a simulation study to compare performance in a range of scenarios including changing mortality rates, predictors with low prevalence and the introduction of a new treatment. Next, the updating strategies were applied to a model for predicting 70-day COVID-19-related mortality using patient data from QResearch, an electronic health records database from general practices in the UK. Results In simulated scenarios with mortality rates changing over time, all updating methods resulted in better calibration than not updating. Moreover, dynamic updating outperformed ad hoc updating. In the simulation scenario with a new predictor and a small updating dataset, Bayesian updating improved the C-index over not updating and refitting. In the motivating example with a rare outcome, no single updating method offered the best performance. Conclusions We found that a dynamic updating process outperformed one-time discrete updating in the simulations. Bayesian updating offered good performance overall, even in scenarios with new predictors and few events. Intercept recalibration was effective in scenarios with smaller sample size and changing baseline hazard. Refitting performance depended on sample size and produced abrupt changes in hazard ratio estimates between periods. |
| first_indexed | 2024-03-08T22:34:14Z |
| format | Article |
| id | doaj.art-50c5e01888bc4ade89fcc7b33e7598a9 |
| institution | Directory Open Access Journal |
| issn | 2397-7523 |
| language | English |
| last_indexed | 2024-03-08T22:34:14Z |
| publishDate | 2023-12-01 |
| publisher | BMC |
| record_format | Article |
| series | Diagnostic and Prognostic Research |
| spelling | doaj.art-50c5e01888bc4ade89fcc7b33e7598a92023-12-17T12:33:22ZengBMCDiagnostic and Prognostic Research2397-75232023-12-017111410.1186/s41512-023-00163-zDynamic updating of clinical survival prediction models in a changing environmentKamaryn T. Tanner0Ruth H. Keogh1Carol A. C. Coupland2Julia Hippisley-Cox3Karla Diaz-Ordaz4Department of Medical Statistics, London School of Hygiene and Tropical MedicineDepartment of Medical Statistics, London School of Hygiene and Tropical MedicineNuffield Department of Primary Health Care Sciences, University of OxfordNuffield Department of Primary Health Care Sciences, University of OxfordDepartment of Statistical Science, University College LondonAbstract Background Over time, the performance of clinical prediction models may deteriorate due to changes in clinical management, data quality, disease risk and/or patient mix. Such prediction models must be updated in order to remain useful. In this study, we investigate dynamic model updating of clinical survival prediction models. In contrast to discrete or one-time updating, dynamic updating refers to a repeated process for updating a prediction model with new data. We aim to extend previous research which focused largely on binary outcome prediction models by concentrating on time-to-event outcomes. We were motivated by the rapidly changing environment seen during the COVID-19 pandemic where mortality rates changed over time and new treatments and vaccines were introduced. Methods We illustrate three methods for dynamic model updating: Bayesian dynamic updating, recalibration, and full refitting. We use a simulation study to compare performance in a range of scenarios including changing mortality rates, predictors with low prevalence and the introduction of a new treatment. Next, the updating strategies were applied to a model for predicting 70-day COVID-19-related mortality using patient data from QResearch, an electronic health records database from general practices in the UK. Results In simulated scenarios with mortality rates changing over time, all updating methods resulted in better calibration than not updating. Moreover, dynamic updating outperformed ad hoc updating. In the simulation scenario with a new predictor and a small updating dataset, Bayesian updating improved the C-index over not updating and refitting. In the motivating example with a rare outcome, no single updating method offered the best performance. Conclusions We found that a dynamic updating process outperformed one-time discrete updating in the simulations. Bayesian updating offered good performance overall, even in scenarios with new predictors and few events. Intercept recalibration was effective in scenarios with smaller sample size and changing baseline hazard. Refitting performance depended on sample size and produced abrupt changes in hazard ratio estimates between periods.https://doi.org/10.1186/s41512-023-00163-zClinical prediction modelsDynamic modelModel updatingSurvival analysis |
| spellingShingle | Kamaryn T. Tanner Ruth H. Keogh Carol A. C. Coupland Julia Hippisley-Cox Karla Diaz-Ordaz Dynamic updating of clinical survival prediction models in a changing environment Diagnostic and Prognostic Research Clinical prediction models Dynamic model Model updating Survival analysis |
| title | Dynamic updating of clinical survival prediction models in a changing environment |
| title_full | Dynamic updating of clinical survival prediction models in a changing environment |
| title_fullStr | Dynamic updating of clinical survival prediction models in a changing environment |
| title_full_unstemmed | Dynamic updating of clinical survival prediction models in a changing environment |
| title_short | Dynamic updating of clinical survival prediction models in a changing environment |
| title_sort | dynamic updating of clinical survival prediction models in a changing environment |
| topic | Clinical prediction models Dynamic model Model updating Survival analysis |
| url | https://doi.org/10.1186/s41512-023-00163-z |
| work_keys_str_mv | AT kamarynttanner dynamicupdatingofclinicalsurvivalpredictionmodelsinachangingenvironment AT ruthhkeogh dynamicupdatingofclinicalsurvivalpredictionmodelsinachangingenvironment AT carolaccoupland dynamicupdatingofclinicalsurvivalpredictionmodelsinachangingenvironment AT juliahippisleycox dynamicupdatingofclinicalsurvivalpredictionmodelsinachangingenvironment AT karladiazordaz dynamicupdatingofclinicalsurvivalpredictionmodelsinachangingenvironment |