There is no such thing as a validated prediction model
Abstract Background Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context? Main body We argue to...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-02-01
|
| Series: | BMC Medicine |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12916-023-02779-w |
| _version_ | 1827983894257860608 |
|---|---|
| author | Ben Van Calster Ewout W. Steyerberg Laure Wynants Maarten van Smeden |
| author_facet | Ben Van Calster Ewout W. Steyerberg Laure Wynants Maarten van Smeden |
| author_sort | Ben Van Calster |
| collection | DOAJ |
| description | Abstract Background Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context? Main body We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models. Conclusion Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making. |
| first_indexed | 2024-04-09T22:53:26Z |
| format | Article |
| id | doaj.art-f9a6641b7f234b999b5884e3d6f677f4 |
| institution | Directory Open Access Journal |
| issn | 1741-7015 |
| language | English |
| last_indexed | 2024-04-09T22:53:26Z |
| publishDate | 2023-02-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medicine |
| spelling | doaj.art-f9a6641b7f234b999b5884e3d6f677f42023-03-22T11:32:43ZengBMCBMC Medicine1741-70152023-02-012111810.1186/s12916-023-02779-wThere is no such thing as a validated prediction modelBen Van Calster0Ewout W. Steyerberg1Laure Wynants2Maarten van Smeden3Department of Development and Regeneration, KU LeuvenDepartment of Development and Regeneration, KU LeuvenDepartment of Development and Regeneration, KU LeuvenJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht UniversityAbstract Background Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context? Main body We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models. Conclusion Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making.https://doi.org/10.1186/s12916-023-02779-wRisk prediction modelsPredictive analyticsInternal validationExternal validationHeterogeneityModel performance |
| spellingShingle | Ben Van Calster Ewout W. Steyerberg Laure Wynants Maarten van Smeden There is no such thing as a validated prediction model BMC Medicine Risk prediction models Predictive analytics Internal validation External validation Heterogeneity Model performance |
| title | There is no such thing as a validated prediction model |
| title_full | There is no such thing as a validated prediction model |
| title_fullStr | There is no such thing as a validated prediction model |
| title_full_unstemmed | There is no such thing as a validated prediction model |
| title_short | There is no such thing as a validated prediction model |
| title_sort | there is no such thing as a validated prediction model |
| topic | Risk prediction models Predictive analytics Internal validation External validation Heterogeneity Model performance |
| url | https://doi.org/10.1186/s12916-023-02779-w |
| work_keys_str_mv | AT benvancalster thereisnosuchthingasavalidatedpredictionmodel AT ewoutwsteyerberg thereisnosuchthingasavalidatedpredictionmodel AT laurewynants thereisnosuchthingasavalidatedpredictionmodel AT maartenvansmeden thereisnosuchthingasavalidatedpredictionmodel |