Validation of prediction models: examining temporal and geographic stability of baseline risk and estimated covariate effects
Abstract Background Stability in baseline risk and estimated predictor effects both geographically and temporally is a desirable property of clinical prediction models. However, this issue has received little attention in the methodological literature. Our objective was to examine methods for assess...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2017-04-01
|
| Series: | Diagnostic and Prognostic Research |
| Subjects: | |
| Online Access: | http://link.springer.com/article/10.1186/s41512-017-0012-3 |
| _version_ | 1818963404814024704 |
|---|---|
| author | Peter C. Austin David van Klaveren Yvonne Vergouwe Daan Nieboer Douglas S. Lee Ewout W. Steyerberg |
| author_facet | Peter C. Austin David van Klaveren Yvonne Vergouwe Daan Nieboer Douglas S. Lee Ewout W. Steyerberg |
| author_sort | Peter C. Austin |
| collection | DOAJ |
| description | Abstract Background Stability in baseline risk and estimated predictor effects both geographically and temporally is a desirable property of clinical prediction models. However, this issue has received little attention in the methodological literature. Our objective was to examine methods for assessing temporal and geographic heterogeneity in baseline risk and predictor effects in prediction models. Methods We studied 14,857 patients hospitalized with heart failure at 90 hospitals in Ontario, Canada, in two time periods. We focussed on geographic and temporal variation in baseline risk (intercept) and predictor effects (regression coefficients) of the EFFECT-HF mortality model for predicting 1-year mortality in patients hospitalized for heart failure. We used random effects logistic regression models for the 14,857 patients. Results The baseline risk of mortality displayed moderate geographic variation, with the hospital-specific probability of 1-year mortality for a reference patient lying between 0.168 and 0.290 for 95% of hospitals. Furthermore, the odds of death were 11% lower in the second period than in the first period. However, we found minimal geographic or temporal variation in predictor effects. Among 11 tests of differences in time for predictor variables, only one had a modestly significant P value (0.03). Conclusions This study illustrates how temporal and geographic heterogeneity of prediction models can be assessed in settings with a large sample of patients from a large number of centers at different time periods. |
| first_indexed | 2024-12-20T12:44:41Z |
| format | Article |
| id | doaj.art-895668a5c3be4eeb86e7cd1a3f635f6c |
| institution | Directory Open Access Journal |
| issn | 2397-7523 |
| language | English |
| last_indexed | 2024-12-20T12:44:41Z |
| publishDate | 2017-04-01 |
| publisher | BMC |
| record_format | Article |
| series | Diagnostic and Prognostic Research |
| spelling | doaj.art-895668a5c3be4eeb86e7cd1a3f635f6c2022-12-21T19:40:20ZengBMCDiagnostic and Prognostic Research2397-75232017-04-01111810.1186/s41512-017-0012-3Validation of prediction models: examining temporal and geographic stability of baseline risk and estimated covariate effectsPeter C. Austin0David van Klaveren1Yvonne Vergouwe2Daan Nieboer3Douglas S. Lee4Ewout W. Steyerberg5Institute for Clinical Evaluative SciencesDepartment of Public Health, Erasmus MC-University Medical Center RotterdamDepartment of Public Health, Erasmus MC-University Medical Center RotterdamDepartment of Public Health, Erasmus MC-University Medical Center RotterdamInstitute for Clinical Evaluative SciencesDepartment of Public Health, Erasmus MC-University Medical Center RotterdamAbstract Background Stability in baseline risk and estimated predictor effects both geographically and temporally is a desirable property of clinical prediction models. However, this issue has received little attention in the methodological literature. Our objective was to examine methods for assessing temporal and geographic heterogeneity in baseline risk and predictor effects in prediction models. Methods We studied 14,857 patients hospitalized with heart failure at 90 hospitals in Ontario, Canada, in two time periods. We focussed on geographic and temporal variation in baseline risk (intercept) and predictor effects (regression coefficients) of the EFFECT-HF mortality model for predicting 1-year mortality in patients hospitalized for heart failure. We used random effects logistic regression models for the 14,857 patients. Results The baseline risk of mortality displayed moderate geographic variation, with the hospital-specific probability of 1-year mortality for a reference patient lying between 0.168 and 0.290 for 95% of hospitals. Furthermore, the odds of death were 11% lower in the second period than in the first period. However, we found minimal geographic or temporal variation in predictor effects. Among 11 tests of differences in time for predictor variables, only one had a modestly significant P value (0.03). Conclusions This study illustrates how temporal and geographic heterogeneity of prediction models can be assessed in settings with a large sample of patients from a large number of centers at different time periods.http://link.springer.com/article/10.1186/s41512-017-0012-3Clinical prediction modelValidationRisk predictionHierarchical regression modelGeographic variationTemporal variation |
| spellingShingle | Peter C. Austin David van Klaveren Yvonne Vergouwe Daan Nieboer Douglas S. Lee Ewout W. Steyerberg Validation of prediction models: examining temporal and geographic stability of baseline risk and estimated covariate effects Diagnostic and Prognostic Research Clinical prediction model Validation Risk prediction Hierarchical regression model Geographic variation Temporal variation |
| title | Validation of prediction models: examining temporal and geographic stability of baseline risk and estimated covariate effects |
| title_full | Validation of prediction models: examining temporal and geographic stability of baseline risk and estimated covariate effects |
| title_fullStr | Validation of prediction models: examining temporal and geographic stability of baseline risk and estimated covariate effects |
| title_full_unstemmed | Validation of prediction models: examining temporal and geographic stability of baseline risk and estimated covariate effects |
| title_short | Validation of prediction models: examining temporal and geographic stability of baseline risk and estimated covariate effects |
| title_sort | validation of prediction models examining temporal and geographic stability of baseline risk and estimated covariate effects |
| topic | Clinical prediction model Validation Risk prediction Hierarchical regression model Geographic variation Temporal variation |
| url | http://link.springer.com/article/10.1186/s41512-017-0012-3 |
| work_keys_str_mv | AT petercaustin validationofpredictionmodelsexaminingtemporalandgeographicstabilityofbaselineriskandestimatedcovariateeffects AT davidvanklaveren validationofpredictionmodelsexaminingtemporalandgeographicstabilityofbaselineriskandestimatedcovariateeffects AT yvonnevergouwe validationofpredictionmodelsexaminingtemporalandgeographicstabilityofbaselineriskandestimatedcovariateeffects AT daannieboer validationofpredictionmodelsexaminingtemporalandgeographicstabilityofbaselineriskandestimatedcovariateeffects AT douglasslee validationofpredictionmodelsexaminingtemporalandgeographicstabilityofbaselineriskandestimatedcovariateeffects AT ewoutwsteyerberg validationofpredictionmodelsexaminingtemporalandgeographicstabilityofbaselineriskandestimatedcovariateeffects |