Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review
Abstract Background Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of m...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2022-12-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-022-01801-8 |
| _version_ | 1798005178651115520 |
|---|---|
| author | M. A. E. Binuya E. G. Engelhardt W. Schats M. K. Schmidt E. W. Steyerberg |
| author_facet | M. A. E. Binuya E. G. Engelhardt W. Schats M. K. Schmidt E. W. Steyerberg |
| author_sort | M. A. E. Binuya |
| collection | DOAJ |
| description | Abstract Background Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of methodological guidance for the evaluation (i.e., validation and impact assessment) and updating of clinical prediction models. Methods We systematically searched nine databases from January 2000 to January 2022 for articles in English with methodological recommendations for the post-derivation stages of interest. Qualitative analysis was used to summarize the 70 selected guidance papers. Results Key aspects for validation are the assessment of statistical performance using measures for discrimination (e.g., C-statistic) and calibration (e.g., calibration-in-the-large and calibration slope). For assessing impact or usefulness in clinical decision-making, recent papers advise using decision-analytic measures (e.g., the Net Benefit) over simplistic classification measures that ignore clinical consequences (e.g., accuracy, overall Net Reclassification Index). Commonly recommended methods for model updating are recalibration (i.e., adjustment of intercept or baseline hazard and/or slope), revision (i.e., re-estimation of individual predictor effects), and extension (i.e., addition of new markers). Additional methodological guidance is needed for newer types of updating (e.g., meta-model and dynamic updating) and machine learning-based models. Conclusion Substantial guidance was found for model evaluation and more conventional updating of regression-based models. An important development in model evaluation is the introduction of a decision-analytic framework for assessing clinical usefulness. Consensus is emerging on methods for model updating. |
| first_indexed | 2024-04-11T12:34:59Z |
| format | Article |
| id | doaj.art-e2d1d4e33e7a4a12b923e6b1df3ba73c |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-04-11T12:34:59Z |
| publishDate | 2022-12-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-e2d1d4e33e7a4a12b923e6b1df3ba73c2022-12-22T04:23:39ZengBMCBMC Medical Research Methodology1471-22882022-12-0122111410.1186/s12874-022-01801-8Methodological guidance for the evaluation and updating of clinical prediction models: a systematic reviewM. A. E. Binuya0E. G. Engelhardt1W. Schats2M. K. Schmidt3E. W. Steyerberg4Division of Molecular Pathology, the Netherlands Cancer Institute – Antoni van Leeuwenhoek HospitalDivision of Molecular Pathology, the Netherlands Cancer Institute – Antoni van Leeuwenhoek HospitalScientific Information Service, The Netherlands Cancer Institute – Antoni van Leeuwenhoek HospitalDivision of Molecular Pathology, the Netherlands Cancer Institute – Antoni van Leeuwenhoek HospitalDepartment of Biomedical Data Sciences, Leiden University Medical CenterAbstract Background Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of methodological guidance for the evaluation (i.e., validation and impact assessment) and updating of clinical prediction models. Methods We systematically searched nine databases from January 2000 to January 2022 for articles in English with methodological recommendations for the post-derivation stages of interest. Qualitative analysis was used to summarize the 70 selected guidance papers. Results Key aspects for validation are the assessment of statistical performance using measures for discrimination (e.g., C-statistic) and calibration (e.g., calibration-in-the-large and calibration slope). For assessing impact or usefulness in clinical decision-making, recent papers advise using decision-analytic measures (e.g., the Net Benefit) over simplistic classification measures that ignore clinical consequences (e.g., accuracy, overall Net Reclassification Index). Commonly recommended methods for model updating are recalibration (i.e., adjustment of intercept or baseline hazard and/or slope), revision (i.e., re-estimation of individual predictor effects), and extension (i.e., addition of new markers). Additional methodological guidance is needed for newer types of updating (e.g., meta-model and dynamic updating) and machine learning-based models. Conclusion Substantial guidance was found for model evaluation and more conventional updating of regression-based models. An important development in model evaluation is the introduction of a decision-analytic framework for assessing clinical usefulness. Consensus is emerging on methods for model updating.https://doi.org/10.1186/s12874-022-01801-8Prediction modelModel evaluationValidationImpact assessmentDiscriminationCalibration |
| spellingShingle | M. A. E. Binuya E. G. Engelhardt W. Schats M. K. Schmidt E. W. Steyerberg Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review BMC Medical Research Methodology Prediction model Model evaluation Validation Impact assessment Discrimination Calibration |
| title | Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review |
| title_full | Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review |
| title_fullStr | Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review |
| title_full_unstemmed | Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review |
| title_short | Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review |
| title_sort | methodological guidance for the evaluation and updating of clinical prediction models a systematic review |
| topic | Prediction model Model evaluation Validation Impact assessment Discrimination Calibration |
| url | https://doi.org/10.1186/s12874-022-01801-8 |
| work_keys_str_mv | AT maebinuya methodologicalguidancefortheevaluationandupdatingofclinicalpredictionmodelsasystematicreview AT egengelhardt methodologicalguidancefortheevaluationandupdatingofclinicalpredictionmodelsasystematicreview AT wschats methodologicalguidancefortheevaluationandupdatingofclinicalpredictionmodelsasystematicreview AT mkschmidt methodologicalguidancefortheevaluationandupdatingofclinicalpredictionmodelsasystematicreview AT ewsteyerberg methodologicalguidancefortheevaluationandupdatingofclinicalpredictionmodelsasystematicreview |