Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease
Abstract Background Stability of risk estimates from prediction models may be highly dependent on the sample size of the dataset available for model derivation. In this paper, we evaluate the stability of cardiovascular disease risk scores for individual patients when using different sample sizes fo...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2020-09-01
|
| Series: | Diagnostic and Prognostic Research |
| Subjects: | |
| Online Access: | http://link.springer.com/article/10.1186/s41512-020-00082-3 |
| _version_ | 1818542234300055552 |
|---|---|
| author | Alexander Pate Richard Emsley Matthew Sperrin Glen P. Martin Tjeerd van Staa |
| author_facet | Alexander Pate Richard Emsley Matthew Sperrin Glen P. Martin Tjeerd van Staa |
| author_sort | Alexander Pate |
| collection | DOAJ |
| description | Abstract Background Stability of risk estimates from prediction models may be highly dependent on the sample size of the dataset available for model derivation. In this paper, we evaluate the stability of cardiovascular disease risk scores for individual patients when using different sample sizes for model derivation; such sample sizes include those similar to models recommended in the national guidelines, and those based on recently published sample size formula for prediction models. Methods We mimicked the process of sampling N patients from a population to develop a risk prediction model by sampling patients from the Clinical Practice Research Datalink. A cardiovascular disease risk prediction model was developed on this sample and used to generate risk scores for an independent cohort of patients. This process was repeated 1000 times, giving a distribution of risks for each patient. N = 100,000, 50,000, 10,000, N min (derived from sample size formula) and N epv10 (meets 10 events per predictor rule) were considered. The 5–95th percentile range of risks across these models was used to evaluate instability. Patients were grouped by a risk derived from a model developed on the entire population (population-derived risk) to summarise results. Results For a sample size of 100,000, the median 5–95th percentile range of risks for patients across the 1000 models was 0.77%, 1.60%, 2.42% and 3.22% for patients with population-derived risks of 4–5%, 9–10%, 14–15% and 19–20% respectively; for N = 10,000, it was 2.49%, 5.23%, 7.92% and 10.59%, and for N using the formula-derived sample size, it was 6.79%, 14.41%, 21.89% and 29.21%. Restricting this analysis to models with high discrimination, good calibration or small mean absolute prediction error reduced the percentile range, but high levels of instability remained. Conclusions Widely used cardiovascular disease risk prediction models suffer from high levels of instability induced by sampling variation. Many models will also suffer from overfitting (a closely linked concept), but at acceptable levels of overfitting, there may still be high levels of instability in individual risk. Stability of risk estimates should be a criterion when determining the minimum sample size to develop models. |
| first_indexed | 2024-12-11T22:19:25Z |
| format | Article |
| id | doaj.art-f4ee15f68da24b14bc1215813b7f5b57 |
| institution | Directory Open Access Journal |
| issn | 2397-7523 |
| language | English |
| last_indexed | 2024-12-11T22:19:25Z |
| publishDate | 2020-09-01 |
| publisher | BMC |
| record_format | Article |
| series | Diagnostic and Prognostic Research |
| spelling | doaj.art-f4ee15f68da24b14bc1215813b7f5b572022-12-22T00:48:29ZengBMCDiagnostic and Prognostic Research2397-75232020-09-014111210.1186/s41512-020-00082-3Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular diseaseAlexander Pate0Richard Emsley1Matthew Sperrin2Glen P. Martin3Tjeerd van Staa4Centre for Health Informatics, School of Health Sciences, Faculty of Biology, Medicine and Health, The University of ManchesterDepartment of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King’s College LondonCentre for Health Informatics, School of Health Sciences, Faculty of Biology, Medicine and Health, The University of ManchesterCentre for Health Informatics, School of Health Sciences, Faculty of Biology, Medicine and Health, The University of ManchesterCentre for Health Informatics, School of Health Sciences, Faculty of Biology, Medicine and Health, The University of ManchesterAbstract Background Stability of risk estimates from prediction models may be highly dependent on the sample size of the dataset available for model derivation. In this paper, we evaluate the stability of cardiovascular disease risk scores for individual patients when using different sample sizes for model derivation; such sample sizes include those similar to models recommended in the national guidelines, and those based on recently published sample size formula for prediction models. Methods We mimicked the process of sampling N patients from a population to develop a risk prediction model by sampling patients from the Clinical Practice Research Datalink. A cardiovascular disease risk prediction model was developed on this sample and used to generate risk scores for an independent cohort of patients. This process was repeated 1000 times, giving a distribution of risks for each patient. N = 100,000, 50,000, 10,000, N min (derived from sample size formula) and N epv10 (meets 10 events per predictor rule) were considered. The 5–95th percentile range of risks across these models was used to evaluate instability. Patients were grouped by a risk derived from a model developed on the entire population (population-derived risk) to summarise results. Results For a sample size of 100,000, the median 5–95th percentile range of risks for patients across the 1000 models was 0.77%, 1.60%, 2.42% and 3.22% for patients with population-derived risks of 4–5%, 9–10%, 14–15% and 19–20% respectively; for N = 10,000, it was 2.49%, 5.23%, 7.92% and 10.59%, and for N using the formula-derived sample size, it was 6.79%, 14.41%, 21.89% and 29.21%. Restricting this analysis to models with high discrimination, good calibration or small mean absolute prediction error reduced the percentile range, but high levels of instability remained. Conclusions Widely used cardiovascular disease risk prediction models suffer from high levels of instability induced by sampling variation. Many models will also suffer from overfitting (a closely linked concept), but at acceptable levels of overfitting, there may still be high levels of instability in individual risk. Stability of risk estimates should be a criterion when determining the minimum sample size to develop models.http://link.springer.com/article/10.1186/s41512-020-00082-3Risk predictionSample sizeStatistical methodsPrecisionStability |
| spellingShingle | Alexander Pate Richard Emsley Matthew Sperrin Glen P. Martin Tjeerd van Staa Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease Diagnostic and Prognostic Research Risk prediction Sample size Statistical methods Precision Stability |
| title | Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease |
| title_full | Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease |
| title_fullStr | Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease |
| title_full_unstemmed | Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease |
| title_short | Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease |
| title_sort | impact of sample size on the stability of risk scores from clinical prediction models a case study in cardiovascular disease |
| topic | Risk prediction Sample size Statistical methods Precision Stability |
| url | http://link.springer.com/article/10.1186/s41512-020-00082-3 |
| work_keys_str_mv | AT alexanderpate impactofsamplesizeonthestabilityofriskscoresfromclinicalpredictionmodelsacasestudyincardiovasculardisease AT richardemsley impactofsamplesizeonthestabilityofriskscoresfromclinicalpredictionmodelsacasestudyincardiovasculardisease AT matthewsperrin impactofsamplesizeonthestabilityofriskscoresfromclinicalpredictionmodelsacasestudyincardiovasculardisease AT glenpmartin impactofsamplesizeonthestabilityofriskscoresfromclinicalpredictionmodelsacasestudyincardiovasculardisease AT tjeerdvanstaa impactofsamplesizeonthestabilityofriskscoresfromclinicalpredictionmodelsacasestudyincardiovasculardisease |