Clinical prediction models and the multiverse of madness
<p><strong>Background</strong> Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice.</p> <p><...
| Main Authors: | , , , , , |
|---|---|
| Format: | Journal article |
| Language: | English |
| Published: |
BioMed Central
2023
|
| _version_ | 1797112175219703808 |
|---|---|
| author | Riley, RD Pate, A Dhiman, P Archer, L Martin, GP Collins, GS |
| author_facet | Riley, RD Pate, A Dhiman, P Archer, L Martin, GP Collins, GS |
| author_sort | Riley, RD |
| collection | OXFORD |
| description | <p><strong>Background</strong> Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice.</p>
<p><strong>Main body</strong> We discuss how the creation of a prediction model (e.g. using regression or machine learning methods) is dependent on the sample and size of data used to develop it—were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created, there exists a multiverse of other potential models for that sample size and, crucially, an individual’s predicted value (e.g. estimated risk) may vary greatly across this multiverse. The more an individual’s prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice.</p>
<p><strong>Conclusions</strong> Instability is concerning as an individual’s predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.</p> |
| first_indexed | 2024-03-07T08:20:26Z |
| format | Journal article |
| id | oxford-uuid:98dc554d-f5c1-48fc-87ff-ad91fcecf868 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T08:20:26Z |
| publishDate | 2023 |
| publisher | BioMed Central |
| record_format | dspace |
| spelling | oxford-uuid:98dc554d-f5c1-48fc-87ff-ad91fcecf8682024-01-26T13:27:38ZClinical prediction models and the multiverse of madnessJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:98dc554d-f5c1-48fc-87ff-ad91fcecf868EnglishSymplectic ElementsBioMed Central2023Riley, RDPate, ADhiman, PArcher, LMartin, GPCollins, GS<p><strong>Background</strong> Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice.</p> <p><strong>Main body</strong> We discuss how the creation of a prediction model (e.g. using regression or machine learning methods) is dependent on the sample and size of data used to develop it—were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created, there exists a multiverse of other potential models for that sample size and, crucially, an individual’s predicted value (e.g. estimated risk) may vary greatly across this multiverse. The more an individual’s prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice.</p> <p><strong>Conclusions</strong> Instability is concerning as an individual’s predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.</p> |
| spellingShingle | Riley, RD Pate, A Dhiman, P Archer, L Martin, GP Collins, GS Clinical prediction models and the multiverse of madness |
| title | Clinical prediction models and the multiverse of madness |
| title_full | Clinical prediction models and the multiverse of madness |
| title_fullStr | Clinical prediction models and the multiverse of madness |
| title_full_unstemmed | Clinical prediction models and the multiverse of madness |
| title_short | Clinical prediction models and the multiverse of madness |
| title_sort | clinical prediction models and the multiverse of madness |
| work_keys_str_mv | AT rileyrd clinicalpredictionmodelsandthemultiverseofmadness AT patea clinicalpredictionmodelsandthemultiverseofmadness AT dhimanp clinicalpredictionmodelsandthemultiverseofmadness AT archerl clinicalpredictionmodelsandthemultiverseofmadness AT martingp clinicalpredictionmodelsandthemultiverseofmadness AT collinsgs clinicalpredictionmodelsandthemultiverseofmadness |