Risk of bias of prognostic models developed using machine learning: a systematic review in oncology
<strong>Background<br></strong> Prognostic models are used widely in the oncology domain to guide medical decision-making. Little is known about the risk of bias of prognostic models developed using machine learning and the barriers to their clinical uptake in the oncology domain....
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Journal article |
| Language: | English |
| Published: |
BioMed Central
2022
|
| _version_ | 1826308366698283008 |
|---|---|
| author | Dhiman, P Ma, J Andaur Navarro, CL Speich, B Bullock, G Damen, JAA Hooft, L Kirtley, S Riley, RD Van Calster, B Moons, KGM Collins, GS |
| author_facet | Dhiman, P Ma, J Andaur Navarro, CL Speich, B Bullock, G Damen, JAA Hooft, L Kirtley, S Riley, RD Van Calster, B Moons, KGM Collins, GS |
| author_sort | Dhiman, P |
| collection | OXFORD |
| description | <strong>Background<br></strong>
Prognostic models are used widely in the oncology domain to guide medical decision-making. Little is known about the risk of bias of prognostic models developed using machine learning and the barriers to their clinical uptake in the oncology domain.
<br><strong>
Methods<br></strong>
We conducted a systematic review and searched MEDLINE and EMBASE databases for oncology-related studies developing a prognostic model using machine learning methods published between 01/01/2019 and 05/09/2019. The primary outcome was risk of bias, judged using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). We described risk of bias overall and for each domain, by development and validation analyses separately.
<br><strong>
Results<br></strong>
We included 62 publications (48 development-only; 14 development with validation). 152 models were developed across all publications and 37 models were validated. 84% (95% CI: 77 to 89) of developed models and 51% (95% CI: 35 to 67) of validated models were at overall high risk of bias. Bias introduced in the analysis was the largest contributor to the overall risk of bias judgement for model development and validation. 123 (81%, 95% CI: 73.8 to 86.4) developed models and 19 (51%, 95% CI: 35.1 to 67.3) validated models were at high risk of bias due to their analysis, mostly due to shortcomings in the analysis including insufficient sample size and split-sample internal validation.
<br><strong>
Conclusions<br></strong>
The quality of machine learning based prognostic models in the oncology domain is poor and most models have a high risk of bias, contraindicating their use in clinical practice. Adherence to better standards is urgently needed, with a focus on sample size estimation and analysis methods, to improve the quality of these models. |
| first_indexed | 2024-03-07T07:18:27Z |
| format | Journal article |
| id | oxford-uuid:7c1e6530-a9c8-49e9-a49d-87969441ea85 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T07:18:27Z |
| publishDate | 2022 |
| publisher | BioMed Central |
| record_format | dspace |
| spelling | oxford-uuid:7c1e6530-a9c8-49e9-a49d-87969441ea852022-09-06T15:55:17ZRisk of bias of prognostic models developed using machine learning: a systematic review in oncologyJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:7c1e6530-a9c8-49e9-a49d-87969441ea85EnglishSymplectic ElementsBioMed Central2022Dhiman, PMa, JAndaur Navarro, CLSpeich, BBullock, GDamen, JAAHooft, LKirtley, SRiley, RDVan Calster, BMoons, KGMCollins, GS<strong>Background<br></strong> Prognostic models are used widely in the oncology domain to guide medical decision-making. Little is known about the risk of bias of prognostic models developed using machine learning and the barriers to their clinical uptake in the oncology domain. <br><strong> Methods<br></strong> We conducted a systematic review and searched MEDLINE and EMBASE databases for oncology-related studies developing a prognostic model using machine learning methods published between 01/01/2019 and 05/09/2019. The primary outcome was risk of bias, judged using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). We described risk of bias overall and for each domain, by development and validation analyses separately. <br><strong> Results<br></strong> We included 62 publications (48 development-only; 14 development with validation). 152 models were developed across all publications and 37 models were validated. 84% (95% CI: 77 to 89) of developed models and 51% (95% CI: 35 to 67) of validated models were at overall high risk of bias. Bias introduced in the analysis was the largest contributor to the overall risk of bias judgement for model development and validation. 123 (81%, 95% CI: 73.8 to 86.4) developed models and 19 (51%, 95% CI: 35.1 to 67.3) validated models were at high risk of bias due to their analysis, mostly due to shortcomings in the analysis including insufficient sample size and split-sample internal validation. <br><strong> Conclusions<br></strong> The quality of machine learning based prognostic models in the oncology domain is poor and most models have a high risk of bias, contraindicating their use in clinical practice. Adherence to better standards is urgently needed, with a focus on sample size estimation and analysis methods, to improve the quality of these models. |
| spellingShingle | Dhiman, P Ma, J Andaur Navarro, CL Speich, B Bullock, G Damen, JAA Hooft, L Kirtley, S Riley, RD Van Calster, B Moons, KGM Collins, GS Risk of bias of prognostic models developed using machine learning: a systematic review in oncology |
| title | Risk of bias of prognostic models developed using machine learning: a systematic review in oncology |
| title_full | Risk of bias of prognostic models developed using machine learning: a systematic review in oncology |
| title_fullStr | Risk of bias of prognostic models developed using machine learning: a systematic review in oncology |
| title_full_unstemmed | Risk of bias of prognostic models developed using machine learning: a systematic review in oncology |
| title_short | Risk of bias of prognostic models developed using machine learning: a systematic review in oncology |
| title_sort | risk of bias of prognostic models developed using machine learning a systematic review in oncology |
| work_keys_str_mv | AT dhimanp riskofbiasofprognosticmodelsdevelopedusingmachinelearningasystematicreviewinoncology AT maj riskofbiasofprognosticmodelsdevelopedusingmachinelearningasystematicreviewinoncology AT andaurnavarrocl riskofbiasofprognosticmodelsdevelopedusingmachinelearningasystematicreviewinoncology AT speichb riskofbiasofprognosticmodelsdevelopedusingmachinelearningasystematicreviewinoncology AT bullockg riskofbiasofprognosticmodelsdevelopedusingmachinelearningasystematicreviewinoncology AT damenjaa riskofbiasofprognosticmodelsdevelopedusingmachinelearningasystematicreviewinoncology AT hooftl riskofbiasofprognosticmodelsdevelopedusingmachinelearningasystematicreviewinoncology AT kirtleys riskofbiasofprognosticmodelsdevelopedusingmachinelearningasystematicreviewinoncology AT rileyrd riskofbiasofprognosticmodelsdevelopedusingmachinelearningasystematicreviewinoncology AT vancalsterb riskofbiasofprognosticmodelsdevelopedusingmachinelearningasystematicreviewinoncology AT moonskgm riskofbiasofprognosticmodelsdevelopedusingmachinelearningasystematicreviewinoncology AT collinsgs riskofbiasofprognosticmodelsdevelopedusingmachinelearningasystematicreviewinoncology |