Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review
<strong>Background<br></strong> While many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. We aim to systematically review the adherence of Machine Learning (...
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Journal article |
| Language: | English |
| Published: |
BioMed Central
2022
|
| _version_ | 1797107644741189632 |
|---|---|
| author | Andaur Navarro, CL Damen, JAA Takada, T Nijman, SWJ Dhiman, P Ma, J Collins, GS Bajpai, R Riley, RD Moons, KGM Hooft, L |
| author_facet | Andaur Navarro, CL Damen, JAA Takada, T Nijman, SWJ Dhiman, P Ma, J Collins, GS Bajpai, R Riley, RD Moons, KGM Hooft, L |
| author_sort | Andaur Navarro, CL |
| collection | OXFORD |
| description | <strong>Background<br></strong>
While many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. We aim to systematically review the adherence of Machine Learning (ML)-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement.
<br><strong>
Methods<br></strong>
We included articles reporting on development or external validation of a multivariable prediction model (either diagnostic or prognostic) developed using supervised ML for individualized predictions across all medical fields. We searched PubMed from 1 January 2018 to 31 December 2019. Data extraction was performed using the 22-item checklist for reporting of prediction model studies (www.TRIPOD-statement.org). We measured the overall adherence per article and per TRIPOD item.
<br><strong>
Results<br></strong>
Our search identified 24,814 articles, of which 152 articles were included: 94 (61.8%) prognostic and 58 (38.2%) diagnostic prediction model studies. Overall, articles adhered to a median of 38.7% (IQR 31.0–46.4%) of TRIPOD items. No article fully adhered to complete reporting of the abstract and very few reported the flow of participants (3.9%, 95% CI 1.8 to 8.3), appropriate title (4.6%, 95% CI 2.2 to 9.2), blinding of predictors (4.6%, 95% CI 2.2 to 9.2), model specification (5.2%, 95% CI 2.4 to 10.8), and model’s predictive performance (5.9%, 95% CI 3.1 to 10.9). There was often complete reporting of source of data (98.0%, 95% CI 94.4 to 99.3) and interpretation of the results (94.7%, 95% CI 90.0 to 97.3).
<br><strong>
Conclusion<br></strong>
Similar to prediction model studies developed using conventional regression-based techniques, the completeness of reporting is poor. Essential information to decide to use the model (i.e. model specification and its performance) is rarely reported. However, some items and sub-items of TRIPOD might be less suitable for ML-based prediction model studies and thus, TRIPOD requires extensions. Overall, there is an urgent need to improve the reporting quality and usability of research to avoid research waste.
<br><strong>
Systematic review registration<br></strong>
PROSPERO, CRD42019161764. |
| first_indexed | 2024-03-07T07:18:59Z |
| format | Journal article |
| id | oxford-uuid:da07eeb5-e49f-4b69-9831-367790f140d8 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T07:18:59Z |
| publishDate | 2022 |
| publisher | BioMed Central |
| record_format | dspace |
| spelling | oxford-uuid:da07eeb5-e49f-4b69-9831-367790f140d82022-09-07T23:21:07ZCompleteness of reporting of clinical prediction models developed using supervised machine learning: a systematic reviewJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:da07eeb5-e49f-4b69-9831-367790f140d8EnglishSymplectic ElementsBioMed Central2022Andaur Navarro, CLDamen, JAATakada, TNijman, SWJDhiman, PMa, JCollins, GSBajpai, RRiley, RDMoons, KGMHooft, L<strong>Background<br></strong> While many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. We aim to systematically review the adherence of Machine Learning (ML)-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement. <br><strong> Methods<br></strong> We included articles reporting on development or external validation of a multivariable prediction model (either diagnostic or prognostic) developed using supervised ML for individualized predictions across all medical fields. We searched PubMed from 1 January 2018 to 31 December 2019. Data extraction was performed using the 22-item checklist for reporting of prediction model studies (www.TRIPOD-statement.org). We measured the overall adherence per article and per TRIPOD item. <br><strong> Results<br></strong> Our search identified 24,814 articles, of which 152 articles were included: 94 (61.8%) prognostic and 58 (38.2%) diagnostic prediction model studies. Overall, articles adhered to a median of 38.7% (IQR 31.0–46.4%) of TRIPOD items. No article fully adhered to complete reporting of the abstract and very few reported the flow of participants (3.9%, 95% CI 1.8 to 8.3), appropriate title (4.6%, 95% CI 2.2 to 9.2), blinding of predictors (4.6%, 95% CI 2.2 to 9.2), model specification (5.2%, 95% CI 2.4 to 10.8), and model’s predictive performance (5.9%, 95% CI 3.1 to 10.9). There was often complete reporting of source of data (98.0%, 95% CI 94.4 to 99.3) and interpretation of the results (94.7%, 95% CI 90.0 to 97.3). <br><strong> Conclusion<br></strong> Similar to prediction model studies developed using conventional regression-based techniques, the completeness of reporting is poor. Essential information to decide to use the model (i.e. model specification and its performance) is rarely reported. However, some items and sub-items of TRIPOD might be less suitable for ML-based prediction model studies and thus, TRIPOD requires extensions. Overall, there is an urgent need to improve the reporting quality and usability of research to avoid research waste. <br><strong> Systematic review registration<br></strong> PROSPERO, CRD42019161764. |
| spellingShingle | Andaur Navarro, CL Damen, JAA Takada, T Nijman, SWJ Dhiman, P Ma, J Collins, GS Bajpai, R Riley, RD Moons, KGM Hooft, L Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review |
| title | Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review |
| title_full | Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review |
| title_fullStr | Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review |
| title_full_unstemmed | Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review |
| title_short | Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review |
| title_sort | completeness of reporting of clinical prediction models developed using supervised machine learning a systematic review |
| work_keys_str_mv | AT andaurnavarrocl completenessofreportingofclinicalpredictionmodelsdevelopedusingsupervisedmachinelearningasystematicreview AT damenjaa completenessofreportingofclinicalpredictionmodelsdevelopedusingsupervisedmachinelearningasystematicreview AT takadat completenessofreportingofclinicalpredictionmodelsdevelopedusingsupervisedmachinelearningasystematicreview AT nijmanswj completenessofreportingofclinicalpredictionmodelsdevelopedusingsupervisedmachinelearningasystematicreview AT dhimanp completenessofreportingofclinicalpredictionmodelsdevelopedusingsupervisedmachinelearningasystematicreview AT maj completenessofreportingofclinicalpredictionmodelsdevelopedusingsupervisedmachinelearningasystematicreview AT collinsgs completenessofreportingofclinicalpredictionmodelsdevelopedusingsupervisedmachinelearningasystematicreview AT bajpair completenessofreportingofclinicalpredictionmodelsdevelopedusingsupervisedmachinelearningasystematicreview AT rileyrd completenessofreportingofclinicalpredictionmodelsdevelopedusingsupervisedmachinelearningasystematicreview AT moonskgm completenessofreportingofclinicalpredictionmodelsdevelopedusingsupervisedmachinelearningasystematicreview AT hooftl completenessofreportingofclinicalpredictionmodelsdevelopedusingsupervisedmachinelearningasystematicreview |