Evidence of questionable research practices in clinical prediction models
Abstract Background Clinical prediction models are widely used in health and medical research. The area under the receiver operating characteristic curve (AUC) is a frequently used estimate to describe the discriminatory ability of a clinical prediction model. The AUC is often interpreted relative t...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-09-01
|
| Series: | BMC Medicine |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12916-023-03048-6 |
| _version_ | 1797452348453289984 |
|---|---|
| author | Nicole White Rex Parsons Gary Collins Adrian Barnett |
| author_facet | Nicole White Rex Parsons Gary Collins Adrian Barnett |
| author_sort | Nicole White |
| collection | DOAJ |
| description | Abstract Background Clinical prediction models are widely used in health and medical research. The area under the receiver operating characteristic curve (AUC) is a frequently used estimate to describe the discriminatory ability of a clinical prediction model. The AUC is often interpreted relative to thresholds, with “good” or “excellent” models defined at 0.7, 0.8 or 0.9. These thresholds may create targets that result in “hacking”, where researchers are motivated to re-analyse their data until they achieve a “good” result. Methods We extracted AUC values from PubMed abstracts to look for evidence of hacking. We used histograms of the AUC values in bins of size 0.01 and compared the observed distribution to a smooth distribution from a spline. Results The distribution of 306,888 AUC values showed clear excesses above the thresholds of 0.7, 0.8 and 0.9 and shortfalls below the thresholds. Conclusions The AUCs for some models are over-inflated, which risks exposing patients to sub-optimal clinical decision-making. Greater modelling transparency is needed, including published protocols, and data and code sharing. |
| first_indexed | 2024-03-09T15:07:28Z |
| format | Article |
| id | doaj.art-036968072a3c4d419929e062609e7cb5 |
| institution | Directory Open Access Journal |
| issn | 1741-7015 |
| language | English |
| last_indexed | 2024-03-09T15:07:28Z |
| publishDate | 2023-09-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medicine |
| spelling | doaj.art-036968072a3c4d419929e062609e7cb52023-11-26T13:34:21ZengBMCBMC Medicine1741-70152023-09-0121111010.1186/s12916-023-03048-6Evidence of questionable research practices in clinical prediction modelsNicole White0Rex Parsons1Gary Collins2Adrian Barnett3Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of TechnologyAustralian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of TechnologyCentre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of OxfordAustralian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of TechnologyAbstract Background Clinical prediction models are widely used in health and medical research. The area under the receiver operating characteristic curve (AUC) is a frequently used estimate to describe the discriminatory ability of a clinical prediction model. The AUC is often interpreted relative to thresholds, with “good” or “excellent” models defined at 0.7, 0.8 or 0.9. These thresholds may create targets that result in “hacking”, where researchers are motivated to re-analyse their data until they achieve a “good” result. Methods We extracted AUC values from PubMed abstracts to look for evidence of hacking. We used histograms of the AUC values in bins of size 0.01 and compared the observed distribution to a smooth distribution from a spline. Results The distribution of 306,888 AUC values showed clear excesses above the thresholds of 0.7, 0.8 and 0.9 and shortfalls below the thresholds. Conclusions The AUCs for some models are over-inflated, which risks exposing patients to sub-optimal clinical decision-making. Greater modelling transparency is needed, including published protocols, and data and code sharing.https://doi.org/10.1186/s12916-023-03048-6Prediction modelArea under curveDiagnosisPrognosisHackingStatistics |
| spellingShingle | Nicole White Rex Parsons Gary Collins Adrian Barnett Evidence of questionable research practices in clinical prediction models BMC Medicine Prediction model Area under curve Diagnosis Prognosis Hacking Statistics |
| title | Evidence of questionable research practices in clinical prediction models |
| title_full | Evidence of questionable research practices in clinical prediction models |
| title_fullStr | Evidence of questionable research practices in clinical prediction models |
| title_full_unstemmed | Evidence of questionable research practices in clinical prediction models |
| title_short | Evidence of questionable research practices in clinical prediction models |
| title_sort | evidence of questionable research practices in clinical prediction models |
| topic | Prediction model Area under curve Diagnosis Prognosis Hacking Statistics |
| url | https://doi.org/10.1186/s12916-023-03048-6 |
| work_keys_str_mv | AT nicolewhite evidenceofquestionableresearchpracticesinclinicalpredictionmodels AT rexparsons evidenceofquestionableresearchpracticesinclinicalpredictionmodels AT garycollins evidenceofquestionableresearchpracticesinclinicalpredictionmodels AT adrianbarnett evidenceofquestionableresearchpracticesinclinicalpredictionmodels |