Identifying unreliable predictions in clinical risk models
Abstract The ability to identify patients who are likely to have an adverse outcome is an essential component of good clinical care. Therefore, predictive risk stratification models play an important role in clinical decision making. Determining whether a given predictive model is suitable for clini...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2020-01-01
|
| Series: | npj Digital Medicine |
| Online Access: | https://doi.org/10.1038/s41746-019-0209-7 |
| _version_ | 1797643043734552576 |
|---|---|
| author | Paul D. Myers Kenney Ng Kristen Severson Uri Kartoun Wangzhi Dai Wei Huang Frederick A. Anderson Collin M. Stultz |
| author_facet | Paul D. Myers Kenney Ng Kristen Severson Uri Kartoun Wangzhi Dai Wei Huang Frederick A. Anderson Collin M. Stultz |
| author_sort | Paul D. Myers |
| collection | DOAJ |
| description | Abstract The ability to identify patients who are likely to have an adverse outcome is an essential component of good clinical care. Therefore, predictive risk stratification models play an important role in clinical decision making. Determining whether a given predictive model is suitable for clinical use usually involves evaluating the model’s performance on large patient datasets using standard statistical measures of success (e.g., accuracy, discriminatory ability). However, as these metrics correspond to averages over patients who have a range of different characteristics, it is difficult to discern whether an individual prediction on a given patient should be trusted using these measures alone. In this paper, we introduce a new method for identifying patient subgroups where a predictive model is expected to be poor, thereby highlighting when a given prediction is misleading and should not be trusted. The resulting “unreliability score” can be computed for any clinical risk model and is suitable in the setting of large class imbalance, a situation often encountered in healthcare settings. Using data from more than 40,000 patients in the Global Registry of Acute Coronary Events (GRACE), we demonstrate that patients with high unreliability scores form a subgroup in which the predictive model has both decreased accuracy and decreased discriminatory ability. |
| first_indexed | 2024-03-11T14:08:58Z |
| format | Article |
| id | doaj.art-2fc1775645bb49caa1fa9a615f60562f |
| institution | Directory Open Access Journal |
| issn | 2398-6352 |
| language | English |
| last_indexed | 2024-03-11T14:08:58Z |
| publishDate | 2020-01-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | npj Digital Medicine |
| spelling | doaj.art-2fc1775645bb49caa1fa9a615f60562f2023-11-02T00:47:33ZengNature Portfolionpj Digital Medicine2398-63522020-01-01311810.1038/s41746-019-0209-7Identifying unreliable predictions in clinical risk modelsPaul D. Myers0Kenney Ng1Kristen Severson2Uri Kartoun3Wangzhi Dai4Wei Huang5Frederick A. Anderson6Collin M. Stultz7Department of Electrical Engineering and Computer Science and Research Laboratory for Electronics, Massachusetts Institute of TechnologyCenter for Computational Health, IBM ResearchCenter for Computational Health, IBM ResearchCenter for Computational Health, IBM ResearchDepartment of Electrical Engineering and Computer Science and Research Laboratory for Electronics, Massachusetts Institute of TechnologyCenter for Outcomes Research, University of Massachusetts Medical SchoolCenter for Outcomes Research, University of Massachusetts Medical SchoolDepartment of Electrical Engineering and Computer Science and Research Laboratory for Electronics, Massachusetts Institute of TechnologyAbstract The ability to identify patients who are likely to have an adverse outcome is an essential component of good clinical care. Therefore, predictive risk stratification models play an important role in clinical decision making. Determining whether a given predictive model is suitable for clinical use usually involves evaluating the model’s performance on large patient datasets using standard statistical measures of success (e.g., accuracy, discriminatory ability). However, as these metrics correspond to averages over patients who have a range of different characteristics, it is difficult to discern whether an individual prediction on a given patient should be trusted using these measures alone. In this paper, we introduce a new method for identifying patient subgroups where a predictive model is expected to be poor, thereby highlighting when a given prediction is misleading and should not be trusted. The resulting “unreliability score” can be computed for any clinical risk model and is suitable in the setting of large class imbalance, a situation often encountered in healthcare settings. Using data from more than 40,000 patients in the Global Registry of Acute Coronary Events (GRACE), we demonstrate that patients with high unreliability scores form a subgroup in which the predictive model has both decreased accuracy and decreased discriminatory ability.https://doi.org/10.1038/s41746-019-0209-7 |
| spellingShingle | Paul D. Myers Kenney Ng Kristen Severson Uri Kartoun Wangzhi Dai Wei Huang Frederick A. Anderson Collin M. Stultz Identifying unreliable predictions in clinical risk models npj Digital Medicine |
| title | Identifying unreliable predictions in clinical risk models |
| title_full | Identifying unreliable predictions in clinical risk models |
| title_fullStr | Identifying unreliable predictions in clinical risk models |
| title_full_unstemmed | Identifying unreliable predictions in clinical risk models |
| title_short | Identifying unreliable predictions in clinical risk models |
| title_sort | identifying unreliable predictions in clinical risk models |
| url | https://doi.org/10.1038/s41746-019-0209-7 |
| work_keys_str_mv | AT pauldmyers identifyingunreliablepredictionsinclinicalriskmodels AT kenneyng identifyingunreliablepredictionsinclinicalriskmodels AT kristenseverson identifyingunreliablepredictionsinclinicalriskmodels AT urikartoun identifyingunreliablepredictionsinclinicalriskmodels AT wangzhidai identifyingunreliablepredictionsinclinicalriskmodels AT weihuang identifyingunreliablepredictionsinclinicalriskmodels AT frederickaanderson identifyingunreliablepredictionsinclinicalriskmodels AT collinmstultz identifyingunreliablepredictionsinclinicalriskmodels |