Racial Differences in Accuracy of Predictive Models for High-Flow Nasal Cannula Failure in COVID-19
OBJECTIVES:. To develop and validate machine learning (ML) models to predict high-flow nasal cannula (HFNC) failure in COVID-19, compare their performance to the respiratory rate-oxygenation (ROX) index, and evaluate model accuracy by self-reported race. DESIGN:. Retrospective cohort study. SETTING:...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wolters Kluwer
2024-03-01
|
| Series: | Critical Care Explorations |
| Online Access: | http://journals.lww.com/10.1097/CCE.0000000000001059 |
| _version_ | 1827180679102726144 |
|---|---|
| author | Philip Yang, MD, MSc Ismail A. Gregory, MD Chad Robichaux, MPH Andre L. Holder, MD, MSc Greg S. Martin, MD, MSc Annette M. Esper, MD, MSc Rishikesan Kamaleswaran, PhD Judy W. Gichoya, MD Sivasubramanium V. Bhavani, MD, MS |
| author_facet | Philip Yang, MD, MSc Ismail A. Gregory, MD Chad Robichaux, MPH Andre L. Holder, MD, MSc Greg S. Martin, MD, MSc Annette M. Esper, MD, MSc Rishikesan Kamaleswaran, PhD Judy W. Gichoya, MD Sivasubramanium V. Bhavani, MD, MS |
| author_sort | Philip Yang, MD, MSc |
| collection | DOAJ |
| description | OBJECTIVES:. To develop and validate machine learning (ML) models to predict high-flow nasal cannula (HFNC) failure in COVID-19, compare their performance to the respiratory rate-oxygenation (ROX) index, and evaluate model accuracy by self-reported race.
DESIGN:. Retrospective cohort study.
SETTING:. Four Emory University Hospitals in Atlanta, GA.
PATIENTS:. Adult patients hospitalized with COVID-19 between March 2020 and April 2022 who received HFNC therapy within 24 hours of ICU admission were included.
INTERVENTIONS:. None.
MEASUREMENTS AND MAIN RESULTS:. Four types of supervised ML models were developed for predicting HFNC failure (defined as intubation or death within 7 d of HFNC initiation), using routine clinical variables from the first 24 hours of ICU admission. Models were trained on the first 60% (n = 594) of admissions and validated on the latter 40% (n = 390) of admissions to simulate prospective implementation. Among 984 patients included, 317 patients (32.2%) developed HFNC failure. eXtreme Gradient Boosting (XGB) model had the highest area under the receiver-operator characteristic curve (AUROC) for predicting HFNC failure (0.707), and was the only model with significantly better performance than the ROX index (AUROC 0.616). XGB model had significantly worse performance in Black patients compared with White patients (AUROC 0.663 vs. 0.808, p = 0.02). Racial differences in the XGB model were reduced and no longer statistically significant when restricted to patients with nonmissing arterial blood gas data, and when XGB model was developed to predict mortality (rather than the composite outcome of failure, which could be influenced by biased clinical decisions for intubation).
CONCLUSIONS:. Our XGB model had better discrimination for predicting HFNC failure in COVID-19 than the ROX index, but had racial differences in accuracy of predictions. Further studies are needed to understand and mitigate potential sources of biases in clinical ML models and to improve their equitability. |
| first_indexed | 2024-04-24T18:49:51Z |
| format | Article |
| id | doaj.art-aa8b5228a84e4813b909d5b356f7c8ec |
| institution | Directory Open Access Journal |
| issn | 2639-8028 |
| language | English |
| last_indexed | 2025-03-21T05:34:45Z |
| publishDate | 2024-03-01 |
| publisher | Wolters Kluwer |
| record_format | Article |
| series | Critical Care Explorations |
| spelling | doaj.art-aa8b5228a84e4813b909d5b356f7c8ec2024-07-25T03:41:46ZengWolters KluwerCritical Care Explorations2639-80282024-03-0163e105910.1097/CCE.0000000000001059202403000-00009Racial Differences in Accuracy of Predictive Models for High-Flow Nasal Cannula Failure in COVID-19Philip Yang, MD, MSc0Ismail A. Gregory, MD1Chad Robichaux, MPH2Andre L. Holder, MD, MSc3Greg S. Martin, MD, MSc4Annette M. Esper, MD, MSc5Rishikesan Kamaleswaran, PhD6Judy W. Gichoya, MD7Sivasubramanium V. Bhavani, MD, MS81 Division of Pulmonary, Allergy, Critical Care, and Sleep Medicine, Emory University, Atlanta, GA.1 Division of Pulmonary, Allergy, Critical Care, and Sleep Medicine, Emory University, Atlanta, GA.2 Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA.1 Division of Pulmonary, Allergy, Critical Care, and Sleep Medicine, Emory University, Atlanta, GA.1 Division of Pulmonary, Allergy, Critical Care, and Sleep Medicine, Emory University, Atlanta, GA.1 Division of Pulmonary, Allergy, Critical Care, and Sleep Medicine, Emory University, Atlanta, GA.2 Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA.4 Department of Radiology and Imaging Sciences, Emory University School of Medicine, Atlanta, GA.1 Division of Pulmonary, Allergy, Critical Care, and Sleep Medicine, Emory University, Atlanta, GA.OBJECTIVES:. To develop and validate machine learning (ML) models to predict high-flow nasal cannula (HFNC) failure in COVID-19, compare their performance to the respiratory rate-oxygenation (ROX) index, and evaluate model accuracy by self-reported race. DESIGN:. Retrospective cohort study. SETTING:. Four Emory University Hospitals in Atlanta, GA. PATIENTS:. Adult patients hospitalized with COVID-19 between March 2020 and April 2022 who received HFNC therapy within 24 hours of ICU admission were included. INTERVENTIONS:. None. MEASUREMENTS AND MAIN RESULTS:. Four types of supervised ML models were developed for predicting HFNC failure (defined as intubation or death within 7 d of HFNC initiation), using routine clinical variables from the first 24 hours of ICU admission. Models were trained on the first 60% (n = 594) of admissions and validated on the latter 40% (n = 390) of admissions to simulate prospective implementation. Among 984 patients included, 317 patients (32.2%) developed HFNC failure. eXtreme Gradient Boosting (XGB) model had the highest area under the receiver-operator characteristic curve (AUROC) for predicting HFNC failure (0.707), and was the only model with significantly better performance than the ROX index (AUROC 0.616). XGB model had significantly worse performance in Black patients compared with White patients (AUROC 0.663 vs. 0.808, p = 0.02). Racial differences in the XGB model were reduced and no longer statistically significant when restricted to patients with nonmissing arterial blood gas data, and when XGB model was developed to predict mortality (rather than the composite outcome of failure, which could be influenced by biased clinical decisions for intubation). CONCLUSIONS:. Our XGB model had better discrimination for predicting HFNC failure in COVID-19 than the ROX index, but had racial differences in accuracy of predictions. Further studies are needed to understand and mitigate potential sources of biases in clinical ML models and to improve their equitability.http://journals.lww.com/10.1097/CCE.0000000000001059 |
| spellingShingle | Philip Yang, MD, MSc Ismail A. Gregory, MD Chad Robichaux, MPH Andre L. Holder, MD, MSc Greg S. Martin, MD, MSc Annette M. Esper, MD, MSc Rishikesan Kamaleswaran, PhD Judy W. Gichoya, MD Sivasubramanium V. Bhavani, MD, MS Racial Differences in Accuracy of Predictive Models for High-Flow Nasal Cannula Failure in COVID-19 Critical Care Explorations |
| title | Racial Differences in Accuracy of Predictive Models for High-Flow Nasal Cannula Failure in COVID-19 |
| title_full | Racial Differences in Accuracy of Predictive Models for High-Flow Nasal Cannula Failure in COVID-19 |
| title_fullStr | Racial Differences in Accuracy of Predictive Models for High-Flow Nasal Cannula Failure in COVID-19 |
| title_full_unstemmed | Racial Differences in Accuracy of Predictive Models for High-Flow Nasal Cannula Failure in COVID-19 |
| title_short | Racial Differences in Accuracy of Predictive Models for High-Flow Nasal Cannula Failure in COVID-19 |
| title_sort | racial differences in accuracy of predictive models for high flow nasal cannula failure in covid 19 |
| url | http://journals.lww.com/10.1097/CCE.0000000000001059 |
| work_keys_str_mv | AT philipyangmdmsc racialdifferencesinaccuracyofpredictivemodelsforhighflownasalcannulafailureincovid19 AT ismailagregorymd racialdifferencesinaccuracyofpredictivemodelsforhighflownasalcannulafailureincovid19 AT chadrobichauxmph racialdifferencesinaccuracyofpredictivemodelsforhighflownasalcannulafailureincovid19 AT andrelholdermdmsc racialdifferencesinaccuracyofpredictivemodelsforhighflownasalcannulafailureincovid19 AT gregsmartinmdmsc racialdifferencesinaccuracyofpredictivemodelsforhighflownasalcannulafailureincovid19 AT annettemespermdmsc racialdifferencesinaccuracyofpredictivemodelsforhighflownasalcannulafailureincovid19 AT rishikesankamaleswaranphd racialdifferencesinaccuracyofpredictivemodelsforhighflownasalcannulafailureincovid19 AT judywgichoyamd racialdifferencesinaccuracyofpredictivemodelsforhighflownasalcannulafailureincovid19 AT sivasubramaniumvbhavanimdms racialdifferencesinaccuracyofpredictivemodelsforhighflownasalcannulafailureincovid19 |