Multisite implementation of a workflow-integrated machine learning system to optimize COVID-19 hospital admission decisions
Abstract Demand has outstripped healthcare supply during the coronavirus disease 2019 (COVID-19) pandemic. Emergency departments (EDs) are tasked with distinguishing patients who require hospital resources from those who may be safely discharged to the community. The novelty and high variability of...
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2022-07-01
|
| Series: | npj Digital Medicine |
| Online Access: | https://doi.org/10.1038/s41746-022-00646-1 |
| _version_ | 1797427911666434048 |
|---|---|
| author | Jeremiah S. Hinson Eili Klein Aria Smith Matthew Toerper Trushar Dungarani David Hager Peter Hill Gabor Kelen Joshua D. Niforatos R. Scott Stephens Alexandra T. Strauss Scott Levin |
| author_facet | Jeremiah S. Hinson Eili Klein Aria Smith Matthew Toerper Trushar Dungarani David Hager Peter Hill Gabor Kelen Joshua D. Niforatos R. Scott Stephens Alexandra T. Strauss Scott Levin |
| author_sort | Jeremiah S. Hinson |
| collection | DOAJ |
| description | Abstract Demand has outstripped healthcare supply during the coronavirus disease 2019 (COVID-19) pandemic. Emergency departments (EDs) are tasked with distinguishing patients who require hospital resources from those who may be safely discharged to the community. The novelty and high variability of COVID-19 have made these determinations challenging. In this study, we developed, implemented and evaluated an electronic health record (EHR) embedded clinical decision support (CDS) system that leverages machine learning (ML) to estimate short-term risk for clinical deterioration in patients with or under investigation for COVID-19. The system translates model-generated risk for critical care needs within 24 h and inpatient care needs within 72 h into rapidly interpretable COVID-19 Deterioration Risk Levels made viewable within ED clinician workflow. ML models were derived in a retrospective cohort of 21,452 ED patients who visited one of five ED study sites and were prospectively validated in 15,670 ED visits that occurred before (n = 4322) or after (n = 11,348) CDS implementation; model performance and numerous patient-oriented outcomes including in-hospital mortality were measured across study periods. Incidence of critical care needs within 24 h and inpatient care needs within 72 h were 10.7% and 22.5%, respectively and were similar across study periods. ML model performance was excellent under all conditions, with AUC ranging from 0.85 to 0.91 for prediction of critical care needs and 0.80–0.90 for inpatient care needs. Total mortality was unchanged across study periods but was reduced among high-risk patients after CDS implementation. |
| first_indexed | 2024-03-09T08:50:26Z |
| format | Article |
| id | doaj.art-f3812fd4046a46cb94d5d87e49f47d5e |
| institution | Directory Open Access Journal |
| issn | 2398-6352 |
| language | English |
| last_indexed | 2024-03-09T08:50:26Z |
| publishDate | 2022-07-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | npj Digital Medicine |
| spelling | doaj.art-f3812fd4046a46cb94d5d87e49f47d5e2023-12-02T14:35:51ZengNature Portfolionpj Digital Medicine2398-63522022-07-015111010.1038/s41746-022-00646-1Multisite implementation of a workflow-integrated machine learning system to optimize COVID-19 hospital admission decisionsJeremiah S. Hinson0Eili Klein1Aria Smith2Matthew Toerper3Trushar Dungarani4David Hager5Peter Hill6Gabor Kelen7Joshua D. Niforatos8R. Scott Stephens9Alexandra T. Strauss10Scott Levin11Department of Emergency Medicine, Johns Hopkins University School of MedicineDepartment of Emergency Medicine, Johns Hopkins University School of MedicineDepartment of Emergency Medicine, Johns Hopkins University School of MedicineDepartment of Emergency Medicine, Johns Hopkins University School of MedicineDepartment of Medicine, Howard County General HospitalDepartment of Medicine, Johns Hopkins University School of MedicineDepartment of Emergency Medicine, Johns Hopkins University School of MedicineDepartment of Emergency Medicine, Johns Hopkins University School of MedicineDepartment of Emergency Medicine, Johns Hopkins University School of MedicineDepartment of Medicine, Johns Hopkins University School of MedicineMalone Center for Engineering in Healthcare, Johns Hopkins University Whiting School of EngineeringDepartment of Emergency Medicine, Johns Hopkins University School of MedicineAbstract Demand has outstripped healthcare supply during the coronavirus disease 2019 (COVID-19) pandemic. Emergency departments (EDs) are tasked with distinguishing patients who require hospital resources from those who may be safely discharged to the community. The novelty and high variability of COVID-19 have made these determinations challenging. In this study, we developed, implemented and evaluated an electronic health record (EHR) embedded clinical decision support (CDS) system that leverages machine learning (ML) to estimate short-term risk for clinical deterioration in patients with or under investigation for COVID-19. The system translates model-generated risk for critical care needs within 24 h and inpatient care needs within 72 h into rapidly interpretable COVID-19 Deterioration Risk Levels made viewable within ED clinician workflow. ML models were derived in a retrospective cohort of 21,452 ED patients who visited one of five ED study sites and were prospectively validated in 15,670 ED visits that occurred before (n = 4322) or after (n = 11,348) CDS implementation; model performance and numerous patient-oriented outcomes including in-hospital mortality were measured across study periods. Incidence of critical care needs within 24 h and inpatient care needs within 72 h were 10.7% and 22.5%, respectively and were similar across study periods. ML model performance was excellent under all conditions, with AUC ranging from 0.85 to 0.91 for prediction of critical care needs and 0.80–0.90 for inpatient care needs. Total mortality was unchanged across study periods but was reduced among high-risk patients after CDS implementation.https://doi.org/10.1038/s41746-022-00646-1 |
| spellingShingle | Jeremiah S. Hinson Eili Klein Aria Smith Matthew Toerper Trushar Dungarani David Hager Peter Hill Gabor Kelen Joshua D. Niforatos R. Scott Stephens Alexandra T. Strauss Scott Levin Multisite implementation of a workflow-integrated machine learning system to optimize COVID-19 hospital admission decisions npj Digital Medicine |
| title | Multisite implementation of a workflow-integrated machine learning system to optimize COVID-19 hospital admission decisions |
| title_full | Multisite implementation of a workflow-integrated machine learning system to optimize COVID-19 hospital admission decisions |
| title_fullStr | Multisite implementation of a workflow-integrated machine learning system to optimize COVID-19 hospital admission decisions |
| title_full_unstemmed | Multisite implementation of a workflow-integrated machine learning system to optimize COVID-19 hospital admission decisions |
| title_short | Multisite implementation of a workflow-integrated machine learning system to optimize COVID-19 hospital admission decisions |
| title_sort | multisite implementation of a workflow integrated machine learning system to optimize covid 19 hospital admission decisions |
| url | https://doi.org/10.1038/s41746-022-00646-1 |
| work_keys_str_mv | AT jeremiahshinson multisiteimplementationofaworkflowintegratedmachinelearningsystemtooptimizecovid19hospitaladmissiondecisions AT eiliklein multisiteimplementationofaworkflowintegratedmachinelearningsystemtooptimizecovid19hospitaladmissiondecisions AT ariasmith multisiteimplementationofaworkflowintegratedmachinelearningsystemtooptimizecovid19hospitaladmissiondecisions AT matthewtoerper multisiteimplementationofaworkflowintegratedmachinelearningsystemtooptimizecovid19hospitaladmissiondecisions AT trushardungarani multisiteimplementationofaworkflowintegratedmachinelearningsystemtooptimizecovid19hospitaladmissiondecisions AT davidhager multisiteimplementationofaworkflowintegratedmachinelearningsystemtooptimizecovid19hospitaladmissiondecisions AT peterhill multisiteimplementationofaworkflowintegratedmachinelearningsystemtooptimizecovid19hospitaladmissiondecisions AT gaborkelen multisiteimplementationofaworkflowintegratedmachinelearningsystemtooptimizecovid19hospitaladmissiondecisions AT joshuadniforatos multisiteimplementationofaworkflowintegratedmachinelearningsystemtooptimizecovid19hospitaladmissiondecisions AT rscottstephens multisiteimplementationofaworkflowintegratedmachinelearningsystemtooptimizecovid19hospitaladmissiondecisions AT alexandratstrauss multisiteimplementationofaworkflowintegratedmachinelearningsystemtooptimizecovid19hospitaladmissiondecisions AT scottlevin multisiteimplementationofaworkflowintegratedmachinelearningsystemtooptimizecovid19hospitaladmissiondecisions |