External validation of a machine learning model to predict hemodynamic instability in intensive care unit
Abstract Background Early prediction model of hemodynamic instability has the potential to improve the critical care, whereas limited external validation on the generalizability. We aimed to independently validate the Hemodynamic Stability Index (HSI), a multi-parameter machine learning model, in pr...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2022-07-01
|
| Series: | Critical Care |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s13054-022-04088-9 |
| _version_ | 1811293728011190272 |
|---|---|
| author | Chiang Dung-Hung Tian Cong Jiang Zeyu Ou-Yang Yu-Shan Lin Yung-Yan |
| author_facet | Chiang Dung-Hung Tian Cong Jiang Zeyu Ou-Yang Yu-Shan Lin Yung-Yan |
| author_sort | Chiang Dung-Hung |
| collection | DOAJ |
| description | Abstract Background Early prediction model of hemodynamic instability has the potential to improve the critical care, whereas limited external validation on the generalizability. We aimed to independently validate the Hemodynamic Stability Index (HSI), a multi-parameter machine learning model, in predicting hemodynamic instability in Asian patients. Method Hemodynamic instability was marked by using inotropic, vasopressor, significant fluid therapy, and/or blood transfusions. This retrospective study included among 15,967 ICU patients who aged 20 years or older (not included 20 years) and stayed in ICU for more than 6 h admitted to Taipei Veteran General Hospital (TPEVGH) between January 1, 2010, and March 31, 2020, of whom hemodynamic instability occurred in 3053 patients (prevalence = 19%). These patients in unstable group received at least one intervention during their ICU stays, and the HSI score of both stable and unstable group was calculated in every hour before intervention. The model performance was assessed using the area under the receiver operating characteristic curve (AUROC) and was compared to single indicators like systolic blood pressure (SBP) and shock index. The hemodynamic instability alarm was set by selecting optimal threshold with high sensitivity, acceptable specificity, and lead time before intervention was calculated to indicate when patients were firstly identified as high risk of hemodynamic instability. Results The AUROC of HSI was 0.76 (95% CI, 0.75–0.77), which performed significantly better than shock Index (0.7; 95% CI, 0.69–0.71) and SBP (0.69; 95% CI, 0.68–0.70). By selecting 0.7 as a threshold, HSI predicted 72% of all 3053 patients who received hemodynamic interventions with 67% in specificity. Time-varying results also showed that HSI score significantly outperformed single indicators even up to 24 h before intervention. And 95% unstable patients can be identified more than 5 h in advance. Conclusions The HSI has acceptable discrimination but underestimates the risk of stable patients in predicting the onset of hemodynamic instability in an external cohort. |
| first_indexed | 2024-04-13T05:06:46Z |
| format | Article |
| id | doaj.art-d765b6c0caca4c06b596d4dbebedc42d |
| institution | Directory Open Access Journal |
| issn | 1364-8535 |
| language | English |
| last_indexed | 2024-04-13T05:06:46Z |
| publishDate | 2022-07-01 |
| publisher | BMC |
| record_format | Article |
| series | Critical Care |
| spelling | doaj.art-d765b6c0caca4c06b596d4dbebedc42d2022-12-22T03:01:10ZengBMCCritical Care1364-85352022-07-0126111010.1186/s13054-022-04088-9External validation of a machine learning model to predict hemodynamic instability in intensive care unitChiang Dung-Hung0Tian Cong1Jiang Zeyu2Ou-Yang Yu-Shan3Lin Yung-Yan4Department of Critical Care Medicine, Taipei Veteran General HospitalPhilips Research ChinaPhilips Research ChinaDepartment of Critical Care Medicine, Taipei Veteran General HospitalDepartment of Critical Care Medicine, Taipei Veteran General HospitalAbstract Background Early prediction model of hemodynamic instability has the potential to improve the critical care, whereas limited external validation on the generalizability. We aimed to independently validate the Hemodynamic Stability Index (HSI), a multi-parameter machine learning model, in predicting hemodynamic instability in Asian patients. Method Hemodynamic instability was marked by using inotropic, vasopressor, significant fluid therapy, and/or blood transfusions. This retrospective study included among 15,967 ICU patients who aged 20 years or older (not included 20 years) and stayed in ICU for more than 6 h admitted to Taipei Veteran General Hospital (TPEVGH) between January 1, 2010, and March 31, 2020, of whom hemodynamic instability occurred in 3053 patients (prevalence = 19%). These patients in unstable group received at least one intervention during their ICU stays, and the HSI score of both stable and unstable group was calculated in every hour before intervention. The model performance was assessed using the area under the receiver operating characteristic curve (AUROC) and was compared to single indicators like systolic blood pressure (SBP) and shock index. The hemodynamic instability alarm was set by selecting optimal threshold with high sensitivity, acceptable specificity, and lead time before intervention was calculated to indicate when patients were firstly identified as high risk of hemodynamic instability. Results The AUROC of HSI was 0.76 (95% CI, 0.75–0.77), which performed significantly better than shock Index (0.7; 95% CI, 0.69–0.71) and SBP (0.69; 95% CI, 0.68–0.70). By selecting 0.7 as a threshold, HSI predicted 72% of all 3053 patients who received hemodynamic interventions with 67% in specificity. Time-varying results also showed that HSI score significantly outperformed single indicators even up to 24 h before intervention. And 95% unstable patients can be identified more than 5 h in advance. Conclusions The HSI has acceptable discrimination but underestimates the risk of stable patients in predicting the onset of hemodynamic instability in an external cohort.https://doi.org/10.1186/s13054-022-04088-9Hemodynamic Stability IndexEarly prediction modelMachine learningClinical decision supportExternal validation |
| spellingShingle | Chiang Dung-Hung Tian Cong Jiang Zeyu Ou-Yang Yu-Shan Lin Yung-Yan External validation of a machine learning model to predict hemodynamic instability in intensive care unit Critical Care Hemodynamic Stability Index Early prediction model Machine learning Clinical decision support External validation |
| title | External validation of a machine learning model to predict hemodynamic instability in intensive care unit |
| title_full | External validation of a machine learning model to predict hemodynamic instability in intensive care unit |
| title_fullStr | External validation of a machine learning model to predict hemodynamic instability in intensive care unit |
| title_full_unstemmed | External validation of a machine learning model to predict hemodynamic instability in intensive care unit |
| title_short | External validation of a machine learning model to predict hemodynamic instability in intensive care unit |
| title_sort | external validation of a machine learning model to predict hemodynamic instability in intensive care unit |
| topic | Hemodynamic Stability Index Early prediction model Machine learning Clinical decision support External validation |
| url | https://doi.org/10.1186/s13054-022-04088-9 |
| work_keys_str_mv | AT chiangdunghung externalvalidationofamachinelearningmodeltopredicthemodynamicinstabilityinintensivecareunit AT tiancong externalvalidationofamachinelearningmodeltopredicthemodynamicinstabilityinintensivecareunit AT jiangzeyu externalvalidationofamachinelearningmodeltopredicthemodynamicinstabilityinintensivecareunit AT ouyangyushan externalvalidationofamachinelearningmodeltopredicthemodynamicinstabilityinintensivecareunit AT linyungyan externalvalidationofamachinelearningmodeltopredicthemodynamicinstabilityinintensivecareunit |