Comparison of machine learning methods with logistic regression analysis in creating predictive models for risk of critical in-hospital events in COVID-19 patients on hospital admission
Abstract Background Machine learning (ML) algorithms have been trained to early predict critical in-hospital events from COVID-19 using patient data at admission, but little is known on how their performance compares with each other and/or with statistical logistic regression (LR). This prospective...
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2022-11-01
|
| Series: | BMC Medical Informatics and Decision Making |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12911-022-02057-4 |
| _version_ | 1811315564255117312 |
|---|---|
| author | Aaron W. Sievering Peter Wohlmuth Nele Geßler Melanie A. Gunawardene Klaus Herrlinger Berthold Bein Dirk Arnold Martin Bergmann Lorenz Nowak Christian Gloeckner Ina Koch Martin Bachmann Christoph U. Herborn Axel Stang |
| author_facet | Aaron W. Sievering Peter Wohlmuth Nele Geßler Melanie A. Gunawardene Klaus Herrlinger Berthold Bein Dirk Arnold Martin Bergmann Lorenz Nowak Christian Gloeckner Ina Koch Martin Bachmann Christoph U. Herborn Axel Stang |
| author_sort | Aaron W. Sievering |
| collection | DOAJ |
| description | Abstract Background Machine learning (ML) algorithms have been trained to early predict critical in-hospital events from COVID-19 using patient data at admission, but little is known on how their performance compares with each other and/or with statistical logistic regression (LR). This prospective multicentre cohort study compares the performance of a LR and five ML models on the contribution of influencing predictors and predictor-to-event relationships on prediction model´s performance. Methods We used 25 baseline variables of 490 COVID-19 patients admitted to 8 hospitals in Germany (March–November 2020) to develop and validate (75/25 random-split) 3 linear (L1 and L2 penalty, elastic net [EN]) and 2 non-linear (support vector machine [SVM] with radial kernel, random forest [RF]) ML approaches for predicting critical events defined by intensive care unit transfer, invasive ventilation and/or death (composite end-point: 181 patients). Models were compared for performance (area-under-the-receiver-operating characteristic-curve [AUC], Brier score) and predictor importance (performance-loss metrics, partial-dependence profiles). Results Models performed close with a small benefit for LR (utilizing restricted cubic splines for non-linearity) and RF (AUC means: 0.763–0.731 [RF–L1]); Brier scores: 0.184–0.197 [LR–L1]). Top ranked predictor variables (consistently highest importance: C-reactive protein) were largely identical across models, except creatinine, which exhibited marginal (L1, L2, EN, SVM) or high/non-linear effects (LR, RF) on events. Conclusions Although the LR and ML models analysed showed no strong differences in performance and the most influencing predictors for COVID-19-related event prediction, our results indicate a predictive benefit from taking account for non-linear predictor-to-event relationships and effects. Future efforts should focus on leveraging data-driven ML technologies from static towards dynamic modelling solutions that continuously learn and adapt to changes in data environments during the evolving pandemic. Trial registration number: NCT04659187. |
| first_indexed | 2024-04-13T11:32:31Z |
| format | Article |
| id | doaj.art-faf7bce8af36413089cdfd88cfc2965c |
| institution | Directory Open Access Journal |
| issn | 1472-6947 |
| language | English |
| last_indexed | 2024-04-13T11:32:31Z |
| publishDate | 2022-11-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Informatics and Decision Making |
| spelling | doaj.art-faf7bce8af36413089cdfd88cfc2965c2022-12-22T02:48:32ZengBMCBMC Medical Informatics and Decision Making1472-69472022-11-0122111410.1186/s12911-022-02057-4Comparison of machine learning methods with logistic regression analysis in creating predictive models for risk of critical in-hospital events in COVID-19 patients on hospital admissionAaron W. Sievering0Peter Wohlmuth1Nele Geßler2Melanie A. Gunawardene3Klaus Herrlinger4Berthold Bein5Dirk Arnold6Martin Bergmann7Lorenz Nowak8Christian Gloeckner9Ina Koch10Martin Bachmann11Christoph U. Herborn12Axel Stang13Semmelweis UniversitySemmelweis UniversitySemmelweis UniversityDepartment of Cardiology and Intensive Care Medicine, Asklepios Hospital St. GeorgDepartment of Internal Medicine, Asklepios Hospital Nord-HeidbergDepartment of Anesthesiology and Intensive Care Medicine, Asklepios Hospital St. GeorgAsklepios TumorzentrumDepartment of Internal Medicine, Cardiology, and Pneumology, Asklepios Hospital WandsbekDepartment of Intensive Care and Ventilation Medicine, Asklepios Hospital München-GautingDepartment of Internal Medicine, Asklepios Hospital OberviechtachBiobank for Pulmonary Diseases, Asklepios Hospital München-GautingDepartment of Intensive Care and Ventilatory Medicine, Asklepios Hospital HarburgSemmelweis UniversitySemmelweis UniversityAbstract Background Machine learning (ML) algorithms have been trained to early predict critical in-hospital events from COVID-19 using patient data at admission, but little is known on how their performance compares with each other and/or with statistical logistic regression (LR). This prospective multicentre cohort study compares the performance of a LR and five ML models on the contribution of influencing predictors and predictor-to-event relationships on prediction model´s performance. Methods We used 25 baseline variables of 490 COVID-19 patients admitted to 8 hospitals in Germany (March–November 2020) to develop and validate (75/25 random-split) 3 linear (L1 and L2 penalty, elastic net [EN]) and 2 non-linear (support vector machine [SVM] with radial kernel, random forest [RF]) ML approaches for predicting critical events defined by intensive care unit transfer, invasive ventilation and/or death (composite end-point: 181 patients). Models were compared for performance (area-under-the-receiver-operating characteristic-curve [AUC], Brier score) and predictor importance (performance-loss metrics, partial-dependence profiles). Results Models performed close with a small benefit for LR (utilizing restricted cubic splines for non-linearity) and RF (AUC means: 0.763–0.731 [RF–L1]); Brier scores: 0.184–0.197 [LR–L1]). Top ranked predictor variables (consistently highest importance: C-reactive protein) were largely identical across models, except creatinine, which exhibited marginal (L1, L2, EN, SVM) or high/non-linear effects (LR, RF) on events. Conclusions Although the LR and ML models analysed showed no strong differences in performance and the most influencing predictors for COVID-19-related event prediction, our results indicate a predictive benefit from taking account for non-linear predictor-to-event relationships and effects. Future efforts should focus on leveraging data-driven ML technologies from static towards dynamic modelling solutions that continuously learn and adapt to changes in data environments during the evolving pandemic. Trial registration number: NCT04659187.https://doi.org/10.1186/s12911-022-02057-4COVID-19Machine learningPredictive modelsCritical event predictionClinical decision-making |
| spellingShingle | Aaron W. Sievering Peter Wohlmuth Nele Geßler Melanie A. Gunawardene Klaus Herrlinger Berthold Bein Dirk Arnold Martin Bergmann Lorenz Nowak Christian Gloeckner Ina Koch Martin Bachmann Christoph U. Herborn Axel Stang Comparison of machine learning methods with logistic regression analysis in creating predictive models for risk of critical in-hospital events in COVID-19 patients on hospital admission BMC Medical Informatics and Decision Making COVID-19 Machine learning Predictive models Critical event prediction Clinical decision-making |
| title | Comparison of machine learning methods with logistic regression analysis in creating predictive models for risk of critical in-hospital events in COVID-19 patients on hospital admission |
| title_full | Comparison of machine learning methods with logistic regression analysis in creating predictive models for risk of critical in-hospital events in COVID-19 patients on hospital admission |
| title_fullStr | Comparison of machine learning methods with logistic regression analysis in creating predictive models for risk of critical in-hospital events in COVID-19 patients on hospital admission |
| title_full_unstemmed | Comparison of machine learning methods with logistic regression analysis in creating predictive models for risk of critical in-hospital events in COVID-19 patients on hospital admission |
| title_short | Comparison of machine learning methods with logistic regression analysis in creating predictive models for risk of critical in-hospital events in COVID-19 patients on hospital admission |
| title_sort | comparison of machine learning methods with logistic regression analysis in creating predictive models for risk of critical in hospital events in covid 19 patients on hospital admission |
| topic | COVID-19 Machine learning Predictive models Critical event prediction Clinical decision-making |
| url | https://doi.org/10.1186/s12911-022-02057-4 |
| work_keys_str_mv | AT aaronwsievering comparisonofmachinelearningmethodswithlogisticregressionanalysisincreatingpredictivemodelsforriskofcriticalinhospitaleventsincovid19patientsonhospitaladmission AT peterwohlmuth comparisonofmachinelearningmethodswithlogisticregressionanalysisincreatingpredictivemodelsforriskofcriticalinhospitaleventsincovid19patientsonhospitaladmission AT nelegeßler comparisonofmachinelearningmethodswithlogisticregressionanalysisincreatingpredictivemodelsforriskofcriticalinhospitaleventsincovid19patientsonhospitaladmission AT melanieagunawardene comparisonofmachinelearningmethodswithlogisticregressionanalysisincreatingpredictivemodelsforriskofcriticalinhospitaleventsincovid19patientsonhospitaladmission AT klausherrlinger comparisonofmachinelearningmethodswithlogisticregressionanalysisincreatingpredictivemodelsforriskofcriticalinhospitaleventsincovid19patientsonhospitaladmission AT bertholdbein comparisonofmachinelearningmethodswithlogisticregressionanalysisincreatingpredictivemodelsforriskofcriticalinhospitaleventsincovid19patientsonhospitaladmission AT dirkarnold comparisonofmachinelearningmethodswithlogisticregressionanalysisincreatingpredictivemodelsforriskofcriticalinhospitaleventsincovid19patientsonhospitaladmission AT martinbergmann comparisonofmachinelearningmethodswithlogisticregressionanalysisincreatingpredictivemodelsforriskofcriticalinhospitaleventsincovid19patientsonhospitaladmission AT lorenznowak comparisonofmachinelearningmethodswithlogisticregressionanalysisincreatingpredictivemodelsforriskofcriticalinhospitaleventsincovid19patientsonhospitaladmission AT christiangloeckner comparisonofmachinelearningmethodswithlogisticregressionanalysisincreatingpredictivemodelsforriskofcriticalinhospitaleventsincovid19patientsonhospitaladmission AT inakoch comparisonofmachinelearningmethodswithlogisticregressionanalysisincreatingpredictivemodelsforriskofcriticalinhospitaleventsincovid19patientsonhospitaladmission AT martinbachmann comparisonofmachinelearningmethodswithlogisticregressionanalysisincreatingpredictivemodelsforriskofcriticalinhospitaleventsincovid19patientsonhospitaladmission AT christophuherborn comparisonofmachinelearningmethodswithlogisticregressionanalysisincreatingpredictivemodelsforriskofcriticalinhospitaleventsincovid19patientsonhospitaladmission AT axelstang comparisonofmachinelearningmethodswithlogisticregressionanalysisincreatingpredictivemodelsforriskofcriticalinhospitaleventsincovid19patientsonhospitaladmission |