Development and validation of an interpretable 3 day intensive care unit readmission prediction model using explainable boosting machines
BackgroundIntensive care unit (ICU) readmissions are associated with mortality and poor outcomes. To improve discharge decisions, machine learning (ML) could help to identify patients at risk of ICU readmission. However, as many models are black boxes, dangerous properties may remain unnoticed. Wide...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-08-01
|
| Series: | Frontiers in Medicine |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fmed.2022.960296/full |
| _version_ | 1811339482320863232 |
|---|---|
| author | Stefan Hegselmann Christian Ertmer Thomas Volkert Antje Gottschalk Martin Dugas Julian Varghese |
| author_facet | Stefan Hegselmann Christian Ertmer Thomas Volkert Antje Gottschalk Martin Dugas Julian Varghese |
| author_sort | Stefan Hegselmann |
| collection | DOAJ |
| description | BackgroundIntensive care unit (ICU) readmissions are associated with mortality and poor outcomes. To improve discharge decisions, machine learning (ML) could help to identify patients at risk of ICU readmission. However, as many models are black boxes, dangerous properties may remain unnoticed. Widely used post hoc explanation methods also have inherent limitations. Few studies are evaluating inherently interpretable ML models for health care and involve clinicians in inspecting the trained model.MethodsAn inherently interpretable model for the prediction of 3 day ICU readmission was developed. We used explainable boosting machines that learn modular risk functions and which have already been shown to be suitable for the health care domain. We created a retrospective cohort of 15,589 ICU stays and 169 variables collected between 2006 and 2019 from the University Hospital Münster. A team of physicians inspected the model, checked the plausibility of each risk function, and removed problematic ones. We collected qualitative feedback during this process and analyzed the reasons for removing risk functions. The performance of the final explainable boosting machine was compared with a validated clinical score and three commonly used ML models. External validation was performed on the widely used Medical Information Mart for Intensive Care version IV database.ResultsThe developed explainable boosting machine used 67 features and showed an area under the precision-recall curve of 0.119 ± 0.020 and an area under the receiver operating characteristic curve of 0.680 ± 0.025. It performed on par with state-of-the-art gradient boosting machines (0.123 ± 0.016, 0.665 ± 0.036) and outperformed the Simplified Acute Physiology Score II (0.084 ± 0.025, 0.607 ± 0.019), logistic regression (0.092 ± 0.026, 0.587 ± 0.016), and recurrent neural networks (0.095 ± 0.008, 0.594 ± 0.027). External validation confirmed that explainable boosting machines (0.221 ± 0.023, 0.760 ± 0.010) performed similarly to gradient boosting machines (0.232 ± 0.029, 0.772 ± 0.018). Evaluation of the model inspection showed that explainable boosting machines can be useful to detect and remove problematic risk functions.ConclusionsWe developed an inherently interpretable ML model for 3 day ICU readmission prediction that reached the state-of-the-art performance of black box models. Our results suggest that for low- to medium-dimensional datasets that are common in health care, it is feasible to develop ML models that allow a high level of human control without sacrificing performance. |
| first_indexed | 2024-04-13T18:27:21Z |
| format | Article |
| id | doaj.art-42d673fc214843d0a8d972bd450510d3 |
| institution | Directory Open Access Journal |
| issn | 2296-858X |
| language | English |
| last_indexed | 2024-04-13T18:27:21Z |
| publishDate | 2022-08-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Medicine |
| spelling | doaj.art-42d673fc214843d0a8d972bd450510d32022-12-22T02:35:12ZengFrontiers Media S.A.Frontiers in Medicine2296-858X2022-08-01910.3389/fmed.2022.960296960296Development and validation of an interpretable 3 day intensive care unit readmission prediction model using explainable boosting machinesStefan Hegselmann0Christian Ertmer1Thomas Volkert2Antje Gottschalk3Martin Dugas4Julian Varghese5Institute of Medical Informatics, University of Münster, Münster, GermanyDepartment of Anesthesiology, Intensive Care and Pain Medicine, University Hospital Münster, Münster, GermanyDepartment of Anesthesiology, Intensive Care and Pain Medicine, University Hospital Münster, Münster, GermanyDepartment of Anesthesiology, Intensive Care and Pain Medicine, University Hospital Münster, Münster, GermanyInstitute of Medical Informatics, Heidelberg University Hospital, Heidelberg, GermanyInstitute of Medical Informatics, University of Münster, Münster, GermanyBackgroundIntensive care unit (ICU) readmissions are associated with mortality and poor outcomes. To improve discharge decisions, machine learning (ML) could help to identify patients at risk of ICU readmission. However, as many models are black boxes, dangerous properties may remain unnoticed. Widely used post hoc explanation methods also have inherent limitations. Few studies are evaluating inherently interpretable ML models for health care and involve clinicians in inspecting the trained model.MethodsAn inherently interpretable model for the prediction of 3 day ICU readmission was developed. We used explainable boosting machines that learn modular risk functions and which have already been shown to be suitable for the health care domain. We created a retrospective cohort of 15,589 ICU stays and 169 variables collected between 2006 and 2019 from the University Hospital Münster. A team of physicians inspected the model, checked the plausibility of each risk function, and removed problematic ones. We collected qualitative feedback during this process and analyzed the reasons for removing risk functions. The performance of the final explainable boosting machine was compared with a validated clinical score and three commonly used ML models. External validation was performed on the widely used Medical Information Mart for Intensive Care version IV database.ResultsThe developed explainable boosting machine used 67 features and showed an area under the precision-recall curve of 0.119 ± 0.020 and an area under the receiver operating characteristic curve of 0.680 ± 0.025. It performed on par with state-of-the-art gradient boosting machines (0.123 ± 0.016, 0.665 ± 0.036) and outperformed the Simplified Acute Physiology Score II (0.084 ± 0.025, 0.607 ± 0.019), logistic regression (0.092 ± 0.026, 0.587 ± 0.016), and recurrent neural networks (0.095 ± 0.008, 0.594 ± 0.027). External validation confirmed that explainable boosting machines (0.221 ± 0.023, 0.760 ± 0.010) performed similarly to gradient boosting machines (0.232 ± 0.029, 0.772 ± 0.018). Evaluation of the model inspection showed that explainable boosting machines can be useful to detect and remove problematic risk functions.ConclusionsWe developed an inherently interpretable ML model for 3 day ICU readmission prediction that reached the state-of-the-art performance of black box models. Our results suggest that for low- to medium-dimensional datasets that are common in health care, it is feasible to develop ML models that allow a high level of human control without sacrificing performance.https://www.frontiersin.org/articles/10.3389/fmed.2022.960296/fullintensive care unitreadmissionartificial intelligencemachine learningexplainable AIinterpretable machine learning |
| spellingShingle | Stefan Hegselmann Christian Ertmer Thomas Volkert Antje Gottschalk Martin Dugas Julian Varghese Development and validation of an interpretable 3 day intensive care unit readmission prediction model using explainable boosting machines Frontiers in Medicine intensive care unit readmission artificial intelligence machine learning explainable AI interpretable machine learning |
| title | Development and validation of an interpretable 3 day intensive care unit readmission prediction model using explainable boosting machines |
| title_full | Development and validation of an interpretable 3 day intensive care unit readmission prediction model using explainable boosting machines |
| title_fullStr | Development and validation of an interpretable 3 day intensive care unit readmission prediction model using explainable boosting machines |
| title_full_unstemmed | Development and validation of an interpretable 3 day intensive care unit readmission prediction model using explainable boosting machines |
| title_short | Development and validation of an interpretable 3 day intensive care unit readmission prediction model using explainable boosting machines |
| title_sort | development and validation of an interpretable 3 day intensive care unit readmission prediction model using explainable boosting machines |
| topic | intensive care unit readmission artificial intelligence machine learning explainable AI interpretable machine learning |
| url | https://www.frontiersin.org/articles/10.3389/fmed.2022.960296/full |
| work_keys_str_mv | AT stefanhegselmann developmentandvalidationofaninterpretable3dayintensivecareunitreadmissionpredictionmodelusingexplainableboostingmachines AT christianertmer developmentandvalidationofaninterpretable3dayintensivecareunitreadmissionpredictionmodelusingexplainableboostingmachines AT thomasvolkert developmentandvalidationofaninterpretable3dayintensivecareunitreadmissionpredictionmodelusingexplainableboostingmachines AT antjegottschalk developmentandvalidationofaninterpretable3dayintensivecareunitreadmissionpredictionmodelusingexplainableboostingmachines AT martindugas developmentandvalidationofaninterpretable3dayintensivecareunitreadmissionpredictionmodelusingexplainableboostingmachines AT julianvarghese developmentandvalidationofaninterpretable3dayintensivecareunitreadmissionpredictionmodelusingexplainableboostingmachines |