The Price of Explainability in Machine Learning Models for 100-Day Readmission Prediction in Heart Failure: Retrospective, Comparative, Machine Learning Study
BackgroundSensitive and interpretable machine learning (ML) models can provide valuable assistance to clinicians in managing patients with heart failure (HF) at discharge by identifying individual factors associated with a high risk of readmission. In this cohort study, we de...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2023-10-01
|
| Series: | Journal of Medical Internet Research |
| Online Access: | https://www.jmir.org/2023/1/e46934 |
| _version_ | 1797648001655635968 |
|---|---|
| author | Amira Soliman Björn Agvall Kobra Etminani Omar Hamed Markus Lingman |
| author_facet | Amira Soliman Björn Agvall Kobra Etminani Omar Hamed Markus Lingman |
| author_sort | Amira Soliman |
| collection | DOAJ |
| description |
BackgroundSensitive and interpretable machine learning (ML) models can provide valuable assistance to clinicians in managing patients with heart failure (HF) at discharge by identifying individual factors associated with a high risk of readmission. In this cohort study, we delve into the factors driving the potential utility of classification models as decision support tools for predicting readmissions in patients with HF.
ObjectiveThe primary objective of this study is to assess the trade-off between using deep learning (DL) and traditional ML models to identify the risk of 100-day readmissions in patients with HF. Additionally, the study aims to provide explanations for the model predictions by highlighting important features both on a global scale across the patient cohort and on a local level for individual patients.
MethodsThe retrospective data for this study were obtained from the Regional Health Care Information Platform in Region Halland, Sweden. The study cohort consisted of patients diagnosed with HF who were over 40 years old and had been hospitalized at least once between 2017 and 2019. Data analysis encompassed the period from January 1, 2017, to December 31, 2019. Two ML models were developed and validated to predict 100-day readmissions, with a focus on the explainability of the model’s decisions. These models were built based on decision trees and recurrent neural architecture. Model explainability was obtained using an ML explainer. The predictive performance of these models was compared against 2 risk assessment tools using multiple performance metrics.
ResultsThe retrospective data set included a total of 15,612 admissions, and within these admissions, readmission occurred in 5597 cases, representing a readmission rate of 35.85%. It is noteworthy that a traditional and explainable model, informed by clinical knowledge, exhibited performance comparable to the DL model and surpassed conventional scoring methods in predicting readmission among patients with HF. The evaluation of predictive model performance was based on commonly used metrics, with an area under the precision-recall curve of 66% for the deep model and 68% for the traditional model on the holdout data set. Importantly, the explanations provided by the traditional model offer actionable insights that have the potential to enhance care planning.
ConclusionsThis study found that a widely used deep prediction model did not outperform an explainable ML model when predicting readmissions among patients with HF. The results suggest that model transparency does not necessarily compromise performance, which could facilitate the clinical adoption of such models. |
| first_indexed | 2024-03-11T15:25:48Z |
| format | Article |
| id | doaj.art-09ba3e434d464023bc09dfb25d79f468 |
| institution | Directory Open Access Journal |
| issn | 1438-8871 |
| language | English |
| last_indexed | 2024-03-11T15:25:48Z |
| publishDate | 2023-10-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | Journal of Medical Internet Research |
| spelling | doaj.art-09ba3e434d464023bc09dfb25d79f4682023-10-27T14:30:42ZengJMIR PublicationsJournal of Medical Internet Research1438-88712023-10-0125e4693410.2196/46934The Price of Explainability in Machine Learning Models for 100-Day Readmission Prediction in Heart Failure: Retrospective, Comparative, Machine Learning StudyAmira Solimanhttps://orcid.org/0000-0002-0264-8762Björn Agvallhttps://orcid.org/0000-0002-3956-6103Kobra Etminanihttps://orcid.org/0000-0003-2006-6229Omar Hamedhttps://orcid.org/0009-0006-2330-2580Markus Lingmanhttps://orcid.org/0000-0002-4068-0341 BackgroundSensitive and interpretable machine learning (ML) models can provide valuable assistance to clinicians in managing patients with heart failure (HF) at discharge by identifying individual factors associated with a high risk of readmission. In this cohort study, we delve into the factors driving the potential utility of classification models as decision support tools for predicting readmissions in patients with HF. ObjectiveThe primary objective of this study is to assess the trade-off between using deep learning (DL) and traditional ML models to identify the risk of 100-day readmissions in patients with HF. Additionally, the study aims to provide explanations for the model predictions by highlighting important features both on a global scale across the patient cohort and on a local level for individual patients. MethodsThe retrospective data for this study were obtained from the Regional Health Care Information Platform in Region Halland, Sweden. The study cohort consisted of patients diagnosed with HF who were over 40 years old and had been hospitalized at least once between 2017 and 2019. Data analysis encompassed the period from January 1, 2017, to December 31, 2019. Two ML models were developed and validated to predict 100-day readmissions, with a focus on the explainability of the model’s decisions. These models were built based on decision trees and recurrent neural architecture. Model explainability was obtained using an ML explainer. The predictive performance of these models was compared against 2 risk assessment tools using multiple performance metrics. ResultsThe retrospective data set included a total of 15,612 admissions, and within these admissions, readmission occurred in 5597 cases, representing a readmission rate of 35.85%. It is noteworthy that a traditional and explainable model, informed by clinical knowledge, exhibited performance comparable to the DL model and surpassed conventional scoring methods in predicting readmission among patients with HF. The evaluation of predictive model performance was based on commonly used metrics, with an area under the precision-recall curve of 66% for the deep model and 68% for the traditional model on the holdout data set. Importantly, the explanations provided by the traditional model offer actionable insights that have the potential to enhance care planning. ConclusionsThis study found that a widely used deep prediction model did not outperform an explainable ML model when predicting readmissions among patients with HF. The results suggest that model transparency does not necessarily compromise performance, which could facilitate the clinical adoption of such models.https://www.jmir.org/2023/1/e46934 |
| spellingShingle | Amira Soliman Björn Agvall Kobra Etminani Omar Hamed Markus Lingman The Price of Explainability in Machine Learning Models for 100-Day Readmission Prediction in Heart Failure: Retrospective, Comparative, Machine Learning Study Journal of Medical Internet Research |
| title | The Price of Explainability in Machine Learning Models for 100-Day Readmission Prediction in Heart Failure: Retrospective, Comparative, Machine Learning Study |
| title_full | The Price of Explainability in Machine Learning Models for 100-Day Readmission Prediction in Heart Failure: Retrospective, Comparative, Machine Learning Study |
| title_fullStr | The Price of Explainability in Machine Learning Models for 100-Day Readmission Prediction in Heart Failure: Retrospective, Comparative, Machine Learning Study |
| title_full_unstemmed | The Price of Explainability in Machine Learning Models for 100-Day Readmission Prediction in Heart Failure: Retrospective, Comparative, Machine Learning Study |
| title_short | The Price of Explainability in Machine Learning Models for 100-Day Readmission Prediction in Heart Failure: Retrospective, Comparative, Machine Learning Study |
| title_sort | price of explainability in machine learning models for 100 day readmission prediction in heart failure retrospective comparative machine learning study |
| url | https://www.jmir.org/2023/1/e46934 |
| work_keys_str_mv | AT amirasoliman thepriceofexplainabilityinmachinelearningmodelsfor100dayreadmissionpredictioninheartfailureretrospectivecomparativemachinelearningstudy AT bjornagvall thepriceofexplainabilityinmachinelearningmodelsfor100dayreadmissionpredictioninheartfailureretrospectivecomparativemachinelearningstudy AT kobraetminani thepriceofexplainabilityinmachinelearningmodelsfor100dayreadmissionpredictioninheartfailureretrospectivecomparativemachinelearningstudy AT omarhamed thepriceofexplainabilityinmachinelearningmodelsfor100dayreadmissionpredictioninheartfailureretrospectivecomparativemachinelearningstudy AT markuslingman thepriceofexplainabilityinmachinelearningmodelsfor100dayreadmissionpredictioninheartfailureretrospectivecomparativemachinelearningstudy AT amirasoliman priceofexplainabilityinmachinelearningmodelsfor100dayreadmissionpredictioninheartfailureretrospectivecomparativemachinelearningstudy AT bjornagvall priceofexplainabilityinmachinelearningmodelsfor100dayreadmissionpredictioninheartfailureretrospectivecomparativemachinelearningstudy AT kobraetminani priceofexplainabilityinmachinelearningmodelsfor100dayreadmissionpredictioninheartfailureretrospectivecomparativemachinelearningstudy AT omarhamed priceofexplainabilityinmachinelearningmodelsfor100dayreadmissionpredictioninheartfailureretrospectivecomparativemachinelearningstudy AT markuslingman priceofexplainabilityinmachinelearningmodelsfor100dayreadmissionpredictioninheartfailureretrospectivecomparativemachinelearningstudy |