Applied forecasting for delayed cerebral ischemia prediction post subarachnoid hemorrhage: Methodological fallacies
Introduction: Delayed Cerebral Ischemia (DCI) is an important cause of morbidity and mortality after aneurysmal Subarachnoid Hemorrhage (aSAH). Researchers have utilized various methods for predicting patients at risk for DCI progression. Methods: An eight-year retrospective review of aSAH patients...
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2022-01-01
|
| Series: | Informatics in Medicine Unlocked |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2352914821002823 |
| _version_ | 1798031301656182784 |
|---|---|
| author | Georgios Alexopoulos Justin Zhang Ioannis Karampelas Maheen Khan Nabiha Quadri Mayur Patel Niel Patel Mohammad Almajali Tobias A. Mattei Joanna Kemp Jeroen Coppens Philippe Mercier |
| author_facet | Georgios Alexopoulos Justin Zhang Ioannis Karampelas Maheen Khan Nabiha Quadri Mayur Patel Niel Patel Mohammad Almajali Tobias A. Mattei Joanna Kemp Jeroen Coppens Philippe Mercier |
| author_sort | Georgios Alexopoulos |
| collection | DOAJ |
| description | Introduction: Delayed Cerebral Ischemia (DCI) is an important cause of morbidity and mortality after aneurysmal Subarachnoid Hemorrhage (aSAH). Researchers have utilized various methods for predicting patients at risk for DCI progression. Methods: An eight-year retrospective review of aSAH patients who presented to St Louis University Hospital. The records were screened for demographic, clinical, and radiographic parameters. DCI was the primary outcome. We identified 16 features to fit various forecasting models and selected the best binary classifier through comprehensive machine learning (ML) workflows. Regression and ensemble tree-based algorithms were utilized, based on their performance on tabular data. We investigated whether a single model could outperform in our dataset. Due to the expected outcome class imbalance (DCI), we selected precision, recall, and F-score as threshold metrics. Precision-recall curves were used for model performance ranking. Results: Of the 213 aSAH patients analyzed, 42 progressed to DCI (19.7%). The mean age was 55.7 years. The outcome variable (DCI) was imbalanced with a class ratio of 1:4. Bivariate analysis revealed two significant associations: The “Hunt-and-Hess scale” (p-value = 0.016), and “Posthemorrhagic hydrocephalus” (p-value < 0.001). The all-relevant important factors during feature selection were: “Fisher scale,” “Modified Fisher scale,” “Hunt-and-Hess scale,” and “Posthemorrhagic hydrocephalus”. “Treatment type” was tentative. The random forests model achieved a pooled accuracy of 71.1% (95%CI: 60.4, 83.4) with an F1-score of 0.484. The best binary classifier utilized extreme gradient boosting while trained on the all-relevant predictors plus “Aneurysm type.” Extreme gradient boosting achieved a predictive accuracy of 84.3% (95%CI: 75.9, 93.4) with an F1-score of 0.684. We describe the challenges that arise during training of a binary classifier on imbalanced datasets, and, while going through an extensive comparison review of similar published studies, we not only demonstrate the model's performance but also identify multiple forecasting methodological fallacies in neurological research. Conclusion: By implementing baseline patient characteristics combined with radiographic grading scales, we built a simple yet robust, highly accurate—but, most importantly—useful binary classifier for DCI prediction. The model is available online, and it can be utilized clinically as an effective forecasting tool (https://georgiosalexopoulos.shinyapps.io/download/). |
| first_indexed | 2024-04-11T19:55:18Z |
| format | Article |
| id | doaj.art-d4f1039244c143a2b6281940cf6eec70 |
| institution | Directory Open Access Journal |
| issn | 2352-9148 |
| language | English |
| last_indexed | 2024-04-11T19:55:18Z |
| publishDate | 2022-01-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Informatics in Medicine Unlocked |
| spelling | doaj.art-d4f1039244c143a2b6281940cf6eec702022-12-22T04:06:12ZengElsevierInformatics in Medicine Unlocked2352-91482022-01-0128100817Applied forecasting for delayed cerebral ischemia prediction post subarachnoid hemorrhage: Methodological fallaciesGeorgios Alexopoulos0Justin Zhang1Ioannis Karampelas2Maheen Khan3Nabiha Quadri4Mayur Patel5Niel Patel6Mohammad Almajali7Tobias A. Mattei8Joanna Kemp9Jeroen Coppens10Philippe Mercier11Department of Neurosurgery, Saint Louis University Hospital, 1201 S. Grand Blvd., St. Louis, Missouri, 63104, USA; School of Medicine, Saint Louis University, 1402 S. Grand Blvd, St. Louis, Missouri, 63104, USA; Corresponding author. Department of Neurosurgery, Saint Louis University Hospital, 1201 S. Grand Blvd., St. Louis, Missouri, 63104, USA.School of Medicine, Saint Louis University, 1402 S. Grand Blvd, St. Louis, Missouri, 63104, USADepartment of Neurosurgery, Banner Neurological Surgery Clinic, 1800 15th St Ste 100B Grand, Greeley, CO, 80631, USADepartment of Neurosurgery, Saint Louis University Hospital, 1201 S. Grand Blvd., St. Louis, Missouri, 63104, USA; School of Medicine, Saint Louis University, 1402 S. Grand Blvd, St. Louis, Missouri, 63104, USADepartment of Neurosurgery, Saint Louis University Hospital, 1201 S. Grand Blvd., St. Louis, Missouri, 63104, USA; School of Medicine, Saint Louis University, 1402 S. Grand Blvd, St. Louis, Missouri, 63104, USASchool of Medicine, Saint Louis University, 1402 S. Grand Blvd, St. Louis, Missouri, 63104, USASchool of Medicine, Saint Louis University, 1402 S. Grand Blvd, St. Louis, Missouri, 63104, USADepartment of Neurology, Saint Louis University Hospital, 1438 S. Grand Blvd., St. Louis, Missouri, 63104, USADepartment of Neurosurgery, Saint Louis University Hospital, 1201 S. Grand Blvd., St. Louis, Missouri, 63104, USA; School of Medicine, Saint Louis University, 1402 S. Grand Blvd, St. Louis, Missouri, 63104, USADepartment of Neurosurgery, Saint Louis University Hospital, 1201 S. Grand Blvd., St. Louis, Missouri, 63104, USA; School of Medicine, Saint Louis University, 1402 S. Grand Blvd, St. Louis, Missouri, 63104, USADepartment of Neurosurgery, Saint Louis University Hospital, 1201 S. Grand Blvd., St. Louis, Missouri, 63104, USA; School of Medicine, Saint Louis University, 1402 S. Grand Blvd, St. Louis, Missouri, 63104, USADepartment of Neurosurgery, Saint Louis University Hospital, 1201 S. Grand Blvd., St. Louis, Missouri, 63104, USA; School of Medicine, Saint Louis University, 1402 S. Grand Blvd, St. Louis, Missouri, 63104, USAIntroduction: Delayed Cerebral Ischemia (DCI) is an important cause of morbidity and mortality after aneurysmal Subarachnoid Hemorrhage (aSAH). Researchers have utilized various methods for predicting patients at risk for DCI progression. Methods: An eight-year retrospective review of aSAH patients who presented to St Louis University Hospital. The records were screened for demographic, clinical, and radiographic parameters. DCI was the primary outcome. We identified 16 features to fit various forecasting models and selected the best binary classifier through comprehensive machine learning (ML) workflows. Regression and ensemble tree-based algorithms were utilized, based on their performance on tabular data. We investigated whether a single model could outperform in our dataset. Due to the expected outcome class imbalance (DCI), we selected precision, recall, and F-score as threshold metrics. Precision-recall curves were used for model performance ranking. Results: Of the 213 aSAH patients analyzed, 42 progressed to DCI (19.7%). The mean age was 55.7 years. The outcome variable (DCI) was imbalanced with a class ratio of 1:4. Bivariate analysis revealed two significant associations: The “Hunt-and-Hess scale” (p-value = 0.016), and “Posthemorrhagic hydrocephalus” (p-value < 0.001). The all-relevant important factors during feature selection were: “Fisher scale,” “Modified Fisher scale,” “Hunt-and-Hess scale,” and “Posthemorrhagic hydrocephalus”. “Treatment type” was tentative. The random forests model achieved a pooled accuracy of 71.1% (95%CI: 60.4, 83.4) with an F1-score of 0.484. The best binary classifier utilized extreme gradient boosting while trained on the all-relevant predictors plus “Aneurysm type.” Extreme gradient boosting achieved a predictive accuracy of 84.3% (95%CI: 75.9, 93.4) with an F1-score of 0.684. We describe the challenges that arise during training of a binary classifier on imbalanced datasets, and, while going through an extensive comparison review of similar published studies, we not only demonstrate the model's performance but also identify multiple forecasting methodological fallacies in neurological research. Conclusion: By implementing baseline patient characteristics combined with radiographic grading scales, we built a simple yet robust, highly accurate—but, most importantly—useful binary classifier for DCI prediction. The model is available online, and it can be utilized clinically as an effective forecasting tool (https://georgiosalexopoulos.shinyapps.io/download/).http://www.sciencedirect.com/science/article/pii/S2352914821002823Delayed cerebral ischemiaClinical vasospasmSubarachnoid hemorrhageApplied predictive modelingApplied forecastingImbalanced binary classification |
| spellingShingle | Georgios Alexopoulos Justin Zhang Ioannis Karampelas Maheen Khan Nabiha Quadri Mayur Patel Niel Patel Mohammad Almajali Tobias A. Mattei Joanna Kemp Jeroen Coppens Philippe Mercier Applied forecasting for delayed cerebral ischemia prediction post subarachnoid hemorrhage: Methodological fallacies Informatics in Medicine Unlocked Delayed cerebral ischemia Clinical vasospasm Subarachnoid hemorrhage Applied predictive modeling Applied forecasting Imbalanced binary classification |
| title | Applied forecasting for delayed cerebral ischemia prediction post subarachnoid hemorrhage: Methodological fallacies |
| title_full | Applied forecasting for delayed cerebral ischemia prediction post subarachnoid hemorrhage: Methodological fallacies |
| title_fullStr | Applied forecasting for delayed cerebral ischemia prediction post subarachnoid hemorrhage: Methodological fallacies |
| title_full_unstemmed | Applied forecasting for delayed cerebral ischemia prediction post subarachnoid hemorrhage: Methodological fallacies |
| title_short | Applied forecasting for delayed cerebral ischemia prediction post subarachnoid hemorrhage: Methodological fallacies |
| title_sort | applied forecasting for delayed cerebral ischemia prediction post subarachnoid hemorrhage methodological fallacies |
| topic | Delayed cerebral ischemia Clinical vasospasm Subarachnoid hemorrhage Applied predictive modeling Applied forecasting Imbalanced binary classification |
| url | http://www.sciencedirect.com/science/article/pii/S2352914821002823 |
| work_keys_str_mv | AT georgiosalexopoulos appliedforecastingfordelayedcerebralischemiapredictionpostsubarachnoidhemorrhagemethodologicalfallacies AT justinzhang appliedforecastingfordelayedcerebralischemiapredictionpostsubarachnoidhemorrhagemethodologicalfallacies AT ioanniskarampelas appliedforecastingfordelayedcerebralischemiapredictionpostsubarachnoidhemorrhagemethodologicalfallacies AT maheenkhan appliedforecastingfordelayedcerebralischemiapredictionpostsubarachnoidhemorrhagemethodologicalfallacies AT nabihaquadri appliedforecastingfordelayedcerebralischemiapredictionpostsubarachnoidhemorrhagemethodologicalfallacies AT mayurpatel appliedforecastingfordelayedcerebralischemiapredictionpostsubarachnoidhemorrhagemethodologicalfallacies AT nielpatel appliedforecastingfordelayedcerebralischemiapredictionpostsubarachnoidhemorrhagemethodologicalfallacies AT mohammadalmajali appliedforecastingfordelayedcerebralischemiapredictionpostsubarachnoidhemorrhagemethodologicalfallacies AT tobiasamattei appliedforecastingfordelayedcerebralischemiapredictionpostsubarachnoidhemorrhagemethodologicalfallacies AT joannakemp appliedforecastingfordelayedcerebralischemiapredictionpostsubarachnoidhemorrhagemethodologicalfallacies AT jeroencoppens appliedforecastingfordelayedcerebralischemiapredictionpostsubarachnoidhemorrhagemethodologicalfallacies AT philippemercier appliedforecastingfordelayedcerebralischemiapredictionpostsubarachnoidhemorrhagemethodologicalfallacies |