Multimodal fusion models for pulmonary embolism mortality prediction
Abstract Pulmonary embolism (PE) is a common, life threatening cardiovascular emergency. Risk stratification is one of the core principles of acute PE management and determines the choice of diagnostic and therapeutic strategies. In routine clinical practice, clinicians rely on the patient’s electro...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2023-05-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-023-34303-8 |
| _version_ | 1797827554950774784 |
|---|---|
| author | Noa Cahan Eyal Klang Edith M. Marom Shelly Soffer Yiftach Barash Evyatar Burshtein Eli Konen Hayit Greenspan |
| author_facet | Noa Cahan Eyal Klang Edith M. Marom Shelly Soffer Yiftach Barash Evyatar Burshtein Eli Konen Hayit Greenspan |
| author_sort | Noa Cahan |
| collection | DOAJ |
| description | Abstract Pulmonary embolism (PE) is a common, life threatening cardiovascular emergency. Risk stratification is one of the core principles of acute PE management and determines the choice of diagnostic and therapeutic strategies. In routine clinical practice, clinicians rely on the patient’s electronic health record (EHR) to provide a context for their medical imaging interpretation. Most deep learning models for radiology applications only consider pixel-value information without the clinical context. Only a few integrate both clinical and imaging data. In this work, we develop and compare multimodal fusion models that can utilize multimodal data by combining both volumetric pixel data and clinical patient data for automatic risk stratification of PE. Our best performing model is an intermediate fusion model that incorporates both bilinear attention and TabNet, and can be trained in an end-to-end manner. The results show that multimodality boosts performance by up to 14% with an area under the curve (AUC) of 0.96 for assessing PE severity, with a sensitivity of 90% and specificity of 94%, thus pointing to the value of using multimodal data to automatically assess PE severity. |
| first_indexed | 2024-04-09T12:50:12Z |
| format | Article |
| id | doaj.art-1c933443b58548fc8d0cbd5145d5895c |
| institution | Directory Open Access Journal |
| issn | 2045-2322 |
| language | English |
| last_indexed | 2024-04-09T12:50:12Z |
| publishDate | 2023-05-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj.art-1c933443b58548fc8d0cbd5145d5895c2023-05-14T11:14:17ZengNature PortfolioScientific Reports2045-23222023-05-0113111510.1038/s41598-023-34303-8Multimodal fusion models for pulmonary embolism mortality predictionNoa Cahan0Eyal Klang1Edith M. Marom2Shelly Soffer3Yiftach Barash4Evyatar Burshtein5Eli Konen6Hayit Greenspan7Department of Biomedical Engineering, Tel-Aviv UniversityDepartment of Diagnostic Imaging, Sheba Medical Center, Ramat Gan, Israel affiliated with the Tel Aviv UniversityDepartment of Diagnostic Imaging, Sheba Medical Center, Ramat Gan, Israel affiliated with the Tel Aviv UniversityDepartment of Diagnostic Imaging, Sheba Medical Center, Ramat Gan, Israel affiliated with the Tel Aviv UniversityDepartment of Diagnostic Imaging, Sheba Medical Center, Ramat Gan, Israel affiliated with the Tel Aviv UniversityDepartment of Biomedical Engineering, Tel-Aviv UniversityDepartment of Diagnostic Imaging, Sheba Medical Center, Ramat Gan, Israel affiliated with the Tel Aviv UniversityDepartment of Biomedical Engineering, Tel-Aviv UniversityAbstract Pulmonary embolism (PE) is a common, life threatening cardiovascular emergency. Risk stratification is one of the core principles of acute PE management and determines the choice of diagnostic and therapeutic strategies. In routine clinical practice, clinicians rely on the patient’s electronic health record (EHR) to provide a context for their medical imaging interpretation. Most deep learning models for radiology applications only consider pixel-value information without the clinical context. Only a few integrate both clinical and imaging data. In this work, we develop and compare multimodal fusion models that can utilize multimodal data by combining both volumetric pixel data and clinical patient data for automatic risk stratification of PE. Our best performing model is an intermediate fusion model that incorporates both bilinear attention and TabNet, and can be trained in an end-to-end manner. The results show that multimodality boosts performance by up to 14% with an area under the curve (AUC) of 0.96 for assessing PE severity, with a sensitivity of 90% and specificity of 94%, thus pointing to the value of using multimodal data to automatically assess PE severity.https://doi.org/10.1038/s41598-023-34303-8 |
| spellingShingle | Noa Cahan Eyal Klang Edith M. Marom Shelly Soffer Yiftach Barash Evyatar Burshtein Eli Konen Hayit Greenspan Multimodal fusion models for pulmonary embolism mortality prediction Scientific Reports |
| title | Multimodal fusion models for pulmonary embolism mortality prediction |
| title_full | Multimodal fusion models for pulmonary embolism mortality prediction |
| title_fullStr | Multimodal fusion models for pulmonary embolism mortality prediction |
| title_full_unstemmed | Multimodal fusion models for pulmonary embolism mortality prediction |
| title_short | Multimodal fusion models for pulmonary embolism mortality prediction |
| title_sort | multimodal fusion models for pulmonary embolism mortality prediction |
| url | https://doi.org/10.1038/s41598-023-34303-8 |
| work_keys_str_mv | AT noacahan multimodalfusionmodelsforpulmonaryembolismmortalityprediction AT eyalklang multimodalfusionmodelsforpulmonaryembolismmortalityprediction AT edithmmarom multimodalfusionmodelsforpulmonaryembolismmortalityprediction AT shellysoffer multimodalfusionmodelsforpulmonaryembolismmortalityprediction AT yiftachbarash multimodalfusionmodelsforpulmonaryembolismmortalityprediction AT evyatarburshtein multimodalfusionmodelsforpulmonaryembolismmortalityprediction AT elikonen multimodalfusionmodelsforpulmonaryembolismmortalityprediction AT hayitgreenspan multimodalfusionmodelsforpulmonaryembolismmortalityprediction |