Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data
Abstract When clinicians assess the prognosis of patients in intensive care, they take imaging and non-imaging data into account. In contrast, many traditional machine learning models rely on only one of these modalities, limiting their potential in medical applications. This work proposes and evalu...
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2023-07-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-023-37835-1 |
| _version_ | 1827910757476466688 |
|---|---|
| author | Firas Khader Jakob Nikolas Kather Gustav Müller-Franzes Tianci Wang Tianyu Han Soroosh Tayebi Arasteh Karim Hamesch Keno Bressem Christoph Haarburger Johannes Stegmaier Christiane Kuhl Sven Nebelung Daniel Truhn |
| author_facet | Firas Khader Jakob Nikolas Kather Gustav Müller-Franzes Tianci Wang Tianyu Han Soroosh Tayebi Arasteh Karim Hamesch Keno Bressem Christoph Haarburger Johannes Stegmaier Christiane Kuhl Sven Nebelung Daniel Truhn |
| author_sort | Firas Khader |
| collection | DOAJ |
| description | Abstract When clinicians assess the prognosis of patients in intensive care, they take imaging and non-imaging data into account. In contrast, many traditional machine learning models rely on only one of these modalities, limiting their potential in medical applications. This work proposes and evaluates a transformer-based neural network as a novel AI architecture that integrates multimodal patient data, i.e., imaging data (chest radiographs) and non-imaging data (clinical data). We evaluate the performance of our model in a retrospective study with 6,125 patients in intensive care. We show that the combined model (area under the receiver operating characteristic curve [AUROC] of 0.863) is superior to the radiographs-only model (AUROC = 0.811, p < 0.001) and the clinical data-only model (AUROC = 0.785, p < 0.001) when tasked with predicting in-hospital survival per patient. Furthermore, we demonstrate that our proposed model is robust in cases where not all (clinical) data points are available. |
| first_indexed | 2024-03-13T01:55:07Z |
| format | Article |
| id | doaj.art-137023c7cd0440f98a47882e0dac9407 |
| institution | Directory Open Access Journal |
| issn | 2045-2322 |
| language | English |
| last_indexed | 2024-03-13T01:55:07Z |
| publishDate | 2023-07-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj.art-137023c7cd0440f98a47882e0dac94072023-07-02T11:16:06ZengNature PortfolioScientific Reports2045-23222023-07-0113111110.1038/s41598-023-37835-1Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging dataFiras Khader0Jakob Nikolas Kather1Gustav Müller-Franzes2Tianci Wang3Tianyu Han4Soroosh Tayebi Arasteh5Karim Hamesch6Keno Bressem7Christoph Haarburger8Johannes Stegmaier9Christiane Kuhl10Sven Nebelung11Daniel Truhn12Department of Diagnostic and Interventional Radiology, University Hospital AachenDepartment of Medicine III, University Hospital AachenDepartment of Diagnostic and Interventional Radiology, University Hospital AachenDepartment of Diagnostic and Interventional Radiology, University Hospital AachenPhysics of Molecular Imaging Systems, Experimental Molecular Imaging, RWTH Aachen UniversityDepartment of Diagnostic and Interventional Radiology, University Hospital AachenDepartment of Medicine III, University Hospital AachenDepartment of Radiology, Charité-University Medicine BerlinOcumeda GmbHInstitute of Imaging and Computer Vision, RWTH Aachen UniversityDepartment of Diagnostic and Interventional Radiology, University Hospital AachenDepartment of Diagnostic and Interventional Radiology, University Hospital AachenDepartment of Diagnostic and Interventional Radiology, University Hospital AachenAbstract When clinicians assess the prognosis of patients in intensive care, they take imaging and non-imaging data into account. In contrast, many traditional machine learning models rely on only one of these modalities, limiting their potential in medical applications. This work proposes and evaluates a transformer-based neural network as a novel AI architecture that integrates multimodal patient data, i.e., imaging data (chest radiographs) and non-imaging data (clinical data). We evaluate the performance of our model in a retrospective study with 6,125 patients in intensive care. We show that the combined model (area under the receiver operating characteristic curve [AUROC] of 0.863) is superior to the radiographs-only model (AUROC = 0.811, p < 0.001) and the clinical data-only model (AUROC = 0.785, p < 0.001) when tasked with predicting in-hospital survival per patient. Furthermore, we demonstrate that our proposed model is robust in cases where not all (clinical) data points are available.https://doi.org/10.1038/s41598-023-37835-1 |
| spellingShingle | Firas Khader Jakob Nikolas Kather Gustav Müller-Franzes Tianci Wang Tianyu Han Soroosh Tayebi Arasteh Karim Hamesch Keno Bressem Christoph Haarburger Johannes Stegmaier Christiane Kuhl Sven Nebelung Daniel Truhn Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data Scientific Reports |
| title | Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data |
| title_full | Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data |
| title_fullStr | Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data |
| title_full_unstemmed | Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data |
| title_short | Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data |
| title_sort | medical transformer for multimodal survival prediction in intensive care integration of imaging and non imaging data |
| url | https://doi.org/10.1038/s41598-023-37835-1 |
| work_keys_str_mv | AT firaskhader medicaltransformerformultimodalsurvivalpredictioninintensivecareintegrationofimagingandnonimagingdata AT jakobnikolaskather medicaltransformerformultimodalsurvivalpredictioninintensivecareintegrationofimagingandnonimagingdata AT gustavmullerfranzes medicaltransformerformultimodalsurvivalpredictioninintensivecareintegrationofimagingandnonimagingdata AT tianciwang medicaltransformerformultimodalsurvivalpredictioninintensivecareintegrationofimagingandnonimagingdata AT tianyuhan medicaltransformerformultimodalsurvivalpredictioninintensivecareintegrationofimagingandnonimagingdata AT sorooshtayebiarasteh medicaltransformerformultimodalsurvivalpredictioninintensivecareintegrationofimagingandnonimagingdata AT karimhamesch medicaltransformerformultimodalsurvivalpredictioninintensivecareintegrationofimagingandnonimagingdata AT kenobressem medicaltransformerformultimodalsurvivalpredictioninintensivecareintegrationofimagingandnonimagingdata AT christophhaarburger medicaltransformerformultimodalsurvivalpredictioninintensivecareintegrationofimagingandnonimagingdata AT johannesstegmaier medicaltransformerformultimodalsurvivalpredictioninintensivecareintegrationofimagingandnonimagingdata AT christianekuhl medicaltransformerformultimodalsurvivalpredictioninintensivecareintegrationofimagingandnonimagingdata AT svennebelung medicaltransformerformultimodalsurvivalpredictioninintensivecareintegrationofimagingandnonimagingdata AT danieltruhn medicaltransformerformultimodalsurvivalpredictioninintensivecareintegrationofimagingandnonimagingdata |