CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities
Sleep abnormalities can have severe health consequences. Automated sleep staging, i.e. labelling the sequence of sleep stages from the patient’s physiological recordings, could simplify the diagnostic process. Previous work on automated sleep staging has achieved great results, mainly rel...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Transactions on Neural Systems and Rehabilitation Engineering |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10400520/ |
| _version_ | 1797299808309870592 |
|---|---|
| author | Konstantinos Kontras Christos Chatzichristos Huy Phan Johan Suykens Maarten De Vos |
| author_facet | Konstantinos Kontras Christos Chatzichristos Huy Phan Johan Suykens Maarten De Vos |
| author_sort | Konstantinos Kontras |
| collection | DOAJ |
| description | Sleep abnormalities can have severe health consequences. Automated sleep staging, i.e. labelling the sequence of sleep stages from the patient’s physiological recordings, could simplify the diagnostic process. Previous work on automated sleep staging has achieved great results, mainly relying on the EEG signal. However, often multiple sources of information are available beyond EEG. This can be particularly beneficial when the EEG recordings are noisy or even missing completely. In this paper, we propose CoRe-Sleep, a Coordinated Representation multimodal fusion network that is particularly focused on improving the robustness of signal analysis on imperfect data. We demonstrate how appropriately handling multimodal information can be the key to achieving such robustness. CoRe-Sleep tolerates noisy or missing modalities segments, allowing training on incomplete data. Additionally, it shows state-of-the-art performance when testing on both multimodal and unimodal data using a single model on SHHS-1, the largest publicly available study that includes sleep stage labels. The results indicate that training the model on multimodal data does positively influence performance when tested on unimodal data. This work aims at bridging the gap between automated analysis tools and their clinical utility. |
| first_indexed | 2024-03-07T22:56:59Z |
| format | Article |
| id | doaj.art-2c5503d35c61408b9d19c82df55cad95 |
| institution | Directory Open Access Journal |
| issn | 1558-0210 |
| language | English |
| last_indexed | 2024-03-07T22:56:59Z |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Transactions on Neural Systems and Rehabilitation Engineering |
| spelling | doaj.art-2c5503d35c61408b9d19c82df55cad952024-02-23T00:00:08ZengIEEEIEEE Transactions on Neural Systems and Rehabilitation Engineering1558-02102024-01-013284084910.1109/TNSRE.2024.335438810400520CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect ModalitiesKonstantinos Kontras0https://orcid.org/0009-0004-4101-0720Christos Chatzichristos1Huy Phan2https://orcid.org/0000-0003-4096-785XJohan Suykens3https://orcid.org/0000-0002-8846-6352Maarten De Vos4https://orcid.org/0000-0002-3482-5145Department of Electrical Engineering (ESAT-Stadius), KU Leuven, Leuven, BelgiumDepartment of Electrical Engineering (ESAT-Stadius), KU Leuven, Leuven, BelgiumSchool of Electronic Engineering and Computer Science, Queen Mary University of London, London, U.K.Department of Electrical Engineering (ESAT-Stadius), KU Leuven, Leuven, BelgiumDepartment of Electrical Engineering (ESAT-Stadius) and the Department of Development and Regeneration, KU Leuven, Leuven, BelgiumSleep abnormalities can have severe health consequences. Automated sleep staging, i.e. labelling the sequence of sleep stages from the patient’s physiological recordings, could simplify the diagnostic process. Previous work on automated sleep staging has achieved great results, mainly relying on the EEG signal. However, often multiple sources of information are available beyond EEG. This can be particularly beneficial when the EEG recordings are noisy or even missing completely. In this paper, we propose CoRe-Sleep, a Coordinated Representation multimodal fusion network that is particularly focused on improving the robustness of signal analysis on imperfect data. We demonstrate how appropriately handling multimodal information can be the key to achieving such robustness. CoRe-Sleep tolerates noisy or missing modalities segments, allowing training on incomplete data. Additionally, it shows state-of-the-art performance when testing on both multimodal and unimodal data using a single model on SHHS-1, the largest publicly available study that includes sleep stage labels. The results indicate that training the model on multimodal data does positively influence performance when tested on unimodal data. This work aims at bridging the gap between automated analysis tools and their clinical utility.https://ieeexplore.ieee.org/document/10400520/Sleep stagingmultimodal fusionimperfect modalitiesincomplete data |
| spellingShingle | Konstantinos Kontras Christos Chatzichristos Huy Phan Johan Suykens Maarten De Vos CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities IEEE Transactions on Neural Systems and Rehabilitation Engineering Sleep staging multimodal fusion imperfect modalities incomplete data |
| title | CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities |
| title_full | CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities |
| title_fullStr | CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities |
| title_full_unstemmed | CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities |
| title_short | CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities |
| title_sort | core sleep a multimodal fusion framework for time series robust to imperfect modalities |
| topic | Sleep staging multimodal fusion imperfect modalities incomplete data |
| url | https://ieeexplore.ieee.org/document/10400520/ |
| work_keys_str_mv | AT konstantinoskontras coresleepamultimodalfusionframeworkfortimeseriesrobusttoimperfectmodalities AT christoschatzichristos coresleepamultimodalfusionframeworkfortimeseriesrobusttoimperfectmodalities AT huyphan coresleepamultimodalfusionframeworkfortimeseriesrobusttoimperfectmodalities AT johansuykens coresleepamultimodalfusionframeworkfortimeseriesrobusttoimperfectmodalities AT maartendevos coresleepamultimodalfusionframeworkfortimeseriesrobusttoimperfectmodalities |