On the Generalization of Sleep Apnea Detection Methods Based on Heart Rate Variability and Machine Learning
Obstructive sleep apnea (OSA) is a respiratory disorder highly correlated with severe cardiovascular diseases that has unleashed the interest of hundreds of experts aiming to overcome the elevated requirements of polysomnography, the gold standard for its detection. In this regard, a variety of algo...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9867980/ |
| _version_ | 1811211360957104128 |
|---|---|
| author | Daniele Padovano Arturo Martinez-Rodrigo Jose M. Pastor Jose J. Rieta Raul Alcaraz |
| author_facet | Daniele Padovano Arturo Martinez-Rodrigo Jose M. Pastor Jose J. Rieta Raul Alcaraz |
| author_sort | Daniele Padovano |
| collection | DOAJ |
| description | Obstructive sleep apnea (OSA) is a respiratory disorder highly correlated with severe cardiovascular diseases that has unleashed the interest of hundreds of experts aiming to overcome the elevated requirements of polysomnography, the gold standard for its detection. In this regard, a variety of algorithms based on heart rate variability (HRV) features and machine learning (ML) classifiers have been recently proposed for epoch-wise OSA detection from the surface electrocardiogram signal. Many researchers have employed freely available databases to assess their methods in a reproducible way, but most were purely tested with cross-validation approaches and even some using solely a single database for training and testing procedures. Hence, although promising values of diagnostic accuracy have been reported by some of these methods, they are suspected to be overestimated and the present work aims to analyze the actual generalization ability of several epoch-wise OSA detectors obtained through a common ML pipeline and typical HRV features. Precisely, the performance of the generated OSA detectors has been compared on two validation approaches, i.e., the widely used epoch-wise, <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>-fold cross-validation and the highly recommended external validation, both considering different combinations of well-known public databases. Regardless of the used ML classifiers and the selected HRV-based features, the external validation results have been 20 to 40% lower than those obtained with cross-validation in terms of accuracy, sensitivity, and specificity. Consequently, these results suggest that ML-based OSA detectors trained with public databases are still not sufficiently general to be employed in clinical practice, as well as that larger, more representative public datasets and the use of external validation are mandatory to improve the generalization ability and to obtain reliable assessment of the true predictive power of these algorithms, respectively. |
| first_indexed | 2024-04-12T05:11:49Z |
| format | Article |
| id | doaj.art-9bc4621ee21a41249f87e53b05ec5651 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-12T05:11:49Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-9bc4621ee21a41249f87e53b05ec56512022-12-22T03:46:44ZengIEEEIEEE Access2169-35362022-01-0110927109272510.1109/ACCESS.2022.32019119867980On the Generalization of Sleep Apnea Detection Methods Based on Heart Rate Variability and Machine LearningDaniele Padovano0https://orcid.org/0000-0003-3838-1438Arturo Martinez-Rodrigo1https://orcid.org/0000-0003-2343-3186Jose M. Pastor2https://orcid.org/0000-0001-5346-996XJose J. Rieta3https://orcid.org/0000-0002-3364-6380Raul Alcaraz4https://orcid.org/0000-0002-0942-3638Research Group in Electronic, Biomedical, and Telecommunication Engineering, Instituto de Tecnologías Audiovisuales, University of Castilla-La Mancha, Cuenca, SpainResearch Group in Electronic, Biomedical, and Telecommunication Engineering, Instituto de Tecnologías Audiovisuales, University of Castilla-La Mancha, Cuenca, SpainResearch Group in Electronic, Biomedical, and Telecommunication Engineering, Instituto de Tecnologías Audiovisuales, University of Castilla-La Mancha, Cuenca, SpainElectronic Engineering Department, BioMIT.org, Universitat Politecnica de Valencia, Valencia, SpainResearch Group in Electronic, Biomedical, and Telecommunication Engineering, Instituto de Tecnologías Audiovisuales, University of Castilla-La Mancha, Cuenca, SpainObstructive sleep apnea (OSA) is a respiratory disorder highly correlated with severe cardiovascular diseases that has unleashed the interest of hundreds of experts aiming to overcome the elevated requirements of polysomnography, the gold standard for its detection. In this regard, a variety of algorithms based on heart rate variability (HRV) features and machine learning (ML) classifiers have been recently proposed for epoch-wise OSA detection from the surface electrocardiogram signal. Many researchers have employed freely available databases to assess their methods in a reproducible way, but most were purely tested with cross-validation approaches and even some using solely a single database for training and testing procedures. Hence, although promising values of diagnostic accuracy have been reported by some of these methods, they are suspected to be overestimated and the present work aims to analyze the actual generalization ability of several epoch-wise OSA detectors obtained through a common ML pipeline and typical HRV features. Precisely, the performance of the generated OSA detectors has been compared on two validation approaches, i.e., the widely used epoch-wise, <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>-fold cross-validation and the highly recommended external validation, both considering different combinations of well-known public databases. Regardless of the used ML classifiers and the selected HRV-based features, the external validation results have been 20 to 40% lower than those obtained with cross-validation in terms of accuracy, sensitivity, and specificity. Consequently, these results suggest that ML-based OSA detectors trained with public databases are still not sufficiently general to be employed in clinical practice, as well as that larger, more representative public datasets and the use of external validation are mandatory to improve the generalization ability and to obtain reliable assessment of the true predictive power of these algorithms, respectively.https://ieeexplore.ieee.org/document/9867980/Electrocardiographyheart rate variabilitymachine learningsleep apnea |
| spellingShingle | Daniele Padovano Arturo Martinez-Rodrigo Jose M. Pastor Jose J. Rieta Raul Alcaraz On the Generalization of Sleep Apnea Detection Methods Based on Heart Rate Variability and Machine Learning IEEE Access Electrocardiography heart rate variability machine learning sleep apnea |
| title | On the Generalization of Sleep Apnea Detection Methods Based on Heart Rate Variability and Machine Learning |
| title_full | On the Generalization of Sleep Apnea Detection Methods Based on Heart Rate Variability and Machine Learning |
| title_fullStr | On the Generalization of Sleep Apnea Detection Methods Based on Heart Rate Variability and Machine Learning |
| title_full_unstemmed | On the Generalization of Sleep Apnea Detection Methods Based on Heart Rate Variability and Machine Learning |
| title_short | On the Generalization of Sleep Apnea Detection Methods Based on Heart Rate Variability and Machine Learning |
| title_sort | on the generalization of sleep apnea detection methods based on heart rate variability and machine learning |
| topic | Electrocardiography heart rate variability machine learning sleep apnea |
| url | https://ieeexplore.ieee.org/document/9867980/ |
| work_keys_str_mv | AT danielepadovano onthegeneralizationofsleepapneadetectionmethodsbasedonheartratevariabilityandmachinelearning AT arturomartinezrodrigo onthegeneralizationofsleepapneadetectionmethodsbasedonheartratevariabilityandmachinelearning AT josempastor onthegeneralizationofsleepapneadetectionmethodsbasedonheartratevariabilityandmachinelearning AT josejrieta onthegeneralizationofsleepapneadetectionmethodsbasedonheartratevariabilityandmachinelearning AT raulalcaraz onthegeneralizationofsleepapneadetectionmethodsbasedonheartratevariabilityandmachinelearning |