Comparison of discrimination and calibration performance of ECG-based machine learning models for prediction of new-onset atrial fibrillation
Abstract Background Machine learning (ML) methods to build prediction models starting from electrocardiogram (ECG) signals are an emerging research field. The aim of the present study is to investigate the performances of two ML approaches based on ECGs for the prediction of new-onset atrial fibrill...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-07-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-023-01989-3 |
| _version_ | 1797774114312683520 |
|---|---|
| author | Giovanni Baj Ilaria Gandin Arjuna Scagnetto Luca Bortolussi Chiara Cappelletto Andrea Di Lenarda Giulia Barbati |
| author_facet | Giovanni Baj Ilaria Gandin Arjuna Scagnetto Luca Bortolussi Chiara Cappelletto Andrea Di Lenarda Giulia Barbati |
| author_sort | Giovanni Baj |
| collection | DOAJ |
| description | Abstract Background Machine learning (ML) methods to build prediction models starting from electrocardiogram (ECG) signals are an emerging research field. The aim of the present study is to investigate the performances of two ML approaches based on ECGs for the prediction of new-onset atrial fibrillation (AF), in terms of discrimination, calibration and sample size dependence. Methods We trained two models to predict new-onset AF: a convolutional neural network (CNN), that takes as input the raw ECG signals, and an eXtreme Gradient Boosting model (XGB), that uses the signal’s extracted features. A penalized logistic regression model (LR) was used as a benchmark. Discrimination was evaluated with the area under the ROC curve, while calibration with the integrated calibration index. We investigated the dependence of models’ performances on the sample size and on class imbalance corrections introduced with random under-sampling. Results CNN's discrimination was the most affected by the sample size, outperforming XGB and LR only around n = 10.000 observations. Calibration showed only a small dependence on the sample size for all the models considered. Balancing the training set with random undersampling did not improve discrimination in any of the models. Instead, the main effect of imbalance corrections was to worsen the models’ calibration (for CNN, integrated calibration index from 0.014 [0.01, 0.018] to 0.17 [0.16, 0.19]). The sample size emerged as a fundamental point for developing the CNN model, especially in terms of discrimination (AUC = 0.75 [0.73, 0.77] when n = 10.000, AUC = 0.80 [0.79, 0.81] when n = 150.000). The effect of the sample size on the other two models was weaker. Imbalance corrections led to poorly calibrated models, for all the approaches considered, reducing the clinical utility of the models. Conclusions Our results suggest that the choice of approach in the analysis of ECG should be based on the amount of data available, preferring more standard models for small datasets. Moreover, imbalance correction methods should be avoided when developing clinical prediction models, where calibration is crucial. |
| first_indexed | 2024-03-12T22:16:18Z |
| format | Article |
| id | doaj.art-d7277f21e217425aae381c97b0abf166 |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-03-12T22:16:18Z |
| publishDate | 2023-07-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-d7277f21e217425aae381c97b0abf1662023-07-23T11:17:51ZengBMCBMC Medical Research Methodology1471-22882023-07-0123111010.1186/s12874-023-01989-3Comparison of discrimination and calibration performance of ECG-based machine learning models for prediction of new-onset atrial fibrillationGiovanni Baj0Ilaria Gandin1Arjuna Scagnetto2Luca Bortolussi3Chiara Cappelletto4Andrea Di Lenarda5Giulia Barbati6Department of Mathematics and Geosciences, University of TriesteDepartment of Medical Sciences, Biostatistics Unit, University of TriesteCardiovascular Center, Territorial Specialistic Department, University Hospital and Health Services of TriesteDepartment of Mathematics and Geosciences, University of TriesteCardiovascular Center, Territorial Specialistic Department, University Hospital and Health Services of TriesteCardiovascular Center, Territorial Specialistic Department, University Hospital and Health Services of TriesteDepartment of Medical Sciences, Biostatistics Unit, University of TriesteAbstract Background Machine learning (ML) methods to build prediction models starting from electrocardiogram (ECG) signals are an emerging research field. The aim of the present study is to investigate the performances of two ML approaches based on ECGs for the prediction of new-onset atrial fibrillation (AF), in terms of discrimination, calibration and sample size dependence. Methods We trained two models to predict new-onset AF: a convolutional neural network (CNN), that takes as input the raw ECG signals, and an eXtreme Gradient Boosting model (XGB), that uses the signal’s extracted features. A penalized logistic regression model (LR) was used as a benchmark. Discrimination was evaluated with the area under the ROC curve, while calibration with the integrated calibration index. We investigated the dependence of models’ performances on the sample size and on class imbalance corrections introduced with random under-sampling. Results CNN's discrimination was the most affected by the sample size, outperforming XGB and LR only around n = 10.000 observations. Calibration showed only a small dependence on the sample size for all the models considered. Balancing the training set with random undersampling did not improve discrimination in any of the models. Instead, the main effect of imbalance corrections was to worsen the models’ calibration (for CNN, integrated calibration index from 0.014 [0.01, 0.018] to 0.17 [0.16, 0.19]). The sample size emerged as a fundamental point for developing the CNN model, especially in terms of discrimination (AUC = 0.75 [0.73, 0.77] when n = 10.000, AUC = 0.80 [0.79, 0.81] when n = 150.000). The effect of the sample size on the other two models was weaker. Imbalance corrections led to poorly calibrated models, for all the approaches considered, reducing the clinical utility of the models. Conclusions Our results suggest that the choice of approach in the analysis of ECG should be based on the amount of data available, preferring more standard models for small datasets. Moreover, imbalance correction methods should be avoided when developing clinical prediction models, where calibration is crucial.https://doi.org/10.1186/s12874-023-01989-3Atrial fibrillationPredictionCalibrationMachine learningDeep learning |
| spellingShingle | Giovanni Baj Ilaria Gandin Arjuna Scagnetto Luca Bortolussi Chiara Cappelletto Andrea Di Lenarda Giulia Barbati Comparison of discrimination and calibration performance of ECG-based machine learning models for prediction of new-onset atrial fibrillation BMC Medical Research Methodology Atrial fibrillation Prediction Calibration Machine learning Deep learning |
| title | Comparison of discrimination and calibration performance of ECG-based machine learning models for prediction of new-onset atrial fibrillation |
| title_full | Comparison of discrimination and calibration performance of ECG-based machine learning models for prediction of new-onset atrial fibrillation |
| title_fullStr | Comparison of discrimination and calibration performance of ECG-based machine learning models for prediction of new-onset atrial fibrillation |
| title_full_unstemmed | Comparison of discrimination and calibration performance of ECG-based machine learning models for prediction of new-onset atrial fibrillation |
| title_short | Comparison of discrimination and calibration performance of ECG-based machine learning models for prediction of new-onset atrial fibrillation |
| title_sort | comparison of discrimination and calibration performance of ecg based machine learning models for prediction of new onset atrial fibrillation |
| topic | Atrial fibrillation Prediction Calibration Machine learning Deep learning |
| url | https://doi.org/10.1186/s12874-023-01989-3 |
| work_keys_str_mv | AT giovannibaj comparisonofdiscriminationandcalibrationperformanceofecgbasedmachinelearningmodelsforpredictionofnewonsetatrialfibrillation AT ilariagandin comparisonofdiscriminationandcalibrationperformanceofecgbasedmachinelearningmodelsforpredictionofnewonsetatrialfibrillation AT arjunascagnetto comparisonofdiscriminationandcalibrationperformanceofecgbasedmachinelearningmodelsforpredictionofnewonsetatrialfibrillation AT lucabortolussi comparisonofdiscriminationandcalibrationperformanceofecgbasedmachinelearningmodelsforpredictionofnewonsetatrialfibrillation AT chiaracappelletto comparisonofdiscriminationandcalibrationperformanceofecgbasedmachinelearningmodelsforpredictionofnewonsetatrialfibrillation AT andreadilenarda comparisonofdiscriminationandcalibrationperformanceofecgbasedmachinelearningmodelsforpredictionofnewonsetatrialfibrillation AT giuliabarbati comparisonofdiscriminationandcalibrationperformanceofecgbasedmachinelearningmodelsforpredictionofnewonsetatrialfibrillation |