Evaluating the Performance of Automated Machine Learning (AutoML) Tools for Heart Disease Diagnosis and Prediction
Globally, over 17 million people annually die from cardiovascular diseases, with heart disease being the leading cause of mortality in the United States. The ever-increasing volume of data related to heart disease opens up possibilities for employing machine learning (ML) techniques in diagnosing an...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-12-01
|
| Series: | AI |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2673-2688/4/4/53 |
| _version_ | 1797382332763602944 |
|---|---|
| author | Lauren M. Paladino Alexander Hughes Alexander Perera Oguzhan Topsakal Tahir Cetin Akinci |
| author_facet | Lauren M. Paladino Alexander Hughes Alexander Perera Oguzhan Topsakal Tahir Cetin Akinci |
| author_sort | Lauren M. Paladino |
| collection | DOAJ |
| description | Globally, over 17 million people annually die from cardiovascular diseases, with heart disease being the leading cause of mortality in the United States. The ever-increasing volume of data related to heart disease opens up possibilities for employing machine learning (ML) techniques in diagnosing and predicting heart conditions. While applying ML demands a certain level of computer science expertise—often a barrier for healthcare professionals—automated machine learning (AutoML) tools significantly lower this barrier. They enable users to construct the most effective ML models without in-depth technical knowledge. Despite their potential, there has been a lack of research comparing the performance of different AutoML tools on heart disease data. Addressing this gap, our study evaluates three AutoML tools—PyCaret, AutoGluon, and AutoKeras—against three datasets (Cleveland, Hungarian, and a combined dataset). To evaluate the efficacy of AutoML against conventional machine learning methodologies, we crafted ten machine learning models using the standard practices of exploratory data analysis (EDA), data cleansing, feature engineering, and others, utilizing the sklearn library. Our toolkit included an array of models—logistic regression, support vector machines, decision trees, random forest, and various ensemble models. Employing 5-fold cross-validation, these traditionally developed models demonstrated accuracy rates spanning from 55% to 60%. This performance is markedly inferior to that of AutoML tools, indicating the latter’s superior capability in generating predictive models. Among AutoML tools, AutoGluon emerged as the superior tool, consistently achieving accuracy rates between 78% and 86% across the datasets. PyCaret’s performance varied, with accuracy rates from 65% to 83%, indicating a dependency on the nature of the dataset. AutoKeras showed the most fluctuation in performance, with accuracies ranging from 54% to 83%. Our findings suggest that AutoML tools can simplify the generation of robust ML models that potentially surpass those crafted through traditional ML methodologies. However, we must also consider the limitations of AutoML tools and explore strategies to overcome them. The successful deployment of high-performance ML models designed via AutoML could revolutionize the treatment and prevention of heart disease globally, significantly impacting patient care. |
| first_indexed | 2024-03-08T21:03:54Z |
| format | Article |
| id | doaj.art-bade48037ecd4e06844ce81b30651540 |
| institution | Directory Open Access Journal |
| issn | 2673-2688 |
| language | English |
| last_indexed | 2024-03-08T21:03:54Z |
| publishDate | 2023-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | AI |
| spelling | doaj.art-bade48037ecd4e06844ce81b306515402023-12-22T13:46:58ZengMDPI AGAI2673-26882023-12-01441036105810.3390/ai4040053Evaluating the Performance of Automated Machine Learning (AutoML) Tools for Heart Disease Diagnosis and PredictionLauren M. Paladino0Alexander Hughes1Alexander Perera2Oguzhan Topsakal3Tahir Cetin Akinci4Department of Computer Science, Florida Polytechnic University, Lakeland, FL 33805, USADepartment of Computer Science, Florida Polytechnic University, Lakeland, FL 33805, USADepartment of Computer Science, Florida Polytechnic University, Lakeland, FL 33805, USADepartment of Computer Science, Florida Polytechnic University, Lakeland, FL 33805, USAWinston Chung Global Energy Center (WCGEC), University of California at Riverside (UCR), Riverside, CA 92521, USAGlobally, over 17 million people annually die from cardiovascular diseases, with heart disease being the leading cause of mortality in the United States. The ever-increasing volume of data related to heart disease opens up possibilities for employing machine learning (ML) techniques in diagnosing and predicting heart conditions. While applying ML demands a certain level of computer science expertise—often a barrier for healthcare professionals—automated machine learning (AutoML) tools significantly lower this barrier. They enable users to construct the most effective ML models without in-depth technical knowledge. Despite their potential, there has been a lack of research comparing the performance of different AutoML tools on heart disease data. Addressing this gap, our study evaluates three AutoML tools—PyCaret, AutoGluon, and AutoKeras—against three datasets (Cleveland, Hungarian, and a combined dataset). To evaluate the efficacy of AutoML against conventional machine learning methodologies, we crafted ten machine learning models using the standard practices of exploratory data analysis (EDA), data cleansing, feature engineering, and others, utilizing the sklearn library. Our toolkit included an array of models—logistic regression, support vector machines, decision trees, random forest, and various ensemble models. Employing 5-fold cross-validation, these traditionally developed models demonstrated accuracy rates spanning from 55% to 60%. This performance is markedly inferior to that of AutoML tools, indicating the latter’s superior capability in generating predictive models. Among AutoML tools, AutoGluon emerged as the superior tool, consistently achieving accuracy rates between 78% and 86% across the datasets. PyCaret’s performance varied, with accuracy rates from 65% to 83%, indicating a dependency on the nature of the dataset. AutoKeras showed the most fluctuation in performance, with accuracies ranging from 54% to 83%. Our findings suggest that AutoML tools can simplify the generation of robust ML models that potentially surpass those crafted through traditional ML methodologies. However, we must also consider the limitations of AutoML tools and explore strategies to overcome them. The successful deployment of high-performance ML models designed via AutoML could revolutionize the treatment and prevention of heart disease globally, significantly impacting patient care.https://www.mdpi.com/2673-2688/4/4/53AutoMLmachine learningcardiovascular diseasecoronary artery diseasediagnosisheart disease |
| spellingShingle | Lauren M. Paladino Alexander Hughes Alexander Perera Oguzhan Topsakal Tahir Cetin Akinci Evaluating the Performance of Automated Machine Learning (AutoML) Tools for Heart Disease Diagnosis and Prediction AI AutoML machine learning cardiovascular disease coronary artery disease diagnosis heart disease |
| title | Evaluating the Performance of Automated Machine Learning (AutoML) Tools for Heart Disease Diagnosis and Prediction |
| title_full | Evaluating the Performance of Automated Machine Learning (AutoML) Tools for Heart Disease Diagnosis and Prediction |
| title_fullStr | Evaluating the Performance of Automated Machine Learning (AutoML) Tools for Heart Disease Diagnosis and Prediction |
| title_full_unstemmed | Evaluating the Performance of Automated Machine Learning (AutoML) Tools for Heart Disease Diagnosis and Prediction |
| title_short | Evaluating the Performance of Automated Machine Learning (AutoML) Tools for Heart Disease Diagnosis and Prediction |
| title_sort | evaluating the performance of automated machine learning automl tools for heart disease diagnosis and prediction |
| topic | AutoML machine learning cardiovascular disease coronary artery disease diagnosis heart disease |
| url | https://www.mdpi.com/2673-2688/4/4/53 |
| work_keys_str_mv | AT laurenmpaladino evaluatingtheperformanceofautomatedmachinelearningautomltoolsforheartdiseasediagnosisandprediction AT alexanderhughes evaluatingtheperformanceofautomatedmachinelearningautomltoolsforheartdiseasediagnosisandprediction AT alexanderperera evaluatingtheperformanceofautomatedmachinelearningautomltoolsforheartdiseasediagnosisandprediction AT oguzhantopsakal evaluatingtheperformanceofautomatedmachinelearningautomltoolsforheartdiseasediagnosisandprediction AT tahircetinakinci evaluatingtheperformanceofautomatedmachinelearningautomltoolsforheartdiseasediagnosisandprediction |