Evaluating methods for risk prediction of Covid-19 mortality in nursing home residents before and after vaccine availability: a retrospective cohort study
Abstract Background SARS-CoV-2 vaccines are effective in reducing hospitalization, COVID-19 symptoms, and COVID-19 mortality for nursing home (NH) residents. We sought to compare the accuracy of various machine learning models, examine changes to model performance, and identify resident characterist...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2024-03-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-024-02189-3 |
| _version_ | 1827301136091054080 |
|---|---|
| author | Komal Aryal Fabrice I. Mowbray Anna Miroshnychenko Ryan P. Strum Darly Dash Michael P. Hillmer Kamil Malikov Andrew P. Costa Aaron Jones |
| author_facet | Komal Aryal Fabrice I. Mowbray Anna Miroshnychenko Ryan P. Strum Darly Dash Michael P. Hillmer Kamil Malikov Andrew P. Costa Aaron Jones |
| author_sort | Komal Aryal |
| collection | DOAJ |
| description | Abstract Background SARS-CoV-2 vaccines are effective in reducing hospitalization, COVID-19 symptoms, and COVID-19 mortality for nursing home (NH) residents. We sought to compare the accuracy of various machine learning models, examine changes to model performance, and identify resident characteristics that have the strongest associations with 30-day COVID-19 mortality, before and after vaccine availability. Methods We conducted a population-based retrospective cohort study analyzing data from all NH facilities across Ontario, Canada. We included all residents diagnosed with SARS-CoV-2 and living in NHs between March 2020 and July 2021. We employed five machine learning algorithms to predict COVID-19 mortality, including logistic regression, LASSO regression, classification and regression trees (CART), random forests, and gradient boosted trees. The discriminative performance of the models was evaluated using the area under the receiver operating characteristic curve (AUC) for each model using 10-fold cross-validation. Model calibration was determined through evaluation of calibration slopes. Variable importance was calculated by repeatedly and randomly permutating the values of each predictor in the dataset and re-evaluating the model’s performance. Results A total of 14,977 NH residents and 20 resident characteristics were included in the model. The cross-validated AUCs were similar across algorithms and ranged from 0.64 to 0.67. Gradient boosted trees and logistic regression had an AUC of 0.67 pre- and post-vaccine availability. CART had the lowest discrimination ability with an AUC of 0.64 pre-vaccine availability, and 0.65 post-vaccine availability. The most influential resident characteristics, irrespective of vaccine availability, included advanced age (≥ 75 years), health instability, functional and cognitive status, sex (male), and polypharmacy. Conclusions The predictive accuracy and discrimination exhibited by all five examined machine learning algorithms were similar. Both logistic regression and gradient boosted trees exhibit comparable performance and display slight superiority over other machine learning algorithms. We observed consistent model performance both before and after vaccine availability. The influence of resident characteristics on COVID-19 mortality remained consistent across time periods, suggesting that changes to pre-vaccination screening practices for high-risk individuals are effective in the post-vaccination era. |
| first_indexed | 2024-04-24T16:16:52Z |
| format | Article |
| id | doaj.art-b3042592d63d4b3e815897ef556bd313 |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-04-24T16:16:52Z |
| publishDate | 2024-03-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-b3042592d63d4b3e815897ef556bd3132024-03-31T11:24:07ZengBMCBMC Medical Research Methodology1471-22882024-03-012411810.1186/s12874-024-02189-3Evaluating methods for risk prediction of Covid-19 mortality in nursing home residents before and after vaccine availability: a retrospective cohort studyKomal Aryal0Fabrice I. Mowbray1Anna Miroshnychenko2Ryan P. Strum3Darly Dash4Michael P. Hillmer5Kamil Malikov6Andrew P. Costa7Aaron Jones8Department of Health Research Methods, Evidence, and Impact, McMaster UniversityCollege of Nursing, Michigan State UniversityDepartment of Health Research Methods, Evidence, and Impact, McMaster UniversityDepartment of Health Research Methods, Evidence, and Impact, McMaster UniversityDepartment of Health Research Methods, Evidence, and Impact, McMaster UniversityInstitute of Health Policy, Management and Evaluation, University of TorontoCapacity Planning and Analytics, Ontario Ministry of HealthDepartment of Health Research Methods, Evidence, and Impact, McMaster UniversityDepartment of Health Research Methods, Evidence, and Impact, McMaster UniversityAbstract Background SARS-CoV-2 vaccines are effective in reducing hospitalization, COVID-19 symptoms, and COVID-19 mortality for nursing home (NH) residents. We sought to compare the accuracy of various machine learning models, examine changes to model performance, and identify resident characteristics that have the strongest associations with 30-day COVID-19 mortality, before and after vaccine availability. Methods We conducted a population-based retrospective cohort study analyzing data from all NH facilities across Ontario, Canada. We included all residents diagnosed with SARS-CoV-2 and living in NHs between March 2020 and July 2021. We employed five machine learning algorithms to predict COVID-19 mortality, including logistic regression, LASSO regression, classification and regression trees (CART), random forests, and gradient boosted trees. The discriminative performance of the models was evaluated using the area under the receiver operating characteristic curve (AUC) for each model using 10-fold cross-validation. Model calibration was determined through evaluation of calibration slopes. Variable importance was calculated by repeatedly and randomly permutating the values of each predictor in the dataset and re-evaluating the model’s performance. Results A total of 14,977 NH residents and 20 resident characteristics were included in the model. The cross-validated AUCs were similar across algorithms and ranged from 0.64 to 0.67. Gradient boosted trees and logistic regression had an AUC of 0.67 pre- and post-vaccine availability. CART had the lowest discrimination ability with an AUC of 0.64 pre-vaccine availability, and 0.65 post-vaccine availability. The most influential resident characteristics, irrespective of vaccine availability, included advanced age (≥ 75 years), health instability, functional and cognitive status, sex (male), and polypharmacy. Conclusions The predictive accuracy and discrimination exhibited by all five examined machine learning algorithms were similar. Both logistic regression and gradient boosted trees exhibit comparable performance and display slight superiority over other machine learning algorithms. We observed consistent model performance both before and after vaccine availability. The influence of resident characteristics on COVID-19 mortality remained consistent across time periods, suggesting that changes to pre-vaccination screening practices for high-risk individuals are effective in the post-vaccination era.https://doi.org/10.1186/s12874-024-02189-3Long-term careNursing homeOlder adultsCOVID-19Prediction modelingCohort study |
| spellingShingle | Komal Aryal Fabrice I. Mowbray Anna Miroshnychenko Ryan P. Strum Darly Dash Michael P. Hillmer Kamil Malikov Andrew P. Costa Aaron Jones Evaluating methods for risk prediction of Covid-19 mortality in nursing home residents before and after vaccine availability: a retrospective cohort study BMC Medical Research Methodology Long-term care Nursing home Older adults COVID-19 Prediction modeling Cohort study |
| title | Evaluating methods for risk prediction of Covid-19 mortality in nursing home residents before and after vaccine availability: a retrospective cohort study |
| title_full | Evaluating methods for risk prediction of Covid-19 mortality in nursing home residents before and after vaccine availability: a retrospective cohort study |
| title_fullStr | Evaluating methods for risk prediction of Covid-19 mortality in nursing home residents before and after vaccine availability: a retrospective cohort study |
| title_full_unstemmed | Evaluating methods for risk prediction of Covid-19 mortality in nursing home residents before and after vaccine availability: a retrospective cohort study |
| title_short | Evaluating methods for risk prediction of Covid-19 mortality in nursing home residents before and after vaccine availability: a retrospective cohort study |
| title_sort | evaluating methods for risk prediction of covid 19 mortality in nursing home residents before and after vaccine availability a retrospective cohort study |
| topic | Long-term care Nursing home Older adults COVID-19 Prediction modeling Cohort study |
| url | https://doi.org/10.1186/s12874-024-02189-3 |
| work_keys_str_mv | AT komalaryal evaluatingmethodsforriskpredictionofcovid19mortalityinnursinghomeresidentsbeforeandaftervaccineavailabilityaretrospectivecohortstudy AT fabriceimowbray evaluatingmethodsforriskpredictionofcovid19mortalityinnursinghomeresidentsbeforeandaftervaccineavailabilityaretrospectivecohortstudy AT annamiroshnychenko evaluatingmethodsforriskpredictionofcovid19mortalityinnursinghomeresidentsbeforeandaftervaccineavailabilityaretrospectivecohortstudy AT ryanpstrum evaluatingmethodsforriskpredictionofcovid19mortalityinnursinghomeresidentsbeforeandaftervaccineavailabilityaretrospectivecohortstudy AT darlydash evaluatingmethodsforriskpredictionofcovid19mortalityinnursinghomeresidentsbeforeandaftervaccineavailabilityaretrospectivecohortstudy AT michaelphillmer evaluatingmethodsforriskpredictionofcovid19mortalityinnursinghomeresidentsbeforeandaftervaccineavailabilityaretrospectivecohortstudy AT kamilmalikov evaluatingmethodsforriskpredictionofcovid19mortalityinnursinghomeresidentsbeforeandaftervaccineavailabilityaretrospectivecohortstudy AT andrewpcosta evaluatingmethodsforriskpredictionofcovid19mortalityinnursinghomeresidentsbeforeandaftervaccineavailabilityaretrospectivecohortstudy AT aaronjones evaluatingmethodsforriskpredictionofcovid19mortalityinnursinghomeresidentsbeforeandaftervaccineavailabilityaretrospectivecohortstudy |