Comparison of State-of-the-Art Neural Network Survival Models with the Pooled Cohort Equations for Cardiovascular Disease Risk Prediction
Abstract Background The Pooled Cohort Equations (PCEs) are race- and sex-specific Cox proportional hazards (PH)-based models used for 10-year atherosclerotic cardiovascular disease (ASCVD) risk prediction with acceptable discrimination. In recent years, neural network models have gained increasing p...
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-01-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-022-01829-w |
| _version_ | 1811175845948030976 |
|---|---|
| author | Yu Deng Lei Liu Hongmei Jiang Yifan Peng Yishu Wei Zhiyang Zhou Yizhen Zhong Yun Zhao Xiaoyun Yang Jingzhi Yu Zhiyong Lu Abel Kho Hongyan Ning Norrina B. Allen John T. Wilkins Kiang Liu Donald M. Lloyd-Jones Lihui Zhao |
| author_facet | Yu Deng Lei Liu Hongmei Jiang Yifan Peng Yishu Wei Zhiyang Zhou Yizhen Zhong Yun Zhao Xiaoyun Yang Jingzhi Yu Zhiyong Lu Abel Kho Hongyan Ning Norrina B. Allen John T. Wilkins Kiang Liu Donald M. Lloyd-Jones Lihui Zhao |
| author_sort | Yu Deng |
| collection | DOAJ |
| description | Abstract Background The Pooled Cohort Equations (PCEs) are race- and sex-specific Cox proportional hazards (PH)-based models used for 10-year atherosclerotic cardiovascular disease (ASCVD) risk prediction with acceptable discrimination. In recent years, neural network models have gained increasing popularity with their success in image recognition and text classification. Various survival neural network models have been proposed by combining survival analysis and neural network architecture to take advantage of the strengths from both. However, the performance of these survival neural network models compared to each other and to PCEs in ASCVD prediction is unknown. Methods In this study, we used 6 cohorts from the Lifetime Risk Pooling Project (with 5 cohorts as training/internal validation and one cohort as external validation) and compared the performance of the PCEs in 10-year ASCVD risk prediction with an all two-way interactions Cox PH model (Cox PH-TWI) and three state-of-the-art neural network survival models including Nnet-survival, Deepsurv, and Cox-nnet. For all the models, we used the same 7 covariates as used in the PCEs. We fitted each of the aforementioned models in white females, white males, black females, and black males, respectively. We evaluated models’ internal and external discrimination power and calibration. Results The training/internal validation sample comprised 23216 individuals. The average age at baseline was 57.8 years old (SD = 9.6); 16% developed ASCVD during average follow-up of 10.50 (SD = 3.02) years. Based on 10 × 10 cross-validation, the method that had the highest C-statistics was Deepsurv (0.7371) for white males, Deepsurv and Cox PH-TWI (0.7972) for white females, PCE (0.6981) for black males, and Deepsurv (0.7886) for black females. In the external validation dataset, Deepsurv (0.7032), Cox-nnet (0.7282), PCE (0.6811), and Deepsurv (0.7316) had the highest C-statistics for white male, white female, black male, and black female population, respectively. Calibration plots showed that in 10 × 10 validation, all models had good calibration in all race and sex groups. In external validation, all models overestimated the risk for 10-year ASCVD. Conclusions We demonstrated the use of the state-of-the-art neural network survival models in ASCVD risk prediction. Neural network survival models had similar if not superior discrimination and calibration compared to PCEs. |
| first_indexed | 2024-04-10T19:42:35Z |
| format | Article |
| id | doaj.art-f3f66d3a1b9246efbc9fde732aaf21ce |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-04-10T19:42:35Z |
| publishDate | 2023-01-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-f3f66d3a1b9246efbc9fde732aaf21ce2023-01-29T12:15:43ZengBMCBMC Medical Research Methodology1471-22882023-01-0123111110.1186/s12874-022-01829-wComparison of State-of-the-Art Neural Network Survival Models with the Pooled Cohort Equations for Cardiovascular Disease Risk PredictionYu Deng0Lei Liu1Hongmei Jiang2Yifan Peng3Yishu Wei4Zhiyang Zhou5Yizhen Zhong6Yun Zhao7Xiaoyun Yang8Jingzhi Yu9Zhiyong Lu10Abel Kho11Hongyan Ning12Norrina B. Allen13John T. Wilkins14Kiang Liu15Donald M. Lloyd-Jones16Lihui Zhao17Center for Health Information Partnerships, Northwestern University Feinberg School of MedicineDivision of Biostatistics, Washington University in St. LouisDepartment of Statistics and Data Science, Northwestern UniversityDepartment of Population Health Sciences, Weill Cornell MedicineDepartment of Statistics and Data Science, Northwestern UniversityDepartment of Statistics, University of ManitobaCenter for Health Information Partnerships, Northwestern University Feinberg School of MedicineDepartment of Computer Science, University of CaliforniaDepartment of Preventive Medicine, Northwestern University Feinberg School of MedicineDepartment of Preventive Medicine, Northwestern University Feinberg School of MedicineNational Center for Biotechnology Information, National Library of Medicine, National Institute of HealthCenter for Health Information Partnerships, Northwestern University Feinberg School of MedicineDepartment of Preventive Medicine, Northwestern University Feinberg School of MedicineDepartment of Preventive Medicine, Northwestern University Feinberg School of MedicineDepartment of Preventive Medicine, Northwestern University Feinberg School of MedicineDepartment of Preventive Medicine, Northwestern University Feinberg School of MedicineDepartment of Preventive Medicine, Northwestern University Feinberg School of MedicineCenter for Health Information Partnerships, Northwestern University Feinberg School of MedicineAbstract Background The Pooled Cohort Equations (PCEs) are race- and sex-specific Cox proportional hazards (PH)-based models used for 10-year atherosclerotic cardiovascular disease (ASCVD) risk prediction with acceptable discrimination. In recent years, neural network models have gained increasing popularity with their success in image recognition and text classification. Various survival neural network models have been proposed by combining survival analysis and neural network architecture to take advantage of the strengths from both. However, the performance of these survival neural network models compared to each other and to PCEs in ASCVD prediction is unknown. Methods In this study, we used 6 cohorts from the Lifetime Risk Pooling Project (with 5 cohorts as training/internal validation and one cohort as external validation) and compared the performance of the PCEs in 10-year ASCVD risk prediction with an all two-way interactions Cox PH model (Cox PH-TWI) and three state-of-the-art neural network survival models including Nnet-survival, Deepsurv, and Cox-nnet. For all the models, we used the same 7 covariates as used in the PCEs. We fitted each of the aforementioned models in white females, white males, black females, and black males, respectively. We evaluated models’ internal and external discrimination power and calibration. Results The training/internal validation sample comprised 23216 individuals. The average age at baseline was 57.8 years old (SD = 9.6); 16% developed ASCVD during average follow-up of 10.50 (SD = 3.02) years. Based on 10 × 10 cross-validation, the method that had the highest C-statistics was Deepsurv (0.7371) for white males, Deepsurv and Cox PH-TWI (0.7972) for white females, PCE (0.6981) for black males, and Deepsurv (0.7886) for black females. In the external validation dataset, Deepsurv (0.7032), Cox-nnet (0.7282), PCE (0.6811), and Deepsurv (0.7316) had the highest C-statistics for white male, white female, black male, and black female population, respectively. Calibration plots showed that in 10 × 10 validation, all models had good calibration in all race and sex groups. In external validation, all models overestimated the risk for 10-year ASCVD. Conclusions We demonstrated the use of the state-of-the-art neural network survival models in ASCVD risk prediction. Neural network survival models had similar if not superior discrimination and calibration compared to PCEs.https://doi.org/10.1186/s12874-022-01829-wArtificial intelligenceCardiovascular diseaseCox regressionDeep learningMachine learningNeural network |
| spellingShingle | Yu Deng Lei Liu Hongmei Jiang Yifan Peng Yishu Wei Zhiyang Zhou Yizhen Zhong Yun Zhao Xiaoyun Yang Jingzhi Yu Zhiyong Lu Abel Kho Hongyan Ning Norrina B. Allen John T. Wilkins Kiang Liu Donald M. Lloyd-Jones Lihui Zhao Comparison of State-of-the-Art Neural Network Survival Models with the Pooled Cohort Equations for Cardiovascular Disease Risk Prediction BMC Medical Research Methodology Artificial intelligence Cardiovascular disease Cox regression Deep learning Machine learning Neural network |
| title | Comparison of State-of-the-Art Neural Network Survival Models with the Pooled Cohort Equations for Cardiovascular Disease Risk Prediction |
| title_full | Comparison of State-of-the-Art Neural Network Survival Models with the Pooled Cohort Equations for Cardiovascular Disease Risk Prediction |
| title_fullStr | Comparison of State-of-the-Art Neural Network Survival Models with the Pooled Cohort Equations for Cardiovascular Disease Risk Prediction |
| title_full_unstemmed | Comparison of State-of-the-Art Neural Network Survival Models with the Pooled Cohort Equations for Cardiovascular Disease Risk Prediction |
| title_short | Comparison of State-of-the-Art Neural Network Survival Models with the Pooled Cohort Equations for Cardiovascular Disease Risk Prediction |
| title_sort | comparison of state of the art neural network survival models with the pooled cohort equations for cardiovascular disease risk prediction |
| topic | Artificial intelligence Cardiovascular disease Cox regression Deep learning Machine learning Neural network |
| url | https://doi.org/10.1186/s12874-022-01829-w |
| work_keys_str_mv | AT yudeng comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT leiliu comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT hongmeijiang comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT yifanpeng comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT yishuwei comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT zhiyangzhou comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT yizhenzhong comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT yunzhao comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT xiaoyunyang comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT jingzhiyu comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT zhiyonglu comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT abelkho comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT hongyanning comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT norrinaballen comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT johntwilkins comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT kiangliu comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT donaldmlloydjones comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction AT lihuizhao comparisonofstateoftheartneuralnetworksurvivalmodelswiththepooledcohortequationsforcardiovasculardiseaseriskprediction |