Dynamic prediction modeling of postoperative mortality among patients undergoing surgical aortic valve replacement in a statewide cohort over a 12-year periodCentral MessagePerspective
Objective: Clinical prediction models for surgical aortic valve replacement mortality, are valuable decision tools but are often limited in their ability to account for changes in medical practice, patient selection, and the risk of outcomes over time. Recent research has identified methods to updat...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-09-01
|
| Series: | JTCVS Open |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666273623001973 |
| _version_ | 1827807237977931776 |
|---|---|
| author | Jackie Pollack, MSc Wei Yang, PhD Erin M. Schnellinger, PhD, MS George J. Arnaoutakis, MD Michael J. Kallan, MS Stephen E. Kimmel, MD, MSCE |
| author_facet | Jackie Pollack, MSc Wei Yang, PhD Erin M. Schnellinger, PhD, MS George J. Arnaoutakis, MD Michael J. Kallan, MS Stephen E. Kimmel, MD, MSCE |
| author_sort | Jackie Pollack, MSc |
| collection | DOAJ |
| description | Objective: Clinical prediction models for surgical aortic valve replacement mortality, are valuable decision tools but are often limited in their ability to account for changes in medical practice, patient selection, and the risk of outcomes over time. Recent research has identified methods to update models as new data accrue, but their effect on model performance has not been rigorously tested. Methods: The study population included 44,546 adults who underwent an isolated surgical aortic valve replacement from January 1, 1999, to December 31, 2018, statewide in Pennsylvania. After chronologically splitting the data into training and validation sets, we compared calibration, discrimination, and accuracy measures amongst a nonupdating model to 2 methods of model updating: calibration regression and the novel dynamic logistic state space model. Results: The risk of mortality decreased significantly during the validation period (P < .01) and the nonupdating model demonstrated poor calibration and reduced accuracy over time. Both updating models maintained better calibration (Hosmer-Lemeshow χ2 statistic) than the nonupdating model: nonupdating (156.5), calibration regression (4.9), and dynamic logistic state space model (8.0). Overall accuracy (Brier score) was consistently better across both updating models: dynamic logistic state space model (0.0252), calibration regression (0.0253), and nonupdating (0.0256). Discrimination improved with the dynamic logistic state space model (area under the curve, 0.696) compared with the nonupdating model (area under the curve, 0.685) and calibration regression method (area under the curve, 0.687). Conclusions: Dynamic model updating can improve model accuracy, discrimination, and calibration. The decision as to which method to use may depend on which measure is most important in each clinical context. Because competing therapies have emerged for valve replacement models, updating may guide clinical decision making. |
| first_indexed | 2024-03-11T21:54:32Z |
| format | Article |
| id | doaj.art-35720ee80693435a8ed2465acc016362 |
| institution | Directory Open Access Journal |
| issn | 2666-2736 |
| language | English |
| last_indexed | 2024-03-11T21:54:32Z |
| publishDate | 2023-09-01 |
| publisher | Elsevier |
| record_format | Article |
| series | JTCVS Open |
| spelling | doaj.art-35720ee80693435a8ed2465acc0163622023-09-26T04:12:37ZengElsevierJTCVS Open2666-27362023-09-011594112Dynamic prediction modeling of postoperative mortality among patients undergoing surgical aortic valve replacement in a statewide cohort over a 12-year periodCentral MessagePerspectiveJackie Pollack, MSc0Wei Yang, PhD1Erin M. Schnellinger, PhD, MS2George J. Arnaoutakis, MD3Michael J. Kallan, MS4Stephen E. Kimmel, MD, MSCE5Department of Epidemiology, College of Public Health and Health Professions and College of Medicine, University of Florida, Gainesville, FlaDepartment of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, PaResearch Department, United Network For Organ Sharing, Richmond, VaDivision of Cardiovascular and Thoracic Surgery, University of Texas at Austin Dell Medical School, Austin, TexCenter for Clinical Epidemiology and Biostatistics, University of Pennsylvania Perelman School of Medicine, Philadelphia, PaDepartment of Epidemiology, College of Public Health and Health Professions and College of Medicine, University of Florida, Gainesville, Fla; Address for reprints: Stephen E. Kimmel, MD, MSCE, Department of Epidemiology, University of Florida, 2004 Mowry Rd, PO Box 100231, Gainesville, FL 32610.Objective: Clinical prediction models for surgical aortic valve replacement mortality, are valuable decision tools but are often limited in their ability to account for changes in medical practice, patient selection, and the risk of outcomes over time. Recent research has identified methods to update models as new data accrue, but their effect on model performance has not been rigorously tested. Methods: The study population included 44,546 adults who underwent an isolated surgical aortic valve replacement from January 1, 1999, to December 31, 2018, statewide in Pennsylvania. After chronologically splitting the data into training and validation sets, we compared calibration, discrimination, and accuracy measures amongst a nonupdating model to 2 methods of model updating: calibration regression and the novel dynamic logistic state space model. Results: The risk of mortality decreased significantly during the validation period (P < .01) and the nonupdating model demonstrated poor calibration and reduced accuracy over time. Both updating models maintained better calibration (Hosmer-Lemeshow χ2 statistic) than the nonupdating model: nonupdating (156.5), calibration regression (4.9), and dynamic logistic state space model (8.0). Overall accuracy (Brier score) was consistently better across both updating models: dynamic logistic state space model (0.0252), calibration regression (0.0253), and nonupdating (0.0256). Discrimination improved with the dynamic logistic state space model (area under the curve, 0.696) compared with the nonupdating model (area under the curve, 0.685) and calibration regression method (area under the curve, 0.687). Conclusions: Dynamic model updating can improve model accuracy, discrimination, and calibration. The decision as to which method to use may depend on which measure is most important in each clinical context. Because competing therapies have emerged for valve replacement models, updating may guide clinical decision making.http://www.sciencedirect.com/science/article/pii/S2666273623001973clinical prediction modelmodel updatingmodel recalibrationsurgical aortic valve replacementdynamic logistic state space model |
| spellingShingle | Jackie Pollack, MSc Wei Yang, PhD Erin M. Schnellinger, PhD, MS George J. Arnaoutakis, MD Michael J. Kallan, MS Stephen E. Kimmel, MD, MSCE Dynamic prediction modeling of postoperative mortality among patients undergoing surgical aortic valve replacement in a statewide cohort over a 12-year periodCentral MessagePerspective JTCVS Open clinical prediction model model updating model recalibration surgical aortic valve replacement dynamic logistic state space model |
| title | Dynamic prediction modeling of postoperative mortality among patients undergoing surgical aortic valve replacement in a statewide cohort over a 12-year periodCentral MessagePerspective |
| title_full | Dynamic prediction modeling of postoperative mortality among patients undergoing surgical aortic valve replacement in a statewide cohort over a 12-year periodCentral MessagePerspective |
| title_fullStr | Dynamic prediction modeling of postoperative mortality among patients undergoing surgical aortic valve replacement in a statewide cohort over a 12-year periodCentral MessagePerspective |
| title_full_unstemmed | Dynamic prediction modeling of postoperative mortality among patients undergoing surgical aortic valve replacement in a statewide cohort over a 12-year periodCentral MessagePerspective |
| title_short | Dynamic prediction modeling of postoperative mortality among patients undergoing surgical aortic valve replacement in a statewide cohort over a 12-year periodCentral MessagePerspective |
| title_sort | dynamic prediction modeling of postoperative mortality among patients undergoing surgical aortic valve replacement in a statewide cohort over a 12 year periodcentral messageperspective |
| topic | clinical prediction model model updating model recalibration surgical aortic valve replacement dynamic logistic state space model |
| url | http://www.sciencedirect.com/science/article/pii/S2666273623001973 |
| work_keys_str_mv | AT jackiepollackmsc dynamicpredictionmodelingofpostoperativemortalityamongpatientsundergoingsurgicalaorticvalvereplacementinastatewidecohortovera12yearperiodcentralmessageperspective AT weiyangphd dynamicpredictionmodelingofpostoperativemortalityamongpatientsundergoingsurgicalaorticvalvereplacementinastatewidecohortovera12yearperiodcentralmessageperspective AT erinmschnellingerphdms dynamicpredictionmodelingofpostoperativemortalityamongpatientsundergoingsurgicalaorticvalvereplacementinastatewidecohortovera12yearperiodcentralmessageperspective AT georgejarnaoutakismd dynamicpredictionmodelingofpostoperativemortalityamongpatientsundergoingsurgicalaorticvalvereplacementinastatewidecohortovera12yearperiodcentralmessageperspective AT michaeljkallanms dynamicpredictionmodelingofpostoperativemortalityamongpatientsundergoingsurgicalaorticvalvereplacementinastatewidecohortovera12yearperiodcentralmessageperspective AT stephenekimmelmdmsce dynamicpredictionmodelingofpostoperativemortalityamongpatientsundergoingsurgicalaorticvalvereplacementinastatewidecohortovera12yearperiodcentralmessageperspective |