Adding propensity scores to pure prediction models fails to improve predictive performance
Background. Propensity score usage seems to be growing in popularity leading researchers to question the possible role of propensity scores in prediction modeling, despite the lack of a theoretical rationale. It is suspected that such requests are due to the lack of differentiation regarding the goa...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
PeerJ Inc.
2013-08-01
|
| Series: | PeerJ |
| Subjects: | |
| Online Access: | https://peerj.com/articles/123.pdf |
| _version_ | 1797419856613605376 |
|---|---|
| author | Amy S. Nowacki Brian J. Wells Changhong Yu Michael W. Kattan |
| author_facet | Amy S. Nowacki Brian J. Wells Changhong Yu Michael W. Kattan |
| author_sort | Amy S. Nowacki |
| collection | DOAJ |
| description | Background. Propensity score usage seems to be growing in popularity leading researchers to question the possible role of propensity scores in prediction modeling, despite the lack of a theoretical rationale. It is suspected that such requests are due to the lack of differentiation regarding the goals of predictive modeling versus causal inference modeling. Therefore, the purpose of this study is to formally examine the effect of propensity scores on predictive performance. Our hypothesis is that a multivariable regression model that adjusts for all covariates will perform as well as or better than those models utilizing propensity scores with respect to model discrimination and calibration.Methods. The most commonly encountered statistical scenarios for medical prediction (logistic and proportional hazards regression) were used to investigate this research question. Random cross-validation was performed 500 times to correct for optimism. The multivariable regression models adjusting for all covariates were compared with models that included adjustment for or weighting with the propensity scores. The methods were compared based on three predictive performance measures: (1) concordance indices; (2) Brier scores; and (3) calibration curves.Results. Multivariable models adjusting for all covariates had the highest average concordance index, the lowest average Brier score, and the best calibration. Propensity score adjustment and inverse probability weighting models without adjustment for all covariates performed worse than full models and failed to improve predictive performance with full covariate adjustment.Conclusion. Propensity score techniques did not improve prediction performance measures beyond multivariable adjustment. Propensity scores are not recommended if the analytical goal is pure prediction modeling. |
| first_indexed | 2024-03-09T06:53:19Z |
| format | Article |
| id | doaj.art-8dae66aabb54418a989f3a05b998b8c6 |
| institution | Directory Open Access Journal |
| issn | 2167-8359 |
| language | English |
| last_indexed | 2024-03-09T06:53:19Z |
| publishDate | 2013-08-01 |
| publisher | PeerJ Inc. |
| record_format | Article |
| series | PeerJ |
| spelling | doaj.art-8dae66aabb54418a989f3a05b998b8c62023-12-03T10:15:35ZengPeerJ Inc.PeerJ2167-83592013-08-011e12310.7717/peerj.123123Adding propensity scores to pure prediction models fails to improve predictive performanceAmy S. Nowacki0Brian J. Wells1Changhong Yu2Michael W. Kattan3Department of Quantitative Health Sciences, Cleveland Clinic, Cleveland, OH, USADepartment of Quantitative Health Sciences, Cleveland Clinic, Cleveland, OH, USADepartment of Quantitative Health Sciences, Cleveland Clinic, Cleveland, OH, USADepartment of Quantitative Health Sciences, Cleveland Clinic, Cleveland, OH, USABackground. Propensity score usage seems to be growing in popularity leading researchers to question the possible role of propensity scores in prediction modeling, despite the lack of a theoretical rationale. It is suspected that such requests are due to the lack of differentiation regarding the goals of predictive modeling versus causal inference modeling. Therefore, the purpose of this study is to formally examine the effect of propensity scores on predictive performance. Our hypothesis is that a multivariable regression model that adjusts for all covariates will perform as well as or better than those models utilizing propensity scores with respect to model discrimination and calibration.Methods. The most commonly encountered statistical scenarios for medical prediction (logistic and proportional hazards regression) were used to investigate this research question. Random cross-validation was performed 500 times to correct for optimism. The multivariable regression models adjusting for all covariates were compared with models that included adjustment for or weighting with the propensity scores. The methods were compared based on three predictive performance measures: (1) concordance indices; (2) Brier scores; and (3) calibration curves.Results. Multivariable models adjusting for all covariates had the highest average concordance index, the lowest average Brier score, and the best calibration. Propensity score adjustment and inverse probability weighting models without adjustment for all covariates performed worse than full models and failed to improve predictive performance with full covariate adjustment.Conclusion. Propensity score techniques did not improve prediction performance measures beyond multivariable adjustment. Propensity scores are not recommended if the analytical goal is pure prediction modeling.https://peerj.com/articles/123.pdfPredictionPropensity scoreCalibration curveConcordance indexMultivariable regression |
| spellingShingle | Amy S. Nowacki Brian J. Wells Changhong Yu Michael W. Kattan Adding propensity scores to pure prediction models fails to improve predictive performance PeerJ Prediction Propensity score Calibration curve Concordance index Multivariable regression |
| title | Adding propensity scores to pure prediction models fails to improve predictive performance |
| title_full | Adding propensity scores to pure prediction models fails to improve predictive performance |
| title_fullStr | Adding propensity scores to pure prediction models fails to improve predictive performance |
| title_full_unstemmed | Adding propensity scores to pure prediction models fails to improve predictive performance |
| title_short | Adding propensity scores to pure prediction models fails to improve predictive performance |
| title_sort | adding propensity scores to pure prediction models fails to improve predictive performance |
| topic | Prediction Propensity score Calibration curve Concordance index Multivariable regression |
| url | https://peerj.com/articles/123.pdf |
| work_keys_str_mv | AT amysnowacki addingpropensityscorestopurepredictionmodelsfailstoimprovepredictiveperformance AT brianjwells addingpropensityscorestopurepredictionmodelsfailstoimprovepredictiveperformance AT changhongyu addingpropensityscorestopurepredictionmodelsfailstoimprovepredictiveperformance AT michaelwkattan addingpropensityscorestopurepredictionmodelsfailstoimprovepredictiveperformance |