Prediction Models for Glaucoma in a Multicenter Electronic Health Records Consortium: The Sight Outcomes Research Collaborative
Purpose: Advances in artificial intelligence have enabled the development of predictive models for glaucoma. However, most work is single-center and uncertainty exists regarding the generalizability of such models. The purpose of this study was to build and evaluate machine learning (ML) approaches...
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2024-05-01
|
| Series: | Ophthalmology Science |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S266691452300177X |
| _version_ | 1797345852367306752 |
|---|---|
| author | Sophia Y. Wang, MD, MS Rohith Ravindranath, MS Joshua D. Stein, MD, MS Sejal Amin Paul A. Edwards Divya Srikumaran Fasika Woreta Jeffrey S. Schultz Anurag Shrivastava Baseer Ahmad Judy Kim Paul Bryar Dustin French Brian L. Vanderbeek Suzann Pershing Sophia Y. Wang Anne M. Lynch Jenna Patnaik Saleha Munir Wuqaas Munir Joshua Stein Lindsey DeLott Brian C. Stagg Barbara Wirostko Brian McMillian Arsham Sheybani |
| author_facet | Sophia Y. Wang, MD, MS Rohith Ravindranath, MS Joshua D. Stein, MD, MS Sejal Amin Paul A. Edwards Divya Srikumaran Fasika Woreta Jeffrey S. Schultz Anurag Shrivastava Baseer Ahmad Judy Kim Paul Bryar Dustin French Brian L. Vanderbeek Suzann Pershing Sophia Y. Wang Anne M. Lynch Jenna Patnaik Saleha Munir Wuqaas Munir Joshua Stein Lindsey DeLott Brian C. Stagg Barbara Wirostko Brian McMillian Arsham Sheybani |
| author_sort | Sophia Y. Wang, MD, MS |
| collection | DOAJ |
| description | Purpose: Advances in artificial intelligence have enabled the development of predictive models for glaucoma. However, most work is single-center and uncertainty exists regarding the generalizability of such models. The purpose of this study was to build and evaluate machine learning (ML) approaches to predict glaucoma progression requiring surgery using data from a large multicenter consortium of electronic health records (EHR). Design: Cohort study. Participants: Thirty-six thousand five hundred forty-eight patients with glaucoma, as identified by International Classification of Diseases (ICD) codes from 6 academic eye centers participating in the Sight OUtcomes Research Collaborative (SOURCE). Methods: We developed ML models to predict whether patients with glaucoma would progress to glaucoma surgery in the coming year (identified by Current Procedural Terminology codes) using the following modeling approaches: (1) penalized logistic regression (lasso, ridge, and elastic net); (2) tree-based models (random forest, gradient boosted machines, and XGBoost), and (3) deep learning models. Model input features included demographics, diagnosis codes, medications, and clinical information (intraocular pressure, visual acuity, refractive status, and central corneal thickness) available from structured EHR data. One site was reserved as an “external site” test set (N = 1550); of the patients from the remaining sites, 10% each were randomly selected to be in development and test sets, with the remaining 27 999 reserved for model training. Main Outcome Measures: Evaluation metrics included area under the receiver operating characteristic curve (AUROC) on the test set and the external site. Results: Six thousand nineteen (16.5%) of 36 548 patients underwent glaucoma surgery. Overall, the AUROC ranged from 0.735 to 0.771 on the random test set and from 0.706 to 0.754 on the external test site, with the XGBoost and random forest model performing best, respectively. There was greatest performance decrease from the random test set to the external test site for the penalized regression models. Conclusions: Machine learning models developed using structured EHR data can reasonably predict whether glaucoma patients will need surgery, with reasonable generalizability to an external site. Additional research is needed to investigate the impact of protected class characteristics such as race or gender on model performance and fairness. Financial Disclosure(s): Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article. |
| first_indexed | 2024-03-08T11:24:43Z |
| format | Article |
| id | doaj.art-6abe49059f6a402aac3a3c51041e5793 |
| institution | Directory Open Access Journal |
| issn | 2666-9145 |
| language | English |
| last_indexed | 2024-03-08T11:24:43Z |
| publishDate | 2024-05-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Ophthalmology Science |
| spelling | doaj.art-6abe49059f6a402aac3a3c51041e57932024-01-26T05:35:33ZengElsevierOphthalmology Science2666-91452024-05-0143100445Prediction Models for Glaucoma in a Multicenter Electronic Health Records Consortium: The Sight Outcomes Research CollaborativeSophia Y. Wang, MD, MS0Rohith Ravindranath, MS1Joshua D. Stein, MD, MS2Sejal AminPaul A. EdwardsDivya SrikumaranFasika WoretaJeffrey S. SchultzAnurag ShrivastavaBaseer AhmadJudy KimPaul BryarDustin FrenchBrian L. VanderbeekSuzann PershingSophia Y. WangAnne M. LynchJenna PatnaikSaleha MunirWuqaas MunirJoshua SteinLindsey DeLottBrian C. StaggBarbara WirostkoBrian McMillianArsham SheybaniDepartment of Ophthalmology, Byers Eye Institute, Stanford University, Palo Alto, California; Correspondence: Sophia Y. Wang, MD, MS, Department of Ophthalmology, Byers Eye Institute, Stanford University, 2370 Watson Ct, Palo Alto, CA 94303.Department of Ophthalmology, Byers Eye Institute, Stanford University, Palo Alto, CaliforniaDepartment of Ophthalmology & Visual Sciences, University of Michigan Kellogg Eye Center, Ann Arbor, MichiganPurpose: Advances in artificial intelligence have enabled the development of predictive models for glaucoma. However, most work is single-center and uncertainty exists regarding the generalizability of such models. The purpose of this study was to build and evaluate machine learning (ML) approaches to predict glaucoma progression requiring surgery using data from a large multicenter consortium of electronic health records (EHR). Design: Cohort study. Participants: Thirty-six thousand five hundred forty-eight patients with glaucoma, as identified by International Classification of Diseases (ICD) codes from 6 academic eye centers participating in the Sight OUtcomes Research Collaborative (SOURCE). Methods: We developed ML models to predict whether patients with glaucoma would progress to glaucoma surgery in the coming year (identified by Current Procedural Terminology codes) using the following modeling approaches: (1) penalized logistic regression (lasso, ridge, and elastic net); (2) tree-based models (random forest, gradient boosted machines, and XGBoost), and (3) deep learning models. Model input features included demographics, diagnosis codes, medications, and clinical information (intraocular pressure, visual acuity, refractive status, and central corneal thickness) available from structured EHR data. One site was reserved as an “external site” test set (N = 1550); of the patients from the remaining sites, 10% each were randomly selected to be in development and test sets, with the remaining 27 999 reserved for model training. Main Outcome Measures: Evaluation metrics included area under the receiver operating characteristic curve (AUROC) on the test set and the external site. Results: Six thousand nineteen (16.5%) of 36 548 patients underwent glaucoma surgery. Overall, the AUROC ranged from 0.735 to 0.771 on the random test set and from 0.706 to 0.754 on the external test site, with the XGBoost and random forest model performing best, respectively. There was greatest performance decrease from the random test set to the external test site for the penalized regression models. Conclusions: Machine learning models developed using structured EHR data can reasonably predict whether glaucoma patients will need surgery, with reasonable generalizability to an external site. Additional research is needed to investigate the impact of protected class characteristics such as race or gender on model performance and fairness. Financial Disclosure(s): Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.http://www.sciencedirect.com/science/article/pii/S266691452300177XMachine learningGlaucomaMulticenter studyDeep learning |
| spellingShingle | Sophia Y. Wang, MD, MS Rohith Ravindranath, MS Joshua D. Stein, MD, MS Sejal Amin Paul A. Edwards Divya Srikumaran Fasika Woreta Jeffrey S. Schultz Anurag Shrivastava Baseer Ahmad Judy Kim Paul Bryar Dustin French Brian L. Vanderbeek Suzann Pershing Sophia Y. Wang Anne M. Lynch Jenna Patnaik Saleha Munir Wuqaas Munir Joshua Stein Lindsey DeLott Brian C. Stagg Barbara Wirostko Brian McMillian Arsham Sheybani Prediction Models for Glaucoma in a Multicenter Electronic Health Records Consortium: The Sight Outcomes Research Collaborative Ophthalmology Science Machine learning Glaucoma Multicenter study Deep learning |
| title | Prediction Models for Glaucoma in a Multicenter Electronic Health Records Consortium: The Sight Outcomes Research Collaborative |
| title_full | Prediction Models for Glaucoma in a Multicenter Electronic Health Records Consortium: The Sight Outcomes Research Collaborative |
| title_fullStr | Prediction Models for Glaucoma in a Multicenter Electronic Health Records Consortium: The Sight Outcomes Research Collaborative |
| title_full_unstemmed | Prediction Models for Glaucoma in a Multicenter Electronic Health Records Consortium: The Sight Outcomes Research Collaborative |
| title_short | Prediction Models for Glaucoma in a Multicenter Electronic Health Records Consortium: The Sight Outcomes Research Collaborative |
| title_sort | prediction models for glaucoma in a multicenter electronic health records consortium the sight outcomes research collaborative |
| topic | Machine learning Glaucoma Multicenter study Deep learning |
| url | http://www.sciencedirect.com/science/article/pii/S266691452300177X |
| work_keys_str_mv | AT sophiaywangmdms predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT rohithravindranathms predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT joshuadsteinmdms predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT sejalamin predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT paulaedwards predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT divyasrikumaran predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT fasikaworeta predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT jeffreysschultz predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT anuragshrivastava predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT baseerahmad predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT judykim predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT paulbryar predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT dustinfrench predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT brianlvanderbeek predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT suzannpershing predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT sophiaywang predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT annemlynch predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT jennapatnaik predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT salehamunir predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT wuqaasmunir predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT joshuastein predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT lindseydelott predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT briancstagg predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT barbarawirostko predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT brianmcmillian predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative AT arshamsheybani predictionmodelsforglaucomainamulticenterelectronichealthrecordsconsortiumthesightoutcomesresearchcollaborative |