Bringing Transparency to Predictive Analytics: A Systematic Comparison of Predictive Modeling Methods in Higher Education
Colleges have increasingly turned to predictive analytics to target at-risk students for additional support. Most of the predictive analytic applications in higher education are proprietary, with private companies offering little transparency about their underlying models. We address this lack of tr...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SAGE Publishing
2021-08-01
|
| Series: | AERA Open |
| Online Access: | https://doi.org/10.1177/23328584211037630 |
| _version_ | 1818888274586894336 |
|---|---|
| author | Kelli A. Bird Benjamin L. Castleman Zachary Mabel Yifeng Song |
| author_facet | Kelli A. Bird Benjamin L. Castleman Zachary Mabel Yifeng Song |
| author_sort | Kelli A. Bird |
| collection | DOAJ |
| description | Colleges have increasingly turned to predictive analytics to target at-risk students for additional support. Most of the predictive analytic applications in higher education are proprietary, with private companies offering little transparency about their underlying models. We address this lack of transparency by systematically comparing two important dimensions: (1) different approaches to sample and variable construction and how these affect model accuracy and (2) how the selection of predictive modeling approaches, ranging from methods many institutional researchers would be familiar with to more complex machine learning methods, affects model performance and the stability of predicted scores. The relative ranking of students’ predicted probability of completing college varies substantially across modeling approaches. While we observe substantial gains in performance from models trained on a sample structured to represent the typical enrollment spells of students and with a robust set of predictors, we observe similar performance between the simplest and the most complex models. |
| first_indexed | 2024-12-19T16:50:31Z |
| format | Article |
| id | doaj.art-16d90a95da174e81a3c6e3b32fbd68d9 |
| institution | Directory Open Access Journal |
| issn | 2332-8584 |
| language | English |
| last_indexed | 2024-12-19T16:50:31Z |
| publishDate | 2021-08-01 |
| publisher | SAGE Publishing |
| record_format | Article |
| series | AERA Open |
| spelling | doaj.art-16d90a95da174e81a3c6e3b32fbd68d92022-12-21T20:13:32ZengSAGE PublishingAERA Open2332-85842021-08-01710.1177/23328584211037630Bringing Transparency to Predictive Analytics: A Systematic Comparison of Predictive Modeling Methods in Higher EducationKelli A. BirdBenjamin L. CastlemanZachary MabelYifeng SongColleges have increasingly turned to predictive analytics to target at-risk students for additional support. Most of the predictive analytic applications in higher education are proprietary, with private companies offering little transparency about their underlying models. We address this lack of transparency by systematically comparing two important dimensions: (1) different approaches to sample and variable construction and how these affect model accuracy and (2) how the selection of predictive modeling approaches, ranging from methods many institutional researchers would be familiar with to more complex machine learning methods, affects model performance and the stability of predicted scores. The relative ranking of students’ predicted probability of completing college varies substantially across modeling approaches. While we observe substantial gains in performance from models trained on a sample structured to represent the typical enrollment spells of students and with a robust set of predictors, we observe similar performance between the simplest and the most complex models.https://doi.org/10.1177/23328584211037630 |
| spellingShingle | Kelli A. Bird Benjamin L. Castleman Zachary Mabel Yifeng Song Bringing Transparency to Predictive Analytics: A Systematic Comparison of Predictive Modeling Methods in Higher Education AERA Open |
| title | Bringing Transparency to Predictive Analytics: A Systematic Comparison of Predictive Modeling Methods in Higher Education |
| title_full | Bringing Transparency to Predictive Analytics: A Systematic Comparison of Predictive Modeling Methods in Higher Education |
| title_fullStr | Bringing Transparency to Predictive Analytics: A Systematic Comparison of Predictive Modeling Methods in Higher Education |
| title_full_unstemmed | Bringing Transparency to Predictive Analytics: A Systematic Comparison of Predictive Modeling Methods in Higher Education |
| title_short | Bringing Transparency to Predictive Analytics: A Systematic Comparison of Predictive Modeling Methods in Higher Education |
| title_sort | bringing transparency to predictive analytics a systematic comparison of predictive modeling methods in higher education |
| url | https://doi.org/10.1177/23328584211037630 |
| work_keys_str_mv | AT kelliabird bringingtransparencytopredictiveanalyticsasystematiccomparisonofpredictivemodelingmethodsinhighereducation AT benjaminlcastleman bringingtransparencytopredictiveanalyticsasystematiccomparisonofpredictivemodelingmethodsinhighereducation AT zacharymabel bringingtransparencytopredictiveanalyticsasystematiccomparisonofpredictivemodelingmethodsinhighereducation AT yifengsong bringingtransparencytopredictiveanalyticsasystematiccomparisonofpredictivemodelingmethodsinhighereducation |