Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction
Abstract Suicide risk prediction models can identify individuals for targeted intervention. Discussions of transparency, explainability, and transportability in machine learning presume complex prediction models with many variables outperform simpler models. We compared random forest, artificial neu...
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2023-03-01
|
| Series: | npj Digital Medicine |
| Online Access: | https://doi.org/10.1038/s41746-023-00772-4 |
| _version_ | 1797426551808065536 |
|---|---|
| author | Susan M. Shortreed Rod L. Walker Eric Johnson Robert Wellman Maricela Cruz Rebecca Ziebell R. Yates Coley Zimri S. Yaseen Sai Dharmarajan Robert B. Penfold Brian K. Ahmedani Rebecca C. Rossom Arne Beck Jennifer M. Boggs Greg E. Simon |
| author_facet | Susan M. Shortreed Rod L. Walker Eric Johnson Robert Wellman Maricela Cruz Rebecca Ziebell R. Yates Coley Zimri S. Yaseen Sai Dharmarajan Robert B. Penfold Brian K. Ahmedani Rebecca C. Rossom Arne Beck Jennifer M. Boggs Greg E. Simon |
| author_sort | Susan M. Shortreed |
| collection | DOAJ |
| description | Abstract Suicide risk prediction models can identify individuals for targeted intervention. Discussions of transparency, explainability, and transportability in machine learning presume complex prediction models with many variables outperform simpler models. We compared random forest, artificial neural network, and ensemble models with 1500 temporally defined predictors to logistic regression models. Data from 25,800,888 mental health visits made by 3,081,420 individuals in 7 health systems were used to train and evaluate suicidal behavior prediction models. Model performance was compared across several measures. All models performed well (area under the receiver operating curve [AUC]: 0.794–0.858). Ensemble models performed best, but improvements over a regression model with 100 predictors were minimal (AUC improvements: 0.006–0.020). Results are consistent across performance metrics and subgroups defined by race, ethnicity, and sex. Our results suggest simpler parametric models, which are easier to implement as part of routine clinical practice, perform comparably to more complex machine learning methods. |
| first_indexed | 2024-03-09T08:31:52Z |
| format | Article |
| id | doaj.art-7d704e4cb32d4b58956d6fbd544d82b4 |
| institution | Directory Open Access Journal |
| issn | 2398-6352 |
| language | English |
| last_indexed | 2024-03-09T08:31:52Z |
| publishDate | 2023-03-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | npj Digital Medicine |
| spelling | doaj.art-7d704e4cb32d4b58956d6fbd544d82b42023-12-02T19:38:19ZengNature Portfolionpj Digital Medicine2398-63522023-03-016112210.1038/s41746-023-00772-4Complex modeling with detailed temporal predictors does not improve health records-based suicide risk predictionSusan M. Shortreed0Rod L. Walker1Eric Johnson2Robert Wellman3Maricela Cruz4Rebecca Ziebell5R. Yates Coley6Zimri S. Yaseen7Sai Dharmarajan8Robert B. Penfold9Brian K. Ahmedani10Rebecca C. Rossom11Arne Beck12Jennifer M. Boggs13Greg E. Simon14Kaiser Permanente Washington Health Research InstituteKaiser Permanente Washington Health Research InstituteKaiser Permanente Washington Health Research InstituteKaiser Permanente Washington Health Research InstituteKaiser Permanente Washington Health Research InstituteKaiser Permanente Washington Health Research InstituteKaiser Permanente Washington Health Research InstituteU.S. Food and Drug AdministrationU.S. Food and Drug AdministrationKaiser Permanente Washington Health Research InstituteCenter for Health Policy & Health Services Research, Henry Ford Health SystemHealthPartners Institute, Division of ResearchKaiser Permanente Colorado Institute for Health ResearchKaiser Permanente Colorado Institute for Health ResearchKaiser Permanente Washington Health Research InstituteAbstract Suicide risk prediction models can identify individuals for targeted intervention. Discussions of transparency, explainability, and transportability in machine learning presume complex prediction models with many variables outperform simpler models. We compared random forest, artificial neural network, and ensemble models with 1500 temporally defined predictors to logistic regression models. Data from 25,800,888 mental health visits made by 3,081,420 individuals in 7 health systems were used to train and evaluate suicidal behavior prediction models. Model performance was compared across several measures. All models performed well (area under the receiver operating curve [AUC]: 0.794–0.858). Ensemble models performed best, but improvements over a regression model with 100 predictors were minimal (AUC improvements: 0.006–0.020). Results are consistent across performance metrics and subgroups defined by race, ethnicity, and sex. Our results suggest simpler parametric models, which are easier to implement as part of routine clinical practice, perform comparably to more complex machine learning methods.https://doi.org/10.1038/s41746-023-00772-4 |
| spellingShingle | Susan M. Shortreed Rod L. Walker Eric Johnson Robert Wellman Maricela Cruz Rebecca Ziebell R. Yates Coley Zimri S. Yaseen Sai Dharmarajan Robert B. Penfold Brian K. Ahmedani Rebecca C. Rossom Arne Beck Jennifer M. Boggs Greg E. Simon Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction npj Digital Medicine |
| title | Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction |
| title_full | Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction |
| title_fullStr | Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction |
| title_full_unstemmed | Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction |
| title_short | Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction |
| title_sort | complex modeling with detailed temporal predictors does not improve health records based suicide risk prediction |
| url | https://doi.org/10.1038/s41746-023-00772-4 |
| work_keys_str_mv | AT susanmshortreed complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction AT rodlwalker complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction AT ericjohnson complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction AT robertwellman complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction AT maricelacruz complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction AT rebeccaziebell complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction AT ryatescoley complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction AT zimrisyaseen complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction AT saidharmarajan complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction AT robertbpenfold complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction AT briankahmedani complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction AT rebeccacrossom complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction AT arnebeck complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction AT jennifermboggs complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction AT gregesimon complexmodelingwithdetailedtemporalpredictorsdoesnotimprovehealthrecordsbasedsuicideriskprediction |