Hard for humans, hard for machines: predicting readmission after psychiatric hospitalization using narrative notes
© 2021, The Author(s). Machine learning has been suggested as a means of identifying individuals at greatest risk for hospital readmission, including psychiatric readmission. We sought to compare the performance of predictive models that use interpretable representations derived via topic modeling t...
| Main Authors: | , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Springer Science and Business Media LLC
2021
|
| Online Access: | https://hdl.handle.net/1721.1/134104 |
| _version_ | 1811078446308130816 |
|---|---|
| author | Boag, William Kovaleva, Olga McCoy, Thomas H Rumshisky, Anna Szolovits, Peter Perlis, Roy H |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Boag, William Kovaleva, Olga McCoy, Thomas H Rumshisky, Anna Szolovits, Peter Perlis, Roy H |
| author_sort | Boag, William |
| collection | MIT |
| description | © 2021, The Author(s). Machine learning has been suggested as a means of identifying individuals at greatest risk for hospital readmission, including psychiatric readmission. We sought to compare the performance of predictive models that use interpretable representations derived via topic modeling to the performance of human experts and nonexperts. We examined all 5076 admissions to a general psychiatry inpatient unit between 2009 and 2016 using electronic health records. We developed multiple models to predict 180-day readmission for these admissions based on features derived from narrative discharge summaries, augmented by baseline sociodemographic and clinical features. We developed models using a training set comprising 70% of the cohort and evaluated on the remaining 30%. Baseline models using demographic features for prediction achieved an area under the curve (AUC) of 0.675 [95% CI 0.674–0.676] on an independent testing set, while language-based models also incorporating bag-of-words features, discharge summaries topics identified by Latent Dirichlet allocation (LDA), and prior psychiatric admissions achieved AUC of 0.726 [95% CI 0.725–0.727]. To characterize the difficulty of the task, we also compared the performance of these classifiers to both expert and nonexpert human raters, with and without feedback, on a subset of 75 test cases. These models outperformed humans on average, including predictions by experienced psychiatrists. Typical note tokens or topics associated with readmission risk were related to pregnancy/postpartum state, family relationships, and psychosis. |
| first_indexed | 2024-09-23T10:59:48Z |
| format | Article |
| id | mit-1721.1/134104 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T10:59:48Z |
| publishDate | 2021 |
| publisher | Springer Science and Business Media LLC |
| record_format | dspace |
| spelling | mit-1721.1/1341042023-02-17T16:45:04Z Hard for humans, hard for machines: predicting readmission after psychiatric hospitalization using narrative notes Boag, William Kovaleva, Olga McCoy, Thomas H Rumshisky, Anna Szolovits, Peter Perlis, Roy H Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory © 2021, The Author(s). Machine learning has been suggested as a means of identifying individuals at greatest risk for hospital readmission, including psychiatric readmission. We sought to compare the performance of predictive models that use interpretable representations derived via topic modeling to the performance of human experts and nonexperts. We examined all 5076 admissions to a general psychiatry inpatient unit between 2009 and 2016 using electronic health records. We developed multiple models to predict 180-day readmission for these admissions based on features derived from narrative discharge summaries, augmented by baseline sociodemographic and clinical features. We developed models using a training set comprising 70% of the cohort and evaluated on the remaining 30%. Baseline models using demographic features for prediction achieved an area under the curve (AUC) of 0.675 [95% CI 0.674–0.676] on an independent testing set, while language-based models also incorporating bag-of-words features, discharge summaries topics identified by Latent Dirichlet allocation (LDA), and prior psychiatric admissions achieved AUC of 0.726 [95% CI 0.725–0.727]. To characterize the difficulty of the task, we also compared the performance of these classifiers to both expert and nonexpert human raters, with and without feedback, on a subset of 75 test cases. These models outperformed humans on average, including predictions by experienced psychiatrists. Typical note tokens or topics associated with readmission risk were related to pregnancy/postpartum state, family relationships, and psychosis. 2021-10-27T19:58:07Z 2021-10-27T19:58:07Z 2021 2021-01-26T19:56:02Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/134104 en 10.1038/s41398-020-01104-w Translational Psychiatry Creative Commons Attribution 4.0 International license https://creativecommons.org/licenses/by/4.0/ application/pdf Springer Science and Business Media LLC Nature |
| spellingShingle | Boag, William Kovaleva, Olga McCoy, Thomas H Rumshisky, Anna Szolovits, Peter Perlis, Roy H Hard for humans, hard for machines: predicting readmission after psychiatric hospitalization using narrative notes |
| title | Hard for humans, hard for machines: predicting readmission after psychiatric hospitalization using narrative notes |
| title_full | Hard for humans, hard for machines: predicting readmission after psychiatric hospitalization using narrative notes |
| title_fullStr | Hard for humans, hard for machines: predicting readmission after psychiatric hospitalization using narrative notes |
| title_full_unstemmed | Hard for humans, hard for machines: predicting readmission after psychiatric hospitalization using narrative notes |
| title_short | Hard for humans, hard for machines: predicting readmission after psychiatric hospitalization using narrative notes |
| title_sort | hard for humans hard for machines predicting readmission after psychiatric hospitalization using narrative notes |
| url | https://hdl.handle.net/1721.1/134104 |
| work_keys_str_mv | AT boagwilliam hardforhumanshardformachinespredictingreadmissionafterpsychiatrichospitalizationusingnarrativenotes AT kovalevaolga hardforhumanshardformachinespredictingreadmissionafterpsychiatrichospitalizationusingnarrativenotes AT mccoythomash hardforhumanshardformachinespredictingreadmissionafterpsychiatrichospitalizationusingnarrativenotes AT rumshiskyanna hardforhumanshardformachinespredictingreadmissionafterpsychiatrichospitalizationusingnarrativenotes AT szolovitspeter hardforhumanshardformachinespredictingreadmissionafterpsychiatrichospitalizationusingnarrativenotes AT perlisroyh hardforhumanshardformachinespredictingreadmissionafterpsychiatrichospitalizationusingnarrativenotes |