Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning
Objective To demonstrate the incremental benefit of using free text data in addition to vital sign and demographic data to identify patients with suspected infection in the emergency department. Methods This was a retrospective, observational cohort study performed at a tertiary academic teac...
| Main Authors: | , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
Public Library of Science
2017
|
| Online Access: | http://hdl.handle.net/1721.1/109959 https://orcid.org/0000-0002-5034-7796 |
| _version_ | 1826204476556443648 |
|---|---|
| author | Horng, Steven Halpern, Yoni Jernite, Yacine Shapiro, Nathan I. Nathanson, Larry A. Sontag, David Alexander |
| author2 | Institute for Medical Engineering and Science |
| author_facet | Institute for Medical Engineering and Science Horng, Steven Halpern, Yoni Jernite, Yacine Shapiro, Nathan I. Nathanson, Larry A. Sontag, David Alexander |
| author_sort | Horng, Steven |
| collection | MIT |
| description | Objective
To demonstrate the incremental benefit of using free text data in addition to vital sign and demographic data to identify patients with suspected infection in the emergency department.
Methods
This was a retrospective, observational cohort study performed at a tertiary academic teaching hospital. All consecutive ED patient visits between 12/17/08 and 2/17/13 were included. No patients were excluded. The primary outcome measure was infection diagnosed in the emergency department defined as a patient having an infection related ED ICD-9-CM discharge diagnosis. Patients were randomly allocated to train (64%), validate (20%), and test (16%) data sets. After preprocessing the free text using bigram and negation detection, we built four models to predict infection, incrementally adding vital signs, chief complaint, and free text nursing assessment. We used two different methods to represent free text: a bag of words model and a topic model. We then used a support vector machine to build the prediction model. We calculated the area under the receiver operating characteristic curve to compare the discriminatory power of each model.
Results
A total of 230,936 patient visits were included in the study. Approximately 14% of patients had the primary outcome of diagnosed infection. The area under the ROC curve (AUC) for the vitals model, which used only vital signs and demographic data, was 0.67 for the training data set, 0.67 for the validation data set, and 0.67 (95% CI 0.65–0.69) for the test data set. The AUC for the chief complaint model which also included demographic and vital sign data was 0.84 for the training data set, 0.83 for the validation data set, and 0.83 (95% CI 0.81–0.84) for the test data set. The best performing methods made use of all of the free text. In particular, the AUC for the bag-of-words model was 0.89 for training data set, 0.86 for the validation data set, and 0.86 (95% CI 0.85–0.87) for the test data set. The AUC for the topic model was 0.86 for the training data set, 0.86 for the validation data set, and 0.85 (95% CI 0.84–0.86) for the test data set.
Conclusion
Compared to previous work that only used structured data such as vital signs and demographic information, utilizing free text drastically improves the discriminatory ability (increase in AUC from 0.67 to 0.86) of identifying infection. |
| first_indexed | 2024-09-23T12:55:21Z |
| format | Article |
| id | mit-1721.1/109959 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T12:55:21Z |
| publishDate | 2017 |
| publisher | Public Library of Science |
| record_format | dspace |
| spelling | mit-1721.1/1099592022-10-01T11:59:05Z Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning Horng, Steven Halpern, Yoni Jernite, Yacine Shapiro, Nathan I. Nathanson, Larry A. Sontag, David Alexander Institute for Medical Engineering and Science Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Sontag, David Alexander Objective To demonstrate the incremental benefit of using free text data in addition to vital sign and demographic data to identify patients with suspected infection in the emergency department. Methods This was a retrospective, observational cohort study performed at a tertiary academic teaching hospital. All consecutive ED patient visits between 12/17/08 and 2/17/13 were included. No patients were excluded. The primary outcome measure was infection diagnosed in the emergency department defined as a patient having an infection related ED ICD-9-CM discharge diagnosis. Patients were randomly allocated to train (64%), validate (20%), and test (16%) data sets. After preprocessing the free text using bigram and negation detection, we built four models to predict infection, incrementally adding vital signs, chief complaint, and free text nursing assessment. We used two different methods to represent free text: a bag of words model and a topic model. We then used a support vector machine to build the prediction model. We calculated the area under the receiver operating characteristic curve to compare the discriminatory power of each model. Results A total of 230,936 patient visits were included in the study. Approximately 14% of patients had the primary outcome of diagnosed infection. The area under the ROC curve (AUC) for the vitals model, which used only vital signs and demographic data, was 0.67 for the training data set, 0.67 for the validation data set, and 0.67 (95% CI 0.65–0.69) for the test data set. The AUC for the chief complaint model which also included demographic and vital sign data was 0.84 for the training data set, 0.83 for the validation data set, and 0.83 (95% CI 0.81–0.84) for the test data set. The best performing methods made use of all of the free text. In particular, the AUC for the bag-of-words model was 0.89 for training data set, 0.86 for the validation data set, and 0.86 (95% CI 0.85–0.87) for the test data set. The AUC for the topic model was 0.86 for the training data set, 0.86 for the validation data set, and 0.85 (95% CI 0.84–0.86) for the test data set. Conclusion Compared to previous work that only used structured data such as vital signs and demographic information, utilizing free text drastically improves the discriminatory ability (increase in AUC from 0.67 to 0.86) of identifying infection. 2017-06-16T17:47:34Z 2017-06-16T17:47:34Z 2017-04 2015-03 Article http://purl.org/eprint/type/JournalArticle 1932-6203 http://hdl.handle.net/1721.1/109959 Horng, Steven; Sontag, David A.; Halpern, Yoni; Jernite, Yacine; Shapiro, Nathan I. and Nathanson, Larry A. “Creating an Automated Trigger for Sepsis Clinical Decision Support at Emergency Department Triage Using Machine Learning.” Edited by Tudor Groza. PLOS ONE 12, no. 4 (April 2017): e0174708 © 2017 Horng et al https://orcid.org/0000-0002-5034-7796 en_US http://dx.doi.org/10.1371/journal.pone.0174708 PLoS ONE Creative Commons Attribution 4.0 International License http://creativecommons.org/licenses/by/4.0/ application/pdf Public Library of Science PLoS |
| spellingShingle | Horng, Steven Halpern, Yoni Jernite, Yacine Shapiro, Nathan I. Nathanson, Larry A. Sontag, David Alexander Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning |
| title | Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning |
| title_full | Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning |
| title_fullStr | Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning |
| title_full_unstemmed | Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning |
| title_short | Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning |
| title_sort | creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning |
| url | http://hdl.handle.net/1721.1/109959 https://orcid.org/0000-0002-5034-7796 |
| work_keys_str_mv | AT horngsteven creatinganautomatedtriggerforsepsisclinicaldecisionsupportatemergencydepartmenttriageusingmachinelearning AT halpernyoni creatinganautomatedtriggerforsepsisclinicaldecisionsupportatemergencydepartmenttriageusingmachinelearning AT jerniteyacine creatinganautomatedtriggerforsepsisclinicaldecisionsupportatemergencydepartmenttriageusingmachinelearning AT shapironathani creatinganautomatedtriggerforsepsisclinicaldecisionsupportatemergencydepartmenttriageusingmachinelearning AT nathansonlarrya creatinganautomatedtriggerforsepsisclinicaldecisionsupportatemergencydepartmenttriageusingmachinelearning AT sontagdavidalexander creatinganautomatedtriggerforsepsisclinicaldecisionsupportatemergencydepartmenttriageusingmachinelearning |