Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study
BackgroundDelirium in hospitalized patients is a syndrome of acute brain dysfunction. Diagnostic (International Classification of Diseases [ICD]) codes are often used in studies using electronic health records (EHRs), but they are inaccurate. ObjectiveWe sought to...
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2022-06-01
|
| Series: | JMIR Formative Research |
| Online Access: | https://formative.jmir.org/2022/6/e33834 |
| _version_ | 1827858230829645824 |
|---|---|
| author | Wendong Ge Haitham Alabsi Aayushee Jain Elissa Ye Haoqi Sun Marta Fernandes Colin Magdamo Ryan A Tesh Sarah I Collens Amy Newhouse Lidia MVR Moura Sahar Zafar John Hsu Oluwaseun Akeju Gregory K Robbins Shibani S Mukerji Sudeshna Das M Brandon Westover |
| author_facet | Wendong Ge Haitham Alabsi Aayushee Jain Elissa Ye Haoqi Sun Marta Fernandes Colin Magdamo Ryan A Tesh Sarah I Collens Amy Newhouse Lidia MVR Moura Sahar Zafar John Hsu Oluwaseun Akeju Gregory K Robbins Shibani S Mukerji Sudeshna Das M Brandon Westover |
| author_sort | Wendong Ge |
| collection | DOAJ |
| description |
BackgroundDelirium in hospitalized patients is a syndrome of acute brain dysfunction. Diagnostic (International Classification of Diseases [ICD]) codes are often used in studies using electronic health records (EHRs), but they are inaccurate.
ObjectiveWe sought to develop a more accurate method using natural language processing (NLP) to detect delirium episodes on the basis of unstructured clinical notes.
MethodsWe collected 1.5 million notes from >10,000 patients from among 9 hospitals. Seven experts iteratively labeled 200,471 sentences. Using these, we trained three NLP classifiers: Support Vector Machine, Recurrent Neural Networks, and Transformer. Testing was performed using an external data set. We also evaluated associations with delirium billing (ICD) codes, medications, orders for restraints and sitters, direct assessments (Confusion Assessment Method [CAM] scores), and in-hospital mortality. F1 scores, confusion matrices, and areas under the receiver operating characteristic curve (AUCs) were used to compare NLP models. We used the φ coefficient to measure associations with other delirium indicators.
ResultsThe transformer NLP performed best on the following parameters: micro F1=0.978, macro F1=0.918, positive AUC=0.984, and negative AUC=0.992. NLP detections exhibited higher correlations (φ) than ICD codes with deliriogenic medications (0.194 vs 0.073 for ICD codes), restraints and sitter orders (0.358 vs 0.177), mortality (0.216 vs 0.000), and CAM scores (0.256 vs –0.028).
ConclusionsClinical notes are an attractive alternative to ICD codes for EHR delirium studies but require automated methods. Our NLP model detects delirium with high accuracy, similar to manual chart review. Our NLP approach can provide more accurate determination of delirium for large-scale EHR-based studies regarding delirium, quality improvement, and clinical trails. |
| first_indexed | 2024-03-12T12:51:57Z |
| format | Article |
| id | doaj.art-bb6606f0948349e6b2113cfa0a866124 |
| institution | Directory Open Access Journal |
| issn | 2561-326X |
| language | English |
| last_indexed | 2024-03-12T12:51:57Z |
| publishDate | 2022-06-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Formative Research |
| spelling | doaj.art-bb6606f0948349e6b2113cfa0a8661242023-08-28T22:22:05ZengJMIR PublicationsJMIR Formative Research2561-326X2022-06-0166e3383410.2196/33834Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational StudyWendong Gehttps://orcid.org/0000-0003-1557-5336Haitham Alabsihttps://orcid.org/0000-0001-6354-4679Aayushee Jainhttps://orcid.org/0000-0002-5018-3234Elissa Yehttps://orcid.org/0000-0003-4851-6543Haoqi Sunhttps://orcid.org/0000-0002-5041-8312Marta Fernandeshttps://orcid.org/0000-0002-7203-2832Colin Magdamohttps://orcid.org/0000-0001-8965-4630Ryan A Teshhttps://orcid.org/0000-0002-6154-6248Sarah I Collenshttps://orcid.org/0000-0001-7010-7266Amy Newhousehttps://orcid.org/0000-0002-7392-4242Lidia MVR Mourahttps://orcid.org/0000-0002-1191-1315Sahar Zafarhttps://orcid.org/0000-0001-5252-5376John Hsuhttps://orcid.org/0000-0001-8244-231XOluwaseun Akejuhttps://orcid.org/0000-0002-6740-1250Gregory K Robbinshttps://orcid.org/0000-0001-7545-5817Shibani S Mukerjihttps://orcid.org/0000-0002-5677-6954Sudeshna Dashttps://orcid.org/0000-0002-9486-6811M Brandon Westoverhttps://orcid.org/0000-0003-4803-312X BackgroundDelirium in hospitalized patients is a syndrome of acute brain dysfunction. Diagnostic (International Classification of Diseases [ICD]) codes are often used in studies using electronic health records (EHRs), but they are inaccurate. ObjectiveWe sought to develop a more accurate method using natural language processing (NLP) to detect delirium episodes on the basis of unstructured clinical notes. MethodsWe collected 1.5 million notes from >10,000 patients from among 9 hospitals. Seven experts iteratively labeled 200,471 sentences. Using these, we trained three NLP classifiers: Support Vector Machine, Recurrent Neural Networks, and Transformer. Testing was performed using an external data set. We also evaluated associations with delirium billing (ICD) codes, medications, orders for restraints and sitters, direct assessments (Confusion Assessment Method [CAM] scores), and in-hospital mortality. F1 scores, confusion matrices, and areas under the receiver operating characteristic curve (AUCs) were used to compare NLP models. We used the φ coefficient to measure associations with other delirium indicators. ResultsThe transformer NLP performed best on the following parameters: micro F1=0.978, macro F1=0.918, positive AUC=0.984, and negative AUC=0.992. NLP detections exhibited higher correlations (φ) than ICD codes with deliriogenic medications (0.194 vs 0.073 for ICD codes), restraints and sitter orders (0.358 vs 0.177), mortality (0.216 vs 0.000), and CAM scores (0.256 vs –0.028). ConclusionsClinical notes are an attractive alternative to ICD codes for EHR delirium studies but require automated methods. Our NLP model detects delirium with high accuracy, similar to manual chart review. Our NLP approach can provide more accurate determination of delirium for large-scale EHR-based studies regarding delirium, quality improvement, and clinical trails.https://formative.jmir.org/2022/6/e33834 |
| spellingShingle | Wendong Ge Haitham Alabsi Aayushee Jain Elissa Ye Haoqi Sun Marta Fernandes Colin Magdamo Ryan A Tesh Sarah I Collens Amy Newhouse Lidia MVR Moura Sahar Zafar John Hsu Oluwaseun Akeju Gregory K Robbins Shibani S Mukerji Sudeshna Das M Brandon Westover Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study JMIR Formative Research |
| title | Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study |
| title_full | Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study |
| title_fullStr | Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study |
| title_full_unstemmed | Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study |
| title_short | Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study |
| title_sort | identifying patients with delirium based on unstructured clinical notes observational study |
| url | https://formative.jmir.org/2022/6/e33834 |
| work_keys_str_mv | AT wendongge identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT haithamalabsi identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT aayusheejain identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT elissaye identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT haoqisun identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT martafernandes identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT colinmagdamo identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT ryanatesh identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT sarahicollens identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT amynewhouse identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT lidiamvrmoura identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT saharzafar identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT johnhsu identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT oluwaseunakeju identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT gregorykrobbins identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT shibanismukerji identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT sudeshnadas identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT mbrandonwestover identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy |