Diagnostic Performance, Triage Safety, and Usability of a Clinical Decision Support System Within a University Hospital Emergency Department: Algorithm Performance and Usability Study
Abstract BackgroundComputerized clinical decision support systems (CDSSs) are increasingly adopted in health care to optimize resources and streamline patient flow. However, they often lack scientific validation against standard medical care. ObjectiveThe purpose of this study was to assess the per...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2023-08-01
|
| Series: | JMIR Medical Informatics |
| Online Access: | https://medinform.jmir.org/2023/1/e46760 |
| _version_ | 1797691629305331712 |
|---|---|
| author | Juhani Määttä Rony Lindell Nick Hayward Susanna Martikainen Katri Honkanen Matias Inkala Petteri Hirvonen Tero J Martikainen |
| author_facet | Juhani Määttä Rony Lindell Nick Hayward Susanna Martikainen Katri Honkanen Matias Inkala Petteri Hirvonen Tero J Martikainen |
| author_sort | Juhani Määttä |
| collection | DOAJ |
| description |
Abstract
BackgroundComputerized clinical decision support systems (CDSSs) are increasingly adopted in health care to optimize resources and streamline patient flow. However, they often lack scientific validation against standard medical care.
ObjectiveThe purpose of this study was to assess the performance, safety, and usability of a CDSS in a university hospital emergency department setting in Kuopio, Finland.
MethodsPatients entering the emergency department were asked to voluntarily participate in this study. Patients aged 17 years or younger, patients with cognitive impairments, and patients who entered the unit in an ambulance or with the need for immediate care were excluded. Patients completed the CDSS web-based form and usability questionnaire when waiting for the triage nurse’s evaluation. The CDSS data were anonymized and did not affect the patients’ usual evaluation or treatment. Retrospectively, 2 medical doctors evaluated the urgency of each patient’s condition by using the triage nurse’s information, and urgent and nonurgent groups were created. The International Statistical Classification of Diseases, Tenth Revision
ResultsIn total, our analyses included 248 patients. Regarding urgency, the mean sensitivities were 85% and 19%, respectively, for urgent and nonurgent cases when assessing the performance of CDSS evaluations in comparison to that of physicians. The mean sensitivities were 85% and 35%, respectively, when comparing the evaluations between the two physicians. Our CDSS did not miss any cases that were evaluated to be emergencies by physicians; thus, all emergency cases evaluated by physicians were evaluated as either urgent cases or emergency cases by the CDSS. In differential diagnosis, the CDSS had an exact match accuracy of 45.5% (97/213). The usability was good, with a mean System Usability Scale score of 78.2 (SD 16.8).
ConclusionsIn a university hospital emergency department setting with a large real-world population, our CDSS was found to be equally as sensitive in urgent patient cases as physicians and was found to have an acceptable differential diagnosis accuracy, with good usability. These results suggest that this CDSS can be safely assessed further in a real-world setting. A CDSS could accelerate triage by providing patient-provided data in advance of patients’ initial consultations and categorize patient cases as urgent and nonurgent cases upon patients' arrival to the emergency department. |
| first_indexed | 2024-03-12T02:15:59Z |
| format | Article |
| id | doaj.art-447d048907034c7fbf953fff3a2a289f |
| institution | Directory Open Access Journal |
| issn | 2291-9694 |
| language | English |
| last_indexed | 2024-03-12T02:15:59Z |
| publishDate | 2023-08-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Medical Informatics |
| spelling | doaj.art-447d048907034c7fbf953fff3a2a289f2023-09-06T08:44:33ZengJMIR PublicationsJMIR Medical Informatics2291-96942023-08-0111e46760e4676010.2196/46760Diagnostic Performance, Triage Safety, and Usability of a Clinical Decision Support System Within a University Hospital Emergency Department: Algorithm Performance and Usability StudyJuhani Määttä0http://orcid.org/0000-0002-2400-9195Rony Lindell1http://orcid.org/0009-0007-4865-1135Nick Hayward2http://orcid.org/0000-0001-9496-7332Susanna Martikainen3http://orcid.org/0000-0003-4616-5751Katri Honkanen4http://orcid.org/0000-0002-0560-0107Matias Inkala5http://orcid.org/0009-0002-5586-9224Petteri Hirvonen6http://orcid.org/0000-0001-8203-5466Tero J Martikainen7http://orcid.org/0000-0002-8151-9845Klinik Healthcare Solutions OyKlinik Healthcare Solutions OyKlinik Healthcare Solutions OyUniversity of Eastern FinlandKuopio University HospitalKuopio University HospitalKlinik Healthcare Solutions OyKuopio University Hospital Abstract BackgroundComputerized clinical decision support systems (CDSSs) are increasingly adopted in health care to optimize resources and streamline patient flow. However, they often lack scientific validation against standard medical care. ObjectiveThe purpose of this study was to assess the performance, safety, and usability of a CDSS in a university hospital emergency department setting in Kuopio, Finland. MethodsPatients entering the emergency department were asked to voluntarily participate in this study. Patients aged 17 years or younger, patients with cognitive impairments, and patients who entered the unit in an ambulance or with the need for immediate care were excluded. Patients completed the CDSS web-based form and usability questionnaire when waiting for the triage nurse’s evaluation. The CDSS data were anonymized and did not affect the patients’ usual evaluation or treatment. Retrospectively, 2 medical doctors evaluated the urgency of each patient’s condition by using the triage nurse’s information, and urgent and nonurgent groups were created. The International Statistical Classification of Diseases, Tenth Revision ResultsIn total, our analyses included 248 patients. Regarding urgency, the mean sensitivities were 85% and 19%, respectively, for urgent and nonurgent cases when assessing the performance of CDSS evaluations in comparison to that of physicians. The mean sensitivities were 85% and 35%, respectively, when comparing the evaluations between the two physicians. Our CDSS did not miss any cases that were evaluated to be emergencies by physicians; thus, all emergency cases evaluated by physicians were evaluated as either urgent cases or emergency cases by the CDSS. In differential diagnosis, the CDSS had an exact match accuracy of 45.5% (97/213). The usability was good, with a mean System Usability Scale score of 78.2 (SD 16.8). ConclusionsIn a university hospital emergency department setting with a large real-world population, our CDSS was found to be equally as sensitive in urgent patient cases as physicians and was found to have an acceptable differential diagnosis accuracy, with good usability. These results suggest that this CDSS can be safely assessed further in a real-world setting. A CDSS could accelerate triage by providing patient-provided data in advance of patients’ initial consultations and categorize patient cases as urgent and nonurgent cases upon patients' arrival to the emergency department.https://medinform.jmir.org/2023/1/e46760 |
| spellingShingle | Juhani Määttä Rony Lindell Nick Hayward Susanna Martikainen Katri Honkanen Matias Inkala Petteri Hirvonen Tero J Martikainen Diagnostic Performance, Triage Safety, and Usability of a Clinical Decision Support System Within a University Hospital Emergency Department: Algorithm Performance and Usability Study JMIR Medical Informatics |
| title | Diagnostic Performance, Triage Safety, and Usability of a Clinical Decision Support System Within a University Hospital Emergency Department: Algorithm Performance and Usability Study |
| title_full | Diagnostic Performance, Triage Safety, and Usability of a Clinical Decision Support System Within a University Hospital Emergency Department: Algorithm Performance and Usability Study |
| title_fullStr | Diagnostic Performance, Triage Safety, and Usability of a Clinical Decision Support System Within a University Hospital Emergency Department: Algorithm Performance and Usability Study |
| title_full_unstemmed | Diagnostic Performance, Triage Safety, and Usability of a Clinical Decision Support System Within a University Hospital Emergency Department: Algorithm Performance and Usability Study |
| title_short | Diagnostic Performance, Triage Safety, and Usability of a Clinical Decision Support System Within a University Hospital Emergency Department: Algorithm Performance and Usability Study |
| title_sort | diagnostic performance triage safety and usability of a clinical decision support system within a university hospital emergency department algorithm performance and usability study |
| url | https://medinform.jmir.org/2023/1/e46760 |
| work_keys_str_mv | AT juhanimaatta diagnosticperformancetriagesafetyandusabilityofaclinicaldecisionsupportsystemwithinauniversityhospitalemergencydepartmentalgorithmperformanceandusabilitystudy AT ronylindell diagnosticperformancetriagesafetyandusabilityofaclinicaldecisionsupportsystemwithinauniversityhospitalemergencydepartmentalgorithmperformanceandusabilitystudy AT nickhayward diagnosticperformancetriagesafetyandusabilityofaclinicaldecisionsupportsystemwithinauniversityhospitalemergencydepartmentalgorithmperformanceandusabilitystudy AT susannamartikainen diagnosticperformancetriagesafetyandusabilityofaclinicaldecisionsupportsystemwithinauniversityhospitalemergencydepartmentalgorithmperformanceandusabilitystudy AT katrihonkanen diagnosticperformancetriagesafetyandusabilityofaclinicaldecisionsupportsystemwithinauniversityhospitalemergencydepartmentalgorithmperformanceandusabilitystudy AT matiasinkala diagnosticperformancetriagesafetyandusabilityofaclinicaldecisionsupportsystemwithinauniversityhospitalemergencydepartmentalgorithmperformanceandusabilitystudy AT petterihirvonen diagnosticperformancetriagesafetyandusabilityofaclinicaldecisionsupportsystemwithinauniversityhospitalemergencydepartmentalgorithmperformanceandusabilitystudy AT terojmartikainen diagnosticperformancetriagesafetyandusabilityofaclinicaldecisionsupportsystemwithinauniversityhospitalemergencydepartmentalgorithmperformanceandusabilitystudy |