314 Large Language Model Approaches to Understand Differences Between Guidelines and Clinician Perception of Best Practices
OBJECTIVES/GOALS: The Clinical Implementation stage in the translational pipeline is hampered by the tension between formal evidence and clinician perceptions. For instance, when guidelines are translated into electronic clinical decision support alerts, they are often ignored. Using advances in LLM...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Cambridge University Press
2024-04-01
|
| Series: | Journal of Clinical and Translational Science |
| Online Access: | https://www.cambridge.org/core/product/identifier/S2059866124002851/type/journal_article |
| _version_ | 1797226956102565888 |
|---|---|
| author | Carrie E. Gold Jorie M. Butler Ithan D. Peltan Julio C. Facelli |
| author_facet | Carrie E. Gold Jorie M. Butler Ithan D. Peltan Julio C. Facelli |
| author_sort | Carrie E. Gold |
| collection | DOAJ |
| description | OBJECTIVES/GOALS: The Clinical Implementation stage in the translational pipeline is hampered by the tension between formal evidence and clinician perceptions. For instance, when guidelines are translated into electronic clinical decision support alerts, they are often ignored. Using advances in LLMs we present a framework to quantify these discrepancies. METHODS/STUDY POPULATION: We hypothesize that ignoring guideline-based alerts may be driven by discordances between clinical guidelines’ deterministic realities and clinician’ perception of clinical reality. Until now this has been very difficult to measure using quantitative methods. We argue that advances in Large Language Models (LLM) provide an avenue for exploring this quantitatively. Here we present the method and preliminary results comparing the responses of BioBERTT from a carefully designed set of questions when the LLM is fine-tuned using either formal guidelines or transcripts of clinicians discussing guidelines and clinical care in the parallel domain. The formal “distance” between the LLM responses is evaluated using quantitative metrics like the Hamming Distance. RESULTS/ANTICIPATED RESULTS: We present a description of the architecture used to prove or disprove our hypothesis. We will present results obtained when training the architecture with data that could be used to test the limits of our hypothesis, by fine-tuning BioBERT with diverse synthetic clinical views, either in agreement or disagreement with the formal guidelines. Results comparing sepsis guideline text with transcripts of interviews with Emergency Department clinicians discussing care practices for sepsis in the ED transcripts will also be considered. Our current emphasis is on securing a wider range of transcripts of clinicians interviewed from different clinical specialties and different clinical settings. While here we focus on clinical guidelines, the framework supports any intervention in the Clinical Implementation stage. DISCUSSION/SIGNIFICANCE: Leveraging recent advances in LLMs, we develop a framework that can quantitatively measure the differences between guidelines and clinician perception of best practices. We demonstrated the functionality of this approach using synthetic data and initiated the collection of clinician transcripts to test the framework in real clinical situations. |
| first_indexed | 2024-04-24T14:33:08Z |
| format | Article |
| id | doaj.art-4823ac0dbb6141c0b2f78057f337e387 |
| institution | Directory Open Access Journal |
| issn | 2059-8661 |
| language | English |
| last_indexed | 2024-04-24T14:33:08Z |
| publishDate | 2024-04-01 |
| publisher | Cambridge University Press |
| record_format | Article |
| series | Journal of Clinical and Translational Science |
| spelling | doaj.art-4823ac0dbb6141c0b2f78057f337e3872024-04-03T02:00:00ZengCambridge University PressJournal of Clinical and Translational Science2059-86612024-04-018969710.1017/cts.2024.285314 Large Language Model Approaches to Understand Differences Between Guidelines and Clinician Perception of Best PracticesCarrie E. Gold0Jorie M. Butler1Ithan D. Peltan2Julio C. Facelli3Department of Biomedical Informatics, The University of Utah, Salt Lake City, UTDepartment of Biomedical Informatics, The University of Utah, Salt Lake City, UTDepartment of Pulmonary and Critical Care Medicine, Intermountain Medical Center, Salt Lake City, UTDepartment of Biomedical Informatics and Clinical and Translational Science Institute, The University of Utah, Salt Lake City, UTOBJECTIVES/GOALS: The Clinical Implementation stage in the translational pipeline is hampered by the tension between formal evidence and clinician perceptions. For instance, when guidelines are translated into electronic clinical decision support alerts, they are often ignored. Using advances in LLMs we present a framework to quantify these discrepancies. METHODS/STUDY POPULATION: We hypothesize that ignoring guideline-based alerts may be driven by discordances between clinical guidelines’ deterministic realities and clinician’ perception of clinical reality. Until now this has been very difficult to measure using quantitative methods. We argue that advances in Large Language Models (LLM) provide an avenue for exploring this quantitatively. Here we present the method and preliminary results comparing the responses of BioBERTT from a carefully designed set of questions when the LLM is fine-tuned using either formal guidelines or transcripts of clinicians discussing guidelines and clinical care in the parallel domain. The formal “distance” between the LLM responses is evaluated using quantitative metrics like the Hamming Distance. RESULTS/ANTICIPATED RESULTS: We present a description of the architecture used to prove or disprove our hypothesis. We will present results obtained when training the architecture with data that could be used to test the limits of our hypothesis, by fine-tuning BioBERT with diverse synthetic clinical views, either in agreement or disagreement with the formal guidelines. Results comparing sepsis guideline text with transcripts of interviews with Emergency Department clinicians discussing care practices for sepsis in the ED transcripts will also be considered. Our current emphasis is on securing a wider range of transcripts of clinicians interviewed from different clinical specialties and different clinical settings. While here we focus on clinical guidelines, the framework supports any intervention in the Clinical Implementation stage. DISCUSSION/SIGNIFICANCE: Leveraging recent advances in LLMs, we develop a framework that can quantitatively measure the differences between guidelines and clinician perception of best practices. We demonstrated the functionality of this approach using synthetic data and initiated the collection of clinician transcripts to test the framework in real clinical situations.https://www.cambridge.org/core/product/identifier/S2059866124002851/type/journal_article |
| spellingShingle | Carrie E. Gold Jorie M. Butler Ithan D. Peltan Julio C. Facelli 314 Large Language Model Approaches to Understand Differences Between Guidelines and Clinician Perception of Best Practices Journal of Clinical and Translational Science |
| title | 314 Large Language Model Approaches to Understand Differences Between Guidelines and Clinician Perception of Best Practices |
| title_full | 314 Large Language Model Approaches to Understand Differences Between Guidelines and Clinician Perception of Best Practices |
| title_fullStr | 314 Large Language Model Approaches to Understand Differences Between Guidelines and Clinician Perception of Best Practices |
| title_full_unstemmed | 314 Large Language Model Approaches to Understand Differences Between Guidelines and Clinician Perception of Best Practices |
| title_short | 314 Large Language Model Approaches to Understand Differences Between Guidelines and Clinician Perception of Best Practices |
| title_sort | 314 large language model approaches to understand differences between guidelines and clinician perception of best practices |
| url | https://www.cambridge.org/core/product/identifier/S2059866124002851/type/journal_article |
| work_keys_str_mv | AT carrieegold 314largelanguagemodelapproachestounderstanddifferencesbetweenguidelinesandclinicianperceptionofbestpractices AT joriembutler 314largelanguagemodelapproachestounderstanddifferencesbetweenguidelinesandclinicianperceptionofbestpractices AT ithandpeltan 314largelanguagemodelapproachestounderstanddifferencesbetweenguidelinesandclinicianperceptionofbestpractices AT juliocfacelli 314largelanguagemodelapproachestounderstanddifferencesbetweenguidelinesandclinicianperceptionofbestpractices |