Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM)
Abstract Background Evaluating patients’ experiences is essential when incorporating the patients’ perspective in improving healthcare. Experiences are mainly collected using closed-ended questions, although the value of open-ended questions is widely recognized. Natural language processing (NLP) ca...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2022-07-01
|
| Series: | BMC Medical Informatics and Decision Making |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12911-022-01923-5 |
| _version_ | 1811218267002372096 |
|---|---|
| author | Marieke M. van Buchem Olaf M. Neve Ilse M. J. Kant Ewout W. Steyerberg Hileen Boosman Erik F. Hensen |
| author_facet | Marieke M. van Buchem Olaf M. Neve Ilse M. J. Kant Ewout W. Steyerberg Hileen Boosman Erik F. Hensen |
| author_sort | Marieke M. van Buchem |
| collection | DOAJ |
| description | Abstract Background Evaluating patients’ experiences is essential when incorporating the patients’ perspective in improving healthcare. Experiences are mainly collected using closed-ended questions, although the value of open-ended questions is widely recognized. Natural language processing (NLP) can automate the analysis of open-ended questions for an efficient approach to patient-centeredness. Methods We developed the Artificial Intelligence Patient-Reported Experience Measures (AI-PREM) tool, consisting of a new, open-ended questionnaire, an NLP pipeline to analyze the answers using sentiment analysis and topic modeling, and a visualization to guide physicians through the results. The questionnaire and NLP pipeline were iteratively developed and validated in a clinical context. Results The final AI-PREM consisted of five open-ended questions about the provided information, personal approach, collaboration between healthcare professionals, organization of care, and other experiences. The AI-PREM was sent to 867 vestibular schwannoma patients, 534 of which responded. The sentiment analysis model attained an F1 score of 0.97 for positive texts and 0.63 for negative texts. There was a 90% overlap between automatically and manually extracted topics. The visualization was hierarchically structured into three stages: the sentiment per question, the topics per sentiment and question, and the original patient responses per topic. Conclusions The AI-PREM tool is a comprehensive method that combines a validated, open-ended questionnaire with a well-performing NLP pipeline and visualization. Thematically organizing and quantifying patient feedback reduces the time invested by healthcare professionals to evaluate and prioritize patient experiences without being confined to the limited answer options of closed-ended questions. |
| first_indexed | 2024-04-12T07:06:58Z |
| format | Article |
| id | doaj.art-2dfec4e469864446ae16575c32435553 |
| institution | Directory Open Access Journal |
| issn | 1472-6947 |
| language | English |
| last_indexed | 2024-04-12T07:06:58Z |
| publishDate | 2022-07-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Informatics and Decision Making |
| spelling | doaj.art-2dfec4e469864446ae16575c324355532022-12-22T03:42:45ZengBMCBMC Medical Informatics and Decision Making1472-69472022-07-0122111110.1186/s12911-022-01923-5Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM)Marieke M. van Buchem0Olaf M. Neve1Ilse M. J. Kant2Ewout W. Steyerberg3Hileen Boosman4Erik F. Hensen5Information Technology & Digital Innovation Department, Leiden University Medical CenterDepartment of Otorhinolaryngology and Head and Neck Surgery, Leiden University Medical CenterInformation Technology & Digital Innovation Department, Leiden University Medical CenterDepartment of Biomedical Data Sciences, Leiden University Medical CenterMorgensDepartment of Otorhinolaryngology and Head and Neck Surgery, Leiden University Medical CenterAbstract Background Evaluating patients’ experiences is essential when incorporating the patients’ perspective in improving healthcare. Experiences are mainly collected using closed-ended questions, although the value of open-ended questions is widely recognized. Natural language processing (NLP) can automate the analysis of open-ended questions for an efficient approach to patient-centeredness. Methods We developed the Artificial Intelligence Patient-Reported Experience Measures (AI-PREM) tool, consisting of a new, open-ended questionnaire, an NLP pipeline to analyze the answers using sentiment analysis and topic modeling, and a visualization to guide physicians through the results. The questionnaire and NLP pipeline were iteratively developed and validated in a clinical context. Results The final AI-PREM consisted of five open-ended questions about the provided information, personal approach, collaboration between healthcare professionals, organization of care, and other experiences. The AI-PREM was sent to 867 vestibular schwannoma patients, 534 of which responded. The sentiment analysis model attained an F1 score of 0.97 for positive texts and 0.63 for negative texts. There was a 90% overlap between automatically and manually extracted topics. The visualization was hierarchically structured into three stages: the sentiment per question, the topics per sentiment and question, and the original patient responses per topic. Conclusions The AI-PREM tool is a comprehensive method that combines a validated, open-ended questionnaire with a well-performing NLP pipeline and visualization. Thematically organizing and quantifying patient feedback reduces the time invested by healthcare professionals to evaluate and prioritize patient experiences without being confined to the limited answer options of closed-ended questions.https://doi.org/10.1186/s12911-022-01923-5Natural language processingSentiment analysisUnsupervised machine learningPatient satisfactionPatient-centered care |
| spellingShingle | Marieke M. van Buchem Olaf M. Neve Ilse M. J. Kant Ewout W. Steyerberg Hileen Boosman Erik F. Hensen Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM) BMC Medical Informatics and Decision Making Natural language processing Sentiment analysis Unsupervised machine learning Patient satisfaction Patient-centered care |
| title | Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM) |
| title_full | Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM) |
| title_fullStr | Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM) |
| title_full_unstemmed | Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM) |
| title_short | Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM) |
| title_sort | analyzing patient experiences using natural language processing development and validation of the artificial intelligence patient reported experience measure ai prem |
| topic | Natural language processing Sentiment analysis Unsupervised machine learning Patient satisfaction Patient-centered care |
| url | https://doi.org/10.1186/s12911-022-01923-5 |
| work_keys_str_mv | AT mariekemvanbuchem analyzingpatientexperiencesusingnaturallanguageprocessingdevelopmentandvalidationoftheartificialintelligencepatientreportedexperiencemeasureaiprem AT olafmneve analyzingpatientexperiencesusingnaturallanguageprocessingdevelopmentandvalidationoftheartificialintelligencepatientreportedexperiencemeasureaiprem AT ilsemjkant analyzingpatientexperiencesusingnaturallanguageprocessingdevelopmentandvalidationoftheartificialintelligencepatientreportedexperiencemeasureaiprem AT ewoutwsteyerberg analyzingpatientexperiencesusingnaturallanguageprocessingdevelopmentandvalidationoftheartificialintelligencepatientreportedexperiencemeasureaiprem AT hileenboosman analyzingpatientexperiencesusingnaturallanguageprocessingdevelopmentandvalidationoftheartificialintelligencepatientreportedexperiencemeasureaiprem AT erikfhensen analyzingpatientexperiencesusingnaturallanguageprocessingdevelopmentandvalidationoftheartificialintelligencepatientreportedexperiencemeasureaiprem |