User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adults
Objectives: The UK government's approach to the pandemic relies on a test, trace and isolate strategy, mainly implemented via the digital NHS Test & Trace Service. Feedback on user experience is central to the successful development of public-facing Services. As the situation dynamicall...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-12-01
|
| Series: | Public Health in Practice |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666535223000472 |
| _version_ | 1797398495286525952 |
|---|---|
| author | P. Bondaronek T. Papakonstantinou C. Stefanidou T. Chadborn |
| author_facet | P. Bondaronek T. Papakonstantinou C. Stefanidou T. Chadborn |
| author_sort | P. Bondaronek |
| collection | DOAJ |
| description | Objectives: The UK government's approach to the pandemic relies on a test, trace and isolate strategy, mainly implemented via the digital NHS Test & Trace Service. Feedback on user experience is central to the successful development of public-facing Services. As the situation dynamically changes and data accumulate, interpretation of feedback by humans becomes time-consuming and unreliable. The specific objectives were to 1) evaluate a human-in-the-loop machine learning technique based on structural topic modelling in terms of its Service ability in the analysis of vast volumes of free-text data, 2) generate actionable themes that can be used to increase user satisfaction of the Service. Methods: We evaluated an unsupervised Topic Modelling approach, testing models with 5–40 topics and differing covariates. Two human coders conducted thematic analysis to interpret the topics. We identified a Structural Topic Model with 25 topics and metadata as covariates as the most appropriate for acquiring insights. Results: Results from analysis of feedback by 37,914 users from May 2020 to March 2021 highlighted issues with the Service falling within three major themes: multiple contacts and incompatible contact method and incompatible contact method, confusion around isolation dates and tracing delays, complex and rigid system. Conclusions: Structural Topic Modelling coupled with thematic analysis was found to be an effective technique to rapidly acquire user insights. Topic modelling can be a quick and cost-effective method to provide high quality, actionable insights from free-text feedback to optimize public health Services. |
| first_indexed | 2024-03-09T01:26:19Z |
| format | Article |
| id | doaj.art-5b77cfbc0e834617ad65c47ea00c3a41 |
| institution | Directory Open Access Journal |
| issn | 2666-5352 |
| language | English |
| last_indexed | 2024-03-09T01:26:19Z |
| publishDate | 2023-12-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Public Health in Practice |
| spelling | doaj.art-5b77cfbc0e834617ad65c47ea00c3a412023-12-10T06:18:19ZengElsevierPublic Health in Practice2666-53522023-12-016100401User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adultsP. Bondaronek0T. Papakonstantinou1C. Stefanidou2T. Chadborn3Office for Health Improvement & Disparities, Department of Health and Social Care, London, SW1H 0EU, United Kingdom; Institute of Health Informatics, University College London, London, NW1 2DA, United Kingdom; Corresponding author. UCL Institute Of Health Informatics, 222 Euston Road, NW1 2DA, London, United Kingdom.Office for Health Improvement & Disparities, Department of Health and Social Care, London, SW1H 0EU, United KingdomOffice for Health Improvement & Disparities, Department of Health and Social Care, London, SW1H 0EU, United KingdomOffice for Health Improvement & Disparities, Department of Health and Social Care, London, SW1H 0EU, United KingdomObjectives: The UK government's approach to the pandemic relies on a test, trace and isolate strategy, mainly implemented via the digital NHS Test & Trace Service. Feedback on user experience is central to the successful development of public-facing Services. As the situation dynamically changes and data accumulate, interpretation of feedback by humans becomes time-consuming and unreliable. The specific objectives were to 1) evaluate a human-in-the-loop machine learning technique based on structural topic modelling in terms of its Service ability in the analysis of vast volumes of free-text data, 2) generate actionable themes that can be used to increase user satisfaction of the Service. Methods: We evaluated an unsupervised Topic Modelling approach, testing models with 5–40 topics and differing covariates. Two human coders conducted thematic analysis to interpret the topics. We identified a Structural Topic Model with 25 topics and metadata as covariates as the most appropriate for acquiring insights. Results: Results from analysis of feedback by 37,914 users from May 2020 to March 2021 highlighted issues with the Service falling within three major themes: multiple contacts and incompatible contact method and incompatible contact method, confusion around isolation dates and tracing delays, complex and rigid system. Conclusions: Structural Topic Modelling coupled with thematic analysis was found to be an effective technique to rapidly acquire user insights. Topic modelling can be a quick and cost-effective method to provide high quality, actionable insights from free-text feedback to optimize public health Services.http://www.sciencedirect.com/science/article/pii/S2666535223000472Public healthMachine learningQualitative dataContact tracingCOVID-19 |
| spellingShingle | P. Bondaronek T. Papakonstantinou C. Stefanidou T. Chadborn User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adults Public Health in Practice Public health Machine learning Qualitative data Contact tracing COVID-19 |
| title | User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adults |
| title_full | User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adults |
| title_fullStr | User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adults |
| title_full_unstemmed | User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adults |
| title_short | User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adults |
| title_sort | user feedback on the nhs test amp trace service during covid 19 the use of machine learning to analyse free text data from 37 914 england adults |
| topic | Public health Machine learning Qualitative data Contact tracing COVID-19 |
| url | http://www.sciencedirect.com/science/article/pii/S2666535223000472 |
| work_keys_str_mv | AT pbondaronek userfeedbackonthenhstestamptraceserviceduringcovid19theuseofmachinelearningtoanalysefreetextdatafrom37914englandadults AT tpapakonstantinou userfeedbackonthenhstestamptraceserviceduringcovid19theuseofmachinelearningtoanalysefreetextdatafrom37914englandadults AT cstefanidou userfeedbackonthenhstestamptraceserviceduringcovid19theuseofmachinelearningtoanalysefreetextdatafrom37914englandadults AT tchadborn userfeedbackonthenhstestamptraceserviceduringcovid19theuseofmachinelearningtoanalysefreetextdatafrom37914englandadults |