User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adults
Objectives: The UK government's approach to the pandemic relies on a test, trace and isolate strategy, mainly implemented via the digital NHS Test & Trace Service. Feedback on user experience is central to the successful development of public-facing Services. As the situation dynamicall...
Main Authors: | , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
Elsevier
2023-12-01
|
Series: | Public Health in Practice |
Subjects: | |
Online Access: | http://www.sciencedirect.com/science/article/pii/S2666535223000472 |
_version_ | 1797398495286525952 |
---|---|
author | P. Bondaronek T. Papakonstantinou C. Stefanidou T. Chadborn |
author_facet | P. Bondaronek T. Papakonstantinou C. Stefanidou T. Chadborn |
author_sort | P. Bondaronek |
collection | DOAJ |
description | Objectives: The UK government's approach to the pandemic relies on a test, trace and isolate strategy, mainly implemented via the digital NHS Test & Trace Service. Feedback on user experience is central to the successful development of public-facing Services. As the situation dynamically changes and data accumulate, interpretation of feedback by humans becomes time-consuming and unreliable. The specific objectives were to 1) evaluate a human-in-the-loop machine learning technique based on structural topic modelling in terms of its Service ability in the analysis of vast volumes of free-text data, 2) generate actionable themes that can be used to increase user satisfaction of the Service. Methods: We evaluated an unsupervised Topic Modelling approach, testing models with 5–40 topics and differing covariates. Two human coders conducted thematic analysis to interpret the topics. We identified a Structural Topic Model with 25 topics and metadata as covariates as the most appropriate for acquiring insights. Results: Results from analysis of feedback by 37,914 users from May 2020 to March 2021 highlighted issues with the Service falling within three major themes: multiple contacts and incompatible contact method and incompatible contact method, confusion around isolation dates and tracing delays, complex and rigid system. Conclusions: Structural Topic Modelling coupled with thematic analysis was found to be an effective technique to rapidly acquire user insights. Topic modelling can be a quick and cost-effective method to provide high quality, actionable insights from free-text feedback to optimize public health Services. |
first_indexed | 2024-03-09T01:26:19Z |
format | Article |
id | doaj.art-5b77cfbc0e834617ad65c47ea00c3a41 |
institution | Directory Open Access Journal |
issn | 2666-5352 |
language | English |
last_indexed | 2024-03-09T01:26:19Z |
publishDate | 2023-12-01 |
publisher | Elsevier |
record_format | Article |
series | Public Health in Practice |
spelling | doaj.art-5b77cfbc0e834617ad65c47ea00c3a412023-12-10T06:18:19ZengElsevierPublic Health in Practice2666-53522023-12-016100401User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adultsP. Bondaronek0T. Papakonstantinou1C. Stefanidou2T. Chadborn3Office for Health Improvement & Disparities, Department of Health and Social Care, London, SW1H 0EU, United Kingdom; Institute of Health Informatics, University College London, London, NW1 2DA, United Kingdom; Corresponding author. UCL Institute Of Health Informatics, 222 Euston Road, NW1 2DA, London, United Kingdom.Office for Health Improvement & Disparities, Department of Health and Social Care, London, SW1H 0EU, United KingdomOffice for Health Improvement & Disparities, Department of Health and Social Care, London, SW1H 0EU, United KingdomOffice for Health Improvement & Disparities, Department of Health and Social Care, London, SW1H 0EU, United KingdomObjectives: The UK government's approach to the pandemic relies on a test, trace and isolate strategy, mainly implemented via the digital NHS Test & Trace Service. Feedback on user experience is central to the successful development of public-facing Services. As the situation dynamically changes and data accumulate, interpretation of feedback by humans becomes time-consuming and unreliable. The specific objectives were to 1) evaluate a human-in-the-loop machine learning technique based on structural topic modelling in terms of its Service ability in the analysis of vast volumes of free-text data, 2) generate actionable themes that can be used to increase user satisfaction of the Service. Methods: We evaluated an unsupervised Topic Modelling approach, testing models with 5–40 topics and differing covariates. Two human coders conducted thematic analysis to interpret the topics. We identified a Structural Topic Model with 25 topics and metadata as covariates as the most appropriate for acquiring insights. Results: Results from analysis of feedback by 37,914 users from May 2020 to March 2021 highlighted issues with the Service falling within three major themes: multiple contacts and incompatible contact method and incompatible contact method, confusion around isolation dates and tracing delays, complex and rigid system. Conclusions: Structural Topic Modelling coupled with thematic analysis was found to be an effective technique to rapidly acquire user insights. Topic modelling can be a quick and cost-effective method to provide high quality, actionable insights from free-text feedback to optimize public health Services.http://www.sciencedirect.com/science/article/pii/S2666535223000472Public healthMachine learningQualitative dataContact tracingCOVID-19 |
spellingShingle | P. Bondaronek T. Papakonstantinou C. Stefanidou T. Chadborn User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adults Public Health in Practice Public health Machine learning Qualitative data Contact tracing COVID-19 |
title | User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adults |
title_full | User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adults |
title_fullStr | User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adults |
title_full_unstemmed | User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adults |
title_short | User feedback on the NHS test & Trace Service during COVID-19: The use of machine learning to analyse free-text data from 37,914 England adults |
title_sort | user feedback on the nhs test amp trace service during covid 19 the use of machine learning to analyse free text data from 37 914 england adults |
topic | Public health Machine learning Qualitative data Contact tracing COVID-19 |
url | http://www.sciencedirect.com/science/article/pii/S2666535223000472 |
work_keys_str_mv | AT pbondaronek userfeedbackonthenhstestamptraceserviceduringcovid19theuseofmachinelearningtoanalysefreetextdatafrom37914englandadults AT tpapakonstantinou userfeedbackonthenhstestamptraceserviceduringcovid19theuseofmachinelearningtoanalysefreetextdatafrom37914englandadults AT cstefanidou userfeedbackonthenhstestamptraceserviceduringcovid19theuseofmachinelearningtoanalysefreetextdatafrom37914englandadults AT tchadborn userfeedbackonthenhstestamptraceserviceduringcovid19theuseofmachinelearningtoanalysefreetextdatafrom37914englandadults |