Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse
BackgroundSocial media has become an established platform for individuals to discuss and debate various subjects, including vaccination. With growing conversations on the web and less than desired maternal vaccination uptake rates, these conversations could provide useful ins...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2021-10-01
|
| Series: | JMIR Medical Informatics |
| Online Access: | https://medinform.jmir.org/2021/10/e29584 |
| _version_ | 1827858944058458112 |
|---|---|
| author | Per E Kummervold Sam Martin Sara Dada Eliz Kilich Chermain Denny Pauline Paterson Heidi J Larson |
| author_facet | Per E Kummervold Sam Martin Sara Dada Eliz Kilich Chermain Denny Pauline Paterson Heidi J Larson |
| author_sort | Per E Kummervold |
| collection | DOAJ |
| description |
BackgroundSocial media has become an established platform for individuals to discuss and debate various subjects, including vaccination. With growing conversations on the web and less than desired maternal vaccination uptake rates, these conversations could provide useful insights to inform future interventions. However, owing to the volume of web-based posts, manual annotation and analysis are difficult and time consuming. Automated processes for this type of analysis, such as natural language processing, have faced challenges in extracting complex stances such as attitudes toward vaccination from large amounts of text.
ObjectiveThe aim of this study is to build upon recent advances in transposer-based machine learning methods and test whether transformer-based machine learning could be used as a tool to assess the stance expressed in social media posts toward vaccination during pregnancy.
MethodsA total of 16,604 tweets posted between November 1, 2018, and April 30, 2019, were selected using keyword searches related to maternal vaccination. After excluding irrelevant tweets, the remaining tweets were coded by 3 individual researchers into the categories Promotional, Discouraging, Ambiguous, and Neutral or No Stance. After creating a final data set of 2722 unique tweets, multiple machine learning techniques were trained on a part of this data set and then tested and compared with the human annotators.
ResultsWe found the accuracy of the machine learning techniques to be 81.8% (F score=0.78) compared with the agreed score among the 3 annotators. For comparison, the accuracies of the individual annotators compared with the final score were 83.3%, 77.9%, and 77.5%.
ConclusionsThis study demonstrates that we are able to achieve close to the same accuracy in categorizing tweets using our machine learning models as could be expected from a single human coder. The potential to use this automated process, which is reliable and accurate, could free valuable time and resources for conducting this analysis, in addition to informing potentially effective and necessary interventions. |
| first_indexed | 2024-03-12T13:02:15Z |
| format | Article |
| id | doaj.art-48022b59bf0d4fa88b70e7eeffa6891d |
| institution | Directory Open Access Journal |
| issn | 2291-9694 |
| language | English |
| last_indexed | 2024-03-12T13:02:15Z |
| publishDate | 2021-10-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Medical Informatics |
| spelling | doaj.art-48022b59bf0d4fa88b70e7eeffa6891d2023-08-28T19:30:04ZengJMIR PublicationsJMIR Medical Informatics2291-96942021-10-01910e2958410.2196/29584Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter DiscoursePer E Kummervoldhttps://orcid.org/0000-0003-1007-0945Sam Martinhttps://orcid.org/0000-0002-4466-8374Sara Dadahttps://orcid.org/0000-0003-3910-1856Eliz Kilichhttps://orcid.org/0000-0003-0928-8293Chermain Dennyhttps://orcid.org/0000-0003-2449-1345Pauline Patersonhttps://orcid.org/0000-0002-4166-8248Heidi J Larsonhttps://orcid.org/0000-0002-8477-7583 BackgroundSocial media has become an established platform for individuals to discuss and debate various subjects, including vaccination. With growing conversations on the web and less than desired maternal vaccination uptake rates, these conversations could provide useful insights to inform future interventions. However, owing to the volume of web-based posts, manual annotation and analysis are difficult and time consuming. Automated processes for this type of analysis, such as natural language processing, have faced challenges in extracting complex stances such as attitudes toward vaccination from large amounts of text. ObjectiveThe aim of this study is to build upon recent advances in transposer-based machine learning methods and test whether transformer-based machine learning could be used as a tool to assess the stance expressed in social media posts toward vaccination during pregnancy. MethodsA total of 16,604 tweets posted between November 1, 2018, and April 30, 2019, were selected using keyword searches related to maternal vaccination. After excluding irrelevant tweets, the remaining tweets were coded by 3 individual researchers into the categories Promotional, Discouraging, Ambiguous, and Neutral or No Stance. After creating a final data set of 2722 unique tweets, multiple machine learning techniques were trained on a part of this data set and then tested and compared with the human annotators. ResultsWe found the accuracy of the machine learning techniques to be 81.8% (F score=0.78) compared with the agreed score among the 3 annotators. For comparison, the accuracies of the individual annotators compared with the final score were 83.3%, 77.9%, and 77.5%. ConclusionsThis study demonstrates that we are able to achieve close to the same accuracy in categorizing tweets using our machine learning models as could be expected from a single human coder. The potential to use this automated process, which is reliable and accurate, could free valuable time and resources for conducting this analysis, in addition to informing potentially effective and necessary interventions.https://medinform.jmir.org/2021/10/e29584 |
| spellingShingle | Per E Kummervold Sam Martin Sara Dada Eliz Kilich Chermain Denny Pauline Paterson Heidi J Larson Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse JMIR Medical Informatics |
| title | Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse |
| title_full | Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse |
| title_fullStr | Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse |
| title_full_unstemmed | Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse |
| title_short | Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse |
| title_sort | categorizing vaccine confidence with a transformer based machine learning model analysis of nuances of vaccine sentiment in twitter discourse |
| url | https://medinform.jmir.org/2021/10/e29584 |
| work_keys_str_mv | AT perekummervold categorizingvaccineconfidencewithatransformerbasedmachinelearningmodelanalysisofnuancesofvaccinesentimentintwitterdiscourse AT sammartin categorizingvaccineconfidencewithatransformerbasedmachinelearningmodelanalysisofnuancesofvaccinesentimentintwitterdiscourse AT saradada categorizingvaccineconfidencewithatransformerbasedmachinelearningmodelanalysisofnuancesofvaccinesentimentintwitterdiscourse AT elizkilich categorizingvaccineconfidencewithatransformerbasedmachinelearningmodelanalysisofnuancesofvaccinesentimentintwitterdiscourse AT chermaindenny categorizingvaccineconfidencewithatransformerbasedmachinelearningmodelanalysisofnuancesofvaccinesentimentintwitterdiscourse AT paulinepaterson categorizingvaccineconfidencewithatransformerbasedmachinelearningmodelanalysisofnuancesofvaccinesentimentintwitterdiscourse AT heidijlarson categorizingvaccineconfidencewithatransformerbasedmachinelearningmodelanalysisofnuancesofvaccinesentimentintwitterdiscourse |