Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system
Abstract Background Machine learning and automation are increasingly used to make the evidence synthesis process faster and more responsive to policymakers’ needs. In systematic reviews of randomized controlled trials (RCTs), risk of bias assessment is a resource-intensive task that typically requir...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2022-06-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-022-01649-y |
| _version_ | 1811247954962415616 |
|---|---|
| author | Patricia Sofia Jacobsen Jardim Christopher James Rose Heather Melanie Ames Jose Francisco Meneses Echavez Stijn Van de Velde Ashley Elizabeth Muller |
| author_facet | Patricia Sofia Jacobsen Jardim Christopher James Rose Heather Melanie Ames Jose Francisco Meneses Echavez Stijn Van de Velde Ashley Elizabeth Muller |
| author_sort | Patricia Sofia Jacobsen Jardim |
| collection | DOAJ |
| description | Abstract Background Machine learning and automation are increasingly used to make the evidence synthesis process faster and more responsive to policymakers’ needs. In systematic reviews of randomized controlled trials (RCTs), risk of bias assessment is a resource-intensive task that typically requires two trained reviewers. One function of RobotReviewer, an off-the-shelf machine learning system, is an automated risk of bias assessment. Methods We assessed the feasibility of adopting RobotReviewer within a national public health institute using a randomized, real-time, user-centered study. The study included 26 RCTs and six reviewers from two projects examining health and social interventions. We randomized these studies to one of two RobotReviewer platforms. We operationalized feasibility as accuracy, time use, and reviewer acceptability. We measured accuracy by the number of corrections made by human reviewers (either to automated assessments or another human reviewer’s assessments). We explored acceptability through group discussions and individual email responses after presenting the quantitative results. Results Reviewers were equally likely to accept judgment by RobotReviewer as each other’s judgement during the consensus process when measured dichotomously; risk ratio 1.02 (95% CI 0.92 to 1.13; p = 0.33). We were not able to compare time use. The acceptability of the program by researchers was mixed. Less experienced reviewers were generally more positive, and they saw more benefits and were able to use the tool more flexibly. Reviewers positioned human input and human-to-human interaction as superior to even a semi-automation of this process. Conclusion Despite being presented with evidence of RobotReviewer’s equal performance to humans, participating reviewers were not interested in modifying standard procedures to include automation. If further studies confirm equal accuracy and reduced time compared to manual practices, we suggest that the benefits of RobotReviewer may support its future implementation as one of two assessors, despite reviewer ambivalence. Future research should study barriers to adopting automated tools and how highly educated and experienced researchers can adapt to a job market that is increasingly challenged by new technologies. |
| first_indexed | 2024-04-12T15:18:27Z |
| format | Article |
| id | doaj.art-6ff612c2b4744404b5ddf27a8fa5349c |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-04-12T15:18:27Z |
| publishDate | 2022-06-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-6ff612c2b4744404b5ddf27a8fa5349c2022-12-22T03:27:33ZengBMCBMC Medical Research Methodology1471-22882022-06-0122111210.1186/s12874-022-01649-yAutomating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning systemPatricia Sofia Jacobsen Jardim0Christopher James Rose1Heather Melanie Ames2Jose Francisco Meneses Echavez3Stijn Van de Velde4Ashley Elizabeth Muller5Division for Health Services, Norwegian Institute of Public HealthDivision for Health Services, Norwegian Institute of Public HealthDivision for Health Services, Norwegian Institute of Public HealthDivision for Health Services, Norwegian Institute of Public HealthDivision for Health Services, Norwegian Institute of Public HealthDivision for Health Services, Norwegian Institute of Public HealthAbstract Background Machine learning and automation are increasingly used to make the evidence synthesis process faster and more responsive to policymakers’ needs. In systematic reviews of randomized controlled trials (RCTs), risk of bias assessment is a resource-intensive task that typically requires two trained reviewers. One function of RobotReviewer, an off-the-shelf machine learning system, is an automated risk of bias assessment. Methods We assessed the feasibility of adopting RobotReviewer within a national public health institute using a randomized, real-time, user-centered study. The study included 26 RCTs and six reviewers from two projects examining health and social interventions. We randomized these studies to one of two RobotReviewer platforms. We operationalized feasibility as accuracy, time use, and reviewer acceptability. We measured accuracy by the number of corrections made by human reviewers (either to automated assessments or another human reviewer’s assessments). We explored acceptability through group discussions and individual email responses after presenting the quantitative results. Results Reviewers were equally likely to accept judgment by RobotReviewer as each other’s judgement during the consensus process when measured dichotomously; risk ratio 1.02 (95% CI 0.92 to 1.13; p = 0.33). We were not able to compare time use. The acceptability of the program by researchers was mixed. Less experienced reviewers were generally more positive, and they saw more benefits and were able to use the tool more flexibly. Reviewers positioned human input and human-to-human interaction as superior to even a semi-automation of this process. Conclusion Despite being presented with evidence of RobotReviewer’s equal performance to humans, participating reviewers were not interested in modifying standard procedures to include automation. If further studies confirm equal accuracy and reduced time compared to manual practices, we suggest that the benefits of RobotReviewer may support its future implementation as one of two assessors, despite reviewer ambivalence. Future research should study barriers to adopting automated tools and how highly educated and experienced researchers can adapt to a job market that is increasingly challenged by new technologies.https://doi.org/10.1186/s12874-022-01649-yRisk of biasMachine learningAutomationEvidence synthesisSystematic reviewHeath technology assessment |
| spellingShingle | Patricia Sofia Jacobsen Jardim Christopher James Rose Heather Melanie Ames Jose Francisco Meneses Echavez Stijn Van de Velde Ashley Elizabeth Muller Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system BMC Medical Research Methodology Risk of bias Machine learning Automation Evidence synthesis Systematic review Heath technology assessment |
| title | Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system |
| title_full | Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system |
| title_fullStr | Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system |
| title_full_unstemmed | Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system |
| title_short | Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system |
| title_sort | automating risk of bias assessment in systematic reviews a real time mixed methods comparison of human researchers to a machine learning system |
| topic | Risk of bias Machine learning Automation Evidence synthesis Systematic review Heath technology assessment |
| url | https://doi.org/10.1186/s12874-022-01649-y |
| work_keys_str_mv | AT patriciasofiajacobsenjardim automatingriskofbiasassessmentinsystematicreviewsarealtimemixedmethodscomparisonofhumanresearcherstoamachinelearningsystem AT christopherjamesrose automatingriskofbiasassessmentinsystematicreviewsarealtimemixedmethodscomparisonofhumanresearcherstoamachinelearningsystem AT heathermelanieames automatingriskofbiasassessmentinsystematicreviewsarealtimemixedmethodscomparisonofhumanresearcherstoamachinelearningsystem AT josefranciscomenesesechavez automatingriskofbiasassessmentinsystematicreviewsarealtimemixedmethodscomparisonofhumanresearcherstoamachinelearningsystem AT stijnvandevelde automatingriskofbiasassessmentinsystematicreviewsarealtimemixedmethodscomparisonofhumanresearcherstoamachinelearningsystem AT ashleyelizabethmuller automatingriskofbiasassessmentinsystematicreviewsarealtimemixedmethodscomparisonofhumanresearcherstoamachinelearningsystem |