The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews
Abstract Background We investigated the feasibility of using a machine learning tool’s relevance predictions to expedite title and abstract screening. Methods We subjected 11 systematic reviews and six rapid reviews to four retrospective screening simulations (automated and semi-automated approaches...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2020-06-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | http://link.springer.com/article/10.1186/s12874-020-01031-w |
| _version_ | 1828433752513052672 |
|---|---|
| author | Allison Gates Michelle Gates Meghan Sebastianski Samantha Guitard Sarah A. Elliott Lisa Hartling |
| author_facet | Allison Gates Michelle Gates Meghan Sebastianski Samantha Guitard Sarah A. Elliott Lisa Hartling |
| author_sort | Allison Gates |
| collection | DOAJ |
| description | Abstract Background We investigated the feasibility of using a machine learning tool’s relevance predictions to expedite title and abstract screening. Methods We subjected 11 systematic reviews and six rapid reviews to four retrospective screening simulations (automated and semi-automated approaches to single-reviewer and dual independent screening) in Abstrackr, a freely-available machine learning software. We calculated the proportion missed, workload savings, and time savings compared to single-reviewer and dual independent screening by human reviewers. We performed cited reference searches to determine if missed studies would be identified via reference list scanning. Results For systematic reviews, the semi-automated, dual independent screening approach provided the best balance of time savings (median (range) 20 (3–82) hours) and reliability (median (range) proportion missed records, 1 (0–14)%). The cited references search identified 59% (n = 10/17) of the records missed. For the rapid reviews, the fully and semi-automated approaches saved time (median (range) 9 (2–18) hours and 3 (1–10) hours, respectively), but less so than for the systematic reviews. The median (range) proportion missed records for both approaches was 6 (0–22)%. Conclusion Using Abstrackr to assist one of two reviewers in systematic reviews saves time with little risk of missing relevant records. Many missed records would be identified via other means. |
| first_indexed | 2024-12-10T18:40:15Z |
| format | Article |
| id | doaj.art-aef08b220f174cf5898b4ffe11ff9288 |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-12-10T18:40:15Z |
| publishDate | 2020-06-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-aef08b220f174cf5898b4ffe11ff92882022-12-22T01:37:40ZengBMCBMC Medical Research Methodology1471-22882020-06-012011910.1186/s12874-020-01031-wThe semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviewsAllison Gates0Michelle Gates1Meghan Sebastianski2Samantha Guitard3Sarah A. Elliott4Lisa Hartling5Alberta Research Centre for Health Evidence, Department of Pediatrics, University of AlbertaAlberta Research Centre for Health Evidence, Department of Pediatrics, University of AlbertaAlberta Strategy for Patient-Oriented Research (SPOR) SUPPORT Unit Knowledge Translation Platform, University of AlbertaAlberta Research Centre for Health Evidence, Department of Pediatrics, University of AlbertaAlberta Research Centre for Health Evidence, Department of Pediatrics, University of AlbertaAlberta Research Centre for Health Evidence, Department of Pediatrics, University of AlbertaAbstract Background We investigated the feasibility of using a machine learning tool’s relevance predictions to expedite title and abstract screening. Methods We subjected 11 systematic reviews and six rapid reviews to four retrospective screening simulations (automated and semi-automated approaches to single-reviewer and dual independent screening) in Abstrackr, a freely-available machine learning software. We calculated the proportion missed, workload savings, and time savings compared to single-reviewer and dual independent screening by human reviewers. We performed cited reference searches to determine if missed studies would be identified via reference list scanning. Results For systematic reviews, the semi-automated, dual independent screening approach provided the best balance of time savings (median (range) 20 (3–82) hours) and reliability (median (range) proportion missed records, 1 (0–14)%). The cited references search identified 59% (n = 10/17) of the records missed. For the rapid reviews, the fully and semi-automated approaches saved time (median (range) 9 (2–18) hours and 3 (1–10) hours, respectively), but less so than for the systematic reviews. The median (range) proportion missed records for both approaches was 6 (0–22)%. Conclusion Using Abstrackr to assist one of two reviewers in systematic reviews saves time with little risk of missing relevant records. Many missed records would be identified via other means.http://link.springer.com/article/10.1186/s12874-020-01031-wSystematic reviewsRapid reviewsMachine learningAutomationEfficiency |
| spellingShingle | Allison Gates Michelle Gates Meghan Sebastianski Samantha Guitard Sarah A. Elliott Lisa Hartling The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews BMC Medical Research Methodology Systematic reviews Rapid reviews Machine learning Automation Efficiency |
| title | The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews |
| title_full | The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews |
| title_fullStr | The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews |
| title_full_unstemmed | The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews |
| title_short | The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews |
| title_sort | semi automation of title and abstract screening a retrospective exploration of ways to leverage abstrackr s relevance predictions in systematic and rapid reviews |
| topic | Systematic reviews Rapid reviews Machine learning Automation Efficiency |
| url | http://link.springer.com/article/10.1186/s12874-020-01031-w |
| work_keys_str_mv | AT allisongates thesemiautomationoftitleandabstractscreeningaretrospectiveexplorationofwaystoleverageabstrackrsrelevancepredictionsinsystematicandrapidreviews AT michellegates thesemiautomationoftitleandabstractscreeningaretrospectiveexplorationofwaystoleverageabstrackrsrelevancepredictionsinsystematicandrapidreviews AT meghansebastianski thesemiautomationoftitleandabstractscreeningaretrospectiveexplorationofwaystoleverageabstrackrsrelevancepredictionsinsystematicandrapidreviews AT samanthaguitard thesemiautomationoftitleandabstractscreeningaretrospectiveexplorationofwaystoleverageabstrackrsrelevancepredictionsinsystematicandrapidreviews AT sarahaelliott thesemiautomationoftitleandabstractscreeningaretrospectiveexplorationofwaystoleverageabstrackrsrelevancepredictionsinsystematicandrapidreviews AT lisahartling thesemiautomationoftitleandabstractscreeningaretrospectiveexplorationofwaystoleverageabstrackrsrelevancepredictionsinsystematicandrapidreviews AT allisongates semiautomationoftitleandabstractscreeningaretrospectiveexplorationofwaystoleverageabstrackrsrelevancepredictionsinsystematicandrapidreviews AT michellegates semiautomationoftitleandabstractscreeningaretrospectiveexplorationofwaystoleverageabstrackrsrelevancepredictionsinsystematicandrapidreviews AT meghansebastianski semiautomationoftitleandabstractscreeningaretrospectiveexplorationofwaystoleverageabstrackrsrelevancepredictionsinsystematicandrapidreviews AT samanthaguitard semiautomationoftitleandabstractscreeningaretrospectiveexplorationofwaystoleverageabstrackrsrelevancepredictionsinsystematicandrapidreviews AT sarahaelliott semiautomationoftitleandabstractscreeningaretrospectiveexplorationofwaystoleverageabstrackrsrelevancepredictionsinsystematicandrapidreviews AT lisahartling semiautomationoftitleandabstractscreeningaretrospectiveexplorationofwaystoleverageabstrackrsrelevancepredictionsinsystematicandrapidreviews |