Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological study
Abstract Objective To investigate the usefulness and performance metrics of three freely-available softwares (Rayyan®, Abstrackr® and Colandr®) for title screening in systematic reviews. Study design and setting In this methodological study, the usefulness of softwares to screen titles in systematic...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-04-01
|
| Series: | Systematic Reviews |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s13643-023-02231-3 |
| _version_ | 1827966421684977664 |
|---|---|
| author | Ana Helena Salles dos Reis Ana Luiza Miranda de Oliveira Carolina Fritsch James Zouch Paulo Ferreira Janaine Cunha Polese |
| author_facet | Ana Helena Salles dos Reis Ana Luiza Miranda de Oliveira Carolina Fritsch James Zouch Paulo Ferreira Janaine Cunha Polese |
| author_sort | Ana Helena Salles dos Reis |
| collection | DOAJ |
| description | Abstract Objective To investigate the usefulness and performance metrics of three freely-available softwares (Rayyan®, Abstrackr® and Colandr®) for title screening in systematic reviews. Study design and setting In this methodological study, the usefulness of softwares to screen titles in systematic reviews was investigated by the comparison between the number of titles identified by software-assisted screening and those by manual screening using a previously published systematic review. To test the performance metrics, sensitivity, specificity, false negative rate, proportion missed, workload and timing savings were calculated. A purposely built survey was used to evaluate the rater's experiences regarding the softwares’ performances. Results Rayyan® was the most sensitive software and raters correctly identified 78% of the true positives. All three softwares were specific and raters correctly identified 99% of the true negatives. They also had similar values for precision, proportion missed, workload and timing savings. Rayyan®, Abstrackr® and Colandr® had 21%, 39% and 34% of false negatives rates, respectively. Rayyan presented the best performance (35/40) according to the raters. Conclusion Rayyan®, Abstrackr® and Colandr® are useful tools and provided good metric performance results for systematic title screening. Rayyan® appears to be the best ranked on the quantitative and on the raters’ perspective evaluation. The most important finding of this study is that the use of software to screen titles does not remove any title that would meet the inclusion criteria for the final review, being valuable resources to facilitate the screening process. |
| first_indexed | 2024-04-09T17:49:44Z |
| format | Article |
| id | doaj.art-d8b0ffa6e4694cff80877d198792eaa3 |
| institution | Directory Open Access Journal |
| issn | 2046-4053 |
| language | English |
| last_indexed | 2024-04-09T17:49:44Z |
| publishDate | 2023-04-01 |
| publisher | BMC |
| record_format | Article |
| series | Systematic Reviews |
| spelling | doaj.art-d8b0ffa6e4694cff80877d198792eaa32023-04-16T11:07:59ZengBMCSystematic Reviews2046-40532023-04-0112111410.1186/s13643-023-02231-3Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological studyAna Helena Salles dos Reis0Ana Luiza Miranda de Oliveira1Carolina Fritsch2James Zouch3Paulo Ferreira4Janaine Cunha Polese5Post-Graduate Program of Health Sciences, Faculdade Ciências Médicas de Minas GeraisPost-Graduate Program of Health Sciences, Faculdade Ciências Médicas de Minas GeraisFaculty of Medicine and Health, School of Health Sciences, Sydney Musculoskeletal Health, The Kolling Institute, The University of SydneyFaculty of Health Sciences, The University of SydneyFaculty of Health Sciences, The University of SydneyPost-Graduate Program of Health Sciences, Faculdade Ciências Médicas de Minas GeraisAbstract Objective To investigate the usefulness and performance metrics of three freely-available softwares (Rayyan®, Abstrackr® and Colandr®) for title screening in systematic reviews. Study design and setting In this methodological study, the usefulness of softwares to screen titles in systematic reviews was investigated by the comparison between the number of titles identified by software-assisted screening and those by manual screening using a previously published systematic review. To test the performance metrics, sensitivity, specificity, false negative rate, proportion missed, workload and timing savings were calculated. A purposely built survey was used to evaluate the rater's experiences regarding the softwares’ performances. Results Rayyan® was the most sensitive software and raters correctly identified 78% of the true positives. All three softwares were specific and raters correctly identified 99% of the true negatives. They also had similar values for precision, proportion missed, workload and timing savings. Rayyan®, Abstrackr® and Colandr® had 21%, 39% and 34% of false negatives rates, respectively. Rayyan presented the best performance (35/40) according to the raters. Conclusion Rayyan®, Abstrackr® and Colandr® are useful tools and provided good metric performance results for systematic title screening. Rayyan® appears to be the best ranked on the quantitative and on the raters’ perspective evaluation. The most important finding of this study is that the use of software to screen titles does not remove any title that would meet the inclusion criteria for the final review, being valuable resources to facilitate the screening process.https://doi.org/10.1186/s13643-023-02231-3Citation screeningText miningMachine learningSoftware toolsUser Experience |
| spellingShingle | Ana Helena Salles dos Reis Ana Luiza Miranda de Oliveira Carolina Fritsch James Zouch Paulo Ferreira Janaine Cunha Polese Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological study Systematic Reviews Citation screening Text mining Machine learning Software tools User Experience |
| title | Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological study |
| title_full | Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological study |
| title_fullStr | Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological study |
| title_full_unstemmed | Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological study |
| title_short | Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological study |
| title_sort | usefulness of machine learning softwares to screen titles of systematic reviews a methodological study |
| topic | Citation screening Text mining Machine learning Software tools User Experience |
| url | https://doi.org/10.1186/s13643-023-02231-3 |
| work_keys_str_mv | AT anahelenasallesdosreis usefulnessofmachinelearningsoftwarestoscreentitlesofsystematicreviewsamethodologicalstudy AT analuizamirandadeoliveira usefulnessofmachinelearningsoftwarestoscreentitlesofsystematicreviewsamethodologicalstudy AT carolinafritsch usefulnessofmachinelearningsoftwarestoscreentitlesofsystematicreviewsamethodologicalstudy AT jameszouch usefulnessofmachinelearningsoftwarestoscreentitlesofsystematicreviewsamethodologicalstudy AT pauloferreira usefulnessofmachinelearningsoftwarestoscreentitlesofsystematicreviewsamethodologicalstudy AT janainecunhapolese usefulnessofmachinelearningsoftwarestoscreentitlesofsystematicreviewsamethodologicalstudy |