Can electronic search engines optimize screening of search results in systematic reviews: an empirical study
<p>Abstract</p> <p>Background</p> <p>Most electronic search efforts directed at identifying primary studies for inclusion in systematic reviews rely on the optimal Boolean search features of search interfaces such as DIALOG<sup>® </sup>and Ovid™. Our objecti...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2006-02-01
|
| Series: | BMC Medical Research Methodology |
| Online Access: | http://www.biomedcentral.com/1471-2288/6/7 |
| _version_ | 1818758574556315648 |
|---|---|
| author | Clifford Tammy J Moher David Barrowman Nicholas J Sampson Margaret Platt Robert W Morrison Andra Klassen Terry P Zhang Li |
| author_facet | Clifford Tammy J Moher David Barrowman Nicholas J Sampson Margaret Platt Robert W Morrison Andra Klassen Terry P Zhang Li |
| author_sort | Clifford Tammy J |
| collection | DOAJ |
| description | <p>Abstract</p> <p>Background</p> <p>Most electronic search efforts directed at identifying primary studies for inclusion in systematic reviews rely on the optimal Boolean search features of search interfaces such as DIALOG<sup>® </sup>and Ovid™. Our objective is to test the ability of an Ultraseek<sup>® </sup>search engine to rank MEDLINE<sup>® </sup>records of the included studies of Cochrane reviews within the top half of all the records retrieved by the Boolean MEDLINE search used by the reviewers.</p> <p>Methods</p> <p>Collections were created using the MEDLINE bibliographic records of included and excluded studies listed in the review and all records retrieved by the MEDLINE search. Records were converted to individual HTML files. Collections of records were indexed and searched through a statistical search engine, Ultraseek, using review-specific search terms. Our data sources, systematic reviews published in the Cochrane library, were included if they reported using at least one phase of the Cochrane Highly Sensitive Search Strategy (HSSS), provided citations for both included and excluded studies and conducted a meta-analysis using a binary outcome measure. Reviews were selected if they yielded between 1000–6000 records when the MEDLINE search strategy was replicated.</p> <p>Results</p> <p>Nine Cochrane reviews were included. Included studies within the Cochrane reviews were found within the first 500 retrieved studies more often than would be expected by chance. Across all reviews, recall of included studies into the top 500 was 0.70. There was no statistically significant difference in ranking when comparing included studies with just the subset of excluded studies listed as excluded in the published review.</p> <p>Conclusion</p> <p>The relevance ranking provided by the search engine was better than expected by chance and shows promise for the preliminary evaluation of large results from Boolean searches. A statistical search engine does not appear to be able to make fine discriminations concerning the relevance of bibliographic records that have been pre-screened by systematic reviewers.</p> |
| first_indexed | 2024-12-18T06:29:00Z |
| format | Article |
| id | doaj.art-a77aa60138bb495287e058529694762c |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-12-18T06:29:00Z |
| publishDate | 2006-02-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-a77aa60138bb495287e058529694762c2022-12-21T21:17:58ZengBMCBMC Medical Research Methodology1471-22882006-02-0161710.1186/1471-2288-6-7Can electronic search engines optimize screening of search results in systematic reviews: an empirical studyClifford Tammy JMoher DavidBarrowman Nicholas JSampson MargaretPlatt Robert WMorrison AndraKlassen Terry PZhang Li<p>Abstract</p> <p>Background</p> <p>Most electronic search efforts directed at identifying primary studies for inclusion in systematic reviews rely on the optimal Boolean search features of search interfaces such as DIALOG<sup>® </sup>and Ovid™. Our objective is to test the ability of an Ultraseek<sup>® </sup>search engine to rank MEDLINE<sup>® </sup>records of the included studies of Cochrane reviews within the top half of all the records retrieved by the Boolean MEDLINE search used by the reviewers.</p> <p>Methods</p> <p>Collections were created using the MEDLINE bibliographic records of included and excluded studies listed in the review and all records retrieved by the MEDLINE search. Records were converted to individual HTML files. Collections of records were indexed and searched through a statistical search engine, Ultraseek, using review-specific search terms. Our data sources, systematic reviews published in the Cochrane library, were included if they reported using at least one phase of the Cochrane Highly Sensitive Search Strategy (HSSS), provided citations for both included and excluded studies and conducted a meta-analysis using a binary outcome measure. Reviews were selected if they yielded between 1000–6000 records when the MEDLINE search strategy was replicated.</p> <p>Results</p> <p>Nine Cochrane reviews were included. Included studies within the Cochrane reviews were found within the first 500 retrieved studies more often than would be expected by chance. Across all reviews, recall of included studies into the top 500 was 0.70. There was no statistically significant difference in ranking when comparing included studies with just the subset of excluded studies listed as excluded in the published review.</p> <p>Conclusion</p> <p>The relevance ranking provided by the search engine was better than expected by chance and shows promise for the preliminary evaluation of large results from Boolean searches. A statistical search engine does not appear to be able to make fine discriminations concerning the relevance of bibliographic records that have been pre-screened by systematic reviewers.</p>http://www.biomedcentral.com/1471-2288/6/7 |
| spellingShingle | Clifford Tammy J Moher David Barrowman Nicholas J Sampson Margaret Platt Robert W Morrison Andra Klassen Terry P Zhang Li Can electronic search engines optimize screening of search results in systematic reviews: an empirical study BMC Medical Research Methodology |
| title | Can electronic search engines optimize screening of search results in systematic reviews: an empirical study |
| title_full | Can electronic search engines optimize screening of search results in systematic reviews: an empirical study |
| title_fullStr | Can electronic search engines optimize screening of search results in systematic reviews: an empirical study |
| title_full_unstemmed | Can electronic search engines optimize screening of search results in systematic reviews: an empirical study |
| title_short | Can electronic search engines optimize screening of search results in systematic reviews: an empirical study |
| title_sort | can electronic search engines optimize screening of search results in systematic reviews an empirical study |
| url | http://www.biomedcentral.com/1471-2288/6/7 |
| work_keys_str_mv | AT cliffordtammyj canelectronicsearchenginesoptimizescreeningofsearchresultsinsystematicreviewsanempiricalstudy AT moherdavid canelectronicsearchenginesoptimizescreeningofsearchresultsinsystematicreviewsanempiricalstudy AT barrowmannicholasj canelectronicsearchenginesoptimizescreeningofsearchresultsinsystematicreviewsanempiricalstudy AT sampsonmargaret canelectronicsearchenginesoptimizescreeningofsearchresultsinsystematicreviewsanempiricalstudy AT plattrobertw canelectronicsearchenginesoptimizescreeningofsearchresultsinsystematicreviewsanempiricalstudy AT morrisonandra canelectronicsearchenginesoptimizescreeningofsearchresultsinsystematicreviewsanempiricalstudy AT klassenterryp canelectronicsearchenginesoptimizescreeningofsearchresultsinsystematicreviewsanempiricalstudy AT zhangli canelectronicsearchenginesoptimizescreeningofsearchresultsinsystematicreviewsanempiricalstudy |