Discrepancies in meta-analyses answering the same clinical question were hard to explain: a meta-epidemiological study
Objectives To systematically explore the methodological factors underpinning discrepancies in the pooled effect estimates from Cochrane reviews (CRs) and non-Cochrane reviews (NCRs) systematic reviews, answering the same clinical question. <br></br>Study Design and Setting Quantitative...
| Main Authors: | , |
|---|---|
| Format: | Journal article |
| Language: | English |
| Published: |
Elsevier
2019
|
| _version_ | 1797102243428696064 |
|---|---|
| author | Hacke, C Nunan, D |
| author_facet | Hacke, C Nunan, D |
| author_sort | Hacke, C |
| collection | OXFORD |
| description | Objectives
To systematically explore the methodological factors underpinning discrepancies in the pooled effect estimates from Cochrane reviews (CRs) and non-Cochrane reviews (NCRs) systematic reviews, answering the same clinical question.
<br></br>Study Design and Setting
Quantitative and qualitative analysis of concordance in effect estimates between meta-analyses from CR and NCR matched on population, intervention, condition, and outcome.
<br></br>Results
We identified 24 matched meta-analyses from 24 CR to 20 NCR reviews (545 randomized controlled trials [RCTs]). Compared to their CR matched-pair, pooled effects from NCR were the same in only one pair, were on average 0.12 log units (13%) higher (P = 0.012), and had a greater than twofold larger effect size in four matched-pairs. Two-thirds of CR (15/24, 70.8%) and 0/20 (0%) NCR were rated to have moderate to high confidence in their results (AMSTAR 2). Differences in pre-defined methods, including search strategy, eligibility criteria, and performance of dual screening, could explain mismatches in included studies. Disagreements in the interpretation of eligibility criteria were identified as reasons underpinning discrepant findings in 14 pairs. 23/24 meta-analyses included at least one study of its match. Only two pairs agreed on the numerical data presented for the same studies. An assessment of 50% of discrepant studies (n = 45) showed that reasons for differences in extracted data could be identified in 15 studies.
<br></br>Conclusion
On average, meta-analyses from NCR reported higher effect estimates compared with meta-analyses from CR answering the same clinical question. Methodological and author judgments and performance are key aspects underpinning poor overlap of included studies and discrepancies in reported effect estimates. The potential impacts on health care policy and clinical practice are far-reaching but still remain unknown. Reinforcing awareness and scrutiny of application of reporting guidelines and improvements in protocol registration are needed. |
| first_indexed | 2024-03-07T06:03:13Z |
| format | Journal article |
| id | oxford-uuid:ecedb76f-0b76-4a94-b97b-b2b6544b48b0 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T06:03:13Z |
| publishDate | 2019 |
| publisher | Elsevier |
| record_format | dspace |
| spelling | oxford-uuid:ecedb76f-0b76-4a94-b97b-b2b6544b48b02022-03-27T11:21:10ZDiscrepancies in meta-analyses answering the same clinical question were hard to explain: a meta-epidemiological studyJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:ecedb76f-0b76-4a94-b97b-b2b6544b48b0EnglishSymplectic ElementsElsevier 2019Hacke, CNunan, DObjectives To systematically explore the methodological factors underpinning discrepancies in the pooled effect estimates from Cochrane reviews (CRs) and non-Cochrane reviews (NCRs) systematic reviews, answering the same clinical question. <br></br>Study Design and Setting Quantitative and qualitative analysis of concordance in effect estimates between meta-analyses from CR and NCR matched on population, intervention, condition, and outcome. <br></br>Results We identified 24 matched meta-analyses from 24 CR to 20 NCR reviews (545 randomized controlled trials [RCTs]). Compared to their CR matched-pair, pooled effects from NCR were the same in only one pair, were on average 0.12 log units (13%) higher (P = 0.012), and had a greater than twofold larger effect size in four matched-pairs. Two-thirds of CR (15/24, 70.8%) and 0/20 (0%) NCR were rated to have moderate to high confidence in their results (AMSTAR 2). Differences in pre-defined methods, including search strategy, eligibility criteria, and performance of dual screening, could explain mismatches in included studies. Disagreements in the interpretation of eligibility criteria were identified as reasons underpinning discrepant findings in 14 pairs. 23/24 meta-analyses included at least one study of its match. Only two pairs agreed on the numerical data presented for the same studies. An assessment of 50% of discrepant studies (n = 45) showed that reasons for differences in extracted data could be identified in 15 studies. <br></br>Conclusion On average, meta-analyses from NCR reported higher effect estimates compared with meta-analyses from CR answering the same clinical question. Methodological and author judgments and performance are key aspects underpinning poor overlap of included studies and discrepancies in reported effect estimates. The potential impacts on health care policy and clinical practice are far-reaching but still remain unknown. Reinforcing awareness and scrutiny of application of reporting guidelines and improvements in protocol registration are needed. |
| spellingShingle | Hacke, C Nunan, D Discrepancies in meta-analyses answering the same clinical question were hard to explain: a meta-epidemiological study |
| title | Discrepancies in meta-analyses answering the same clinical question were hard to explain: a meta-epidemiological study |
| title_full | Discrepancies in meta-analyses answering the same clinical question were hard to explain: a meta-epidemiological study |
| title_fullStr | Discrepancies in meta-analyses answering the same clinical question were hard to explain: a meta-epidemiological study |
| title_full_unstemmed | Discrepancies in meta-analyses answering the same clinical question were hard to explain: a meta-epidemiological study |
| title_short | Discrepancies in meta-analyses answering the same clinical question were hard to explain: a meta-epidemiological study |
| title_sort | discrepancies in meta analyses answering the same clinical question were hard to explain a meta epidemiological study |
| work_keys_str_mv | AT hackec discrepanciesinmetaanalysesansweringthesameclinicalquestionwerehardtoexplainametaepidemiologicalstudy AT nunand discrepanciesinmetaanalysesansweringthesameclinicalquestionwerehardtoexplainametaepidemiologicalstudy |