An empirical evaluation of the impact scenario of pooling bodies of evidence from randomized controlled trials and cohort studies in medical research
Abstract Background Randomized controlled trials (RCTs) and cohort studies are the most common study design types used to assess treatment effects of medical interventions. We aimed to hypothetically pool bodies of evidence (BoE) from RCTs with matched BoE from cohort studies included in the same sy...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2022-10-01
|
| Series: | BMC Medicine |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12916-022-02559-y |
| _version_ | 1811257054805884928 |
|---|---|
| author | Nils Bröckelmann Julia Stadelmaier Louisa Harms Charlotte Kubiak Jessica Beyerbach Martin Wolkewitz Jörg J. Meerpohl Lukas Schwingshackl |
| author_facet | Nils Bröckelmann Julia Stadelmaier Louisa Harms Charlotte Kubiak Jessica Beyerbach Martin Wolkewitz Jörg J. Meerpohl Lukas Schwingshackl |
| author_sort | Nils Bröckelmann |
| collection | DOAJ |
| description | Abstract Background Randomized controlled trials (RCTs) and cohort studies are the most common study design types used to assess treatment effects of medical interventions. We aimed to hypothetically pool bodies of evidence (BoE) from RCTs with matched BoE from cohort studies included in the same systematic review. Methods BoE derived from systematic reviews of RCTs and cohort studies published in the 13 medical journals with the highest impact factor were considered. We re-analyzed effect estimates of the included systematic reviews by pooling BoE from RCTs with BoE from cohort studies using random and common effects models. We evaluated statistical heterogeneity, 95% prediction intervals, weight of BoE from RCTs to the pooled estimate, and whether integration of BoE from cohort studies modified the conclusion from BoE of RCTs. Results Overall, 118 BoE-pairs based on 653 RCTs and 804 cohort studies were pooled. By pooling BoE from RCTs and cohort studies with a random effects model, for 61 (51.7%) out of 118 BoE-pairs, the 95% confidence interval (CI) excludes no effect. By pooling BoE from RCTs and cohort studies, the median I 2 was 48%, and the median contributed percentage weight of RCTs to the pooled estimates was 40%. The direction of effect between BoE from RCTs and pooled effect estimates was mainly concordant (79.7%). The integration of BoE from cohort studies modified the conclusion (by examining the 95% CI) from BoE of RCTs in 32 (27%) of the 118 BoE-pairs, but the direction of effect was mainly concordant (88%). Conclusions Our findings provide insights for the potential impact of pooling both BoE in systematic reviews. In medical research, it is often important to rely on both evidence of RCTs and cohort studies to get a whole picture of an investigated intervention-disease association. A decision for or against pooling different study designs should also always take into account, for example, PI/ECO similarity, risk of bias, coherence of effect estimates, and also the trustworthiness of the evidence. Overall, there is a need for more research on the influence of those issues on potential pooling. |
| first_indexed | 2024-04-12T17:50:35Z |
| format | Article |
| id | doaj.art-1b44874cb7924b8dbe000029cf524971 |
| institution | Directory Open Access Journal |
| issn | 1741-7015 |
| language | English |
| last_indexed | 2024-04-12T17:50:35Z |
| publishDate | 2022-10-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medicine |
| spelling | doaj.art-1b44874cb7924b8dbe000029cf5249712022-12-22T03:22:30ZengBMCBMC Medicine1741-70152022-10-0120112210.1186/s12916-022-02559-yAn empirical evaluation of the impact scenario of pooling bodies of evidence from randomized controlled trials and cohort studies in medical researchNils Bröckelmann0Julia Stadelmaier1Louisa Harms2Charlotte Kubiak3Jessica Beyerbach4Martin Wolkewitz5Jörg J. Meerpohl6Lukas Schwingshackl7Institute for Evidence in Medicine, Medical Center - University of Freiburg, Faculty of Medicine, University of FreiburgInstitute for Evidence in Medicine, Medical Center - University of Freiburg, Faculty of Medicine, University of FreiburgInstitute for Evidence in Medicine, Medical Center - University of Freiburg, Faculty of Medicine, University of FreiburgInstitute for Evidence in Medicine, Medical Center - University of Freiburg, Faculty of Medicine, University of FreiburgInstitute for Evidence in Medicine, Medical Center - University of Freiburg, Faculty of Medicine, University of FreiburgInstitute of Medical Biometry and Statistics, Faculty of Medicine and Medical Center - University of FreiburgInstitute for Evidence in Medicine, Medical Center - University of Freiburg, Faculty of Medicine, University of FreiburgInstitute for Evidence in Medicine, Medical Center - University of Freiburg, Faculty of Medicine, University of FreiburgAbstract Background Randomized controlled trials (RCTs) and cohort studies are the most common study design types used to assess treatment effects of medical interventions. We aimed to hypothetically pool bodies of evidence (BoE) from RCTs with matched BoE from cohort studies included in the same systematic review. Methods BoE derived from systematic reviews of RCTs and cohort studies published in the 13 medical journals with the highest impact factor were considered. We re-analyzed effect estimates of the included systematic reviews by pooling BoE from RCTs with BoE from cohort studies using random and common effects models. We evaluated statistical heterogeneity, 95% prediction intervals, weight of BoE from RCTs to the pooled estimate, and whether integration of BoE from cohort studies modified the conclusion from BoE of RCTs. Results Overall, 118 BoE-pairs based on 653 RCTs and 804 cohort studies were pooled. By pooling BoE from RCTs and cohort studies with a random effects model, for 61 (51.7%) out of 118 BoE-pairs, the 95% confidence interval (CI) excludes no effect. By pooling BoE from RCTs and cohort studies, the median I 2 was 48%, and the median contributed percentage weight of RCTs to the pooled estimates was 40%. The direction of effect between BoE from RCTs and pooled effect estimates was mainly concordant (79.7%). The integration of BoE from cohort studies modified the conclusion (by examining the 95% CI) from BoE of RCTs in 32 (27%) of the 118 BoE-pairs, but the direction of effect was mainly concordant (88%). Conclusions Our findings provide insights for the potential impact of pooling both BoE in systematic reviews. In medical research, it is often important to rely on both evidence of RCTs and cohort studies to get a whole picture of an investigated intervention-disease association. A decision for or against pooling different study designs should also always take into account, for example, PI/ECO similarity, risk of bias, coherence of effect estimates, and also the trustworthiness of the evidence. Overall, there is a need for more research on the influence of those issues on potential pooling.https://doi.org/10.1186/s12916-022-02559-yGeneral medicinePoolingMeta-analysisCohort studiesRandomized controlled trials |
| spellingShingle | Nils Bröckelmann Julia Stadelmaier Louisa Harms Charlotte Kubiak Jessica Beyerbach Martin Wolkewitz Jörg J. Meerpohl Lukas Schwingshackl An empirical evaluation of the impact scenario of pooling bodies of evidence from randomized controlled trials and cohort studies in medical research BMC Medicine General medicine Pooling Meta-analysis Cohort studies Randomized controlled trials |
| title | An empirical evaluation of the impact scenario of pooling bodies of evidence from randomized controlled trials and cohort studies in medical research |
| title_full | An empirical evaluation of the impact scenario of pooling bodies of evidence from randomized controlled trials and cohort studies in medical research |
| title_fullStr | An empirical evaluation of the impact scenario of pooling bodies of evidence from randomized controlled trials and cohort studies in medical research |
| title_full_unstemmed | An empirical evaluation of the impact scenario of pooling bodies of evidence from randomized controlled trials and cohort studies in medical research |
| title_short | An empirical evaluation of the impact scenario of pooling bodies of evidence from randomized controlled trials and cohort studies in medical research |
| title_sort | empirical evaluation of the impact scenario of pooling bodies of evidence from randomized controlled trials and cohort studies in medical research |
| topic | General medicine Pooling Meta-analysis Cohort studies Randomized controlled trials |
| url | https://doi.org/10.1186/s12916-022-02559-y |
| work_keys_str_mv | AT nilsbrockelmann anempiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT juliastadelmaier anempiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT louisaharms anempiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT charlottekubiak anempiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT jessicabeyerbach anempiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT martinwolkewitz anempiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT jorgjmeerpohl anempiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT lukasschwingshackl anempiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT nilsbrockelmann empiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT juliastadelmaier empiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT louisaharms empiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT charlottekubiak empiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT jessicabeyerbach empiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT martinwolkewitz empiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT jorgjmeerpohl empiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch AT lukasschwingshackl empiricalevaluationoftheimpactscenarioofpoolingbodiesofevidencefromrandomizedcontrolledtrialsandcohortstudiesinmedicalresearch |