Using public clinical trial reports to probe non-experimental causal inference methods
Abstract Background Non-experimental studies (also known as observational studies) are valuable for estimating the effects of various medical interventions, but are notoriously difficult to evaluate because the methods used in non-experimental studies require untestable assumptions. This lack of int...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-09-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-023-02025-0 |
| _version_ | 1797558780606545920 |
|---|---|
| author | Ethan Steinberg Nikolaos Ignatiadis Steve Yadlowsky Yizhe Xu Nigam Shah |
| author_facet | Ethan Steinberg Nikolaos Ignatiadis Steve Yadlowsky Yizhe Xu Nigam Shah |
| author_sort | Ethan Steinberg |
| collection | DOAJ |
| description | Abstract Background Non-experimental studies (also known as observational studies) are valuable for estimating the effects of various medical interventions, but are notoriously difficult to evaluate because the methods used in non-experimental studies require untestable assumptions. This lack of intrinsic verifiability makes it difficult both to compare different non-experimental study methods and to trust the results of any particular non-experimental study. Methods We introduce TrialProbe, a data resource and statistical framework for the evaluation of non-experimental methods. We first collect a dataset of pseudo “ground truths” about the relative effects of drugs by using empirical Bayesian techniques to analyze adverse events recorded in public clinical trial reports. We then develop a framework for evaluating non-experimental methods against that ground truth by measuring concordance between the non-experimental effect estimates and the estimates derived from clinical trials. As a demonstration of our approach, we also perform an example methods evaluation between propensity score matching, inverse propensity score weighting, and an unadjusted approach on a large national insurance claims dataset. Results From the 33,701 clinical trial records in our version of the ClinicalTrials.gov dataset, we are able to extract 12,967 unique drug/drug adverse event comparisons to form a ground truth set. During our corresponding methods evaluation, we are able to use that reference set to demonstrate that both propensity score matching and inverse propensity score weighting can produce estimates that have high concordance with clinical trial results and substantially outperform an unadjusted baseline. Conclusions We find that TrialProbe is an effective approach for probing non-experimental study methods, being able to generate large ground truth sets that are able to distinguish how well non-experimental methods perform in real world observational data. |
| first_indexed | 2024-03-10T17:36:13Z |
| format | Article |
| id | doaj.art-1a75e03d4d66424d8d726dd131d0d17a |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-03-10T17:36:13Z |
| publishDate | 2023-09-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-1a75e03d4d66424d8d726dd131d0d17a2023-11-20T09:49:40ZengBMCBMC Medical Research Methodology1471-22882023-09-0123111310.1186/s12874-023-02025-0Using public clinical trial reports to probe non-experimental causal inference methodsEthan Steinberg0Nikolaos Ignatiadis1Steve Yadlowsky2Yizhe Xu3Nigam Shah4Center for Biomedical Informatics Research, Stanford UniversityDepartment of Statistics, University of ChicagoGoogle Research, GoogleCenter for Biomedical Informatics Research, Stanford UniversityCenter for Biomedical Informatics Research, Stanford UniversityAbstract Background Non-experimental studies (also known as observational studies) are valuable for estimating the effects of various medical interventions, but are notoriously difficult to evaluate because the methods used in non-experimental studies require untestable assumptions. This lack of intrinsic verifiability makes it difficult both to compare different non-experimental study methods and to trust the results of any particular non-experimental study. Methods We introduce TrialProbe, a data resource and statistical framework for the evaluation of non-experimental methods. We first collect a dataset of pseudo “ground truths” about the relative effects of drugs by using empirical Bayesian techniques to analyze adverse events recorded in public clinical trial reports. We then develop a framework for evaluating non-experimental methods against that ground truth by measuring concordance between the non-experimental effect estimates and the estimates derived from clinical trials. As a demonstration of our approach, we also perform an example methods evaluation between propensity score matching, inverse propensity score weighting, and an unadjusted approach on a large national insurance claims dataset. Results From the 33,701 clinical trial records in our version of the ClinicalTrials.gov dataset, we are able to extract 12,967 unique drug/drug adverse event comparisons to form a ground truth set. During our corresponding methods evaluation, we are able to use that reference set to demonstrate that both propensity score matching and inverse propensity score weighting can produce estimates that have high concordance with clinical trial results and substantially outperform an unadjusted baseline. Conclusions We find that TrialProbe is an effective approach for probing non-experimental study methods, being able to generate large ground truth sets that are able to distinguish how well non-experimental methods perform in real world observational data.https://doi.org/10.1186/s12874-023-02025-0Causal inferenceMeta-analysisClinical trialsMethod evaluation |
| spellingShingle | Ethan Steinberg Nikolaos Ignatiadis Steve Yadlowsky Yizhe Xu Nigam Shah Using public clinical trial reports to probe non-experimental causal inference methods BMC Medical Research Methodology Causal inference Meta-analysis Clinical trials Method evaluation |
| title | Using public clinical trial reports to probe non-experimental causal inference methods |
| title_full | Using public clinical trial reports to probe non-experimental causal inference methods |
| title_fullStr | Using public clinical trial reports to probe non-experimental causal inference methods |
| title_full_unstemmed | Using public clinical trial reports to probe non-experimental causal inference methods |
| title_short | Using public clinical trial reports to probe non-experimental causal inference methods |
| title_sort | using public clinical trial reports to probe non experimental causal inference methods |
| topic | Causal inference Meta-analysis Clinical trials Method evaluation |
| url | https://doi.org/10.1186/s12874-023-02025-0 |
| work_keys_str_mv | AT ethansteinberg usingpublicclinicaltrialreportstoprobenonexperimentalcausalinferencemethods AT nikolaosignatiadis usingpublicclinicaltrialreportstoprobenonexperimentalcausalinferencemethods AT steveyadlowsky usingpublicclinicaltrialreportstoprobenonexperimentalcausalinferencemethods AT yizhexu usingpublicclinicaltrialreportstoprobenonexperimentalcausalinferencemethods AT nigamshah usingpublicclinicaltrialreportstoprobenonexperimentalcausalinferencemethods |