poster
Agreement of Treatment Effect Estimates From Observational Studies and Randomized Clinical Trials Evaluating therapeutics for COVID-19
keywords:
quality of trials
pandemic science
quality of the literature
Objective To systematically identify, match, and compare
treatment effect estimates and study demographic
characteristics from observational studies and randomized
clinical trials (RCTs) evaluating the same COVID-19
therapeutics, comparators, and outcomes.
Design In this meta-epidemiological study, individual RCTs
or meta-analyses of RCTs reported in a BMJ living review
directly comparing any of the 3 most frequently studied
therapeutic interventions for COVID-19 (hydroxychloroquine-
chloroquine, lopinavir-ritonavir, or dexamethasone) were
identified for any safety and efficacy outcomes.1
Using theEpistemonikos “Living OVerview of Evidence” evidence
database, individual observational studies evaluating the
same interventions, comparisons, and outcomes reported in
the BMJ review were identified. Treatment effect estimates
from observational studies were identified, standardized, and,
when possible, meta-analyzed to match individual RCTs or
meta-analyses of RCTs with the same interventions,
comparisons, and outcomes (ie, matched pairs). The direction
and statistical significance (both P < .05 or P ≥ .05) of
treatment effect estimates and the distribution of study
demographic characteristics from matched pairs were then
compared.
Results Seventeen new, independent meta-analyses of
observational studies were conducted of hydroxychloroquine-
chloroquine, lopinavir-ritonavir, or dexamethasone vs an
active or placebo comparator for any safety or efficacy
outcomes and were matched and compared with 17 meta-
analyses of RCTs reported in the BMJ review. Ten additional
matched pairs with only 1 observational study and/or only 1
RCT were identified. Across all 27 matched pairs, 22 included
any demographic and clinical data for all individual studies.
All 22 matched pairs had studies with overlapping
distributions of sex, age, and disease severity. Overall, 21
(78%) of the 27 matched pairs had effect estimates that
agreed in terms of direction and statistical significance
(Table 42). Higher levels of concordance were observed
among the 17 matched pairs consisting of meta-analyses of
observational studies and meta-analyses of RCTs (14 82%)
than among the 10 matched pairs consisting of only 1
observational study and/or only 1 RCT (7 70%). The 18
matched pairs with relative treatment effect estimates also
had higher levels of agreement (16 89%) than the 9 matched
pairs with continuous treatment effect estimates (5 56%).
Although 37 (80%) of the 46 individual observational studies
referenced at least 1 RCT, only 12 (32%) of the 37 relevant
RCTs were referenced by at least 1 observational study.
Conclusions More than three-quarters of the matched pairs
had treatment effects that were in agreement. Meta-analyses
of observational studies and RCTs evaluating therapeutics for
the treatment of COVID-19 more often than not have
summary treatment effect estimates that are in agreement in
terms of direction and statistical significance. Although
concerns have been raised about the evidence produced by
individual observational studies evaluating therapeutics for
COVID-19,2 meta-analyzed evidence from observational
studies may complement evidence collected from RCTs.
References
1. Siemieniuk RA, Bartoszko JJ, Ge L, et al. Drug treatments
for COVID-19: living systematic review and network meta-
analysis. BMJ. 2020;370:m2980. doi:10.1136/bmj.m2980
2. Jung RG, Di Santo P, Clifford C, et al. Methodological
quality of COVID-19 clinical research. Nat Commun.
2021;12(1):943. doi:10.1038/s41467-021-21220-5
Conflicts of Interest Disclosures Joseph J. Ross is the US
Outreach and Associate Research Editor at The BMJ and currently
receives research support through Yale University from Johnson
& Johnson to develop methods of clinical trial data sharing, from
the Medical Device Innovation Consortium as part of the National
Evaluation System for Health Technology, from the US Food
and Drug Administration for the Yale–Mayo Clinic Center for
Excellence in Regulatory Science and Innovation program (grant
U01FD005938), from the Agency for Healthcare Research and
Quality (grant R01HS022882), from the National Heart, Lung,
and Blood Institute of the National Institutes of Health (grants
R01HS025164 and R01HL144644), and from the Laura and John
Arnold Foundation to establish the Good Pharma Scorecard at
Bioethics International; in addition, Joseph J. Ross is an expert
witness at the request of the relator’s attorneys, the Greene Law
Firm, in a qui tam suit alleging violations of the False Claims Act
and Anti-Kickback Statute against Biogen Inc. Joshua D. Wallach
currently receives research support from the US Food and Drug
Administration, the National Institute on Alcohol Abuse and
Alcoholism of the National Institutes of Health under award
K01AA028258, and through Yale University from Johnson &
Johnson to develop methods of clinical trial data sharing. No other
disclosures were reported.
Funding/Support Osman Moneer received support from the US
Food and Drug Administration through the Yale–Mayo Center for
Excellence in Regulatory Science and Innovation Scholars Program.
Role of the Funder/Sponsor The funder had no role in the
design and conduct of the study; collection, management, analysis,
and interpretation of the data; preparation, review, or approval of
the abstract; and decision to submit the abstract for presentation.