Objective To systematically identify, match, and compare treatment effect estimates and study demographic characteristics from observational studies and randomized clinical trials (RCTs) evaluating the same COVID-19 therapeutics, comparators, and outcomes.

Design In this meta-epidemiological study, individual RCTs or meta-analyses of RCTs reported in a BMJ living review directly comparing any of the 3 most frequently studied therapeutic interventions for COVID-19 (hydroxychloroquine- chloroquine, lopinavir-ritonavir, or dexamethasone) were identified for any safety and efficacy outcomes.1 Using theEpistemonikos “Living OVerview of Evidence” evidence database, individual observational studies evaluating the same interventions, comparisons, and outcomes reported in the BMJ review were identified. Treatment effect estimates from observational studies were identified, standardized, and, when possible, meta-analyzed to match individual RCTs or meta-analyses of RCTs with the same interventions, comparisons, and outcomes (ie, matched pairs). The direction and statistical significance (both P < .05 or P ≥ .05) of treatment effect estimates and the distribution of study demographic characteristics from matched pairs were then compared.

Results Seventeen new, independent meta-analyses of observational studies were conducted of hydroxychloroquine- chloroquine, lopinavir-ritonavir, or dexamethasone vs an active or placebo comparator for any safety or efficacy outcomes and were matched and compared with 17 meta- analyses of RCTs reported in the BMJ review. Ten additional matched pairs with only 1 observational study and/or only 1 RCT were identified. Across all 27 matched pairs, 22 included any demographic and clinical data for all individual studies. All 22 matched pairs had studies with overlapping distributions of sex, age, and disease severity. Overall, 21 (78%) of the 27 matched pairs had effect estimates that agreed in terms of direction and statistical significance (Table 42). Higher levels of concordance were observed among the 17 matched pairs consisting of meta-analyses of observational studies and meta-analyses of RCTs (14 82%) than among the 10 matched pairs consisting of only 1 observational study and/or only 1 RCT (7 70%). The 18 matched pairs with relative treatment effect estimates also had higher levels of agreement (16 89%) than the 9 matched pairs with continuous treatment effect estimates (5 56%). Although 37 (80%) of the 46 individual observational studies referenced at least 1 RCT, only 12 (32%) of the 37 relevant RCTs were referenced by at least 1 observational study.



Conclusions More than three-quarters of the matched pairs had treatment effects that were in agreement. Meta-analyses of observational studies and RCTs evaluating therapeutics for the treatment of COVID-19 more often than not have summary treatment effect estimates that are in agreement in terms of direction and statistical significance. Although concerns have been raised about the evidence produced by individual observational studies evaluating therapeutics for COVID-19,2 meta-analyzed evidence from observational studies may complement evidence collected from RCTs.

References 1. Siemieniuk RA, Bartoszko JJ, Ge L, et al. Drug treatments for COVID-19: living systematic review and network meta- analysis. BMJ. 2020;370:m2980. doi:10.1136/bmj.m2980

2. Jung RG, Di Santo P, Clifford C, et al. Methodological quality of COVID-19 clinical research. Nat Commun. 2021;12(1):943. doi:10.1038/s41467-021-21220-5

Conflicts of Interest Disclosures Joseph J. Ross is the US Outreach and Associate Research Editor at The BMJ and currently receives research support through Yale University from Johnson & Johnson to develop methods of clinical trial data sharing, from the Medical Device Innovation Consortium as part of the National Evaluation System for Health Technology, from the US Food and Drug Administration for the Yale–Mayo Clinic Center for Excellence in Regulatory Science and Innovation program (grant U01FD005938), from the Agency for Healthcare Research and Quality (grant R01HS022882), from the National Heart, Lung, and Blood Institute of the National Institutes of Health (grants R01HS025164 and R01HL144644), and from the Laura and John Arnold Foundation to establish the Good Pharma Scorecard at Bioethics International; in addition, Joseph J. Ross is an expert witness at the request of the relator’s attorneys, the Greene Law Firm, in a qui tam suit alleging violations of the False Claims Act and Anti-Kickback Statute against Biogen Inc. Joshua D. Wallach currently receives research support from the US Food and Drug Administration, the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under award K01AA028258, and through Yale University from Johnson & Johnson to develop methods of clinical trial data sharing. No other disclosures were reported.

Funding/Support Osman Moneer received support from the US Food and Drug Administration through the Yale–Mayo Center for Excellence in Regulatory Science and Innovation Scholars Program.

Role of the Funder/Sponsor The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.