technical paper
Assessment of Concordance Between Reports of Clinical Studies Posted as medRxiv Preprints and Corresponding Publications in Peer Reviewed Journals
keywords:
dissemination of information
preprints
peer review
Objective The study characteristics, results, and
interpretations described in preprints of clinical studies that
are subsequently published in high-impact journals are
broadly concordant.1. Given that studies published in
high-impact journals may represent the highest-quality
studies, it is necessary to evaluate concordance for a larger
sample of clinical studies posted as preprints and
subsequently published in any journal, regardless of impact
factor (IF).
Design In this cross-sectional analysis, preprints posted on
medRxiv in September 2020 were identified. Four evaluators
determined how many preprints were subsequently published
in peer-reviewed journals as of March 2022, calculating the
time from preprint posting to publication. For preprints with
multiple versions, the most recent version prior to journal
acceptance was selected. Preprints updated after journal
acceptance were excluded. For preprint-journal article pairs
describing clinical trials, observational studies, and meta-
analyses that measured health-related outcomes, sample size,
primary end points, corresponding results, and overarching
conclusions were abstracted. Results from primary end points
were considered concordant only if they contained the same
information or had numerical equivalence (eg, identical effect
size estimates and 95% CIs or P values from inferential
analyses). Rates of concordance were compared between
preprints and corresponding journal articles overall and by
focus on COVID-19 and journal IF.
Results Among 1399 preprints first posted on medRxiv in
September 2020, there were 623 modeling studies (44.5%),
280 observational studies (20.0%), 90 systematic reviews or
meta-analyses (6.4%), 42 clinical trials (3.0%), and 364
articles with other study designs (26.0%). As of March 2022,
there were 680 preprints (48.6%) published a median (IQR)
of 5 (3-7) months after preprint posting. Among 331 preprint-
publication pairs describing clinical trials, observational
studies, or meta-analyses, 182 pairs (55.0%) were related to
COVID-19. Of 325 pairs reporting sample sizes in both
sources, 290 pairs (89.2%) were concordant. Of 35 pairs with
discordant sample sizes, 20 pairs (57.1%) had larger samples
in the journal publication. There were 328 pairs (99.1%) with
concordant and 3 pairs (0.8%) with discordant primary end
points. Among 329 pairs in which results for primary end
points could be compared, 290 pairs (88.1%) were
concordant. Two-thirds of 39 discordant pairs (26 pairs
66.7%) had effect size estimates in the same direction and
were statistically consistent. Overall, 323 pairs (97.6%) had
concordant study interpretations, including 32 of 39 pairs
with discordant primary end point results (82.1%). Pairs with
corresponding publications in journals with an IF of 10 or
higher had lower concordance rates for sample size (17 of 24
pairs 70.8% vs 235 of 258 pairs 91.1%; P = .01) and results
(16 of 23 pairs 70.0% vs 235 of 261 pairs 90.0%; P = .004).
Conclusions Most clinical studies posted as preprints on
medRxiv and subsequently published in peer-reviewed
journals had concordant study characteristics, results, and
interpretations, similar to what has been previously observed
among preprints published in the highest-impact journals.1
Reference
1. Shi X, Ross JS, Amancharla N, Niforatos JD, Krumholz
HM, Wallach JD. Assessment of concordance and
discordance among clinical studies posted as preprints and
subsequently published in high-impact journals. JAMA Netw
Open. 2021;4(3):e212110. doi:10.1001/
jamanetworkopen.2021.2110
Conflict of Interest Disclosures Guneet Janda receives
research funding from the Yale-Mayo Clinic Center for Excellence
in Regulatory Science and Innovation (CERSI) Scholars Award
Program. Vishal Khetpal serves in an advisory role for Necessary
Ventures. Xiaoting Shi is supported by the China Scholarship
Council. Joseph S. Ross is a former associate editor of JAMA
Internal Medicine and a current research editor at The BMJ;
receives research support through Yale University from Johnson
& Johnson to develop methods of clinical trial data sharing, the
Medical Device Innovation Consortium as part of the National
Evaluation System for Health Technology, the US Food and Drug
Administration (FDA) for CERSI, the Agency for Healthcare
Research and Quality, the National Heart, Lung, and Blood Institute
of the National Institutes of Health (NIH), and the Laura and John
Arnold Foundation to establish the Good Pharma Scorecard at
Bioethics International; and is an expert witness at the request of
relator’s attorneys, the Greene Law Firm, in a qui tam suit alleging
violations of the False Claims Act and the Anti-Kickback Statute
against Biogen. Joseph S. Ross reported cofounding medRxiv, the
preprint server for health sciences research, and Joshua D. Wallach
reported being a medRxiv affiliate; these individuals did not receive
compensation for their roles. Joshua D. Wallach is supported by the
National Institute on Alcohol Abuse and Alcoholism of the NIH.