United States

Peer Review Congress 2022

Morning Welcome: Theo Bloom | VIDEO

Welcome to Enago See the Future 2021 (Global). To access the complimentary content,[ please register here](https://www.enago.com/see-the-future/conference-2021/register/).

https://www.enago.com/see-the-future/conference-2021/register/

Enago See the Future 2021 (Global - English)

Research publication never slowed down while the Earth stood still as COVID crisscrossed international borders in multiple waves. The numbers of submissions and publications continuously grew while most productions of modern societies dwindled. At the same time, the long-foretold paradigm shift has happened before our eyes. Modes of publication and research collaboration have undergone rapid fundamental changes. In these uncertain times, change and greater openness remain the only constant. See the Future 2021 Virtual Conference will focus on these vital aspects of research publication and once again challenge our preconceptions. 

Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.
 
 **Welcome!**
 
 The JAMA Network, The BMJ, and METRICS welcome you to the Ninth International Congress on Peer Review and Scientific Publication. We have continued our efforts to broaden the scope of the Congress to all aspects of peer review and publication—from funding to postpublication—and to all sciences.
 
 We will have 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination. There are 50 plenary session research presentations and 5 plenary invited talks. Each plenary session research presentation will be followed by equal time for discussion and questions from the audience. In addition, there are 125 posters. All posters, including virtual posters, are available to view throughout the meeting. In-person posters will also be presented at designated times on September 9 and 10. If you are attending virtually, please visit the individual poster pages and ask your questions in the chat box. See the link to Posters on the left.
 
 We hope you will take an active part in the program, as we depend on your participation in the discussion sessions to make the Congress a success.

 [Peer Review Congress Organizers, Directors, and Advisory Board](https://peerreviewcongress.org/organizers-and-advisory-board/)
 
 [Health and Safety Policies](https://web.cvent.com/event/1b505794-6a0b-428c-a44c-6c46c4c50e99/websitePage:152962bb-4567-4561-9d81-21ebfa97f487)

![](https://assets.underline.io/uploads/markdown_image/1/image/7c9f0143dee383780cc4670269ade132.png)

A PDF of the schedule is available [here](https://drive.google.com/file/d/1-Jd191W-nxbWcN1prdX5o9nlSfeo9_-H/view?usp=sharing) and a flipbook version of the complete program and abstracts is included below. See also the link to the interactive schedule to the left.

<iframe width="700px" height="425px" src="https://online.fliphtml5.com/ebtyf/nlfn/#p=1" frameborder="0" allowfullscreen allowtransparency></iframe>

Registration is required to attend this event, please use the URL linked below to secure your ticket. Registration will close on Friday, September 2 at 5:00 PM CT. 

[**Register Here**]( https://web.cvent.com/event/1b505794-6a0b-428c-a44c-6c46c4c50e99/summary)

Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.



**Objective** Diverse efforts are underway to reform the journal
peer review system. In combination with a growing interest in
open science practices, open peer review (OPR) has become
of central concern to multiple stakeholders within the
scholarly communication process. However, what OPR is
understood to encompass and how effective its individual
elements are in meeting the expectations of the peer review
system is uncertain. Through a discussion of the latest
evidence1-3 on uptake and efficacy of these varieties of OPR,
the goal of this talk is to orient the audience on the state of
the art regarding this growing area of peer review innovation.

**Design** This talk will introduce the aims of OPR, especially in
its relation to the broader open science agenda, and through
critical discussion of its key traits (open identities, open
reports, and open participation), and the various ways in
which they can be combined to modify journal review
processes, especially reflecting on progress made in the last
years. Results from various strands of work using both
qualitative and quantitative approaches will be presented.

**Results** The talk will present definitions of OPR, guidelines
for implementation, and critical reflection of the advantages
and disadvantages. The latter will include presentation of first
results from a new systematic review of evidence on OPR
uptake, attitudes, and efficacy to synthesize what we know on
what works in which circumstances, reflect on important
barriers to further implementation, and call for research on
key areas where more evidence is needed.

**Conclusions** Open peer review is growing fast, yet key
questions regarding uptake and efficacy persist. As a
community, we need to be open to these challenges and
commit to shared research to address these questions.

**References**
1. Tennant JP, Ross-Hellauer T. The limitations to our
understanding of peer review. Res Integ Peer Rev. Published
online April 30, 2020. doi:10.1186/s41073-020-00092-1 
 2. Ross-Hellauer T, Görögh E. Guidelines for open peer
review implementation. Res Integ Peer Rev. Published online
February 27, 2019. doi:10.1186/s41073-019-0063-9 
 3. Ross-Hellauer T. What is open peer review? a systematic
review. F1000Research. Published online April 27, 2017.
doi:10.12688/f1000research.11369.2

**Conflict of Interest Disclosures** None reported.

**Acknowledgments** This presentation reports findings from a
range of articles produced by the author between 2017 and 2022
in collaboration with coauthors Serge Horbach, Jon Tennant,
Birgit Schmidt, and Edith Görögh, whose contributions the author
gratefully acknowledges.

![](https://assets.underline.io/uploads/markdown_image/1/image/39d87b98aeff51a20c32a6b4cfbcf869.png)

Peer Review in the Age of Open Science, Tony Ross-Hellauer

**Objective** Most principal investigators support the concept
of data sharing in principle, but few commit to sharing data in
practice.1
One way shown to reduce this gap is if major
stakeholders across the research life cycle implement policies
to recommend or require data sharing. The objective of this
study was to determine the prevalence and characteristics of
data sharing policies. 
 **Design** This was a cross-sectional study of data sharing
policies of health research funders, research ethics
committees, clinical trial registries, peer-reviewed scientific
journals, and research data repositories. It included the 55
largest private and 55 largest public and philanthropic health
research funders by annual health research expenditure, all
national ethics committees, all clinical trial registries, the 5
highest-impact peer-reviewed scientific journals by journal
impact factor for each of the 59 fields of clinical medicine, and
all research data repositories in clinical medicine.
Investigators reviewed all official websites, online reports,
and gray literature information sources of stakeholders for
the presence of a data sharing policy. If present, investigators
assessed its magnitude of support for data sharing. If it
recommended or required data sharing, investigators
assessed its characteristics. All data were abstracted in
duplicate by 2 independent investigators who compared the
relevant information against structured criteria on a
prepiloted data extraction form and resolved disagreements
by discussion and a third investigator. 
 **Results** Overall, 110 health research funders, 124 national
ethics committees, 18 clinical trial registries, 273 peer-
reviewed scientific journals, and 410 research data
repositories were included. More than half of health research
funders either recommended (15 [15%]) or required (45
[41%]) data sharing. These policies typically applied to all
data from only interventional studies, with justified
exceptions, and specified data to be shared before a
predetermined period with independent committee–
approved investigators for research proposal–approved
purposes via third-party websites. Only 4 national ethics
committees (3%) recommended data sharing. These policies
typically applied to all studies, with justified exceptions, and
specified data to be shared via third-party websites. Only 1
clinical trial registry (6%) required data sharing. This policy
applied to only interventional studies, with justified
exceptions, and specified data to be shared via third-party
websites. Almost two-thirds of peer-reviewed scientific
journals either recommended (120 [44%]) or required (52
[19%]) data sharing. These policies typically applied to only
some data from all studies, with unjustified exceptions, and
specified data to be shared with anyone for any purpose via
third-party websites. Few research data repositories
recommended (26 [6%]) or required (24 [6%]) data sharing.
These policies typically applied to all data from all studies,
with unjustified exceptions, and specified data to be shared
with anyone for any purpose and via third-party websites. 
 **Conclusions** Data sharing imperatives were not met by
most stakeholders. 
 **Reference**
1. Tan AC, Askie LM, Hunter KE, Barba A, Simes RJ, Seidler
AL. Data sharing—trialists’ plans at registration, attitudes,
barriers and facilitators: a cohort study and cross-sectional
survey. Res Synth Methods. 2021;12(5):641-657. doi:10.1002/
jrsm.1500 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/b062b29c31f26a7db4f5c577352415ef.png)

Prevalence and Characteristics of Data Sharing Policies Across the Health Research Life Cycle: Funders, Ethics Committees, Trial Registries, Journals, and Data Repositories | VIDEO

**Objective** Numerous metaresearch studies have investigated
rates and predictors of data and code sharing in medicine.
However, these studies have often been narrow in scope,
focusing on some important aspects and predictors of sharing
but not others. A systematic review and individual participant
data (IPD) meta-analysis of this corpus of research is being
conducted to provide an expansive picture of how availability
rates have changed over time in medicine and what factors
are associated with sharing. 
 **Design** Ovid Embase, Ovid MEDLINE, MetaArXiv, medRxiv,
and bioRxiv were searched up to July 1, 2021, for
metaresearch studies that investigated data sharing, code
sharing, or both among a sample of scientific articles
presenting original research from the medical and health
sciences (ie, primary articles). Two authors independently
screened records and assessed risk of bias in the included
studies. Key outcomes of interest included the prevalence of
affirmative sharing declarations (declared availability) and
availability as confirmed by the metaresearch authors (actual
availability). The association between data and code
availability and several factors (eg, year published, journal
policy) were also examined. IPD were collected or requested
from authors of eligible studies. A 2-stage approach to IPD
meta-analysis was performed, with outcomes pooled using
the Hartung-Knapp-Sidik-Jonkman method for random-
effects meta-analysis.1 The review methods were preregistered
on the Open Science Framework 2 and are described in a
detailed review protocol. 3 
 **Results** A total of 4970 potential studies were identified, of
which 101 were eligible for the review, 28 of which did not
publicly share any IPD. Eligible studies examined a median
(IQR) of 203 (125-398) primary articles published between
1987 and 2020 across 32 unique medical disciplines. To date,
data from 36 studies (including 7750 primary articles) have
been processed. Only 1 study was classified as low risk of bias.
Meta-analysis revealed declared and actual data availability
rates of 9% (95% CI, 6%-14%; 23 studies) and 3% (95% CI,
1%-6%; 26 studies), respectively, since 2015, with no
significant differences between rates when compared with the
preceding 5-year period. The same finding was also noted for
code sharing (all <1%). Early results also indicate that only
35% (95% CI, 18%-55%; 5 studies) and 16% (95% CI, 10%-
22%; 2 studies) of authors complied with mandatory data and
code sharing policies, respectively. Comparatively, 13% (95%
CI, 0-37%; 6 studies) and 8% (95% CI, 0-50%; 4 studies) of
authors submitting to journals with policies encouraging
sharing or no policy made data available, respectively. 
 **Conclusions** Preliminary analysis suggests that data and
code sharing in medicine remains uncommon and occurs at a
rate much lower than expected if journal policies were
followed. We recommend future research to explore why
sharing rates and compliance with mandatory policies are low
as well as strategies for how this might be improved. 
 **References**
1. IntHout J, Ioannidis JP, Borm GF. The Hartung-Knapp-
Sidik-Jonkman method for random effects meta-analysis is
straightforward and considerably outperforms the standard
DerSimonian-Laird method. BMC Med Res Methodol.
2014;14(1):25. doi:10.1186/1471-2288-14-25 
 2. Hamilton DG, Fraser H, Fidler F, Rowhani-Farid A, Hong
K, Page MJ. Rates and predictors of data and code sharing in
the medical and health sciences: a systematic review and
individual participant data meta-analysis. Open Science
Framework. May 28, 2021. doi:10.17605/OSF.IO/7SX8U 
 3. Hamilton DG, Fraser H, Fidler F, et al. Rates and
predictors of data and code sharing in the medical and health
sciences: protocol for a systematic review and individual
participant data meta-analysis. F1000Research. 2021;10:491.
doi:10.12688/f1000research.53874.2 
 **Conflict of Interest Disclosures** Daniel G. Hamilton is a board
member of the Association of Interdisciplinary Meta-research
and Open Science (AIMOS) and a PhD candidate supported by
an Australian Commonwealth Government Research Training
Program Scholarship. The Laura and John Arnold Foundation
funds the Restoring Invisible and Abandoned Trials (RIAT) Support
Center, which supports the salary of Kyungwan Hong and Anisa
Rowhani-Farid. Kyungwan Hong was supported in 2020 by the
US Food and Drug Administration (FDA) of the US Department of
Health and Human Services (HHS) as part of a financial assistance
award U01FD005946, funded by FDA/HHS. The project contents
are those of Kyungwan Hong and do not necessarily represent
the official views of, nor an endorsement by, FDA/HHS or the US
government. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/7b810d5d04b00406f2f3bc85aa89d39b.png)

Frequency of Data and Code Sharing in Medical Research: An Individual Participant Data Meta-analysis of Metaresearch Studies | VIDEO

**Objective** The Yale Open Data Access (YODA) Project
enables researchers to access shared participant-level clinical
research data for independent secondary and replication
studies.1
The project, because it requires an application,
provides an opportunity to determine how published analyses
compare with the initial aims and the degree to which any
deviance is noted in the publications. Accordingly, the
objective of this study was to evaluate the concordance among
the included trials, the study objectives, and the statistical
methods specified in researchers’ requests to the YODA
Project for Johnson & Johnson clinical trial data, the primary
YODA data sharing partner, and their corresponding
publications. 
 **Design** In this cross-sectional study, all approved YODA
requests for Johnson & Johnson pharmaceutical or medical
device data that had 1 corresponding English-language
publication or more were identified (from the first request in
2018 to October 29, 2021). From each request-publication
pair, the primary objectives were classified as fully, partially,
or not at all concordant. Primary and secondary end points
were classified as fully concordant, partially concordant (≥ 1
additional primary or secondary end point in the request or
publication), or discordant (≥ 1 secondary end point dropped
or converted to a primary end point, primary end point
converted to a secondary end point, or secondary and primary
end points swapped). Given that slight methodological
changes may have been necessary once researchers had
access to the shared data, statistical methods were classified
as concordant if the pairs described the same broad
methodological approaches. 
 **Results** Forty-eight requests on the YODA Project website
with 1 publication or more in a peer-reviewed journal were
identified. Of the 48 request-publication pairs, 33 (68.8%)
had a fully concordant overarching study objective, and 13
(27.1%) had a partially concordant overarching study
objective (**Table 17**). There were 28 pairs (58.3%) for which
all of the requested trials were included in the analyses
described in the publications; 17 pairs (35.4%) had articles
that included fewer trials than the number of trials specified
in the request. There were 31 pairs (64.6%) with fully
concordant primary end points and 25 pairs (52.1%) with
fully concordant secondary end points. Only 1 pair had fully
concordant primary and secondary end points. Most pairs (39
[81.3%]) had concordant statistical methods; there were no
pairs that were fully concordant across all proposal details. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/22ad7ae763f4548533a3f37f3631572b.png) 
 **Conclusions:** None of the YODA Project requests were fully
concordant with their corresponding publications describing
the completed research, most often because fewer trials were
used than requested. These findings suggest that investigators
using data from data sharing platforms should explain
deviations from the data requests in their publications and
that research reviewers should compare and evaluate the
consistency between the prespecified requests and
publications. 
 **Reference**
1. Ross JS, Waldstreicher J, Bamford S, et al. Overview and
experience of the YODA Project with clinical trial data sharing
after 5 years. Sci Data. 2018;5(1):180268. doi:10.1038/
sdata.2018.268 
 **Conflict of Interest Disclosures** Joseph S. Ross and Harlan M.
Krumholz report being cofounders of the Yale Open Data Access
(YODA) Project, and Cary P. Gross and Joshua D. Wallach report
being YODA project affiliates. Joseph S. Ross is a former associate
editor of JAMA Internal Medicine and a current research editor at
The BMJ and receives research support through Yale University
from Johnson & Johnson to develop methods of clinical trial data
sharing, from the Medical Device Innovation Consortium as part
of the National Evaluation System for Health Technology, from the
US Food and Drug Administration (FDA) for the Yale–Mayo Clinic
Center for Excellence in Regulatory Science and Innovation program
(grant U01FD005938), from the Agency for Healthcare Research
and Quality (grant R01HS022882), from the National Heart, Lung,
and Blood Institute of the National Institutes of Health (NIH)
(grants R01HS025164 and R01HL144644), and from the Laura and
John Arnold Foundation to establish the Good Pharma Scorecard at
Bioethics International; in addition, he is an expert witness at the
request of the relator’s attorneys, the Greene Law Firm, in a qui tam
suit alleging violations of the False Claims Act and Anti-Kickback
Statute against Biogen Inc. Cary P. Gross has received research
funding through Yale University from the National Comprehensive
Cancer Network Foundation (funded by AstraZeneca) and Johnson
& Johnson to help devise and implement new approaches to sharing
clinical trial data and from Genentech. Karla Childers, Stephen
Bamford, and Joanne Waldstreicher are employees and stockholders
of Johnson & Johnson. Harlan M. Krumholz reports that he has
contracts through Yale New Haven Hospital with the Centers for
Medicare & Medicaid Services to support quality measurement
programs and through Yale University with UnitedHealth Group to
engage in collaborative research. He was a recipient of a research
grant through Yale University from Medtronic for data sharing,
from the FDA to develop methods for postmarket surveillance of
medical devices, from Johnson & Johnson to support data sharing,
and from the Shenzhen Center for Health Information for work to
advance intelligent disease prevention and health promotion; he
is an advisor to the National Center for Cardiovascular Diseases
in Beijing, China; was an expert witness for the Arnold & Porter
Law Firm for work related to the Sanofi clopidogrel litigation; and
is an expert witness for the Martin/Baughman Law Firm for work
related to the Cook Celect inferior vena cava (IVC) filter litigation
and related to C. R. Bard Recovery IVC filter litigation and for the
Siegfried and Jensen Law Firm for work related to Vioxx litigation;
he chairs a cardiac scientific advisory board for UnitedHealth; was a
member of the IBM Watson Health Life Sciences Board; is a member
of the advisory board for Element Science, the health care advisory
board for Facebook, and the physician advisory board for Aetna;
he is the cofounder of HugoHealth, a personal health information
platform, and cofounder of Refactor Health, an enterprise health
care artificial intelligence–augmented data management company;
he is a venture partner at F-Prime. Joshua D. Wallach is supported
by the FDA and the National Institute on Alcohol Abuse and
Alcoholism of the NIH under award 1K01AA028258. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/5ee6ce285d0f679f0d85e4b74a285f73.png)

Assessment of Concordance Between Yale Open Data Access (YODA) Project Data Requests and Corresponding Publications

**Objective** Preprint servers offer a means to disseminate
research reports before or concurrent with peer-review.1
medRxiv, an independent, not-for-profit preprint server for
clinical and health science research introduced in June 2019,
grew substantially in the context of the COVID-19 pandemic.2
Submissions, preprints posted, and user downloads for
medRxiv since launch were characterized. 
 **Design** This analysis used data from the medRxiv website,
internal data, and Altmetric.com from June 11, 2019 (launch),
through December 31, 2021. Submissions, postings, abstract
views, downloads, comments, and withdrawals were assessed.
The posting rate was calculated as the percentage of
submissions that were posted after passing screening criteria,
including that the submission represents scientific research
(not a narrative review, commentary, or case report). In
addition, all posted preprints with Altmetric scores greater
than 1000 were identified. Published journal articles
corresponding to posted preprints were identified through
routine, automated searches of PubMed and CrossRef. 
 **Results** As of December 31, 2021, there were 33,342
submissions to medRxiv, 27,674 (83.0%) of which were
subsequently posted after screening: 913 in 2019, 14,070 in
2020, and 12,691 in 2021. Among these, 6165 (22.3%) had
been revised at least once and 4227 (15.3%) were
simultaneously submitted to journals as part of the M2J
program. Overall, 16,465 preprints (59.5%) described
COVID-19 research. In total, 47 posted preprints (0.17%)
were subsequently withdrawn, 30 of which were COVID-19–
related. Preprints have thus far been posted by 156,290
unique authors from 151 countries, most commonly from the
United States, the United Kingdom, and China. As of
December 31, 2021, there were 51,943,342 downloads and
132,900,392 abstract views: 107,772 and 241,528 in 2019;
27,963,915 and 61,613,928 in 2020; and 23,871,655 and
71,044,936 in 2021. The median (IQR) number of downloads
per preprint was 451 (261-904) and abstract views per
preprint was 1659 (1055-2955). There have been 8394 total
user comments on preprints and 2199 preprints (7.9%) have
at least 1 user comment. There were 312 preprints (1.1%) with
an Altmetric score greater than 1000, all but 1 of which was
COVID-19–related; the median (IQR) Altmetric score was 2
(0-11). Thus far, 10,041 preprints (36.3%) were subsequently
published in 2316 peer-reviewed journals, with a median
interval between preprint posting and journal publication of
140 days, including 566 (62.0%), 6615 (47.0%), and 2860
(22.5%) that were published after being posted in 2019, 2020,
and 2021, respectively. 
 **Conclusions** medRxiv grew rapidly since its launch,
particularly for COVID-19–related research. The preprint
server is an active repository for clinical and health science
research; future research should also account for peer
communication through social media. 
 **References**
1. Krumholz HM, Ross JS, Otto CM. Will research preprints
improve healthcare for patients? BMJ. 2018;362:k3628.
doi:10.1136/bmj.k3628 
 2. Krumholz HM, Bloom T, Sever R, et al. Submissions and
downloads of preprints in the first year of medRxiv. JAMA.
2020;324:1903-1905. doi:10.1001/jama.2020.17529 
 **Conflict of Interest Disclosures** All authors disclose being
cofounders of medRxiv. Joseph Ross is a former associate editor of
JAMA Internal Medicine, a current research editor at The BMJ, and
receives research support through Yale University from Johnson
& Johnson to develop methods of clinical trial data sharing, from
the Medical Device Innovation Consortium as part of the National
Evaluation System for Health Technology (NEST), from the Food
and Drug Administration for the Yale-Mayo Clinic Center for
Excellence in Regulatory Science and Innovation (CERSI) program
(U01FD005938), from the Agency for Healthcare Research and
Quality (R01HS022882), from the National Heart, Lung, and Blood
Institute of the National Institutes of Health (NIH) (R01HS025164,
R01HL144644), and from the Laura and John Arnold Foundation
to establish the Good Pharma Scorecard at Bioethics International;
in addition, Joseph Ross is an expert witness at the request of
relator’s attorneys, the Greene Law Firm, in a qui tam suit alleging
violations of the False Claims Act and Anti-Kickback Statute against
Biogen Inc. Richard Sever reports being the assistant director
of Cold Spring Harbor Laboratory Press and a director of Life
Science Alliance LLC. Theodora Bloom is employed full time by
The BMJ; reports chairing the scientific advisory board of EMBL-
EBI Literature Services (http://www.ebi.ac.uk/services/literature),
being on the Board of Managers of AIP Publishing (https://
publishing.aip.org/), and being European coordinator for the Peer
Review Congress (https://peerreviewcongress.org/organizers-and-board.html). Samantha Hindle is cofounder of PREreview,
an initiative to support and train early-career researchers in peer
review using preprints. John R. Inglis reports being the executive
director and publisher of Cold Spring Harbor Laboratory Press, a
director of Life Science Alliance LLC, and a member of the advisory
board of MIT Press. Harlan M. Krumholz reports that he is a
cofounder of Refactor Health and HugoHealth, is associated with
contracts, through Yale New Haven Hospital from the Centers for
Medicare & Medicaid Services and through Yale University from
Johnson & Johnson, and has received expenses and/or personal
fees from UnitedHealth, Element Science, Aetna, Reality Labs,
Tesseract/4Catalyst, the Siegfried and Jensen Law Firm, Arnold
and Porter Law Firm, Martin/Baughman Law Firm, and F-Prime.
Theodora Bloom is a member of the Peer Review Congress Advisory
Board but was not involved in the review or decision for this
abstract. 
 **Funding/Support** This work was supported in part by funds
from Chan Zuckerberg Initiative through the Cold Spring Harbor
Laboratory. 
 **Role of the Funder/Sponsor** The funder had no role in the
design and conduct of the study; collection, management, analysis,
and interpretation of the data; preparation, review, or approval of
the abstract; and decision to submit the abstract for presentation. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/fd2659f5a1cc9c89d56264a9af7e0cee.png)

medRxiv Preprint Submissions, Posts, and Key Metrics, 2019-2021 | VIDEO

**Objective** The study characteristics, results, and
interpretations described in preprints of clinical studies that
are subsequently published in high-impact journals are
broadly concordant.1. Given that studies published in
high-impact journals may represent the highest-quality
studies, it is necessary to evaluate concordance for a larger
sample of clinical studies posted as preprints and
subsequently published in any journal, regardless of impact
factor (IF). 
 **Design** In this cross-sectional analysis, preprints posted on
medRxiv in September 2020 were identified. Four evaluators
determined how many preprints were subsequently published
in peer-reviewed journals as of March 2022, calculating the
time from preprint posting to publication. For preprints with
multiple versions, the most recent version prior to journal
acceptance was selected. Preprints updated after journal
acceptance were excluded. For preprint-journal article pairs
describing clinical trials, observational studies, and meta-
analyses that measured health-related outcomes, sample size,
primary end points, corresponding results, and overarching
conclusions were abstracted. Results from primary end points
were considered concordant only if they contained the same
information or had numerical equivalence (eg, identical effect
size estimates and 95% CIs or P values from inferential
analyses). Rates of concordance were compared between
preprints and corresponding journal articles overall and by
focus on COVID-19 and journal IF. 
 **Results** Among 1399 preprints first posted on medRxiv in
September 2020, there were 623 modeling studies (44.5%),
280 observational studies (20.0%), 90 systematic reviews or
meta-analyses (6.4%), 42 clinical trials (3.0%), and 364
articles with other study designs (26.0%). As of March 2022,
there were 680 preprints (48.6%) published a median (IQR)
of 5 (3-7) months after preprint posting. Among 331 preprint-
publication pairs describing clinical trials, observational
studies, or meta-analyses, 182 pairs (55.0%) were related to
COVID-19. Of 325 pairs reporting sample sizes in both
sources, 290 pairs (89.2%) were concordant. Of 35 pairs with
discordant sample sizes, 20 pairs (57.1%) had larger samples
in the journal publication. There were 328 pairs (99.1%) with
concordant and 3 pairs (0.8%) with discordant primary end
points. Among 329 pairs in which results for primary end
points could be compared, 290 pairs (88.1%) were
concordant. Two-thirds of 39 discordant pairs (26 pairs
[66.7%]) had effect size estimates in the same direction and
were statistically consistent. Overall, 323 pairs (97.6%) had
concordant study interpretations, including 32 of 39 pairs
with discordant primary end point results (82.1%). Pairs with
corresponding publications in journals with an IF of 10 or
higher had lower concordance rates for sample size (17 of 24
pairs [70.8%] vs 235 of 258 pairs [91.1%]; P = .01) and results
(16 of 23 pairs [70.0%] vs 235 of 261 pairs [90.0%]; P = .004). 
 **Conclusions** Most clinical studies posted as preprints on
medRxiv and subsequently published in peer-reviewed
journals had concordant study characteristics, results, and
interpretations, similar to what has been previously observed
among preprints published in the highest-impact journals.1 
 **Reference**
1. Shi X, Ross JS, Amancharla N, Niforatos JD, Krumholz
HM, Wallach JD. Assessment of concordance and
discordance among clinical studies posted as preprints and
subsequently published in high-impact journals. JAMA Netw
Open. 2021;4(3):e212110. doi:10.1001/
jamanetworkopen.2021.2110 
 **Conflict of Interest Disclosures** Guneet Janda receives
research funding from the Yale-Mayo Clinic Center for Excellence
in Regulatory Science and Innovation (CERSI) Scholars Award
Program. Vishal Khetpal serves in an advisory role for Necessary
Ventures. Xiaoting Shi is supported by the China Scholarship
Council. Joseph S. Ross is a former associate editor of JAMA
Internal Medicine and a current research editor at The BMJ;
receives research support through Yale University from Johnson
& Johnson to develop methods of clinical trial data sharing, the
Medical Device Innovation Consortium as part of the National
Evaluation System for Health Technology, the US Food and Drug
Administration (FDA) for CERSI, the Agency for Healthcare
Research and Quality, the National Heart, Lung, and Blood Institute
of the National Institutes of Health (NIH), and the Laura and John
Arnold Foundation to establish the Good Pharma Scorecard at
Bioethics International; and is an expert witness at the request of
relator’s attorneys, the Greene Law Firm, in a qui tam suit alleging
violations of the False Claims Act and the Anti-Kickback Statute
against Biogen. Joseph S. Ross reported cofounding medRxiv, the
preprint server for health sciences research, and Joshua D. Wallach
reported being a medRxiv affiliate; these individuals did not receive
compensation for their roles. Joshua D. Wallach is supported by the
National Institute on Alcohol Abuse and Alcoholism of the NIH. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/de0b40c65b691c2b25b42fc570bde7dd.png)

Assessment of Concordance Between Reports of Clinical Studies Posted as medRxiv Preprints and Corresponding Publications in Peer Reviewed Journals

**Objective** Previous studies showed high levels of similarity
between preprints and their subsequent peer-reviewed
journal publications.1,2 The goal of this study was to analyze
the extent of similarity for preprint-publication pairs in the
field of epidemiology. 
 **Design** This cohort study documented differences between
bioRxiv epidemiology preprints with only 1 preprint version
and their subsequent journal version. Preprints were
classified as “epidemiology” by their submitting authors.
From inception of the preprint server through December 31,
2019, there were 622 such preprints. Sample size calculation
for precision using a 95% confidence level and an 8% margin
of error yielded a requirement of 121 preprints, which were
then randomly sampled from the 622. Changes between
preprint-publication pairs were highlighted using the
Microsoft Word function compare two versions of a
document. Any changes that occurred were noted and
classified. 
 **Results** The 121 bioRxiv epidemiology preprints were later
published in 73 different journals (median [IQR] impact
factor, 4 [2.9-6.9]) with a median (IQR) time from preprint to
publication of 204 (131-243) days. Of the 121 pairs, 31 (26%)
had differences in their titles, 8 (7%) in the number or order
of authors, 31 (26%) in the number of tables, 28 (23%) in the
number of figures, 102 (84%) in the number of references, 54
(44%) in acknowledgment descriptions, 74 (61%) in conflict
of interest declarations, and 49 (40%) in data sharing
statements. Regarding main content, 109 (90%) had changes
in the abstract, 7 of which (6%) reported different P values;
106 (88%) had changes in the introduction section, 37 of
which (31%) altered descriptions of their objectives; 120
(99%) had changes in their methods section, 9 of which (7%)
had changes in their sample size; 115 (95%) had changes in
their results section, with 82 (68%) adding or removing (parts
of) results; and 116 (96%) had changes in the their discussion
sections, with 65 (54%) adding limitations in their journal
versions and 12 (10%) exhibiting substantive changes to main
results in the first sentence of their discussion (**Table 18**). 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/df0f88dbc2efc26ec79b92d02b994fae.png)
**Conclusions** This study shows that almost all aspects of
epidemiological preprints were slightly changed in their
journal publication versions, with 10% of preprints changing
their main findings. Further research is needed to determine
who requested those changes and why, whether changes were
associated with the quality of the study or the expertise of
those requesting them, and whether changes led to increases
in validity, transparency, or readability. 
 **References**
1. Klein M, Broadwell P, Farb SE, Grappone T. Comparing
published scientific journal articles to their pre-print versions.
Int J Digital Librar. 2019;20:335-350. doi:10.1007/s00799-
018-0234-1 
 2. Shi X, Ross JS, Amancharla N, Niforatos JD, Krumholz
HM, Wallach JD. Assessment of concordance and
discordance among clinical studies posted as preprints and
subsequently published in high-impact journals. JAMA Netw
Open. 2021;4(3):e212110. doi:10.1001/
jamanetworkopen.2021.2110 
 **Conflict of Interest Disclosures** IJsbrand Jan Aalbersberg is
senior vice president of research integrity at Elsevier. Mario Malički
is a co–editor in chief of Research Integrity and Peer Review. Lex
Bouter, John P. A. Ioannidis, and Steven N. Goodman are members
of the Peer Review Congress Advisory Board but were not involved
in the review or decision for this abstract. 
 **Funding/Support** Elsevier funding was awarded to Stanford
University for a Meta-Research Innovation Center at Stanford
postdoctoral position that supported Mario Malički’s work on the
project. 
 **Role of the Funder/Sponsor** IJsbrand Jan Aalbersberg is an
employee of Elsevier and had a role in the design and conduct of
the study; management and interpretation of the data; review and
approval of the abstract; and decision to submit the abstract for
presentation. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/a26ea1f51a55475081eaafd588687ed4.png)

Comparison of Reports of Epidemiology Studies Posted as bioRxiv Preprints and Published in Peer Reviewed Journals

**Objective** This study aimed to describe and explore the
content of comments received by preprints posted in 2020 to
bioRxiv and medRxiv. It extended a previous study1
thatassessed comments on bioRxiv until 2019, considering the
increased attention received by preprints after the beginning
of the COVID-19 pandemic.2 
 **Design** This was an observational qualitative study. The list
of preprints posted to bioRxiv and medRxiv and the number
of comments received by them were obtained using each
platform’s application program interface. After excluding
preprints with no comments and those with more than 20
comments (for feasibility and avoiding overrepresentation of
individual preprints), eligible preprints were randomly
sampled and their comments were assigned to 3 reviewers.
Content analysis of comments was based on a structured
form, with predefined categories based on previous research
on preprint comments1 and on a scoping review of peer
review.3 If 2 of 3 reviewers agreed, the classification was
considered final; in case all 3 disagreed, a fourth reviewer
made the final call. Neither manuscripts nor the comment
appropriateness or validity in regard to the manuscript text
were assessed. 
 **Results** A total of 1037 preprints were assessed, with a total
of 1921 comments (mean [SD], 1.8 [1.9] comments per
preprint); 439 comments (22.8%) were replies to previous
comments, leaving 1482 comments available for classification
based on our prespecified categories (**Table 19**). A total of
14.6% (95% CI, 12.7%-16.5%) of reader comments were
considered not to be about the content of the preprints
(n = 192). Comments from preprint authors (n = 165)
accounted for 11.1% (95% CI, 9.5%-12.7%) of the remaining
comments, and mostly addressed publication status (n = 89);
38% (95% CI, 35.2%-40.9%) of reader (nonauthor) comments
included compliments (n = 428), 61.7% (95% CI, 58.8%-
64.5%) included criticisms, corrections, or suggestions
(n = 694), and 34.9% (95% CI, 32.1%-37.7%) included
questions (n = 393). Although compliments were mostly
general (n = 279), criticisms, corrections, or suggestions
largely addressed specific points, such as interpretation
(n = 286), methodologic design (n = 267), materials/data
collection methods (n = 238), and analyses (n = 228). Most
questions were about information not provided in the
preprint (n = 170) or about materials/data collection
(n = 166). 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/845bcb02b0a10dc793691ce6a41aba5d.png) 
 **Conclusions** As shown before,1
only a small percentage of
preprints received comments on their respective preprint
platforms, but this study found that the content of these
comments showed similarities with the type of content
typically expected from journal-elicited peer review as
previously described.3
These results may help generate
hypotheses to inform future research on preprints and peer
review. 
 **References**
1. Malički M, Costello J, Alperin JP, Maggio LA. Analysis of
single comments left for bioRxiv preprints till September
2019. Biochem Med (Zagreb). Published online April 15,
2021. doi:10.11613/BM.2021.020201 
 2. Fraser N, Brierley L, Dey G, et al. The evolving role of
preprints in the dissemination of COVID-19 research and
their impact on the science communication landscape. PLoS
Biol. Published online April 2, 2021. doi:10.1371/journal.
pbio.3000959 
 3. Glonti K, Cauchi D, Cobo E, Boutron I, Moher D, Hren D. A
scoping review on the roles and tasks of peer reviewers in the
manuscript review process in biomedical journals. BMC Med.
Published online June 20, 2019. doi:10.1186/s12916-019-
1347-0 
 **Conflict of Interest Disclosures** None reported. 
 **Funding/Support** Clarissa França Dias Carneiro received a
scholarship from the Conselho Nacional de Desenvolvimento
Científico e Tecnológico (CNPq). Olavo B. Amaral received funding
from FAPERJ (E-26/200.824/2021), CNPq (310813/2021-2), and
the Serrapilheira Institute. 
 **Role of the Funder/Sponsor** The funders had no direct role
in the design and conduct of the study; collection, management,
analysis, and interpretation of the data; preparation, review, or
approval of the abstract; and decision to submit the abstract for
presentation. 
 **Acknowledgment** The authors thank Richard Abdill for his
contributions to data collection. 
 **Additional Information** Mario Malički and Olavo B. Amaral are
co–corresponding authors. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/410f058e1bd446ecd5f8d3992a429a99.png)

Content Analysis of Comments on bioRxiv and medRxiv Preprints

**Objective** To compare media attention, Twitter engagement,
and citations among COVID-19–related clinical trial preprints
and corresponding peer-reviewed publications. 
 **Design** Preprints of clinical trials were included in this
cross-sectional study if they reported results related to the
treatment or prevention of COVID-19 and were uploaded to
an open access preprint server indexed by the National
Institutes of Health iSearch COVID-19 portfolio in 2020. Two
investigators, including a medical librarian, independently
searched Medline, Google Scholar, and Embase to identify
peer-reviewed publications corresponding to the included
preprints. Altmetric data were used to quantify media
mentions and Twitter interactions within 3 months after
publication for both preprints and peer-reviewed
manuscripts. Citation counts were also recorded using Web of
Science. Descriptive statistics were reported for the included
trials, and linear regression was used to assess associations
between study characteristics and media mentions, Twitter
interactions, and citation counts. 
 **Results** Of 22,615 preprints screened for eligibility, 145 were
included. MedRxiv was the source for most eligible preprints
(n =100; 69%). Peer reviewed publications matching 118 of
145 preprints (81%) were found. Sixty-eight preprints (47%)
received media citations within 3 months of publication
(median [IQR] number of mentions per preprint, 0 [0-9];
maximum mentions, 568), and 118 (81%) had Twitter
interactions (median [IQR] tweets per preprint, 28 [4-202];
maximum mentions, 18,177). One hundred preprints (69%)
were cited in published literature (median [IQR] citations, 2
[0-12]). Among 118 preprints with matching peer-reviewed
publications, Altmetric and citation data were unavailable for
5 (4%). Preprints received more media mentions than the
corresponding peer-reviewed publications in 33 cases (28%),
equal mentions in 38 cases (32%), and fewer media mentions
in 42 cases (36%). Twitter mentions were greater for
preprints than peer-reviewed publications in 54 cases (46%),
equal in 2 (2%), and fewer for 62 (53%). In 26 cases (22%),
preprints received more citations than peer-reviewed
publications; citation counts were equal in 6 cases (5%) and
in 85 cases (72%), the peer-reviewed version received more
citations. Study size, government funding, failure to
prospectively register, and positive study results were most
often associated with increased media mentions, tweets, and
citations (**Table 20**). 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/59664a9a5f5c8dc866bb6ab1f830560e.png) 
 **Conclusions** Although peer-reviewed publications had more
media, Twitter, and citation activity than corresponding
preprints in most cases, it was not uncommon for preprints to
receive more attention than peer-reviewed publications.
Measures of trial reliability or quality were generally not
associated with increases in these metrics. 
 **Conflict of Interest Disclosures** Christopher W. Jones reported
receiving grants from AstraZeneca, Abbott, Vapotherm, and
Ophirex outside the submitted work. Timothy F. Platts-Mills is an
employee of Ophirex. No other disclosures were reported. 
 **Funding/Support** This work was supported by the US
Department of Health and Human Services (HHS) Office of
Research Integrity (grant ORIIR200066-01-00). 
 **Role of the Funder/Sponsor** The sponsor had no role in
the design and conduct of the study; collection, management,
analysis, and interpretation of the data; preparation, review, or
approval of the manuscript; and decision to submit the abstract for
presentation. 
 **Additional Information** The contents of this manuscript are
those of the authors and do not represent the official views of, nor
an endorsement by, the Office of the Assistant Secretary for Health,
the HHS, or the US government. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/3af6ee0204355a3795cd4012c2ef7123.png)

Media Attention, Twitter Engagement, and Citations of COVID-19 Clinical Trial Preprints and Their Corresponding Peer-Reviewed Publications

**Objective** The COVID-19 pandemic may have signaled a
positive shift in attitudes toward sharing individual-level
patient data (IPD).1 This project aimed to obtain IPD from
trialists of randomized clinical trials (RCTs) of
pharmacological treatments for COVID-19 for the purposes of
conducting an IPD meta-analysis under the COVID-NMA
initiative.2

**Design** This single cohort study evaluated the effectiveness
of accessing IPD from trialists of COVID-19 RCTs through
email requests and online data repositories. Participants were
the corresponding authors of RCTs of pharmacological trials
for the treatment of COVID-19 who published their findings
in a preprint or peer-reviewed journal between March 2020
and May 2021. Corresponding authors were emailed once,
with at least 2 reminders over 9 months (November 2020 to
September 2021) regardless of their data sharing statement.
Online data repositories (including Vivli, Yale University
Open Data Access Project, and ClinicalStudyDataRequest.
com) were also searched. The primary outcome was the
proportion of studies for which IPD were accessed; IPD was
defined as a data set obtained directly from trialists or those
that were made accessible online. The project team
collaborated with Vivli, a global data sharing platform.

**Results** Fifty-six of 229 COVID-19 RCT trialists (24%)
shared their IPD by the end of December 2021. Of these, 18
(32%) declared their positive willingness to share in the
registry and 42 (75%) in their preprint or publication. Of
those trials that did not share their data, 53 (31%) declared
their positive willingness to share in the registry and 100
(58%) in their preprint or publication. Stratified by funding,
trials that shared vs did not share IPD received funding from
sources that were public or nonprofit (shared: 13 [23%]; not
shared: 79 [46%]), private (shared: 10 [18%]; not shared: 36
[21%]), mixed public/nonprofit and private (shared: 20
[36%]; not shared: 37 [21%]), no funding (shared: 11 [20%];
not shared: 10 [6%]), and not reported or unclear (shared: 2
[4%]; not shared: 11 [6%]). Trials that shared vs did not share
IPD were based in high-income countries (shared: 19 [34%];
not shared: 45 [26%]), low- to middle-income countries
(shared: 31 [55%]; not shared: 117 [68%]), and both high-
income and low- to middle-income countries (shared: 6
[11%]; not shared: 11 [6%]).

**Conclusions** Despite the positive shift in attitudes toward
sharing IPD, less than one-quarter of trialists shared their
IPD. Furthermore, data sharing statements often did not line
up with the ability to obtain the data. This study emphasizes
the need to mandate and/or reward timely data sharing while
addressing remaining administrative, resource infrastructure,
and cultural obstacles.

**References**
1. Larson K, Sim I, von Isenburg M, et al. COVID-19
interventional trials: analysis of data sharing intentions
during a time of pandemic. Contemp Clin Trials.
2022;115:106709. doi:10.1016/j.cct.2022.106709 
 2. Boutron I, Chaimani A, Meerpohl JJ, et al; COVID-NMA
Consortium. The COVID-NMA project: building an evidence
ecosystem for the COVID-19 pandemic. Ann Intern Med.
2022;173(12):1015-1017. doi:10.7326/M20-5261

**Conflict of Interest Disclosures** Isabelle Boutron is a member
of the Peer Review Congress Advisory Board but was not involved
in the review or decision for this abstract. No other disclosures were
reported.

**Funding/Support** This study, as part of the COVID-NMA
initiative, received funding from Université de Paris, Assistance
Publique Hôpitaux de Paris, INSERM, Cochrane France (Ministry
of Health), the French Ministry of Higher Education and Research,
Agence Nationale de la Recherche, and the World Health
Organization.

**Additional Information** Isabelle Boutron is a co–corresponding
author.

![](https://assets.underline.io/uploads/markdown_image/1/image/e347bbf33644d08756b4e91337063f86.png)

Downloads

Next from Peer Review Congress 2022

Peer Review in the Age of Open Science, Tony Ross-Hellauer

Similar lecture

International research collaboration in the post-COVID era

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from Peer Review Congress 2022

Peer Review in the Age of Open Science, Tony Ross-Hellauer

Similar lecture

International research collaboration in the post-COVID era

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads