**Objective** Previous research has indicated that 642 of 1116
surveyed researchers (58%) received unprofessional
comments in peer review reports at least once in their
professional life1
and that 179 of 1491 review reports (12%) in
the fields of ecology and evolution and behavioral medicine
contained unprofessional comments.2
It was the goal of this
study to estimate the percentage of manuscripts with at least
1 unprofessional comment made in a broad sample of peer
review reports across disciplines.

**Design** This was a cross-sectional study of review reports
available in the PEERE database, which covers 10 years of
peer review records across all impact factor quartiles and peer
review types and includes journals from the fields of life
sciences, health and medicine, physical sciences, and social
sciences and economics.3
Sample size calculation indicated
that 380 of 297,026 manuscripts contained in the PEERE
database should be analyzed to detect a prevalence of 1% of
manuscripts (range, 0%-2%) having at least 1 unprofessional
comment. Randomized stratified sampling was used to
preserve scholarly field distribution. All review reports of 1147
sampled manuscripts were then extracted. Reading of the
reports was done independently by 3 researchers, who
marked instances of unprofessional comments. Disagreement
between researchers was resolved through consensus.

**Results** Of 1147 analyzed review reports, 13 (1.1%) contained
at least 1 unprofessional comment (**Table 69**). All were
found in review reports of different manuscripts; authors of
13 of 380 analyzed manuscripts (3.4%) received
unprofessional comment(s) during peer review. All instances of unprofessional comments were found among detailed
review reports.

![](https://assets.underline.io/uploads/markdown_image/1/image/927965422185a5ad94b93f332d616f58.png)

**Conclusions** In the study sample, 3.4% of submitted
manuscripts received at least 1 unprofessional comment
during peer review. Although the PEERE database presents
the largest collection of confidential review reports shared by
publishers, it might not be representative of all journals.
Nevertheless, to the best of our knowledge, this is the first
study of unprofessional comments that used random
sampling across scholarly disciplines. Future studies should
evaluate the effects of these comments on the authors, assess
whether and how they respond to them, and explore
automated approaches for detecting unprofessional
comments.

**References**
1. Silbiger NJ, Stubler AD. Unprofessional peer reviews
disproportionately harm underrepresented groups in STEM.
Peer J. 2019;7:e8247. doi:10.7717/peerj.8247 
 2. Gerwing TG, Gerwing AMA, Avery-Gomm S, Choi CY,
Clements JC, Rash JA. Quantifying professionalism in peer
review. Res Integ Peer Rev. 2020;5(1):1-8. doi:10.1186/
s41073-020-00096-x 
 3. Squazzoni F, Bravo G, Farjam M, Marusic A, Mehmani B,
Willis M, et al. Peer review and gender bias: a study on 145
scholarly journals. Sci Advanc. 2021;7(2):eabd0299.
doi:10.1126/sciadv.abd0299

**Conflict of Interest Disclosures** Mario Malički is the co-editor
in chief of Research Integrity and Peer Review. Bahar Mehmani
is reviewer experience lead at Elsevier. Other authors declare no
competing interests. Steven Goodman and Bahar Mehmani are
members of the Peer Review Congress Advisory Board but were not
involved in the review or decision for this abstract.

**Funding/Support** The work by Francisco Grimaldo, Daniel
García-Costa, and Elena Álvarez-García is supported by the Spanish
Ministry of Science, Innovation and Universities, the Spanish State
Research Agency, and the European Regional Development Fund
under project RTI2018-095820-B-I00.

**Role of Funder/Sponsor** The funders had no role in the design
and conduct of the study; collection, management, analysis, and
interpretation of the data; preparation, review, or approval of the
abstract; and decision to submit the abstract for presentation.

![](https://assets.underline.io/uploads/markdown_image/1/image/7c93947ec5afedcbc7585ef447a0000b.png)

Unprofessional Comments in Peer Review Reports Across Scholarly Disciplines

**Objective** Previous studies showed high levels of similarity
between preprints and their subsequent peer-reviewed
journal publications.1,2 The goal of this study was to analyze
the extent of similarity for preprint-publication pairs in the
field of epidemiology. 
 **Design** This cohort study documented differences between
bioRxiv epidemiology preprints with only 1 preprint version
and their subsequent journal version. Preprints were
classified as “epidemiology” by their submitting authors.
From inception of the preprint server through December 31,
2019, there were 622 such preprints. Sample size calculation
for precision using a 95% confidence level and an 8% margin
of error yielded a requirement of 121 preprints, which were
then randomly sampled from the 622. Changes between
preprint-publication pairs were highlighted using the
Microsoft Word function compare two versions of a
document. Any changes that occurred were noted and
classified. 
 **Results** The 121 bioRxiv epidemiology preprints were later
published in 73 different journals (median [IQR] impact
factor, 4 [2.9-6.9]) with a median (IQR) time from preprint to
publication of 204 (131-243) days. Of the 121 pairs, 31 (26%)
had differences in their titles, 8 (7%) in the number or order
of authors, 31 (26%) in the number of tables, 28 (23%) in the
number of figures, 102 (84%) in the number of references, 54
(44%) in acknowledgment descriptions, 74 (61%) in conflict
of interest declarations, and 49 (40%) in data sharing
statements. Regarding main content, 109 (90%) had changes
in the abstract, 7 of which (6%) reported different P values;
106 (88%) had changes in the introduction section, 37 of
which (31%) altered descriptions of their objectives; 120
(99%) had changes in their methods section, 9 of which (7%)
had changes in their sample size; 115 (95%) had changes in
their results section, with 82 (68%) adding or removing (parts
of) results; and 116 (96%) had changes in the their discussion
sections, with 65 (54%) adding limitations in their journal
versions and 12 (10%) exhibiting substantive changes to main
results in the first sentence of their discussion (**Table 18**). 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/df0f88dbc2efc26ec79b92d02b994fae.png)
**Conclusions** This study shows that almost all aspects of
epidemiological preprints were slightly changed in their
journal publication versions, with 10% of preprints changing
their main findings. Further research is needed to determine
who requested those changes and why, whether changes were
associated with the quality of the study or the expertise of
those requesting them, and whether changes led to increases
in validity, transparency, or readability. 
 **References**
1. Klein M, Broadwell P, Farb SE, Grappone T. Comparing
published scientific journal articles to their pre-print versions.
Int J Digital Librar. 2019;20:335-350. doi:10.1007/s00799-
018-0234-1 
 2. Shi X, Ross JS, Amancharla N, Niforatos JD, Krumholz
HM, Wallach JD. Assessment of concordance and
discordance among clinical studies posted as preprints and
subsequently published in high-impact journals. JAMA Netw
Open. 2021;4(3):e212110. doi:10.1001/
jamanetworkopen.2021.2110 
 **Conflict of Interest Disclosures** IJsbrand Jan Aalbersberg is
senior vice president of research integrity at Elsevier. Mario Malički
is a co–editor in chief of Research Integrity and Peer Review. Lex
Bouter, John P. A. Ioannidis, and Steven N. Goodman are members
of the Peer Review Congress Advisory Board but were not involved
in the review or decision for this abstract. 
 **Funding/Support** Elsevier funding was awarded to Stanford
University for a Meta-Research Innovation Center at Stanford
postdoctoral position that supported Mario Malički’s work on the
project. 
 **Role of the Funder/Sponsor** IJsbrand Jan Aalbersberg is an
employee of Elsevier and had a role in the design and conduct of
the study; management and interpretation of the data; review and
approval of the abstract; and decision to submit the abstract for
presentation. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/a26ea1f51a55475081eaafd588687ed4.png)

Comparison of Reports of Epidemiology Studies Posted as bioRxiv Preprints and Published in Peer Reviewed Journals

**Objective** Transparency and Openness Promotion (TOP)
guidelines were published in 2014, and since then, more than
5000 journals and 80 organizations have become TOP
signatories.1 However, attitudes toward all TOP guideline
recommendations have not been systematically assessed. The
goal of this study was to assess differences in attitudes and
experiences of authors, reviewers, and editors about
conducting and reporting research (TOP guidelines), their
research environment, and perceived prevalence of
responsible and detrimental research practices. 
 **Design** A survey was sent in 2018 to 100,000 randomly
selected corresponding authors of articles indexed in Scopus,
with 2 reminder rounds. The survey was designed for this
study and consisted of 38 questions using 5-item rating scales
(eg, from strongly agree to strongly disagree) and 10
(categorical) sociodemographic questions. Responses were
presented as absolute numbers and percentages based on
number of respondents per question. Ordinal regression
analyses were used to explore associations between answers
to questions and sociodemographic characteristics. All
analyses were conducted in Stata, version 13, and P ≤ .001
was considered statistically significant. 
 **Results** Study response rate was 4.9% (3659 of 74,749
delivered emails) and included responses from 1389 authors,
1833 reviewers, and 434 editors. Respondents came from 126
countries, had a median age of 44 years (IQR, 35-55 years),
and were most commonly from physical sciences (1034
[33%]) and life sciences (796 [25%]). There were no
significant differences between authors, reviewers, and
editors in their attitudes toward TOP guidelines, but some
TOP recommendations (eg, study preregistration) were not
supported. A majority (3462 [97%]) of respondents (strongly)
agreed that researchers must appropriately cite study data,
methods, and materials; 2675 (74%) that authors must follow
appropriate reporting guidelines; 2174 (60%) that researchers
must share data; 797 (23%) that authors must include the full
data analysis plan in study preregistration; and 751 (21%) that
studies must be preregistered. One fifth of respondents (701
[20%]) admitted sacrificing the quality of their publications
for quantity, and 492 (14%) reported funders interfering in
their study design or reporting. Undeserved authorship was
perceived by all groups as the most prevalent detrimental
research practice, while fabrication, falsification, plagiarism,
and not citing prior relevant research were seen as more
prevalent by editors than authors or reviewers (**Table 38**).
Additional findings are available elsewhere.2 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/a43c3e0462b4e713c2360345ed290fe8.png) 
 **Conclusions** There were no differences in attitudes between
authors, reviewers, and editors toward specific TOP
recommendations, although some recommendations were not
supported. Respondents’ perceptions of their research
environments and of prevalence of detrimental research
practices indicated that there is still much room for
improvement. Without agreement and involvement of all
stakeholders, it is unlikely that TOP recommendations will
become standard practice. This study is limited by the overall
low response rate, and therefore, the results might not be
generalizable. 
 **References**
1. Nosek BA, Alter G, Banks GC, et al. Promoting an open
research culture. Science. 2015;348(6242):1422-1425.
doi:10.1126/science.aab2374 
 2. Malički M, Aalbersberg IJ, Bouter L, Mulligan A, ter Riet G.
Transparency in conducting and reporting research: a survey
of authors, reviewers, and editors across scholarly disciplines.
Research Square. Preprint posted online January 26, 2022.
doi:10.21203/rs.3.rs-1296644/v1 
 **Conflict of Interest Disclosures** Mario Malički is co–Editor
in Chief of Research Integrity and Peer Review. IJsbrand Jan
Aalbersberg is Senior Vice President of Research Integrity at
Elsevier. Lex Bouter is a member of the Peer Review Congress
Advisory Board but was not involved in the review or decision for
this abstract. Adrian Mulligan is a research director for Customer
Insights at Elsevier. No other disclosures were reported. 
 **Funding/Support** Elsevier funded the work of Mario Malički in
this study. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/bc2e457f5ab6defe02b60b3dc9c4bfd6.png)

Attitudes and Experiences of Authors, Reviewers, and Editors About Responsible and Detrimental Research Practices and the Transparency and Openness Promotion Guidelines Across Scholarly Disciplines

**Objective** The ability of peer review to improve the scientific
endeavor (eg, the conduct, reporting, and validity of study
findings) has been questioned,1 and calls have been made to
showcase changes that occurred to each study due to peer
review.2 Until such transparency is achieved, identification
and synthesis of studies that analyzed differences between
preprints or manuscript versions submitted to journals and
peer-reviewed publications is being undertaken. 
 **Design** In this stage of the living systematic review, studies
were identified based on authors’ knowledge of the field and
by checking all research at peer review conferences
(presented as podium presentations or posters in the
European Union and USA). References and citations of
identified studies were then checked. For all studies, the
following was extracted: year of publication, sampling
method, conflict of interest, funding, data and protocol
sharing, number of analyzed version pairs, sample size
calculation, scholarly discipline, method used to compare
versions, variables (ie, manuscript sections) analyzed for
changes, and metric with which the changes were quantified
or qualitatively classified. 
 **Results** Of 25 studies published from 1990 through the end
of 2021, 16 (64%) analyzed changes between submitted and
published papers and 9 (36%) between preprints and
published papers. Most commonly, changes were analyzed by
filling out questionnaires or scales separately for each of the 2
manuscript versions (11 [44%]) or by manual comparison of
the 2 manuscript versions (6 [24%]). The median number of
analyzed version pairs was 59 (IQR, 41-122). Most studies
analyzed changes that occurred in health (18 [72%]) or social
sciences (4 [16%]) manuscripts. Overall, studies’ conclusions
indicated very high similarity between version pairs, with the
largest changes occurring in introduction and discussion
sections. Examples of items for which most changes were
found are presented in **Table 13**. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/8c97dc5a11f6a5f830a8e977dad133da.png) 
 **Conclusions** The current results indicate that submitted or
preprinted manuscript versions and their peer-reviewed
journal version are very similar, with main (analysis) methods
and main findings rarely changing. Quantification of these
results is pending. Large differences between studies, type of
manuscript changes, and methods with which they were
measured indicate greater need for collaboration in the peer
review field and creation of the core outcomes measures for
manuscript version changes. 
 **References**
1. Tennant JP, Ross-Hellauer T. The limitations to our
understanding of peer review. Res Integr Peer Rev.
2020;5(1):6. doi:10.1186/s41073-020-00092-1 
 2. Limbu S. Building trust in peer review: a Q&A with Dr
Mario Malički. BioMed Central. September 18, 2020.
Accessed June 24, 2022. http://blogs.biomedcentral.com/on-medicine/2020/09/18/building-trust-in-peer-review-a-qa-with-dr-mario-malicki/ 
 **Conflict of Interest Disclosures** IJsbrand Jan Aalbersberg is
senior vice president of research integrity at Elsevier. Mario Malički
is a co–editor in chief of Research Integrity and Peer Review. Lex
Bouter, John P. A. Ioannidis, and Steven N. Goodman are members
of the Peer Review Congress Advisory Board but were not involved
in the review or decision for this abstract. 
 **Funding/Support** Elsevier funding was awarded to Stanford
University for a METRICS postdoctoral position that supported
Mario Malički’s work on the project. 
 **Role of the Funder/Sponsor** IJsbrand Jan Aalbersberg is an
employee of Elsevier and had a role in the design and conduct of
the study; management and interpretation of the data; review and
approval of the abstract; and decision to submit the abstract for
presentation. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/285fdcf7b8c697b89f8022f686600131.png)

A Synthesis of Studies on Changes Manuscripts Underwent Between Submission or Preprint Posting and Peer-Reviewed Journal Publication

Peer Review Metrics And Data Analysis

Peering Into Peer Review

Between 21 May 2016 and 9 September 2019, 1,983 bioRxiv preprints received a single public comment on the bioRxiv website. More than two thirds of those comments were posted by non-authors (n=1,366, 69%) while the remainder were posted by the preprint’s authors (n=617, 31%). Overall, the non-author comments were longer than comments posted by the authors (median number of words was 38, IQR 17 to 83, for non-authors vs 18, IQR 11 to 32, for authors, Mann-Whitney test, P<0.001), and they were posted a median of 23 days (IQR 3 to 117) after the preprints. In comparison, authors’ comments were posted after a median of 91 days (IQR 3 to 23, Mann-Whitney test, P<0.001). Twelve percent of non-author’s comments (n=168) were full review reports resembling those traditionally submitted during the journal peer review process. Comments not resembling full peer review reports most commonly praised the preprint (n=577, 42%), made suggestions on how to improve it (n=399, 29%), or criticized some aspect of the preprint (n=226, 17%). Authors’ comments most commonly contained updates about the preprint’s publication status (n=354, 57%), additional information on the study (n=158, 26%), or solicited feedback for the preprint (n=65, 11%).

From amazing work to I beg to differ: analysis of bioRxiv preprints that received one public comment till September 2019

This is an inperson poster session. If you are attending virtually please visit the individual poster pages and ask your questions in the chat box.

PS2 - Peer Review

poster

Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.
 
 **Welcome!**
 
 The JAMA Network, The BMJ, and METRICS welcome you to the Ninth International Congress on Peer Review and Scientific Publication. We have continued our efforts to broaden the scope of the Congress to all aspects of peer review and publication—from funding to postpublication—and to all sciences.
 
 We will have 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination. There are 50 plenary session research presentations and 5 plenary invited talks. Each plenary session research presentation will be followed by equal time for discussion and questions from the audience. In addition, there are 125 posters. All posters, including virtual posters, are available to view throughout the meeting. In-person posters will also be presented at designated times on September 9 and 10. If you are attending virtually, please visit the individual poster pages and ask your questions in the chat box. See the link to Posters on the left.
 
 We hope you will take an active part in the program, as we depend on your participation in the discussion sessions to make the Congress a success.

 [Peer Review Congress Organizers, Directors, and Advisory Board](https://peerreviewcongress.org/organizers-and-advisory-board/)
 
 [Health and Safety Policies](https://web.cvent.com/event/1b505794-6a0b-428c-a44c-6c46c4c50e99/websitePage:152962bb-4567-4561-9d81-21ebfa97f487)

![](https://assets.underline.io/uploads/markdown_image/1/image/7c9f0143dee383780cc4670269ade132.png)

A PDF of the schedule is available [here](https://drive.google.com/file/d/1-Jd191W-nxbWcN1prdX5o9nlSfeo9_-H/view?usp=sharing) and a flipbook version of the complete program and abstracts is included below. See also the link to the interactive schedule to the left.

<iframe width="700px" height="425px" src="https://online.fliphtml5.com/ebtyf/nlfn/#p=1" frameborder="0" allowfullscreen allowtransparency></iframe>

Registration is required to attend this event, please use the URL linked below to secure your ticket. Registration will close on Friday, September 2 at 5:00 PM CT. 

[**Register Here**]( https://web.cvent.com/event/1b505794-6a0b-428c-a44c-6c46c4c50e99/summary)

Peer Review Congress 2022

Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.



Preprints

technical paper

PS1 - Misconduct

Peer Review

2. Peer review metrics and data analysis

Following the success of the 1st PEERE International Conference on Peer Review, the PEERE COST Action follow-up community organised the second edition of the PEERE International Conference on Peer Review  and was celebrated virtually on 29 September – 1 October 2020.

The conference welcomed any contribution (e.g., original research, position papers, literature reviews) that used any method (e.g., quantitative, experimental or qualitative) to investigate peer review in a variety of scientific domains (e.g., scholarly journals, funding agencies, research assessment). Interdisciplinary research is especially welcome.

The program is published here: http://tiny.cc/peere2020 


PEERE

The second edition of the PEERE International Conference on Peer Review aims to provide a forum for scholars, practitioners and science stakeholders to share research, models, tools and experience on peer review in different fields, e.g., medicine, computer science, social sciences and humanities. It aims to stimulate the use of evidence-based research in the design and implementation of peer review in a variety of fields and encourage more systematic research.

Mario Malicki

7

5

SHORT BIO

Presentations

Unprofessional Comments in Peer Review Reports Across Scholarly Disciplines

Comparison of Reports of Epidemiology Studies Posted as bioRxiv Preprints and Published in Peer Reviewed Journals

Attitudes and Experiences of Authors, Reviewers, and Editors About Responsible and Detrimental Research Practices and the Transparency and Openness Promotion Guidelines Across Scholarly Disciplines

A Synthesis of Studies on Changes Manuscripts Underwent Between Submission or Preprint Posting and Peer-Reviewed Journal Publication

Peer Review Metrics And Data Analysis

Peering Into Peer Review

From amazing work to I beg to differ: analysis of bioRxiv preprints that received one public comment till September 2019

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES