technical paper

Peer Review Congress 2022

September 10, 2022

Chicago, United States

Content Analysis of Comments on bioRxiv and medRxiv Preprints


peer review process and models


peer review

Objective This study aimed to describe and explore the content of comments received by preprints posted in 2020 to bioRxiv and medRxiv. It extended a previous study1 thatassessed comments on bioRxiv until 2019, considering the increased attention received by preprints after the beginning of the COVID-19 pandemic.2

Design This was an observational qualitative study. The list of preprints posted to bioRxiv and medRxiv and the number of comments received by them were obtained using each platform’s application program interface. After excluding preprints with no comments and those with more than 20 comments (for feasibility and avoiding overrepresentation of individual preprints), eligible preprints were randomly sampled and their comments were assigned to 3 reviewers. Content analysis of comments was based on a structured form, with predefined categories based on previous research on preprint comments1 and on a scoping review of peer review.3 If 2 of 3 reviewers agreed, the classification was considered final; in case all 3 disagreed, a fourth reviewer made the final call. Neither manuscripts nor the comment appropriateness or validity in regard to the manuscript text were assessed.

Results A total of 1037 preprints were assessed, with a total of 1921 comments (mean SD, 1.8 1.9 comments per preprint); 439 comments (22.8%) were replies to previous comments, leaving 1482 comments available for classification based on our prespecified categories (Table 19). A total of 14.6% (95% CI, 12.7%-16.5%) of reader comments were considered not to be about the content of the preprints (n = 192). Comments from preprint authors (n = 165) accounted for 11.1% (95% CI, 9.5%-12.7%) of the remaining comments, and mostly addressed publication status (n = 89); 38% (95% CI, 35.2%-40.9%) of reader (nonauthor) comments included compliments (n = 428), 61.7% (95% CI, 58.8%- 64.5%) included criticisms, corrections, or suggestions (n = 694), and 34.9% (95% CI, 32.1%-37.7%) included questions (n = 393). Although compliments were mostly general (n = 279), criticisms, corrections, or suggestions largely addressed specific points, such as interpretation (n = 286), methodologic design (n = 267), materials/data collection methods (n = 238), and analyses (n = 228). Most questions were about information not provided in the preprint (n = 170) or about materials/data collection (n = 166).

Conclusions As shown before,1 only a small percentage of preprints received comments on their respective preprint platforms, but this study found that the content of these comments showed similarities with the type of content typically expected from journal-elicited peer review as previously described.3 These results may help generate hypotheses to inform future research on preprints and peer review.

References 1. Malički M, Costello J, Alperin JP, Maggio LA. Analysis of single comments left for bioRxiv preprints till September 2019. Biochem Med (Zagreb). Published online April 15, 2021. doi:10.11613/BM.2021.020201

2. Fraser N, Brierley L, Dey G, et al. The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape. PLoS Biol. Published online April 2, 2021. doi:10.1371/journal. pbio.3000959

3. Glonti K, Cauchi D, Cobo E, Boutron I, Moher D, Hren D. A scoping review on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals. BMC Med. Published online June 20, 2019. doi:10.1186/s12916-019- 1347-0

Conflict of Interest Disclosures None reported.

Funding/Support Clarissa França Dias Carneiro received a scholarship from the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq). Olavo B. Amaral received funding from FAPERJ (E-26/200.824/2021), CNPq (310813/2021-2), and the Serrapilheira Institute.

Role of the Funder/Sponsor The funders had no direct role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.

Acknowledgment The authors thank Richard Abdill for his contributions to data collection.

Additional Information Mario Malički and Olavo B. Amaral are co–corresponding authors.


SlidesTranscript English (automatic)

Next from Peer Review Congress 2022

Media Attention, Twitter Engagement, and Citations of COVID-19 Clinical Trial Preprints and Their Corresponding Peer-Reviewed Publications
technical paper

Media Attention, Twitter Engagement, and Citations of COVID-19 Clinical Trial Preprints and Their Corresponding Peer-Reviewed Publications

Peer Review Congress 2022

Emily Inwards

10 September 2022

Similar lecture

Does social influence matter in peer review?
technical paper

Does social influence matter in peer review?


+3Thomas Feliciani
Thomas Feliciani and 5 other authors

30 September 2020

Stay up to date with the latest Underline news!

Select topic of interest (you can select more than one)


  • All Lectures
  • For Librarians
  • Resource Center
  • Free Trial
Underline Science, Inc.
1216 Broadway, 2nd Floor, New York, NY 10001, USA

© 2023 Underline - All rights reserved