Objective A constructive peer review can bring a fresh perspective to help improve an author’s work. Until recently, there has been little guidance on what constitutes a good or poor peer review and no empirical study about researchers’ experiences with unhelpful or upsetting reviews.1-3 This pilot study aimed to explore the types of poor peer review practices (PPRP) experienced by authors and assess their association with authors’ ability to disseminate research.

Design An anonymous 51-question survey was designed through Qualtrics. An invitation to complete the survey was emailed to a random sample of 500 researchers funded by the National Institutes of Health in 2018 (with the replacement of undeliverable email addresses) and posted to a bioethics discussion forum (mcw-bioethics@mailman.mcw.edu) with approximately 600 members. Three mailings were performed between April 5 and April 19, 2022. The study was determined to be exempt by the University of Pennsylvania institutional review board. A 28-item list of PPRP was developed following a literature review; respondents’ experiences were assessed with yes-no and open-ended questions. The PPRP was assessed through author responses to Likert scale questions. Exploratory nonparametric analyses of respondents’ reported experiences and demographic characteristics were performed using Stata, version 12.1 (StataCorp LLC). Data were reported according to the Consensus-Based Checklist for Reporting of Survey Studies (CROSS).

Results A total of 112 researchers completed the surveys, approximately 10% of those solicited. Respondents were predominantly male (59 of 109 54%), held a PhD (87 of 112 78%), were older than 50 years (73 of 111 66%), identified as White race (92 of 106 87%), published more than 50 peer-reviewed papers in their career (71 of 111 64%), were trained in the humanities or social sciences (61 of 112 54%), and conducted primarily empirical research (68 of 111 61%). The mean number of PPRP experienced per author was 12.5 of 28 (44.6%) (range, 0-27; 95% CI, 11.2-13.8). A total of 57% (63 of 111) of authors admitted abandoning a manuscript after receiving unfair peer reviews; 67% (74 of 111) of authors sometimes or often received insightful peer reviews that improved the quality of their final papers. An exploratory univariate analysis of the association between the total number of PPRP reported per author and demographic characteristics and reported positive peer review experiences is presented in Table 62. This analysis suggests areas for future study, particularly the need to explore types of helpful reviews.

Conclusions This pilot study was the first to date to assess researchers’ experiences with PPRP. The low response rate suggests high selectivity and bias and that respondents report experiences with an extensive range of PPRP. The study lays the groundwork for future research, which may be valuable for improving peer review quality.

References

1. Smith R. Peer review: a flawed process at the heart of science and journals. J R Soc Med. 2006;99(4):178-182. doi:10.1258/jrsm.99.4.178
2. Mavrogenis AF, Quaile A, Scarlat MM. The good, the bad and the rude peer-review. Int Orthop. 2020;44(3):413-415. doi:10.1007/s00264-020-04504-1 3.Silbiger NJ, Stubler AD. Unprofessional peer reviews disproportionately harm underrepresented groups in STEM. PeerJ. 2019;7:e8247. doi:10.7717/peerj.8247

Conflict of Interest Disclosures None reported.