poster
Concordance Between Peer Reviewers' Recommendations and Editorial Decision-Making at The Journal of Pediatrics
keywords:
editorial and peer review process
peer review
Objective To examine concordance between peer reviewers’
recommendations and editorial decisions for a general
pediatric journal.
Design This was a cross-sectional observational study of a
convenience sample of peer-review recommendations for
articles submitted to The Journal of Pediatrics reported using
the Strengthening the Reporting of Observational Studies in
Epidemiology (STROBE) reporting guideline. After
submission, an editor made a preliminary assessment and
invited peer reviewers. Responding reviewers were asked to
provide a recommendation for acceptance, revision, or
rejection. When recommending acceptance or revision,
reviewers were asked to select a priority score for publication
on a 4-level Likert scale (low, medium low, medium high,
high). After assessing the reviews, editors decided either to
move the article forward (accept or invite a revision) or to
reject. Manuscripts were included in the study if the original
submission was initially reviewed by at least 2 invited
reviewers. The primary outcome of the study was the final
disposition (accept or reject). Reviewer recommendations
and Likert scores in response to the original submission were
compared with final dispositions using univariable and
multivariable regression. For the multivariable analysis, the
highest reviewer priority score was dichotomized as high/
medium high vs. medium low/low. Differences between
priority scores were dichotomized as matched or differing by
1 Likert level vs differing by 2 or 3 Likert levels.
Results The sample included 1016 articles sent for review
between June 16, 2020, and October 27, 2021, with at least 2
completed reviews in response to the original submission. At
least 1 reviewer recommended rejection for 393 of 1016
manuscripts (38.7%). When reviewers agreed to reject, all
manuscripts were rejected (65 of 65); however, when
reviewers disagreed on rejection, 36 of 328 manuscripts were
accepted for publication (11%; P = .005) and 1 is pending to
date. Among articles with a reviewer recommendation for
rejection and at least 1 reviewer priority score (n=294), high
or medium high priority scores were associated with a final
decision to accept the article (odds ratio OR,11.37; 95% CI,
3.94-32.85; P < .001). Reviewers agreed to move the article
forward for 623 of 1016 manuscripts (61.3%) with 395
ultimately accepted for publication. Of these 623, 494 (79.3%)
had at least 2 reviewer priority scores and a final disposition.
The highest reviewer priority score was significantly
associated with the acceptance rate (Table 34). In
multivariable regression, higher priority score category (OR,
12.59; 95% CI, 6.84-23.14) and level of reviewer agreement
on priority scores (OR, 2.04; 95% CI, 1.15-3.6) were both
significantly associated with article acceptance rates.
Conclusions Both reviewer recommendations on
acceptance or rejection and priority scores for publication
appeared to be influential in editorial decision-making.
However, editors were frequently required to arbitrate mixed
reviewer recommendations and make independent decisions.
Conflict of Interest Disclosures None reported.