Peer Review Congress 2022

September 10, 2022

Chicago, United States

Mitigating Subjectivity in Peer Review via Artificial Intelligence


peer review

artificial intelligence

Objective Peer review incurs a problem of subjectivity (also called commensuration bias)1,2: different reviewers put differing emphasis on the various criteria when combining them to make an overall recommendation. Such subjectivity and resulting inconsistencies in the reviews are said to contribute to arbitrariness of the peer review process.1,2 In a top-tier venue’s peer review process, 2 types of inconsistencies due to such subjectivity were quantified, and an artificial intelligence (AI)–based strategy3 was deployed to mitigate such subjectivity.

Design This study was conducted in the final decision phase of the Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence 2022, which is a top scientific conference in artificial intelligence that reviews full papers, is a terminal publication venue, and is considered at par with journals. Reviewers were asked to evaluate papers on 8 criteria and also provide an overall recommendation. The issue of subjectivity pertains to different reviewers using different mappings from the criteria to their overall recommendations. Subjectivity may also lead to the following inconsistency. For any review, r, let cr denote the vector of criteria scores and or denote the numeric overall recommendation. For any criteria score vectors c and c′, c > c′ if every entry of c is at least as large as the corresponding entry of c′ and at least 1 entry of c is strictly larger than the corresponding entry of c′. A pair of reviews, r and s, are said to be inconsistent if cr > cs and or ≤ os or if cr = cs and or ≠ os. They are strongly inconsistent if cr > cs and or < os . An AI technique3 was deployed to mitigate such subjectivity. From the reviews, the AI learned a single mapping from criteria scores to overall recommendations that was representative of the entire set of reviews. Then, it applied the learned mapping to the criteria scores in each review to obtain an updated overall recommendation associated with each review. It mitigated subjectivity by virtue of the fact that the updated overall recommendations for all reviews were obtained via the same mapping. The design of the AI also ensured that the updated overall scores did not have the aforementioned inconsistencies.

Results There were 177 reviews in which the difference between the original overall recommendation provided by the reviewer and the updated overall recommendation computed by the AI was 2 or greater (on a 10-point scale); the (equivalents of the) associate editors for these reviews were notified. The maximum such difference was 7. The number of inconsistencies in the reviews is shown in Table 52.

Conclusions A large number of inconsistencies were found across reviewers in how criteria are mapped to overall recommendations. An AI method3 was successfully deployed to mitigate subjectivity in peer reviews.


  1. Lee CJ. Commensuration bias in peer review. Philos Sci. 2015;82(5):1272-1283. doi:10.1086/683652
  2. Kerr S, Tolliver J, Petree D. Manuscript characteristics which influence acceptance for management and social science journals. Acad Manage J. 1977;20(1):132-141. doi:10.5465/255467
  3. Noothigattu R, Shah N, Procaccia A. Loss functions, axioms, and peer review. J Artif Intelligence Res. 2021;70:1481-1515. doi:10.1613/jair.1.12554

Conflict of Interest Disclosures None reported.

Funding/Support This work was supported by the US National Science Foundation CAREER award 1942124, which supports research on the fundamentals of learning from people with applications to peer review.

Next from Peer Review Congress 2022

Analysis of Gender Representation, Authorship Inflation, and Institutional Affiliation in Abstract Acceptance, 2017-2021

Analysis of Gender Representation, Authorship Inflation, and Institutional Affiliation in Abstract Acceptance, 2017-2021

Peer Review Congress 2022

Carisa Cooney

10 September 2022

Similar lecture

AI Meets Real-Time: Addressing Real-World Complexities in Graph Response-Time Analysis
technical paper

AI Meets Real-Time: Addressing Real-World Complexities in Graph Response-Time Analysis


+1James AndersonTanya AmertSergey Voronov
Sergey Voronov and 3 other authors

09 December 2021

Stay up to date with the latest Underline news!

Select topic of interest (you can select more than one)


  • All Lectures
  • For Librarians
  • Resource Center
  • Free Trial
Underline Science, Inc.
1216 Broadway, 2nd Floor, New York, NY 10001, USA

© 2023 Underline - All rights reserved