Peer Review Congress 2022

September 11, 2022

Chicago, United States

Assessment of Grant Peer Reviewers' Tolerance for Risk in Research Proposals


funding/grant peer review

peer review


Objective Grant peer review relies on scientists’ ability to evaluate research proposals’ quality. Such judgments are sometimes beyond reviewers’ discriminatory power, leading to reliance on subjective biases, including preferences for lower-risk, incremental projects.1,2 However, reviewers’ research risk tolerance during grant peer review has not been well studied.

Design In late 2020 to early 2021, participants were recruited through email invitation using the American Institute of Biological Sciences’ reviewer databases and the National Institutes of Health’s panel rosters. A cross-sectional, prospective study was conducted of peer reviewers’ numeric evaluations of mock primary reviewers’ comments of a theoretical research proposal in which the level and sources of risks and weaknesses were manipulated. Specifically, there were 4 scenarios, with risks associated with either the investigator (limited experience in leading independent research; n = 199), the approach (lacking pilot studies; n = 205), both investigator and approach (n = 201), or neither (control; n = 605). Each participant evaluated the control and 1 randomly chosen manipulated risk scenario (the order of presentation was randomized). Risk tolerance was measured by the 13-item Openness to Experience scale from the NEO Five-Factor Inventory 3.3

Results Violin plots of overall and criteria scoring data from 605 participants (lower scores are more positive) showed that the experimental manipulation was effective: Scenarios with manipulated risk received more mediocre overall scores and more dispersed scoring distributions than the control scenario (Figure 32). Ordinal logistic regression models (Nagelkerke R2 = 0.78) suggest that the manipulated risks in these scenarios (compared with control) were strongly associated with participants’ overall scores, with odds ratios of 15.0 (95% CI, 3.4-66.0) for the investigator risk scenario, 21.7 (95% CI, 5.5-85.4) for the approach risk scenario, and 36.2 (95% CI, 8.5-154.4) for the scenario with both risks. The approach criterion score was associated with the overall score for all scenarios; significance and innovation criteria scores had stronger associations with overall scores in the control scenario. Openness to experience was not associated with scores, but differing levels of reviewer-reported scoring leniency were observed.

Conclusions These data suggest that the evaluation of risks dominates reviewers’ evaluation of research proposals and is an important source of interreviewer variability. A weakness is that this study did not involve actual grant proposals. Training peer reviewers to consider risks as potential assets (eg, innovation) may represent an area of useful intervention for science advancement.

References 1. Lamont M. How Professors Think: Inside the Curious World of Academic Judgment. Harvard University Press; 2009. 2. Luukkonen T. Conservatism and risk-taking in peer review: emerging ERC practices. Res Eval. 2012;21(1):48-60. doi:10.1093/reseval/rvs001 3. McCrae RR, Costa PT. NEO Inventories: Professional Manual. PAR, Inc; 2010.

Conflict of Interest Disclosures None reported.

Funding/Support This work was supported by the Social, Behavioral, and Economic Sciences Office of Multidisciplinary Activities at the National Science Foundation under the standard grant award numbers 1951132 and 1951251.



Next from Peer Review Congress 2022

Assessing PubMed Metatag Usage for Plain Language Summary Discoverability

Assessing PubMed Metatag Usage for Plain Language Summary Discoverability

Peer Review Congress 2022

Slavka Baronikova

11 September 2022

Similar lecture

Man, senior, targeting prestigious journals: Is this the identikit of the big fish in the peer review pond? An empirical study on 157 scholarly journals
technical paper

Man, senior, targeting prestigious journals: Is this the identikit of the big fish in the peer review pond? An empirical study on 157 scholarly journals


+1Giangiacomo BravoFlaminio Squazzoni
Flaminio Squazzoni and 3 other authors

30 September 2020

Stay up to date with the latest Underline news!

Select topic of interest (you can select more than one)


  • All Lectures
  • For Librarians
  • Resource Center
  • Free Trial
Underline Science, Inc.
1216 Broadway, 2nd Floor, New York, NY 10001, USA

© 2023 Underline - All rights reserved