VIDEO DOI: https://doi.org/10.48448/7ekt-xa96

technical paper

Peer Review Congress 2022

September 08, 2022

Chicago, United States

A Bayesian Approach to Reduce Bias in the Ranking of Peer-Reviewed Grant Proposals Submitted to the Swiss National Science Foundation

keywords:

funding/grant peer review

statistics

bias

Objective Funding agencies rely on peer review to select grant proposals deserving of funding. Peer review has limitations and may be biased. The interrater reliability between reviewers and panels is low, particularly for proposals near the funding line. A method for ranking grant proposals was developed to reduce bias and accommodate the uncertainty in evaluation and funding decisions.

Design Proposals submitted to a call were peer reviewed and later discussed in evaluation panels. After the discussion, all panel members electronically rated each proposal on a scale from 1 to 9. Panel members with conflicts of interest did not participate in the discussion or vote. Simple averages can introduce bias if panel members with distinct behaviors (eg, very critical or very generous) cannot vote. Furthermore, there may be uncertainty in ranking. A ranking approach based on a flexible bayesian hierarchical model (BHM) was developed to avoid the long and biased discussions of proposals around the funding line and was compared with standard procedure. The BHM accounted for the correlated data structure due to the same panel members voting on a set of proposals and modeled explicitly the uncertainty present at different stages of the evaluation process. As such, the BHM described the whole distribution of the rank of each proposal. The 50% credible intervals around the ranks helped assign the proposals into 3 groups: accepted, random selection, and rejected. The random selection group was composed of proposals of similar quality near the funding line. The approach was flexible and could take special cases into account. For example, if proposals were discussed in subpanels, the further level of dependency could be accounted for in the model. The convergence of the BHM was investigated using Gelman-Rubin convergence diagnostics.

Results The approach was simulated in the Swiss National Science Foundation (SNSF) early career fellowship scheme call of February 2020. A total of 181 fellowship grants were submitted to 5 disciplinary panels—humanities (23 fellowship grants), social sciences (38), medicine (35), biology (35), and STEM (50); 79 were discussed in the panel, and 32 were funded (Figure 12). A funding line was drawn based on the available budget. A random selection group was identified for 2 panels: medicine and STEM. Applications to other funding schemes will be presented.



Conclusions In this study, a method to address the limitations of peer review of good but not outstanding proposals was developed. The bayesian ranking approach ensured a transparent translation of votes into a ranking of the proposals. The approach was extensively discussed with stakeholders in 2021, and the Research Council of the SNSF recently decided to adopt it in most of its funding schemes later in 2022.

Conflict of Interest Disclosures Matthias Egger is the president of the Research Council of the Swiss National Science Foundation. No other disclosures were reported.

Downloads

SlidesTranscript English (automatic)

Next from Peer Review Congress 2022

Epidemiology of Scientific Output During the COVID-19 Pandemic | VIDEO
technical paper

Epidemiology of Scientific Output During the COVID-19 Pandemic | VIDEO

Peer Review Congress 2022

Anne Yang

08 September 2022

Similar lecture

Characterizing News Portrayal of Civil Unrest in Hong Kong, 1998–2020
workshop paper

Characterizing News Portrayal of Civil Unrest in Hong Kong, 1998–2020

IJCNLP-AACL 2021

Rohit Bhattacharya and 1 other author

02 August 2021

Stay up to date with the latest Underline news!

Select topic of interest (you can select more than one)

PRESENTATIONS

  • All Lectures
  • For Librarians
  • Resource Center
  • Free Trial
Underline Science, Inc.
1216 Broadway, 2nd Floor, New York, NY 10001, USA

© 2023 Underline - All rights reserved