technical paper
A Bayesian Approach to Reduce Bias in the Ranking of Peer-Reviewed Grant Proposals Submitted to the Swiss National Science Foundation
keywords:
funding/grant peer review
statistics
bias
Objective Funding agencies rely on peer review to select
grant proposals deserving of funding. Peer review has
limitations and may be biased. The interrater reliability
between reviewers and panels is low, particularly for
proposals near the funding line. A method for ranking grant
proposals was developed to reduce bias and accommodate the
uncertainty in evaluation and funding decisions.
Design Proposals submitted to a call were peer reviewed and
later discussed in evaluation panels. After the discussion, all
panel members electronically rated each proposal on a scale
from 1 to 9. Panel members with conflicts of interest did not
participate in the discussion or vote. Simple averages can
introduce bias if panel members with distinct behaviors (eg,
very critical or very generous) cannot vote. Furthermore,
there may be uncertainty in ranking. A ranking approach
based on a flexible bayesian hierarchical model (BHM) was
developed to avoid the long and biased discussions of
proposals around the funding line and was compared with
standard procedure. The BHM accounted for the correlated
data structure due to the same panel members voting on a set
of proposals and modeled explicitly the uncertainty present at
different stages of the evaluation process. As such, the BHM
described the whole distribution of the rank of each proposal.
The 50% credible intervals around the ranks helped assign
the proposals into 3 groups: accepted, random selection, and
rejected. The random selection group was composed of
proposals of similar quality near the funding line. The
approach was flexible and could take special cases into
account. For example, if proposals were discussed in
subpanels, the further level of dependency could be accounted
for in the model. The convergence of the BHM was
investigated using Gelman-Rubin convergence diagnostics.
Results The approach was simulated in the Swiss National
Science Foundation (SNSF) early career fellowship scheme
call of February 2020. A total of 181 fellowship grants were
submitted to 5 disciplinary panels—humanities (23 fellowship
grants), social sciences (38), medicine (35), biology (35), and
STEM (50); 79 were discussed in the panel, and 32 were
funded (Figure 12). A funding line was drawn based on the
available budget. A random selection group was identified for
2 panels: medicine and STEM. Applications to other funding
schemes will be presented.
Conclusions In this study, a method to address the
limitations of peer review of good but not outstanding
proposals was developed. The bayesian ranking approach
ensured a transparent translation of votes into a ranking of
the proposals. The approach was extensively discussed with
stakeholders in 2021, and the Research Council of the SNSF
recently decided to adopt it in most of its funding schemes
later in 2022.
Conflict of Interest Disclosures Matthias Egger is the president
of the Research Council of the Swiss National Science Foundation.
No other disclosures were reported.