"Introduction
In this study, we tested the effect of receiving a short individual feedback report compared to a short general feedback report on the agreement between reviewers.

Methods
A total of 42 reviewers at the Norwegian foundation Dam were randomly assigned to receive either a general feedback report or an individual feedback report. The general feedback group received one report before the start of the reviews that contained general information about the previous call the reviewers took part in. In the individual feedback group, the reviewers received two reports, one before the review period (based on the previous call) and one during the period (based on the current call). In the individual feedback group, the reviewers were presented with detailed information on their scoring compared with the review committee as a whole. The main outcomes were the proportion of agreement in the eligibility assessment and the score difference between pairs of reviewers assessing the same application.

Results
A total of 2398 paired reviews were included in the analysis. There was no difference between the two groups in score difference. There was a significant difference between the two groups in the proportion of absolute agreement on whether the application was eligible for the funding program or not, with the general feedback group demonstrating a higher rate of agreement. However, the overall levels of agreement remained critically low in both groups and for both outcomes.

Conclusion
The results did not support our hypothesis but are in line with related studies in journal peer review. The use of a convenience sample and the limited sample size is are important limitations in this study. The low levels of agreement remain a major concern in grant peer review and research to identify factors contributing to this, as well as development and testing of interventions aiming to increase agreement rates, are still needed.

The study was preregistered at OSF.io/n4fq3"

Feedback to improve agreement in grant peer review

<br>



**Chair:** Marco Seeber, University of Agder


4. Peer review of grant proposals

technical paper

Following the success of the 1st PEERE International Conference on Peer Review, the PEERE COST Action follow-up community organised the second edition of the PEERE International Conference on Peer Review  and was celebrated virtually on 29 September – 1 October 2020.

The conference welcomed any contribution (e.g., original research, position papers, literature reviews) that used any method (e.g., quantitative, experimental or qualitative) to investigate peer review in a variety of scientific domains (e.g., scholarly journals, funding agencies, research assessment). Interdisciplinary research is especially welcome.

The program is published here: http://tiny.cc/peere2020 


PEERE

The second edition of the PEERE International Conference on Peer Review aims to provide a forum for scholars, practitioners and science stakeholders to share research, models, tools and experience on peer review in different fields, e.g., medicine, computer science, social sciences and humanities. It aims to stimulate the use of evidence-based research in the design and implementation of peer review in a variety of fields and encourage more systematic research.

Jan-Ole Hesselberg

1

SHORT BIO

OTHER AFFILIATIONS

Presentations

Feedback to improve agreement in grant peer review

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES