Spain

We explore which characteristics of a reviewer affect disagreement with the other reviewers in research proposal evaluation, studying 52,488 proposals submitted to the Horizon 2020 Marie Skłodowska-Curie Actions (MSCA) and 1,939 proposals to the European Cooperation in Science and Technology Actions (COST). We find that experience in previous calls significantly reduces disagreement, and in MSCA - IF, an inverted U relationship between the number of proposals evaluated in a given call and disagreement, with a remarkable decrease of disagreement above thirteen proposals evaluated. The results suggest that reviewing experience improve reliability, curbing unwarranted disagreement by gradually fine-tuning reviewers’ evaluation.

PEERE

Does reviewers&#39; traits affect disagreement in proposal evaluation?

research funding

economics of science

sociology of science

horizon 2020

msca

project evaluation

cost

experience

research evaluation

disagreement

reliability

### Welcome to the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL) 
Continuing its mission of expanding and involving the science community of all European countries, EACL had selected the Ukrainian community for the 16th EACL, which took place online due to the COVID pandemic. Unfortunately, the ongoing war made the organisation in Kyiv impossible. Considering the importance of physical interaction among researchers, especially after the restrictions imposed by the COVID pandemic, in addition to an online mode, the conference will be held in Dubrovnik, Croatia, from 2 to 6 of May, 2023. The original aim of strengthening the connection with the Ukrainian community will not change as our program will feature a dedicated session and a workshop to highlight work on Ukrainian language technologies. As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language.

You need to register for the conference in order to access this site. Please visit https://2023.eacl.org/registration for more information.

EACL 2023

As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language.

Does reviewers' traits affect disagreement in proposal evaluation?

technical paper

Following the success of the 1st PEERE International Conference on Peer Review, the PEERE COST Action follow-up community organised the second edition of the PEERE International Conference on Peer Review  and was celebrated virtually on 29 September – 1 October 2020.

The conference welcomed any contribution (e.g., original research, position papers, literature reviews) that used any method (e.g., quantitative, experimental or qualitative) to investigate peer review in a variety of scientific domains (e.g., scholarly journals, funding agencies, research assessment). Interdisciplinary research is especially welcome.

The program is published here: http://tiny.cc/peere2020 


The second edition of the PEERE International Conference on Peer Review aims to provide a forum for scholars, practitioners and science stakeholders to share research, models, tools and experience on peer review in different fields, e.g., medicine, computer science, social sciences and humanities. It aims to stimulate the use of evidence-based research in the design and implementation of peer review in a variety of fields and encourage more systematic research.

The organization of peer review processes in grant proposals evaluation is rarely studied. We analysed 77 thousand proposals from the Marie Curie programme, submitted over a period of 12 years, with the aim to get insights on the reliability of the process. Mean scores, Average Deviation (AD) indices, used as a measure of inter-rater agreement, and interrupted time series analysis were calculated to study changes in reviewers’ scores and level of agreement. Our results indicate that scores given by and agreement among reviewers may be more dependent on the type of grants or scientific panels than on the way the evaluation process is organised, with important economic and environmental implications for policy-making in research grants evaluation.

Reviewers' agreement and scores in Marie Curie proposals from 2007 to 2018

Background: Every year, thousands of research proposals are submitted to funding organisations in the hope of securing funding for research. These applications go through a number of assessment stages in which decisions are made about whether to decline the application, progress the application or fund the application (1–3). Funding organisations often have limited funds, large numbers of application submissions, and a need to demonstrate impactful spending, and so many of these applications go through multiple iterations and not all will be funded. 
Feedback to the applicants about these decisions are a vital part of the scientific process. Applicants rely on feedback to improve the quality of their application for securing successful funding and/or as a learning opportunity to facilitate the chance of success in future applications. It is therefore important that applicants receive constructive feedback in order to improve the quality of their application, change the content of their application or to understand why their application was or was not successful (was it competitive but there were stronger applications, or were significant improvements required?). A lack of constructive feedback could not only impact on professional development for researchers but could introduce unnecessary burden and negative experiences of researchers as applicants, in what is already a labour intensive research activity (1,4–7). 
For the National Institute for Health Research (NIHR) Evaluation, Trials and Studies Coordinating Centre (NETSCC), applicants receive written feedback by the research programme funding committees at both stage 1 and stage 2 of the application process. Stage 1 feedback is provided for applications that are progressed to stage 2, as well as for those that were unsuccessful. For applications that successfully progress to stage 2, feedback is again provided for both successful and unsuccessful applications at this second stage. Committees are encouraged to consider providing targeted feedback for each application rather than using generic phrases. Feedback points are drafted in the funding committee meetings and are provided to the applicants in written bullet point format. NIHR applicants also have access to guidance on what to include in their applications and the assessment criteria against which the application will be assessed. 

In order to support applicants throughout the funding process, it is vital that the guidance and feedback provided is of high quality, constructive and consistent. It is therefore important to understand the key components of the feedback provided to applicants at Stage 1 and 2 and how these are perceived by applicants. This will lead to increased knowledge on the quality and value of the feedback provided to applicants, whether improvements are needed regarding the type of feedback given and whether this is consistent across the NIHR research programmes managed by NETSCC, promoting the adding value in research approach and transparency agenda. 

Objectives: The aim is to identify and compare the key components of feedback that are provided to applicants by NIHR research programmes, whether these components are consistent across different sources of feedback (external peer review, funding committees at stage 1, funding committees at stage 2, guidance documents) and how the feedback is perceived by applicants.

Methods: To gain a NIHR funder perspective of feedback provided, qualitative content analysis of feedback and guidance documents from the Health Technology Assessment (HTA), Public Health Research (PHR), Efficacy and Mechanism Evaluation (EME), and Health Services and Delivery Research (HSDR) funding programmes was conducted. Following ethical approval, purposeful sampling was used to select commissioned-led and researcher-led, and successful and unsuccessful research applications from March 2018 and March 2019. To date, a total of 84 applicants across the four funding programmes have been contacted to gain approvals to access the feedback data. Using a phased approach, (1) a coding framework was created using inductive content analysis of external peer reviews, stage 1 and stage 2 funding committee feedback from the HTA programme, and (2) then applied to the stage 2 funding committee feedback from the remaining funding programmes using deductive content analysis. (3) The final phase will compare the key components across funding programmes and to the guidance criteria provided for each programme. 
To gain an NIHR applicant perspective about feedback received from funding committees, an online survey was conducted in October 2019. Following ethical approval, the survey used purposeful sampling and targeted chief investigators of applications submitted to HTA, EME, PHR and HSDR research programmes during March 2018 and March 2019 to ensure the sample reached commissioned-led, researcher-led and successful and unsuccessful applicants. The survey was distributed via email to 770 applicants and was also published on Twitter. The survey used both closed and open questions to explore how useful the feedback was and considerations for improvements for both stages 1 and 2. A mixed methods analysis using descriptive statistics and inductive thematic analysis was conducted on closed and open question responses.
Results: From the NETSCC perspective, qualitative content analysis of feedback and guidance documents is on-going. To date, feedback documents relating to 25 HTA applications have been analysed. From this, a coding framework has been created. Seven key components (with sub-components) were identified from the feedback given to HTA applicants:
(1) Acceptability to patients and professionals: feedback on patient and public involvement in the study design.
(2) Team and infrastructure: feedback on justification of the roles and time of the team as well as potential conflicts of interest for the team.
(3) Justification of project: feedback on the scientific rationale, relevance and need of the project.
(4) Study design: feedback was primarily around clearly defining and justifying study parameters, outcomes and sample size.
(5) Risks and contingencies: feedback highlighted the need for consideration to be given to recruitment, retention and attrition, and feasibility of the project.
(6) Value for money and costings: feedback around cost elements of the application.
(7) Pathway to impact and implementation: feedback around implementation plans and whether project will have clinical impact.
The most frequently reported components were study design; risks and contingencies; and team and infrastructure. The majority of the components were consistently reported across the funding committee and external peer reviewer comments, except for the justification of project component, which was only provided in feedback to applicants at stage 1. Recruitment for documents in the remaining research programmes commenced in October 2019. To date, we have recruited 23 applicants across PHR (8), EME (6) and HSDR (9), and the coding framework is being applied to stage 2 funding committee feedback and guidance documents. 
From the applicant perspective, 147 applicants completed the online survey (response rate of 19%). After removing data from applications under review, non-NETSCC research programmes and incomplete datasets, a final sample of 115 applicant data was analysed. Overall, more than half of applicants agreed that the feedback provided by the funding committee was useful (52%) and of benefit (53%). However, more than half of applicants also disagreed that the feedback was of good quality (54%) or that it was helpful for developing future applications or work (57%). Responses differed depending on whether the applicant had been successful in securing funding, with at least 77% of funded applicants responding positively about the feedback and less than 37% of unsuccessful applicants responding positively. Moreover, feedback at each stage was more likely to be seen positively if the application was moved on to the next stage/funded whilst feedback at the point of rejection was more likely to be considered negatively. For example, unsuccessful applicants at stage 2 regarded stage 1 feedback positively, but regarded stage 2 feedback negatively, finding it unhelpful and unclear on areas for development. The open-ended responses also suggested that feedback could be more detailed and directive to facilitate changes and that it would be helpful to get some indication of how important each comment was (is it mandatory and which to focus more attention on). In contrast to the findings from the NIHR funder perspective, feedback from the applicant perspective showed that feedback is not always seen as being consistent across stages, with guidance, briefs or within the same feedback document. 
Discussion: The findings will lead to increased knowledge on the quality, transparency and consistency of feedback provided to applicants, whether this is consistent across research programmes and how useful the feedback is to applicants. The preliminary findings show that key components provided in feedback by the different NIHR research programmes were consistently reported across both the funding committee and external peer reviewer comments. However, applicants reported that the content of the feedback was not always consistent across both stages and within stages, and that although applicants found this feedback useful overall, there are a number of areas in which feedback and guidance could be improved. Once the full analysis has been completed, recommendations can then be made for NIHR research programme guidance and feedback to applicants that supports the chance of a successful application, promoting the adding value in research approach and transparency agenda. 
This work is part of a wider series of studies being undertaken by the NIHR Research on Research team on peer review and decision-making in funding allocation. This wider work includes the studies reported here, a realist synthesis to identify contexts in which different decision-making processes work, a survey with international funders to understand current funding practices, qualitative analysis to understand stakeholders’ expectations and an observational study of funding committees and will provide much needed evidence to help inform funders in peer review and decision-making activities.

References:
1. Guthrie S, Ghiga I, Wooding S. What do we know about grant peer review in the health sciences? An updated review of the literature and six case studies. Santa Monica, CA; 2018. 
2. Ismail S, Farrands A, Wooding S, Europe R. Evaluating grant peer review in the health sciences. A review of the literature. Cambridge, UK; 2009. 
3. Publons part of the Web of Science Group. Grant Review in Focus. Global State of Peer Review Series. 2019. 
4. Gluckman P. Which science to fund: time to review peer review? 2012;(December):11. 
5. Graves N, Barnett AG, Clarke P. Cutting random funding decisions. Nature. 2011;469(7330):299. 
6. Herbert DL, Coveney J, Clarke P, Graves N, Barnett AG. The impact of funding deadlines on personal workloads, stress and family relationships: a qualitative study of Australian researchers. BMJ Open. 2014;4(3):e004462. 
7. Herbert DL, Barnett AG, Clarke P, Graves N. On the time spent preparing grant proposals: an observational study of Australian researchers. BMJ Open. 2013;3(5):1–6. 



Identification and comparison of key criteria of funding decision feedback to applicants: A funder and applicant perspective

We aimed to investigate patterns of characteristics for scholars who review for predatory journals and for legitimate journals. To do so, we matched representative samples of predatory journals and legitimate journals of the Cabells Scholarly Analytics’ journal lists with the Publons database. For reviewers of matched reviews, we analysed reviewing and publishing behaviour. We studied 183,743 unique Publons reviews which were claimed by 19,598 reviewers. The vast majority of reviews were claimed for legitimate journals, only few reviews were submitted for predatory journals. Most scholars never or only occasionally claimed reviews for predatory journals on Publons. Only small numbers of scholars submitted reviews predominantly or exclusively for predatory journals. The latter group of scholars showed a younger academic age and had fewer publications and fewer reviews than the first group. To tackle predatory journals, a holistic approach is needed that takes into account the entire research workflow.

Who reviews for predatory and legitimate journals? A study on reviewer characteristics

Peer review is also used when deciding about awards and prizes. We are interested in the question whether gender differences exist in these decision-making processes. Using a case of awarding cum laude for PhD theses, we find a strong gender effect, as men have a twice as high probability to receive cum laude then women do, even after controlling for quality differences.

Gender bias in granting awards: cum laude for PhD theses

Between 21 May 2016 and 9 September 2019, 1,983 bioRxiv preprints received a single public comment on the bioRxiv website. More than two thirds of those comments were posted by non-authors (n=1,366, 69%) while the remainder were posted by the preprint’s authors (n=617, 31%). Overall, the non-author comments were longer than comments posted by the authors (median number of words was 38, IQR 17 to 83, for non-authors vs 18, IQR 11 to 32, for authors, Mann-Whitney test, P<0.001), and they were posted a median of 23 days (IQR 3 to 117) after the preprints. In comparison, authors’ comments were posted after a median of 91 days (IQR 3 to 23, Mann-Whitney test, P<0.001). Twelve percent of non-author’s comments (n=168) were full review reports resembling those traditionally submitted during the journal peer review process. Comments not resembling full peer review reports most commonly praised the preprint (n=577, 42%), made suggestions on how to improve it (n=399, 29%), or criticized some aspect of the preprint (n=226, 17%). Authors’ comments most commonly contained updates about the preprint’s publication status (n=354, 57%), additional information on the study (n=158, 26%), or solicited feedback for the preprint (n=65, 11%).

From amazing work to I beg to differ: analysis of bioRxiv preprints that received one public comment till September 2019

Legal scholarship has an ambiguous identity, somewhere between the humanities and the social sciences, having features in common with both. It shares many of the characteristics of social sciences since the law is a social phenomenon, but when normativity and legal certainty are concerned, legal scholarship is probably closer to humanities (Peruginelli & Faro, 2018). At the same time law is distinct from both. There is an interesting debate on the role of academic legal research in which consideration is given to 'law as a practical discipline', 'law as humanities' and 'law as social sciences' (Siems & Mac Sithigh, 2012). Because of the peculiarities of legal scholarship, it is not easy to assess legal journals according to the standard quantitative metrics applied in social sciences, and qualitative assessment of the published content is more appropriate. Therefore, we compared the peer review process employed by legal journals in three countries, Croatia, Italy and Spain, to find out their characteristics. 
In this study, 34 Croatian, 36 Spanish and 40 Italian law journals (selected randomly from the 153 top ranking Italian law journals) were analysed. For each journal, we collected basic set of metadata, as well as a document providing the description of the peer-review process. We collected 107 documents in four languages (English, Croatian, Italian and Spanish) which were analysed in order to find out characteristics of the peer review process. For text analysis, we developed an extensive categorization dictionary. Nine main categories in the Categorization dictionary are peer review (process and reviewers), reviewer's characteristics (academic level, autonomy, competence), reviewer's provenance (external, internal), peer review blindness/openness (single-blind, double-blind, open), number of reviewers (one, two), evaluation criteria for submissions (originality, methods, relevance, clarity, accuracy, etc.), peer review outcome (report, acceptance, rejection), ethical issues (editorial standards and codes, conflict of interests, confidentiality, research integrity), manuscript type (original scientific article, professional article, etc.).
The most frequent categories are "peer review", "submissions' evaluation criteria" and "manuscript type", present in more than 90% of the collected documents. "Reviewer's competences" are the most represented subcategory in "“reviewer's characteristics“, as high level of reviewers expertise in the legal sciences is important for law journals. 73% of analyzed journals are declaring anonymous peer review, 25% double-blind peer review, and only one journal employ single-blind peer review. 42% of documents contained information on "two reviewers", 8% "one reviewer" and 7% "three or more reviewers".
Ethical issues are certainly under-represented in documents describing peer review process and editorial policies issued by Italian, Spanish and Croatian law journals. Only 32% of the documents mention conflict of interests, and all other subcategories are present even less than 30%. 
Croatian legal journals are addressing research misconduct and different publication types more than the other two countries. Also, they consider manuscript originality as very important, but some important submission evaluation criteria, reviewer characteristics and peer review outcomes are not present. On the other hand, Spanish legal journals are addressing more reviewer's competences, but some important ethical issues are missing. Italian legal journals take care about the reviewer's competences, academic level and objectivity, as well as anonymity of the peer review process, but expected peer review outcomes and ethical issues should be mentioned in a greater extent.
References
Gutwirth, S. (2009). The evaluation of legal science. The Vl.I.R.-model for integral quality assessment of research in law : what next ? In H. Eisendrath & J. P. Van Bendegem (Eds.), It takes two to do science. The puzzling interactions between science and society (pp. 69–80). VUBPress.
Peruginelli, G., & Faro, S. (2018). Research quality evaluation: The case of legal studies. In The Evaluation of Research in Social Sciences and Humanities (pp. 103–129). Springer Cham.
Siems, M., & Mac Sithigh, D. (2012). Mapping Legal Research. Cambridge Law Journal, 71(3), 651–676.




Peer Review in Legal journals

The received wisdom of online user experience is that text must be kept to an absolute minimum if you are to hold your users’ attention. It came as a shock to us, then, when the long-winded, legally-worded question asking referees to allow us to pass their reviewer activity for Springer Nature journals on to Publons produced an opt-in rate of 56% — substantially higher than the opt-in of 37% for the same service for ORCID offered more succinctly. Based on user experience studies carried out for ORCiD opt-in, we believe that part of the reason for this is that the greater detail given during opt-in improves the opt-in rate.

Do Words Matter? Increasing buy-in by building trust with peer reviewers on Publons and ORCID

Peer review has multiple functions. On the one hand, it helps screening the quality of manuscripts for publication. On the other, it allows authors to increase the value of their manuscripts by constructive feedback. However, measuring the developmental value of peer review is difficult as it requires complete, large-scale manuscript data rarely available. With complete data from ten journals from the Royal Society from 2006 to 2017 in various fields of research, we estimated the effect of the number of reviewers and the reviewer score on manuscript changes throughout peer review. We found that peer review improves manuscripts of moderate initial quality often leading them to receive more citations than manuscripts initially receiving higher reviewer scores. Most beneficial changes include the last sections of manuscripts, i.e., discussion/conclusions, and especially references, whose changes were beneficial in terms of citations. The developmental effect of peer review is marginal whenever more than three reviewers assess the same manuscript. Our findings suggest that peer review can essentially work both as a quality screening and a developmental process.

Peer Review Improves Manuscripts of Moderate Initial Quality. A Study on Ten Journals from the Royal Society (2006-2017)

Finding good peer reviewers is a difficult task. In Decentralized Science1project we are designing and developing a tool to improve the quality, fairness and reliability of academic peer reviewing. Our approach relies in opening peer review, giving transparency to the peer reviewing process using decentralized technologies such as Blockchain. During our ongoing product design research we gained interesting insights about the peer reviewing selection process, and how editors currently deal with it. Our research methods are oriented towards the development of a software tool. We use Lean Design and Agile development principles, favoring fast iterative learning over the precision and completeness of more formal approaches. This contribution shares what we learned in the process about how editors deal with peer reviewer selection: from their need sand complains to their tricks, including some of their confessions. It also explains how we embraced this insights to improve our current prototype design.

The challenges of finding peer reviewers: Insights from our product design research

A presentation of a taxonomy for peer review that will be used to describe the peer review process to authors and readers in a consistent and transparent way.

Downloads

Next from PEERE

Reviewers' agreement and scores in Marie Curie proposals from 2007 to 2018

Similar lecture

Why Don't You Do It Right? Analysing Annotators' Disagreement in Subjective Tasks

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from PEERE

Reviewers' agreement and scores in Marie Curie proposals from 2007 to 2018

Similar lecture

Why Don't You Do It Right? Analysing Annotators' Disagreement in Subjective Tasks

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads