A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports 
Darko Hren1, David G. Pina2, Cristopher Norman3, Ana Marušić4

1University of Split Faculty of Humanities and Social Sciences, Split, Croatia
2Research Executive Agency, European Commission, Brussels, Belgium
3LIMSI, CNRS, Université Paris Saclay, Paris, France
4University of Split School of Medicine, Split, Croatia

Objectives:
To identify themes in the content of reviewers' evaluation reports about proposals submitted to the FP7 Marie Curie Initial Training Networks calls and to explore the content areas in reviewers' comments that differentiate successful from unsuccessful proposals.

Methods:
We qualitatively analyzed the contents of reviewers' comments to obtain the most common themes within evaluation reports. Given a large amount of textual data, we used a two-step approach.
The first step was automated to help reduce the data to the manageable size. We used K-Means clustering to identify the clusters of statements within sections of the evaluation reports. Clusters were visualized in 2-dimensional space using TensorBoard Embedding Projector (https://projector.tensorflow.org/). Principal components analysis (PCA) was used for the visualization of clusters and cosine distances between clusters as a reference to cluster distances.
In the second step, the clusters were further grouped into themes using empirical estimates from PCA in step 1 and qualitative analysis based on cluster terms and content. To identify clusters that could potentially be grouped into themes, we used cosine distances lower than 1 as an arbitrary indicator of cluster proximity. Next, we inspected the content of the obtained clusters (terms and example statements) to verify whether and how groups of clusters could be interpreted as broader themes. Inspection of content had precedence over empirical data, i.e. if there was no meaningful overlap in cluster terms and content, even clusters that had cosine distance below 1 were not combined.
Finally, we registered the frequencies of obtained themes within each evaluation report (i.e. a number of sentences containing terms belonging to a specific theme) to inspect potential differences between proposals with different decision status.

Results:
We analyzed the content of reviews of 3666 grant proposals submitted to Marie Curie Initial Training Networks from the FP7 PEOPLE programme in 2008 (n=886), 2010 (n=857), 2011 (n=909), and 2012 (n=1013).
Each review had four sections of criteria (C1-Science & Technology Quality, C2-Training, C3-Implementation, and C4-Impact), and each section was divided into two parts (Strengths and Weaknesses). Therefore, 12 separate K-Means cluster analyses were performed in the first step, and we identified between 9 and 12 clusters explaining between 32.7 and 51.6% of the variance. 
In the second step, we identified between 29 themes within different sections of the evaluation reports (Figure 1A-D). Evaluation reports of proposals from the main and reserve lists included more themes related to strengths and fewer themes related to weaknesses than those of rejected proposals. Themes related to strengths were generally more frequent than those related to weaknesses. Most prominent themes related to strengths were: Research programme and objectives and Multidisciplinary context (C1), Consistency of the research and training programme and Career development (C2), Capacities of research and training (C3), and Outreach activities and Career prospects (C4). Most prominent themes related to weaknesses were: Research programme and objectives and Research methodology (C1), Engagement of ESRs and Training characteristics (C2), Capacities of research and training (C3), and Exploitation and dissemination (C4).

Conclusions:
We used a novel approach which combined machine learning and qualitative inquiry to identify the most common themes of the evaluation reports of proposals submitted to the FP7 Marie Curie Initial Training Networks calls. The themes reflected general evaluation criteria applied by the European Commission, but also specific themes that distinguished successful from unsuccessful proposals thus demonstrating the validity of the selection process.
The obtained results may be used by researchers preparing to apply for grants.



A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports

The Horizon 2020 Marie Skłodowska-Curie Actions (MSCA) Individual Fellowships (IF) funding programme is one of the largest and most competitive EU research programme (around 15% success rate). It would be expected that the applications of funded proposals have detected ethical issues well. The aim of this study was to describe the ethics evaluation process in the MSCA IF funding scheme . We first assessed the ethical issues identified by the applicants when preparing their proposals and compared these to the ethics issues identified by ethics experts during the ethics evaluation performed for those proposals retained for funding. In total, we analyzed 3074 proposals from three MSCA IF calls ( 2016, 2017 and 2018). Applicants identified almost two times fewer ethical issues compared to the experts performing the ethics review. Most of the ethics issues were identified in proposals from the Life Sciences and Social Sciences and Humanities research panels, which represented more than 60% of all proposals. Out of all 3074 proposals which were main listed and accepted for funding, 2225 (72.4%) received conditional ethics clearance, pending satisfaction of requirements defined by the ethics experts. The most prevalent requirements were within the Protection of personal data, Environment, Health and Safety, Humans, and non-EU countries categories. Our results emphasize that successful applicants to the MSCA IF programme, consisting mostly of postdoctoral researchers, are not adequately trained or knowledgeable about the ethics issues related to their proposed research, and that problem persists over time. Training in research ethics and proper guidance by the host institutions are paramount for narrowing the gap between the applicants’ ethics self-assessments and the results of the ethics review procedure. 


Ethics assessment of H2020 Marie Skłodowska-CUrie individual fellowships

We explore which characteristics of a reviewer affect disagreement with the other reviewers in research proposal evaluation, studying 52,488 proposals submitted to the Horizon 2020 Marie Skłodowska-Curie Actions (MSCA) and 1,939 proposals to the European Cooperation in Science and Technology Actions (COST). We find that experience in previous calls significantly reduces disagreement, and in MSCA - IF, an inverted U relationship between the number of proposals evaluated in a given call and disagreement, with a remarkable decrease of disagreement above thirteen proposals evaluated. The results suggest that reviewing experience improve reliability, curbing unwarranted disagreement by gradually fine-tuning reviewers’ evaluation.

Does reviewers' traits affect disagreement in proposal evaluation?

The organization of peer review processes in grant proposals evaluation is rarely studied. We analysed 77 thousand proposals from the Marie Curie programme, submitted over a period of 12 years, with the aim to get insights on the reliability of the process. Mean scores, Average Deviation (AD) indices, used as a measure of inter-rater agreement, and interrupted time series analysis were calculated to study changes in reviewers’ scores and level of agreement. Our results indicate that scores given by and agreement among reviewers may be more dependent on the type of grants or scientific panels than on the way the evaluation process is organised, with important economic and environmental implications for policy-making in research grants evaluation.

David Pina

4

5

SHORT BIO

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES