**Objective** The peer review system used by academic journals
to assess the quality of submitted papers is constantly in need
of new technologies that can help reviewers and increase the
objectivity of the process. The aim of the present study was to
develop a comprehensive list of words related to research
methods and statistics, which could then be used to
automatically study the content and the quantity and types of
methodological and statistical issues identified in peer review
reports.

**Design** The terms for the list were identified from different
glossaries.1,2 The terms were organized into subcategories,
some of which were more related to methodological aspects
(eg, study design, sampling, or procedures) or statistical
aspects (eg, parametric descriptive parameters or data
presentation) of research. The list was refined and enriched
with terms using a computational language model. The final
list and the preregistration can be found at the Open Science
Framework website (https://osf.io/d34b9/). The occurrence
of terms in the PEERE database3
of peer review reports, for
which data on reviewer gender, continental region, review
recommendation, journal impact factor, and subject area (N
= 496,928) were available, was counted. Regression models
were created to determine the frequency of the terms used
based on peer review characteristics.

**Results** The mean length of a peer review was 196 words
(95% CI, 108-330 words). A total of 26.8% of review reports
in the sample contained at least 1 methodological term, while
70.7% contained at least 1 statistical term. It was more likely
that the review contained methodological terms if the
reviewer was female and resided in North America or Oceania
compared with other continents, if the recommendation was
to reject the article or send it for major revisions, if the
journal was from the social sciences and humanities, if the
journal was in the top impact factor quartile, and if the article
was a longer peer review. More words related to statistics
were observed when the reviewer was female, for reviews
recommending any revisions, for journals from health and
medical sciences, in journals in the top impact factor quartile,
and in longer reviews. Mixed regression models indicated
that the scientific area was more strongly associated with
words related to statistics than were reviewer
recommendations.

**Conclusions** The application of this newly developed list
showed that methodological and statistical terms are not a
common topic in peer review and that they are not always
present in review reports from different fields. The list
successfully covered research terms because the most-
identified terms were related to categories relevant for most
types of research. Future analysis should try to understand
the context of the occurrence of words from the list to confirm
the validity of the list of terms.

**References**

1. Stark PB. Glossary of Statistical Terms. Accessed March 23,
2021. https://www.stat.berkeley.edu/~stark/SticiGui/Text/
gloss.htm
2. Framework for Open and Reproducible Research Training.
The Glossary of Terms. Accessed July 12, 2021. https://docs.
google.com/document/d/1oNsHSRmgAsDGUKvq5joqBpv
HeRmDJKldH2kSyJovHhk/edit#heading=h.2l8ssl89e1ey
3. Squazzoni F, Ahrweiler P, Barros T, et al. Unlock ways to
share data on peer review. Nature. 2020;578(7796):512-514.
doi:10.1038/d41586-020-00500-y

**Conflict of Interest Disclosures** Ana Marušić is a member of
the Peer Review Congress Advisory Board by was not involved in the
editorial review or decision for this abstract. No other disclosures
reported.

**Funding/Support** This study was funded by the Croatian Science
Foundation “Professionalism in Health–Decision making in
practice and research” under grant agreement IP-2019-04-4882.
Richard A. Klein received funding from a Replication Studies grant
(401.18.053) from the Netherlands Organization for Scientific
Research, and a Consolidator Grant (IMPROVE) from the European
Research Council (grant 726361). Francisco Grimaldo and Daniel
Garcia-Costa are partially supported by the Spanish Ministry of
Science, Innovation and Universities, the Spanish State Research
Agency, and the European Regional Development Fund under
project RTI2018-095820-B-I00.

**Role of the Funder/Sponsor** The funders had no role in the
design and conduct of the study; collection, management, analysis,
and interpretation of the data; preparation, review, or approval of
the abstract; and decision to submit the abstract for presentation.

![](https://assets.underline.io/uploads/markdown_image/1/image/eba25e8a44f7081b702973e5ddf9a854.png)

Development of a List to Detect Statistical and Methodological Terms in Peer Reviews: A Cross-Sectional Study

**Objective** The evaluation of European Union research grant
proposals consists of 2 consecutive steps: (1) individual expert
assessment and (2) consensus evaluation made by multiple
reviewers. The result is an evaluation summary report, and
previous studies have established this approach as a stable
procedure in the assessment of research grants.1,2 In 2020,
numerical scores were replaced by textual comments in the
individual expert assessment. The objective was to compare
the linguistic characteristics of the comments for Excellence,
Impact and Implementation criteria in evaluation reports of
Marie Skłodowska-Curie Actions’ Innovative Training
Networks (ITN) proposals submitted in 2019 and 2020 to
assess whether the removal of numerical scoring affected the
structure of individuals evaluation report textual comments
and evaluation outcome. 
 **Design** In this observational study for which data were
collected in fall 2022, all ITN proposals submitted in 2019
and 2020 were considered. Information was collected about
proposal scores and outcome, evaluation panel, and textual
comments of the individual expert evaluations on all
proposals submitted to the call. Linguistic characteristics of
experts’ comments were assessed using the Linguistic Inquiry
and Word Count software, a program that counts words
related to different psychological states and phenomena and
gives a score that is a proportion of the specific category in the
entire text. We used logistic regression to compare differences
between the 2 call years, in which proposal variables
(proposal status, word count for research excellence
weaknesses, word count for implementation strengths, and
negative effect levels for implementation strengths) were
factors and the year of the call was criterion, with the
significance level set at P < .001. 
 **Results** The number of proposals was similar in 2019 (n =
1554) and 2020 (n = 1503). The proportion of accepted
proposals was slightly higher in 2020 (148 [9.85%]) than in
2019 (128 [8.24%]) (**Table 37**). In logistic regression,
experts’ comments from 2020 differed from 2019 proposals
in 2 linguistic domains. Comments in the Excellence section
related to weaknesses had a greater number of words in the
description of the proposal (**Table 37**). The comments on
strengths in Implementation for proposals from 2020 had
slightly more words and lower negative tone or words related
to negative emotions, such as wrong, suffer, and sad
(**Table 37**). All factors jointly explained around only 4% of
the variance of the criterion. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/a21802046ca78c985d81a241cf135c85.png) 
 **Conclusions** It seems that removing numerical scoring in
the evaluation of ITN proposals at the stage of the individual
assessment had little effect on the linguistic characteristics of
the experts’ comments, because all differences were marginal
and we analyzed the whole proposal cohort.
**References** 
 1. Pina DG, Buljan I, Hren D, Marušić A. A retrospective
analysis of the peer review of more than 75,000 Marie Curie
proposals between 2007 and 2018. eLife. 2021;10:e59338.
doi:10.7554/eLife.59338 
 2. Buljan I, Pina DG, Marušić A. Ethics issues identified by
applicants and ethics experts in Horizon 2020 grant
proposals. F1000Res. 2021;10:471. doi:10.12688/
f1000research.52965.2 
 **Conflict of Interest Disclosures** David G. Pina is employed by
the European Research Executive Agency. Ana Marušić occasionally
serves as an ethics expert for the European Research Executive
Agency and is an advisory board member of the Peer Review
Congress but was not involved in the review or decision of this
abstract. Ivan Buljan and Antonia Mijatović declare no competing
interests. 
 **Funding/Support** This study was funded by the Croatian Science
Foundation “Professionalism in Health – Decision making in
practice and research” (ProDeM) under grant agreement IP-2019-
04-4882. 
 **Role of the Funder/Sponsor** The funders had no role in design
and conduct of the study; collection, management, analysis, and
interpretation of the data; preparation, review, or approval of the
abstract; and decision to submit the abstract for presentation. 
 **Disclaimer** All views expressed in this abstract are strictly those
of the authors and may in no circumstances be regarded as an
official position of the European Research Executive Agency or the
European Commission. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/dba03205861e88086430823162f000b1.png)

Comparison of Evaluations of Grant Proposals With and Without Numerical Scoring Submitted to Marie Sklodowska-Curie Actions Innovative Training Networks

Lightning Session 1

Peer Review Of Grant Proposals

Peer review is often criticized for being flawed, subjective and biased, but research into peer review is still lacking studies on peer review reports. In our study, in which text-analysis software was used to determine the linguistic characteristics of 472,449 peer review reports we compared a range of characteristics (including analytical tone, authenticity, clout, three measures of sentiment, and morality) as interaction of reviewer recommendation, area of research, type of peer review and reviewer gender. Our main conclusion was that the reviewer recommendation had the biggest impact on the linguistic characteristics of reports, and that area of research, type of peer review and reviewer gender had little or no impact. The lack of influence of research area, type of review or reviewer gender on the linguistic characteristics is a sign of the robustness of peer review.

Large-scale language analysis of peer review reports

A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports 
Darko Hren1, David G. Pina2, Cristopher Norman3, Ana Marušić4

1University of Split Faculty of Humanities and Social Sciences, Split, Croatia
2Research Executive Agency, European Commission, Brussels, Belgium
3LIMSI, CNRS, Université Paris Saclay, Paris, France
4University of Split School of Medicine, Split, Croatia

Objectives:
To identify themes in the content of reviewers' evaluation reports about proposals submitted to the FP7 Marie Curie Initial Training Networks calls and to explore the content areas in reviewers' comments that differentiate successful from unsuccessful proposals.

Methods:
We qualitatively analyzed the contents of reviewers' comments to obtain the most common themes within evaluation reports. Given a large amount of textual data, we used a two-step approach.
The first step was automated to help reduce the data to the manageable size. We used K-Means clustering to identify the clusters of statements within sections of the evaluation reports. Clusters were visualized in 2-dimensional space using TensorBoard Embedding Projector (https://projector.tensorflow.org/). Principal components analysis (PCA) was used for the visualization of clusters and cosine distances between clusters as a reference to cluster distances.
In the second step, the clusters were further grouped into themes using empirical estimates from PCA in step 1 and qualitative analysis based on cluster terms and content. To identify clusters that could potentially be grouped into themes, we used cosine distances lower than 1 as an arbitrary indicator of cluster proximity. Next, we inspected the content of the obtained clusters (terms and example statements) to verify whether and how groups of clusters could be interpreted as broader themes. Inspection of content had precedence over empirical data, i.e. if there was no meaningful overlap in cluster terms and content, even clusters that had cosine distance below 1 were not combined.
Finally, we registered the frequencies of obtained themes within each evaluation report (i.e. a number of sentences containing terms belonging to a specific theme) to inspect potential differences between proposals with different decision status.

Results:
We analyzed the content of reviews of 3666 grant proposals submitted to Marie Curie Initial Training Networks from the FP7 PEOPLE programme in 2008 (n=886), 2010 (n=857), 2011 (n=909), and 2012 (n=1013).
Each review had four sections of criteria (C1-Science & Technology Quality, C2-Training, C3-Implementation, and C4-Impact), and each section was divided into two parts (Strengths and Weaknesses). Therefore, 12 separate K-Means cluster analyses were performed in the first step, and we identified between 9 and 12 clusters explaining between 32.7 and 51.6% of the variance. 
In the second step, we identified between 29 themes within different sections of the evaluation reports (Figure 1A-D). Evaluation reports of proposals from the main and reserve lists included more themes related to strengths and fewer themes related to weaknesses than those of rejected proposals. Themes related to strengths were generally more frequent than those related to weaknesses. Most prominent themes related to strengths were: Research programme and objectives and Multidisciplinary context (C1), Consistency of the research and training programme and Career development (C2), Capacities of research and training (C3), and Outreach activities and Career prospects (C4). Most prominent themes related to weaknesses were: Research programme and objectives and Research methodology (C1), Engagement of ESRs and Training characteristics (C2), Capacities of research and training (C3), and Exploitation and dissemination (C4).

Conclusions:
We used a novel approach which combined machine learning and qualitative inquiry to identify the most common themes of the evaluation reports of proposals submitted to the FP7 Marie Curie Initial Training Networks calls. The themes reflected general evaluation criteria applied by the European Commission, but also specific themes that distinguished successful from unsuccessful proposals thus demonstrating the validity of the selection process.
The obtained results may be used by researchers preparing to apply for grants.



A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports

The Horizon 2020 Marie Skłodowska-Curie Actions (MSCA) Individual Fellowships (IF) funding programme is one of the largest and most competitive EU research programme (around 15% success rate). It would be expected that the applications of funded proposals have detected ethical issues well. The aim of this study was to describe the ethics evaluation process in the MSCA IF funding scheme . We first assessed the ethical issues identified by the applicants when preparing their proposals and compared these to the ethics issues identified by ethics experts during the ethics evaluation performed for those proposals retained for funding. In total, we analyzed 3074 proposals from three MSCA IF calls ( 2016, 2017 and 2018). Applicants identified almost two times fewer ethical issues compared to the experts performing the ethics review. Most of the ethics issues were identified in proposals from the Life Sciences and Social Sciences and Humanities research panels, which represented more than 60% of all proposals. Out of all 3074 proposals which were main listed and accepted for funding, 2225 (72.4%) received conditional ethics clearance, pending satisfaction of requirements defined by the ethics experts. The most prevalent requirements were within the Protection of personal data, Environment, Health and Safety, Humans, and non-EU countries categories. Our results emphasize that successful applicants to the MSCA IF programme, consisting mostly of postdoctoral researchers, are not adequately trained or knowledgeable about the ethics issues related to their proposed research, and that problem persists over time. Training in research ethics and proper guidance by the host institutions are paramount for narrowing the gap between the applicants’ ethics self-assessments and the results of the ethics review procedure. 


Ethics assessment of H2020 Marie Skłodowska-CUrie individual fellowships

We explore which characteristics of a reviewer affect disagreement with the other reviewers in research proposal evaluation, studying 52,488 proposals submitted to the Horizon 2020 Marie Skłodowska-Curie Actions (MSCA) and 1,939 proposals to the European Cooperation in Science and Technology Actions (COST). We find that experience in previous calls significantly reduces disagreement, and in MSCA - IF, an inverted U relationship between the number of proposals evaluated in a given call and disagreement, with a remarkable decrease of disagreement above thirteen proposals evaluated. The results suggest that reviewing experience improve reliability, curbing unwarranted disagreement by gradually fine-tuning reviewers’ evaluation.

Does reviewers' traits affect disagreement in proposal evaluation?

The organization of peer review processes in grant proposals evaluation is rarely studied. We analysed 77 thousand proposals from the Marie Curie programme, submitted over a period of 12 years, with the aim to get insights on the reliability of the process. Mean scores, Average Deviation (AD) indices, used as a measure of inter-rater agreement, and interrupted time series analysis were calculated to study changes in reviewers’ scores and level of agreement. Our results indicate that scores given by and agreement among reviewers may be more dependent on the type of grants or scientific panels than on the way the evaluation process is organised, with important economic and environmental implications for policy-making in research grants evaluation.

Reviewers' agreement and scores in Marie Curie proposals from 2007 to 2018

This is virtual poster session. Please visit the individual poster pages and ask your questions in the chat box.

Virtual Poster - Peer Review

poster

Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.
 
 **Welcome!**
 
 The JAMA Network, The BMJ, and METRICS welcome you to the Ninth International Congress on Peer Review and Scientific Publication. We have continued our efforts to broaden the scope of the Congress to all aspects of peer review and publication—from funding to postpublication—and to all sciences.
 
 We will have 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination. There are 50 plenary session research presentations and 5 plenary invited talks. Each plenary session research presentation will be followed by equal time for discussion and questions from the audience. In addition, there are 125 posters. All posters, including virtual posters, are available to view throughout the meeting. In-person posters will also be presented at designated times on September 9 and 10. If you are attending virtually, please visit the individual poster pages and ask your questions in the chat box. See the link to Posters on the left.
 
 We hope you will take an active part in the program, as we depend on your participation in the discussion sessions to make the Congress a success.

 [Peer Review Congress Organizers, Directors, and Advisory Board](https://peerreviewcongress.org/organizers-and-advisory-board/)
 
 [Health and Safety Policies](https://web.cvent.com/event/1b505794-6a0b-428c-a44c-6c46c4c50e99/websitePage:152962bb-4567-4561-9d81-21ebfa97f487)

![](https://assets.underline.io/uploads/markdown_image/1/image/7c9f0143dee383780cc4670269ade132.png)

A PDF of the schedule is available [here](https://drive.google.com/file/d/1-Jd191W-nxbWcN1prdX5o9nlSfeo9_-H/view?usp=sharing) and a flipbook version of the complete program and abstracts is included below. See also the link to the interactive schedule to the left.

<iframe width="700px" height="425px" src="https://online.fliphtml5.com/ebtyf/nlfn/#p=1" frameborder="0" allowfullscreen allowtransparency></iframe>

Registration is required to attend this event, please use the URL linked below to secure your ticket. Registration will close on Friday, September 2 at 5:00 PM CT. 

[**Register Here**]( https://web.cvent.com/event/1b505794-6a0b-428c-a44c-6c46c4c50e99/summary)

Peer Review Congress 2022

Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.



This is an inperson poster session. If you are attending virtually please visit the individual poster pages and ask your questions in the chat box.

PS1 - Funding/Sponsorship

**Chair:** Ana Marušić, University of Split

Lightning session 1

technical paper

Following the success of the 1st PEERE International Conference on Peer Review, the PEERE COST Action follow-up community organised the second edition of the PEERE International Conference on Peer Review  and was celebrated virtually on 29 September – 1 October 2020.

The conference welcomed any contribution (e.g., original research, position papers, literature reviews) that used any method (e.g., quantitative, experimental or qualitative) to investigate peer review in a variety of scientific domains (e.g., scholarly journals, funding agencies, research assessment). Interdisciplinary research is especially welcome.

The program is published here: http://tiny.cc/peere2020 


PEERE

The second edition of the PEERE International Conference on Peer Review aims to provide a forum for scholars, practitioners and science stakeholders to share research, models, tools and experience on peer review in different fields, e.g., medicine, computer science, social sciences and humanities. It aims to stimulate the use of evidence-based research in the design and implementation of peer review in a variety of fields and encourage more systematic research.

Ana Marusic

9

14

SHORT BIO

OTHER AFFILIATIONS

Presentations

Development of a List to Detect Statistical and Methodological Terms in Peer Reviews: A Cross-Sectional Study

Comparison of Evaluations of Grant Proposals With and Without Numerical Scoring Submitted to Marie Sklodowska-Curie Actions Innovative Training Networks

Lightning Session 1

Peer Review Of Grant Proposals

Large-scale language analysis of peer review reports

A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports

Ethics assessment of H2020 Marie Skłodowska-CUrie individual fellowships

Does reviewers' traits affect disagreement in proposal evaluation?

Reviewers' agreement and scores in Marie Curie proposals from 2007 to 2018

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES