<p>Luka Ursić,<sup>1,2</sup> Nensi Bralić,<sup>1</sup> Giovanni Spitale,<sup>2</sup> Federico Germani,<sup>2</sup> Ana Marušić<sup>1</sup></p>
<h4>Objective</h4>
<p> Divisive discourse observed in news coverage or social media posts by researchers and the general public during the H1N1 and COVID-19 pandemics contributed to polarization and politicization, rapid shifts in sentiment, and concerning rates of misinformation.<sup>1-3</sup> We aimed to study the sentiment and natural language processing (NLP) analysis of editorial material in medical journals discussing the health care system response to the 2 pandemics.</p>
<h4>Design</h4>
<p> Using a systematic review design, we searched Web of Science and PubMed up to May 22, 2024, for editorials, viewpoints, and similar opinion pieces discussing the following segments of the health care system response to the 2 pandemics: nonpharmaceutical interventions; misinformation or disinformation; health care resource allocation and management; health care system preparedness; mandates, policies, and guidance related to research, education, or new technologies; and lessons learned. We limited our search to 2009 to 2014 for the H1N1 pandemic and 2019 to 2024 for the COVID-19 pandemic. Three researchers screened the titles, abstracts, and full texts of the retrieved records for eligibility in pairs, resolving discrepancies through discussion. We extracted the text from the included records’ PDFs using Python and performed a sentiment analysis using the Language Inquiry and Word Count 2022 software. We also used NLP approaches (flat lemmatization, convoluted lemmatization, rule-based autocoding, and cosine similarity) to determine the most frequently used lemmas in the 2 groups of editorials and explore topics based on their co-occurrence. We conducted inferential analyses at the level of the full sample and across specific categories (ie, stratified by pandemic, author country, and context of response). While we could not set a hypothesis for the exploratory NLP analysis, we hypothesized that in terms of sentiment, the editorials published during the COVID-19 pandemic would have higher scores for negative tone and emotion, personal and person-centered language, certitude, and all-or-none thinking and lower scores for analytical thinking.</p>
<h4>Results </h4>
<p>Following the screening process (<b>Figure 25-0979</b>), we included 2954 editorials in the final dataset. We performed a preliminary analysis of 200 editorials: 25 for the H1N1 pandemic and 175 for the COVID-19 pandemic. In contrast to our initial hypothesis, the editorials for the H1N1 pandemic had higher values for negative tone (median, 2.21 [IQR, 1.74-3.07] vs 1.69 [IQR, 1.22-2.31]; <i>P</i> = .007) and certitude (median, 0.39 [IQR, 0.16-0.66] vs 0.23 [IQR, 0.12-0.37]; <i>P</i> = .01). There were no differences in negative emotion, analytical thinking, or all-or-none thinking.</p>

![](https://assets.underline.io/markdown_image/1/image/d658c3c3e26c29b586c4a178cb5264eb.png)

<h4>Conclusions </h4>
<p>In this preliminary analysis, the editorials for the H1N1 pandemic contained more words connoting negative tone and certitude than those for the COVID-19 pandemic. These findings suggest that the researchers authoring these editorials expressed a more negative sentiment toward the public health response to the H1N1 pandemic and that they were seemingly more certain about their position.</p>
<h4>References</h4>
<p>1. Schmidt H. Pandemics and politics: analyzing the politicization and polarization of pandemic-related reporting. <i>Newsp Res J</i>. 2023;44(1):26-52. doi:10.1177/07395329221095850</p>
<p>2. Lwin MO, Lu J, Sheldenkar A, et al. Global sentiments surrounding the COVID-19 pandemic on Twitter: analysis of Twitter trends. <i>JMIR Public Health Surveill</i>. 2020;6(2):e19447. doi:10.2196/19447</p>
<p>3. Sule S, DaCosta MC, DeCou E, Gilson C, Wallace K, Goff SL. Communication of COVID-19 misinformation on social media by physicians in the US. <i>JAMA Netw Open</i>. 2023;6(8):e2328928. doi:10.1001/jamanetworkopen.2023.28928</p>
<sup>1</sup>Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia, luka.ursic@mefst.hr; <sup>2</sup>Institute of Biomedical Ethics and History of Medicine, University of Zürich, Zürich, Switzerland.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Ana Marušić is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Funding/Support </h4>
<p>This study was funded by the Croatian Science Foundation under grant agreement IP-2019-04-4882 and the University of Zürich Institute of Biomedical Ethics and History of Medicine under a Stehr-Boldt fellowship given to Luka Ursić.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funders had no role in the design of this study, its execution, analyses, interpretation of the data, or decision to submit results.</p>


Analysis of Editorials on the Response to the H1N1 and COVID-19 Pandemics

<p>Ana Marušić<sup>1</sup></p>
<h4>Importance </h4>
<p>Authorship is the currency of academia and research, but it is not only about recognition. It brings accountability and transparency and can have ethical and legal implications.</p>
<h4>Observations </h4>
<p>Starting from the call for contributors to replace authors by Drummond Rennie and his colleagues almost 30 years ago,<sup>1</sup> the evolution of scientific authorship and contributorship models will be presented. Current challenges to authorship will also be explored, from paper mills to artificial intelligence.</p>
<h4>Conclusions </h4>
<p>Authorship has not been replaced by contributorship and will need a systemic approach to balance scientific credit, career advancement, and ethical responsibility.</p>
<h4>Reference</h4>
<p>1. Rennie D, Yank V, Emanuel L. When authorship fails: a proposal to make contributors accountable. <i>JAMA</i>. 1997;278(7):579-585. doi:10.1001/jama.278.7.579</p>

<p><sup>1</sup>University of Split School of Medicine, Split, Croatia; ana.marusic@mefst.hr.</p>
<h4>Conflict of Interest Disclosures</h4> Ana Marušić is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.


Forward to the Past - Making Contributors Accountable

**Objective** The peer review system used by academic journals
to assess the quality of submitted papers is constantly in need
of new technologies that can help reviewers and increase the
objectivity of the process. The aim of the present study was to
develop a comprehensive list of words related to research
methods and statistics, which could then be used to
automatically study the content and the quantity and types of
methodological and statistical issues identified in peer review
reports.

**Design** The terms for the list were identified from different
glossaries.1,2 The terms were organized into subcategories,
some of which were more related to methodological aspects
(eg, study design, sampling, or procedures) or statistical
aspects (eg, parametric descriptive parameters or data
presentation) of research. The list was refined and enriched
with terms using a computational language model. The final
list and the preregistration can be found at the Open Science
Framework website (https://osf.io/d34b9/). The occurrence
of terms in the PEERE database3
of peer review reports, for
which data on reviewer gender, continental region, review
recommendation, journal impact factor, and subject area (N
= 496,928) were available, was counted. Regression models
were created to determine the frequency of the terms used
based on peer review characteristics.

**Results** The mean length of a peer review was 196 words
(95% CI, 108-330 words). A total of 26.8% of review reports
in the sample contained at least 1 methodological term, while
70.7% contained at least 1 statistical term. It was more likely
that the review contained methodological terms if the
reviewer was female and resided in North America or Oceania
compared with other continents, if the recommendation was
to reject the article or send it for major revisions, if the
journal was from the social sciences and humanities, if the
journal was in the top impact factor quartile, and if the article
was a longer peer review. More words related to statistics
were observed when the reviewer was female, for reviews
recommending any revisions, for journals from health and
medical sciences, in journals in the top impact factor quartile,
and in longer reviews. Mixed regression models indicated
that the scientific area was more strongly associated with
words related to statistics than were reviewer
recommendations.

**Conclusions** The application of this newly developed list
showed that methodological and statistical terms are not a
common topic in peer review and that they are not always
present in review reports from different fields. The list
successfully covered research terms because the most-
identified terms were related to categories relevant for most
types of research. Future analysis should try to understand
the context of the occurrence of words from the list to confirm
the validity of the list of terms.

**References**

1. Stark PB. Glossary of Statistical Terms. Accessed March 23,
2021. https://www.stat.berkeley.edu/~stark/SticiGui/Text/
gloss.htm
2. Framework for Open and Reproducible Research Training.
The Glossary of Terms. Accessed July 12, 2021. https://docs.
google.com/document/d/1oNsHSRmgAsDGUKvq5joqBpv
HeRmDJKldH2kSyJovHhk/edit#heading=h.2l8ssl89e1ey
3. Squazzoni F, Ahrweiler P, Barros T, et al. Unlock ways to
share data on peer review. Nature. 2020;578(7796):512-514.
doi:10.1038/d41586-020-00500-y

**Conflict of Interest Disclosures** Ana Marušić is a member of
the Peer Review Congress Advisory Board by was not involved in the
editorial review or decision for this abstract. No other disclosures
reported.

**Funding/Support** This study was funded by the Croatian Science
Foundation “Professionalism in Health–Decision making in
practice and research” under grant agreement IP-2019-04-4882.
Richard A. Klein received funding from a Replication Studies grant
(401.18.053) from the Netherlands Organization for Scientific
Research, and a Consolidator Grant (IMPROVE) from the European
Research Council (grant 726361). Francisco Grimaldo and Daniel
Garcia-Costa are partially supported by the Spanish Ministry of
Science, Innovation and Universities, the Spanish State Research
Agency, and the European Regional Development Fund under
project RTI2018-095820-B-I00.

**Role of the Funder/Sponsor** The funders had no role in the
design and conduct of the study; collection, management, analysis,
and interpretation of the data; preparation, review, or approval of
the abstract; and decision to submit the abstract for presentation.

![](https://assets.underline.io/uploads/markdown_image/1/image/eba25e8a44f7081b702973e5ddf9a854.png)

Development of a List to Detect Statistical and Methodological Terms in Peer Reviews: A Cross-Sectional Study

**Objective** The evaluation of European Union research grant
proposals consists of 2 consecutive steps: (1) individual expert
assessment and (2) consensus evaluation made by multiple
reviewers. The result is an evaluation summary report, and
previous studies have established this approach as a stable
procedure in the assessment of research grants.1,2 In 2020,
numerical scores were replaced by textual comments in the
individual expert assessment. The objective was to compare
the linguistic characteristics of the comments for Excellence,
Impact and Implementation criteria in evaluation reports of
Marie Skłodowska-Curie Actions’ Innovative Training
Networks (ITN) proposals submitted in 2019 and 2020 to
assess whether the removal of numerical scoring affected the
structure of individuals evaluation report textual comments
and evaluation outcome.<br>
<br>**Design** In this observational study for which data were
collected in fall 2022, all ITN proposals submitted in 2019
and 2020 were considered. Information was collected about
proposal scores and outcome, evaluation panel, and textual
comments of the individual expert evaluations on all
proposals submitted to the call. Linguistic characteristics of
experts’ comments were assessed using the Linguistic Inquiry
and Word Count software, a program that counts words
related to different psychological states and phenomena and
gives a score that is a proportion of the specific category in the
entire text. We used logistic regression to compare differences
between the 2 call years, in which proposal variables
(proposal status, word count for research excellence
weaknesses, word count for implementation strengths, and
negative effect levels for implementation strengths) were
factors and the year of the call was criterion, with the
significance level set at P < .001.<br>
<br>**Results** The number of proposals was similar in 2019 (n =
1554) and 2020 (n = 1503). The proportion of accepted
proposals was slightly higher in 2020 (148 [9.85%]) than in
2019 (128 [8.24%]) (**Table 37**). In logistic regression,
experts’ comments from 2020 differed from 2019 proposals
in 2 linguistic domains. Comments in the Excellence section
related to weaknesses had a greater number of words in the
description of the proposal (**Table 37**). The comments on
strengths in Implementation for proposals from 2020 had
slightly more words and lower negative tone or words related
to negative emotions, such as wrong, suffer, and sad
(**Table 37**). All factors jointly explained around only 4% of
the variance of the criterion.<br>
<br>![](https://assets.underline.io/uploads/markdown_image/1/image/a21802046ca78c985d81a241cf135c85.png)<br>
<br>**Conclusions** It seems that removing numerical scoring in
the evaluation of ITN proposals at the stage of the individual
assessment had little effect on the linguistic characteristics of
the experts’ comments, because all differences were marginal
and we analyzed the whole proposal cohort.
**References**<br>
<br>1. Pina DG, Buljan I, Hren D, Marušić A. A retrospective
analysis of the peer review of more than 75,000 Marie Curie
proposals between 2007 and 2018. eLife. 2021;10:e59338.
doi:10.7554/eLife.59338<br>
<br>2. Buljan I, Pina DG, Marušić A. Ethics issues identified by
applicants and ethics experts in Horizon 2020 grant
proposals. F1000Res. 2021;10:471. doi:10.12688/
f1000research.52965.2<br>
<br>**Conflict of Interest Disclosures** David G. Pina is employed by
the European Research Executive Agency. Ana Marušić occasionally
serves as an ethics expert for the European Research Executive
Agency and is an advisory board member of the Peer Review
Congress but was not involved in the review or decision of this
abstract. Ivan Buljan and Antonia Mijatović declare no competing
interests.<br>
<br>**Funding/Support** This study was funded by the Croatian Science
Foundation “Professionalism in Health – Decision making in
practice and research” (ProDeM) under grant agreement IP-2019-
04-4882.<br>
<br>**Role of the Funder/Sponsor** The funders had no role in design
and conduct of the study; collection, management, analysis, and
interpretation of the data; preparation, review, or approval of the
abstract; and decision to submit the abstract for presentation.<br>
<br>**Disclaimer** All views expressed in this abstract are strictly those
of the authors and may in no circumstances be regarded as an
official position of the European Research Executive Agency or the
European Commission.<br>
<br>![](https://assets.underline.io/uploads/markdown_image/1/image/dba03205861e88086430823162f000b1.png)

Comparison of Evaluations of Grant Proposals With and Without Numerical Scoring Submitted to Marie Sklodowska-Curie Actions Innovative Training Networks

Lightning Session 1

Peer Review Of Grant Proposals

Peer review is often criticized for being flawed, subjective and biased, but research into peer review is still lacking studies on peer review reports. In our study, in which text-analysis software was used to determine the linguistic characteristics of 472,449 peer review reports we compared a range of characteristics (including analytical tone, authenticity, clout, three measures of sentiment, and morality) as interaction of reviewer recommendation, area of research, type of peer review and reviewer gender. Our main conclusion was that the reviewer recommendation had the biggest impact on the linguistic characteristics of reports, and that area of research, type of peer review and reviewer gender had little or no impact. The lack of influence of research area, type of review or reviewer gender on the linguistic characteristics is a sign of the robustness of peer review.

Large-scale language analysis of peer review reports

A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports 
Darko Hren1, David G. Pina2, Cristopher Norman3, Ana Marušić4

1University of Split Faculty of Humanities and Social Sciences, Split, Croatia
2Research Executive Agency, European Commission, Brussels, Belgium
3LIMSI, CNRS, Université Paris Saclay, Paris, France
4University of Split School of Medicine, Split, Croatia

Objectives:
To identify themes in the content of reviewers' evaluation reports about proposals submitted to the FP7 Marie Curie Initial Training Networks calls and to explore the content areas in reviewers' comments that differentiate successful from unsuccessful proposals.

Methods:
We qualitatively analyzed the contents of reviewers' comments to obtain the most common themes within evaluation reports. Given a large amount of textual data, we used a two-step approach.
The first step was automated to help reduce the data to the manageable size. We used K-Means clustering to identify the clusters of statements within sections of the evaluation reports. Clusters were visualized in 2-dimensional space using TensorBoard Embedding Projector (https://projector.tensorflow.org/). Principal components analysis (PCA) was used for the visualization of clusters and cosine distances between clusters as a reference to cluster distances.
In the second step, the clusters were further grouped into themes using empirical estimates from PCA in step 1 and qualitative analysis based on cluster terms and content. To identify clusters that could potentially be grouped into themes, we used cosine distances lower than 1 as an arbitrary indicator of cluster proximity. Next, we inspected the content of the obtained clusters (terms and example statements) to verify whether and how groups of clusters could be interpreted as broader themes. Inspection of content had precedence over empirical data, i.e. if there was no meaningful overlap in cluster terms and content, even clusters that had cosine distance below 1 were not combined.
Finally, we registered the frequencies of obtained themes within each evaluation report (i.e. a number of sentences containing terms belonging to a specific theme) to inspect potential differences between proposals with different decision status.

Results:
We analyzed the content of reviews of 3666 grant proposals submitted to Marie Curie Initial Training Networks from the FP7 PEOPLE programme in 2008 (n=886), 2010 (n=857), 2011 (n=909), and 2012 (n=1013).
Each review had four sections of criteria (C1-Science & Technology Quality, C2-Training, C3-Implementation, and C4-Impact), and each section was divided into two parts (Strengths and Weaknesses). Therefore, 12 separate K-Means cluster analyses were performed in the first step, and we identified between 9 and 12 clusters explaining between 32.7 and 51.6% of the variance. 
In the second step, we identified between 29 themes within different sections of the evaluation reports (Figure 1A-D). Evaluation reports of proposals from the main and reserve lists included more themes related to strengths and fewer themes related to weaknesses than those of rejected proposals. Themes related to strengths were generally more frequent than those related to weaknesses. Most prominent themes related to strengths were: Research programme and objectives and Multidisciplinary context (C1), Consistency of the research and training programme and Career development (C2), Capacities of research and training (C3), and Outreach activities and Career prospects (C4). Most prominent themes related to weaknesses were: Research programme and objectives and Research methodology (C1), Engagement of ESRs and Training characteristics (C2), Capacities of research and training (C3), and Exploitation and dissemination (C4).

Conclusions:
We used a novel approach which combined machine learning and qualitative inquiry to identify the most common themes of the evaluation reports of proposals submitted to the FP7 Marie Curie Initial Training Networks calls. The themes reflected general evaluation criteria applied by the European Commission, but also specific themes that distinguished successful from unsuccessful proposals thus demonstrating the validity of the selection process.
The obtained results may be used by researchers preparing to apply for grants.



A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports

The Horizon 2020 Marie Skłodowska-Curie Actions (MSCA) Individual Fellowships (IF) funding programme is one of the largest and most competitive EU research programme (around 15% success rate). It would be expected that the applications of funded proposals have detected ethical issues well. The aim of this study was to describe the ethics evaluation process in the MSCA IF funding scheme . We first assessed the ethical issues identified by the applicants when preparing their proposals and compared these to the ethics issues identified by ethics experts during the ethics evaluation performed for those proposals retained for funding. In total, we analyzed 3074 proposals from three MSCA IF calls ( 2016, 2017 and 2018). Applicants identified almost two times fewer ethical issues compared to the experts performing the ethics review. Most of the ethics issues were identified in proposals from the Life Sciences and Social Sciences and Humanities research panels, which represented more than 60% of all proposals. Out of all 3074 proposals which were main listed and accepted for funding, 2225 (72.4%) received conditional ethics clearance, pending satisfaction of requirements defined by the ethics experts. The most prevalent requirements were within the Protection of personal data, Environment, Health and Safety, Humans, and non-EU countries categories. Our results emphasize that successful applicants to the MSCA IF programme, consisting mostly of postdoctoral researchers, are not adequately trained or knowledgeable about the ethics issues related to their proposed research, and that problem persists over time. Training in research ethics and proper guidance by the host institutions are paramount for narrowing the gap between the applicants’ ethics self-assessments and the results of the ethics review procedure. 


Ethics assessment of H2020 Marie Skłodowska-CUrie individual fellowships

We explore which characteristics of a reviewer affect disagreement with the other reviewers in research proposal evaluation, studying 52,488 proposals submitted to the Horizon 2020 Marie Skłodowska-Curie Actions (MSCA) and 1,939 proposals to the European Cooperation in Science and Technology Actions (COST). We find that experience in previous calls significantly reduces disagreement, and in MSCA - IF, an inverted U relationship between the number of proposals evaluated in a given call and disagreement, with a remarkable decrease of disagreement above thirteen proposals evaluated. The results suggest that reviewing experience improve reliability, curbing unwarranted disagreement by gradually fine-tuning reviewers’ evaluation.

Does reviewers' traits affect disagreement in proposal evaluation?

The organization of peer review processes in grant proposals evaluation is rarely studied. We analysed 77 thousand proposals from the Marie Curie programme, submitted over a period of 12 years, with the aim to get insights on the reliability of the process. Mean scores, Average Deviation (AD) indices, used as a measure of inter-rater agreement, and interrupted time series analysis were calculated to study changes in reviewers’ scores and level of agreement. Our results indicate that scores given by and agreement among reviewers may be more dependent on the type of grants or scientific panels than on the way the evaluation process is organised, with important economic and environmental implications for policy-making in research grants evaluation.

Ana Marusic

11

14

SHORT BIO

OTHER AFFILIATIONS

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES