Spain

PEERE

Peer Review Of Grant Proposals

Following the success of the 1st PEERE International Conference on Peer Review, the PEERE COST Action follow-up community organised the second edition of the PEERE International Conference on Peer Review  and was celebrated virtually on 29 September – 1 October 2020.

The conference welcomed any contribution (e.g., original research, position papers, literature reviews) that used any method (e.g., quantitative, experimental or qualitative) to investigate peer review in a variety of scientific domains (e.g., scholarly journals, funding agencies, research assessment). Interdisciplinary research is especially welcome.

The program is published here: http://tiny.cc/peere2020 


The second edition of the PEERE International Conference on Peer Review aims to provide a forum for scholars, practitioners and science stakeholders to share research, models, tools and experience on peer review in different fields, e.g., medicine, computer science, social sciences and humanities. It aims to stimulate the use of evidence-based research in the design and implementation of peer review in a variety of fields and encourage more systematic research.

technical paper

Problem
One of the mechanisms proposed to improve reporting of research results is the installation of reporting guidelines in scholarly journals. These guidelines have been issued in a variety of research contexts and with a diverse set of aims, including improving identification and validation of biomedical research materials. Considering these journal initiatives from a science policy perspective, they constitute rule-based or ‘regulatory’ policy, attempting to change researchers’ behaviour through regulation. Rules require policing, which is often expensive, and rules tend to fail if they lack community support. 
Approach
To analyse the effectiveness of journal reporting guidelines we assessed such guidelines in two contexts within biomedical science. We first analysed reporting guidelines in the context of misidentified cell lines, which constitutes a long-known and large-scale issue in biomedical research. Secondly, we assessed guidelines in the context of antibody validation.
Using bibliometric techniques, we set out to study how journal guidelines affect the number of journal articles reporting on misidentified cell lines and the provision of sufficient antibody validation information. For both cases, we compared journals with and without relevant reporting guidelines, as well as reporting before and after the introduction of guidelines.
Results
In the case of cell lines, we observed a decrease in the percentage of articles that use misidentified cell lines in journals with guidelines. However, this decrease already starts before the implementation of the guidelines and does accelerate when the guidelines are implemented. Similarly, we also find a decrease in the number of articles on misidentified cells in journals not introducing guidelines. In addition, the percentage of articles that use misidentified cell lines is higher in journals with guidelines than in those without.
Conclusions
Our results suggest that journal reporting guidelines are an only moderately effective mechanism to steer individual behaviour. In addition, the context and precise format in which they are implemented highly influences the likelihood of guidelines actually leading to improved reporting. In our studies, the implementation of guidelines seems to have resulted in symbolic or minimal compliance. The observed trend in reporting on misidentified cells, suggests that journal guidelines only formalise reporting behaviour that has already been adopted by the community.

Biomedical journal reporting guidelines: what they can and cannot do.

Journal peer review plays a central role in the self-regulatory capability of science. However, there are quite some differences in the expectations of peer review. For example, some argue that ‘safeguarding the scientific integrity of published articles’ is one of the core responsibilities. Thus, some expect peer review to assure both the quality and integrity of research, where others argue that the system was never designed to, nor suited to do so [1]. 
Due to the many expectations, the peer review process has developed in such a way that it currently comes in many shapes and flavours [1] [2]. Scholars studying scientific publishing have also expressed worries about the opaqueness of editorial policies, hindering further research on the different ways of organising the peer review process [3]. In addition, some key players in scientific publishing have expressed their willingness to address these issues, but they encounter a lack of convincing information to warrant editorial policy changes [3]. To address these concerns, we are establishing a Platform for Responsible Editorial Policies (PREP).
First and foremost, PREP will provide insight into specific journals’ peer review procedures in the form of a database based on a dozen of questions. Among others, these will include the level of anonymity of authors and reviewers, whether digital tools, such as plagiarism scanners, are used, and the timing of peer review within the research process. Traditionally, peer review takes place between submission and publication of a manuscript. However, recently, two new forms of peer review timing have emerged; post-publication review, in the form of preprint services, and pre-submission, in the form of registered reports. The answers of 361 journals to the dozen of questions on their peer review procedures will form the start of the Open Access Editorial Procedures Database [4].
PREP invites other editors to provide the relevant information for their journals. Based on the answers to the dozen questions, PREP will include tailored suggestions for potential improvements to peer review procedures, including issues particularly relevant to the journal’s research area (such as the review of statistics) [3].
PREP will also suggest possible improvements on how to improve transparency of the peer review procedures and editorial policies such as policies on corrections and retractions, in line with the transparency declaration (https://www.ru.nl/science/isis/research/transparency-declaration/). To further help journals to become transparent about their editorial policies, the platform will generate textual material that can be used on a journal’s webpage to foster transparency about their peer review procedures based on the responses to the dozen questions about current review procedures.
In addition, PREP will provide web-friendly information about different review procedures (e.g. single-, and double-blind procedures, open review, or registered reports), including the rationale for their development and the evidence-base for their effectiveness in the literature. This information including infographics will be freely downloadable and can thus be used for information, training, and educational purposes. 
Finally, PREP will be established in line with the Leiden manifesto in cooperation with Centre for Science and Technology Studies (CWTS), and will advise on responsible research evaluation [5]. 
In conclusion, PREP envisions contributing to the integrity and trustworthiness of science via increased transparency. This builds on the general trend in science and the publication system including forces towards open science [2]. In addition, PREP’s database enables further research on peer review procedures, and in time could provide better and more detailed recommendations to journal editors and academic publishers.


Platform for Responsible Editorial Policies

In the absence of gold standard for evaluating quality of peer review, considerable attention has been focused on studying reviewer agreement via inter-rater reliability (IRR) which can be thought of as the correlation between scores of different reviewers given to the same grant proposal. Noting that it is not uncommon for IRR in grant peer review studies to be estimated from some range-restricted subset of submissions, we use statistical methods and data analysis of real peer review data to illustrate behavior of such local IRR estimates when only fractions of top-quality proposal submissions are considered. We demonstrate that local IRR estimates are smaller than those obtained from all submissions and that zero local IRR estimates are quite plausible. We note that, from a measurement perspective, when reviewers are asked to differentiate among grant proposals across the whole range of submissions, only IRR measures that correspond to the complete range of submissions are warranted. We recommend against using local IRR estimates in those situations. Moreover, if review scores are intended to be used for differentiating among top proposals, we recommend peer review administrators and researchers to align review procedures with their intended measurement.

Local versus global inter-rater reliability for evaluating the internal validity of grant peer review: Considerations of measurement

In peer review, reviewers often have formal and informal occasions to interact with other reviewers of the same grant proposal or submission. The former happens, for example, when grant proposals are evaluated by sitting review panels who are tasked with collectively deciding the fate of a particular proposal. Despite the scholarly consensus that interactions (and thus social influence) can alter the outcome of collective decision making, to date social influence has rarely been a theoretical ingredient in simulation models of peer review. In our contribution we use a simple agent-based model of peer review to show that social influence processes can indeed change the dynamics and outcome of peer review. With our simulation experiment we showcase the incorporation of social influence processes in peer review models as a promising direction for future research.

Does social influence matter in peer review?

This paper describes the pilot project to introduce a new Open Data policy to three IOP Publishing journals as part of IOP Publishing’s commitment to increasing transparency and support for open science. For the largest and longest established of these journals an investigation was undertaken to understand the impact of this change on authors and the editorial staff by examining changes in the revised submission processing time and how often manuscripts were returned to the author. The paper identifies an overall increase in revised manuscript submission processing at the start of the trial that is influenced by the method used by authors to share data. The rate of un-submitting manuscripts was also found to be increased showing an impact on authors. The paper also shows that the processing time quickly returned to normal levels once mitigating actions were taken. The increase in the rate at which manuscripts were returned to the author was also reduced but not completely removed.

Introducing a Data Accessibility Policy for journals at IOP Publishing: Measuring the impact on authors and editorial teams

Peer review is a cornerstone of the science system. Previous studies have found that peer review in the social sciences primarily prompted changes in the theoretical framing of papers and contextualisation of results, while the methodological sections were largely untouched, with associated patterns of change in reference lists. We ask whether this focus on theory over methods also occurs in other fields. To assess this, we matched preprints from Dimensions with published articles from Web of Science, PLOS and Scopus to identify 6,002 preprint-publication pairs across all fields. We then compared their reference lists and calculated the proportion of retained, added and removed references, and also identified in which sections of publications the references were cited. We found that the medical sciences added and removed significantly more references between preprint and publication than did the natural sciences, which had more bibliographically stable manuscripts. By proportion, the introduction and discussion sections underwent the most bibliographic change in both fields, however by ratio of added to unchanged references, the methodology and results section had greater change. Our results suggest there are important differences between fields in how reference lists change between preprint and publication, and that the focus of theory over methodology in the social sciences may not extend to the natural sciences or medical sciences.

Changing the shoulders I am standing on: Describing the changes that occurred in publications' reference lists after peer review

Preprints – generally defined as a publicly available version of a research manuscript prior to formal assessment and publication via a journal-mediated peer review process – are becoming a common feature in some research communities. Accelerated sharing of research findings, creating a citable research output to show evidence of research productivity, creating opportunities for community feedback and collaboration, and increased attention and citation are some of the ways preprints can benefit researchers (References 1-4). Preprints also create opportunities for publishers seeking to address pain points associated with lengthy peer review processes. To this end, since October 2018, Research Square has collaborated with Springer Nature to offer In Review, a preprint service that integrates preprint posting with ongoing journal peer review and provides transparency into the peer review process. In Review helps Nature authors rapidly share their research findings while also providing journal-mediated quality control checks on the posted preprint and transparency into the peer-review process. 

1) https://jamanetwork.com/journals/jama/fullarticle/2670247
2) https://arxiv.org/abs/1805.05238
3) https://arxiv.org/abs/astro-ph/0411275
4) https://elifesciences.org/articles/52646

In Review on Research Square: a journal-integrated preprint system for Springer Nature

Sentiment analysis methods have been developed to automatically extract opinions from texts, e.g. online reviews. Here we explore the idea of adopting sentiment analysis to extract grant reviewers' opinions. This paper examines the relationship between the sentiments of grant reviews and reviewed proposals’ funding decisions made by agencies. We define peer reviews’ prediction as to what degree reviews’ positiveness/negativeness, via sentiment extraction, can indicate proposals’ chance of success in receiving funding. Building on a corpus of peer review texts and related documents from an Irish science funding agency, we conduct sentiment extraction by manual and two simple automated coding (TextBlob and VADER). A comparison between the three results has been made. We find the manual and automated coding results to be overall consistent, but manual coding is still the most accurate and reliable for peer review studies. Furthermore, the three coding results show strong prediction of proposals’ funding decisions, which indicates that reviewers' opinions are largely adopted by agencies.

Peer Reviews' Prediction in Proposals' Funding Success: A Sentiment Analysis of Grant Reviews

Peer review is often criticized for being flawed, subjective and biased, but research into peer review is still lacking studies on peer review reports. In our study, in which text-analysis software was used to determine the linguistic characteristics of 472,449 peer review reports we compared a range of characteristics (including analytical tone, authenticity, clout, three measures of sentiment, and morality) as interaction of reviewer recommendation, area of research, type of peer review and reviewer gender. Our main conclusion was that the reviewer recommendation had the biggest impact on the linguistic characteristics of reports, and that area of research, type of peer review and reviewer gender had little or no impact. The lack of influence of research area, type of review or reviewer gender on the linguistic characteristics is a sign of the robustness of peer review.

Peer Review Of Grant Proposals

Downloads

Next from PEERE

Biomedical journal reporting guidelines: what they can and cannot do.

Similar lecture

Policy Lessons From Social Simulation

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Peer Review Of Grant Proposals

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from PEERE

Biomedical journal reporting guidelines: what they can and cannot do.

Similar lecture

Policy Lessons From Social Simulation

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads