Problem
One of the mechanisms proposed to improve reporting of research results is the installation of reporting guidelines in scholarly journals. These guidelines have been issued in a variety of research contexts and with a diverse set of aims, including improving identification and validation of biomedical research materials. Considering these journal initiatives from a science policy perspective, they constitute rule-based or ‘regulatory’ policy, attempting to change researchers’ behaviour through regulation. Rules require policing, which is often expensive, and rules tend to fail if they lack community support. 
Approach
To analyse the effectiveness of journal reporting guidelines we assessed such guidelines in two contexts within biomedical science. We first analysed reporting guidelines in the context of misidentified cell lines, which constitutes a long-known and large-scale issue in biomedical research. Secondly, we assessed guidelines in the context of antibody validation.
Using bibliometric techniques, we set out to study how journal guidelines affect the number of journal articles reporting on misidentified cell lines and the provision of sufficient antibody validation information. For both cases, we compared journals with and without relevant reporting guidelines, as well as reporting before and after the introduction of guidelines.
Results
In the case of cell lines, we observed a decrease in the percentage of articles that use misidentified cell lines in journals with guidelines. However, this decrease already starts before the implementation of the guidelines and does accelerate when the guidelines are implemented. Similarly, we also find a decrease in the number of articles on misidentified cells in journals not introducing guidelines. In addition, the percentage of articles that use misidentified cell lines is higher in journals with guidelines than in those without.
Conclusions
Our results suggest that journal reporting guidelines are an only moderately effective mechanism to steer individual behaviour. In addition, the context and precise format in which they are implemented highly influences the likelihood of guidelines actually leading to improved reporting. In our studies, the implementation of guidelines seems to have resulted in symbolic or minimal compliance. The observed trend in reporting on misidentified cells, suggests that journal guidelines only formalise reporting behaviour that has already been adopted by the community.

Biomedical journal reporting guidelines: what they can and cannot do.

Journal peer review plays a central role in the self-regulatory capability of science. However, there are quite some differences in the expectations of peer review. For example, some argue that ‘safeguarding the scientific integrity of published articles’ is one of the core responsibilities. Thus, some expect peer review to assure both the quality and integrity of research, where others argue that the system was never designed to, nor suited to do so [1]. 
Due to the many expectations, the peer review process has developed in such a way that it currently comes in many shapes and flavours [1] [2]. Scholars studying scientific publishing have also expressed worries about the opaqueness of editorial policies, hindering further research on the different ways of organising the peer review process [3]. In addition, some key players in scientific publishing have expressed their willingness to address these issues, but they encounter a lack of convincing information to warrant editorial policy changes [3]. To address these concerns, we are establishing a Platform for Responsible Editorial Policies (PREP).
First and foremost, PREP will provide insight into specific journals’ peer review procedures in the form of a database based on a dozen of questions. Among others, these will include the level of anonymity of authors and reviewers, whether digital tools, such as plagiarism scanners, are used, and the timing of peer review within the research process. Traditionally, peer review takes place between submission and publication of a manuscript. However, recently, two new forms of peer review timing have emerged; post-publication review, in the form of preprint services, and pre-submission, in the form of registered reports. The answers of 361 journals to the dozen of questions on their peer review procedures will form the start of the Open Access Editorial Procedures Database [4].
PREP invites other editors to provide the relevant information for their journals. Based on the answers to the dozen questions, PREP will include tailored suggestions for potential improvements to peer review procedures, including issues particularly relevant to the journal’s research area (such as the review of statistics) [3].
PREP will also suggest possible improvements on how to improve transparency of the peer review procedures and editorial policies such as policies on corrections and retractions, in line with the transparency declaration (https://www.ru.nl/science/isis/research/transparency-declaration/). To further help journals to become transparent about their editorial policies, the platform will generate textual material that can be used on a journal’s webpage to foster transparency about their peer review procedures based on the responses to the dozen questions about current review procedures.
In addition, PREP will provide web-friendly information about different review procedures (e.g. single-, and double-blind procedures, open review, or registered reports), including the rationale for their development and the evidence-base for their effectiveness in the literature. This information including infographics will be freely downloadable and can thus be used for information, training, and educational purposes. 
Finally, PREP will be established in line with the Leiden manifesto in cooperation with Centre for Science and Technology Studies (CWTS), and will advise on responsible research evaluation [5]. 
In conclusion, PREP envisions contributing to the integrity and trustworthiness of science via increased transparency. This builds on the general trend in science and the publication system including forces towards open science [2]. In addition, PREP’s database enables further research on peer review procedures, and in time could provide better and more detailed recommendations to journal editors and academic publishers.


Willem Halffman

2

2

SHORT BIO

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES