United States

&lt;p&gt;Tais Freire Galvão,&lt;sup&gt;1,2&lt;/sup&gt; Everton Nunes da Silva,&lt;sup&gt;2&lt;/sup&gt; Jorge Otávio Maia Barreto,&lt;sup&gt;3&lt;/sup&gt; Marcus Tolentino Silva&lt;sup&gt;4&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective&lt;/h4&gt;
&lt;p&gt; Detailed descriptions of journals’ workflows exist,&lt;sup&gt;1&lt;/sup&gt; but empirical studies assessing their effects are scarce. In 2024, a new workflow was implemented at &lt;i&gt;Epidemiologia e Serviços de Saúde: revista do SUS&lt;/i&gt; (&lt;i&gt;RESS&lt;/i&gt;) aimed at getting timely and quality reviews. This study assessed the effect of a new workflow on the journal’s time to decision-making.&lt;/p&gt;
&lt;h4&gt;Design&lt;/h4&gt;
&lt;p&gt; This before and after study using historical controls assessed submissions to &lt;i&gt;RESS&lt;/i&gt; from June to December 2024 (intervention) and June to December 2023 (control), followed up until January 2025. The intervention started in June 2024, based on inclusion of a peer review administrator in charge of reviewers’ assignments and adding new and trained associate editors&lt;sup&gt;2&lt;/sup&gt;; previous evaluation by the editor in chief (scope) and scientific editor (methodological quality); and checklists to standardize reviews.&lt;sup&gt;3&lt;/sup&gt; The comparator was the previous workflow, centralized on the associate editor’s assignment of reviewers, and final decisions made by the editor in chief and a member of the editorial committee. Outcomes were processing time from submission to decision, number of reviewers invited, and reviews received. We summarized means and SDs or medians and IQRs. Normality was tested using the Shapiro-Wilk test and square root or logarithmic transformations preceded t-tests, with the Levene test used to assess variances’ homogeneity. In unsuitable t-tests, the Mann-Whitney test was applied. Residual normality and heteroscedasticity were examined by Breusch-Pagan test. We adjusted by the length of the paper (full or short) to minimize potential confounding.&lt;/p&gt;
&lt;h4&gt;Results&lt;/h4&gt;
&lt;p&gt; Out of 928 submissions in 2024, 218 proceeded to peer review (140 ongoing until cohort closure) and 78 had a decision (36 accepted, 42 rejected). In 2023, from 970 submissions, 77 were peer reviewed and resulted in 28 acceptances and 49 rejections (&lt;i&gt;P&lt;/i&gt; = .22). Mean (SD) time in days to decision was significantly shorter in the intervention (62.8 [40.5]) than in the controls (130.1 [64.5]; &lt;i&gt;P&lt;/i&gt; &amp;lt; .001), which was also faster in the intervention according to acceptance (92.8 [39.8] vs 133.5 [61.6]; &lt;i&gt;P&lt;/i&gt; = .002) or rejection (37.1 [16.0] vs 128.1 [66.7]; &lt;i&gt;P&lt;/i&gt; &amp;lt; .001). The mean (SD) number of invited reviewers per manuscript was higher in the intervention (10.1 [6.3]) than in the controls (7.8 [5.5]; &lt;i&gt;P&lt;/i&gt; = .02), without significant difference according to decision when compared with the historical controls (approved, 10.0 [7.0] vs 6.3 [2.8]; &lt;i&gt;P&lt;/i&gt; = .07; rejected, 10.3 [5.8] vs 8.6 [6.5]; adjusted &lt;i&gt;P&lt;/i&gt; = .11). The median (IQR) number of reviews received was also higher (4 [4-4] vs 2 [2-3]; &lt;i&gt;P&lt;/i&gt; &amp;lt; .001), and was similarly higher when comparing decisions across groups (approved, 4 [4-4] vs 2 [2-3]; &lt;i&gt;P&lt;/i&gt; &amp;lt; .001; rejected, 4 [4-4] vs 2 [2-3]; &lt;i&gt;P&lt;/i&gt; &amp;lt; .001) (&lt;b&gt;Table 25-0909&lt;/b&gt;).&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/c5b3c1bf721bf5ccfd8c212b70f07e17.png)

&lt;h4&gt;Conclusions&lt;/h4&gt;
&lt;p&gt; The new workflow reduced the duration of editorial processing and allowed a higher number of peer reviews to inform decision-making. The outcomes were not defined before the intervention was planned; the length of follow-up, new editors, and submission system were confounders that may have influenced the findings.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Mark H, Ragon T, Funning G, et al. Editorial workflow of a community-led, all-volunteer scientific journal: lessons from the launch of Seismica. &lt;i&gt;Seismica&lt;/i&gt;. 2023;2(2):1091. doi:10.26443/seismica.v2i2.1091&lt;/p&gt;
&lt;p&gt;&lt;span lang=&quot;pt-PT&quot;&gt;2. Galvão TF, Silva EN, Araújo WN, Barreto JOM. Editorial improvement in &lt;/span&gt;&lt;i&gt;Epidemiologia e Serviços de Saúde&lt;/i&gt;&lt;span lang=&quot;pt-PT&quot;&gt; in 2024. &lt;/span&gt;&lt;i&gt;Epidemiol Serv Saude&lt;/i&gt;&lt;span lang=&quot;pt-PT&quot;&gt;. 2024;33:e20241002. doi:&lt;/span&gt;&lt;a href=&quot;https://doi.org/10.1590/S2237-96222024v33e20241002.en&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot; lang=&quot;pt-PT&quot;&gt;10.1590/S2237-96222024v33e20241002.en&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;span lang=&quot;pt-PT&quot;&gt;3. Silva MT, Galv&lt;/span&gt;&lt;span lang=&quot;ar-SA&quot;&gt;ã&lt;/span&gt;&lt;span lang=&quot;pt-PT&quot;&gt;o TF. Systematization of peer review in &lt;/span&gt;&lt;i&gt;Epidemiologia e Serviços de Saúde&lt;/i&gt;&lt;span lang=&quot;pt-PT&quot;&gt;. &lt;/span&gt;&lt;i&gt;Epidemiol Serv Saude&lt;/i&gt;&lt;span lang=&quot;pt-PT&quot;&gt;. 2024;33:e20241001. doi:&lt;/span&gt;&lt;a href=&quot;https://doi.org/10.1590/S2237-96222024v33e20241001.en&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot; lang=&quot;pt-PT&quot;&gt;10.1590/S2237-96222024v33e20241001.en&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p class=&quot;abstract_conflict&quot; lang=&quot;fr-FR&quot;&gt;&lt;sup&gt;1&lt;/sup&gt;Faculdade de Ciências Farmacêuticas&lt;span lang=&quot;pt-PT&quot;&gt;, Universidade Estadual de Campinas, Campinas, Brazil, taisgalvao@gmail.com; &lt;/span&gt;&lt;sup&gt;2&lt;/sup&gt;&lt;span lang=&quot;pt-PT&quot;&gt;Secretaria de Vigilância em Saúde e Ambiente, Ministério da Saúde&lt;/span&gt;, Brasília, Brazil; &lt;sup&gt;3&lt;/sup&gt;&lt;span lang=&quot;pt-PT&quot;&gt;Faculdade d&lt;/span&gt;e Ciências e Tecnologias em Saúde, Universidade de Brasília, Brasília, Brazil; &lt;sup&gt;4&lt;/sup&gt;&lt;span lang=&quot;pt-PT&quot;&gt;Gerência Regional de Brasília, Fundação Oswaldo Cruz, &lt;/span&gt;Brasília, Brazil; &lt;sup&gt;5&lt;/sup&gt;&lt;span lang=&quot;pt-PT&quot;&gt;Faculdade d&lt;/span&gt;e Ciências da Saúde, Universidade de Brasília, Brasília, Brazil.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures &lt;/h4&gt;
&lt;p&gt;Tais Freire Galv&lt;span lang=&quot;ar-SA&quot;&gt;ã&lt;/span&gt;o, Everton Nunes da Silva, and Jorge Otávio Maia Barreto are editors of &lt;i&gt;Epidemiologia e Serviços de Saúde: revista do SUS&lt;/i&gt;. Marcus Tolentino Silva has no conflicts of interest to declare. Tais Freire Galv&lt;span lang=&quot;ar-SA&quot;&gt;ã&lt;/span&gt;o is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.&lt;/p&gt;


The 10th Peer Review Congress

Effects of Peer Review and Editorial Workflows in Decision-Making at a Diamond Open Access Journal

peer review process and models

editorial and peer review process

peer review

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Tais Freire Galvão,<sup>1,2</sup> Everton Nunes da Silva,<sup>2</sup> Jorge Otávio Maia Barreto,<sup>3</sup> Marcus Tolentino Silva<sup>4</sup></p>
<h4>Objective</h4>
<p> Detailed descriptions of journals’ workflows exist,<sup>1</sup> but empirical studies assessing their effects are scarce. In 2024, a new workflow was implemented at <i>Epidemiologia e Serviços de Saúde: revista do SUS</i> (<i>RESS</i>) aimed at getting timely and quality reviews. This study assessed the effect of a new workflow on the journal’s time to decision-making.</p>
<h4>Design</h4>
<p> This before and after study using historical controls assessed submissions to <i>RESS</i> from June to December 2024 (intervention) and June to December 2023 (control), followed up until January 2025. The intervention started in June 2024, based on inclusion of a peer review administrator in charge of reviewers’ assignments and adding new and trained associate editors<sup>2</sup>; previous evaluation by the editor in chief (scope) and scientific editor (methodological quality); and checklists to standardize reviews.<sup>3</sup> The comparator was the previous workflow, centralized on the associate editor’s assignment of reviewers, and final decisions made by the editor in chief and a member of the editorial committee. Outcomes were processing time from submission to decision, number of reviewers invited, and reviews received. We summarized means and SDs or medians and IQRs. Normality was tested using the Shapiro-Wilk test and square root or logarithmic transformations preceded t-tests, with the Levene test used to assess variances’ homogeneity. In unsuitable t-tests, the Mann-Whitney test was applied. Residual normality and heteroscedasticity were examined by Breusch-Pagan test. We adjusted by the length of the paper (full or short) to minimize potential confounding.</p>
<h4>Results</h4>
<p> Out of 928 submissions in 2024, 218 proceeded to peer review (140 ongoing until cohort closure) and 78 had a decision (36 accepted, 42 rejected). In 2023, from 970 submissions, 77 were peer reviewed and resulted in 28 acceptances and 49 rejections (<i>P</i> = .22). Mean (SD) time in days to decision was significantly shorter in the intervention (62.8 [40.5]) than in the controls (130.1 [64.5]; <i>P</i> &lt; .001), which was also faster in the intervention according to acceptance (92.8 [39.8] vs 133.5 [61.6]; <i>P</i> = .002) or rejection (37.1 [16.0] vs 128.1 [66.7]; <i>P</i> &lt; .001). The mean (SD) number of invited reviewers per manuscript was higher in the intervention (10.1 [6.3]) than in the controls (7.8 [5.5]; <i>P</i> = .02), without significant difference according to decision when compared with the historical controls (approved, 10.0 [7.0] vs 6.3 [2.8]; <i>P</i> = .07; rejected, 10.3 [5.8] vs 8.6 [6.5]; adjusted <i>P</i> = .11). The median (IQR) number of reviews received was also higher (4 [4-4] vs 2 [2-3]; <i>P</i> &lt; .001), and was similarly higher when comparing decisions across groups (approved, 4 [4-4] vs 2 [2-3]; <i>P</i> &lt; .001; rejected, 4 [4-4] vs 2 [2-3]; <i>P</i> &lt; .001) (<b>Table 25-0909</b>).</p>

![](https://assets.underline.io/markdown_image/1/image/c5b3c1bf721bf5ccfd8c212b70f07e17.png)

<h4>Conclusions</h4>
<p> The new workflow reduced the duration of editorial processing and allowed a higher number of peer reviews to inform decision-making. The outcomes were not defined before the intervention was planned; the length of follow-up, new editors, and submission system were confounders that may have influenced the findings.</p>
<h4>References</h4>
<p>1. Mark H, Ragon T, Funning G, et al. Editorial workflow of a community-led, all-volunteer scientific journal: lessons from the launch of Seismica. <i>Seismica</i>. 2023;2(2):1091. doi:10.26443/seismica.v2i2.1091</p>
<p><span lang="pt-PT">2. Galvão TF, Silva EN, Araújo WN, Barreto JOM. Editorial improvement in </span><i>Epidemiologia e Serviços de Saúde</i><span lang="pt-PT"> in 2024. </span><i>Epidemiol Serv Saude</i><span lang="pt-PT">. 2024;33:e20241002. doi:</span><a href="https://doi.org/10.1590/S2237-96222024v33e20241002.en"><span class="Hyperlink CharOverride-6" lang="pt-PT">10.1590/S2237-96222024v33e20241002.en</span></a></p>
<p><span lang="pt-PT">3. Silva MT, Galv</span><span lang="ar-SA">ã</span><span lang="pt-PT">o TF. Systematization of peer review in </span><i>Epidemiologia e Serviços de Saúde</i><span lang="pt-PT">. </span><i>Epidemiol Serv Saude</i><span lang="pt-PT">. 2024;33:e20241001. doi:</span><a href="https://doi.org/10.1590/S2237-96222024v33e20241001.en"><span class="Hyperlink CharOverride-6" lang="pt-PT">10.1590/S2237-96222024v33e20241001.en</span></a></p>
<p class="abstract_conflict" lang="fr-FR"><sup>1</sup>Faculdade de Ciências Farmacêuticas<span lang="pt-PT">, Universidade Estadual de Campinas, Campinas, Brazil, taisgalvao@gmail.com; </span><sup>2</sup><span lang="pt-PT">Secretaria de Vigilância em Saúde e Ambiente, Ministério da Saúde</span>, Brasília, Brazil; <sup>3</sup><span lang="pt-PT">Faculdade d</span>e Ciências e Tecnologias em Saúde, Universidade de Brasília, Brasília, Brazil; <sup>4</sup><span lang="pt-PT">Gerência Regional de Brasília, Fundação Oswaldo Cruz, </span>Brasília, Brazil; <sup>5</sup><span lang="pt-PT">Faculdade d</span>e Ciências da Saúde, Universidade de Brasília, Brasília, Brazil.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Tais Freire Galv<span lang="ar-SA">ã</span>o, Everton Nunes da Silva, and Jorge Otávio Maia Barreto are editors of <i>Epidemiologia e Serviços de Saúde: revista do SUS</i>. Marcus Tolentino Silva has no conflicts of interest to declare. Tais Freire Galv<span lang="ar-SA">ã</span>o is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>


poster

<p>Hillary Wnfried Ramirez,<sup>1,2,3</sup> Malena Chiaborelli,<sup>1,2,3</sup> Christof M. Schönenberger,<sup>1</sup> Katie Mellor,<sup>4,5</sup> Alexandra N. Griessbach,<sup>1</sup> Paula Dhiman,<sup>4,6</sup> Pooja Gandhi,<sup>7</sup> Szimonetta Lohner,<sup>8,9</sup> Arnav Agarwal,<sup>10,11</sup> Ayodele Odutayo,<sup>5,12</sup> Michael M. Schlussel,<sup>4,6</sup> Philippe Ravaud,<sup>13,14</sup> David Moher,<sup>15,16 </sup>Matthias Briel,<sup>1,10</sup> Isabelle Boutron,<sup>13,14</sup> Sally Hopewell,<sup>4</sup> Sara Schroter,<sup>17,18 </sup>Benjamin Speich<sup>1,4</sup></p>
<h4>Objective </h4>
<p>Two randomized controlled trials (RCTs) conducted at journal level have shown that reminding peer reviewers about the 10 most important and underreported reporting items did not improve the reporting quality in published articles.<sup>1</sup> With this pooled in-depth analysis of peer reviewer reports, we aimed to assess at what stage the intervention failed.</p>
<h4>Design</h4>
<p> A subsample of peer reviewer reports from the control group (receiving no reminder) and the intervention group (receiving a reminder of the 10 most important reporting items) were analyzed. In brief, 2 blinded authors independently extracted from peer reviewer reports how many of the 10 key reporting items were flagged by peer reviewers for clarification. The main outcome of this analysis was the mean proportion of the 10 selected reporting items for which at least 1 peer reviewer requested clarification, assessed at the manuscript level. Furthermore, we assessed how many requested changes were later adequately reported in published articles.</p>
<h4>Results </h4>
<p>Across the RCTs, we had access to peer reviewer reports for 533 manuscripts (265 in the intervention group, assessing comments from 740 peer reviewers; and 268 in the control group, assessing comments from 719 peer reviewers). Our results indicate that reviewers in the intervention group requested clarification on more reporting items than those in the control group. Overall, reviewers in the intervention group flagged 21.1% of the 10 reporting items for clarification compared with 13.1% in the control group (mean difference, 8.0 percentage points (pp); 95% CI, 4.9-11.1 pp). However, the overall mean difference between groups was diluted from 8.0 to 4.2 pp when only assessing accepted and published articles and decreased even further to 2.6 pp when only considering changes that were then implemented by authors of manuscripts (<b>Table 25-0956</b>). Approximately 55% of reporting items that were criticized by peer reviewers were later adequately reported in the published article (intervention group, 173 of 308 [56.2%]; control group, 105 of 197 [53.3%]).</p>

![](https://assets.underline.io/markdown_image/1/image/80fe50628cb162743e45faadd6604b8b.png)

<h4>Conclusions</h4>
<p> Reminding peer reviewers to check reporting items increased their focus on reporting guidelines, leading to more reporting-related requests in their review reports. However, the effect was diluted during the peer review process (particularly due to rejected articles and requests not being implemented by authors). Journals should therefore make sure that requested clarifications are adequately addressed in revised manuscripts.</p>
<h4>Reference</h4>
<p>1. Speich B, Mann E, Schönenberger CM, et al. Reminding peer reviewers of reporting guideline items to improve completeness in published articles: primary results of 2 randomized trials. <i>JAMA Netw Open</i>. 2023;6(6):e2317651.</p>
<sup>1</sup>CLEAR Methods Center, Division of Clinical Epidemiology, Department Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland, benjamin.speich@usb.ch; <sup>2</sup>Swiss Tropical and Public Health Institute, Basel, Switzerland; <sup>3</sup>University of Basel, Basel, Switzerland; <sup>4</sup>Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK; <sup>5</sup>Clinical Outcomes Assessment, Clarivate, London, UK; <sup>6</sup>The EQUATOR Network, Oxford, UK; <sup>7</sup>Department of Communication Sciences and Disorders, Faculty of Rehabilitation Medicine, University of Alberta, Edmonton, AL, Canada; <sup>8</sup>Cochrane Hungary, Medical School, University of Pécs, Pécs, Hungary; <sup>9</sup>MTA–PTE Lendület “Momentum” Evidence in Medicine Research Group, Department of Public Health Medicine, Medical School, University of Pécs, Pécs, Hungary; <sup>10</sup>Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada; <sup>11</sup>Division of General Internal Medicine, Department of Medicine, McMaster University, Hamilton, ON, Canada; <sup>12</sup>Division of Nephrology, Toronto General Hospital, University Health Network, Toronto, ON, Canada; <sup>13</sup>Centre d’Épidémiologie Clinique, Hôpital Hôtel-Dieu, Assistance Publique Hôpitaux de Paris, Paris, France;<sup>14</sup>Université de Paris, CRESS, Inserm, INRA, Paris, France; <sup>15</sup>Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada; <sup>16</sup>Faculty of Medicine, School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada; <sup>17</sup><i>The BMJ</i>, London, UK; <sup>18</sup>Faculty of Public Health &amp; Policy, London School of Hygiene &amp; Tropical Medicine, London, UK.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Benjamin Speich and Matthias Briel reported receiving unrestricted grants from Moderna for studies unrelated to the presented work. Sara Schroter is employed by BMJ Publishing Group. Katie Mellor is employed by Clarivate. David Moher, Sally Hopewell, and Isabelle Boutron are members of the Consolidated Standards for Reporting Trials (CONSORT) executive board and authors of the CONSORT 2010 statement. David Moher is an author of the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) 2013 statement. David Moher, Michael M. Schlussel, Paula Dhiman, and Philippe Ravaud are members of the Enhancing the Quality and Transparency of Research (EQUATOR) network. Isabelle Boutron and David Moher are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Funding/Support </h4>
<p>Benjamin Speich was supported by a Return Postdoc.Mobility (P4P4PM_194496) grant from the Swiss National Science Foundation. Christof M. Schönenberger was funded by the Janggen Pöhn Foundation and the Swiss National Science Foundation (MD-PhD grant No. 323530_221860). Szimonetta Lohner was supported by the Hungarian Academy of Sciences (MTA) within the framework of the Lendület Programme.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.</p>
<h4>Additional Information</h4>
<p> Both trials (CONSORT-PR and SPIRIT-PR) were prospectively registered on Open Science Framework (<a href="https://osf.io/c4hn8"><span class="Hyperlink CharOverride-6">https://osf.io/c4hn8</span></a> and <a href="https://osf.io/z2hm9"><span class="Hyperlink CharOverride-6">https://osf.io/z2hm9</span></a>).</p>


Reminding Peer Reviewers to Comment on Reporting Items as Instructed by the Journal? An Analysis of 2 Randomized Trials

<p>Mario Malički,<sup>1,2,3</sup> Ana Jerončić,<sup>4</sup> Gerben ter Riet,<sup>5,6</sup> Lex Bouter,<sup>7,8</sup> John P. A. Ioannidis,<sup>2,3,9,10</sup> IJsbrand Jan Aalbersberg,<sup>11 </sup>Steven N. Goodman<sup>1,2,3,9,12</sup> </p>
<h4>Objective </h4>
<p>Previous research has indicated a knowledge gap on changes brought on by peer review and journal publishing.<sup>1</sup> We are conducting a living evidence synthesis of studies that analyzed differences between manuscript versions (eg, preprinted or submitted versions vs peer reviewed journal versions).</p>
<h4>Design </h4>
<p>This is a living synthesis of studies with meta-analyses of proportions (following PRISMA guidelines for reporting).<sup>2</sup> Studies were identified based on authors’ knowledge of the field, from 18 previous systematic reviews on peer review,<sup>3</sup> and by checking all research presented at the peer review conferences. All references and citations of identified studies (using Google Scholar up to January 2025) were checked. For all included studies that reported on changes, we extracted descriptive variables (including year of publication, sampling method, change comparison method, and data, wherever available, on how many analyzed version-pairs had changes for 11 outcomes). We assessed risk of bias for each outcome based on 3 domains: sample selection, measurement reliability, and conflicts of interest. To pool the data, we used random effects meta-analysis with Freeman-Tukey transformed proportions and reported the <i>I</i><sup>2</sup> index of heterogeneity.</p>
<h4>Results </h4>
<p>We identified 67 studies published from 1978 through the end of 2024, of which 31 (46%) analyzed changes between preprint and journal versions, 23 (34%) between submitted and accepted or published versions, 8 (12%) between rejected versions and those later published in other journals, 4 (6%) between multiple version sets, and 1 (1.5%) between different preprinted versions. Most studies, 45 (67%), analyzed changes manually, 13 (19%) used computational methods, and 9 (12%) combined manual and computational methods. The median number of analyzed version-pairs was 113 (IQR, 51-429). By discipline, 43 (64%) studies looked only at health research, 6 (9%) at life sciences, 6 (9%) at social sciences, 4 (6%) at physical sciences, and 8 (12%) at multiple disciplines. Meta-analyses showed the highest frequency of changes in title, authorship, conflicts of interest, and numerical results, with the lowest frequency in study conclusions (<b>Table 25-1077</b>).</p>

![](https://assets.underline.io/markdown_image/1/image/e3c3d82a0d010bd96adfaadbf9984c49.png)

<h4>Conclusions </h4>
<p>Current evidence indicates that while different manuscript sections experience varying degrees of changes from their submitted, preprinted, or rejected versions to their peer-reviewed journal version, the primary conclusions rarely change. Our results in this stage are limited by not conducting a systematic search of bibliographic databases and lack of (identified) data points from many subdisciplines, especially arts and humanities, as well as studies that explored possible factors associated with the extent of changes (eg, researchers’ or editors’ career stage or prestige, quality of initial manuscript version).</p>
<h4>References</h4>
<p>1. Tennant JP, Ross-Hellauer T. The limitations to our understanding of peer review. <i>Res Integr Peer Rev</i>. 2020;5:6. doi:10.1186/s41073-020-00092-1</p>
<p>2. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. <i>BMJ</i>. 2021;372:n71. doi:10.1136/bmj.n71</p>
<p>3. European Association of Science Editors. What is peer review? December 28, 2024. Accessed July 15, 2025. <a href="https://ease.org.uk/communities/peer-review-committee/peer-review-toolkit/What-is-peer-review/"><span class="Hyperlink CharOverride-6">https://ease.org.uk/communities/peer-review-committee/peer-review-toolkit/What-is-peer-review/</span></a></p>
<sup>1</sup>Stanford Program on Research Rigor and Reproducibility (SPORR), Stanford University, Stanford, California, US, mmalicki@stanford.edu; <sup>2</sup>Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, California, US; <sup>3</sup>Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, US; <sup>4</sup>Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia; <a id="_idTextAnchor069"></a><sup>5</sup>Urban Vitality Centre of Expertise, Amsterdam University of Applied Sciences, Amsterdam, The Netherlands; <sup>6</sup>Amsterdam University Medical Centers, Department of Cardiology, Amsterdam, The Netherlands; <sup>7</sup>Department of Philosophy, Faculty of Humanities, Vrije Universiteit, Amsterdam, The Netherlands; <sup>8</sup>Amsterdam University Medical Centers, Department of Epidemiology and Data Science, Amsterdam, The Netherlands; <a id="_idTextAnchor070"></a><sup>9</sup>Department of Medicine, Stanford University School of Medicine, Stanford, California, US<a id="_idTextAnchor071"></a>; <sup>10</sup>Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California, US; <sup>11</sup>Independent scholar; <sup>12</sup>Department of Health Policy, Stanford University School of Medicine, Stanford, California, US.</p>
<p class="abstract_conflict" lang="fr-FR"><span class="text-bold">Conflict of Interest Disclosures </span>Mario Malički was a co–editor in chief of Research Integrity and Peer Review, in which 1 of the included studies was published (he was not involved in the processing or decision making for that study, and Research Integrity and Peer Review full peer review histories are available to the public on the journal website). Mario Malički was also an external reviewer for 1 study. Gerben ter Riet is a co-author of 1 of the included studies and Steven N. Goodman is a co-author of 1 of the included studies. In both of these cases, data extractions and primary data analyses were done by Mario Malički and Ana Jerončić. IJsbrand Jan Aalbersberg was a Senior Vice President of Research Integrity at Elsevier. John P. A. Ioannidis, Lex Bouter, and Steven N. Goodman are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract.</p>
<h4>Funding/Support</h4>
<p> The work on this project started in 2020, and in 2020 Elsevier funding was awarded to Stanford University for a 1-year METRICS postdoctoral position that supported Dr Malički’s work. From 2021 to 2025, Dr Malički’s work has been supported by the Stanford School of Medicine Research Office, which also supports the work of Dr Goodman. At the start of the project, IJsbrand Jan Aalbersberg was an employee of Elsevier.</p>
<h4>Acknowledgments </h4>
<p>In 2022’s Peer Review Congress, we presented preliminary findings (<a href="https://peerreviewcongress.org/abstract/a-synthesis-of-studies-on-changes-manuscripts-underwent-between-submission-or-preprint-posting-and-peer-reviewed-journal-publication/"><span class="Hyperlink CharOverride-6">https://peerreviewcongress.org/abstract/a-synthesis-of-studies-on-changes-manuscripts-underwent-between-submission-or-preprint-posting-and-peer-reviewed-journal-publication/</span></a>) that covered 25 studies published between 1990 and 2021. The current results include studies published before 1990 and up to January 2025.</p>
<h4>Additional Information</h4>
<p> A website with computationally reproducible results from this living review will be available soon.</p>


Differences Between Manuscript Versions: A Living Review and Series of Meta-Analyses

<p>Melecia Miller,<sup>1</sup> <span lang="it-IT">Vera Nezgovorova,</span><sup>2</sup> Ilana Kersh,<sup>3</sup> Mohamed I. Elsaid,<sup>4</sup> Marina Broitman<sup>2</sup></p>
<h4>Objective</h4>
<p> This study’s goal was to develop a way to assess patient review quality based on author and editor reactions to reviews. We used patient reviews of draft final research reports (DFRRs) of Patient Centered Outcomes Research Institute (PCORI)–funded research that were peer reviewed by clinical, statistical, and patient reviewers.<sup>1</sup> Review quality ratings were based on prior research on PCORI peer review and other research on patient reviewers.<sup>2,3</sup> We hypothesized that specific, substantive changes to the report and patient-focused comments would be associated with a favorable mention of the patient review by the editor as well as a substantive response from the author.</p>
<h4>Design</h4>
<p> Pilot testing on 13 patient reviews for 13 clinical research DFRRs submitted from 2021 to 2022 was completed in 2025. We conducted a content analysis of patient reviewer comments and developed a coding scheme based on prior research. Coding was performed at the sentence level and included 2 categories: (1) specificity and constructiveness of the comment and (2) patient focus of the comment (<b>Table 25-1110</b>). Each review was coded independently by 2 raters each for category 1 and category 2, with a final Cohen κ of 0.83 and 0.66, respectively, indicating moderate to substantial interrater agreement. Coding discrepancies between reviewers received the most conservative (ie, lowest value) code. Raters assessed whether the editor mentioned the patient review at least once in their review summary and either encouraged (code 1) or discouraged (code 2) an author response vs no mention (code 0). Author responses to reviewer comments were coded by paragraph indicating whether authors responded directly to patient-reviewer comments (code 1) or made changes to the report because of the comments (code 2) vs neither (code 0). We used hierarchical modeling to calculate the odds of editors encouraging authors to respond (code 1 vs codes 0 or 2) and mixed-effects logistic regression to calculate the odds of authors responding substantively (codes 1 or 2 vs code 0) to patient reviews with more specific, constructive comments.</p>
<h4>Results</h4>
<p> <b>Table 25-1110</b> shows counts for associations between category 1 and category 2. More specific and constructive comments and patient-focused comments were not associated with higher odds of the editor mentioning the patient review (N = 13; odds ratio [OR], 0.36; 95% CI, 0.28-0.45 and OR, 0.65; 95% CI 0.54-0.78, respectively). Patient review paragraphs with more than 50% specific and constructive comments were associated with higher odds of authors responding substantively, compared with other comment types (N = 147; OR, 4.00; 95% CI, 1.67-9.59; <i>P</i> = .002).</p>
<a href="/wp-content/uploads/2025_conference/tables/prc25-1110t1.png"> <img loading="lazy" src="/wp-content/uploads/2025_conference/tables/prc25-1110t1.png"> </a>
<h4>Conclusions</h4>
<p> This pilot study shows the feasibility of establishing codes rating the quality of patient reviews and determining whether higher quality patient reviews are associated with mentions by the editor and author responses. The results are limited because of the sample size and the uniqueness of DFRRs.</p>
<h4>References</h4>
<p>1. Broitman M, Sox HC, Slutsky J. A model for public access to trustworthy and comprehensive reporting of research. <i>JAMA</i>. 2019;321(15):1453-1454. doi:10.1001/jama.2019.2807</p>
<p>2. Ghazaryan E, Broitman M, Sox H. Differences in the style and quantity of reviewer comments in structured vs unstructured peer review forms. Poster presented at: Ninth International Congress on Peer Review and Scientific Publication; September 10, 2022; Chicago, IL.</p>
<p>3. Huml AM, Albert JM, Beltran JM, et al. Community members as reviewers of medical journal manuscripts: a randomized controlled trial. <i>J Gen Intern Med</i>. 2022;38(6):1393-401. doi:10.1007/s11606-022-07802-z</p>
<sup>1</sup>KnowledgeWorks Global, Ltd., 410 MN-25, Brainerd, MN, US; <span lang="it-IT">mmiller@originreview.org; </span><sup>2</sup>Patient-Centered Outcomes Research Institute, Washington, DC, US; <sup>3</sup>University of Pittsburgh, Pittsburgh, PA, US; <sup>4</sup>The Ohio State University College of Medicine, Columbus, Ohio, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>


Quality of Patient Reviewer Comments and Association With Author and Editor Responses

<p>Matthew B. Stanbrook,<sup>1</sup> Shannon Charlebois,<sup>1</sup> George A. Tomlinson,<sup>1</sup> Meredith Weinhold<sup>1</sup></p>
<h4>Objective</h4>
<p> The opportunity to appeal a journal’s decision to reject a manuscript enhances procedural fairness toward authors and allows editors to correct errors in judgment but wastes time for both authors and editors when unsuccessful. We sought to identify factors associated with outcomes of appeals of research manuscripts at the <i>Canadian Medical Association Journal</i> (<i>CMAJ</i>), a general medical journal that explicitly highlights the option to appeal in rejection letters.</p>
<h4>Design</h4>
<p> We conducted a retrospective cohort study of all appeal requests for manuscripts initially rejected after submission to the Research section of <i>CMAJ</i> between 2007 and 2024. We obtained data on manuscript characteristics and editorial decisions from <i>CMAJ</i>’s manuscript submission database (ScholarOne). Three editors independently reviewed the text of the initial manuscript decision letters to identify reasons for rejection, with disagreements resolved by consensus. We included manuscripts rejected at any stage of the editorial process. The primary outcome was whether appeal requests were granted or denied. A secondary outcome was whether appealed manuscripts were ultimately published. We estimated associations of multiple prespecified characteristics with appeal outcomes using bayesian logistic regression.</p>
<h4>Results</h4>
<p> Among 14,351 submitted manuscripts, 13,419 were rejected, and 616 of these rejections (4.6%) were appealed. Among appeals, 214 (34.7%) were granted, of which 58 (27.1% of granted appeals and 9.4% of all appeals) were ultimately published. Compared with manuscripts rejected without external peer review, granted appeals were more likely for manuscripts rejected after peer review without editorial discussion (adjusted odds ratio [AOR], 2.71; 95% credible interval [CrI], 1.40-5.27) but similarly likely for manuscripts rejected after peer review and editorial discussion (AOR, 1.19; 95% CrI, 0.74-1.91) (<b>Table 25-1161</b>). Granted appeals were more likely if rejection letters included positive comments (AOR, 1.82; 95% CrI, 1.21-2.76) or cited lack of clarity (AOR, 1.84; 95% CrI, 1.12-3.04) and less likely if corresponding authors were based outside Canada (AOR, 0.38; 95% CrI, 0.24-0.59). The odds of an appeal being granted also varied significantly based on the individual editor making the appeal decision (median AOR, 1.90; 95% CrI, 1.24-4.37). Among factors associated with appeals being granted, only positive comments (AOR, 1.98; 95% CrI, 1.10-3.59) were also significantly associated with successful publication after appeal. However, appealed manuscripts rejected after peer review and editorial discussion were more likely to be published than were those rejected without peer review (AOR, 4.93; 95% CrI, 2.47-10.25).</p>

![](https://assets.underline.io/markdown_image/1/image/ca4a766cb379b500fce70628c15c12cb.png)

<h4>Conclusions</h4>
<p> Characteristics of authors, editors, editorial process, and decision communication were associated with successful appeals of initially rejected research manuscripts. Lack of awareness of the small fraction of appeals that ultimately achieve publication may raise false expectations among authors.</p>
<sup>1</sup><i>Canadian Medical Association Journal </i>(<i>CMAJ</i>), Toronto, Ontario, Canada, matthew.stanbrook@cmaj.ca.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> All authors are employees or paid consultants of CMAJ. Matthew B. Stanbrook, Shannon Charlebois, and George A. Tomlinson participated in editorial decision making for some of the manuscripts included in the study. No other disclosures were reported.</p>


Factors Associated With Outcomes of Appeals of Manuscripts Initially Rejected by a General Medical Journal

<p>Daniel S. Evanko,<sup>1</sup> Deondre Jordan<sup>1</sup></p>
<h4>Objective </h4>
<p>Journals ask authors to comply with many reporting requirements to improve study transparency, reproducibility, and reuse. Manual monitoring and enforcement of requirements is difficult, and reporting checklists or vendor-based solutions can be untargeted, burdensome, expensive, or of limited effectiveness. For example, since June 2020 the American Association for Cancer Research has used a methods review tool (SciScore) to help improve author compliance with reporting requirements at its 10 journals. This tool greatly increased author assignment of research resource identifiers (RRIDs),<sup>1</sup> but compliance plateaued at approximately 30% of original research articles published in the association journals. This cross-sectional study describes a simple, inexpensive, automated approach to further overcome limitations of common approaches and quantitatively evaluates its performance using 3 example interventions.</p>
<h4>Design </h4>
<p>Custom Python scripts coupled with structured language queries were used for targeted extraction of manuscript data from the EJournal Press journal submission system. For RRID reporting compliance, once every day the submission system was queried for manuscripts that completed initial quality control checks and did not contain the text RRID. A templated e-mail alert was sent to the corresponding author that explained what RRIDs are, why they are important, and how to add them. The system was also adapted with a custom supervised machine-learning method to identify studies that generated new sequencing data or that were computationally intensive to encourage authors to deposit sequence data and provide computer code used in their study and used to send different alerts for these manuscripts at different designated points in the editorial process.</p>
<h4>Results </h4>
<p>The new automated RRID process was activated August 6, 2024, at all the association journals, resulting in missing-RRID alerts sent for 84% (4949 of 5899) of manuscripts at completion of initial quality control checks through the end of 2024. Performance was assessed by searching for “RRID” in published articles as done previously.<sup>1</sup> An immediate sharp increase in articles containing RRIDs was observed that quickly increased the percentage of published research articles containing RRIDs from 33% (176 of 526) in April through July to 65% (330 of 505) in November through February <b>(Figure 25-1181</b>). Targeted alerts to authors of manuscripts asking them to deposit sequencing data (sent after peer review started) or supply code in a computational science software (Code Ocean) capsule (sent after a revise decision was sent to the author) were turned on on August 26, 2024, and resulted in sequence deposition alerts sent for 43% (1416 of 3261) of manuscripts and Code Ocean alerts sent for 39% (614 of 1594) of manuscripts through April 30, 2025. The sequencing alert triggered more than 100 author e-mail communications about deposition, demonstrating author engagement with this request and allowing resolution early in the editorial process. The percentage of accepted manuscripts with linked Code Ocean capsules increased from 1.4% (21 of 1555) in 2024 to 4.6% (25 of 547) in the January to May 2025 period, a more than 3-fold increase.</p>

![](https://assets.underline.io/markdown_image/1/image/87ac32761435115531ebf1a5f6ddec09.png)

<h4>Conclusions </h4>
<p>This fully automatic and highly targeted process engages with authors in a context-dependent way at predefined times in the editorial process that are appropriate for the specific communication, thus achieving substantial author engagement and improved compliance with journal requirements.</p>
<h4>Reference</h4>
<p>1. Roelandse M, Ozykurt IB, Evanko D, Bandrowski A. Assessing the effectiveness of SciScore in supporting the reproducibility of scientific research. <i>Sci Ed</i>. 2023;46:46-52.</p>
<sup>1</sup>American Association for Cancer Research, Philadelphia, PA, US, daniel.evanko@aacr.org.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Daniel S. Evanko is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>


Automated targeted emails for improving author compliance with study reporting requirements and other editorial processes

<p>Franki Y. H. Kung,<sup>1</sup> Mariam Aly,<sup>2</sup> Shahana Ansari,<sup>3</sup> Eliana Colunga,<sup>4</sup> M. J. Crockett,<sup>5,6</sup> Amanda B. Diekman,<sup>3</sup> Pablo Gomez,<sup>7</sup> Paul C. McKee,<sup>8</sup> Miriam Pérez,<sup>9</sup> Sarah M. Stilwell,<sup>10</sup> Matthew Goldrick<sup>11,12</sup></p>

<h4>Objective</h4> Accumulating data suggest peer reviews can exhibit bias<i> </i>(ie, systematically devaluing certain topics, methods, or the study of nondominant populations<sup>1</sup>) and can be unfair with respect to process (eg, decisions are made inconsistently) or people (eg, authors are disrespected).<sup>2</sup> Recent studies have shown that biases and unfair behaviors in peer review tend to undermine the self-reported experiences and career progression of researchers, including those from underserved populations, such as women, nonbinary individuals, and people from racial and ethnic groups other than White.<sup>1-3</sup> This study examined whether peer review exercises increased psychology and neuroscience trainees’ perceived readiness to respond to biased and unfair reviewers.</p>

<h4>Design </h4>Single-session synchronous online trainings (conducted during 2023-2024) allowed for trainer-trainee and trainee-trainee interactions. Trainees were recruited for this study through social media, targeted academic organizations, and graduate programs. Trainers (professors and a postdoctoral fellow with multiple years of peer review experience) reviewed processes occurring between journal submission and receipt of a decision letter. Trainees were then given a simplified, anonymized decision letter and reviews. Exercises and discussions illustrated how trainees could plan manuscript revisions and compose responses to critiques. Finally, trainees were given examples of biased and unfair reviews, along with strategies for responding to them while acknowledging the power dynamics inherent to review. Discussion included how to maintain motivation when encountering such reviews. Participants were asked to complete surveys before and after online training in which they rated their ability to perform the following targeted skills on a scale from 1 (very little skill) to 5 (great deal of skill): define and identify bias in peer reviews; define and identify unfair language in peer reviews; respond to reviews using fair language; respond to biased or unfair reviews; and promote an inclusive and equitable peer-review culture.</p>

<h4>Results </h4>A total of 64 participants (42 PhD students [66%], 10 predoctoral students [16%], 8 postdoctoral trainees [13%], and 4 other trainees [5%]) rated their level of skill for 5 targeted skills. Ordinal regressions showed participants had significantly higher ratings for all skills at posttest. For example, the mean (SD) self-ratings for the skill to respond to biased on unfair reviews increased from 2.0 (1.01) on the pretest to 4.09 (0.89) on the posttest (<b>Table 25-0893</b>).</p>

![](https://assets.underline.io/markdown_image/1/image/f801de28833ed0c4470dce4f2815a32b.png)

<h4>Conclusions </h4>
<p>After these peer review exercises, psychology and neuroscience trainees had an increase in their perceived readiness to confront biased and unfair reviews. Work is ongoing to scale up this training using on-demand materials and further assess training impact, including measures beyond trainee self-reports.</p>
<h4>References</h4>
<p>1. Rogers LO, Moffitt U, McLean KC, Syed M. Research as resistance: naming and dismantling the master narrative of “good” science. <i>Am Psychol</i>. 2024;79:484-496. doi:10.1037/amp0001246</p>
<p>2. Silbiger NJ, Stubler AD. Unprofessional peer reviews disproportionately harm underrepresented groups in STEM. <i>PeerJ</i>. 2019;7:e8247. doi:10.7717/peerj.8247</p>
<p>3. Aly M, Colunga E, Crockett MJ, et al. Changing the culture of peer review for a more inclusive and equitable psychological science. <i>J Exp Psychol Gen</i>. 2023;152:3546-3565. doi:10.1037/xge0001461</p>
<sup>1</sup>Department of Psychological Sciences, Purdue University, West Lafayette, IN, US; <sup>2</sup>Department of Psychology, University of California Berkeley, Berkeley, CA, US; <sup>3</sup>Department of Psychological and Brain Sciences, Indiana University Bloomington, Bloomington, IN, US; <sup>4</sup>Department of Psychology and Neuroscience, University of Colorado Boulder, City, CO, US; <sup>5</sup>Department of Psychology, Princeton University, Princeton, NJ, US; <sup>6</sup>Department of Psychology, University Center for Human Values, Princeton University, Princeton, NJ, US; <sup>7</sup>Department of Psychology, Skidmore College, Saratoga Springs, NY, US; <sup>8</sup>Department of Psychology and Neuroscience, Duke University, Durham, NC, US; <sup>9</sup>Department of Psychology, North Park University, Chicago, IL, US; <sup>10</sup>Department of Health Behavior and Health Equity, University of Michigan, Ann Arbor, MI, US; <sup>11</sup>Department of Linguistics, Northwestern University, Evanston, IL, US, matt-goldrick@northwestern.edu; <sup>12</sup>Cognitive Science Program, Northwestern University, Evanston, IL, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This work was supported by grants from the National Science Foundation (DGE-2224777, DGE-2224779, DGE-2436430). </p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funder of the study had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Additional Information</h4>
<p> Training materials are freely available (<a href="https://osf.io/658ev"><span class="Hyperlink CharOverride-6">https://osf.io/658ev</span></a>).</p>


Peer Review Exercises to Enhance Trainees' Readiness to Confront Unfair or Biased Reviews

<p>Christopher Baethge,<sup>1,2</sup>Hannah Jergas<sup>3</sup></p>
<h4>Objective</h4>
<p> Although quotations are central to scientific publications, they often misrepresent the cited reference.<sup>1</sup> Inaccurate quotations may mislead readers, undermine the line of argument of a paper, and threaten the integrity of the scientific record.<sup>2,3</sup> We updated an earlier meta-analysis<sup>1</sup> to assess improvement in recent years.</p>
<h4>Design</h4>
<p> This systematic review and meta-analysis, preregistered on Open Science Framework (OSF), was conducted following the Cochrane Handbook and PRISMA statement. It focuses on quotation inaccuracy—errors of content—as opposed to bibliographical errors, often referred to as citation errors. We searched MEDLINE, PubMed Central via PubMed, and Web of Science from January 1, 2014, through October 1, 2023, for studies on quotation errors in the medical literature, with no date, language, or source restrictions, and we hand-searched all reference lists of included studies. Literature screening, data extraction, and risk of bias estimation were carried out by 2 authors independently. We adopted the original studies’ definitions<sup>1</sup> of major inaccuracies (ie, not at all in accordance with the claim of the original authors), minor inaccuracies (ie, inconsistencies and factual errors not severe enough to contradict a statement by citing authors), secondary quotations (ie, quotations of sources referencing an original study), and total quotation errors. Summary proportions of quotation errors were estimated in random-effects meta-analyses (log-transformed proportions and DL τ<sup>2</sup> estimates), with heterogeneity measures, 95% confidence intervals, and 95% prediction intervals. Time trends were investigated in meta-regressions.</p>
<h4>Results</h4>
<p> Based on 46 studies included, with 32,074 quotations/references checked, 16.9% were incorrect (95% CI, 14.1%-20.0%). Of these, 8.0% (95% CI, 6.4%-10.0%) were major inaccuracies and 7.8% (95% CI, 5.7%-10.5%) were minor inaccuracies. Heterogeneity throughout all meta-analyses was high (<b>Table 24-0811</b>). Rates of total errors have not improved during the last decade (slope: −0.002; 95% CI, −0.03 to 0.02; <i>P</i> = .85) and neither have rates of major errors (slope: 0.002; 95% CI, −0.02 to 0.03; <i>P</i> = .89). There were no associations with risk of bias, publication bias, number of references, or medical specialty (surgical, nonsurgical, other), but the total error rate was statistically significantly (<i>P</i> = .04; binomial, n = 25) and negatively correlated with Journal Impact Factor (−0.25; Spearman, n = 23). Interrater reliability was sufficient (Cohen κ: 0.73), and summary estimates were supported by sensitivity analyses employing arcsine square root data transformation and Sidik and Jonkman τ<sup>2</sup> estimates.</p>

![](https://assets.underline.io/markdown_image/1/image/6088d5f3f296b8d165f7d47e3340260a.png)

<h4>Conclusions</h4>
<p> Quotation inaccuracy remains prevalent in scientific medical texts. Despite seemingly broad awareness among journal editors and probably among many authors, there is no evidence of improvement in recent years. It is uncertain how the utilization of artificial intelligence in writing articles and in double-checking quotation inaccuracy will impact the problem, but human effort will likely remain important in creating and vetting quotes—on the part of authors, editors, reviewers, and readers.</p>
<h4>References</h4>
<p>1. Jergas H, Baethge C. Quotation accuracy in medical journal articles—a systematic review and meta-analysis. <i>PeerJ</i>. Published online October 27, 2015. doi:10.7717/peerj.1364</p>
<p>2. Baethge C. Importance, errors, and patterns of quotations to psychiatric original articles. <i>Pharmacopsychiatry</i>. 2020;53(6):247-255. doi:10.1055/a-1167-35673</p>
<p>3. Peoples N, Østbye T, Yan LL. Burden of proof: combating inaccurate citation in biomedical literature. <i>BMJ</i>. Published online November 6, 2023. doi:10.1136/bmj-2023-076441</p>
<sup>1</sup><i>Deutsches Ärzteblatt </i>and<i> Deutsches Ärzteblatt International</i>, Editorial Offices, Cologne, Germany, <span lang="de-DE">baethge@aerzteblatt.de; </span><sup>2</sup>Department of Psychiatry and Psychotherapy, Faculty of Medicine, University of Cologne, Cologne, Germany; <sup>3</sup>Department of Neurology, Faculty of Medicine, University of Cologne, Cologne, Germany.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Both authors have published on this topic before. Christopher Baethge is employed by a general medical journal (<i>Deutsches Arzteblatt, Deutsches Arzteblatt International</i>).</p>
<h4>Acknowledgment</h4>
<p> The authors gratefully acknowledge Joan Albert Hammerstein’s help in screening the literature.</p>
<h4>Additional Information</h4>
<p> This study is registered on Open Science Framework (OSF) at <a href="https://osf.io/95rej/?view_only=8c5d2b51a8814278bff1fec1e40d0bb9"><span class="Hyperlink CharOverride-6">https://osf.io/95rej/?view_only=8c5d2b51a8814278bff1fec1e40d0bb9</span></a></p>


Quotation inaccuracy in medicine - A systematic review and meta-analysis

<p>Seba Qussini,<sup>1,2</sup> Farizah Mezer Anami,<sup>2</sup> Kris Dierickx<sup>1</sup></p>
<h4>Objective </h4>
<p>Many peer review attributes are widely criticized and remain poorly investigated, particularly in the context of proposals’ peer review.<sup>1</sup> This study aims to explore the diverse perspectives of stakeholders regarding the role of unmasking in funding proposals’ peer review and the implications of open peer review as part of the recent open science movement, specifically in biomedical research proposals’ peer review.</p>
<h4>Design </h4>
<p>To describe participants’ perspectives as constructed through their recent experiences, we have conducted a generic descriptive qualitative study<sup>2</sup> within a constructivist paradigm, using semistructured interviews to gather insights from reviewers, applicants, and peer review scholars. A total of 23 participants were selected through purposive and snowball sampling from funding agencies in Belgium and Qatar with whom no prior relationship had been established. Interviews were conducted between June 2024 and February 2025 by the first author (S.Q.). Transcribed interviews were analyzed according to the 6-step thematic framework analysis described by Braun and Clarke.<sup>3</sup> Initially, autogenerated transcripts were read and checked for in-depth familiarization with the data, which was followed by a line-by-line inductive coding, conducted iteratively after each set of 2 interviews. We followed the Consolidated Criteria for Reporting Qualitative Research (COREQ) guidelines.</p>
<h4>Results </h4>
<p>Codes with shared characteristics were grouped into categories, and ultimately 3 overarching themes were generated: (1) the importance of increased transparency in fund allocation procedures while maintaining anonymized reviewer identities; (2) open peer review as a feasible approach for enhancing transparency and accountability in funding proposals’ peer review; and (3) a growing critical stance toward traditional peer review systems, calling for alternative models like baseline or lottery funding procedures. Collectively, the results shed light on the perceived advantages and limitations of different peer review models and provide an understanding of how unmasking identities influences the fairness and objectivity of fund allocation decisions.</p>
<h4>Conclusions </h4>
<p>There is persistent preference for double-anonymous review among the scientific community; however, researchers are increasingly aware of the shortcoming of anonymized review, especially in light of current challenges within the funding landscape. Simultaneously, they recognize the importance of greater openness in peer review and increased transparency in fund allocation procedures.</p>
<h4>References</h4>
<p>1. Qussini S, MacDonald RS, Shahbal S, Dierickx K. Blinding models for scientific peer-review of biomedical research proposals: a systematic review. <i>J Empir Res Hum Res Ethics</i>. 2023;18(4):250-262. doi:10.1177/15562646231191424</p>
<p>2. Qussini S, Dierickx K. Multi-stakeholder perspectives on current attitudes toward (un)blinding reviewers’ identity in biomedical research proposals peer review: a qualitative study. OSF Registries. <a href="https://osf.io/dp65f"><span class="Hyperlink CharOverride-6">https://osf.io/dp65f</span></a></p>
<p>3. Clarke V, Braun V. Thematic analysis. <i>J Pos Psychol</i>. 2017;12(3):297-298. doi:10.1080/17439760.2016.1262613</p>
<sup>1</sup>Centre for Biomedical Ethics and Law, Faculty of Medicine, KU Leuven, Leuven, Belgium, seba.qussini@student.kuleuven.be; <sup>2</sup>Hamad Medical Corporation, The Medical Research Center, Doha, Qatar.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> The publication and conference fees will be funded by the Medical Research Center at Hamad Medical Corporation through grant MRC-01-24-305.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The Medical Research Center had no role in the design, analysis, or interpretation of this study.</p>


Multi-Stakeholder Perspectives on Current Attitudes Toward Unmasking Reviewers' Identity in Biomedical Research Proposals Peer Review: A Qualitative Study

<p>Jong-Wook Ban,<sup>1</sup> Hans Lund,<sup>2</sup> Karen Robinson,<sup>3</sup> Ida Svege,<sup>4</sup> Jan-Ole Hesselberg<sup>5,6</sup></p>
<h4>Objective</h4>
<p> Many have argued that the need for new research should be determined through a systematic examination of existing evidence to maximize its value.<sup>1,2</sup> Our study evaluated the outcomes of following such arguments on funding applications. We tested a hypothesis that funding applications justifying new research with a systematic review would receive higher scores than those without such justification.</p>
<h4>Design</h4>
<p> We analyzed funding applications submitted to Stiftelsen Dam, one of Norway’s largest health foundations, during the 2024 cycle. We used multiple regression to test our hypothesis while controlling for the influence of the following 5 covariables: requested funding amount, principal investigator’s birth year, principal investigator’s gender, project’s health domain, and research methods. </p>
<h4>Results</h4>
<p> Of 420 applications received, 193 (46.0%) cited a systematic review, 181 (43.1%) referenced a published or unpublished study, and 46 (11.0%) referred to applicants’ own experience as justification for new research. The median requested funding amount was kr3,000,000 ($267,796), with an IQR of kr2,801,000 to kr3,000,000. There were 158 (37.6%) men and 247 (58.8%) women principal investigators, and 15 (3.6%) who did not disclose their gender. The median birth year of the principal investigators was 1976 (IQR, 1968-1982). Funding applications included research projects in mental health (122 [29.0%]), physical health (197 [46.9%]), well-being (42 [10.0%]), disability (18 [4.3%]), and other domains (41 [9.8%]). Qualitative, mixed, and quantitative methods were used in 53 (12.6%), 145 (34.5%), and 222 (52.9%) applications, respectively. The median funding application score was 4.8 (IQR, 4.4-5.1) on a 7-point scale, where 1 represents poor and 7 represents excellent. Applications that cited a systematic review as justification received scores 0.19 (95% CI, 0.03-0.35) points higher than those that relied on the applicants’ own experience (<b>Figure 25-1129</b>). This could have raised an application with an average score by approximately 100 places in a ranking of 1716 applications. Applications that cited a systematic review received scores 0.12 (95% CI, 0.02-0.22) points higher than those that cited other types of evidence. </p>

![](https://assets.underline.io/markdown_image/1/image/679eec507dc3d8912ae0b0169dacb029.png)

<h4>Conclusions</h4>
<p> Our exploratory analysis showed an association between citing a systematic review and higher funding application scores. Our study has the following limitations. First, we could only control for variables available in our data. So, it was impossible to exclude the influence of other potential confounders, such as study design. Second, we evaluated funding application scores rather than funding outcomes. We could not evaluate the funding outcomes due to the very small number of applications receiving funding each year. Because Stiftelsen Dam heavily relies on funding application scores to decide on funding, we used those scores as a surrogate for funding outcomes.</p>
<h4>References</h4>
<p>1. Chalmers I, Bracken MB, Djulbegovic B, et al. How to increase value and reduce waste when research priorities are set. <i>Lancet</i>. 2014;383(9912):156-165. doi:10.1016/S0140-6736(13)62229-1</p>
<p>2. Lund H, Brunnhuber K, Juhl C, et al. Towards evidence based research. <i>BMJ</i>. 2016;355:i5440. doi:10.1136/bmj.i5440</p>
<sup>1</sup>Section of Hospital Medicine, University of Chicago, Chicago, IL, US, jban@uchicago.edu; <sup>2</sup>Section of Evidence-Based Practice, Western Norway University of Applied Sciences, Bergen, Norway; <sup>3</sup>Johns Hopkins Evidence-Based Practice Center, Johns Hopkins University, Baltimore, MD, US; <sup>4</sup>Nordic Institute for Studies in Innovation, Research and Education, Oslo, Norway; <sup>5</sup>Stiftelsen Dam, Oslo, Norway; <sup>6</sup>Department of Psychology, University of Oslo, Oslo, Norway.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Ida Svege reported being the former Head of Program Development (Leder for programutvikling) of Stiftelsen Dam. Jan-Ole Hesselberg reported being the current Program Manager (Programsjef) of Stiftelsen Dam. </p>


The influence of using a systematic review to justify new research on funding application score

<p>Markus K. Heinemann,<sup>1</sup><span lang="de-DE"> </span>Andreas Boening,<sup>2</sup><span lang="de-DE"> Kazunori Okabe,</span><sup>3</sup><span lang="de-DE"> Jessica Bogensberger</span><sup>4</sup></p>
<h4>Objective</h4>
<p> In 2013, a cardiothoracic surgical journal introduced plagiarism detection (Crossref, powered by iThenticate). After a phase of 2 years during which all submissions were screened, since January 2015 only returned manuscripts after revision were checked by default. Primary submissions were then screened at the discretion of the Editor in Chief (EiC) only. The aim of this study was to report the 12-year experience.</p>
<h4>Design</h4>
<p> All manuscripts submitted between January 1, 2013, and December 31, 2024, and branded by the plagiarism search engine with a similarity index above 33% were analyzed regarding the originator of the screening (EiC or revision default), reasons (actual plagiarism or self-plagiarism, which is a form of duplication), similarity percentage, and final decision. Descriptive statistics were applied. All updates of the search tool were implemented.</p>
<h4>Results</h4>
<p> During the investigation period, 4083 manuscripts were submitted. A total of 99 (2.4%) showed a similarity index greater than 33%. The mean percentage was 53.3% (minimum, 34%; maximum, 99%; median, 50%; SD, ±14.8). Plagiarism and duplication search was prompted by the EiC in 48 papers and by automated default in 51 papers. Self-plagiarized or duplicated content was by the same authors in 71 of 99 papers (71.7%), by different authors in 18 (18.2%), and both in 10 (10.1%). Fifty-five manuscripts (55.6%) remained unpublished: 17 rejected without review, 34 rejected, and 4 withdrawn. Forty-four of 99 (44.4%) were accepted after revisions. There were 4 attempts at duplicate publication, all rejected. Preprints were discovered in 3 cases, with 1 accepted after revision: the preprint had been uploaded by a different journal for review and was rejected there, but then not deleted. The 44 manuscripts finally accepted contained previously published sequences by the same authors in 36, by different authors in 4, and both in 4. In 25 cases, the EiC rendered the similarity uncritical (eg, materials and methods section). The most frequent necessary correction was adequate citation. No adequate citation was given for self-plagiarized/duplicated content in 23 cases and for content by different authors in 4.</p>
<h4>Conclusions</h4>
<p> Although the overall rate of plagiarism and duplication can be considered low, scientific misconduct was detected and its publication prevented in a few cases. In many instances, duplicated passages contained similar material and methods previously published by the same group. Deficiencies in adequate citation are a problem and necessitate continued education of authors. Selective plagiarism and duplication screening remains an important tool, although chosen thresholds of similarity remain arbitrary. Further improvements of sensitivity are expected, especially with the implementation of machine learning advances. Editorial decisions, however, cannot be made on automated findings alone, but must be put into personal perspective.</p>
<sup>1</sup>German Society for Thoracic and Cardiovascular Surgery (DGTHG), Germany, heinemann@uni-mainz.de; <sup>2</sup><span lang="de-DE">Universitaetsklinik Giessen, Germany; </span><sup>3</sup><span lang="de-DE">Bell Land General Hospital, Osaka, Japan; </span><sup>4</sup>Thieme Publishers, Stuttgart, Germany.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Jessica Bogensberger is an employee of Thieme Publishers, the publishing house of the journal investigated. All other authors have no conflict of interest to declare.</p>


Next from The 10th Peer Review Congress

Reminding Peer Reviewers to Comment on Reporting Items as Instructed by the Journal? An Analysis of 2 Randomized Trials

Similar lecture

[V] In-Person Peer Review Training to Improve Preparedness to Evaluate Manuscripts

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES