United States

&lt;p&gt;Mario Malički,&lt;sup&gt;1,2,3&lt;/sup&gt; Ana Jerončić,&lt;sup&gt;4&lt;/sup&gt; Gerben ter Riet,&lt;sup&gt;5,6&lt;/sup&gt; Lex Bouter,&lt;sup&gt;7,8&lt;/sup&gt; John P. A. Ioannidis,&lt;sup&gt;2,3,9,10&lt;/sup&gt; IJsbrand Jan Aalbersberg,&lt;sup&gt;11 &lt;/sup&gt;Steven N. Goodman&lt;sup&gt;1,2,3,9,12&lt;/sup&gt; &lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;Previous research has indicated a knowledge gap on changes brought on by peer review and journal publishing.&lt;sup&gt;1&lt;/sup&gt; We are conducting a living evidence synthesis of studies that analyzed differences between manuscript versions (eg, preprinted or submitted versions vs peer reviewed journal versions).&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;This is a living synthesis of studies with meta-analyses of proportions (following PRISMA guidelines for reporting).&lt;sup&gt;2&lt;/sup&gt; Studies were identified based on authors’ knowledge of the field, from 18 previous systematic reviews on peer review,&lt;sup&gt;3&lt;/sup&gt; and by checking all research presented at the peer review conferences. All references and citations of identified studies (using Google Scholar up to January 2025) were checked. For all included studies that reported on changes, we extracted descriptive variables (including year of publication, sampling method, change comparison method, and data, wherever available, on how many analyzed version-pairs had changes for 11 outcomes). We assessed risk of bias for each outcome based on 3 domains: sample selection, measurement reliability, and conflicts of interest. To pool the data, we used random effects meta-analysis with Freeman-Tukey transformed proportions and reported the &lt;i&gt;I&lt;/i&gt;&lt;sup&gt;2&lt;/sup&gt; index of heterogeneity.&lt;/p&gt;
&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;We identified 67 studies published from 1978 through the end of 2024, of which 31 (46%) analyzed changes between preprint and journal versions, 23 (34%) between submitted and accepted or published versions, 8 (12%) between rejected versions and those later published in other journals, 4 (6%) between multiple version sets, and 1 (1.5%) between different preprinted versions. Most studies, 45 (67%), analyzed changes manually, 13 (19%) used computational methods, and 9 (12%) combined manual and computational methods. The median number of analyzed version-pairs was 113 (IQR, 51-429). By discipline, 43 (64%) studies looked only at health research, 6 (9%) at life sciences, 6 (9%) at social sciences, 4 (6%) at physical sciences, and 8 (12%) at multiple disciplines. Meta-analyses showed the highest frequency of changes in title, authorship, conflicts of interest, and numerical results, with the lowest frequency in study conclusions (&lt;b&gt;Table 25-1077&lt;/b&gt;).&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/e3c3d82a0d010bd96adfaadbf9984c49.png)

&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;Current evidence indicates that while different manuscript sections experience varying degrees of changes from their submitted, preprinted, or rejected versions to their peer-reviewed journal version, the primary conclusions rarely change. Our results in this stage are limited by not conducting a systematic search of bibliographic databases and lack of (identified) data points from many subdisciplines, especially arts and humanities, as well as studies that explored possible factors associated with the extent of changes (eg, researchers’ or editors’ career stage or prestige, quality of initial manuscript version).&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Tennant JP, Ross-Hellauer T. The limitations to our understanding of peer review. &lt;i&gt;Res Integr Peer Rev&lt;/i&gt;. 2020;5:6. doi:10.1186/s41073-020-00092-1&lt;/p&gt;
&lt;p&gt;2. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. &lt;i&gt;BMJ&lt;/i&gt;. 2021;372:n71. doi:10.1136/bmj.n71&lt;/p&gt;
&lt;p&gt;3. European Association of Science Editors. What is peer review? December 28, 2024. Accessed July 15, 2025. &lt;a href=&quot;https://ease.org.uk/communities/peer-review-committee/peer-review-toolkit/What-is-peer-review/&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://ease.org.uk/communities/peer-review-committee/peer-review-toolkit/What-is-peer-review/&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Stanford Program on Research Rigor and Reproducibility (SPORR), Stanford University, Stanford, California, US, mmalicki@stanford.edu; &lt;sup&gt;2&lt;/sup&gt;Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, California, US; &lt;sup&gt;3&lt;/sup&gt;Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, US; &lt;sup&gt;4&lt;/sup&gt;Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia; &lt;a id=&quot;_idTextAnchor069&quot;&gt;&lt;/a&gt;&lt;sup&gt;5&lt;/sup&gt;Urban Vitality Centre of Expertise, Amsterdam University of Applied Sciences, Amsterdam, The Netherlands; &lt;sup&gt;6&lt;/sup&gt;Amsterdam University Medical Centers, Department of Cardiology, Amsterdam, The Netherlands; &lt;sup&gt;7&lt;/sup&gt;Department of Philosophy, Faculty of Humanities, Vrije Universiteit, Amsterdam, The Netherlands; &lt;sup&gt;8&lt;/sup&gt;Amsterdam University Medical Centers, Department of Epidemiology and Data Science, Amsterdam, The Netherlands; &lt;a id=&quot;_idTextAnchor070&quot;&gt;&lt;/a&gt;&lt;sup&gt;9&lt;/sup&gt;Department of Medicine, Stanford University School of Medicine, Stanford, California, US&lt;a id=&quot;_idTextAnchor071&quot;&gt;&lt;/a&gt;; &lt;sup&gt;10&lt;/sup&gt;Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California, US; &lt;sup&gt;11&lt;/sup&gt;Independent scholar; &lt;sup&gt;12&lt;/sup&gt;Department of Health Policy, Stanford University School of Medicine, Stanford, California, US.&lt;/p&gt;
&lt;p class=&quot;abstract_conflict&quot; lang=&quot;fr-FR&quot;&gt;&lt;span class=&quot;text-bold&quot;&gt;Conflict of Interest Disclosures &lt;/span&gt;Mario Malički was a co–editor in chief of Research Integrity and Peer Review, in which 1 of the included studies was published (he was not involved in the processing or decision making for that study, and Research Integrity and Peer Review full peer review histories are available to the public on the journal website). Mario Malički was also an external reviewer for 1 study. Gerben ter Riet is a co-author of 1 of the included studies and Steven N. Goodman is a co-author of 1 of the included studies. In both of these cases, data extractions and primary data analyses were done by Mario Malički and Ana Jerončić. IJsbrand Jan Aalbersberg was a Senior Vice President of Research Integrity at Elsevier. John P. A. Ioannidis, Lex Bouter, and Steven N. Goodman are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract.&lt;/p&gt;
&lt;h4&gt;Funding/Support&lt;/h4&gt;
&lt;p&gt; The work on this project started in 2020, and in 2020 Elsevier funding was awarded to Stanford University for a 1-year METRICS postdoctoral position that supported Dr Malički’s work. From 2021 to 2025, Dr Malički’s work has been supported by the Stanford School of Medicine Research Office, which also supports the work of Dr Goodman. At the start of the project, IJsbrand Jan Aalbersberg was an employee of Elsevier.&lt;/p&gt;
&lt;h4&gt;Acknowledgments &lt;/h4&gt;
&lt;p&gt;In 2022’s Peer Review Congress, we presented preliminary findings (&lt;a href=&quot;https://peerreviewcongress.org/abstract/a-synthesis-of-studies-on-changes-manuscripts-underwent-between-submission-or-preprint-posting-and-peer-reviewed-journal-publication/&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://peerreviewcongress.org/abstract/a-synthesis-of-studies-on-changes-manuscripts-underwent-between-submission-or-preprint-posting-and-peer-reviewed-journal-publication/&lt;/span&gt;&lt;/a&gt;) that covered 25 studies published between 1990 and 2021. The current results include studies published before 1990 and up to January 2025.&lt;/p&gt;
&lt;h4&gt;Additional Information&lt;/h4&gt;
&lt;p&gt; A website with computationally reproducible results from this living review will be available soon.&lt;/p&gt;


The 10th Peer Review Congress

Differences Between Manuscript Versions: A Living Review and Series of Meta-Analyses

editorial and peer review process

preprints

peer review

<p>Mario Malički,<sup>1,2,3</sup> Ana Jerončić,<sup>4</sup> Gerben ter Riet,<sup>5,6</sup> Lex Bouter,<sup>7,8</sup> John P. A. Ioannidis,<sup>2,3,9,10</sup> IJsbrand Jan Aalbersberg,<sup>11 </sup>Steven N. Goodman<sup>1,2,3,9,12</sup> </p>
<h4>Objective </h4>
<p>Previous research has indicated a knowledge gap on changes brought on by peer review and journal publishing.<sup>1</sup> We are conducting a living evidence synthesis of studies that analyzed differences between manuscript versions (eg, preprinted or submitted versions vs peer reviewed journal versions).</p>
<h4>Design </h4>
<p>This is a living synthesis of studies with meta-analyses of proportions (following PRISMA guidelines for reporting).<sup>2</sup> Studies were identified based on authors’ knowledge of the field, from 18 previous systematic reviews on peer review,<sup>3</sup> and by checking all research presented at the peer review conferences. All references and citations of identified studies (using Google Scholar up to January 2025) were checked. For all included studies that reported on changes, we extracted descriptive variables (including year of publication, sampling method, change comparison method, and data, wherever available, on how many analyzed version-pairs had changes for 11 outcomes). We assessed risk of bias for each outcome based on 3 domains: sample selection, measurement reliability, and conflicts of interest. To pool the data, we used random effects meta-analysis with Freeman-Tukey transformed proportions and reported the <i>I</i><sup>2</sup> index of heterogeneity.</p>
<h4>Results </h4>
<p>We identified 67 studies published from 1978 through the end of 2024, of which 31 (46%) analyzed changes between preprint and journal versions, 23 (34%) between submitted and accepted or published versions, 8 (12%) between rejected versions and those later published in other journals, 4 (6%) between multiple version sets, and 1 (1.5%) between different preprinted versions. Most studies, 45 (67%), analyzed changes manually, 13 (19%) used computational methods, and 9 (12%) combined manual and computational methods. The median number of analyzed version-pairs was 113 (IQR, 51-429). By discipline, 43 (64%) studies looked only at health research, 6 (9%) at life sciences, 6 (9%) at social sciences, 4 (6%) at physical sciences, and 8 (12%) at multiple disciplines. Meta-analyses showed the highest frequency of changes in title, authorship, conflicts of interest, and numerical results, with the lowest frequency in study conclusions (<b>Table 25-1077</b>).</p>

![](https://assets.underline.io/markdown_image/1/image/e3c3d82a0d010bd96adfaadbf9984c49.png)

<h4>Conclusions </h4>
<p>Current evidence indicates that while different manuscript sections experience varying degrees of changes from their submitted, preprinted, or rejected versions to their peer-reviewed journal version, the primary conclusions rarely change. Our results in this stage are limited by not conducting a systematic search of bibliographic databases and lack of (identified) data points from many subdisciplines, especially arts and humanities, as well as studies that explored possible factors associated with the extent of changes (eg, researchers’ or editors’ career stage or prestige, quality of initial manuscript version).</p>
<h4>References</h4>
<p>1. Tennant JP, Ross-Hellauer T. The limitations to our understanding of peer review. <i>Res Integr Peer Rev</i>. 2020;5:6. doi:10.1186/s41073-020-00092-1</p>
<p>2. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. <i>BMJ</i>. 2021;372:n71. doi:10.1136/bmj.n71</p>
<p>3. European Association of Science Editors. What is peer review? December 28, 2024. Accessed July 15, 2025. <a href="https://ease.org.uk/communities/peer-review-committee/peer-review-toolkit/What-is-peer-review/"><span class="Hyperlink CharOverride-6">https://ease.org.uk/communities/peer-review-committee/peer-review-toolkit/What-is-peer-review/</span></a></p>
<sup>1</sup>Stanford Program on Research Rigor and Reproducibility (SPORR), Stanford University, Stanford, California, US, mmalicki@stanford.edu; <sup>2</sup>Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, California, US; <sup>3</sup>Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, US; <sup>4</sup>Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia; <a id="_idTextAnchor069"></a><sup>5</sup>Urban Vitality Centre of Expertise, Amsterdam University of Applied Sciences, Amsterdam, The Netherlands; <sup>6</sup>Amsterdam University Medical Centers, Department of Cardiology, Amsterdam, The Netherlands; <sup>7</sup>Department of Philosophy, Faculty of Humanities, Vrije Universiteit, Amsterdam, The Netherlands; <sup>8</sup>Amsterdam University Medical Centers, Department of Epidemiology and Data Science, Amsterdam, The Netherlands; <a id="_idTextAnchor070"></a><sup>9</sup>Department of Medicine, Stanford University School of Medicine, Stanford, California, US<a id="_idTextAnchor071"></a>; <sup>10</sup>Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California, US; <sup>11</sup>Independent scholar; <sup>12</sup>Department of Health Policy, Stanford University School of Medicine, Stanford, California, US.</p>
<p class="abstract_conflict" lang="fr-FR"><span class="text-bold">Conflict of Interest Disclosures </span>Mario Malički was a co–editor in chief of Research Integrity and Peer Review, in which 1 of the included studies was published (he was not involved in the processing or decision making for that study, and Research Integrity and Peer Review full peer review histories are available to the public on the journal website). Mario Malički was also an external reviewer for 1 study. Gerben ter Riet is a co-author of 1 of the included studies and Steven N. Goodman is a co-author of 1 of the included studies. In both of these cases, data extractions and primary data analyses were done by Mario Malički and Ana Jerončić. IJsbrand Jan Aalbersberg was a Senior Vice President of Research Integrity at Elsevier. John P. A. Ioannidis, Lex Bouter, and Steven N. Goodman are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract.</p>
<h4>Funding/Support</h4>
<p> The work on this project started in 2020, and in 2020 Elsevier funding was awarded to Stanford University for a 1-year METRICS postdoctoral position that supported Dr Malički’s work. From 2021 to 2025, Dr Malički’s work has been supported by the Stanford School of Medicine Research Office, which also supports the work of Dr Goodman. At the start of the project, IJsbrand Jan Aalbersberg was an employee of Elsevier.</p>
<h4>Acknowledgments </h4>
<p>In 2022’s Peer Review Congress, we presented preliminary findings (<a href="https://peerreviewcongress.org/abstract/a-synthesis-of-studies-on-changes-manuscripts-underwent-between-submission-or-preprint-posting-and-peer-reviewed-journal-publication/"><span class="Hyperlink CharOverride-6">https://peerreviewcongress.org/abstract/a-synthesis-of-studies-on-changes-manuscripts-underwent-between-submission-or-preprint-posting-and-peer-reviewed-journal-publication/</span></a>) that covered 25 studies published between 1990 and 2021. The current results include studies published before 1990 and up to January 2025.</p>
<h4>Additional Information</h4>
<p> A website with computationally reproducible results from this living review will be available soon.</p>


poster

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Melecia Miller,<sup>1</sup> <span lang="it-IT">Vera Nezgovorova,</span><sup>2</sup> Ilana Kersh,<sup>3</sup> Mohamed I. Elsaid,<sup>4</sup> Marina Broitman<sup>2</sup></p>
<h4>Objective</h4>
<p> This study’s goal was to develop a way to assess patient review quality based on author and editor reactions to reviews. We used patient reviews of draft final research reports (DFRRs) of Patient Centered Outcomes Research Institute (PCORI)–funded research that were peer reviewed by clinical, statistical, and patient reviewers.<sup>1</sup> Review quality ratings were based on prior research on PCORI peer review and other research on patient reviewers.<sup>2,3</sup> We hypothesized that specific, substantive changes to the report and patient-focused comments would be associated with a favorable mention of the patient review by the editor as well as a substantive response from the author.</p>
<h4>Design</h4>
<p> Pilot testing on 13 patient reviews for 13 clinical research DFRRs submitted from 2021 to 2022 was completed in 2025. We conducted a content analysis of patient reviewer comments and developed a coding scheme based on prior research. Coding was performed at the sentence level and included 2 categories: (1) specificity and constructiveness of the comment and (2) patient focus of the comment (<b>Table 25-1110</b>). Each review was coded independently by 2 raters each for category 1 and category 2, with a final Cohen κ of 0.83 and 0.66, respectively, indicating moderate to substantial interrater agreement. Coding discrepancies between reviewers received the most conservative (ie, lowest value) code. Raters assessed whether the editor mentioned the patient review at least once in their review summary and either encouraged (code 1) or discouraged (code 2) an author response vs no mention (code 0). Author responses to reviewer comments were coded by paragraph indicating whether authors responded directly to patient-reviewer comments (code 1) or made changes to the report because of the comments (code 2) vs neither (code 0). We used hierarchical modeling to calculate the odds of editors encouraging authors to respond (code 1 vs codes 0 or 2) and mixed-effects logistic regression to calculate the odds of authors responding substantively (codes 1 or 2 vs code 0) to patient reviews with more specific, constructive comments.</p>
<h4>Results</h4>
<p> <b>Table 25-1110</b> shows counts for associations between category 1 and category 2. More specific and constructive comments and patient-focused comments were not associated with higher odds of the editor mentioning the patient review (N = 13; odds ratio [OR], 0.36; 95% CI, 0.28-0.45 and OR, 0.65; 95% CI 0.54-0.78, respectively). Patient review paragraphs with more than 50% specific and constructive comments were associated with higher odds of authors responding substantively, compared with other comment types (N = 147; OR, 4.00; 95% CI, 1.67-9.59; <i>P</i> = .002).</p>
<a href="/wp-content/uploads/2025_conference/tables/prc25-1110t1.png"> <img loading="lazy" src="/wp-content/uploads/2025_conference/tables/prc25-1110t1.png"> </a>
<h4>Conclusions</h4>
<p> This pilot study shows the feasibility of establishing codes rating the quality of patient reviews and determining whether higher quality patient reviews are associated with mentions by the editor and author responses. The results are limited because of the sample size and the uniqueness of DFRRs.</p>
<h4>References</h4>
<p>1. Broitman M, Sox HC, Slutsky J. A model for public access to trustworthy and comprehensive reporting of research. <i>JAMA</i>. 2019;321(15):1453-1454. doi:10.1001/jama.2019.2807</p>
<p>2. Ghazaryan E, Broitman M, Sox H. Differences in the style and quantity of reviewer comments in structured vs unstructured peer review forms. Poster presented at: Ninth International Congress on Peer Review and Scientific Publication; September 10, 2022; Chicago, IL.</p>
<p>3. Huml AM, Albert JM, Beltran JM, et al. Community members as reviewers of medical journal manuscripts: a randomized controlled trial. <i>J Gen Intern Med</i>. 2022;38(6):1393-401. doi:10.1007/s11606-022-07802-z</p>
<sup>1</sup>KnowledgeWorks Global, Ltd., 410 MN-25, Brainerd, MN, US; <span lang="it-IT">mmiller@originreview.org; </span><sup>2</sup>Patient-Centered Outcomes Research Institute, Washington, DC, US; <sup>3</sup>University of Pittsburgh, Pittsburgh, PA, US; <sup>4</sup>The Ohio State University College of Medicine, Columbus, Ohio, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>


Quality of Patient Reviewer Comments and Association With Author and Editor Responses

<p>Matthew B. Stanbrook,<sup>1</sup> Shannon Charlebois,<sup>1</sup> George A. Tomlinson,<sup>1</sup> Meredith Weinhold<sup>1</sup></p>
<h4>Objective</h4>
<p> The opportunity to appeal a journal’s decision to reject a manuscript enhances procedural fairness toward authors and allows editors to correct errors in judgment but wastes time for both authors and editors when unsuccessful. We sought to identify factors associated with outcomes of appeals of research manuscripts at the <i>Canadian Medical Association Journal</i> (<i>CMAJ</i>), a general medical journal that explicitly highlights the option to appeal in rejection letters.</p>
<h4>Design</h4>
<p> We conducted a retrospective cohort study of all appeal requests for manuscripts initially rejected after submission to the Research section of <i>CMAJ</i> between 2007 and 2024. We obtained data on manuscript characteristics and editorial decisions from <i>CMAJ</i>’s manuscript submission database (ScholarOne). Three editors independently reviewed the text of the initial manuscript decision letters to identify reasons for rejection, with disagreements resolved by consensus. We included manuscripts rejected at any stage of the editorial process. The primary outcome was whether appeal requests were granted or denied. A secondary outcome was whether appealed manuscripts were ultimately published. We estimated associations of multiple prespecified characteristics with appeal outcomes using bayesian logistic regression.</p>
<h4>Results</h4>
<p> Among 14,351 submitted manuscripts, 13,419 were rejected, and 616 of these rejections (4.6%) were appealed. Among appeals, 214 (34.7%) were granted, of which 58 (27.1% of granted appeals and 9.4% of all appeals) were ultimately published. Compared with manuscripts rejected without external peer review, granted appeals were more likely for manuscripts rejected after peer review without editorial discussion (adjusted odds ratio [AOR], 2.71; 95% credible interval [CrI], 1.40-5.27) but similarly likely for manuscripts rejected after peer review and editorial discussion (AOR, 1.19; 95% CrI, 0.74-1.91) (<b>Table 25-1161</b>). Granted appeals were more likely if rejection letters included positive comments (AOR, 1.82; 95% CrI, 1.21-2.76) or cited lack of clarity (AOR, 1.84; 95% CrI, 1.12-3.04) and less likely if corresponding authors were based outside Canada (AOR, 0.38; 95% CrI, 0.24-0.59). The odds of an appeal being granted also varied significantly based on the individual editor making the appeal decision (median AOR, 1.90; 95% CrI, 1.24-4.37). Among factors associated with appeals being granted, only positive comments (AOR, 1.98; 95% CrI, 1.10-3.59) were also significantly associated with successful publication after appeal. However, appealed manuscripts rejected after peer review and editorial discussion were more likely to be published than were those rejected without peer review (AOR, 4.93; 95% CrI, 2.47-10.25).</p>

![](https://assets.underline.io/markdown_image/1/image/ca4a766cb379b500fce70628c15c12cb.png)

<h4>Conclusions</h4>
<p> Characteristics of authors, editors, editorial process, and decision communication were associated with successful appeals of initially rejected research manuscripts. Lack of awareness of the small fraction of appeals that ultimately achieve publication may raise false expectations among authors.</p>
<sup>1</sup><i>Canadian Medical Association Journal </i>(<i>CMAJ</i>), Toronto, Ontario, Canada, matthew.stanbrook@cmaj.ca.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> All authors are employees or paid consultants of CMAJ. Matthew B. Stanbrook, Shannon Charlebois, and George A. Tomlinson participated in editorial decision making for some of the manuscripts included in the study. No other disclosures were reported.</p>


Factors Associated With Outcomes of Appeals of Manuscripts Initially Rejected by a General Medical Journal

<p>Daniel S. Evanko,<sup>1</sup> Deondre Jordan<sup>1</sup></p>
<h4>Objective </h4>
<p>Journals ask authors to comply with many reporting requirements to improve study transparency, reproducibility, and reuse. Manual monitoring and enforcement of requirements is difficult, and reporting checklists or vendor-based solutions can be untargeted, burdensome, expensive, or of limited effectiveness. For example, since June 2020 the American Association for Cancer Research has used a methods review tool (SciScore) to help improve author compliance with reporting requirements at its 10 journals. This tool greatly increased author assignment of research resource identifiers (RRIDs),<sup>1</sup> but compliance plateaued at approximately 30% of original research articles published in the association journals. This cross-sectional study describes a simple, inexpensive, automated approach to further overcome limitations of common approaches and quantitatively evaluates its performance using 3 example interventions.</p>
<h4>Design </h4>
<p>Custom Python scripts coupled with structured language queries were used for targeted extraction of manuscript data from the EJournal Press journal submission system. For RRID reporting compliance, once every day the submission system was queried for manuscripts that completed initial quality control checks and did not contain the text RRID. A templated e-mail alert was sent to the corresponding author that explained what RRIDs are, why they are important, and how to add them. The system was also adapted with a custom supervised machine-learning method to identify studies that generated new sequencing data or that were computationally intensive to encourage authors to deposit sequence data and provide computer code used in their study and used to send different alerts for these manuscripts at different designated points in the editorial process.</p>
<h4>Results </h4>
<p>The new automated RRID process was activated August 6, 2024, at all the association journals, resulting in missing-RRID alerts sent for 84% (4949 of 5899) of manuscripts at completion of initial quality control checks through the end of 2024. Performance was assessed by searching for “RRID” in published articles as done previously.<sup>1</sup> An immediate sharp increase in articles containing RRIDs was observed that quickly increased the percentage of published research articles containing RRIDs from 33% (176 of 526) in April through July to 65% (330 of 505) in November through February <b>(Figure 25-1181</b>). Targeted alerts to authors of manuscripts asking them to deposit sequencing data (sent after peer review started) or supply code in a computational science software (Code Ocean) capsule (sent after a revise decision was sent to the author) were turned on on August 26, 2024, and resulted in sequence deposition alerts sent for 43% (1416 of 3261) of manuscripts and Code Ocean alerts sent for 39% (614 of 1594) of manuscripts through April 30, 2025. The sequencing alert triggered more than 100 author e-mail communications about deposition, demonstrating author engagement with this request and allowing resolution early in the editorial process. The percentage of accepted manuscripts with linked Code Ocean capsules increased from 1.4% (21 of 1555) in 2024 to 4.6% (25 of 547) in the January to May 2025 period, a more than 3-fold increase.</p>

![](https://assets.underline.io/markdown_image/1/image/87ac32761435115531ebf1a5f6ddec09.png)

<h4>Conclusions </h4>
<p>This fully automatic and highly targeted process engages with authors in a context-dependent way at predefined times in the editorial process that are appropriate for the specific communication, thus achieving substantial author engagement and improved compliance with journal requirements.</p>
<h4>Reference</h4>
<p>1. Roelandse M, Ozykurt IB, Evanko D, Bandrowski A. Assessing the effectiveness of SciScore in supporting the reproducibility of scientific research. <i>Sci Ed</i>. 2023;46:46-52.</p>
<sup>1</sup>American Association for Cancer Research, Philadelphia, PA, US, daniel.evanko@aacr.org.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Daniel S. Evanko is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>


Automated targeted emails for improving author compliance with study reporting requirements and other editorial processes

<p>Franki Y. H. Kung,<sup>1</sup> Mariam Aly,<sup>2</sup> Shahana Ansari,<sup>3</sup> Eliana Colunga,<sup>4</sup> M. J. Crockett,<sup>5,6</sup> Amanda B. Diekman,<sup>3</sup> Pablo Gomez,<sup>7</sup> Paul C. McKee,<sup>8</sup> Miriam Pérez,<sup>9</sup> Sarah M. Stilwell,<sup>10</sup> Matthew Goldrick<sup>11,12</sup></p>

<h4>Objective</h4> Accumulating data suggest peer reviews can exhibit bias<i> </i>(ie, systematically devaluing certain topics, methods, or the study of nondominant populations<sup>1</sup>) and can be unfair with respect to process (eg, decisions are made inconsistently) or people (eg, authors are disrespected).<sup>2</sup> Recent studies have shown that biases and unfair behaviors in peer review tend to undermine the self-reported experiences and career progression of researchers, including those from underserved populations, such as women, nonbinary individuals, and people from racial and ethnic groups other than White.<sup>1-3</sup> This study examined whether peer review exercises increased psychology and neuroscience trainees’ perceived readiness to respond to biased and unfair reviewers.</p>

<h4>Design </h4>Single-session synchronous online trainings (conducted during 2023-2024) allowed for trainer-trainee and trainee-trainee interactions. Trainees were recruited for this study through social media, targeted academic organizations, and graduate programs. Trainers (professors and a postdoctoral fellow with multiple years of peer review experience) reviewed processes occurring between journal submission and receipt of a decision letter. Trainees were then given a simplified, anonymized decision letter and reviews. Exercises and discussions illustrated how trainees could plan manuscript revisions and compose responses to critiques. Finally, trainees were given examples of biased and unfair reviews, along with strategies for responding to them while acknowledging the power dynamics inherent to review. Discussion included how to maintain motivation when encountering such reviews. Participants were asked to complete surveys before and after online training in which they rated their ability to perform the following targeted skills on a scale from 1 (very little skill) to 5 (great deal of skill): define and identify bias in peer reviews; define and identify unfair language in peer reviews; respond to reviews using fair language; respond to biased or unfair reviews; and promote an inclusive and equitable peer-review culture.</p>

<h4>Results </h4>A total of 64 participants (42 PhD students [66%], 10 predoctoral students [16%], 8 postdoctoral trainees [13%], and 4 other trainees [5%]) rated their level of skill for 5 targeted skills. Ordinal regressions showed participants had significantly higher ratings for all skills at posttest. For example, the mean (SD) self-ratings for the skill to respond to biased on unfair reviews increased from 2.0 (1.01) on the pretest to 4.09 (0.89) on the posttest (<b>Table 25-0893</b>).</p>

![](https://assets.underline.io/markdown_image/1/image/f801de28833ed0c4470dce4f2815a32b.png)

<h4>Conclusions </h4>
<p>After these peer review exercises, psychology and neuroscience trainees had an increase in their perceived readiness to confront biased and unfair reviews. Work is ongoing to scale up this training using on-demand materials and further assess training impact, including measures beyond trainee self-reports.</p>
<h4>References</h4>
<p>1. Rogers LO, Moffitt U, McLean KC, Syed M. Research as resistance: naming and dismantling the master narrative of “good” science. <i>Am Psychol</i>. 2024;79:484-496. doi:10.1037/amp0001246</p>
<p>2. Silbiger NJ, Stubler AD. Unprofessional peer reviews disproportionately harm underrepresented groups in STEM. <i>PeerJ</i>. 2019;7:e8247. doi:10.7717/peerj.8247</p>
<p>3. Aly M, Colunga E, Crockett MJ, et al. Changing the culture of peer review for a more inclusive and equitable psychological science. <i>J Exp Psychol Gen</i>. 2023;152:3546-3565. doi:10.1037/xge0001461</p>
<sup>1</sup>Department of Psychological Sciences, Purdue University, West Lafayette, IN, US; <sup>2</sup>Department of Psychology, University of California Berkeley, Berkeley, CA, US; <sup>3</sup>Department of Psychological and Brain Sciences, Indiana University Bloomington, Bloomington, IN, US; <sup>4</sup>Department of Psychology and Neuroscience, University of Colorado Boulder, City, CO, US; <sup>5</sup>Department of Psychology, Princeton University, Princeton, NJ, US; <sup>6</sup>Department of Psychology, University Center for Human Values, Princeton University, Princeton, NJ, US; <sup>7</sup>Department of Psychology, Skidmore College, Saratoga Springs, NY, US; <sup>8</sup>Department of Psychology and Neuroscience, Duke University, Durham, NC, US; <sup>9</sup>Department of Psychology, North Park University, Chicago, IL, US; <sup>10</sup>Department of Health Behavior and Health Equity, University of Michigan, Ann Arbor, MI, US; <sup>11</sup>Department of Linguistics, Northwestern University, Evanston, IL, US, matt-goldrick@northwestern.edu; <sup>12</sup>Cognitive Science Program, Northwestern University, Evanston, IL, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This work was supported by grants from the National Science Foundation (DGE-2224777, DGE-2224779, DGE-2436430). </p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funder of the study had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Additional Information</h4>
<p> Training materials are freely available (<a href="https://osf.io/658ev"><span class="Hyperlink CharOverride-6">https://osf.io/658ev</span></a>).</p>


Peer Review Exercises to Enhance Trainees' Readiness to Confront Unfair or Biased Reviews

<p>Christopher Baethge,<sup>1,2</sup>Hannah Jergas<sup>3</sup></p>
<h4>Objective</h4>
<p> Although quotations are central to scientific publications, they often misrepresent the cited reference.<sup>1</sup> Inaccurate quotations may mislead readers, undermine the line of argument of a paper, and threaten the integrity of the scientific record.<sup>2,3</sup> We updated an earlier meta-analysis<sup>1</sup> to assess improvement in recent years.</p>
<h4>Design</h4>
<p> This systematic review and meta-analysis, preregistered on Open Science Framework (OSF), was conducted following the Cochrane Handbook and PRISMA statement. It focuses on quotation inaccuracy—errors of content—as opposed to bibliographical errors, often referred to as citation errors. We searched MEDLINE, PubMed Central via PubMed, and Web of Science from January 1, 2014, through October 1, 2023, for studies on quotation errors in the medical literature, with no date, language, or source restrictions, and we hand-searched all reference lists of included studies. Literature screening, data extraction, and risk of bias estimation were carried out by 2 authors independently. We adopted the original studies’ definitions<sup>1</sup> of major inaccuracies (ie, not at all in accordance with the claim of the original authors), minor inaccuracies (ie, inconsistencies and factual errors not severe enough to contradict a statement by citing authors), secondary quotations (ie, quotations of sources referencing an original study), and total quotation errors. Summary proportions of quotation errors were estimated in random-effects meta-analyses (log-transformed proportions and DL τ<sup>2</sup> estimates), with heterogeneity measures, 95% confidence intervals, and 95% prediction intervals. Time trends were investigated in meta-regressions.</p>
<h4>Results</h4>
<p> Based on 46 studies included, with 32,074 quotations/references checked, 16.9% were incorrect (95% CI, 14.1%-20.0%). Of these, 8.0% (95% CI, 6.4%-10.0%) were major inaccuracies and 7.8% (95% CI, 5.7%-10.5%) were minor inaccuracies. Heterogeneity throughout all meta-analyses was high (<b>Table 24-0811</b>). Rates of total errors have not improved during the last decade (slope: −0.002; 95% CI, −0.03 to 0.02; <i>P</i> = .85) and neither have rates of major errors (slope: 0.002; 95% CI, −0.02 to 0.03; <i>P</i> = .89). There were no associations with risk of bias, publication bias, number of references, or medical specialty (surgical, nonsurgical, other), but the total error rate was statistically significantly (<i>P</i> = .04; binomial, n = 25) and negatively correlated with Journal Impact Factor (−0.25; Spearman, n = 23). Interrater reliability was sufficient (Cohen κ: 0.73), and summary estimates were supported by sensitivity analyses employing arcsine square root data transformation and Sidik and Jonkman τ<sup>2</sup> estimates.</p>

![](https://assets.underline.io/markdown_image/1/image/6088d5f3f296b8d165f7d47e3340260a.png)

<h4>Conclusions</h4>
<p> Quotation inaccuracy remains prevalent in scientific medical texts. Despite seemingly broad awareness among journal editors and probably among many authors, there is no evidence of improvement in recent years. It is uncertain how the utilization of artificial intelligence in writing articles and in double-checking quotation inaccuracy will impact the problem, but human effort will likely remain important in creating and vetting quotes—on the part of authors, editors, reviewers, and readers.</p>
<h4>References</h4>
<p>1. Jergas H, Baethge C. Quotation accuracy in medical journal articles—a systematic review and meta-analysis. <i>PeerJ</i>. Published online October 27, 2015. doi:10.7717/peerj.1364</p>
<p>2. Baethge C. Importance, errors, and patterns of quotations to psychiatric original articles. <i>Pharmacopsychiatry</i>. 2020;53(6):247-255. doi:10.1055/a-1167-35673</p>
<p>3. Peoples N, Østbye T, Yan LL. Burden of proof: combating inaccurate citation in biomedical literature. <i>BMJ</i>. Published online November 6, 2023. doi:10.1136/bmj-2023-076441</p>
<sup>1</sup><i>Deutsches Ärzteblatt </i>and<i> Deutsches Ärzteblatt International</i>, Editorial Offices, Cologne, Germany, <span lang="de-DE">baethge@aerzteblatt.de; </span><sup>2</sup>Department of Psychiatry and Psychotherapy, Faculty of Medicine, University of Cologne, Cologne, Germany; <sup>3</sup>Department of Neurology, Faculty of Medicine, University of Cologne, Cologne, Germany.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Both authors have published on this topic before. Christopher Baethge is employed by a general medical journal (<i>Deutsches Arzteblatt, Deutsches Arzteblatt International</i>).</p>
<h4>Acknowledgment</h4>
<p> The authors gratefully acknowledge Joan Albert Hammerstein’s help in screening the literature.</p>
<h4>Additional Information</h4>
<p> This study is registered on Open Science Framework (OSF) at <a href="https://osf.io/95rej/?view_only=8c5d2b51a8814278bff1fec1e40d0bb9"><span class="Hyperlink CharOverride-6">https://osf.io/95rej/?view_only=8c5d2b51a8814278bff1fec1e40d0bb9</span></a></p>


Quotation inaccuracy in medicine - A systematic review and meta-analysis

<p>Seba Qussini,<sup>1,2</sup> Farizah Mezer Anami,<sup>2</sup> Kris Dierickx<sup>1</sup></p>
<h4>Objective </h4>
<p>Many peer review attributes are widely criticized and remain poorly investigated, particularly in the context of proposals’ peer review.<sup>1</sup> This study aims to explore the diverse perspectives of stakeholders regarding the role of unmasking in funding proposals’ peer review and the implications of open peer review as part of the recent open science movement, specifically in biomedical research proposals’ peer review.</p>
<h4>Design </h4>
<p>To describe participants’ perspectives as constructed through their recent experiences, we have conducted a generic descriptive qualitative study<sup>2</sup> within a constructivist paradigm, using semistructured interviews to gather insights from reviewers, applicants, and peer review scholars. A total of 23 participants were selected through purposive and snowball sampling from funding agencies in Belgium and Qatar with whom no prior relationship had been established. Interviews were conducted between June 2024 and February 2025 by the first author (S.Q.). Transcribed interviews were analyzed according to the 6-step thematic framework analysis described by Braun and Clarke.<sup>3</sup> Initially, autogenerated transcripts were read and checked for in-depth familiarization with the data, which was followed by a line-by-line inductive coding, conducted iteratively after each set of 2 interviews. We followed the Consolidated Criteria for Reporting Qualitative Research (COREQ) guidelines.</p>
<h4>Results </h4>
<p>Codes with shared characteristics were grouped into categories, and ultimately 3 overarching themes were generated: (1) the importance of increased transparency in fund allocation procedures while maintaining anonymized reviewer identities; (2) open peer review as a feasible approach for enhancing transparency and accountability in funding proposals’ peer review; and (3) a growing critical stance toward traditional peer review systems, calling for alternative models like baseline or lottery funding procedures. Collectively, the results shed light on the perceived advantages and limitations of different peer review models and provide an understanding of how unmasking identities influences the fairness and objectivity of fund allocation decisions.</p>
<h4>Conclusions </h4>
<p>There is persistent preference for double-anonymous review among the scientific community; however, researchers are increasingly aware of the shortcoming of anonymized review, especially in light of current challenges within the funding landscape. Simultaneously, they recognize the importance of greater openness in peer review and increased transparency in fund allocation procedures.</p>
<h4>References</h4>
<p>1. Qussini S, MacDonald RS, Shahbal S, Dierickx K. Blinding models for scientific peer-review of biomedical research proposals: a systematic review. <i>J Empir Res Hum Res Ethics</i>. 2023;18(4):250-262. doi:10.1177/15562646231191424</p>
<p>2. Qussini S, Dierickx K. Multi-stakeholder perspectives on current attitudes toward (un)blinding reviewers’ identity in biomedical research proposals peer review: a qualitative study. OSF Registries. <a href="https://osf.io/dp65f"><span class="Hyperlink CharOverride-6">https://osf.io/dp65f</span></a></p>
<p>3. Clarke V, Braun V. Thematic analysis. <i>J Pos Psychol</i>. 2017;12(3):297-298. doi:10.1080/17439760.2016.1262613</p>
<sup>1</sup>Centre for Biomedical Ethics and Law, Faculty of Medicine, KU Leuven, Leuven, Belgium, seba.qussini@student.kuleuven.be; <sup>2</sup>Hamad Medical Corporation, The Medical Research Center, Doha, Qatar.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> The publication and conference fees will be funded by the Medical Research Center at Hamad Medical Corporation through grant MRC-01-24-305.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The Medical Research Center had no role in the design, analysis, or interpretation of this study.</p>


Multi-Stakeholder Perspectives on Current Attitudes Toward Unmasking Reviewers' Identity in Biomedical Research Proposals Peer Review: A Qualitative Study

<p>Jong-Wook Ban,<sup>1</sup> Hans Lund,<sup>2</sup> Karen Robinson,<sup>3</sup> Ida Svege,<sup>4</sup> Jan-Ole Hesselberg<sup>5,6</sup></p>
<h4>Objective</h4>
<p> Many have argued that the need for new research should be determined through a systematic examination of existing evidence to maximize its value.<sup>1,2</sup> Our study evaluated the outcomes of following such arguments on funding applications. We tested a hypothesis that funding applications justifying new research with a systematic review would receive higher scores than those without such justification.</p>
<h4>Design</h4>
<p> We analyzed funding applications submitted to Stiftelsen Dam, one of Norway’s largest health foundations, during the 2024 cycle. We used multiple regression to test our hypothesis while controlling for the influence of the following 5 covariables: requested funding amount, principal investigator’s birth year, principal investigator’s gender, project’s health domain, and research methods. </p>
<h4>Results</h4>
<p> Of 420 applications received, 193 (46.0%) cited a systematic review, 181 (43.1%) referenced a published or unpublished study, and 46 (11.0%) referred to applicants’ own experience as justification for new research. The median requested funding amount was kr3,000,000 ($267,796), with an IQR of kr2,801,000 to kr3,000,000. There were 158 (37.6%) men and 247 (58.8%) women principal investigators, and 15 (3.6%) who did not disclose their gender. The median birth year of the principal investigators was 1976 (IQR, 1968-1982). Funding applications included research projects in mental health (122 [29.0%]), physical health (197 [46.9%]), well-being (42 [10.0%]), disability (18 [4.3%]), and other domains (41 [9.8%]). Qualitative, mixed, and quantitative methods were used in 53 (12.6%), 145 (34.5%), and 222 (52.9%) applications, respectively. The median funding application score was 4.8 (IQR, 4.4-5.1) on a 7-point scale, where 1 represents poor and 7 represents excellent. Applications that cited a systematic review as justification received scores 0.19 (95% CI, 0.03-0.35) points higher than those that relied on the applicants’ own experience (<b>Figure 25-1129</b>). This could have raised an application with an average score by approximately 100 places in a ranking of 1716 applications. Applications that cited a systematic review received scores 0.12 (95% CI, 0.02-0.22) points higher than those that cited other types of evidence. </p>

![](https://assets.underline.io/markdown_image/1/image/679eec507dc3d8912ae0b0169dacb029.png)

<h4>Conclusions</h4>
<p> Our exploratory analysis showed an association between citing a systematic review and higher funding application scores. Our study has the following limitations. First, we could only control for variables available in our data. So, it was impossible to exclude the influence of other potential confounders, such as study design. Second, we evaluated funding application scores rather than funding outcomes. We could not evaluate the funding outcomes due to the very small number of applications receiving funding each year. Because Stiftelsen Dam heavily relies on funding application scores to decide on funding, we used those scores as a surrogate for funding outcomes.</p>
<h4>References</h4>
<p>1. Chalmers I, Bracken MB, Djulbegovic B, et al. How to increase value and reduce waste when research priorities are set. <i>Lancet</i>. 2014;383(9912):156-165. doi:10.1016/S0140-6736(13)62229-1</p>
<p>2. Lund H, Brunnhuber K, Juhl C, et al. Towards evidence based research. <i>BMJ</i>. 2016;355:i5440. doi:10.1136/bmj.i5440</p>
<sup>1</sup>Section of Hospital Medicine, University of Chicago, Chicago, IL, US, jban@uchicago.edu; <sup>2</sup>Section of Evidence-Based Practice, Western Norway University of Applied Sciences, Bergen, Norway; <sup>3</sup>Johns Hopkins Evidence-Based Practice Center, Johns Hopkins University, Baltimore, MD, US; <sup>4</sup>Nordic Institute for Studies in Innovation, Research and Education, Oslo, Norway; <sup>5</sup>Stiftelsen Dam, Oslo, Norway; <sup>6</sup>Department of Psychology, University of Oslo, Oslo, Norway.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Ida Svege reported being the former Head of Program Development (Leder for programutvikling) of Stiftelsen Dam. Jan-Ole Hesselberg reported being the current Program Manager (Programsjef) of Stiftelsen Dam. </p>


The influence of using a systematic review to justify new research on funding application score

<p>Markus K. Heinemann,<sup>1</sup><span lang="de-DE"> </span>Andreas Boening,<sup>2</sup><span lang="de-DE"> Kazunori Okabe,</span><sup>3</sup><span lang="de-DE"> Jessica Bogensberger</span><sup>4</sup></p>
<h4>Objective</h4>
<p> In 2013, a cardiothoracic surgical journal introduced plagiarism detection (Crossref, powered by iThenticate). After a phase of 2 years during which all submissions were screened, since January 2015 only returned manuscripts after revision were checked by default. Primary submissions were then screened at the discretion of the Editor in Chief (EiC) only. The aim of this study was to report the 12-year experience.</p>
<h4>Design</h4>
<p> All manuscripts submitted between January 1, 2013, and December 31, 2024, and branded by the plagiarism search engine with a similarity index above 33% were analyzed regarding the originator of the screening (EiC or revision default), reasons (actual plagiarism or self-plagiarism, which is a form of duplication), similarity percentage, and final decision. Descriptive statistics were applied. All updates of the search tool were implemented.</p>
<h4>Results</h4>
<p> During the investigation period, 4083 manuscripts were submitted. A total of 99 (2.4%) showed a similarity index greater than 33%. The mean percentage was 53.3% (minimum, 34%; maximum, 99%; median, 50%; SD, ±14.8). Plagiarism and duplication search was prompted by the EiC in 48 papers and by automated default in 51 papers. Self-plagiarized or duplicated content was by the same authors in 71 of 99 papers (71.7%), by different authors in 18 (18.2%), and both in 10 (10.1%). Fifty-five manuscripts (55.6%) remained unpublished: 17 rejected without review, 34 rejected, and 4 withdrawn. Forty-four of 99 (44.4%) were accepted after revisions. There were 4 attempts at duplicate publication, all rejected. Preprints were discovered in 3 cases, with 1 accepted after revision: the preprint had been uploaded by a different journal for review and was rejected there, but then not deleted. The 44 manuscripts finally accepted contained previously published sequences by the same authors in 36, by different authors in 4, and both in 4. In 25 cases, the EiC rendered the similarity uncritical (eg, materials and methods section). The most frequent necessary correction was adequate citation. No adequate citation was given for self-plagiarized/duplicated content in 23 cases and for content by different authors in 4.</p>
<h4>Conclusions</h4>
<p> Although the overall rate of plagiarism and duplication can be considered low, scientific misconduct was detected and its publication prevented in a few cases. In many instances, duplicated passages contained similar material and methods previously published by the same group. Deficiencies in adequate citation are a problem and necessitate continued education of authors. Selective plagiarism and duplication screening remains an important tool, although chosen thresholds of similarity remain arbitrary. Further improvements of sensitivity are expected, especially with the implementation of machine learning advances. Editorial decisions, however, cannot be made on automated findings alone, but must be put into personal perspective.</p>
<sup>1</sup>German Society for Thoracic and Cardiovascular Surgery (DGTHG), Germany, heinemann@uni-mainz.de; <sup>2</sup><span lang="de-DE">Universitaetsklinik Giessen, Germany; </span><sup>3</sup><span lang="de-DE">Bell Land General Hospital, Osaka, Japan; </span><sup>4</sup>Thieme Publishers, Stuttgart, Germany.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Jessica Bogensberger is an employee of Thieme Publishers, the publishing house of the journal investigated. All other authors have no conflict of interest to declare.</p>


Experience With 12 Years of Plagiarism Screening

<p>Jhih-Yi Hsieh,<sup>1</sup> Aditi Raghunathan,<sup>1</sup> Nihar B. Shah<sup>1</sup></p>
<h4>Objective </h4>
<p>Collusion rings—researchers who manipulate reviewer assignments to review each other’s work—pose major challenges for machine learning and artificial intelligence conferences. As submissions grow substantially, automated reviewer assignment algorithms<sup>1</sup> have become common, often relying on the text similarity of reviewers’ past papers and the author’s submission.<sup>2</sup> While this text similarity is generally considered collusion safe,<sup>3</sup> this work investigated potential manipulations and proposes mitigations.</p>
<h4>Design </h4>
<p>For a colluding author and reviewer, we designed a realistic 2-step attack procedure: (1) reviewer profile curation, in which the colluding reviewer kept only 1 past paper that was the most similar to the author’s submission for text matching; and (2) submission abstract modification, in which the colluding author modified their submission’s abstract by adding background sentences related to the themes in the reviewer’s profile and inserting keywords that increased the text similarity to the reviewer. For scalability, we used a large language model (gpt-4-0125-preview [OpenAI]) to modify abstracts, although colluding authors in practice could manually refine them to reduce suspicion. We curated a dataset (3123 papers; 7900 reviewers) of the NeurIPS 2023 conference, which reviewed full papers (not abstracts). We then sampled reviewer-paper pairs where the reviewer initially ranked 20th, 101st, 501st, or 1001st among all reviewers by similarity, simulated collusion via our attack, and reported if the reviewer became highly ranked afterward. We proposed 2 hypotheses to improve robustness. (1) Requiring more past papers in reviewer profiles reduces attack success. (2) When a reviewer has multiple past papers in their profile, the similarities between each past paper and the submission are first calculated, then aggregated by taking the mean or maximum. We hypothesized that mean aggregation is more robust than maximum because averaging reduces manipulation effects. We tested these hypotheses via 2 ablations: (1) enforcing 10 papers in reviewer profiles and (2) comparing mean vs maximum under attack. Lastly, a randomized controlled trial tested for the identifiability of adversarial abstracts by unsuspecting human reviewers. Participants were randomly assigned either the benign or adversarial abstract to review.</p>
<h4>Results </h4>
<p>The attack was highly effective (<b>Table 25-0911</b>); 67% of reviewers initially ranked 1001st became top 5, despite not working on similar topics. Both hypotheses on robustness were confirmed. When reviewers selected 10 past papers (100 samples), only 19% of similarity rankings were in the top 5 after the attack. For mean vs maximum aggregation (100 samples each), 32% and 49% of rankings were in the top 5 after attack, respectively. Lastly, in the randomized controlled trial (116 samples), while participants had more complaints for manipulated papers, surprisingly, benign papers elicited similar complaints, suggesting plausible deniability.</p>

![](https://assets.underline.io/markdown_image/1/image/899f0f6d7dd7d88e89c0954872b543ac.png)

<h4>Conclusions </h4>
<p>This study found that conference reviewer assignments based on text matching are vulnerable to colluders and identified methods to improve robustness.</p>
<h4>References</h4>
<p>1. Shah NB. Challenges, experiments, and computational solutions in peer review. <i>Commun ACM</i>. June 2022. Accessed July 11, 2025. <a href="https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf"><span class="Hyperlink CharOverride-6">https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf</span></a></p>
<p>2. Cohan A, Feldman S, Beltagy I, Downey D, Weld D. SPECTER: document-level representation learning using citation-informed transformers. In: <i>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</i>. 2020, 2270-2282. doi:10.18653/v1/2020.acl-main.20</p>
<p>3. Wu R, Guo C, Wu F, Kidambi R, Van Der Maaten L, Weinberger K. Making paper reviewing robust to bid manipulation attacks. In: <i>International Conference on Machine Learning</i>. 2021: 11240-11250.</p>
<sup>1</sup>Carnegie Mellon University, Pittsburgh, PA, US, jhihyi.hsieh@gmail.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Jhih-Yi Hsieh, Aditi Raghunathan, and Nihar B. Shah are employees of Carnegie Mellon University. Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>
<h4>Funding/Support </h4>
<p>This study was funded by the J.P. Morgan Research Scholar Program (ONR N000142212181, NSF 2200410, 1942124, 2310758), the AI2050 program at Schmidt Sciences (G2264481), the Google Research Scholar Program, Apple, Cisco, and Open Philanthropy.</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funders played no role in the design and execution of the experiment, data analysis, or preparation of the abstract.</p>
<h4>Additional Information </h4>
<p>gpt-4-0125-preview (OpenAI) was used between April and September 2024 to generate modified abstracts without human supervision. Jhih-Yi Hsieh takes responsibility for the integrity of the content generated.</p>


Vulnerability of Automated Text-Matching-Based Reviewer Assignments to Collusions

<p>Cyril Labbé,<sup>1</sup> Qinyue Liu,<sup>1</sup> Amira Barhoumi,<sup>1</sup> Olessya Miroshnichenko<sup>1</sup></p>
<h4>Objective</h4>
<p> For various reasons, duplicate publications are often considered problematic.<sup>1,2</sup> We tested a method to semiautomatically detect such publications from publicly available metadata registered with Crossref by publisher.<sup>3</sup></p>
<h4>Design</h4>
<p> We first queried Crossref using ISSNs or ISBNs to build a set of (random) digital object identifiers (DOIs) with abstracts for duplicates to be searched. For each DOI, Crossref was queried again to retrieve a new set of 200 DOIs with similar titles containing potential duplicates of publications in the first set. We then identified abstracts in the second set that were nearly identical to those in the first set, differing by only a few words. When DOIs of apparent duplicates were not registered by preprint platforms (eg, bioRxiv, medRxiv), we would then automatically analyze when DOIs were registered by a different publisher, had differences in authorship, or were published in journals or books. The procedure was applied to the following sets of DOIs chosen to represent a wide variety of publishers for which abstracts are registered with Crossref: <i>Science</i> publications (2023-2024), <i>PLOS One</i> (2023-2024), <i>International Journal of Molecular Sciences</i> (2024), <i>BioMed Research International</i> (2024), <i>Scientific Reports</i> (2024), and IGI Global (DOIs included in books published in 2024).</p>
<h4>Results</h4>
<p> The vast majority of duplicates found were not problematic, as they typically originated from preprint versions (<b>Table 25-0986</b>). For example, of 375 duplicated abstracts for articles published in <i>PLOS One</i>, 4 were not preprint publications. Moreover, close inspection revealed that 3 duplicates were republication of abstracts only. Only the 1 remaining abstract could be seen as problematic, as it was a duplication of the same exact content in 2 different journals, both open access.</p>

![](https://assets.underline.io/markdown_image/1/image/e018f0b6fdc9855b01f8f955fd195208.png)

<h4>Conclusions</h4>
<p> One limitation of the process is that despite having the exact same abstract, the body of the publications might be different. For many publications, abstracts are not registered with Crossref or sometimes are registered with errors. It is also difficult to identify when one of the DOIs represents a republication with an abstract alone. Abstracts publicly available at Crossref can be used to detect problematic duplicate publications as a cost-effective alternative to plagiarism detection tools. This experiment emphasizes the importance for publishers to answer positively to the Initiative for Open Abstracts (I4OA), which calls on all scholarly publishers to<span lang="fr-FR"> open the abstracts of their</span> published works and, where possible, to submit them to Crossref.</p>
<h4>References</h4>
<p>1. Tramèr MR, Reynolds DJ, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: a case study. <i>BMJ</i>. 1997;315(7109):635-640. doi:10.1136/bmj.315.7109.635</p>
<p>2. Errami M, Hicks JM, Fisher W, et al. Déjà vu—a study of duplicate citations in Medline. <i>Bioinformatics</i>. 2008;24(2):243-249. doi:10.1093/bioinformatics/btm574</p>
<p>3. van Eck NJ, Waltman L. Crossref as a source of open bibliographic metadata. <i>MetaArXiv</i>. Preprint posted online May 12, 2025. doi:10.31222/osf.io/smxe5</p>
<sup>1</sup>Université Grenoble Alpes, French National Centre for Scientific Research, Grenoble INP, Laboratoire d’Informatique de Grenoble, Grenoble, France, cyril.labbe@imag.fr.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> We acknowledge the NanoBubbles project that has received Synergy grant funding from the European Research Council within the European Union’s Horizon 2020 program (grant agreement 951393).</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funders had no role in this research.</p>


Next from The 10th Peer Review Congress

Quality of Patient Reviewer Comments and Association With Author and Editor Responses

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES