United States

&lt;p&gt;Jhih-Yi Hsieh,&lt;sup&gt;1&lt;/sup&gt; Aditi Raghunathan,&lt;sup&gt;1&lt;/sup&gt; Nihar B. Shah&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;Collusion rings—researchers who manipulate reviewer assignments to review each other’s work—pose major challenges for machine learning and artificial intelligence conferences. As submissions grow substantially, automated reviewer assignment algorithms&lt;sup&gt;1&lt;/sup&gt; have become common, often relying on the text similarity of reviewers’ past papers and the author’s submission.&lt;sup&gt;2&lt;/sup&gt; While this text similarity is generally considered collusion safe,&lt;sup&gt;3&lt;/sup&gt; this work investigated potential manipulations and proposes mitigations.&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;For a colluding author and reviewer, we designed a realistic 2-step attack procedure: (1) reviewer profile curation, in which the colluding reviewer kept only 1 past paper that was the most similar to the author’s submission for text matching; and (2) submission abstract modification, in which the colluding author modified their submission’s abstract by adding background sentences related to the themes in the reviewer’s profile and inserting keywords that increased the text similarity to the reviewer. For scalability, we used a large language model (gpt-4-0125-preview [OpenAI]) to modify abstracts, although colluding authors in practice could manually refine them to reduce suspicion. We curated a dataset (3123 papers; 7900 reviewers) of the NeurIPS 2023 conference, which reviewed full papers (not abstracts). We then sampled reviewer-paper pairs where the reviewer initially ranked 20th, 101st, 501st, or 1001st among all reviewers by similarity, simulated collusion via our attack, and reported if the reviewer became highly ranked afterward. We proposed 2 hypotheses to improve robustness. (1) Requiring more past papers in reviewer profiles reduces attack success. (2) When a reviewer has multiple past papers in their profile, the similarities between each past paper and the submission are first calculated, then aggregated by taking the mean or maximum. We hypothesized that mean aggregation is more robust than maximum because averaging reduces manipulation effects. We tested these hypotheses via 2 ablations: (1) enforcing 10 papers in reviewer profiles and (2) comparing mean vs maximum under attack. Lastly, a randomized controlled trial tested for the identifiability of adversarial abstracts by unsuspecting human reviewers. Participants were randomly assigned either the benign or adversarial abstract to review.&lt;/p&gt;
&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;The attack was highly effective (&lt;b&gt;Table 25-0911&lt;/b&gt;); 67% of reviewers initially ranked 1001st became top 5, despite not working on similar topics. Both hypotheses on robustness were confirmed. When reviewers selected 10 past papers (100 samples), only 19% of similarity rankings were in the top 5 after the attack. For mean vs maximum aggregation (100 samples each), 32% and 49% of rankings were in the top 5 after attack, respectively. Lastly, in the randomized controlled trial (116 samples), while participants had more complaints for manipulated papers, surprisingly, benign papers elicited similar complaints, suggesting plausible deniability.&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/899f0f6d7dd7d88e89c0954872b543ac.png)

&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;This study found that conference reviewer assignments based on text matching are vulnerable to colluders and identified methods to improve robustness.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Shah NB. Challenges, experiments, and computational solutions in peer review. &lt;i&gt;Commun ACM&lt;/i&gt;. June 2022. Accessed July 11, 2025. &lt;a href=&quot;https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;2. Cohan A, Feldman S, Beltagy I, Downey D, Weld D. SPECTER: document-level representation learning using citation-informed transformers. In: &lt;i&gt;Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics&lt;/i&gt;. 2020, 2270-2282. doi:10.18653/v1/2020.acl-main.20&lt;/p&gt;
&lt;p&gt;3. Wu R, Guo C, Wu F, Kidambi R, Van Der Maaten L, Weinberger K. Making paper reviewing robust to bid manipulation attacks. In: &lt;i&gt;International Conference on Machine Learning&lt;/i&gt;. 2021: 11240-11250.&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Carnegie Mellon University, Pittsburgh, PA, US, jhihyi.hsieh@gmail.com.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt;
&lt;p&gt; Jhih-Yi Hsieh, Aditi Raghunathan, and Nihar B. Shah are employees of Carnegie Mellon University. Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.&lt;/p&gt;
&lt;h4&gt;Funding/Support &lt;/h4&gt;
&lt;p&gt;This study was funded by the J.P. Morgan Research Scholar Program (ONR N000142212181, NSF 2200410, 1942124, 2310758), the AI2050 program at Schmidt Sciences (G2264481), the Google Research Scholar Program, Apple, Cisco, and Open Philanthropy.&lt;/p&gt;
&lt;h4&gt;Role of the Funder/Sponsor &lt;/h4&gt;
&lt;p&gt;The funders played no role in the design and execution of the experiment, data analysis, or preparation of the abstract.&lt;/p&gt;
&lt;h4&gt;Additional Information &lt;/h4&gt;
&lt;p&gt;gpt-4-0125-preview (OpenAI) was used between April and September 2024 to generate modified abstracts without human supervision. Jhih-Yi Hsieh takes responsibility for the integrity of the content generated.&lt;/p&gt;


The 10th Peer Review Congress

Vulnerability of Automated Text-Matching-Based Reviewer Assignments to Collusions

misconduct

peer review

research integrity

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Jhih-Yi Hsieh,<sup>1</sup> Aditi Raghunathan,<sup>1</sup> Nihar B. Shah<sup>1</sup></p>
<h4>Objective </h4>
<p>Collusion rings—researchers who manipulate reviewer assignments to review each other’s work—pose major challenges for machine learning and artificial intelligence conferences. As submissions grow substantially, automated reviewer assignment algorithms<sup>1</sup> have become common, often relying on the text similarity of reviewers’ past papers and the author’s submission.<sup>2</sup> While this text similarity is generally considered collusion safe,<sup>3</sup> this work investigated potential manipulations and proposes mitigations.</p>
<h4>Design </h4>
<p>For a colluding author and reviewer, we designed a realistic 2-step attack procedure: (1) reviewer profile curation, in which the colluding reviewer kept only 1 past paper that was the most similar to the author’s submission for text matching; and (2) submission abstract modification, in which the colluding author modified their submission’s abstract by adding background sentences related to the themes in the reviewer’s profile and inserting keywords that increased the text similarity to the reviewer. For scalability, we used a large language model (gpt-4-0125-preview [OpenAI]) to modify abstracts, although colluding authors in practice could manually refine them to reduce suspicion. We curated a dataset (3123 papers; 7900 reviewers) of the NeurIPS 2023 conference, which reviewed full papers (not abstracts). We then sampled reviewer-paper pairs where the reviewer initially ranked 20th, 101st, 501st, or 1001st among all reviewers by similarity, simulated collusion via our attack, and reported if the reviewer became highly ranked afterward. We proposed 2 hypotheses to improve robustness. (1) Requiring more past papers in reviewer profiles reduces attack success. (2) When a reviewer has multiple past papers in their profile, the similarities between each past paper and the submission are first calculated, then aggregated by taking the mean or maximum. We hypothesized that mean aggregation is more robust than maximum because averaging reduces manipulation effects. We tested these hypotheses via 2 ablations: (1) enforcing 10 papers in reviewer profiles and (2) comparing mean vs maximum under attack. Lastly, a randomized controlled trial tested for the identifiability of adversarial abstracts by unsuspecting human reviewers. Participants were randomly assigned either the benign or adversarial abstract to review.</p>
<h4>Results </h4>
<p>The attack was highly effective (<b>Table 25-0911</b>); 67% of reviewers initially ranked 1001st became top 5, despite not working on similar topics. Both hypotheses on robustness were confirmed. When reviewers selected 10 past papers (100 samples), only 19% of similarity rankings were in the top 5 after the attack. For mean vs maximum aggregation (100 samples each), 32% and 49% of rankings were in the top 5 after attack, respectively. Lastly, in the randomized controlled trial (116 samples), while participants had more complaints for manipulated papers, surprisingly, benign papers elicited similar complaints, suggesting plausible deniability.</p>

![](https://assets.underline.io/markdown_image/1/image/899f0f6d7dd7d88e89c0954872b543ac.png)

<h4>Conclusions </h4>
<p>This study found that conference reviewer assignments based on text matching are vulnerable to colluders and identified methods to improve robustness.</p>
<h4>References</h4>
<p>1. Shah NB. Challenges, experiments, and computational solutions in peer review. <i>Commun ACM</i>. June 2022. Accessed July 11, 2025. <a href="https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf"><span class="Hyperlink CharOverride-6">https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf</span></a></p>
<p>2. Cohan A, Feldman S, Beltagy I, Downey D, Weld D. SPECTER: document-level representation learning using citation-informed transformers. In: <i>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</i>. 2020, 2270-2282. doi:10.18653/v1/2020.acl-main.20</p>
<p>3. Wu R, Guo C, Wu F, Kidambi R, Van Der Maaten L, Weinberger K. Making paper reviewing robust to bid manipulation attacks. In: <i>International Conference on Machine Learning</i>. 2021: 11240-11250.</p>
<sup>1</sup>Carnegie Mellon University, Pittsburgh, PA, US, jhihyi.hsieh@gmail.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Jhih-Yi Hsieh, Aditi Raghunathan, and Nihar B. Shah are employees of Carnegie Mellon University. Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>
<h4>Funding/Support </h4>
<p>This study was funded by the J.P. Morgan Research Scholar Program (ONR N000142212181, NSF 2200410, 1942124, 2310758), the AI2050 program at Schmidt Sciences (G2264481), the Google Research Scholar Program, Apple, Cisco, and Open Philanthropy.</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funders played no role in the design and execution of the experiment, data analysis, or preparation of the abstract.</p>
<h4>Additional Information </h4>
<p>gpt-4-0125-preview (OpenAI) was used between April and September 2024 to generate modified abstracts without human supervision. Jhih-Yi Hsieh takes responsibility for the integrity of the content generated.</p>


poster

<p>Cyril Labbé,<sup>1</sup> Qinyue Liu,<sup>1</sup> Amira Barhoumi,<sup>1</sup> Olessya Miroshnichenko<sup>1</sup></p>
<h4>Objective</h4>
<p> For various reasons, duplicate publications are often considered problematic.<sup>1,2</sup> We tested a method to semiautomatically detect such publications from publicly available metadata registered with Crossref by publisher.<sup>3</sup></p>
<h4>Design</h4>
<p> We first queried Crossref using ISSNs or ISBNs to build a set of (random) digital object identifiers (DOIs) with abstracts for duplicates to be searched. For each DOI, Crossref was queried again to retrieve a new set of 200 DOIs with similar titles containing potential duplicates of publications in the first set. We then identified abstracts in the second set that were nearly identical to those in the first set, differing by only a few words. When DOIs of apparent duplicates were not registered by preprint platforms (eg, bioRxiv, medRxiv), we would then automatically analyze when DOIs were registered by a different publisher, had differences in authorship, or were published in journals or books. The procedure was applied to the following sets of DOIs chosen to represent a wide variety of publishers for which abstracts are registered with Crossref: <i>Science</i> publications (2023-2024), <i>PLOS One</i> (2023-2024), <i>International Journal of Molecular Sciences</i> (2024), <i>BioMed Research International</i> (2024), <i>Scientific Reports</i> (2024), and IGI Global (DOIs included in books published in 2024).</p>
<h4>Results</h4>
<p> The vast majority of duplicates found were not problematic, as they typically originated from preprint versions (<b>Table 25-0986</b>). For example, of 375 duplicated abstracts for articles published in <i>PLOS One</i>, 4 were not preprint publications. Moreover, close inspection revealed that 3 duplicates were republication of abstracts only. Only the 1 remaining abstract could be seen as problematic, as it was a duplication of the same exact content in 2 different journals, both open access.</p>

![](https://assets.underline.io/markdown_image/1/image/e018f0b6fdc9855b01f8f955fd195208.png)

<h4>Conclusions</h4>
<p> One limitation of the process is that despite having the exact same abstract, the body of the publications might be different. For many publications, abstracts are not registered with Crossref or sometimes are registered with errors. It is also difficult to identify when one of the DOIs represents a republication with an abstract alone. Abstracts publicly available at Crossref can be used to detect problematic duplicate publications as a cost-effective alternative to plagiarism detection tools. This experiment emphasizes the importance for publishers to answer positively to the Initiative for Open Abstracts (I4OA), which calls on all scholarly publishers to<span lang="fr-FR"> open the abstracts of their</span> published works and, where possible, to submit them to Crossref.</p>
<h4>References</h4>
<p>1. Tramèr MR, Reynolds DJ, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: a case study. <i>BMJ</i>. 1997;315(7109):635-640. doi:10.1136/bmj.315.7109.635</p>
<p>2. Errami M, Hicks JM, Fisher W, et al. Déjà vu—a study of duplicate citations in Medline. <i>Bioinformatics</i>. 2008;24(2):243-249. doi:10.1093/bioinformatics/btm574</p>
<p>3. van Eck NJ, Waltman L. Crossref as a source of open bibliographic metadata. <i>MetaArXiv</i>. Preprint posted online May 12, 2025. doi:10.31222/osf.io/smxe5</p>
<sup>1</sup>Université Grenoble Alpes, French National Centre for Scientific Research, Grenoble INP, Laboratoire d’Informatique de Grenoble, Grenoble, France, cyril.labbe@imag.fr.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> We acknowledge the NanoBubbles project that has received Synergy grant funding from the European Research Council within the European Union’s Horizon 2020 program (grant agreement 951393).</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funders had no role in this research.</p>


Identifying Potential Duplicate Publications in the Scientific Literature Using Crossref

<p>Wendeline Swart,<sup>1</sup> Ophélie Fraisier-Vannier,<sup>1</sup> Guillaume Cabanac<sup>1,2</sup></p>
<h4>Objective </h4>
<p>Tortured phrases<sup>1</sup> in scientific articles, such as <i>amino corrosive</i> instead of <i>amino acid</i>, are potential signals of scientific misconduct. They alter established expressions using synonyms to avoid plagiarism detection. We highlighted the presence of tortured phrases in 1323 conference proceedings and identified the most impacted conference.</p>
<h4>Design </h4>
<p>We used the Problematic Paper Screener (PPS),<sup>2</sup> a tool we developed to detect research integrity issues, to identify conference proceedings published containing tortured articles, defined as articles with 5 or more tortured phrases or fewer than 5 tortured phrases but assessed as problematic by a human. We selected the conference with the largest number of tortured articles and probed with Dimensions (a bibliometric platform) and PubPeer (a postpublication peer review platform) the production of the experts listed in its proceedings. We counted the experts’ numbers of retractions, expressions of concern, and other red flags.</p>
<h4>Results</h4>
<p> The PPS tabulated 20,066 tortured articles published from 2005 to 2024 by 344 publishers. The conference proceedings subset comprised 7518 articles (37%) mainly published by the Institute of Electrical and Electronics Engineers (IEEE) (6156 [82%]; presented in 1348 conferences), AIP Publishing (648 [9%]), IOP Publishing (364 [5%]), and 12 other publishers.<sup>3</sup> The most problematic conference was the IEEE International Conference on Optimization, Computing, and Wireless Communication held in Ethiopia in January 2024, featuring 324 peer-reviewed articles affiliated with India and China only. We identified problematic content in the proceedings using 2 clues: (1) nonsensical text, found in 260 articles (80%) with a mean 11 tortured phrases (<b>Figure 25-1029</b>), and (2) questionable expertise in 2 listed groups. The organizing committee (34 individuals) included 6 persons whose work had been commented on on PubPeer for various concerns on 31 articles, 21 of which have been retracted. Concerns included tortured phrases, suspicion of purchase of an authorship position, plagiarized sections, nonsensical equations, and irrelevant references. The reviewers (134 individuals) included 31 persons (23%) who authored 67 articles that were commented on on PubPeer for integrity issues, 8 of which had an expression of concern and 9 of which have been retracted. Eight reviewers have had several retractions, including 1 reviewer with 3 retractions due to violations of IEEE policy on authorship.</p>

![](https://assets.underline.io/markdown_image/1/image/225f2b9bf6aa303b9b83081c41f1ee00.png)

<h4>Conclusions </h4>
<p>As of February 2025, IEEE tagged with an expression of concern 15 of 260 articles (6%) flagged by the PPS (with a mean of 4 tortured phrases per article) and reported on PubPeer. IEEE had not issued expressions of concern or other corrective labels for articles with more tortured phrases. Limitations of this analysis include the focus on a single conference and the possibility that the PPS may identify false positives or less serious problems that do not require an expression of concern or retraction. However, reassessment may be needed for many of the 20,066 PPS-flagged articles.</p>
<h4>References</h4>
<p>1. Cabanac G, Labbé C, Magazinov A. Tortured phrases: a dubious writing style emerging in science—evidence of critical issues affecting established journals. <i>arXiv</i>. Preprint posted online July 12, 2021. doi:10.48550/arXiv.2107.06751</p>
<p>2. Cabanac G, Labbé C, Magazinov A. The ‘Problematic Paper Screener’ automatically selects suspect publications for post-publication (re)assessment. <i>arXiv</i>. Preprint posted online October 7, 2022. doi:10.48550/arXiv.2210.04895</p>
<p>3. Swart W, Cabanac G. Year after year: Tortured conference series thriving in computer science. <i>arXiv</i>. Preprint posted online October 17, 2023. doi:10.48550/arXiv.2401.02422</p>
<sup>1</sup>Université de Toulouse, IRIT (UMR 5505 CNRS), Toulouse, France, guillaume.cabanac@univ-tlse3.fr; <sup>2</sup>Institut Universitaire de France, Paris, France.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Guillaume Cabanac is the administrator of the Problematic Paper Screener, a public platform that uses metadata from Digital Science and PubPeer via no-cost agreements, and has been in touch with most of the major publishers and their integrity officers, offering pro bono consulting regarding detection tools to various actors in the field, including Clear Skies, Morressier, River Valley, Signals, and STM. No other disclosures were reported.</p>
<h4>Funding/Support </h4>
<p>Guillaume Cabanac received funding from the Institut Universitaire de France and the NanoBubbles project, which received Synergy grant funding from the European Research Council within the European Union’s Horizon 2020 program (grant agreement No. 951393).</p>


Tortured Phrases as a Sign of Possible Misconduct in Proceedings from an Engineering Conference

<p>Giovanni Colavizza,<sup>1,2</sup> Lauren Cadwallader,<sup>3</sup> Iain Hrynaszkiewicz<sup>3</sup></p>
<h4>Objective </h4>
<p>This research explored whether articles that demonstrate 1 or more open science practices received more citations on average than similar articles that do not demonstrate open science practices by applying an existing linear regression model<sup>1</sup> to an openly available dataset.</p>
<h4>Design </h4>
<p>This study applied a previously published citation prediction model<sup>1</sup> with data from openly available sources including OpenAlex, DataCite, CrossRef, and the French Open Science Monitor dataset,<sup>2</sup> which includes all research publications from open data sources with at least 1 French author. This study evaluated whether open science practices in publications are associated with changes in citation impact. A total of 479,700 journal articles were analyzed, which were from across 10 broad categories of research domains, have at least 1 French author, were published in 2020-2022, and for which the metadata, including citation counts, could be extracted from the publicly available sources at the time of this study (July 2024). The citation prediction method controlled for factors known to influence citations such as article age, open access status, research discipline, length of reference lists, and numbers of authors. Associations were assessed with an ordinary least squares model.</p>
<h4>Results </h4>
<p>The analysis showed that a citation advantage was associated with all 3 open science practices (data, code, and preprint sharing), both individually and cumulatively. Posting a preprint had the largest association with citations and was correlated with a 19.0% higher number of citations, compared with articles that did not have preprints. The associations with sharing research data and code were smaller—and correlated with 14.3% and 13.5% higher number of citations, respectively. These advantages were cumulative, that is, articles with a preprint, shared research data, and shared code attracted a mean of 46.8% more citations than articles that do not share these outputs. Variation in citation advantage was seen across disciplines (<b>Table 25-0867</b>). For example, articles that had preprint postings had a 1.0% higher number of citations in the fields of earth, ecology, energy and 62.5% more citations in the humanities, compared with articles in the same disciplines that did not have preprints. Study limitations include the dataset bias toward French publications (we might observe different citation correlations in a more international sample), and the results for the data and code sharing citation advantage were not directly comparable with the results in previous studies due to differences in the definition and calculation of open science practices.</p>

![](https://assets.underline.io/markdown_image/1/image/acaf7e3c6dc29e934136a937db6aade0.png)

<h4>Conclusion </h4>
<p>This study explored the French Open Science Monitor dataset and investigated whether open science indicators were associated with a citation premium received by the publications that follow them. To our knowledge, this is the largest scale analysis of such a question to date using nearly 500,000 publications from many scientific domains, which found a positive association between the open science practices of data, code, and preprint sharing and postpublication citation counts.</p>
<h4>References</h4>
<p>1. Colavizza G, Cadwallader L, LaFlamme M, et al. An analysis of the effects of sharing research data, code, and preprints on citations. <i>PLOS One</i><span lang="fr-FR">. 2024;19(10): e0311493. doi:10.1371/journal.pone.0311493</span></p>
<p><span lang="fr-FR">2. Ministère de l’Enseignement supérieur, de la Recherche et de l’Innovation. Baromètre de la science ouverte (général). Accessed July 15, 2024. </span><a href="https://data.enseignementsup-recherche.gouv.fr/explore/dataset/open-access-monitor-france/information."><span class="Hyperlink CharOverride-6" lang="fr-FR">https://data.enseignementsup-recherche.gouv.fr/explore/dataset/open-access-monitor-france/information</span></a></p>
<sup>1</sup>University of Bologna, Bologna, Italy; <sup>2</sup>University of Copenhagen, Copenhagen, Denmark; <sup>3</sup>PLOS, Cambridge, UK, lcadwallader@plos.org.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> PLOS works to promote adoption of open science practices and provided funding for this study for the collection, management, analysis, and interpretation of the data by Giovanni Colavizza. PLOS also provided support in the form of salaries for authors Lauren Cadwallader and Iain Hrynaszkiewicz. PLOS also had a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, and approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Additional Information</h4>
<p> Data and code necessary to reproduce the results are available: <a href="https://figshare.com/articles/dataset/A_study_on_the_citation_impact_of_Open_Science_Indicators_in_the_French_Open_Science_Monitor/27822663"><span class="Hyperlink CharOverride-6">https://doi.org/10.6084/m9.figshare.27822663</span></a></p>


Citations of Articles With Open Science Indicators in the French Open Science Monitor Dataset

<p>Stephan B. Bruns,<sup>1</sup> Helmut Herwartz,<sup>2</sup> John P. A. Ioannidis,<sup>3</sup> Chris-Gabriel Islam,<sup>2</sup> Fabian H. C. Raters<sup>2</sup></p>
<h4>Objective </h4>
<p>The economics literature predominantly reports statistical values in data tables. This study developed a tool for extraction of statistical values in tables to identify reporting errors and to analyze the association between journal data and code availability policies and article characteristics.</p>
<h4>Design </h4>
<p>DORIS (Diagnosis of Reporting Errors in Scraped Tables)<sup>1,2</sup> automatically extracts statistical values (eg, coefficients and SEs) from tables via web scraping in R to extract tables from HTML articles and text mining in Python to interpret data. This analysis included the top 50 economics journals that provided articles in HTML from 1998 to 2016. Tests were divided into 2 categories: main (first 3 table rows from main tables) and nonmain (all other rows from main tables and other tables). We used a staggered difference-in-differences design to assess the association between data and code availability policies and article outcomes<sup>3</sup>: reporting errors using a dummy for inconsistency altering significance level; statistical significance proxied by <i>z</i>-value, presence of visual indicators of statistical significance, and type of statistical value reported; logarithmized number of tests per article, tables with tests, and a dummy for appendix position; and citations measured by the logarithmized citation count for the first 7 years.</p>
<h4>Results </h4>
<p>The study included 578,132 statistical tests (median: 27 per table) from 15,725 tables (median: 4 per article) in 3746 articles (median: 76 per journal) that were published in 31 journals. Based on a sample of 3068 statistical tests, DORIS had a false discovery rate of 1.2%. Analysis revealed that 547 of 3677 articles (14.8%) had at least 1 reporting error in main tests altering the statistical significance of findings. The errors were prevalently oriented toward lowering <i>P</i> values and claiming statistical significance. Assessment of data and code availability policies included 21 journals and 535,838 tests. These policies were associated with slightly less emphasis on statistical significance (smaller z-values: −0.16 [90% CI, −0.33 to 0], <i>P</i> = .10; fewer statistical significance symbols: −4 [90% CI, −6 to −2] percentage points [pp]; <i>P</i> = .004; fewer tests reported with a focus on significance (<i>t</i>, <i>z</i>, or <i>P </i>values instead of SEs and CIs): −8 [90% CI, −13 to −3] pp; <i>P</i> = .004), more rigor in reporting (more tests: 11% [90% CI, −1% to 22%]; <i>P</i> = .12; more tables: 15% [90% CI, 7% to 23%]; <i>P</i> = .002; more appendix tables: 4 [90% CI, 2 to 7] pp; <i>P</i> = .009), and fewer reporting errors in nonmain tests (−0.2 [90% CI, −0.4 to 0] pp; <i>P</i> = .08), but there was no difference in citations (−9% [90% CI, −21% to 4%]; <i>P</i> = .25).</p>
<h4>Conclusions </h4>
<p>DORIS may be used in the peer review process in the economics literature to improve article quality and to generate large scale data for future meta-research. Data and code availability policies may help improve the reliability of published economics research.</p>
<h4>References</h4>
<p>1. Better Papers. Home page. Accessed June 12, 2025. <a href="https://betterpapers.org/%23sec-faqs"><span class="Hyperlink CharOverride-6">https://betterpapers.org/#sec-faqs</span></a></p>
<p>2. Bruns SB, Herwartz H, Ioannidis JPA, Islam C-G, Raters FHC. Statistical reporting errors in economics. <i>MetaArXiv</i>. Preprint posted online September 01, 2023. doi:10.31222/osf.io/mbx62</p>
<p>3. Borusyak, K, Jaravel, X, Spiess, J. Revisiting event-study designs: robust and efficient estimation. <i>Rev Econ Studies</i>. 2024;91(6):3253-3285. doi:10.1093/restud/rdae007</p>
<sup>1</sup>Hasselt University, Hasselt, Belgium, stephan.b.bruns@gmail.com; <sup>2</sup>Georg August University Göttingen, Göttingen, Germany; <sup>3</sup>Meta-Research Innovation Center at Stanford (METRICS), Stanford, CA, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> John P. A. Ioannidis is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Funding/Support</h4>
<p> Funding was received from the German Research Foundation (DFG) under the project “Replications in Empirical Economics: Necessity, Incentives and Impact,” with follow up project “Selective Reporting and the Evolving Research Landscape in Economics” (project number: 405039391) and from Hasselt University within the framework of BOF BILA (BOF21BL08).</p>
<h4>Role of Funder/Sponsor</h4>
<p> DFG and BOF BILA funding helped in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>


Automated Interpretation of Statistical Tables to Assess Reporting Errors and Associations With Open Science Policies in Economic Journals

<p>Constant Vinatier,<sup>1</sup> Ayu Putu Madri Dewi,<sup>2</sup> Gwénaël Dumont,<sup>1</sup>Tracey Weissgerber,<sup>3</sup>Vladislav Nachev,<sup>3</sup>Gowri Gopalakrishna,<sup>2,3,4</sup> Maud Scheidecker,<sup>1</sup> François-Joseph Arnault,<sup>1</sup> Nicholas J. DeVito,<sup>5</sup> Guillaume Freyermuth,<sup>6</sup> Mathieu Acher,<sup>6,10</sup> Gauthier Le Bartz Lyan,<sup>6</sup> Inge Stegeman,<sup>7,8</sup>Mariska M. G. Leeflang,<sup>2</sup>F. Naudet<sup>9,10</sup></p>
<h4>Objective </h4>
<p>Despite open science policies in major biomedical journals, adherence remains uncertain. This study evaluated automated tools, from regular expressions to large language models (LLMs), for assessing core open science practices in leading biomedical journals.</p>
<h4>Design </h4>
<p>We retrospectively assessed research articles from a sample of 10 major generalist medical journals (<i>Annals of-Internal Medicine</i>, <i>BMJ</i>, <i>BMC Medicine</i>, <i>Canadian Medical Association Journal</i> [<i>CMAJ</i>], <i>JAMA</i>, <i>JAMA</i> <i>Network Open</i>, <i>Lancet</i>, <i>Nature Medicine</i>, <i>New England Journal of Medicine</i>, and <i>PLoS Medicine</i>) from 2020 to 2023. Articles were retrieved via PubMed using a Peer Review of Electronic Search Strategies (PRESS) search strategy. The database comprised random samples of 103 randomized controlled trials (RCTs), 98 meta-analyses (MAs), and 111 other research articles (RAs). We evaluated 13 open science practices, including study registration, data sharing, and protocol sharing (open access or upon request). Each article was evaluated by 2 independent raters, with any disagreements resolved by a third rater. Seven different automated tools—rtransparent, oddpub, ctRegistries, ContriBot, DataSeer, SciScore, and an LLM (Llama 3-70B)—were used. Diagnostic accuracies were estimated using sensitivities, specificities, F1 scores, and LR+ and LR-.</p>
<h4>Results </h4>
<p>Manual extraction in the 312 articles identified registration in 98% (101/103) of RCTs, 69% (68/98) of MAs, and 18% (20/111) of RAs. Open data were present in 6% (6/103) of RCTs, 36% (35/98) of MAs, and 13% (15/111) of RAs and accessible upon request in 78% (80/103), 41% (40/98), and 59% (66/111), respectively. Protocols were openly available in 84% (87/103) of RCTs, 64% (63/98) of MAs, and 20% (22/111) of RAs and accessible upon request in 5% (5/103), 3% (3/98), and 3% (3/111), respectively. The accuracy of automated tools varied depending on the practice evaluated, with F1 scores ranging from 1.00 (Conflict of Interest statement, rtransparent) to 0.16 (SciScore, registration). For study registration, a simple tool using regular expressions, such as rtransparent, demonstrated good sensitivity (77%; 95% CI, 70%-83%) and high specificity (93%; 95% CI, 88%-97%). Data sharing detection remained challenging; for instance, rtransparent detects data sharing with a sensitivity of 74% (95% CI, 68%-80%) and a specificity of 59% (95% CI, 46%-70%). Different diagnostic accuracies were observed depending on the type of research and the journal, likely due to different formatting standards. All results are shown in <b>Table 25-1025</b>. Limitations include the declarative nature of some practices (eg, data sharing).</p>

![](https://assets.underline.io/markdown_image/1/image/f6bf4287ea9763381f2096129dbea4e8.png)

<h4>Conclusions </h4>
<p>Our study provides a detailed description of core open science practices across leading biomedical journals. It also highlights current challenges regarding the accuracy of automated tools in detecting these practices. While these tools likely provide valuable insights into overall practices, it is crucial to remain aware of the potential ranking biases introduced by these tools, as well as their limitations in providing detailed feedback for individual studies.</p>
<sup>1</sup>Univ Rennes, Inserm, EHESP, Irset (Institut de recherche en santé, environnement et travail), UMR_S 1085, Rennes, France, constant.vinatier1@gmail.com; <sup>2</sup>Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, the Netherlands; <sup>3</sup>QUEST Center for Responsible Research, Berlin Institute of Health at Charité–Universitätsmedizin Berlin, Berlin, Germany; <sup>4</sup>Department of Epidemiology, Faculty of Health, Medicine, and Life Sciences, Maastricht University, Maastricht, the Netherlands; <sup>5</sup>Bennett Institute for Applied Data Science, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK; <sup>6</sup>Univ Rennes, IRISA, Inria, CNRS, Rennes, France; <sup>7</sup>Department of Otorhinolaryngology and Head and Neck Surgery, University Medical Center Utrecht, Utrecht, the Netherlands; <sup>8</sup>Brain Center, University Medical Center Utrecht, Utrecht, the Netherlands; <sup>9</sup>Univ Rennes, CHU Rennes, Inserm, EHESP, Irset (Institut de recherche en santé, environnement et travail), UMR_S 1085, Rennes, France; <sup>10</sup>Institut Universitaire de France (IUF), France.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> As part of the OSIRIS project, this work was supported by the European Union’s Horizon Europe Research and Innovation Program under grant agreement number 101094725. Constant Vinatier, Ayu Putu Madri Dewi, Gowri Gopalakrishna, Nicholas J. DeVito, Inge Stegeman, Mariska M. G. Leeflang, and <span lang="fi-FI">F. Naudet</span> are members of this project.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.</p>


Detection of Open Science Practices in Major Medical Journals: A Survey and Diagnostic Accuracy of Automatic Tools Using Sensitivity and Specificity

<p>Luka Ursić,<sup>1,2</sup> Nensi Bralić,<sup>1</sup> Giovanni Spitale,<sup>2</sup> Federico Germani,<sup>2</sup> Ana Marušić<sup>1</sup></p>
<h4>Objective</h4>
<p> Divisive discourse observed in news coverage or social media posts by researchers and the general public during the H1N1 and COVID-19 pandemics contributed to polarization and politicization, rapid shifts in sentiment, and concerning rates of misinformation.<sup>1-3</sup> We aimed to study the sentiment and natural language processing (NLP) analysis of editorial material in medical journals discussing the health care system response to the 2 pandemics.</p>
<h4>Design</h4>
<p> Using a systematic review design, we searched Web of Science and PubMed up to May 22, 2024, for editorials, viewpoints, and similar opinion pieces discussing the following segments of the health care system response to the 2 pandemics: nonpharmaceutical interventions; misinformation or disinformation; health care resource allocation and management; health care system preparedness; mandates, policies, and guidance related to research, education, or new technologies; and lessons learned. We limited our search to 2009 to 2014 for the H1N1 pandemic and 2019 to 2024 for the COVID-19 pandemic. Three researchers screened the titles, abstracts, and full texts of the retrieved records for eligibility in pairs, resolving discrepancies through discussion. We extracted the text from the included records’ PDFs using Python and performed a sentiment analysis using the Language Inquiry and Word Count 2022 software. We also used NLP approaches (flat lemmatization, convoluted lemmatization, rule-based autocoding, and cosine similarity) to determine the most frequently used lemmas in the 2 groups of editorials and explore topics based on their co-occurrence. We conducted inferential analyses at the level of the full sample and across specific categories (ie, stratified by pandemic, author country, and context of response). While we could not set a hypothesis for the exploratory NLP analysis, we hypothesized that in terms of sentiment, the editorials published during the COVID-19 pandemic would have higher scores for negative tone and emotion, personal and person-centered language, certitude, and all-or-none thinking and lower scores for analytical thinking.</p>
<h4>Results </h4>
<p>Following the screening process (<b>Figure 25-0979</b>), we included 2954 editorials in the final dataset. We performed a preliminary analysis of 200 editorials: 25 for the H1N1 pandemic and 175 for the COVID-19 pandemic. In contrast to our initial hypothesis, the editorials for the H1N1 pandemic had higher values for negative tone (median, 2.21 [IQR, 1.74-3.07] vs 1.69 [IQR, 1.22-2.31]; <i>P</i> = .007) and certitude (median, 0.39 [IQR, 0.16-0.66] vs 0.23 [IQR, 0.12-0.37]; <i>P</i> = .01). There were no differences in negative emotion, analytical thinking, or all-or-none thinking.</p>

![](https://assets.underline.io/markdown_image/1/image/d658c3c3e26c29b586c4a178cb5264eb.png)

<h4>Conclusions </h4>
<p>In this preliminary analysis, the editorials for the H1N1 pandemic contained more words connoting negative tone and certitude than those for the COVID-19 pandemic. These findings suggest that the researchers authoring these editorials expressed a more negative sentiment toward the public health response to the H1N1 pandemic and that they were seemingly more certain about their position.</p>
<h4>References</h4>
<p>1. Schmidt H. Pandemics and politics: analyzing the politicization and polarization of pandemic-related reporting. <i>Newsp Res J</i>. 2023;44(1):26-52. doi:10.1177/07395329221095850</p>
<p>2. Lwin MO, Lu J, Sheldenkar A, et al. Global sentiments surrounding the COVID-19 pandemic on Twitter: analysis of Twitter trends. <i>JMIR Public Health Surveill</i>. 2020;6(2):e19447. doi:10.2196/19447</p>
<p>3. Sule S, DaCosta MC, DeCou E, Gilson C, Wallace K, Goff SL. Communication of COVID-19 misinformation on social media by physicians in the US. <i>JAMA Netw Open</i>. 2023;6(8):e2328928. doi:10.1001/jamanetworkopen.2023.28928</p>
<sup>1</sup>Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia, luka.ursic@mefst.hr; <sup>2</sup>Institute of Biomedical Ethics and History of Medicine, University of Zürich, Zürich, Switzerland.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Ana Marušić is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Funding/Support </h4>
<p>This study was funded by the Croatian Science Foundation under grant agreement IP-2019-04-4882 and the University of Zürich Institute of Biomedical Ethics and History of Medicine under a Stehr-Boldt fellowship given to Luka Ursić.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funders had no role in the design of this study, its execution, analyses, interpretation of the data, or decision to submit results.</p>


Analysis of Editorials on the Response to the H1N1 and COVID-19 Pandemics

<p>Caitlin J. Bakker,<sup>1,2</sup> Erin E. Reardon,<sup>3</sup> Sarah Jane Brown,<sup>4</sup> Nicole Theis-Mahon,<sup>4</sup> Sara Schroter,<sup>5,6</sup> Lex Bouter,<sup>7,8</sup> Maurice P. Zeegers<sup>2</sup></p>
<h4>Objective </h4>
<p>The scientific community responded rapidly to COVID-19, producing over 200,000 publications in 1 year.<sup>1</sup> This speed brought challenges, including a higher retraction rate.<sup>2</sup> While retraction helps correct the scientific record, retracted status may be inconsistently presented and notices may be incomplete.<sup>3</sup> During a pandemic, inconsistency and incompleteness can have immediate and long-term health impacts. We evaluated retraction presentation consistency and notice completeness for COVID-19 vs non–COVID-19 publications. We describe reasons for retraction and time from publication to retraction.</p>
<h4>Design </h4>
<p>In March 2023, we retrieved retracted publications categorized as research articles or clinical studies in the subject area of public health and safety from Retraction Watch. A previous study focused on all retracted publications in this period<sup>3</sup>; this is a sub-study comparing COVID-19 with non–COVID-19 publications. Between April 28 and June 6, 2023, we assessed consistency in 11 databases (Academia.edu, CINAHL, Embase.com, Ovid Embase, Ovid Medline, PubMed, ResearchGate, SciHub, Scopus, Web of Science, and publisher websites) using 12 criteria from the International Committee of Medical Journal Editors and National Library of Medicine and notice completeness using 17 criteria from Retraction Watch and the Committee on Publication Ethics. Criteria were scored 0 if missing or 1 if present; partial scores were assigned when evaluating multicomponent criteria, such as bidirectional links. To ensure consistent scoring, a random subset of 21% (92 of 441) of retracted publications were independently reviewed by 2 researchers. Each researcher extracted data using Qualtrics forms, and scoring discrepancies were resolved by consensus. Following this calibration phase, remaining publications were extracted by a single reviewer. Kruskal-Wallis tests assessed differences in scores between COVID-19 and non–COVID-19 publications. </p>
<h4>Results</h4>
<p> Of 441 publications, 47 were about COVID-19 and 394 were not. COVID-19 publications were published between 2019 and 2022, while non-COVID 19 publications were published between 1978 and 2022. COVID-19 publications were most frequently retracted due to concerns about reliability of data or results (15 [31.9%]) compared with plagiarism (82 [20.8%]) for non–COVID-19 publications. The median time between publication and retraction was 120 (IQR, 15-196) days for COVID-19 publications and 326 (IQR, 124-789) days for non–COVID-19 publications (<i>P</i> &lt; .001). Across 11 databases, 41.2% (110 of 267) of records retrieved for retracted COVID-19 publications were marked as retracted compared with 47.6% (1225 of 2574) for non–COVID-19 publications. There was no statistically significant difference between consistency or completeness scores for COVID-19 vs non–COVID-19 retracted publications (<b>Table 25-1062</b>). No publications met all criteria.</p>

![](https://assets.underline.io/markdown_image/1/image/edc419e41ec09f04154b15739dea0699.png)

<h4>Conclusions </h4>
<p>Incomplete and inconsistent information poses challenges for researchers and practitioners, undermining trust in scientific literature. We found no association between publications being about COVID-19 and the consistency or completeness of retraction information; however, publications about COVID-19 appeared to be retracted more quickly.</p>
<h4>References</h4>
<p>1. Shimray SR. Research done wrong: a comprehensive investigation of retracted publications in COVID-19. <i>Account Res</i>. 2022;30(7):393-406. doi:10.1080/08989621.2021.2014327</p>
<p>2. Yeo-Teh NSL, Tang BL. An alarming retraction rate for scientific publications on coronavirus disease 2019 (COVID-19). <i>Account Res</i>. 2020;28(1):47-53. doi:10.1080/08989621.2020.1782203</p>
<p>3. Bakker CJ, Reardon EE, Brown SJ, et al. Identification of retracted publications and completeness of retraction notices in public health. <i>J Clin Epidemiol</i>. 2024;173:111427. doi:10.1016/j.jclinepi.2024.111427</p>
<sup>1</sup>University of Regina, Regina, Saskatchewan, Canada, caitlin.bakker@uregina.ca; <sup>2</sup>Maastricht University, Maastricht, the Netherlands; <sup>3</sup>Emory University, Atlanta, GA, US; <sup>4</sup>University of Minnesota, Minneapolis, MN, US; <sup>5</sup><i>BMJ</i>, London, UK; <sup>6</sup>London School of Hygiene and Tropical Medicine, London, UK; <sup>7</sup>Amsterdam University Medical Center, Amsterdam, the Netherlands; <sup>8</sup>Vrije Universiteit Amsterdam, Amsterdam, the Netherlands.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Caitlin J. Bakker is cochair of the National Information Standards Organization Communication of Retractions, Removals and Expressions of Concern Standing Committee. Lex Bouter is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Funding/Support</h4>
<p> This research is part of an ongoing PhD collaboration between The BMJ (British Medical Journal) and the team Meta-Research at Maastricht University (UM) on the responsible conduct of publishing scientific research. The BMJ is published by BMJ Group, a wholly owned subsidiary of the British Medical Association. UM is a public legal entity in the Netherlands. This study is part of Caitlin Bakker’s self-funded BMJ/UM PhD. No exchange of funds has taken place for this research project.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The authors are wholly responsible for the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Disclaimer</h4>
<p> All authors express their own opinions and not necessarily that of their employers.</p>


Consistency and Completeness of Retractions in Public Health Research on COVID-19

<p>Annie Whamond,<sup>1</sup> Adrian G. Barnett,<sup>2</sup> Jennifer A. Byrne<sup>1,3</sup></p>
<h4>Objective</h4>
<p> Paper mills are unethical organizations that provide low-value or fraudulent content to client authors.<sup>1</sup> Although image manipulation and incorrect reagent detection tools are available,<sup>1</sup> these approaches can require reviewers and readers to learn new skills. Scaled manuscript production through templates may also result in paper mill articles showing other common features. We aimed to help readers identify possible indicators of paper mill support by simply scanning article text and references. We focused on discussion sections for this study.</p>
<h4>Design</h4>
<p> We used an exploratory, cross-sectional design<sup>2</sup> to develop a descriptive analysis of discussion text and references in high Impact Factor (IF) cancer journals, defined as IF of 7 or higher for journal categories oncology, biochemistry and molecular biology, or cell biology. We downloaded the Retraction Watch database on October 16, 2024, and filtered for retraction reason <i>paper mill</i>, subject <i>cancer</i>, and journals on our high IF list. To identify comparison article cohorts, we randomly sampled original cancer articles in (1) the same 8 journals as retracted articles or (2) 8 independent journals with no paper mill retractions and high IF maintained for 20 years or longer. For each article, we recorded the percentages of references first cited in the introduction, methods, results, or discussion section. Focusing on discussion sections, we classified each sentence as providing background, summary, comparison, interpretation or implication, limitation, or future direction<sup>3</sup> and recorded whether individual discussion sentences cited new references. </p>
<h4>Results </h4>
<p>We found 22 retracted paper mill articles from 8 journals published between June 21, 2016, and June 16, 2022. Both comparison groups (50 articles each) were restricted to articles published between January 1, 2016, and October 31, 2024. Articles in all 3 groups included similar numbers of total references per article (median [IQR]: 42 [36-46] retracted articles; 47 [36-55] articles in same journals; 51 [45-60] articles in independent journals) and percentages of discussion sentences per article (median [IQR]: 34 [30-38] retracted articles; 40 [30-48] same journals; 34 [26-43] independent journals). Analyses of discussion sections indicated that retracted paper mill articles included higher percentages of references that were first cited in the discussion and higher percentages of discussion sentences that described background information and cited new references (<b>Figure 25-1022</b>).</p>

![](https://assets.underline.io/markdown_image/1/image/d20ae00ed6b6328fe12ed01f027e066e.png)

<h4>Conclusions</h4>
<p> Our analyses suggest that some discussion sections in retracted paper mill cancer research articles reiterate background information that is supported by new and possibly superfluous references. While recognizing that genuine studies also cite new references in discussion sections, superficial and redundant second introductions in discussion sections could help readers identify potentially problematic articles in high IF cancer journals, particularly when combined with other features of paper mill support.<sup>1</sup></p>
<h4>References</h4>
<p>1. Byrne JA, Abalkina A, Akinduro-Aje O, et al. A call for research to address the threat of paper mills. <i>PLoS Biol</i>. 2024;22(11):e3002931. doi:10.1371/journal.pbio.3002931</p>
<p>2. Kesmodel US. Cross-sectional studies—what are they good for? <i>Acta Obstet Gynecol Scand</i>. 2018;97(4):388-393. doi:10.1111/aogs.13331</p>
<p>3. Toronto CE, Remington R. Discussion and conclusion. In: <i>A Step-by-Step Guide to Conducting an Integrative Review</i>. Toronto CE, Remington R, eds. Springer International; 2020:71-84.</p>
<sup>1</sup>School of Medical Sciences, Faculty of Medicine and Health, University of Sydney, Sydney, New South Wales, Australia, jennifer.byrne@health.nsw.gov.au; <sup>2</sup>School of Public Health and Social Work, Queensland University of Technology, Kelvin Grove, Queensland, Australia; <sup>3</sup>NSW Health Statewide Biobank, NSW Health Pathology, Camperdown, New South Wales, Australia.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> Adrian G. Barnett and Jennifer A. Byrne acknowledge grant funding from the National Health and Medical Research Council of Australia (ideas grant APP2029249). This grant supports Annie Whamond’s PhD candidature.</p>
<h4>Role of Funder/Sponsor</h4>
<p> The funding body played no role in the study design, data collection, management, analysis, or interpretation, and will play no role in the writing of any report, or the decision to submit the report for publication.</p>


Analysis of Cancer Research Discussion Text and References in High-Impact Factor Journals for Possible Indicators of Paper Mills

<p>Alexandre Clausse,<sup>1</sup> Guillaume Cabanac,<sup>1,2</sup> Pascal Cuxac,<sup>3</sup> Cyril Labbé<sup>4</sup></p>
<h4>Objective</h4>
<p> The Problematic Paper Screener (PPS) features 10 detectors to identify problematic articles in the scientific record.<sup>1</sup> The detectors search through metadata, references, or textual contents. The tortured phrases<sup>2</sup> detector uses a list of more than 7000 identified suspect expressions (<b>Table 25-1159</b>) to query a database (Dimensions; Digital Science) with approximately 130 million scientific articles.<sup>3</sup> The PPS displays the screening results on a public website. In this qualitative study, we introduce an alternative screening algorithm based on regular expressions (regex) and benchmark its effectiveness. Publishers are welcome to use this stand-alone algorithm that does not require a subscription to the database that we used. </p>

![](https://assets.underline.io/markdown_image/1/image/8b16462ed041c6b4f82712aec9be879f.png)

<h4>Design</h4>
<p> As of May 2025, we designed an algorithm that uses regex based on the PPS fingerprints list in the database to capture different forms of tortured phrases, such as “man-made consciousness,” “profound learning AND deep learning,” and “128 pieces”~5’ [128 bits] (ie, a tortured phrase with components in a 5-word sliding window). This approach is independent from search engines, as it matches fingerprints against the textual content of each article, ignoring their metadata, figure and table contents (including captions), and references. We benchmarked this new approach on the Hindawi extensible markup language (XML) corpus, which contains several hundred articles with tortured phrases, focusing on articles published from 2020 to 2022—the period with the most PPS-flagged articles (n <i>= </i>3400).</p>
<h4>Results</h4>
<p> The regex-based algorithm flagged 2455 problematic articles, with 1948 also flagged by PPS, yielding a 58% overlap. After removing duplicates, 401 articles flagged by the PPS were missing from the Hindawi XML corpus, amounting to 3371 problematic articles in 139 different journals. The evaluation is ongoing; we analyzed the top 200 results and found 48 false-positive results due to database indexing issues related to special characters and figures and PPS querying issues. We also found 52 false-negative results due to fingerprints list issues and because we excluded figures, tables, and references. Overall, we obtained 100 true-positive results, as both detectors (ie, PPS and regex-based algorithm) extracted expressions on the fingerprints list from the same articles.</p>
<h4>Conclusions</h4>
<p> The regex-based algorithm yielded results comparable to the current PPS screening process. Other configurations should be tested, including figures, tables, and references in the screening, as tortured phrases can also appear there. We invite publishers to use this regex-based approach to screen incoming manuscripts. We noticed several querying issues with the database that we used, and some fingerprints need to be redesigned. We will investigate further the validation assessment, as there are still 3351 articles to be reassessed. This research could contribute to improving the regex and updating the PPS fingerprints list, while offering detailed feedback and bug reports to the database developer.</p>
<h4>References</h4>
<p>1. Cabanac G, Labbé C, Magazinov A. The ‘Problematic Paper Screener’ automatically selects suspect publications for post-publication (re)assessment. <i>arXiv</i>. Preprint posted online October 7, 2022. doi:10.48550/arXiv.2210.04895</p>
<p>2. Cabanac G, Labbé C, Magazinov A. Tortured phrases: A dubious writing style emerging in science—evidence of critical issues affecting established journals. <i>arXiv</i>. Preprint posted online July 12, 2021. doi:10.48550/arXiv.2107.06751</p>
<p>3. Herzog C, Hook D, Konkiel S. Bringing down barriers between scientometricians and data. <i>Quantitative Science Studies</i>. 2020;1:387-395. doi:10.1162/qss_a_00020</p>
<sup>1</sup>Université de Toulouse, IRIT (UMR 5505), Toulouse, France, alexandre.clausse@irit.fr; <sup>2</sup>Institut Universitaire de France, Paris, France; <sup>3</sup>INIST CNRS, UAR 67, Vandoeuvre-lès-Nancy, France; <sup>4</sup>Université Grenoble Alpes, CNRS, Grenoble INP, LIG UMR 5217, Grenoble, France.</p>
<h4>Conflicts of Interest Disclosures </h4>
<p>Guillaume Cabanac is the administrator of the Problematic Paper Screener, a public platform that uses metadata from Digital Science and PubPeer via no-cost agreements. Guillaume Cabanac and Cyril Labbé have been in touch with most of the major publishers and their integrity officers, offering pro bono consulting regarding detection tools to various actors in the field, including Clear Skies, Morressier, River Valley, Signals, and STM.</p>
<h4>Funding/Support </h4>
<p>Alexandre Clausse, Guillaume Cabanac, and Cyril Labbé received funding from the European Research Council. Guillaume Cabanac received funding from the Institut Universitaire de France.</p>


Screening articles for tortured phrases with a regex-based detector

<p>Melissa L. Rethlefsen,<sup>1</sup> Carrie Price,<sup>2</sup> Sara Schroter<sup>3</sup></p>
<h4>Objective</h4>
<p> To explore the perspectives of librarians and information specialists (LISs) who participated in a randomized controlled trial of the effect of LIS involvement on reporting quality of systematic reviews (SRs).<sup>1</sup></p>
<h4>Design</h4>
<p> We surveyed LISs who completed a peer review of an SR in a trial conducted in <i>BMJ</i>, <i>BMJ Open</i>, and <i>BMJ Medicine</i> from January 3, 2023, to January 2, 2024. LISs were not told they were peer reviewing manuscripts as part of a trial but were sent invitations to review as part of the usual process. The questionnaire sought to understand their experience, what aspects of manuscripts they focused on, perceived impact on editorial decision-making and authors’ revisions, and willingness to peer review again. To better understand factors that might impact decisions to review again, we contacted 27 respondents to participate in a focus group concentrating on facilitators and barriers to peer reviewing SRs. </p>
<h4>Results</h4>
<p> Of the 88 LISs invited to participate in the survey, 70 (79.5%) responded. Most respondents had 6 or more years of experience as an LIS (67/70; 95.7%) and advising researchers on doing SRs (55/70; 78.6%) and had peer reviewed for a journal prior to the study (57/70; 81.4%). Most focused on the search and SR methods when reviewing but also commented on aspects such as research question formulation, plagiarism, and study results and conclusions. Two-thirds (44/66; 66.7%) believed they impacted editors’ decision-making and 59.1% (39/66) believed they impacted the authors’ revisions. Only 3 factors were considered extremely or very likely to impact their decision to review again: their schedule and/or lack of time, review turnaround time, and their sense of professional duty (<b>Table 24-0830</b>). Seventeen of 27 invited LISs (63.0%) participated in a focus group. Time was the primary barrier identified in the focus groups, followed by a sense of intimidation. LISs reported being motivated by feeling valued by editors, the enjoyment of peer reviewing, the desire to improve SR quality, and peer review as a learning experience. Several expressed surprise and delight at being asked to peer review for the journals.</p>

![](https://assets.underline.io/markdown_image/1/image/c26ed0c4df287a1d953b680e49fbcda6.png)

<h4>Conclusions</h4>
<p> A select sample of highly engaged LIS respondents believed they made a difference through their peer reviews and said they were very likely to agree to peer review in the future. LISs may be an underused peer reviewing resource with methodological experience that can help editors make decisions and improve the quality of SRs.</p>
<h4>Reference</h4>
<p>1. Rethlefsen ML, Schroter S, Bouter LM, et al. Improving peer review of systematic reviews and related review types by involving librarians and information specialists as methodological peer reviewers: a randomised controlled trial. <i>BMJ Evid Based Med</i>. Published online March 11, 2025. doi:10.1136/bmjebm-2024-113527</p>
<sup>1</sup>Health Sciences Library &amp; Informatics Center, University of New Mexico, Albuquerque, NM, US, mlrethlefsen@gmail.com; <sup>2</sup>ToxStrategies, A Blue Ridge Life Sciences Company, Baltimore, MD, US; <sup>3</sup>BMJ Publishing Group, London, UK.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>This study follows up on a study that was conducted as part of Melissa L. Rethlefsen’s self-funded PhD project registered at Maastricht University, the Netherlands, in collaboration with BMJ Publishing Group. Carrie Price was employed at the National Institutes of Health Library at the time of this research. Sara Schroter is a full-time employee of BMJ Publishing Group but is not involved in decision-making on individual research submissions.</p>


Downloads

Next from The 10th Peer Review Congress

Identifying Potential Duplicate Publications in the Scientific Literature Using Crossref

Similar lecture

Diversity Among Reviewers Assigned to Evaluate a Paper as a Factor in Diversifying Perspective and Improving the Peer Review Process in Computer Science

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES