United States

&lt;p&gt;Leslie D. McIntosh,&lt;sup&gt;1&lt;/sup&gt;&lt;span lang=&quot;ar-SA&quot;&gt; Hélène Draux&lt;/span&gt;,&lt;sup&gt;1&lt;/sup&gt; Elizabeth Smee,&lt;sup&gt;1&lt;/sup&gt; Cynthia Hudson Vitale&lt;sup&gt;2&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;To understand the broad impact of one questionable network on publishing, including the publishers, countries, and institutions involved and affected by the network’s activity.&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;We examined the Pharmakon Neuroscience Research Network (PNN), a questionable entity listed in funding statements, acknowledgments, and author affiliations in articles published between 2019 and 2023. We selected the PNN because it was the first potential broad-scale case of scientific manipulation identified by one of the authors (L.D.M), and analyzing the PNN has helped the development of patterns to discover other unusual collaboration networks. All documents containing “Pharmakon Neuroscience” anywhere in the title, abstract, author affiliations, and full text were selected using Dimensions.&lt;sup&gt;1&lt;/sup&gt; We excluded retraction and correction notices; articles for which a peer reviewer identified their organization as PNN; and articles with more than 25 authors. We inspected the metadata associated with the articles, notably the organizations related to the articles. We situated these practices within the Taxonomy of Scientific Manipulation,&lt;sup&gt;2&lt;/sup&gt; systematically identifying (1) who was involved (authors, institutions, and publishers), (2) where&lt;span class=&quot;CharOverride-9&quot;&gt; &lt;/span&gt;mechanisms were used to manipulate science (journals, countries, and institutions), and (3) how the science was manipulated (methods for deceiving scholarly communication).&lt;/p&gt;
&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;Pharmakon Neuroscience was referenced in 140 published articles as of January 2025. After applying exclusion criteria, 123 articles remained. These articles included more than 6000 citations (citation ratio of 51 citations per article). The articles had 361 unique authors, including 29 without author identifiers (eg, ORCiD, Dimensions). The PNN involved 56 journals across 12 publishers, with Springer Nature (n = 29), Elsevier (n = 26), and Bentham Science Publishers (n = 25) having the most articles involved out of the total (n = 123). The involved authors were affiliated with 40 countries and 232 organizations. Of these organizations, 212 had research identifiers (eg, Research Organization Registry, Global Research Identifier Database), and 20 did not. Three of the organizations appeared to be personal homes, even though they were claimed as businesses or educational institutions. Additionally, one publication&lt;sup&gt;3&lt;/sup&gt; highlighted the PNN as an affiliation among the “top 25 most predictive and impactful authors.”&lt;/p&gt;
&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;This investigation leveraged publication databases and advanced forensic scientometric techniques to highlight critical research integrity and security vulnerabilities in a questionable research network. We use the term “questionable network” because this co-authorship network appears to have grown too rapidly to have developed naturally from scholarly affiliations over time. The impact of the PNN’s work might be legitimized through future studies and become seen as normative behavior. Although the most prolific PNN author and organization stem from one country, there is a global impact not confined to high- or low-income countries. While one individual appears central to the network, a single orchestrator does not imply sole responsibility. Instead, this case underscores the complexity of manipulating the scientific ecosystem, where questionable actors exploit the scholarly communication system through nonverifiable funders and organizations and artificially inflated citation metrics.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Hook DW, Porter SJ, Herzog C. Dimensions: building context for search and evaluation. &lt;i&gt;Front Res Metr Anal&lt;/i&gt;. 2018;3:23. doi:10.3389/frma.2018.00023&lt;/p&gt;
&lt;p&gt;2. McIntosh LD, White W, Vitale CH. Unveiling deception: establishing a taxonomic framework for disinformation within scientific discourse. &lt;i&gt;arXiv&lt;/i&gt;. Preprint posted online November 19, 2023. doi:10.48550/arxiv.2311.11344&lt;/p&gt;
&lt;p&gt;3. Sab MC, Ahmed KM. Tracing Bangladesh’s scientific footprints: a scientometric expedition into inflammation pharmacology. &lt;i&gt;Inform Res Comm&lt;/i&gt;. 2024;1(1):9-21. doi:10.5530/irc.1.1.4&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Digital Science, London, UK, l.mcintosh@digital-science.com; &lt;sup&gt;2&lt;/sup&gt;Johns Hopkins University, Baltimore, MD, US. 
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt; Leslie D. McIntosh, Hélène Draux, and Elizabeth Smee are employees of Digital Science, a research technology company.


The 10th Peer Review Congress

How a Questionable Research Network Manipulated Scholarly Publishing

misinformation and disinformation

and scientometrics

informatics

misconduct

bibliometrics

<p>Leslie D. McIntosh,<sup>1</sup><span lang="ar-SA"> Hélène Draux</span>,<sup>1</sup> Elizabeth Smee,<sup>1</sup> Cynthia Hudson Vitale<sup>2</sup></p>
<h4>Objective </h4>
<p>To understand the broad impact of one questionable network on publishing, including the publishers, countries, and institutions involved and affected by the network’s activity.</p>
<h4>Design </h4>
<p>We examined the Pharmakon Neuroscience Research Network (PNN), a questionable entity listed in funding statements, acknowledgments, and author affiliations in articles published between 2019 and 2023. We selected the PNN because it was the first potential broad-scale case of scientific manipulation identified by one of the authors (L.D.M), and analyzing the PNN has helped the development of patterns to discover other unusual collaboration networks. All documents containing “Pharmakon Neuroscience” anywhere in the title, abstract, author affiliations, and full text were selected using Dimensions.<sup>1</sup> We excluded retraction and correction notices; articles for which a peer reviewer identified their organization as PNN; and articles with more than 25 authors. We inspected the metadata associated with the articles, notably the organizations related to the articles. We situated these practices within the Taxonomy of Scientific Manipulation,<sup>2</sup> systematically identifying (1) who was involved (authors, institutions, and publishers), (2) where<span class="CharOverride-9"> </span>mechanisms were used to manipulate science (journals, countries, and institutions), and (3) how the science was manipulated (methods for deceiving scholarly communication).</p>
<h4>Results </h4>
<p>Pharmakon Neuroscience was referenced in 140 published articles as of January 2025. After applying exclusion criteria, 123 articles remained. These articles included more than 6000 citations (citation ratio of 51 citations per article). The articles had 361 unique authors, including 29 without author identifiers (eg, ORCiD, Dimensions). The PNN involved 56 journals across 12 publishers, with Springer Nature (n = 29), Elsevier (n = 26), and Bentham Science Publishers (n = 25) having the most articles involved out of the total (n = 123). The involved authors were affiliated with 40 countries and 232 organizations. Of these organizations, 212 had research identifiers (eg, Research Organization Registry, Global Research Identifier Database), and 20 did not. Three of the organizations appeared to be personal homes, even though they were claimed as businesses or educational institutions. Additionally, one publication<sup>3</sup> highlighted the PNN as an affiliation among the “top 25 most predictive and impactful authors.”</p>
<h4>Conclusions </h4>
<p>This investigation leveraged publication databases and advanced forensic scientometric techniques to highlight critical research integrity and security vulnerabilities in a questionable research network. We use the term “questionable network” because this co-authorship network appears to have grown too rapidly to have developed naturally from scholarly affiliations over time. The impact of the PNN’s work might be legitimized through future studies and become seen as normative behavior. Although the most prolific PNN author and organization stem from one country, there is a global impact not confined to high- or low-income countries. While one individual appears central to the network, a single orchestrator does not imply sole responsibility. Instead, this case underscores the complexity of manipulating the scientific ecosystem, where questionable actors exploit the scholarly communication system through nonverifiable funders and organizations and artificially inflated citation metrics.</p>
<h4>References</h4>
<p>1. Hook DW, Porter SJ, Herzog C. Dimensions: building context for search and evaluation. <i>Front Res Metr Anal</i>. 2018;3:23. doi:10.3389/frma.2018.00023</p>
<p>2. McIntosh LD, White W, Vitale CH. Unveiling deception: establishing a taxonomic framework for disinformation within scientific discourse. <i>arXiv</i>. Preprint posted online November 19, 2023. doi:10.48550/arxiv.2311.11344</p>
<p>3. Sab MC, Ahmed KM. Tracing Bangladesh’s scientific footprints: a scientometric expedition into inflammation pharmacology. <i>Inform Res Comm</i>. 2024;1(1):9-21. doi:10.5530/irc.1.1.4</p>
<sup>1</sup>Digital Science, London, UK, l.mcintosh@digital-science.com; <sup>2</sup>Johns Hopkins University, Baltimore, MD, US. 
<h4>Conflict of Interest Disclosures</h4> Leslie D. McIntosh, Hélène Draux, and Elizabeth Smee are employees of Digital Science, a research technology company.


technical paper

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Michael O. Mensah,<sup>1</sup> Anand R. Habib,<sup>2</sup> Jacob Kendall-Taylor,<sup>3</sup> Mya Roberson,<sup>4</sup> Kanade Shinkai,<sup>4</sup> Annette Flanagin,<sup>3</sup> Preeti Malani<sup>3</sup></p>
<h4>Objective </h4>
<p>In March 2023, the JAMA Network invited peer reviewers to check a box when concerned about elements of equity, diversity, and inclusion (EDI) in manuscripts, and to explain such concerns in confidential comments to the editor. We examined peer reviewers’ use of the EDI concerns checkbox and categorized concerns described in confidential comments.</p>
<h4>Design </h4>
<p>In this cross-sectional, mixed-methods analysis, we initially examined all reviewed manuscripts for 13 JAMA Network journals between March 28, 2023, and December 31, 2024, and determined counts and percentages of reviews that utilized an EDI concerns checkbox. As a case study, we then used rigorous and accelerated data reduction (RADaR) to analyze deidentified, confidential reviewer comments for <i>JAMA Dermatology</i> manuscripts. RADaR develops all-inclusive data tables, then iteratively reduces those tables until only research question–relevant data remain.<sup>1</sup> Two former <i>JAMA</i> editorial fellows independently classified confidential comments as either relevant or irrelevant to EDI and then inductively coded comments. The coders agreed on which code(s) best described each comment and then distilled prominent categories among all comments. This study was deemed exempt by Yale School of Medicine’s institutional review board and follows STROBE and SRQR guidelines.</p>
<h4>Results </h4>
<p>Among 39,151 reviews across all JAMA Network journals, 5.3% (2075 of 39,151) utilized the EDI concerns checkbox, including 5.5% (1897 of 34,542) of research manuscript reviews. <i>JAMA Otolaryngology</i> reviews utilized the EDI checkbox least often (2.63% [29 of 1104] overall; 2.7% [26 of 948] of research manuscripts). <i>JAMA Pediatrics</i> reviews utilized the EDI checkbox most often (8.14% [146 of 1793]; 8.0% [126 of 1583] of research manuscripts). <i>JAMA Dermatology</i> reviews utilized the EDI checkbox at the median frequency among all JAMA Network journals (4.84% [91 of 1879]; 5.4% [78 of 1439] of research manuscripts) (<b>Table 25-1090</b>). Among <i>JAMA Dermatology </i>reviews utilizing the EDI checkbox, 34.1% (31 of 91) had confidential comments relevant to EDI, 50.5% (46 of 91) were irrelevant to EDI, and 15.4% (14 of 91) had no comment. Regarding EDI-relevant confidential comments, categories of responses included lack of clarity regarding race and ethnicity categories (48.4% [15]), statistical or methodological concerns (16.1% [5]), skin tone representation (9.7% [3]), ambiguity in language around demographic identifiers (9.7% [3]), and inappropriate use of gender ascribed by observers (6.5% [2]).</p>

![](https://assets.underline.io/markdown_image/1/image/d05d9cd83a84ebec0b4dfb72e01f8163.png)

<h4>Conclusions </h4>
<p>Overall,<span class="CharOverride-4"> </span>the EDI concerns checkbox was infrequently used across the JAMA Network journals. Among <i>JAMA Dermatology</i> reviews using the EDI concerns checkbox, less than half had confidential comments relevant to EDI, perhaps because instructions to reviewers did not define EDI concerns. Prominent categories among EDI-relevant comments included concerns about proper usage of race and ethnicity categories, absence of photos depicting dermatologic conditions in darker skin tones, and use of inclusive language regarding sex and gender, consistent with equity issues previously noted by the American Academy of Dermatology.<sup>2</sup> Future work could explore ways of systematically capturing EDI concerns from reviewers and their eventual influence on manuscripts accepted for publication.</p>
<h4>References</h4>
<p>1. Watkins DC. Rapid and rigorous qualitative data analysis: the “RADaR” technique for applied research. <i>Int J Qualitative Methods</i>. 2017;16(1):1609406917712131. doi:10.1177/1609406917712131</p>
<p>2. American Academy of Dermatology Association. Diversity and the academy. Accessed June 8, 2025. <a href="https://www.aad.org/member/career/diversity">https://www.aad.org/member/career/diversity</a></p>
<sup>1</sup>Department of Psychiatry, Yale School of Medicine, New Haven, CT, US, michael.mensah@yale.edu; <sup>2</sup>National Clinician Scholars Program, Yale School of Medicine, New Haven, CT, US; <sup>3</sup>JAMA and the JAMA Network, Chicago, IL, US; <sup>4</sup>JAMA Dermatology, Chicago, IL, US. 
<h4>Conflict of Interest Disclosures </h4>
Michael O. Mensah is co-editor of the Race and Mental Health Equity column in Psychiatric Services, unrelated to this work. Mya Roberson receives consulting fees from the National Committee for Quality Assurance outside the scope of the submitted work. Annette Flanagin is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.


An Analysis of Equity, Diversity, and Inclusion Concerns From JAMA Network Peer Reviewers

<p>Anne Lasinsky,<sup>1</sup> James Wrightson,<sup>2</sup> Matthew Hogel,<sup>3</sup> Alannah Brown,<sup>3</sup> Adrian Mota,<sup>3</sup> Karim M. Khan,<sup>1,2,4</sup> Clare L. Ardern<sup>5,6</sup></p>
<h4>Objective</h4>
<p> A small number of Canadian research funders have implemented interventions to address known biases in grant peer review. Equalization, which aims to match the proportion of funded grant applications to the proportion of submitted applications from specific underrepresented groups, is one such intervention. In 2016, the Canadian Institutes of Health Research (CIHR) implemented equalization for early career researcher (ECR) principal applicants in its Project Grant Competition. In 2021, equalization expanded to female principal applicants and French-language applicants. The objective of this study was to describe the outcome of equalization in 2022.</p>
<h4>Design</h4>
<p> This was a retrospective analysis of the number of funded grants in the spring 2022 CIHR Project Grant Competition. The equalization intervention was applied to applications submitted by ECR applicants, female principal applicants, and applications in French. To apply the intervention, first, scores for all applications submitted to the Project Grant Competition, which were reviewed by 59 committees, were converted to a percentage rank to account for scoring differences across committees. CIHR funded applications in rank order from the top percent rank, as far down as the competition budget allowed, and intervened to ensure that the next-ranked applications from each underrepresented applicant group were funded. Equalization matched the proportion of applications funded to the proportion of applications submitted by each group. No grants were defunded. Any grants that were equalized were added to the pool of funded grants—they did not displace another applicant’s score-win. This study descriptively analyzed routinely collected data from CIHR. The main outcome was the number of grant applications funded for each underrepresented applicant group, with and without equalization. The secondary outcome was the proportion of funded applications for each underrepresented applicant group, with and without equalization.</p>
<h4>Results</h4>
<p> There were 2095 applications submitted to the spring 2022 Project Grant Competition. Before equalization, 370 applications (17.7%) were funded. After equalization, 405 applications (19.3%) were funded with a total of CAD$325 million. ECR principal applicants submitted 580 applications (27.7%), female principal applicants submitted 774 applications (36.9%), and 25 applications (1.2%) were submitted in French). After equalization, 21 additional applications from ECR principal applicants and 22 applications from female principal applicants were funded (<b>Table 25-1092</b>). The funding success rates increased from 16.9% to 20.7% for ECR principal applicants and 17.5% to 20.3% for female principal applicants. One additional French-language application was funded with equalization; the success rate for French-language applicants increased from 17.4% to 21.7%.</p>

![](https://assets.underline.io/markdown_image/1/image/7ed3f1babc4b418643dbb1605b2d00b4.png)

<h4>Conclusions</h4>
<p> In the spring 2022 CIHR Project Grant Competition, equalization increased the number of health research grants awarded and the funding success rate for ECR and female principal applicants, and for applications submitted in French.</p>
<h4>Affiliations</h4> <sup>1</sup>School of Kinesiology, The University of British Columbia, Vancouver, Canada; <sup>2</sup>Department of Family Practice, The University of British Columbia, Vancouver, Canada; <sup>3</sup>Canadian Institutes of Health Research, Ottawa, Canada; <sup>4</sup>Canadian Institutes of Health Research-Institute of Musculoskeletal Health and Arthritis, Vancouver, Canada; <sup>5</sup>Department of Physical Therapy, The University of British Columbia, Vancouver, Canada, clare.ardern@ ubc.ca; <sup>6</sup>Sport and Exercise Medicine Research Centre, La Trobe University, Melbourne, Australia. <h4>Conflict of Interest Disclosures</h4> Matthew Hogel is Deputy Director, Funding Analytics at the Canadian Institutes of Health Research (CIHR). Alannah Brown is Senior Advisor to the Associate Vice-President at CIHR. Adrian Mota is Acting Vice President, Research—Programs at CIHR. Karim M. Khan is Scientific Director for CIHR’s Institute of Musculoskeletal Health and Arthritis (20172025). No other conflicts were reported. <h4>Funding/Support</h4> This work was supported by a CIHR Research Operating Grant (Scientific Directors) held by Karim M. Khan. CIHR’s Funding Analytics coordinated data management and analysis as part of its mandate to foster and deliver high quality peer review for health research in Canada. <h4>Role of the Funder/Sponsor</h4> CIHR did not participate in preparing the abstract, nor in the decision to submit the abstract for presentation.



Assessment of an Intervention to Equalize the Proportion of Funded Grant Applications for Underrepresented Groups at the Canadian Institutes of Health Research

<p>Noémie Aubert Bonn,<sup>1,2</sup> Lukas Hughes-Noehrer<sup>1</sup></p>
<h4>Objective </h4>
<p>The impact that research environments have on the quality of research has gained heightened visibility in the past few years. The Research Excellence Framework (REF), a UK-wide funding exercise that assesses higher education institutions (HEIs) on the excellence of their research, includes research environment as part of the elements it assesses. Recently, the weight of research environment was increased to one-fourth of the total score, bringing it closer to the other elements of research outputs and impacts. However, measuring positive research environments is challenging and clear indicators are still hard to delineate.</p>
<h4>Design </h4>
<p>To identify indicators that may be associated with positive research environments, we conducted a qualitative analysis of environment statements submitted to REF 2021.<sup>1</sup> We sampled 34 four-star statements drawn from different units of assessment (ie, academic disciplines) available openly on the REF website. Four-star scores are given to statements that describe “an environment that is conducive to producing research of world-leading quality and enabling outstanding impact, in terms of its vitality and sustainability.” Using these statements, 1 researcher conducted a mix of inductive (first 17 statements in NVivo 12) and deductive (last 17 statements) thematic analysis to extract the potential indicators of positive research environments. These indicators were organized in core themes, discussed with the research team, and compared with existing recommendations, policies, and literature on healthy research environments to build an exhaustive overview of possible indicators.</p>
<h4>Results </h4>
<p>The analysis  identified a vast array of elements, which were used to showcase positive research environments. We organized the elements extracted in indicators providing details on people, such as diversity metrics, staff and student characteristics, and measures to improve equity, diversity, and inclusion; indicators demonstrating positive research cultures, such as responsible leadership, collaborative culture, research integrity, and open research; and indicators demonstrating positive research environments, such as capacity-building measures, staff and financial support, employment security and progression, and responsible assessment practices (<b>Table 25-0889</b>). The breadth of indicators identified showcased differences and similarities in how different institutions and units of assessment qualified their research environments.</p>

![](https://assets.underline.io/markdown_image/1/image/8f8b88702af17ad7b2039178c62af22e.png)

<h4>Conclusions </h4>
<p>Our analysis provided insights on the breadth of elements that can come into play in creating a positive research environment. Despite being drawn from the UK landscape, our findings could be relevant to a broad diversity of research settings. For example, our findings could help support more comprehensive research assessment or they could help inspire concrete actions to improve research environments in HEIs.</p>
<h4>Reference</h4>
<p>1. Research Excellence Framework. Results and submissions. May 12, 2022. Accessed June 24, 2025. <a href="https://results2021.ref.ac.uk/">https://results2021.ref.ac.uk/</a></p>
<sup>1</sup>The University of Manchester, Department of Computer Science, Faculty of Science and Engineering, Manchester, UK, noemie. aubertbonn@manchester.ac.uk; <sup>2</sup>Hasselt University, Department of Healthcare & Ethics, Faculty of Medicine, Hasselt, Belgium. 
<h4>Conflict of Interest Disclosures</h4> Noémie Aubert Bonn was employed as a senior policy advisor with Research England from 2022 to 2023. 
<h4>Funding/Support</h4> The conceptualization, design, and part of the analysis was carried out and supported during Noémie Aubert Bonn’s employment at Research England. The analysis software subscription was provided by Hasselt University, Belgium. <h4>Role of the Funder/Sponsor</h4> The conclusions of the study are independent from Research England. 
<h4>Acknowledgment</h4> We wish to thank all the members of the People, Culture and Environment team at Research England who provided feedback and discussion throughout this project, particularly Catriona Firth, Claire Fraser, Steven Hill, Myles Furr, Marie-Helene Nienaltowski, Duncan Shermer, and Nicole Dixon. We also wish to thank everyone within UK Research and Innovation and The University of Manchester who provided feedback and insights on this project.


Extracting Research Environment Indicators From the UK Research Excellence Framework 2021 Statements

<p>Ahmad Sofi-Mahmudi,<sup>1</sup> Hesam Salmabadi<sup>2</sup></p>
<h4>Objective</h4>
<p> To evaluate the association between countries’ democratic status and scientific article retraction rates while exploring potential factors influencing this association.</p>
<h4>Design</h4>
<p> This retrospective cohort study analyzed publication and retraction data from 2006 to 2023. Data sources included the Retraction Watch Database for retraction data (n = 64,264 retractions), the Economist Intelligence Unit’s Democracy Index for democratic status, and SCImago for country-specific publication outputs (n = 59,385,751 publications). Our primary outcome was retraction rate (retractions per 10,000 publications) rather than absolute retraction counts, to account for varying publication volumes across countries. Using bayesian hierarchical negative binomial regression models with multiple imputations for missing data, we examined the association between the democracy index and article retractions while controlling for key confounders, including gross domestic product (GDP) per capita, research investment as percentage of GDP, English language proficiency scores, corruption control, government effectiveness, regulatory quality, rule of law, press freedom, and international collaboration rates, across 167 countries.</p>
<h4>Results</h4>
<p> The analysis of 167 countries demonstrated a median democracy index score of 5.71 (range, 1.05-9.81) and a median scientific publication retraction rate of 5.3 per 10,000 publications (range, 0-214.8). Higher democracy scores were associated with lower retraction rates (adjusted coefficient, −0.46; 95% credible interval [CrI], −0.59 to −0.32) after controlling for confounders (<b>Figure 25-1106</b>). Among institutional factors, rule of law showed a positive association (0.40; 95% CrI, 0.17-0.63), whereas corruption control demonstrated a negative association (−0.54; 95% CrI, −0.72 to −0.36) with retraction rates. The association between democracy and retractions remained stable across periods (interaction coefficient, −0.03; 95% CrI, −0.46 to 0.39), although press freedom effects varied significantly between the 2006 to 2012 and 2013 to 2023 periods (interaction coefficient, −1.40; 95% CrI, −1.77 to −1.03). GDP per capita (0.09; 95% CrI, 0.01-0.17) and government effectiveness (0.21; 95% CrI, 0.01-0.41) were positively associated with retraction rates, suggesting better detection and correction mechanisms in wealthier, more effectively governed countries.</p>

![](https://assets.underline.io/markdown_image/1/image/7d8b6c98d8de5538c342378e51b17ad2.png)

<h4>Conclusions</h4>
<p> Democratic status shows a negative association with scientific article retraction rates, although this association is complex and mediated by institutional factors. The findings suggest that while democratic societies may have lower rates of problematic research requiring retraction, they also demonstrate stronger institutional capacity for detecting and addressing research misconduct.</p>
<h4>Affiliations</h4> <sup>1</sup>Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada, a.sofimahmudi@ gmail.com; <sup>2</sup>Department of Environmental Sciences, University of Quebec in Trois-Rivières, Trois-Rivières, Quebec, Canada. <h4>Conflict of Interest Disclosures</h4> Ahmad Sofi-Mahmudi is an employee of Cytel Canada Health Inc. <h4>Additional Information</h4> We used Claude, 3.5 Sonnet (claude-35-sonnet-20241022) through its website for refining the analysis codes. Also, it was used as a grammar and flow checker of the abstract.


Retractions and Democracy Index Scores Across 167 Countries

<p>João Phillipe Cardenuto,<sup>1</sup> Daniel Moreira,<sup>2</sup> Anderson Rocha<sup>1</sup></p>
<h4>Objective</h4>
<p> To quantitatively analyze the types, contexts, and manipulation methods of problematic images that lead to retractions of scientific articles.</p>
<h4>Design</h4>
<p> This cross-sectional study analyzed retracted articles flagged for problematic image manipulation (eg, image duplication) in the Retraction Watch Database<sup>1</sup> (56,716 entries as of October 4, 2024). We focused on entries containing the term <i>image</i> in the retraction reason (8002 entries) and further refined the dataset to those discussed on PubPeer<sup>2</sup> (2078 after duplicate removal) to gain more detailed insights into the image problems. Data extracted included figure types (eg, microscopy, gel blot), the context of image misuse (eg, within-article, between-article), and the type of manipulation (eg, duplication, splicing).</p>
<h4>Results</h4>
<p> The Table presents our results (<b>Table 25-0965</b>). Gel blots (eg, Western blots) were the most frequently cited image type in problematic retractions, appearing in 1074 articles (51.68%). Between-article image reuse, where an image and its associated data are duplicated across different publications, was the most common context of misuse, identified in 1241 cases (59.72%). Notably, 982 retractions (47.28%) were attributed to paper mills. Image duplication was the predominant cause of retraction, accounting for 1827 cases (87.92%). Only 1 retraction was attributed to computer- or artificial intelligence–generated manipulation. While our analysis did not filter by the biomedical area, most problematic images originated from the biomedical domain.</p>

![](https://assets.underline.io/markdown_image/1/image/0773fe066647557e1654c0bd3b16d9a9.png)

<h4>Conclusions</h4>
<p> This study highlights the prevalence of gel blot images and between-article image duplication in retracted articles, indicating a potential benefit from specialized tools to detect such issues. During our analysis, we noticed a frequent lack of detailed and standardized information in retraction notices, which hinders efforts to understand and prevent the presented problems. While PubPeer data offer valuable insights when the retraction notices fail to do so, PubPeer posts are not official documents and may exhibit biases from their authors, which could result in speculative claims about an article. Because of that, to facilitate research and improve the integrity of the scientific record, future research should focus on discussing and developing better guidelines for comprehensive retraction notices that may even support computer-aided solutions.</p>
<h4>References</h4>
<p>1. The Retraction Watch Database. Accessed January 31, 2025. <a href="http://retractiondatabase.org/">http://retractiondatabase.org/</a></p>
<p>2. PubPeer. Accessed January 31, 2025. <a href="https://pubpeer.com"><span class="CharOverride-10">https://pubpeer.com</span></a></p>
<p>
  <sup>1</sup>Artificial Intelligence Lab, Recod.ai, Institute of Computing, Universidade Estadual de Campinas, Campinas, São Paulo, Brazil, phillipe.cardenuto@ic.unicamp.br; 
  <sup>2</sup>Department of Computer Science, Loyola University Chicago, Chicago, IL, US.
</p>

<h4>Conflict of Interest Disclosures</h4>
<p>None reported.</p>

<h4>Funding/Support</h4>
<p>This study was supported by the National Council for Scientific and Technological Development–CNPq (grant No. 442229/2024-0) and by the São Paulo Research Foundation–FAPESP (grant No. 2023/12865-8).</p>

<h4>Role of the Funder/Sponsor</h4>
<p>The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the abstract.</p>


Characterizing Problematic Images in Retracted Scientific Articles

<p>Reese A. K. Richardson,<sup>1,2</sup> Jeonghyun Moon,<sup>1</sup> Spencer S. Hong,<sup>1,3</sup> Luis A. Nunes Amaral<sup>1,2,4,5,6</sup></p>
<h4>Objective</h4>
<p> Materials science and engineering (MSE) research has, for the most part, escaped the doubts raised about the reliability of scientific literature by recent large-scale replication studies in psychology and cancer biology. However, users on postpublication peer review sites have recently identified dozens of articles where the make and model of the scanning electron microscope (SEM) listed in the text does not match the instrument’s metadata visible in the images in the published article.</p>
<h4>Design </h4>
<p>To systematically investigate this potential risk to the MSE literature, we developed a semiautomated approach using optical character recognition to scan published figures for instrumental metadata banners and check the information contained in these banners on instrument manufacturer and model against the SEM instrument identified in the text. We validated our approach by evaluating model performance on known instances of SEM misidentification and manual reevaluation of novel instances of SEM misidentification returned by the pipeline. This analysis was performed in January 2024.</p>
<h4>Results</h4>
<p> Starting from a set of 1,067,102 articles published since 2010 (as indexed by OpenAlex, downloaded on February 27, 2023) in 50 MSE journals (selected according to the publisher’s permissiveness of text and data mining and to represent a broad range of subdisciplines and impact factors), we identified 11,314 articles for which SEM make and model could be identified in an image’s metadata. For 2400 of those articles (21.2%), the image metadata did not match the SEM manufacturer or model listed in the text, and for another 2799 (24.7%), at least some of the instruments used in the study were not reported. We found that articles with SEM misidentification were more likely to have existing PubPeer comments than articles without (43 of 2400 [1.8%] vs 984 of 171,646 [0.5%]; <i>P</i> = 9.98 × 10<sup>-15</sup>) and that electrochemistry articles that featured SEM misidentification, compared with those that did not have SEM misidentification, were more likely to make a crucial error in their determination of optical band gap (41 of 154 [26.6%] vs 100 of 751 [13.3%]; <i>P</i> = 3.35 × 10<sup>-5</sup>).</p>
<h4>Conclusions</h4>
<p> Our results suggest that SEM misidentification can be a valuable indicator of irreproducibility in the results of extant scientific articles. We recommend that peer reviewers, before and after publication, remain attentive to the accurate reporting of instruments used in scientific articles. Our selection criteria for journals and the high false-negative rate of our pipeline limit the extent to which our findings are representative of the MSE literature at large. Unexplained textual similarities (eg, the same unlikely typo appearing in dozens of articles) common to many of these articles suggest the involvement of paper mills, organizations that mass produce, sell authorship on, and publish fraudulent scientific manuscripts at scale.</p>
<p>
  <sup>1</sup>Department of Engineering Sciences and Applied Mathematics, Northwestern University, Evanston, IL US, richardsonr43@gmail.com; <sup>2</sup>Department of Molecular Biosciences, Northwestern University, Evanston, IL US; <sup>3</sup>Department of Chemical and Biological Engineering, Northwestern University, Evanston, IL US; <sup>4</sup>Department of Physics and Astronomy, Northwestern University, Evanston, IL, US; <sup>5</sup>Northwestern Institute on Complex Systems, Northwestern University, Evanston, IL, US; <sup>6</sup>NSF-Simons National Institute for Theory and Mathematics in Biology, Chicago, IL, US.
</p>

<h4>Conflict of Interest Disclosures</h4>
<p>None reported.</p>

<h4>Funding/Support</h4>
<p>
  Reese A. K. Richardson was supported in part by the National Institutes of Health Training grant T32GM008449 through Northwestern University’s Biotechnology Training Program; Reese A. K. Richardson gratefully acknowledges funding from the Dr. John N. Nicholson fellowship from Northwestern University and Moderna Inc. Spencer S. Hong gratefully acknowledges support from the Ryan Fellowship and the International Institute for Nanotechnology at Northwestern University; Luis A. Nunes Amaral gratefully acknowledges funding from SCISIPBIO: a data-science approach to evaluating the likelihood of fraud and error in published studies (grant No. 1956338). 

<h4>Role of the Funder/Sponsor</h4> The funders had no role in the design or execution of this study.



Misidentification of Scanning Electron Microscope Instruments in the Peer-Reviewed Materials Science and Engineering Literature

<p>Iakovos Evdaimon,<sup>1</sup> John P. A. Ioannidis,<sup>2</sup> Giannis Nikolentzos,<sup>3</sup> Michail Chatzianastasis,<sup>1</sup> George Panagopoulos,<sup>4</sup> Michalis Vazirgiannis<sup>1,5</sup></p>
<h4>Objective</h4>
<p> Citation counts are widely used to evaluate research influence.<sup>1</sup> The h-index, while popular, does not account for how citations are distributed across articles or authors. This is critical when citations are strategically concentrated to boost perceived influence. We studied extreme citation concentration patterns that may distort bibliometric evaluations and merit further scrutiny.<sup>2</sup></p>
<h4>Design</h4>
We analyzed Scopus data up to March 2023, focusing on 1,496,680 authors with more than 5 full articles and at least 1000 citations. We used Scopus-calculated h-indices and citation counts and introduced 3 indicators of extreme citation concentration: C/h2 ratio, A<i><sub>50%C</sub></i> , and A<i><sub>50</sub></i> . For C/h2 ratio, a small value (first percentile, 2.45) indicates that the citations of the author may have been strategically placed to increase the h-index. This metric normalizes citation volume by what would be expected if the h-index reflected broad citation distribution. A<i><sub>50%C</sub></i> measured the number of distinct citing authors contributing at least 50% of an author’s total citations. Extremely low values (first percentile, 5) suggest reliance on few citing sources, possibly from self-citation or closed groups. A<i><sub>50</sub></i> measured the number of coauthors with whom an author has published more than 50 articles. Extremely high values (99th percentile, more than 7) may indicate large-scale collaborations and dense citation clusters. Thresholds were based on the 1% extremes of each metric’s distribution. Authors in physics and astronomy were excluded from A<i><sub>50</sub></i> and A<sub>50%C</sub> analyses due to the influence of nuclear and particle physics. Field definitions followed the ScienceMetrix classification.
<h4>Results</h4>
<p> In the C/h<sup>2</sup> analysis, 14,967 authors (1%) fell below 2.45. These authors had a median of 63 articles, all with h-indices exceeding 21. Fields such as chemistry and earth and environmental sciences showed 1.5-fold higher representation of authors with extremely low C/h<sup>2</sup> scores (<b>Table 25-0974</b>). In the A<i><sub>50%C</sub></i> analysis, 11,277 authors (1%) had 5 or fewer citing authors accounting for 50% of their citations. The median number of articles was 104, and most authors had an h-index greater than 20. Fields with the highest enrichment included chemistry and mathematics and statistics. In the A<i><sub>50</sub></i> analysis, 12,015 authors (1%) had more than 7 highly recurrent coauthors (50 or more shared publications). These authors were exceptionally prolific (median, 596 articles) and had very high h-indices (median, 56; 12.8% with an h-index greater than 100). A total of 73.3% were in clinical medicine. Co-occurrence analysis revealed a strong association between low C/h<sup>2</sup> and low A<i><sub>50%C</sub></i> values (odds ratio, 6.4; 95% CI, 5.9-6.9). There was modest coexistence for extremely low A<i><sub>50%C</sub></i> and extremely high A<i><sub>50</sub></i> values (odds ratio, 1.5; 95% CI, 1.3-1.7). Extremely low C/h<sup>2</sup> very rarely co-occurred with extremely high A<i><sub>50</sub></i> values (odds ratio, 0.09; 95% CI, 0.05-0.16).</p>

![](https://assets.underline.io/markdown_image/1/image/74d68afd39138897dab9923e970655c4.png)

<h4>Conclusions</h4>
<p> Extreme patterns of citation concentration can substantially distort citation-based metrics, especially the h-index. While not implying misconduct, these metrics highlight outliers where citation behavior diverges from norms. Such metrics may serve as red flags for further analysis.</p>
<h4>References</h4>
<p>1. Bornmann L. How are excellent (highly cited) papers defined in bibliometrics? a quantitative analysis of the literature. <i>Res Eval</i>. 2014;23(2):166-173. doi:10.1093/reseval/rvu002</p>
<p>2. Waltman L, van Eck NJ, van Leeuwen TN, Visser MS, van Raan AFJ. Towards a new crown indicator: some theoretical considerations. <i>J Informetrics</i>. 2011;5(1):37-47. doi:10.1016/j.joi.2010.08.001</p>
<sup>1</sup>LIX, École Polytechnique, Institut Polytechnique de Paris, Palaiseau, France; <sup>2</sup>Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, US, jioannid@stanford.edu; <sup>3</sup>Department of Informatics and Telecommunications, University of Peloponnese, Tripoli, Greece; <sup>4</sup>Department of Computer Science, University of Luxembourg, Esch-sur-Alzette, Luxembourg; <sup>5</sup>Mohamed bin Zayed University of Artificial Intelligence, Masdar City, Abu Dhabi, United Arab Emirates.
<h4>Conflict of Interest Disclosures</h4> John P. A. Ioannidis is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported. 
<h4>Additional Information</h4> Iakovos Evdaimon is a cocorresponding author (ievdaimon7@gmail.com).



Indicators of Small-Scale and Large-Scale Citation Concentration Patterns

<p>Reese A. K. Richardson,<sup>1,2</sup> Spencer S. Hong,<sup>1,3</sup> Jennifer A. Byrne,<sup>4,5</sup> Thomas Stoeger,<sup>6,7,8</sup> Luís A. Nunes Amaral<sup>1,2,9,10,11</sup></p>
<h4>Objective </h4>
<p>Some suggest that the ease of communication provided by the internet and open access publishing has created the conditions for the emergence of entities (including paper mills, brokers, and predatory publishers) that facilitate systematic scientific fraud. However, little is understood about the organization of these entities as well as how they react to and evade science integrity measures. Here, we sought to demonstrate that large networks that produce scientific fraud at scale can be identified by the footprints they have left in the published scientific literature. These footprints can be analyzed to glean insights about their organizational structure and operations.</p>
<h4>Design </h4>
<p>Our work consisted of observational case studies making use of article metadata from OpenAlex and the complete corpora of <i>PLOS One </i>and Hindawi articles; all PubPeer comments made before January 2024; historical indexing data from MEDLINE, Scopus, and Web of Science; and archived webpages obtained with the Internet Archive’s Wayback Machine. Our case studies addressed networks of compromised editors at large mega-journals, analysis of article provenance through observations of interarticle image duplication, a longitudinal study of the operations of a large broker organization, a comparison of revision and retraction rates in closely related biomedical subfields, and an overall assessment of growth rates of systematic scientific fraud. This analysis was performed from June to September 2024.</p>
<h4>Results </h4>
<p>We identified 45 editors at <i>PLOS One</i>, 53 editors at 10 Hindawi journals, and 205 Institute of Electrical and Electronics Engineers (IEEE) conferences that handled articles that were eventually retracted, articles with PubPeer comments, or articles featuring tortured phrases far more often than expected by chance (1-sided Poisson binomial test, Benjamini-Hochberg false discovery rate, &lt;0.05). We characterized a network of image duplication spanning 2213 articles. We catalogued the involvement of a broker organization with an evolving portfolio spanning 188 journals over 7 years. We found similar rates of revision in subfields of RNA biology but retraction rates orders of magnitude apart. We found that while publication rates of scientific articles doubled approximately every 15 years and the publication rates of eventually retracted articles every 3 years, articles of likely paper mill provenance were published at a rate that doubled less than every 2 years.</p>
<h4>Conclusions </h4>
<p>Here, we demonstrated through case studies that (1) individuals have colluded to publish problematic papers in a number of journals; (2) brokers can ensure publication in infiltrated journals at scale; and (3) within a field of science, not all subfields are equally targeted for scientific fraud. Our results revealed some of the strategies that enable entities promoting scientific fraud to evade interventions. Our final analysis suggested that this ability to evade interventions is enabling the number of fraudulent publications to grow at a rate far outpacing that of legitimate science.</p>
<h4>Author Affiliations</h4>
<sup>1</sup>Department of Engineering Sciences & Applied Mathematics, Northwestern University, Evanston, IL, US, richardsonr43@gmail.com; <sup>2</sup>Department of Molecular Biosciences, Northwestern University, Evanston, IL, US; <sup>3</sup>Department of Chemical & Biological Engineering, Northwestern University, Evanston, IL, US; <sup>4</sup>School of Medical Sciences, Faculty of Medicine and Health, The University of Sydney, NSW, Australia; <sup>5</sup>NSW Health Statewide Biobank, NSW Health Pathology, Camperdown, NSW, Australia; <sup>6</sup>The Potocsnak Longevity Institute, Northwestern University, Chicago, IL, US; <sup>7</sup>Simpson Querrey Lung Institute for Translational Science, Northwestern University, Chicago, IL, US; <sup>8</sup>Division of Pulmonary and Critical Care, Northwestern University Feinberg School of Medicine, Chicago, IL, US; <sup>9</sup>Department of Physics and Astronomy, Northwestern University, Evanston, IL, US; <sup>10</sup>Northwestern Institute on Complex Systems, Northwestern University, Evanston, IL, US; <sup>11</sup>NSF-Simons National Institute for Theory and Mathematics in Biology, Chicago, IL, US.
<h4>Conflict of Interest Disclosures</h4>
Luís A. Nunes Amaral is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.

<h4>Funding/Support</h4>
Reese A. K. Richardson was supported in part by a National Institutes of Health Training Grant (T32GM008449) through Northwestern University’s Biotechnology Training Program and receives funding from the Dr John N. Nicholson Fellowship from Northwestern University and Moderna. Spencer S. Hong receives support from the Ryan Fellowship and the International Institute for Nanotechnology at Northwestern University. Jennifer A. Byrne receives funding from a National Health and Medical Research Council of Australia Ideas grant (APP1184263) and Rewarding Research Success funding from the Faculty of Medicine and Health at The University of Sydney. Thomas Stoeger receives funding from the National Institute on Aging, Integrative Multi-Scale Systems Analysis of Gene-Expression-Driven Aging Morbidity (R00AG068544), National Institute of Allergy and Infectious Diseases (AI135964), and Successful Clinical Response In Pneumonia Therapy (SCRIPT) Systems Biology Center. Luís A. Nunes Amaral and Thomas Stoeger receive funding from SCISIPBIO: a data-science approach to evaluating the likelihood of fraud and error in published studies (1956338).
<h4>Role of the Funder/Sponsor</h4>
The funders had no role in the design or execution of this study.



Scale and Resilience in Organizations Enabling Systematic Scientific Fraud

<p>Anna Abalkina,<sup>1</sup> Svetlana Kleiner<sup>2</sup></p>
<h4>Objective </h4>
<p>Paper mills pose a significant challenge to the scientific community, yet knowledge of paper mills remains fragmented. An estimated 400,000 studies originate from paper mills,<sup>1</sup> although only 56,000 have been retracted or corrected. This study investigated the patterns of paper-mill articles and the challenges of their retraction, using a sample from Tanu.pro,<sup>2</sup> one of Europe’s largest paper mills.</p>
<h4>Design</h4>
<p> Tanu.pro creates unique emails to submit papers, often leading to discrepancies between the country domains and authors’ affiliations. Employing a snowball method, the study identified more than 60 suspicious email domains, such as tanu.pro (which gave the paper mill its title in the literature). To retrieve Tanu.pro articles, we conducted a full-text search in Scopus on these domains on December 14, 2024. Combining both outsider and insider perspectives, we analyzed the retraction challenges, focusing on Tanu.pro papers published in Springer journals. A sample of 77 Springer articles was used to examine scholars’ positions and compare the results with a control group of 77 randomly selected articles without paper-mill hallmarks but with the matching indicators (year, journal, number of coauthors). The ranks and positions of all authors in both groups were examined, where possible, at the time of submission.</p>
<h4>Results </h4>
<p>The study detected 1517 papers published between 2017 and 2025 in 380 journals and coauthored by more than 4500 scholars from 46 countries and over 460 universities. While a problematic email was not definitive proof of malpractice, there was evidence that some Tanu.pro papers were associated with an author number anomaly (<b>Figure 25-1113</b>), poor empirical analysis, irrelevant citations, data fabrication, errors, fake reviewer suggestions, compromised special issues, and translated plagiarism. The Springer articles were initially identified by authors’ emails; an additional selection criterion was the presence of suggested reviewers. Tanu.pro papers had a higher proportion of scholars occupying top university positions, such as professors or department heads, than in the control group (44% vs 30%). A problematic email as the only evidence posed challenges for action by journals when other issues are absent: 20 of the 77 Tanu.pro articles (26%) could not be retracted under current Committee on Publication Ethics (COPE) guidelines. Editors also felt more reluctant to retract when the only red flags were emails and unused suggested reviewers.</p>

![](https://assets.underline.io/markdown_image/1/image/05c9b5fc3034d8ef288f41a58baaaf9c.png)

<h4>Conclusions </h4>
<p>The study has shown that specific paper mill patterns are useful when investigating problematic articles. However, some red flags are insufficient grounds for paper retraction due to limited clarity of COPE guidelines that place some articles in the gray zone if they do not feature strong grounds for retraction. More retractable categories are necessary so that COPE members can mark dubious research more easily, eg, undisclosed third-party submission. Although our recommendations are limited by the analysis of one paper mill, COPE guidelines need regular updating considering patterns of other paper mills.</p>
<h4>References</h4>
<p>1. Van Noorden R. How big is science’s fake-paper problem? <i>Nature</i>. 2023;623(7987):466-467. doi:10.1038/d41586-023-03464-x</p>
<p>2. Bishop D, Abalkina A. Paper mills: a novel form of publishing malpractice affecting psychology. <i>Meta-Psy</i>. March 17, 2024. <a href="https://open.lnu.se/index.php/metapsychology/article/view/3422">https://open.lnu.se/index.php/metapsychology/article/view/3422</a></p>
<h4>Affiliations</h4> <sup>1</sup>Freie Universität Berlin, Berlin, Germany, abalkina@gmail.com; <sup>2</sup>Springer Nature, Dordrecht, the Netherlands. <h4>Conflict of Interest Disclosures</h4> None reported.


Patterns of Paper Mill Papers and Retraction Challenges

<p>Renee V. Hoch,<sup>1</sup> Emily J. Chenette<sup>1</sup></p>
<h4>Objective </h4>
<p><i>PLOS One </i>has experienced a sharp rise in paper mills and poor-quality submissions since 2021 and has taken an agile, multifaceted preventive approach to address these issues at scale. This quality improvement study evaluated trends in large-scale integrity issues (ie, megacases) at <i>PLOS One</i> between 2021 and 2024 and the impacts of preventive initiatives on the journal’s ability to address megacase concerns prior to publication.</p>
<h4>Design</h4>
<p> We analyzed temporal and geographic trends, issue types, and research areas for megacases detected at <i>PLOS One</i> between 2021 and 2024. We then evaluated changes to desk reject rates following specific interventions as well as changes to the pre- vs postpublication impacts of megacases over time.</p>
<h4>Results</h4>
<p> Between 2021 and 2024, <i>PLOS One</i> detected 50 megacases that cumulatively involved more than 7000 submissions and publications, with concerns including paper mills (42%; 21 of 50), authorship (22%; 11 of 50), peer review (52%; 26 of 50), and policy compliance (12%; 6 of 50). Megacases spanned multiple research areas, with the highest representation in the physical sciences and engineering (42%; 21 of 50) and medicine and public health (18%; 9 of 50). Most megacases involved submissions from geographically clustered author groups; megacases were received from 19 countries, with the highest representation from Pakistan (20%; 10 of 50), China (18%; 9 of 50), and India (10%; 5 of 50). The number of megacases detected by PLOS increased from 2 (214 articles) in 2021 to 25 (4310 articles) in 2024, and during this time the journal’s desk reject rate more than doubled, from 13% (5188 of 40,548) in 2021 to 27% (16,160 of 59,425) in 2024. In 2024 alone, we rejected more than 3000 submissions flagged for megacase concerns. Desk reject data in periods before and after specific preventive interventions indicated that each of these efforts increased the journal’s ability to detect and/or reject concerning content (<b>Table 25-0967</b>
	). We quantified pre- vs postpublication megacase impacts using a preventive efficacy metric, defined as the number of submissions flagged for megacase concerns (ie, prepublication impact) divided by the number of affected publications (ie, postpublication impact). <i>PLOS One</i>’s preventive efficacy increased from 0.70 in 2022 to 9.09 in 2024; the cumulative number of affected articles increased substantially during this time, but articles of concern were increasingly rejected prepublication, and the postpublication impact actually decreased (530 affected publications in 2022 vs 427 affected publications in 2024).</p>
	
![](https://assets.underline.io/markdown_image/1/image/b1311984a7ead8b037c781f0545fc4ac.png)

<h4>Conclusions</h4>
<p> Large-scale integrity issues have risen considerably in recent years, affecting a broad array of subject areas and geographies. Through varied approaches, <i>PLOS One</i> has improved prepublication detection and rejection of content linked to megacases. This reduces the volume of unreliable publications, the resources required to address these integrity issues, and the impacts of megacases on peer review. Such scalable preventive approaches are crucial to the long-term sustainability of integrity work in an era of increasing paper mill activity.</p>
<h4>Affiliations</h4> <sup>1</sup>PLOS (Public Library of Science), Cambridge, UK, and San Francisco, CA, US, rhoch@plos.org. <h4>Conflict of Interest Disclosures</h4> Renee V. Hoch is an employee of PLOS; the head of the PLOS Publication Ethics team; a member of the Committee on Publication Ethics (COPE) Council, a United2Act Working Group, and an STM Task & Finish Working Group that developed guidance pertaining to large-scale integrity cases; a former member of the COPE Paper Mills Working Group; and a coauthor of COPE guidance on addressing large-scale paper mill issues. Emily J. Chenette is an employee of PLOS and is the editor in chief of <i>PLOS One</i> and head of PLOS editorial board services. <h4>Acknowledgment</h4> We thank the <i>PLOS One</i> Editorial and Publication Ethics teams for their work identifying and addressing large-scale cases, the PLOS editorial board management team for their help investigating and offboarding editorial board members of concern, and the PLOS Digital team for their help developing data analytics and technology tools needed for this work. Thanks also to Suzanne Farley, Rebecca Kirk, Zena Nyakoojo, and Dena Emmerson for their input on this abstract. <h4>Additional Information</h4> PLOS data and resources were used for this study, and others within PLOS Editorial provided input and contributed to the work underlying the datasets. PLOS was not otherwise involved in the reported study.



Downloads

Next from The 10th Peer Review Congress

An Analysis of Equity, Diversity, and Inclusion Concerns From JAMA Network Peer Reviewers

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from The 10th Peer Review Congress

An Analysis of Equity, Diversity, and Inclusion Concerns From JAMA Network Peer Reviewers

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads