United States

&lt;p&gt;Nicola Di Girolamo,&lt;sup&gt;1,2&lt;/sup&gt; Reint Meursinge Reynders,&lt;sup&gt;3,4&lt;/sup&gt; Ugo Di Girolamo&lt;sup&gt;5&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;Consolidated Standard of Reporting of Trials (CONSORT) checklists help authors to adhere to reporting standards and allow editors and peer reviewers to ensure that critical information is reported in a randomized controlled trial (RCT) at a glance.&lt;sup&gt;1&lt;/sup&gt; We pragmatically assessed the ability of 3 artificial intelligence (AI) large language model (LLM) chatbots provided with minimal instructions to properly complete a CONSORT checklist from manuscripts reporting RCTs before peer review in 2025.&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;This cross-sectional study was registered on the Open Science Framework.&lt;sup&gt;2&lt;/sup&gt; PDFs of manuscripts reporting RCTs before peer review that were consecutively published in &lt;i&gt;The BMJ&lt;/i&gt; were extracted in reverse-chronological order for a total of 50 manuscripts. &lt;i&gt;The BMJ&lt;/i&gt; granted permission for this study. In February 2025, each PDF was uploaded exactly as submitted by the authors to 3 LLMs (ChatGPT-4o, Gemini Advanced 2.0 Flash, and Claude 3.5 Sonnet), which were instructed to complete the CONSORT checklist in a table format,&lt;sup&gt;3&lt;/sup&gt; including the reporting of the 37 CONSORT items and extraction of the text where the item was reported. The number of iterations and total time (seconds) required to obtain a visually acceptable table were recorded. Sensitivity, specificity, and accuracy for each LLM were calculated with human manual extraction as the reference standard. Manual extraction was performed by 1 operator (N.D.) with expertise in RCT methods. Generalized linear mixed models were built to evaluate the effect of PDF characteristics (number of pages, file size, and number of words) on the disagreements between AI and human assessment.&lt;/p&gt;
&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;Of the 1850 CONSORT items, ChatGPT provided decisions for 1776 (96.0%); 1159 of these were in agreement with human extraction (overall accuracy, 65.3% [95% CI, 63.0%-67.5%]; sensitivity, 61.3% [95% CI, 55.0%-67.3%]; specificity, 65.9% [95% CI, 63.5%-68.3%]). Gemini provided decisions for 1629 items (88.0%), of which 1521 were in agreement with human extraction (accuracy, 93.4% [95% CI, 92.1%-94.5%]; sensitivity, 97.6% [95% CI, 94.8%-99.1%]; specificity, 92.6% [95% CI, 91.1%-94.0%]). Claude provided decisions for 1480 items (80.0%), of which 1375 were in agreement with human extraction (accuracy, 92.9% [95% CI, 91.5%- 94.2%]; sensitivity, 56.1% [95% CI, 49.3%-62.8%]; specificity, 99.4% [95% CI, 98.8%-99.7%]). Accuracy of LLMs was particularly low for certain CONSORT items (&lt;b&gt;Figure 25-1152&lt;/b&gt;). There were no significant associations between PDF characteristics and AI and human disagreements. Trial PDFs were a mean of 49 pages and 17,268 words. LLMs were able to compile visually acceptable CONSORT checklists on the first iteration in 76.7% (115 of 150) attempts: ChatGPT for 40 RCTs, Gemini for 36 RCTs, and Claude for 39 RCTs. Generation of a CONSORT checklist required a mean of 75 seconds, 38 seconds, and 47 seconds for ChatGPT, Gemini, and Claude, respectively (range, 16-249 seconds).&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/39e42ee902f1b80f6a363027b6df1d1e.png)

&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;In early 2025, the LLMs Gemini and Claude, but not ChatGPT, were able to generate accurate CONSORT checklists from PDFs of manuscripts reporting RCTs before peer review in over 90% of cases, given minimal instructions and often in less than 1 minute.&lt;/p&gt;
&lt;h4&gt;References&lt;a id=&quot;_idTextAnchor008&quot;&gt;&lt;/a&gt;&lt;a id=&quot;_idTextAnchor009&quot;&gt;&lt;/a&gt;&lt;/h4&gt;
&lt;p&gt;1. Moher D, Jones A, Lepage L; CONSORT Group (Consolidated Standards for Reporting of Trials). Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. &lt;i&gt;JAMA&lt;/i&gt;. 2001;285(15):1992-1995. doi:10.1001/jama.285.15.1992&lt;/p&gt;
&lt;p&gt;2. Di Girolamo N, Meursinge Reynders R, Di Girolamo U. Comparison of different AI large language models for extraction of CONSORT items from submitted randomized controlled trials before peer-review. Open Science Framework. Registered February 13, 2025. Accessed July 5, 2025. &lt;a href=&quot;https://osf.io/5v6h2&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://osf.io/5v6h2&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;3. Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. &lt;i&gt;BMJ&lt;/i&gt;. 2010;340:c332. doi:10.1136/bmj.c332&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Department of Clinical Sciences, College of Veterinary Medicine, Cornell University, Ithaca, NY, US, nd374@cornell.edu; &lt;sup&gt;2&lt;/sup&gt;Journal of Small Animal Practice, British Small Animal Veterinary Association, Gloucestershire, UK; &lt;sup&gt;3&lt;/sup&gt;Department of Oral and Maxillofacial Surgery, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, the Netherlands; &lt;sup&gt;4&lt;/sup&gt;Private Practice of Orthodontics, Milan, Italy; &lt;sup&gt;5&lt;/sup&gt;Compass, New York, NY, US.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures &lt;/h4&gt;
&lt;p&gt;Nicola Di Girolamo is editor in chief of 2 peer-reviewed journals, one published by Elsevier and one by Wiley. No other disclosures were reported.&lt;/p&gt;


The 10th Peer Review Congress

Pragmatic Assessment of Different AI Large Language Models for Extraction of CONSORT Items From Randomized Controlled Trials Before Peer Review

artificial intelligence and large language models

quality of trials

peer review

<p>Nicola Di Girolamo,<sup>1,2</sup> Reint Meursinge Reynders,<sup>3,4</sup> Ugo Di Girolamo<sup>5</sup></p>
<h4>Objective </h4>
<p>Consolidated Standard of Reporting of Trials (CONSORT) checklists help authors to adhere to reporting standards and allow editors and peer reviewers to ensure that critical information is reported in a randomized controlled trial (RCT) at a glance.<sup>1</sup> We pragmatically assessed the ability of 3 artificial intelligence (AI) large language model (LLM) chatbots provided with minimal instructions to properly complete a CONSORT checklist from manuscripts reporting RCTs before peer review in 2025.</p>
<h4>Design </h4>
<p>This cross-sectional study was registered on the Open Science Framework.<sup>2</sup> PDFs of manuscripts reporting RCTs before peer review that were consecutively published in <i>The BMJ</i> were extracted in reverse-chronological order for a total of 50 manuscripts. <i>The BMJ</i> granted permission for this study. In February 2025, each PDF was uploaded exactly as submitted by the authors to 3 LLMs (ChatGPT-4o, Gemini Advanced 2.0 Flash, and Claude 3.5 Sonnet), which were instructed to complete the CONSORT checklist in a table format,<sup>3</sup> including the reporting of the 37 CONSORT items and extraction of the text where the item was reported. The number of iterations and total time (seconds) required to obtain a visually acceptable table were recorded. Sensitivity, specificity, and accuracy for each LLM were calculated with human manual extraction as the reference standard. Manual extraction was performed by 1 operator (N.D.) with expertise in RCT methods. Generalized linear mixed models were built to evaluate the effect of PDF characteristics (number of pages, file size, and number of words) on the disagreements between AI and human assessment.</p>
<h4>Results </h4>
<p>Of the 1850 CONSORT items, ChatGPT provided decisions for 1776 (96.0%); 1159 of these were in agreement with human extraction (overall accuracy, 65.3% [95% CI, 63.0%-67.5%]; sensitivity, 61.3% [95% CI, 55.0%-67.3%]; specificity, 65.9% [95% CI, 63.5%-68.3%]). Gemini provided decisions for 1629 items (88.0%), of which 1521 were in agreement with human extraction (accuracy, 93.4% [95% CI, 92.1%-94.5%]; sensitivity, 97.6% [95% CI, 94.8%-99.1%]; specificity, 92.6% [95% CI, 91.1%-94.0%]). Claude provided decisions for 1480 items (80.0%), of which 1375 were in agreement with human extraction (accuracy, 92.9% [95% CI, 91.5%- 94.2%]; sensitivity, 56.1% [95% CI, 49.3%-62.8%]; specificity, 99.4% [95% CI, 98.8%-99.7%]). Accuracy of LLMs was particularly low for certain CONSORT items (<b>Figure 25-1152</b>). There were no significant associations between PDF characteristics and AI and human disagreements. Trial PDFs were a mean of 49 pages and 17,268 words. LLMs were able to compile visually acceptable CONSORT checklists on the first iteration in 76.7% (115 of 150) attempts: ChatGPT for 40 RCTs, Gemini for 36 RCTs, and Claude for 39 RCTs. Generation of a CONSORT checklist required a mean of 75 seconds, 38 seconds, and 47 seconds for ChatGPT, Gemini, and Claude, respectively (range, 16-249 seconds).</p>

![](https://assets.underline.io/markdown_image/1/image/39e42ee902f1b80f6a363027b6df1d1e.png)

<h4>Conclusions </h4>
<p>In early 2025, the LLMs Gemini and Claude, but not ChatGPT, were able to generate accurate CONSORT checklists from PDFs of manuscripts reporting RCTs before peer review in over 90% of cases, given minimal instructions and often in less than 1 minute.</p>
<h4>References<a id="_idTextAnchor008"></a><a id="_idTextAnchor009"></a></h4>
<p>1. Moher D, Jones A, Lepage L; CONSORT Group (Consolidated Standards for Reporting of Trials). Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. <i>JAMA</i>. 2001;285(15):1992-1995. doi:10.1001/jama.285.15.1992</p>
<p>2. Di Girolamo N, Meursinge Reynders R, Di Girolamo U. Comparison of different AI large language models for extraction of CONSORT items from submitted randomized controlled trials before peer-review. Open Science Framework. Registered February 13, 2025. Accessed July 5, 2025. <a href="https://osf.io/5v6h2"><span class="Hyperlink CharOverride-6">https://osf.io/5v6h2</span></a></p>
<p>3. Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. <i>BMJ</i>. 2010;340:c332. doi:10.1136/bmj.c332</p>
<sup>1</sup>Department of Clinical Sciences, College of Veterinary Medicine, Cornell University, Ithaca, NY, US, nd374@cornell.edu; <sup>2</sup>Journal of Small Animal Practice, British Small Animal Veterinary Association, Gloucestershire, UK; <sup>3</sup>Department of Oral and Maxillofacial Surgery, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, the Netherlands; <sup>4</sup>Private Practice of Orthodontics, Milan, Italy; <sup>5</sup>Compass, New York, NY, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Nicola Di Girolamo is editor in chief of 2 peer-reviewed journals, one published by Elsevier and one by Wiley. No other disclosures were reported.</p>


poster

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Josh Major,<sup>1</sup> Kurt Mahnken,<sup>1</sup> Alec Young,<sup>1</sup> Cameron O’Brien,<sup>1</sup> Andrew V. Tran,<sup>1</sup> Patrick Crotty,<sup>1</sup> Alicia Ito Ford,<sup>1,2</sup> Matt Vassar<sup>1,2</sup></p>
<h4>Objective</h4>
<p> The integration of artificial intelligence (AI) in scientific research presents both opportunities and challenges, particularly concerning transparency, ethics, and reproducibility.<sup>1</sup> This study aimed to evaluate how leading orthopedic and sports medicine (OSM) journals address 2 distinct outcomes: (1) the extent to which AI is permitted in research and publication and (2) the degree to which OSM journals are endorsing AI-specific reporting guidelines (RGs). The primary outcome was to assess the presence of AI-related policies, including requirements for AI disclosure, acceptance or prohibition of AI-generated images and content, and AI’s role in manuscript writing and authorship. Secondary outcomes examined the endorsement of AI-specific RGs and the relationship between AI policies and Journal Impact Factor.</p>
<h4>Design</h4>
<p> We conducted a cross-sectional analysis of the top 100 OSM journals ranked by the 2023 SCImago Journal Rank (SJR) indicator. Eligibility criteria included actively publishing peer-reviewed clinical journals with publicly accessible Instructions for Authors in English language. On September 29, 2024, 2 investigators independently screened, extracted, and reconciled data in a masked, duplicate manner. Data were extracted from journals’ Instructions for Authors pages and affiliated publisher websites where indicated. If a journal had no mention of AI policies or AI-specific RGs, the editorial teams were contacted directly to inquire about these gaps. Biserial correlation analyses in R version 4.4.1 (The R Foundation) and RStudio (Posit) examined AI policy relationships with Journal Impact Factor and SJR.</p>
<h4>Results</h4>
<p> The initial search yielded 319 journals, from which the top 100 were analyzed. Of these, 94 mentioned AI in their Instructions for Authors, with all 94 requiring disclosure of AI use and prohibiting AI as an author (<b>Table 25-1175</b>). Additionally, AI-assisted manuscript writing was permitted in all cases, while 87% allowed AI-assisted content generation and 64% permitted AI-generated images. Despite this widespread recognition of AI policies, AI-specific RGs were endorsed in only 1% of journals, recommending the use of the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) RG. No significant associations were observed between Journal Impact Factor and the presence of AI-related editorial policies. Additionally, no differences were found among the AI policies in the 4 quartiles of the OSM journals in this study.</p>
<a href="/wp-content/uploads/2025_conference/tables/prc25-1175t1.png"> <img loading="lazy" src="/wp-content/uploads/2025_conference/tables/prc25-1175t1.png"> </a>
<h4>Conclusions</h4>
<p> While some AI use in research is widely acknowledged by OSM journals, the lack of consistency in policies regarding AI-generated images and content suggest an area for improvement to standardize how AI can be applied in these journals. Furthermore, the lack of endorsement for AI-specific RGs suggests a critical gap in ensuring the transparency and methodological rigor of AI-integrated research. To address these issues, we recommend that OSM journals establish clear AI policies and endorse AI-specific RGs. The International Committee of Medical Journal Editors, Committee on Publication Ethics, and World Association of Medical Editors have taken initial steps; however, implementation may be difficult.<sup>2</sup> Implementing these measures will bridge existing gaps, promote transparency, and improve research quality in scientific publishing.</p>
<h4>References</h4>
<p>1. Prasana P, Mandal PK, Hussain D, et al. Intersection of orthopaedics and artificial intelligence: a review. <i>SSR Inst Int J Life Sci</i>. 2024;10(3):5544-5552. doi:10.21276/ssr-iijls.2024.10.3.21</p>
<p>2. Simera I, Moher D, Hoey J, Schulz KF, Altman DG. The EQUATOR Network and reporting guidelines: helping to achieve high standards in reporting health research studies. <i>Maturitas</i>. 2009;63(1):4-6. doi:10.1016/j.maturitas.2009.03.011</p>
<sup>1</sup>Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK, US, youngalec9.r@gmail.com; <sup>2</sup>Department of Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, OK, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Alicia Ito Ford reports receipt of funding from the Center for Integrative Research on Childhood Adversity, the Oklahoma Shared Clinical and Translational Resources, and internal grants from Oklahoma State University and Oklahoma State University Center for Health Sciences outside of the present work. Matt Vassar reports receipt of funding from the National Institute on Drug Abuse, the National Institute on Alcohol Abuse and Alcoholism, the US Office of Research Integrity, Oklahoma Center for Advancement of Science and Technology, and internal grants from Oklahoma State University Center for Health Sciences outside of the present work. No other disclosures were reported.</p>
<h4>Additional Information</h4>
<p> Andrew V. Tran is a co–corresponding author (andrewtranresearch@gmail.com).</p>


Artificial Intelligence Editorial Policies and Reporting Standards in Orthopedic and Sports Medicine Journals

<p>Heather Goodell,<sup>1</sup> Christine Beaty,<sup>1</sup> Jonathan Schultz,<sup>1</sup> Shilpi Mehra,<sup>2</sup> Chirag Jay Patel<sup>3</sup></p>
<h4>Objective</h4>
<p> The increase in abstract submissions to major scientific conferences requires efficient and reliable methods to ensure compliance with research integrity standards, which are essential for maintaining the credibility of science. Traditional review methods may not fully capture key issues, requiring a structured approach to evaluate submissions before final acceptance. This study presents a review process that uses artificial intelligence (AI) with human validation to assess the integrity of abstracts submitted to the American Heart Association (AHA) Scientific Sessions.</p>
<h4>Design</h4>
<p> An integrity evaluation process was applied by combining AI-driven analysis with human review of abstracts submitted for the 2024 AHA Scientific Sessions. An AI tool (Paperpal Preflight for Editorial Desk) was used to identify research integrity issues, including AI-generated text, author expertise misalignment, and reference accuracy. Subject matter experts then reviewed abstracts flagged for validation. AI checks categorized abstracts into <i>passed</i>, <i>warning</i>, or <i>critical</i> levels, with those not marked as <i>passed</i> requiring further human review.</p>
<h4>Results</h4>
<p> Among the 8477 submitted abstracts that were analyzed, 42 were flagged with a warning for integrity concerns (AI-generated text, author expertise misalignment, and reference accuracy); human reviewers cleared 3 of these. Further analysis revealed that 167 authors submitted 15 or more abstracts, 63 authors submitted 20 or more, and 13 authors submitted more than 30 abstracts. A total of 2624 abstracts had at least 1 author with 10 or more submissions, and 1438 abstracts had at least 1 author with 15 or more submissions. The number of submissions was considered when assessing research integrity concerns. Analysis of author submission counts helped flag questionable cases for closer review and the identification of potentially suspect activities.</p>
<h4>Conclusions</h4>
<p> The hybrid AI-human review process for abstract submissions was demonstrated to be an efficient and accurate evaluation method that should be scalable. By addressing concerns related to author submission volume and content errors, this approach can strengthen conference proceedings and support broader efforts to enhance research quality and trust in scientific dissemination. Future efforts will focus on refining and customizing AI models and optimizing reviewer workflows to further enhance this process.</p>
<sup>1</sup>American Heart Association, Dallas, TX, US; <sup>2</sup>Cactus Communications Pvt Ltd, Mumbai, India; <sup>3</sup>Cactus Communications Inc, Princeton, NJ, US; chirag.patel@cactusglobal.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Heather Goodell, Christine Beaty, and Jonathan Schultz are employed by the American Heart Association. Shilpi Mehra and Chirag Jay Patel are employed by Cactus Communications, which owns Paperpal Preflight for Editorial Desk. No other disclosures were reported.</p>


Enhancing Research Integrity in Abstract Submissions With a Hybrid AI-Human Post-Review Process

<p>Ting Gao,<sup>1</sup> Xiuyuan Hao<sup>2</sup></p>
<h4>Objective</h4>
<p> Authorship confers credit and has important academic, social, and financial implications.<sup>1</sup> Although certain journals, such as <i>JAMA</i>, enforce strict authorship policies, the recognition of equal contributions by 2 or more co–first and/or co–corresponding authors is increasingly common in biomedical publishing.<sup>2,3</sup> Both the <i>Chinese Medical Journal</i> (<i>CMJ</i>) and <i>JAMA</i> are peer-reviewed general medical journals covering all major medical disciplines. This study evaluated the prevalence of co–first and co–corresponding authors in <i>CMJ</i> and <i>JAMA</i>.</p>
<h4>Design</h4>
<p> From January 1 to January 21, 2025, we examined all articles published in <i>CMJ</i> and <i>JAMA</i> in 2024, excluding News, Editors’ Notes, Corrigendum/Correction, Humanities, and miscellaneous sections, to assess the frequency of co–first and co–corresponding authorship. Other characteristics examined included article type, subject, and geographic location of corresponding authors. A multivariate logistic regression model was used to identify characteristics associated with co–first or co–corresponding authors.</p>
<h4>Results</h4>
<p> Of the 473 articles published in <i>CMJ</i> in 2024, 233 (49.3%) included co–first authors and 180 (38.1%) included co–corresponding authors. <i>JAMA</i> published 1011 articles in 2024; among them, 42 (4.2%) had co–first authors and 18 (1.8%) had co–corresponding authors. Original articles most frequently featured co–first and co–corresponding authors in both journals (<b>Table 25-0882</b>). In <i>CMJ</i>, gastroenterology was the most frequent subject associated with both forms of shared authorship. In <i>JAMA</i>, public health had the highest number of co–first authors, while cardiology had the most co–first and co–corresponding authors. In <i>CMJ</i>, most corresponding authors of articles with co–first or co–corresponding authorship were based in China. Logistic regression analysis revealed that article type, subject, and corresponding author’s geographic location were not significantly associated with shared authorship status in <i>CMJ</i>. By contrast, in <i>JAMA</i>, article type was significantly associated with both co–first and co–corresponding authorship, and corresponding author’s geographic location was significantly associated with co–corresponding authorship. <i>CMJ</i> showed a higher prevalence of co–first (χ² = 431.3; <i>P</i> &lt;.001) and co–corresponding (χ² = 363.6; <i>P</i> &lt; .001) authorship than <i>JAMA</i>.</p>

![](https://assets.underline.io/markdown_image/1/image/432ce7332177690f74ec1e349e31502a.png)

<h4>Conclusions</h4>
<p> This study demonstrates a significant difference in the prevalence of co–first and co–corresponding authorship between <i>CMJ</i> and <i>JAMA</i>.<i> </i>China and the United States are among the leading countries in scientific output. Compared with US-based authors, Chinese authors were more likely to designate co–first and co–corresponding authors. This trend may reflect the academic, social, and economic significance of shared authorship roles within the Chinese research evaluation system. While journals such as <i>JAMA</i> impose restrictions on shared authorship, increasing multidisciplinary collaborations and the rise of team science have led others, such as <i>CMJ</i>, to adopt more flexible policies. Establishing a standardized framework for managing co–first and co–corresponding authorship is needed.</p>
<h4>References</h4>
<p>1. International Committee of Medical Journal Editors. Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. Updated May 2023. Accessed January 18, 2025. <a href="https://www.icmje.org/recommendations/"><span class="Hyperlink CharOverride-6">https://www.icmje.org/recommendations/</span></a></p>
<p>2. Aakhus E, Mitra N, Lautenbach E, Joffe S. Gender and byline placement of co-first authors in clinical and basic science journals with high impact factors. <i>JAMA</i>. 2018;319(6): 610-611. doi:10.1001/jama.2017.18672</p>
<p>3. Hosseini M. Equal co-authorship practices: Review and recommendations. <i>Sci Eng Ethics</i>. 2020;26(3):1133-1148. doi:10.1007/s11948-020-00183-8</p>
<sup>1</sup>Editorial Department, <i>Chinese Medical Journal</i>, Chinese Medical Association Publishing House, Beijing, China; <sup>2</sup>Editorial Department, <i>Chinese Medical Journal</i>, Chinese Medical Association Publishing House, Beijing, China, haoxiuyuan@163.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


Co-First authors and Co-Corresponding Authors in the Chinese Medical Journal and JAMA

<p>Cecilia Munguti,<sup>1</sup> James Kiilu,<sup>1</sup> James Kigera,<sup>1</sup> Michael Mwachiro<sup>1</sup></p>
<h4>Objective</h4>
<p> The International Committee of Medical Journal Editors (ICMJE) authorship criteria ensure appropriate credit for significant research contributions, while the Contributor Role Taxonomy (CRediT) framework offers a detailed account of individual scholarly contributions. Despite these frameworks, honorary authorship remains widespread in health sciences,<sup>1</sup> and surveys from Africa have documented its prevalence and the difficulty of applying authorship standards.<sup>2</sup> This study evaluated the fulfillment of ICMJE authorship criteria and CRediT contributions in manuscripts submitted between 2017 and 2023.</p>
<h4>Design</h4>
<p> This retrospective study analyzed the fulfillment of ICMJE authorship criteria and author-reported CRediT contributor roles for manuscripts submitted to <i>Annals of African Surgery</i>, a journal that endorses the ICMJE authorship criteria, between 2017 and 2023. The evaluated ICMJE criteria included criteria 1 (intellectual input) and 2 (manuscript drafting or critical revision) because these are the only author-level items captured as discrete checkboxes in the submission system. Criteria 3 and 4 were not assessed because they are captured as a collective attestation in a signed mandatory licence agreement required at submission. Data were retrieved from the journal’s online manuscript management system. CRediT roles were scored using a weighted system: 3 points for lead, 2 for equal, and 1 for supporting contributions across 14 roles, with a maximum possible score of 42 converted to a percentage. Descriptive statistics summarized ICMJE compliance and CRediT role distribution.</p>
<h4>Results</h4>
<p> A total of 1619 authors and 448 manuscripts with complete authorship data were evaluated. Among the authors, 1027 (63.4%) met both ICMJE criteria 1 and 2, 460 (28.4%) met only 1 criterion, and 132 (8.2%) met none. Of the 1619 authors, 709 (43.8%) were supervisors; among these supervisor authors, 523 (73.8%) met both criteria, 124 (17.5%) met only 1 criterion, and 62 (8.7%) failed to meet any. A total of 9610 CRediT contributor roles were analyzed. On weighted CRediT scores, 275 (17%) authors had a contribution score above 50%, while 971 (60%) scored 25% or less. Among 709 supervisors, 241 (34%) had scores above 50% and 269 (38%) scored 25% or less. Funding acquisition was credited to 296 authors (18.3%). Among the supervisor authors, 248 (35.0%) reported funding acquisition, accounting for 83.8% of all authors who claimed that role.</p>
<h4>Conclusions</h4>
<p> Using ICMJE criteria, supervisor involvement appeared to be limited, suggesting a high prevalence of guest authorship. However, weighted CRediT scores indicated that supervisors often contributed significantly, more so than other coauthors, highlighting a disconnect between traditional authorship criteria and actual contributions. Although limited to 1 journal, this study demonstrates the value of integrating authorship criteria audits with CRediT data, while noting the potential for checkbox behavior by submitting authors who may not grasp the significance of their declarations.</p>
<h4>References</h4>
<p>1. Meursinge Reynders RA, ter Riet G, Di Girolamo N, Cavagnetto D, Malički M. Honorary authorship is highly prevalent in health sciences: systematic review and meta-analysis of surveys. <i>Sci Rep</i>. 2024;14:4385. doi:10.1038/s41598-024-31526-2</p>
<p>2. Breet E, Botha J, Horn L, Swartz L. Academic and scientific authorship practices: a survey among South African researchers. <i>J Empir Res Hum Res Ethics</i>. 2018;13(5):412-420. doi:10.1177/1556264618789253</p>
<sup>1</sup><i>Annals of African Surgery</i>, Nairobi, Kenya, ceciliamunguti@annalsofafricansurgery.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> James Kigera is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Additional Information</h4>
<p> James Kiilu is a co–corresponding author (jameskiilu@annalsofafricansurgery.com).</p>


Authorship and Contributorship Criteria and Practices at the Annals of African Surgery

<p>Mohammad Hosseini,<sup>1</sup> Sofie Elaine Adams,<sup>2</sup> Yensi Flores,<sup>3</sup> Kathleen Hall Jamieson,<sup>2</sup> Joerg Heber,<sup>4</sup> Jennifer Heimberg,<sup>5</sup><span lang="ar-SA"> Véronique Kiermer,</span><sup>6</sup> Arthur Lupia,<sup>7</sup> Ana Marušić,<sup>8</sup> Beau Nielsen,<sup>5</sup> Magdalena Skipper,<sup>9</sup> Geeta Swamy,<sup>10</sup> Susan Wolf<sup>11</sup></p>
<h4>Objective </h4>
<p>Although authorship is a cornerstone of the reward system of science, it is often fraught with disagreements and questionable practices.<sup>1</sup> A working group (including authors of this abstract and additional experts) was convened in January 2024 by the National Academy of Sciences (NAS) Strategic Council on Research Excellence, Integrity, and Trust to examine the impact of previous recommendations on institutional and journal authorship policies.<sup>1,2</sup></p>
<h4>Design </h4>
<p>Our analysis consisted of 2 parts completed in 2024 and 2025. First, we examined authorship policies of the AAAS, AGU, APS, <i>BMJ</i>, <i>Cell</i>, ICMJE, IEEE, JACS, <i>Nature</i>, <i>NEJM</i>, PLOS, <i>PNAS</i>, and SAGE (represented in the group members who authored the NAS 2018 recommendation<sup>2</sup>) to examine the extent to which they had adapted policies. This review captured trends regarding authorship guidelines, use of ORCIDs, and the CRediT taxonomy. Second, we searched the websites of all R1 (very high research activity) institutions (search terms <i>authorship</i>, <i>authorship guidelines</i>, <i>research guidelines</i>, and <i>research integrity</i>) in the US (n = 146) to determine whether credit and accountability or responsibility were mentioned in authorship guidelines and, if so, whether they were cast as related concepts. We then checked whether guidelines included links to external instructions (eg, ICMJE) and encouraged meetings to discuss authorship. Both phases were followed by brainstorming sessions among working group members.</p>
<h4>Results </h4>
<p>Of the examined journal groups, 5 adopted ICMJE guidelines, 2 referenced the NAS 2018 recommendations, and others had different recommendations. Seven groups recommended using the CRediT taxonomy, but only 3 required ORCIDs for all authors. Of the 146 R1 institutions, 59 included a credit-accountability link in their guidelines, but in only 38 cases did the credit-accountability link appear at the top of the guideline. Guidelines of 60 institutions suggested authors meet proactively to discuss authorship.</p>
<h4>Conclusions </h4>
<p>Anchoring authorship in credit (give credit where it is due) and accountability (authors are accountable for the integrity of the work) principles<sup>2</sup> readily explains why questionable practices (eg, gifts, ghost writing, and coercive authorship) are problematic and helps determine when authorship credit is warranted. Providing a clear description of contributions reinforces the application of these principles and fosters transparency as an important complementary principle.<sup>3</sup> While journals should choose authorship guidelines that suit their context, encouraging the use of ORCID and taxonomies such as CRediT are beneficial. Aligning institutional guidelines with the credit-accountability framework offers an opportunity to improve authorship practices. We plan to engage with institutional leaders to explore ways to support these improvements (eg, by offering educational materials or courses, providing mechanisms to resolve conflicts, and offering a safe environment for discussing authorship).</p>
<h4>References</h4>
<p>1. National Academies of Sciences, Engineering, and Medicine. <i>Fostering Integrity in Research</i>. 2017; National Academies Press.<a href="https://doi.org/10.17226/21896"><span class="Hyperlink CharOverride-6"> doi:10.17226/21896</span></a></p>
<p>2. McNutt MK, Bradford M, Drazen JM, et al. Transparency in authors’ contributions and responsibilities to promote integrity in scientific publication. <i>Proc Natl Acad Sci U S A</i>. 2018;115(11):2557-2560.<a href="https://doi.org/10.1073/pnas.1715374115"><span class="Hyperlink CharOverride-6"> doi:10.1073/pnas.1715374115</span></a></p>
<p>3. Allen L, Scott J, Brand A, Hlava M, Altman M. Publishing: credit where credit is due. <i>Nature.</i> 2014;508(7496):312-313.<a href="https://doi.org/10.1038/508312a"><span class="Hyperlink CharOverride-6"> doi:10.1038/508312a</span></a></p>
<sup>1</sup>Northwestern University, US, mohammad.hosseini@northwestern.edu; <sup>2</sup>University of Pennsylvania, US; <sup>3</sup>University College Cork, Ireland; <sup>4</sup>Lawrence Berkeley National Laboratory, US; <sup>5</sup>National Academy of Sciences, US; <sup>6</sup>PLOS, US; <sup>7</sup>University of Michigan, US; <sup>8</sup>University of Split, Croatia; <sup>9</sup>Nature, UK; <sup>10</sup>Duke University, US; <sup>11</sup>University of Minnesota, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>The authors declare no conflicting interests. Véronique Kiermer and Ana Marušić are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract.</p>
<h4>Funding/Support </h4>
<p>This work is funded by the National Academy of Sciences (NAS). Mohammad Hosseini is funded by the National Institutes of Health’s (NIH) National Center for Advancing Translational Sciences (UM1TR005121).</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funders have not played a role in the design, analysis, decision to publish, or preparation of the manuscript, and views expressed here do not represent the views and opinions of the NAS, NIH, or US government.</p>


Integration of Credit and Accountability Principles in Authorship Policies of Science Journals and Research Institutions

<p>Stephan D. Fihn,<sup>1,3</sup> Roy H. Perlis,<sup>2,3</sup> Jacob Kendall-Taylor,<sup>3</sup> Annette Flanagin<sup>3</sup></p>
<h4>Objective</h4>
<p> The major challenges for editors of journals include attracting submissions of high-quality manuscripts and recruiting reviewers.<sup>1</sup> In light of a dearth of relevant research, we sought to ascertain the association between publishing manuscripts as an author and participation in peer review for the same journal.</p>
<h4>Design</h4>
<p> We conducted a cross-sectional study using data extracted about authors and reviewers for <i>JAMA Network Open</i>, a high-volume open access general medical journal, for the years 2019 to 2023, including number of articles submitted and published as first or corresponding author during this period, numbers of requests to review and response (submitted, declined, no response), and quality of review submitted as determined by reviewing editor on a 5-point Likert scale (excellent = 1 to poor = 5). Data were analyzed using R 4.4.2.</p>
<h4>Results</h4>
<p> Among 38,683 invited reviewers, 30,082 (77.8%) submitted no manuscripts; 4014 (10.6%) submitted ≥1 article as first or corresponding author but had no published articles; and 4497 (11.6%) submitted ≥1 article as first or corresponding author and had ≥1 publication. <b>Table 25-1097</b> summarizes review requests to these individuals and compares their rates of accepting reviews and ratings among those who returned reviews. The mean (SD) number of invitations was greatest among published reviewers (5.1 [9.5]), followed by those with submissions but no publications (3.4 [6.1]), and then individuals with no submissions (2.7 [5.9]) (<i>P</i> &lt; .007 by analysis of variance). Reviewers with publications were significantly more likely to accept invitations and significantly less likely to ignore invitations compared with the other 2 groups (<i>P</i> &lt; .01 by analysis of variance and post hoc pairwise test). The absolute difference in proportion of accepted invitations was 26.2% (95% CI, 24.8%-27.6%; <i>P</i> &lt; .001) between published and never-submitted reviewers and 11.6% (95% CI, 9.7%-13.5%; <i>P</i> &lt; .001) between published and nonpublished submitters. Among those who returned reviews, quality (mean rating) was significantly higher among reviewers with publications (absolute difference from nonsubmitters: 0.25; 95% CI, −0.29 to −0.20; <i>P</i> &lt; .001). Differences between submitters without publications (mean difference: 0.03) and never-submitters (mean difference: 2.5%) were minimal for both rating metrics (<i>P</i> &gt; .50).</p>
<a href="/wp-content/uploads/2025_conference/tables/prc25-1097t1.png"> <img loading="lazy" src="/wp-content/uploads/2025_conference/tables/prc25-1097t1.png"> </a>
<h4>Conclusions </h4>
<p>Authors who published in <i>JAMA Network Open </i>were significantly more likely to accept an invitation to review even though they received nearly twice as many requests as those who did not publish. Moreover, the quality of their reviews was significantly greater. Individuals who submitted manuscripts but did not publish were intermediate between these 2 groups in terms of accepting invitations to reviews and quality of reviews. These results suggest that investigators may be willing to commit more of their academic effort to journals that publish their work. Limitations of this analysis include the inability to determine the relative timing of publication versus review; restriction of the analysis to a single high-volume journal; exclusion of co-authors; and limitation to a 4-year time frame.</p>
<h4>Reference</h4>
<p>1. Zupanc GKH. “It is becoming increasingly difficult to find reviewers”—myths and facts about peer review. J Comp Physiol A Neuroethol Sens Neural Behav Physiol. 2024;210(1):1-5. doi:10.1007/s00359-023-01642-w</p>
<sup>1</sup>Department of Medicine, University Washington, Seattle, WA, US; steve.fihn@jamanetwork.org; <sup>2</sup>Department of Psychiatry, Massachusetts General Hospital and Harvard Medical School, Boston, MA, US; <sup>3</sup>JAMA Network, Chicago, IL, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Stephan D. Fihn (SDF) and Roy H. Perlis (RHP) are paid as consultants for work as editors of JAMA Network Open (SDF and RHP) and JAMA+ AI (RHP). Jacob Kendall-Taylor and Annette Flanagin are on the editorial staff of the JAMA Network. Annette Flanagin is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. None of the authors have any other conflicts of interest to disclose.</p>
<h4>Funding/Support </h4>
<p>The authors received no direct funding for this work apart from that described above. </p>


Authors Who Publish in a Journal and Likelihood to Serve as Reviewers

<p>Sidney Xiang,<sup>1</sup> Daniel M. Romero,<sup>1,2,3</sup> Misha Teplitskiy<sup>1</sup></p>
<h4>Objective</h4>
<p> Prior literature on interdisciplinary research evaluation does not include prepublication manuscripts or account for differential evaluation over multiple dimensions of interdisciplinarity. Based on management science literature, we introduce 2 novel dimensions: topic interdisciplinarity, which may incur evaluation penalties by cutting across disciplinary standards and threatening symbolic boundaries, and knowledge-base interdisciplinarity, which may incur benefits by combining larger pools of nonredundant information. Evaluations may also depend on how these dimensions align with each other and the audience. Our study tested these hypotheses using a large-scale dataset of journal submissions.</p>
<h4>Design</h4>
<p> We performed a case-control study using administrative data from the Institute of Physics Publishing, a major STEM publisher. The data included 128,950 manuscripts (accepted and rejected) submitted to 62 physical science journals from 2018 to 2022. We calculated topic interdisciplinarity with concept tags of the manuscript’s associated OpenAlex record<sup>1</sup> and knowledge-base interdisciplinarity with concept tags of the manuscript’s references. For each type, we quantified interdisciplinarity using the integration index,<sup>2</sup> ranging from 0 to 1. Logistic regression was used to model the relationship between both types of interdisciplinarity and 2 evaluation outcomes: final decision and review positivity, controlling for journal, manuscript type, year, number of authors, number of authors’ prior citations, number of authors’ prior publications, and number of references. We retained observations with both interdisciplinarities and all covariates.</p>
<h4>Results</h4>
<p> The marginal effect of a 1-SD increase in knowledge-base interdisciplinarity was a 0.9 (0.7) percentage point higher acceptance (positive review) probability, while a 1-SD increase in topic interdisciplinarity corresponded to a 1.2 (0.4) percentage point lower acceptance (positive review) probability. Interactions revealed that high knowledge-base interdisciplinarity attenuated the negative relationship between topic interdisciplinarity and acceptance. We did not see a symmetric effect when high knowledge-base interdisciplinarity was accompanied by high topic interdisciplinarity. When disaggregating model results by journal interdisciplinarity according to the publisher’s 2022 product catalogue,<sup>3</sup> journals categorized by the publisher as interdisciplinary or multisubject had positive associations between both types of interdisciplinarity and evaluation outcomes, whereas monodisciplinary journals had negative associations. Our results were robust to the inclusion and transformation of manuscript and team covariates.</p>

![](https://assets.underline.io/markdown_image/1/image/3f8d7212732299f660aee5d48fd7f1cb.png)

<h4>Conclusions</h4>
<p> Our findings indicate that different dimensions of interdisciplinarity have different relationships to evaluation and that, for successful publication, authors must attend to alignment between types of interdisciplinarity within the manuscript and between the manuscript and journal audience. Limitations include the scope of journals in our dataset (STEM only) and the reliance on matching submissions to OpenAlex records.</p>
<h4>References</h4>
<p>1. Priem J, Piwowar H, Orr R. OpenAlex: a fully-open index of scholarly works, authors, venues, institutions, and concepts. arXiv. Preprint posted online May 4, 2022. doi:10.48550/ARXIV.2205.01833</p>
<p>2. Porter AL, Cohen AS, David Roessner J, Perreault M. Measuring researcher interdisciplinarity. Scientometrics. 2007;72(1):117-147. doi:10.1007/s11192-007-1700-5</p>
<p>3. IOP Publishing. Product catalogue 2022. <a href="https://ioppublishing.org/wp-content/uploads/2022/03/IOPP-Catalogue-2022.pdf"><span class="Hyperlink CharOverride-6">https://ioppublishing.org/wp-content/uploads/2022/03/IOPP-Catalogue-2022.pdf</span></a></p>
<sup>1</sup>University of Michigan School of Information, Ann Arbor, MI, US, tepl@umich.edu; <sup>2</sup>University of Michigan Department of Electrical Engineering and Computer Science, Ann Arbor, MI, US; <sup>3</sup>University of Michigan Center for the Study of Complex Systems, Ann Arbor, MI, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


Topic and Knowledge-base Interdisciplinarity in Manuscripts Submitted to Physical Science Journals vs Editorial Decision and Reviewer Positivity

<p>Neil Millar,<sup>1</sup> Brian Budgell<sup>2</sup></p>
<h4>Objective</h4>
<p> Promotional language (hereafter, <i>hype</i>) in scientific writing has increased significantly over the past 40 years.<sup>1</sup> Analyses of grant proposals have found the incidence of hype to be positively associated with funding success<sup>2</sup>; however, evidence of causality is still lacking. Furthermore, an underpowered pilot study suggested that hype may bias clinicians’ perceptions and evaluations of evidence.<sup>3</sup> This study tested the hypothesis that students’ appraisal of biomedical research abstracts is influenced by hype.</p>
<h4>Design</h4>
<p> A double-blind randomized controlled trial was conducted to assess readers’ evaluation of hyped vs nonhyped abstracts. Four structured abstracts were selected from leading spinal care journals, controlling for length and absence of hype. Hyped abstracts were modified to include 6 common hype terms, such as “carefully designed” to highlight methodological rigor, “this is the first study to” to emphasize novelty, “convincing evidence” to amplify results, and “conducted by an experienced radiologist” to emphasize researcher competence. Inclusion criteria were that participants were second-year or higher chiropractic students recruited from institutions in the UK, US, and Canada. A blinded researcher randomly selected folders for students to evaluate. Participants rated abstracts on a 10-point scale, using 4 criteria: implementation likelihood, rigor, novelty, and researcher competence. A linear mixed-effects model was used to assess whether evaluations were influenced by hype, accounting for variability across participants and abstracts.</p>
<h4>Results</h4>
<p> This ongoing study began on April 2, 2024. As of January 2025, 36 students at 3 institutions had completed the rating task. Mixed-effects modeling, including interactions between condition and criterion, showed no significant differences between hyped and original abstracts across all evaluation criteria (estimate, 0.067; <i>P</i> &gt; .05) (<b>Figure 25-0927</b>). While novelty received consistently lower ratings (estimate, −1.479; <i>P</i> &lt; .001), the influence of hype did not vary significantly by criterion, as interaction terms were nonsignificant (eg, condition × novelty interaction [estimate, 0.517; <i>P</i> &gt; .05]), indicating minimal measurable effect of promotional language.</p>

![](https://assets.underline.io/markdown_image/1/image/ed072eece70a4477cfa47ac6438bf2df.png)

<h4>Conclusions </h4>
<p>Within this experimental paradigm, hype did not appear to bias readers’ evaluation of the scientific merit of the literature. These findings are discussed in relation to evidence-based medicine and bias on the part of researchers, reviewers, editors, and other stakeholders. Limitations of the study include the small sample size reported to date and restriction of the experimental texts to abstracts.</p>
<h4>References</h4>
<p>1. Millar N, Batalo B, Budgell B. Trends in the use of promotional language (hype) in National Institutes of Health funding opportunity announcements, 1992-2020. <i>JAMA Netw Open</i>. 2022;5(11):e2243221. doi:10.1001/jamanetworkopen.2022.43221</p>
<p>2. Qiu HS, Peng H, Fosse HB, Woodruff TK, Uzzi B. Use of promotional language in grant applications and grant success. <i>JAMA Netw Open</i>. 2024;7(12):e2448696. doi:10.1001/jamanetworkopen.2024.48696</p>
<p>3. Millar N, Budgell B. Impact of hype on clinicians’ evaluation of trials—a pilot study. <i>J Can Chiropr Assoc</i>. 2023;67(1):38-49.</p>
<sup>1</sup>Faculty of Engineering, Information and Systems, University of Tsukuba, Tsukuba, Ibaraki, Japan; <sup>2</sup>Canadian Memorial Chiropractic College, Toronto, Ontario, Canada, bbudgell@cmcc.ca.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>
<h4>Funding/Support </h4>
<p>This study was supported by grant 21K02919 from the Japan Society for the Promotion of Science.</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funding organization had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>


The Influence of Promotional Language on Evaluations of Biomedical Literature: A Randomized Controlled Trial

<p>Vincent Yuan,<sup>1</sup> Aidan Christopher Tan<sup>2</sup> </p>
<h4>Objective </h4>
<p>Our study aimed to determine the prevalence, types, and facilitators of “spin” and other selective reporting practices in recently published medical artificial intelligence (AI) research.</p>
<h4>Design </h4>
<p>This was a cross-sectional meta-research study of interventional and observational studies related to medical AI. Medical AI was defined as computational models (eg, machine learning, deep learning, natural language processing) that amplify or augment health care–related tasks (eg, diagnosis, prognosis, risk stratification).<sup>1</sup> Studies were identified by searching MEDLINE, Embase, and the Cochrane Library. Studies were included if they were observational or interventional studies related to medical AI published in 2023, English-language, and full text. Spin was defined as the distortion of findings to present the performance of a medical AI tool as more beneficial than it appears. Examples of spin include misrepresentation or omission of significant or nonsignificant results and selective focus on abstract conclusions. Other study characteristics investigated included the journal, academic qualifications of first authors, funding, and competing interests. The features of spin were derived from coding manuals of Boutron et al<sup>2</sup> and Andaur Navarro et al.<sup>3</sup> Spin was categorized as either misleading interpretation, referring to an overemphasis of results, or misleading transportability, referring to an exaggeration of a model’s clinical applicability. Instances of spin were abstracted by manually reviewing the Results and Discussion sections of abstracts and body and extracting the data into a prepiloted data collection form. This form was modified from SPIN-Prediction Models to exclude or include certain indicators, with the addition of custom items based on previous spin studies. Data were collected by 1 reviewer, and a second reviewer independently recollected the data in instances of uncertainty.</p>
<h4>Results </h4>
<p>A total of 423 studies were included (<b>Figure 25-0950</b>). Slightly more than one-third (35% [n = 149]) of studies described AI as superior to the comparator. Nine percent of studies (n = 38) contained leading words in the title, and more than half (56% [n = 236]) selectively focused abstract conclusions on significant results. In almost half (48% [n = 203]) of study abstracts, qualifiers (eg, <i>might</i>, <i>very</i>) were present to subtly suggest stronger findings than likely warranted. The majority of studies (71% [n = 300]) contained strong statements (eg, <i>clearly shows</i>) in the body to emphasise the AI model’s performance, accuracy, or effectiveness. Twenty-four percent (n = 101) of studies made recommendations for clinical practice in the body without external validation in the same study.</p>

![](https://assets.underline.io/markdown_image/1/image/8d8ebe9fade171778923d51191e80204.png)

<h4>Conclusions </h4>
<p>Our findings suggest that selective reporting practices, including spin, are prevalent in medical AI research. These practices can misinform readers of the technical performance and real-world applicability of AI tools. This is especially crucial for titles and abstracts because these sections often influence first impressions and inform clinical decision-making.</p>
<h4>References</h4>
<p>1. Bajwa J, Munir U, Nori A, Williams B. Artificial intelligence in healthcare: transforming the practice of medicine. Future Healthc J. 2021;8(2):e188-e194. doi:10.7861/fhj.2021-0095 </p>
<p>2. Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010;303(20):2058-2064. doi:10.1001/jama.2010.651 </p>
<p>3. Andaur Navarro CL, Damen JAA, Ghannad M, et al. SPIN-PM: a consensus framework to evaluate the presence of spin in studies on prediction models. J Clin Epidemiol. 2024;170:111364. doi:10.1016/j.jclinepi.2024.111364 </p>
<sup>1</sup>School of Clinical Medicine, Faculty of Medicine and Health, University of New South Wales, Sydney, NSW, Australia,<i> </i>vincent.yuan1@student.unsw.edu.au; <sup>2</sup>Sydney School of Public Health, Faculty of Medicine and Health, University of Sydney, Sydney, NSW, Australia.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


Unraveling the 'Spin' and Selective Reporting in Medical AI Research: A Cross-Sectional Meta-Research Study

<p>Joe D. Menke,<sup>1</sup> Mengfei Lan,<sup>1</sup> Halil Kilicoglu<sup>1</sup></p>
<h4>Objective </h4>
<p>The importance of self-acknowledged limitations (SALs) has long been recognized.<sup>1</sup> Prior work detecting<sup>2</sup> and categorizing<sup>3</sup> limitation statements using natural language processing (NLP) has made large-scale assessments of limitation reporting possible. By comparing SALs in research articles with limitations identified by systematic reviewers, this study aimed to evaluate the quality and coverage of SALs in relation to risk of bias.</p>
<h4>Design</h4>
<p> We constructed a dataset pairing SALs with limitations discussed by reviewers through risk-of-bias assessments. Articles from the Cochrane Database of Systematic Reviews available in the open-access subset of PubMed Central were downloaded in September 2024. For papers cited in these reviews, full texts were downloaded when available, and existing NLP models<sup>3</sup> were used to extract and classify sentences discussing limitations (December 2024). Identifying information and risk-of-bias assessments for cited articles were extracted from reviews’ Characteristics of Included Studies tables when a Risk of Bias section was present; reviews without this section were excluded. For living reviews, duplicate assessments were removed to retain only 1 assessment per cited article across versions of a living review. Risk-of-bias statements, originally in tabular form (eg, “Allocation concealment [Selection bias] High risk CBA; no allocation concealment done.”), were rephrased into natural language using a large language model (Llama3.3-70B). Low-risk components were excluded, retaining only high- and unclear-risk items. A random subset was manually reviewed to check for systematic issues, which did not reveal concerning patterns or hallucinations. Paraphrased information from reviews and SALs from cited papers were classified using NLP models that recognized limitations sentences and types.<sup>3</sup> We analyzed and compared the distribution of categorized limitations mentioned in both groups in an effort to assess the quality of SALs.</p>
<h4>Results</h4>
<p> From 145 reviews, we extracted and classified limitations statements for 803 cited articles (5.54 articles per review). From risk-of-bias tables, 3172 limitations were extracted (3.95 per cited article). From 750 cited articles, 136,290 sentences were processed and classified (170 per article), yielding 5616 SAL sentences (7.49 per article). No SALs were found in 53 articles. Categorized data are reported in <b>Table 25-1017</b>. Outcome measures were the most common SAL (69.12% of articles), while reviewers noted missing data as the most common high-risk limitation (33.75% of articles) and blinding as the most common unclear risk (48.19% of articles). On average, 56.67% of reviewer-identified limitations were not self-acknowledged, including 38.10% of high-risk and 53.01% of unclear-risk limitations.</p>

![](https://assets.underline.io/markdown_image/1/image/94ce029a42605d4892ff758a6a004f27.png)

<h4>Conclusions</h4>
<p> We present categorized limitations data and compared SALs with reviewer-derived limitations, showing that more than one-third of high-risk limitations are not self-acknowledged. Our findings may be used to better understand the quality of existing self-acknowledged limitations as well as highlight categories that are potentially underreported by authors.</p>
<h4>References</h4>
<p>1. Ioannidis JP. Limitations are not properly acknowledged in the scientific literature. J Clin Epidemiol. 2007;60(4):324-329. doi:10.1016/j.jclinepi.2006.09.011</p>
<p>2. Kilicoglu H, Rosemblat G, Malički M, Ter Riet G. Automatic recognition of self-acknowledged limitations in clinical research literature. J Am Med Inform Assoc. 2018;25(7):855-861. doi:10.1093/jamia/ocy038</p>
<p>3. Lan M, Cheng M, Hoang L, Ter Riet G, Kilicoglu H. Automatic categorization of self-acknowledged limitations in randomized controlled trial publications. J Biomed Inform. 2024;152:104628. doi:10.1016/j.jbi.2024.104628</p>
<sup>1</sup>School of Information Sciences, University of Illinois Urbana-Champaign, Champaign, IL, US, jmenke2@illinois.edu.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This work was supported by the National Library of Medicine of the National Institutes of Health (NIH; R01LM014079).</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funder had no role in the study design or in the collection, analysis, or interpretation of data; writing of the report; or decision to submit the abstract for publication.</p>
<h4>Additional Information</h4>
<p> The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.</p>


Next from The 10th Peer Review Congress

Artificial Intelligence Editorial Policies and Reporting Standards in Orthopedic and Sports Medicine Journals

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES