United States

&lt;p&gt;Fangwen Zhou,&lt;sup&gt;1&lt;/sup&gt; Muhammad Afzal,&lt;sup&gt;2&lt;/sup&gt; Rick Parrish,&lt;sup&gt;1&lt;/sup&gt; Ashirbani Saha,&lt;sup&gt;3&lt;/sup&gt; R. Brian Haynes,&lt;sup&gt;1&lt;/sup&gt; Alfonso Iorio,&lt;sup&gt;1,4&lt;/sup&gt; Cynthia Lokker&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective&lt;/h4&gt;
&lt;p&gt; Deep learning models for classifying published clinical literature to support critical appraisal have garnered wide attention.&lt;sup&gt;1&lt;/sup&gt; To automate the critical appraisal workflow for McMaster’s Premium Literature Service (PLUS), a gold-standard database manually labeled by experts was used to develop well-performing models to classify the rigor of randomized controlled trials (RCTs).&lt;sup&gt;2&lt;/sup&gt; However, due to the complexity of language models, the lack of transparency remains an important concern. This study explored Shapley Additive Explanations (SHAP)&lt;sup&gt;3&lt;/sup&gt; to better understand how a deep learning model makes classification decisions.&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;Details regarding the development and evaluation of the rigor classifiers are published elsewhere.&lt;sup&gt;2&lt;/sup&gt; Briefly, classifiers were trained with titles and abstracts of original RCTs, that met or did not meet PLUS criteria for rigor (randomization, ≥10 participants per group, ≥80% follow-up, clinically important outcomes, and preplanned subgroup analyses, if applicable). Models were trained on 53,219 PLUS articles from 2003 to 2023, randomly split 80:10:10 into train, validate, and test sets. Articles in Clinical Hedges, a similar database that preceded PLUS, and PLUS articles from 2024 were used for external testing. A top-performing BioLinkBERT model, with an area under the receiver operating characteristic curve of 94%, was selected for this study. The SHAP partition explainer determined important tokens (words/subwords) for articles from the validate and test datasets, Clinical Hedges, and PLUS 2024. Tokens were combined into words for ease of interpretation, and their SHAP values were summed. The mean SHAP values, which indicate the aggregated average marginal contribution to model output over all samples, for the most impactful words with 100 or more occurrences were examined.&lt;/p&gt;
&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;Overall, 6,207,935 words were analyzed, of which 49,387 were unique. &lt;b&gt;Figure 25-0848&lt;/b&gt; shows the 15 most impactful unique words for positive (rigorous) and negative (nonrigorous) classes. Terms such as &lt;i&gt;noninferiority&lt;/i&gt; positively influenced rigor predictions with a mean SHAP value of 0.00904 (95% CI, 0.00844-0.00965), whereas &lt;i&gt;nonrandomized&lt;/i&gt; (mean SHAP value, −0.02510; 95% CI, −0.03204 to −0.01816) had negative impacts. These results generally align with the manual appraisal criteria. However, tokens apparently unrelated to the criteria, such as tiotropium (mean SHAP value, 0.00571; 95% CI, 0.00508-0.00634), also had sizable impacts, indicating that they may have been correlated with higher rigor during manual appraisal. These patterns were learned and applied by the model, indicating a certain degree of overfitting.&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/83c754796d3484f54b7a4aacbdad1110.png)

&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;This study demonstrates that SHAP helps understand which features influence a deep learning model’s rigor assessment of an RCT. Identifying influential features increases confidence in the model assessments and generalizability and reduces unease about the black-box nature of these models. Future work should explore SHAP with other critical appraisal tools and datasets and potentially integrate it into machine learning systems to improve user trust and model accountability in evidence synthesis and critical appraisal workflows.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Lokker C, Bagheri E, Abdelkader W, et al. Deep learning to refine the identification of high-quality clinical research articles from the biomedical literature: performance evaluation. &lt;i&gt;J Biomed Inform&lt;/i&gt;. 2023;142:104384. doi:10.1016/j.jbi.2023.104384&lt;/p&gt;
&lt;p&gt;2. Zhou F, Parrish R, Afzal M, et al. Benchmarking domain-specific pretrained language models to identify the best model for methodological rigor in clinical studies. &lt;i&gt;J Biomed Inform&lt;/i&gt;. 2025;166:104825. doi:10.1016/j.jbi.2025.104825&lt;/p&gt;
&lt;p&gt;3. Lundberg S, Lee SI. A unified approach to interpreting model predictions. &lt;i&gt;arXiv&lt;/i&gt;. Preprint posted online May 22, 2017. doi:10.48550/arXiv.1705.07874&lt;/p&gt;

&lt;sup&gt;1&lt;/sup&gt;Health Information Research Unit, Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada, lokkerc@mcmaster.ca; &lt;sup&gt;2&lt;/sup&gt;Faculty of Computing, Engineering and the Built Environment, Birmingham City University, Birmingham, UK; &lt;sup&gt;3&lt;/sup&gt;Department of Oncology, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; &lt;sup&gt;4&lt;/sup&gt;Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada.
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt; McMaster University, a nonprofit public academic institution, has contracts through the Health Information Research Unit under the supervision of Alfonso Iorio and R. Brian Haynes. These contracts involve professional and commercial publishers to provide newly published studies, which are critically appraised for research methodology and assessed for clinical relevance as part of the McMaster Premium Literature Service (McMaster PLUS). Cynthia Lokker and Rick Parrish receive partial compensation, and R. Brian Haynes is remunerated for supervisory responsibilities and royalties. Ashirbani Saha, Fangwen Zhou, and Muhammad Afzal are not affiliated with McMaster PLUS.
&lt;h4&gt;Funding/Support&lt;/h4&gt; Fangwen Zhou was funded through the Mitacs Business Strategy Internship grant (IT42947) with matching funds from EBSCO Canada.
&lt;h4&gt;Role of the Funder/Sponsor&lt;/h4&gt; The funders were not involved in the design and conduct of the study, collection, management, analysis, and interpretation of the data, preparation, review, or approval of the abstract, and decision to submit the abstract for presentation.
&lt;h4&gt;Acknowledgment&lt;/h4&gt; We thank the Digital Research Alliance of Canada for providing the computational resources.


The 10th Peer Review Congress

Understanding How a Language Model Assesses the Quality of Randomized Controlled Trials: Applying Shapley Additive Explanations to Encoder Transformer Classification Models

artificial intelligence and large language models

ethics and ethical concerns

quality of the literature

<p>Fangwen Zhou,<sup>1</sup> Muhammad Afzal,<sup>2</sup> Rick Parrish,<sup>1</sup> Ashirbani Saha,<sup>3</sup> R. Brian Haynes,<sup>1</sup> Alfonso Iorio,<sup>1,4</sup> Cynthia Lokker<sup>1</sup></p>
<h4>Objective</h4>
<p> Deep learning models for classifying published clinical literature to support critical appraisal have garnered wide attention.<sup>1</sup> To automate the critical appraisal workflow for McMaster’s Premium Literature Service (PLUS), a gold-standard database manually labeled by experts was used to develop well-performing models to classify the rigor of randomized controlled trials (RCTs).<sup>2</sup> However, due to the complexity of language models, the lack of transparency remains an important concern. This study explored Shapley Additive Explanations (SHAP)<sup>3</sup> to better understand how a deep learning model makes classification decisions.</p>
<h4>Design </h4>
<p>Details regarding the development and evaluation of the rigor classifiers are published elsewhere.<sup>2</sup> Briefly, classifiers were trained with titles and abstracts of original RCTs, that met or did not meet PLUS criteria for rigor (randomization, ≥10 participants per group, ≥80% follow-up, clinically important outcomes, and preplanned subgroup analyses, if applicable). Models were trained on 53,219 PLUS articles from 2003 to 2023, randomly split 80:10:10 into train, validate, and test sets. Articles in Clinical Hedges, a similar database that preceded PLUS, and PLUS articles from 2024 were used for external testing. A top-performing BioLinkBERT model, with an area under the receiver operating characteristic curve of 94%, was selected for this study. The SHAP partition explainer determined important tokens (words/subwords) for articles from the validate and test datasets, Clinical Hedges, and PLUS 2024. Tokens were combined into words for ease of interpretation, and their SHAP values were summed. The mean SHAP values, which indicate the aggregated average marginal contribution to model output over all samples, for the most impactful words with 100 or more occurrences were examined.</p>
<h4>Results </h4>
<p>Overall, 6,207,935 words were analyzed, of which 49,387 were unique. <b>Figure 25-0848</b> shows the 15 most impactful unique words for positive (rigorous) and negative (nonrigorous) classes. Terms such as <i>noninferiority</i> positively influenced rigor predictions with a mean SHAP value of 0.00904 (95% CI, 0.00844-0.00965), whereas <i>nonrandomized</i> (mean SHAP value, −0.02510; 95% CI, −0.03204 to −0.01816) had negative impacts. These results generally align with the manual appraisal criteria. However, tokens apparently unrelated to the criteria, such as tiotropium (mean SHAP value, 0.00571; 95% CI, 0.00508-0.00634), also had sizable impacts, indicating that they may have been correlated with higher rigor during manual appraisal. These patterns were learned and applied by the model, indicating a certain degree of overfitting.</p>

![](https://assets.underline.io/markdown_image/1/image/83c754796d3484f54b7a4aacbdad1110.png)

<h4>Conclusions </h4>
<p>This study demonstrates that SHAP helps understand which features influence a deep learning model’s rigor assessment of an RCT. Identifying influential features increases confidence in the model assessments and generalizability and reduces unease about the black-box nature of these models. Future work should explore SHAP with other critical appraisal tools and datasets and potentially integrate it into machine learning systems to improve user trust and model accountability in evidence synthesis and critical appraisal workflows.</p>
<h4>References</h4>
<p>1. Lokker C, Bagheri E, Abdelkader W, et al. Deep learning to refine the identification of high-quality clinical research articles from the biomedical literature: performance evaluation. <i>J Biomed Inform</i>. 2023;142:104384. doi:10.1016/j.jbi.2023.104384</p>
<p>2. Zhou F, Parrish R, Afzal M, et al. Benchmarking domain-specific pretrained language models to identify the best model for methodological rigor in clinical studies. <i>J Biomed Inform</i>. 2025;166:104825. doi:10.1016/j.jbi.2025.104825</p>
<p>3. Lundberg S, Lee SI. A unified approach to interpreting model predictions. <i>arXiv</i>. Preprint posted online May 22, 2017. doi:10.48550/arXiv.1705.07874</p>

<sup>1</sup>Health Information Research Unit, Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada, lokkerc@mcmaster.ca; <sup>2</sup>Faculty of Computing, Engineering and the Built Environment, Birmingham City University, Birmingham, UK; <sup>3</sup>Department of Oncology, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; <sup>4</sup>Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada.
<h4>Conflict of Interest Disclosures</h4> McMaster University, a nonprofit public academic institution, has contracts through the Health Information Research Unit under the supervision of Alfonso Iorio and R. Brian Haynes. These contracts involve professional and commercial publishers to provide newly published studies, which are critically appraised for research methodology and assessed for clinical relevance as part of the McMaster Premium Literature Service (McMaster PLUS). Cynthia Lokker and Rick Parrish receive partial compensation, and R. Brian Haynes is remunerated for supervisory responsibilities and royalties. Ashirbani Saha, Fangwen Zhou, and Muhammad Afzal are not affiliated with McMaster PLUS.
<h4>Funding/Support</h4> Fangwen Zhou was funded through the Mitacs Business Strategy Internship grant (IT42947) with matching funds from EBSCO Canada.
<h4>Role of the Funder/Sponsor</h4> The funders were not involved in the design and conduct of the study, collection, management, analysis, and interpretation of the data, preparation, review, or approval of the abstract, and decision to submit the abstract for presentation.
<h4>Acknowledgment</h4> We thank the Digital Research Alliance of Canada for providing the computational resources.


technical paper

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Xiangji Ying,<sup>1</sup> Colby J. Vorland,<sup>2</sup> Kiran Ninan,<sup>1</sup> Jean-Pierre Oberste,<sup>1</sup> Andrew W. Brown,<sup>3</sup> Riaz Qureshi,<sup>4</sup> Sirui Zhang,<sup>5</sup> Nicholas J. DeVito,<sup>6</sup> Matthew Page,<sup>7</sup> Ian J. Saldanha,<sup>8</sup> Halil Kilicoglu,<sup>9</sup> Evan Mayo-Wilson<sup>1</sup></p>
<h4>Objective </h4>
<p>Registries, such as ClinicalTrials.gov, and guidelines, including SPIRIT 2025 (Standard Protocol Items: Recommendations for Interventional Trials) and CONSORT 2025 (Consolidated Standards of Reporting Trials), define clinical trial outcomes using 5 elements: domain, measurement, metric, aggregation method, and time point. Changes between prospective registration and results reporting can introduce bias. Readers can manually compare trial documents, but doing so is resource intensive. Automated methods could facilitate checking and improve peer review. We used a chatbot to define outcomes registered on ClinicalTrials.gov, identify changes between prospective registrations and registry results, and describe those changes.</p>
<h4>Design </h4>
<p>We conducted a cross-sectional study using prospectively registered, completed randomized clinical trials from ClinicalTrials.gov (January 2000-January 2024). Building on the 5-element framework, we developed rules and categories to define outcomes and outcome changes. We used GPT-4o, o1, and o3 mini (Open AI) and developed and optimized prompts using a training set of 225 trials (2221 outcomes). We validated performance on 150 trials (1459 outcomes). We divided the task into 16 structured subtasks in the following steps: (1) defining outcomes in both versions, (2) matching outcomes, and (3) detecting changes in outcome elements. We provided cosine and Jaccard distances to help the chatbot match and compare outcomes. We evaluated all outcomes using the 3 chatbot models in January 2025. We selected the answer given by at least 2 of the models. Two human raters independently evaluated the chatbot results. We present preliminary findings based on unreconciled ratings of changes (final findings will be presented at the conference).</p>
<h4>Results </h4>
<p>The 150 validation trials reported a mean (SD) of 6.9 (5.0) outcomes in prospective registrations and 8.4 (6.0) outcomes in final versions. The final versions included 832 outcomes matched with prospective registrations and 428 additional outcomes and omitted 199 outcomes from prospective versions. We achieved 99.8% (95% CI, 99.4%-99.9%) accuracy in matching outcomes between versions (<b>Table 25-1142</b>). Accuracy for detecting changes was 87.9% (95% CI, 85.5%-90.1%) for measurement, 88.1% (95% CI, 85.7%-90.2%) for metric, 87.2% (95% CI, 84.8%-89.4%) for aggregation method, 94.5% (95% CI, 92.7%-95.9%) for cutoff, and 94.8% (95% CI, 93.1%-96.2%) for time point. On average, the language learning models completed the entire process (ie, matching, defining, comparing) in approximately 2 minutes per trial. It took humans 27 minutes per trial to evaluate the chatbot’s response.</p>

![](https://assets.underline.io/markdown_image/1/image/8c0ccf28450d3d85a18008523d330a86.png)

<h4>Conclusions </h4>
<p>A prompt-based approach was highly accurate in defining clinical trial outcomes and identifying outcome changes in ClinicalTrials.gov. This approach could be expanded to identify changes between registrations and manuscripts. Although it did not achieve perfect accuracy, our prompt-based approach could help editors and peer reviewers detect likely discrepancies that warrant further review. </p>

<sup>1</sup>Department of Epidemiology, University of North Carolina Gillings School of Global Public Health, Chapel Hill, NC, US, evan.mayo-wilson@unc.edu; <sup>2</sup>Department of Epidemiology and Biostatistics, Indiana University School of Public Health–Bloomington, IN, US; <sup>3</sup>Department of Biostatistics, University of Arkansas for Medical Sciences; Arkansas Children’s Research Institute, Little Rock, AR, US; <sup>4</sup>Department of Ophthalmology, School of Medicine; Department of Epidemiology, School of Public Health, University of Colorado Anschutz Medical Campus, Denver, CO, US; <sup>5</sup>Department of Epidemiology, School of Public Health, Brown University, Providence, RI, US; <sup>6</sup>The Bennett Institute for Applied Data Science, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK; <sup>7</sup>Methods in Evidence Synthesis Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia; <sup>8</sup>Departments of Epidemiology (Primary) and Health Policy and Management (Joint), Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, US; <sup>9</sup>School of Information Sciences, University of Illinois at Urbana-Champaign, IL, US.

<h4>Conflict of Interest Disclosures</h4> None reported.
<h4>Funding/Support</h4> National Library of Medicine, National Institutes of Health (R01LM014079).
<h4>Role of Funder/Sponsor</h4> The funder played no roles in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation. The views expressed do not necessarily reflect the views of the National Institutes of Health.
<h4>Additional Information</h4> Our protocol is publicly available and we will share prompts, code, and datasets on the Open Science Framework (https://osf.io/2tyh3/).


Using GPT to Identify Changes in Clinical Trial Outcomes Registered on ClinicalTrials.gov

<p>Lukas Hughes-Noehrer,<sup>1</sup> Noémie Aubert Bonn<sup>1</sup></p>
<h4>Objective </h4>
<p>Open and transparent research practices are essential for high-quality research. Despite growing efforts to promote these practices through new funding policies, a rethinking of evaluation criteria, and investment in support and infrastructure, widespread adoption across disciplines remains challenging. Comprehensive data and analysis of the perceived risks and barriers to implementing open research practices within UK higher education institutions (HEIs) is still lacking.</p>
<h4>Design </h4>
<p>In 2025, we conducted an inductive thematic analysis of openly available data from the UK Reproducibility Network Open and Transparent Research Practices Survey,<sup>1</sup> which was conducted between December 2022 and April 2023. The survey addressed 14 topics related to open research practices (eg, research coproduction, open-source code and software, reproducibility, open access, FAIR principles [findability, accessibility, interoperability, and reusability], and conflicts of interests), with respondents answering questions on practice, support, and approaches to each practice. This analysis focuses on responses to the open-text questions specifically referring to<i> </i>risks or barriers to researchers or to their fields of study across all 14 topics. Following SRQR guidelines,<sup>2</sup> we ensured transparency in our analytic process. One researcher analyzed the data and developed initial codes, which have been reviewed by a second coder, before agreeing on collaboratively refined themes through iterative discussion. We used NVivo, version 14 (Lumivero) for analysis and data management. Reflexivity was maintained through regular team debriefs. Ethical approval was not required as the data were publicly available and anonymized. </p>
<h4>Results </h4>
<p>The survey received 2567 responses from research-active staff at 15 UK HEIs. Of these, 511 (20%) responses were complete. Participants reported barriers and risks to the 14 open research topics (<b>Table 25-0863</b>), which we grouped into 3 overarching categories: personal, systemic and institutional, and ethical and quality. In the personal category, respondents highlighted lacking motivation to undertake open research practices, with some perceiving that such practices should not rely on researchers and others perceiving such practices as a mere tick-box exercise. Respondents also feared that open research practices would expose errors and make them vulnerable to criticism. In the systemic and institutional category, respondents described barriers that made them unable to practice open research, such as methodological or disciplinary barriers. They also highlighted missing infrastructure and a lack of resources. Others also mentioned that institutions did not support them adequately, did not provide appropriate training, and did not recognize or incentivize open research practices, making it difficult for them to engage in open research. On an ethical and quality level, respondents worried that open research practices introduced new ethical concerns, or even that they detract from high-quality research and innovation.</p>

![](https://assets.underline.io/markdown_image/1/image/1d5e47286cd4c89ef1c9c68d0e982600.png)

<h4>Conclusions </h4>
<p>These findings underscore critical challenges hindering the adoption of open research practices in UK HEIs. Addressing these barriers requires enhanced institutional support, targeted training, and robust incentives to empower researchers. Mitigating these challenges could significantly improve the quality and transparency of academic research.</p>
<h4>References</h4>
<p>1. Hughes-Noehrer L, Aubert Bonn N, De Maria M, et al. UK Reproducibility Network open and transparent research practices survey dataset. <i>Sci Data</i>. 2024;11:912. doi:10.1038/s41597-024-03786-z</p>
<p>2. O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. <i>Acad Med</i>. 2014;89(9):1245-1251. doi:10.1097/ACM.0000000000000388</p>

<sup>1</sup>Department of Computer Science, The University of Manchester, Manchester, UK, lukas.noehrer@manchester.ac.uk.
<h4>Conflict of Interest Disclosures</h4> None reported.
<h4>Funding/Support</h4> This work was partially funded by the Research England Development Fund. Research England had no further involvement in the research life cycle.
<h4>Acknowledgment</h4> We want to thank all members of the United Kingdom Reproducibility Network who have contributed to the Open and Transparent Research Practices survey.


Perceived Risks and Barriers to Open Research Practices in UK Higher Education

<p>Ayu Putu Madri Dewi,<sup>1</sup> Melissa L. Rethlefsen,<sup>2</sup> Sara Schroter,<sup>3,4</sup> Florian Naudet,<sup>5,6</sup> Nicholas J. DeVito,<sup>7</sup> Constant Vinatier,<sup>5</sup> Inge Stegeman,<sup>8,9</sup> Mariska Leeflang,<sup>1</sup> Gowri Gopalakrishna<sup>1,10</sup></p>
<h4>Objective</h4>
<p> <i>The BMJ</i> is one of the first journals to require authors to share analytic codes from all studies as well as data from all trials,<sup>1</sup> but whether this enhances reproducibility has not been tested. We will assess whether adding an open science checklist to the journal peer review process improves the reproducibility of scientific findings in submitted manuscripts to <i>The BMJ</i> and <i>BMJ Open</i>.</p>
<h4>Design</h4>
<p> This study is registered on Open Science Framework<sup>2</sup> and follows the SPIRIT guidelines. Reproducibility was defined as obtaining the same primary result when rerunning analyses using the original data and code. We used a 2-arm parallel randomized controlled trial design, equal allocation, and random block randomization (block sizes of 4, 6, and 8; generated in R version 4.3.2 [The R Foundation]). We hypothesized that the intervention will increase reproducibility of primary outcomes from 50% in the control arm to 70% in the intervention arm (90% power; α = .05). Our minimum sample size is 260 manuscripts (minimum 130 manuscripts per arm). Eligible manuscripts include quantitative observational or experimental designs and systematic reviews containing a meta-analysis. In the intervention arm, the research team completed an open science checklist assessing whether a set of predefined open science items<sup>3</sup> (<b>Table 25-1009</b>) were reported in the manuscripts. The completed checklist was shared with the authors as part of the standard peer review process. Manuscripts in the control arm followed the usual peer review process. The primary outcome was the difference in reproducibility of the authors’ reported primary outcomes between the 2 arms, as assessed by blinded outcome assessors. The secondary outcome was the proportion of manuscripts with openly shared data and code. If the data, code, or methods were unavailable or only available on request, authors were not contacted, as they were blinded to the study, and the outcome assessors did not proceed. Randomization began on March 3, 2025, with outcome assessments conducted after the first manuscript revision.</p>

![](https://assets.underline.io/markdown_image/1/image/d5ae7ca2c0806a938675560f7bc7064a.png)

<h4>Results</h4>
<p> By April 18, 2025, we had completed randomization of 403 manuscripts. This includes open science intervention in 22 of 43 manuscripts from <i>The BMJ</i> and 179 of 360 manuscripts from <i>BMJ Open</i>. As of June 5, 2025, 78 manuscripts had received a decision letter, including 39 rejections, while the remainder proceeded to reproducibility assessment. Analysis is ongoing.</p>
<h4>Conclusions</h4>
<p> This study will provide robust evidence on whether incorporating an open science checklist into the peer review process improves the reproducibility of scientific findings.</p>
<h4>References</h4>
<p>1. Loder E, Macdonald H, Bloom T, Abbasi K. Mandatory data and code sharing for research published by <i>The BMJ</i>. <i>BMJ</i>. 2024;384:q324. doi:10.1136/bmj.q324</p>
<p>2. Dewi APM, Rethlefsen ML, Schroter S, et al. Measuring the efficacy of an intervention to improve reproducibility of scientific manuscripts: study protocol for a randomized controlled trial. OSF. Accessed July 2, 2025. <a href="https://osf.io/b5g6y/%20">https://osf.io/b5g6y/</a> </p>
<p>3. Cobey KD, Haustein S, Brehaut J, et al. Community consensus on core open science practices to monitor in biomedicine. <i>PLoS Biol</i>. 2023;21(1):e3001949. doi:10.1371/journal.pbio.3001949</p>

<sup>1</sup>Epidemiology and Data Science Department, Amsterdam UMC, Amsterdam, the Netherlands; <sup>2</sup>Health Sciences Library and Informatics Center, University of New Mexico, Albuquerque, NM, US; <sup>3</sup>BMJ Publishing Group, London, UK; <sup>4</sup>Faculty of Public Health and Policy, London School of Hygiene and Tropical Medicine, London, UK; <sup>5</sup>Univeristy of Rennes, CHU Rennes, INSERM, EHESP, Irset (Institut de recherche en santé, environnement et travail), UMR_S 1085, Rennes, France; <sup>6</sup>Institut Universitaire de France, Paris, France; <sup>7</sup>Bennett Institute for Applied Data Science, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK, nicholas.devito@phc.ox.ac.uk; <sup>8</sup>Department of Otorhinolaryngology and Head & Neck Surgery, University Medical Center Utrecht, Utrecht, the Netherlands; <sup>9</sup>Brain Center, University Medical Center Utrecht, Utrecht, the Netherlands; <sup>10</sup>Department of Epidemiology, Faculty of Health, Medicine, and Life Sciences Maastricht University, Maastricht, the Netherlands.
<h4>Conflict of Interest Disclosures</h4> None reported.


Use of an Open Science Checklist and Reproducibility of Findings: A Randomized Controlled Trial

<p>Benjamin Speich,<sup>1</sup> Ala Taji Heravi,<sup>1</sup> Johannes M. Schwenke,<sup>1</sup> Christof M. Schönenberger,<sup>1</sup> Lena Hausheer,<sup>1</sup> Dmitry Gryaznov,<sup>1</sup> Jason W. Busse,<sup>2,3</sup> Manuela Covino,<sup>1</sup> Szimonetta Lohner,<sup>4</sup> Malena Chiaborelli,<sup>1</sup> Ruben<sup> </sup>Ramirez,<sup>1</sup> Ramon Saccilotto,<sup>5</sup> Erik von Elm,<sup>6</sup> Arnav Agarwal,<sup>3</sup> Julian Hirt,<sup>7</sup> David Mall,<sup>1</sup> Alain Amstutz,<sup>1</sup> Selina Epp,<sup>1</sup> Dominik Mertz,<sup>3</sup> Anette Blümle,<sup>8</sup> Belinda von Niederhäusern,<sup>9</sup> Ayodele Odutayo,<sup>10</sup> Alexandra N. Griessbach,<sup>1</sup> Sally Hopewell,<sup>11</sup> Matthias Briel,<sup>1,3</sup> for the ASPIRE Study Group</p>
<h4>Objective</h4>
<p> Previous studies found that approximately one-third of randomized clinical trials (RCTs) were discontinued prematurely and that the most common reason for stopping early was poor recruitment of participants. To minimize research waste, it is crucial that all RCTs are registered and make their results available. Hence, we aimed to (1) assess the fate of RCTs approved by ethics committees in 2016 in terms of nonregistration, discontinuation, and nonpublication and (2) examine RCT characteristics associated with discontinuation due to poor recruitment and nonpublication of RCT results.</p>
<h4>Design</h4>
<p> We had access to 347 RCT protocols that were approved in 2016 by research ethics committees in the UK, Switzerland, Germany, and Canada. Key trial characteristics were extracted from approved trial protocols. <span lang="de-DE">Trial registration was verified using registration numbers from the protocol and by searching the </span>World Health Organization<span lang="de-DE"> International Clinical Trials Registry Platform, ClinicalTrials.gov, </span>European Union Clinical Trials Register<span lang="de-DE">, ISRCTN, and Google; trials were deemed nonregistered if not found. We searched for full-text publications in PubMed, Google Scholar, and Scopus. Searches for both registration and publication used (</span>1<span lang="de-DE">) full titles</span>,<span lang="de-DE"> (</span>2<span lang="de-DE">) short titles</span>,<span lang="de-DE"> (</span>3<span lang="de-DE">) study acronyms</span>,<span lang="de-DE"> and (</span>4<span lang="de-DE">) the study population and intervention. </span>All searches were conducted in duplicate (last search, July 2024). In case the status of an RCT was unclear, we contacted the corresponding ethics committee or the principal investigator. We reported the proportion of nonregistered RCTs, discontinued RCTs (including reason for early discontinuation), and nonpublished RCTs (considering peer-reviewed publications and results in trial registries). </p>
<h4>Results</h4>
<p> Of the 347 included RCTs from 2016, 20 (5.8%) were nonregistered (industry-sponsored RCTs, 5 of 181 [2.8%]; nonindustry RCTs, 15 of 166 [9.0%]). Approximately one-third of RCTs (108 of 347 [31.1%]) were discontinued, most often due to poor recruitment (49 of 108 [45.4%]). A total of 276 of 347 RCTs (79.5%) made their results available at any source, 249 (71.8%) as a peer-reviewed publication, and 170 (49.0%) in the trial registry. Discontinued RCTs had lower result availability than completed RCTs (discontinued, 74 of 108 [68.5%]; completed, 202 of 226 [89.4%]). Results from industry-sponsored trials were more often available compared with nonindustry trials (92% vs 66%). This difference was driven by the fact that only 17 of 166 nonindustry RCTs (10.2%) reported results in trial registries compared with 153 of 181 industry trials (84.5%). The status of 4% of RCTs remained unclear despite our efforts to contact investigators.</p>
<h4>Conclusions</h4>
<p> Nonregistration, premature discontinuation due to poor recruitment, and nonpublication of RCT results remain major challenges, especially for nonindustry trials.</p>

<sup>1</sup>Division of Clinical Epidemiology, Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland, johannesmanteo.schwenke@usb.ch; <sup>2</sup>Department of Anesthesia, McMaster University, Hamilton, Ontario, Canada; <sup>3</sup>Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; <sup>4</sup>MTA–PTE Lendület Momentum Evidence in Medicine Research Group, Medical School, University of Pécs, Pécs, Hungary; <sup>5</sup>Clinical Trial Unit, Department of Clinical Research, University of Basel and University Hospital Basel, Basel, Switzerland; <sup>6</sup>Cochrane Switzerland, Centre for Primary Care and Public Health (Unisante), University of Lausanne, Lausanne, Switzerland; <sup>7</sup>Research Center for Clinical Neuroimmunology and Neuroscience Basel, University Hospital Basel and University of Basel, Basel, Switzerland; <sup>8</sup>Clinical Trials Unit, Faculty of Medicine and Medical Center, University of Freiburg, Freiburg, Germany; <sup>9</sup>Roche Pharma AG, Grenzach-Wyhlen, Germany; <sup>10</sup>University Health Network, Division of Nephrology, Department of Medicine, University of Toronto, Toronto, Ontario, Canada; <sup>11</sup>Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK.
<h4>Conflict of Interest Disclosures</h4> Benjamin Speich and Matthias Briel received unrestricted grants from Moderna for the conduct of the COVERALL-2 and COVERALL-3 study. Benjamin Speich has received honoraria from Moderna and Roche for presenting study results not related to this work. Johannes M. Schwenke is paid by the Swiss National Science Foundation for work unrelated to this project. No other disclosures were reported.
<h4>Funding/Support</h4> The study is supported by the Swiss Federal Office of Public Health (Matthias Briel).
<h4>Role of the Funder/Sponsor</h4> The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.
<h4>Group Information</h4> The ASPIRE Study Group included Dmitry Gryaznov, Belinda von Niederhäusern, Benjamin Speich, Benjamin Kasenda, Elena Ojeda-Ruiz, Anette Blümle, Stefan Schandelmaier, Dominik Mertz, Ayodele Odutayo, Yuki Tomonaga, Alain Amstutz, Christiane Pauli-Magnus, Viktoria Gloy, Szimonetta Lohner, Karin Bischoff, Katharina Wollmann, Laura Rehner, Joerg J. Meerpohl, Alain Nordmann, Katharina Klatte, Nilabh Ghosh, Ala Taji Heravi, Jacqueline Wong, Ngai Chow, Patrick Jiho Hong, Kimberly McCord, Sirintip Sricharoenchai, Jason W. Busse, Lukas Kübler, Pooja Gandhi, Zsuzsanna Kontar, Julia Hüllstrung, Mona Elafy, Arnav Agarwal, Ramon Saccilotto, Alexandra N. Griessbach, Christof Schönenberger, Matthias Schwenkglenks, Giusi Moffa, Lars G. Hemkens, Sally Hopewell, Erik von Elm, and Matthias Briel.
<h4>Additional Information</h4> Matthias Briel is a co–corresponding author (matthias.briel@usb.ch).


Nonregistration, Discontinuation, and Nonpublication of Randomized Trials in Switzerland, the UK, Germany, and Canada: An Updated Meta-Research Study

<p>David Blanco,<sup>1</sup> Elizabeth Loder,<sup>2,3</sup> Sophie Cook,<sup>3</sup> Martí Casals,<sup>4,5</sup> Jordi Cortés,<sup>6</sup><span lang="ar-SA"> Aïda Cadellans-Arroniz,</span><sup>1</sup> Victor Zárate,<sup>1</sup> Ella Hubbard,<sup>3</sup> Sara Schroter<sup>3,7</sup></p>
<h4>Objective </h4>
<p>In 2005, the International Committee of Medical Journal Editors (ICMJE) mandated that all clinical trials be properly registered—that is, registered in an approved registry at or before the enrollment of the first participant.<sup>1</sup> Despite improved registration rates, many trials published in journals claiming adherence to ICMJE registration policy remain improperly registered.<sup>2</sup> This study aimed to (1) identify variables associated with improper registration; (2) examine types of registration deficiencies, publication status, and disclosure of registration issues in improperly registered trials; and (3) assess authors’ claims of proper registration.</p>
<h4>Design </h4>
<p>This observational study analyzed 239 improperly and 239 properly registered trials submitted to <i>The BMJ</i> (2019-2023). For objective 1, we collected data on trial design, setting, sample size, intervention type, number of authors, corresponding author region, mention of Consolidated Standards of Reporting Trials (CONSORT) guidelines, submission year, and funding source. For objective 2, we examined improperly registered trials submitted between 2019 and 2021, focusing on registration deficiency type (retrospective registration in approved registry, registration in unapproved registry, or no registration), registration delay (for retrospectively registered trials), and publication status. For published trials, we recorded Journal Impact Factor, ICMJE affiliation, time to publication, and disclosure of registration issues. For objective 3, we assessed authors’ claims of prospective registration for improperly registered trials. Analyses included multivariable logistic regression and descriptive statistics.</p>
<h4>Results </h4>
<p>Several variables were associated with reduced odds of improper registration: larger sample size (101-500 vs 1-100; odds ratio [OR], 0.43 [95% CI, 0.22-0.84]), corresponding authors from Oceania (reference: Europe; OR, 0.35 [95% CI, 0.14-0.82]), more authors (10 vs 1; OR, 0.53 [95% CI, 0.32-0.87]), CONSORT mention (OR, 0.22 [95% CI, 0.06-0.67]), recent submissions (2021-2023 vs 2019-2020; OR, 0.63 [95% CI, 0.42-0.96]), and funding (eg, nonprofit vs no funding; OR, 0.20 [95% CI, 0.09-0.41]). Trials with authors from Asia (OR, 1.75 [95% CI, 1.07-2.89]) had higher odds of improper registration. Of 176 improperly registered trials, 82.4% (n = 145) were retrospectively registered in approved registries (median delay, 6.54 months; Q1, 2.50 months; Q3, 18.67 months), 13.1% (n = 23) were unregistered, and 4.5% (n = 8) used unapproved registries. Most (88.1% [n = 155 trials]) were later published, including 1 in <i>The BMJ</i>. Among these, 89.0% (n = 138) appeared in journals with an Impact Factor (median, 5.39; Q1, 3.98; Q3, 10.40) and 62.0% (n = 96) in journals claiming adherence with ICMJE registration policy. Median time to publication was 12 months (Q1, 8 months; Q3, 19 months). Only about one-sixth explicitly acknowledged the registration issue at the time of publication. Of 72 responses to <i>The BMJ</i>’s submission question on prospective registration (2021-2023), 83.3% (n = 60) incorrectly claimed compliance.</p>
<h4>Conclusions </h4>
<p>We identified variables associated with improper trial registration. Retrospective registration was common, and most improperly registered trials rejected by <i>The BMJ</i> were later published in other ICMJE-affiliated journals. ICMJE journals should strengthen processes to better identify and reject improperly registered trials.</p>
<h4>References</h4>
<p>1. International Committee of Medical Journal Editors. Clinical trials. Accessed January 23, 2025. <a href="https://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/clinical-trial-registration.html">https://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/clinical-trial-registration.html</a></p>
<p>2. Trinquart L, Dunn AG, Bourgeois FT. Registration of published randomized trials: a systematic review and meta-analysis. <i>BMC Med</i>. 2018;16(1):173. doi:10.1186/s12916-018-1168-6</p>

<sup>1</sup>Department of Physiotherapy, Universitat Internacional de Catalunya, Barcelona, Spain, dblanco@uic.es; <sup>2</sup>Department of Neurology, Brigham and Women’s Hospital, Boston, MA, US; <sup>3</sup>The BMJ, London, UK; <sup>4</sup>National Institute of Physical Education of Catalonia, University of Barcelona, Barcelona, Spain; <sup>5</sup>Sport and Physical Activity Studies Centre, Faculty of Medicine, University of Vic - Central University of Catalonia, Barcelona, Spain; <sup>6</sup>Biostatistics and Bioinformatics Research Group, Department of Statistics and Operations Research and Institute for Research and Innovation in Health, Universitat Politècnica de Catalunya, Barcelona, Spain; <sup>7</sup>Faculty of Public Health and Policy, London School of Hygiene & Tropical Medicine, London, UK.
<h4>Conflict of Interest Disclosures</h4> Sophie Cook and Sara Schroter are full-time employees of the BMJ Group and Elizabeth Loder is a part-time employee.


Factors Associated With Improper Clinical Trial Registration, Registration Deficiencies, and Publication Status of Submissions to the BMJ

<p>Eun Hye Lee,<sup>1,2,3,4</sup> San Lee,<sup>1,3,5</sup> Jae Il Shin,<sup>3,4,6</sup> John P. A. Ioannidis<sup>1,7,8,9</sup></p>
<h4>Objective</h4>
<p> East Asia has become a major hub for clinical trials.<sup>1</sup> Although a few studies have compared ClinicalTrials.gov with local registries in specific disease domains, comprehensive comparisons remain limited.<sup>2</sup> This study analyzes registered clinical trial trends in East Asia during 2014 to 2025 and compares them with trends in the US.</p>
<h4>Design</h4>
<p> We extracted clinical trial data through the World Health Organization–operated International Clinical Trials Registry Platform (ICTRP), which integrates ClinicalTrials.gov and local registries. Because Taiwan’s registry is not part of the ICTRP, we collected its data separately. We analyzed overall clinical trial trends and filtered randomized clinical trials (RCTs), categorizing them by location, target size, and disease category. Trials conducted in only 1 country were categorized as domestic, whereas those conducted in 2 or more countries were classified as international. We explored the proportion of studies registered prospectively, defined as within 1 month of the trial start date.<sup>3</sup></p>
<h4>Results</h4>
<p> China experienced rapid growth in clinical trials, surpassing both Japan and the US in total trials and RCTs (<b>Figure 25-1020</b>). In China, the number of registered trials increased markedly from 2578 total trials (1521 RCTs) in 2014 to 16,612 (7798 RCTs) in 2023. In contrast, the US showed a more modest increase from 8841 trials (4056 RCTs) in 2014 to 9100 (4619 RCTs) in 2023. Except for China, all other countries experienced a postpandemic decline, with annual trial registrations decreasing by approximately 3% to 10% in recent years.<span class="CharOverride-18"> </span>Neoplastic diseases and cardiovascular and metabolic diseases accounted for the largest percentages, each representing approximately 15% to 25% of all registered trials. In the US, mental health trials increased from 11.9% in 2014 to 16.0% in 2023, reflecting growing interest in this area. In 2023, China’s RCTs were predominantly domestic, with international trials making up only 1.8% (140) of the total, while 13.1% (603) of RCTs in the US were international. Domestic RCTs were predominantly small-sized trials (&lt;100 participants), while international RCTs were more commonly medium-sized (100-499 participants) or large-sized (≥500 participants) trials. In 2023, 75% of China’s RCTs (5841 of 7798) and 81% of Japan’s RCTs (1081 of 1339) were domestic trials registered in local registries, highlighting the predominant use of local registries in both countries. Although the trend has increased over the past decade, a considerable percentage of RCTs, ranging from 15% to 35% across countries, were still not registered prospectively in 2023. We recorded some inconsistencies and missing information across existing registries. Updated data to mid-2025 will be presented at the congress.</p>

![](https://assets.underline.io/markdown_image/1/image/f1904b3cba2f1d83b5fb2807bec4e17d.png)

<h4>Conclusions</h4>
<p> The study highlights the increasing prominence of East Asia, particularly China, in registered clinical trials, though with a primarily domestic focus. Improvements in existing trial registries covering East Asia are desirable.</p>
<h4>References</h4>
<p>1. Ali S, Egunsola O, Babar ZUD, Hasan SS. Clinical trials in Asia: a World Health Organization database study. <i>Perspect Clin Res.</i> 2019;10(3):121-124. doi:10.4103/picr.PICR_109_18</p>
<p>2. Doi M, Yukawa K, Sato H. Characteristics of Asian 4 countries on cancer clinical trials registered in the international clinical trials registry platform between 2005 and 2018. <i>Chin Clin Oncol</i>. 2021;10(3):28. doi:10.21037/cco-21-17</p>
<p>3. Klatte K, Sluka c, Gloy V, et al. Towards full clinical trial registration and results publication: longitudinal meta-research study in Northwestern and Central Switzerland. <i>BMC Med Res Methodol</i>. 2023;23(1):27. doi:10.1186/s12874-023-01840-9</p>

<sup>1</sup>Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, US, jioannid@stanford.edu; <sup>2</sup>Division of Pulmonology, Allergy and Critical Care Medicine, Department of Internal Medicine, Yongin Severance Hospital, Yonsei University College of Medicine, Yongin-si, Gyeonggi-do, South Korea; <sup>3</sup>The Center for Medical Education Training and Professional Development, Yonsei Donggok Medical Education Institute, Seoul, Korea; <sup>4</sup>Severance Underwood Meta-Research Center, Institute of Convergence Science, Yonsei University, Seoul, South Korea; <sup>5</sup>Department of Psychiatry and the Institute of Behavioral Science in Medicine, Yonsei University College of Medicine, Seoul, South Korea; <sup>6</sup>Department of Pediatrics, Yonsei University College of Medicine, Seoul, South Korea; <sup>7</sup>Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Stanford, CA, US; <sup>8</sup>Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, US; <sup>9</sup>Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, US.
<h4>Conflict of Interest Disclosures</h4> John P. A. Ioannidis is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.
<h4>Funding/Support</h4> The work of John P. A. Ioannidis is supported by an unrestricted gift from Sue and Bob O’Donnell to Stanford University.
<h4>Role of the Funder/Sponsor</h4> The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
<h4>Additional Information</h4> A language model based on Bidirectional Encoder Representations from Transformers (BERT) was employed on December 30, 2024, to classify disease categories in RCTs. Specifically, PubMedBERT, a version pretrained on biomedical literature from PubMed, was used to automate the disease classification of clinical trials. San Lee takes responsibility for the integrity of the content. Eun Hye Lee and San Lee contributed equally to this study.


Registered Clinical Trial Trends in East Asia and the United States, 2014 to 2025

<p>Aidan Christopher Tan,<sup>1,2</sup> Yiyi Lin,<sup>1</sup> Michellie Lian,<sup>1</sup> Zhilin Ren,<sup>1</sup> Tony Lian,<sup>1</sup> Vincent Yuan,<sup>3</sup> Angela Webster,<sup>1</sup> Anna Lene Seidler<sup>1</sup></p>
<h4>Objective</h4>
<p> We aimed to describe researcher adherence to journal data sharing policies across health research. This included describing what factors increase intention to share data and how journal policies requiring data sharing impact intended data sharing.</p>
<h4>Design</h4>
<p> This was a cross-sectional study of all original research published in the highest-impact medical journals in 2022. Journals were included if they ranked among the top 5 in Impact Factor in 59 fields of medicine in the 2020 Journal Citation Reports, had a data sharing policy that either required or recommended sharing data, and published original research. Journal websites were searched to identify all original research published in 2022. Data were manually collected on study characteristics (study type, industry involvement, and COVID-19 relevance) and data sharing plans (data sharing statements and data sharing intentions extracted from publicly available research protocols and articles). Articles were descriptively analyzed by these journal data sharing policies, study characteristics, and data sharing plans.</p>
<h4>Results</h4>
<p> Of 134 journals included, 36 (27%) required data sharing and 98 (73%) recommended data sharing. The analysis included 1868 interventional studies (1383 randomized clinical trials [74%] and 485 nonrandomized trials [26%]) and 10,368 observational studies (4814 cohort studies [46%], 4166 cross-sectional studies [40%], 1100 case-control studies [11%], and 288 case series/reports [3%]). Publicly available research protocols were available for 1993 of 10,243 studies (16%), of which 1153 (58%) had an initial data sharing statement. Only 1023 interventional studies (55%) and 4511 observational studies (44%) in journals that recommended or required data sharing actually intended to share data. Most studies only intended to share data underlying the published results with researchers, for purposes and by mechanisms at the discretion of and subject to approval by the principal investigator and without supporting documents or specified timeframes. Factors associated with increased intention to share data included journal policies that required data sharing and data sharing statements as well as industry involvement and COVID-19 relevance for interventional studies. For journals with policies that required data sharing, researchers were more likely to share more data (ie, all data collected during the study) and supporting documents (ie, supporting documents) in a timelier manner (ie, immediately following publication and indefinitely) to more people (ie, to anyone who wishes to access the data) for more purposes (ie, for any purpose) and by easier means (ie, with unrestricted access through a third-party website).</p>
<h4>Conclusions</h4>
<p> Journals that required data sharing had higher rates of studies that intended to share data compared with journals that only recommended data sharing. However, most studies that intended to share data had restrictions on sharing. In addition to requiring data sharing and data sharing statements, journals should define and explain data sharing and individual participant data, better review data sharing statements and the reasonableness of justifications to not share data, and provide additional guidance on the operationalization of data sharing in accordance with best practice.</p>

<sup>1</sup>NHMRC Clinical Trials Centre, University of Sydney, Sydney, Australia, aidan.tan@sydney.edu.au; <sup>2</sup>Sydney School of Public Health, University of Sydney, Sydney, Australia, <sup>3</sup>University of New South Wales, Sydney, Australia.
<h4>Conflict of Interest Disclosures</h4> None reported.
<h4>Funding/Support</h4> This work was supported by a research contract from CT:QI.
<h4>Role of the Funder/Sponsor</h4> CT:QI had no role in the study design, conduct, analysis, or reporting.


Researcher Adherence to Journal Data Sharing Policies: A Meta-Research Study

<p>Robert Thibault,<sup>1,2</sup> Dana E. Cobb-Lewis,<sup>1,2</sup> Matt Lewis,<sup>1,2</sup> Devin Snyder,<sup>1,2</sup> Cornelis Blauwendraat,<sup>1,2</sup> Sonya Dumanis<sup>1,2</sup></p>
<h4>Objective </h4>
<p>We examined whether a funder-led open science policy and compliance workflow could increase the deposition and unambiguous identification of research data, code, protocols, and key laboratory materials (eg, cell lines, antibodies).</p>
<h4>Design </h4>
<p>A cross-sectional study was conducted on articles supported by the Aligning Science Across Parkinson’s (ASAP) Collaborative Research Network (CRN) and published between January 1, 2024, and April 30, 2025 (N = 102). The study was exploratory, and thus was not registered, and does not present inferential statistics. ASAP has a thorough Open Science Policy<sup>1</sup> and Compliance Workflow,<sup>2</sup> which require grantees to send manuscript drafts to ASAP staff, post preprints, deposit research outputs, and unambiguously identify research inputs. The workflow is managed by an ASAP staff member who integrates automated and manual assessments to provide a grantee with systematic feedback outlining the actions required to align their manuscript draft with the ASAP Open Science Policy.</p>
<h4>Results </h4>
<p>Between the version of a manuscript first shared with ASAP staff and the associated final publication, there were substantial increases in the deposition of newly generated datasets (21% for the first version to 88% for the published version), unambiguous identification of reused datasets (72% to 86%), deposition of newly generated code (15% to 72%), unambiguous identification of software used (35% to 79%), registration of newly generated key lab materials (22% to 78%), unambiguous identification of key lab materials used (44% to 87%), and deposition of newly generated protocols and unambiguous identification of existing protocols (35% to 81%; data on protocols was collapsed) (<b>Table 25-0892</b>). Of the included publications, 98% (100 of 102) had an associated preprint. Preprints were posted a median (IQR) of 4 (−7 to 50) days before submission to the journal in which they were eventually published and 236 (150-328) days before they were published in a journal. To facilitate this workflow, grantees share each manuscript alongside a Key Resource Table that lists persistent identifiers for the associated datasets, code, software, protocols, and key lab materials. Collating these Key Resource Tables provides a living and near-comprehensive log of all the inputs and outputs from ASAP CRN–funded research.</p>

![](https://assets.underline.io/markdown_image/1/image/42436f84ba0deebd5496f77519d9878a.png)

<h4>Conclusions </h4>
<p>A funder-led intervention to monitor and support a robust open science policy can foster the posting of preprints and the sharing of data, code, protocols, and key lab materials.</p>
<h4>References</h4>
<p>1. Thibault R. ASAP Open Science Policy Handbook. Zenodo. 2024. doi:10.5281/zenodo.13769766</p>
<p>2. Cobb-Lewis DE, Snyder D, Dumanis S, et al. Investing in open science: key considerations for funders. <i>bioRxiv</i>. 2024:12.09.627554. doi:10.1101/2024.12.09.627554</p>

<sup>1</sup>Aligning Science Across Parkinson’s (ASAP), Chevy Chase, MD, US, openscience@parkinsonsroadmap.org; <sup>2</sup>Coalition for Aligning Science (CAS), Chevy Chase, MD, US.
<h4>Conflict of Interest Disclosures</h4> All authors are employees or contractors of CAS, the managing organization for ASAP.
<h4>Funding/Support</h4> The authors conducted this research as part of their employment or contract with CAS. No additional funding was sought or acquired to conduct this research.
<h4>Acknowledgment</h4> We thank the Michael J. Fox Foundation for Parkinson’s Research for their partnership in implementing the ASAP research initiative and for their assistance with the Open Science Compliance Workflow. We thank DataSeer for their active part in the development and conduct of the ASAP Open Science Compliance Workflow.
<h4>Additional Information</h4> The authors’ ORCIDs are as follows: Robert Thibault, https://orcid.org/0000-0002-6561-3962; Dana E. Cobb-Lewis, https://orcid.org/0000-0002-4104-0311; Matt Lewis, https://orcid.org/0000-0002-9091-5516; Devin Snyder, https://orcid.org/0000-0001-8528-7538; Cornelis Blauwendraat, https://orcid.org/0000-0001-9358-8111; Sonya Dumanis, https://orcid.org/0000-0002-3345-9497. ChatGPT-4o (OpenAI) was used between February 10, 2025, and June 10, 2025, to help write the R Markdown script used to analyze the data. Robert Thibault takes responsibility for the integrity of the content generated. Data, codebooks, code, results, and instructions for how to reproduce our results are available on GitHub at https://github.com/robert-thibault/peer-review-congress-abstract and have been assigned a persistent identifier via Zenodo at https://doi.org/10.5281/zenodo.15832432. The ASAP Open Science Policy is available at https://doi.org/10.5281/zenodo.13769765. A preprint detailing the ASAP Open Science Compliance Workflow is available at https://doi.org/10.1101/2024.12.09.627554.


A Funder-Led Intervention to Increase the Sharing of Data, Code, Protocols, and Key Laboratory Materials

<p>Kyobin Hwang,<sup>1</sup> Zexing Song,<sup>2,3</sup> Marsida Stafa,<sup>3</sup> Jodie Chiu<sup>,4</sup><sup> </sup>An-Wen Chan<sup>1,3,5</sup></p>
<p><h4>Objective</h4> We aimed to determine how often medical journals have policies requiring clinical trial registration, adherence to reporting guidelines, and participant-level data sharing. We also evaluated associations between journal characteristics and the existence of such policies for clinical trial manuscripts.</p>
<p><h4>Design </h4>A PubMed search using the clinical trial filter was conducted to identify journals that published at least 20 trials in 2023. We extracted publicly available data from journal websites, including policies on trial registration, adherence to reporting guidelines, trial protocol submission requirements, and availability of participant-level data. A practice was classified as required if the policy used words such as must, need, or should. Policies using language such as encouraged and preferred were classified as recommended practices. For each journal, we recorded the 2023 Clarivate impact factor, journal scope (general vs specialty), and publishing model (purely open access vs other). We calculated descriptive statistics to summarize the prevalence of transparency policies and used multivariable logistic regression to assess the association between journal characteristics and policy requirements.</p>
<h4>Results</h4>
<p> Among 380 included journals, 320 (84%) required trial registration, 11 (3%) recommended it, and 49 (13%) did not mention it. Adherence to a reporting guideline for clinical trials was required by 251 journals (66%), recommended by 74 (19%), and not mentioned by 55 (14%). Trial protocol submission was required by 118 (31%), recommended by 110 (29%), and not mentioned in 152 (40%). Public availability of participant-level datasets was required by 104 (27%), recommended by 218 (57%), and not mentioned by 58 (15%). A description of the data sharing plan was required by 212 journals (55.8%), recommended by 110 (28.9%), and not mentioned by 58 (15.3%). Purely open access journals had a 4-fold higher odds of requiring trial registration (adjusted odds ratio [AOR], 4.02; 95% CI, 1.41-15.59) and protocol submission (AOR, 4.13; 95% CI, 2.17-8.37) and 2.5 times higher odds of requiring public sharing of participant-level data (AOR, 2.46; 95% CI, 1.03-7.04) compared with other journals (<b>Table 25-1183</b>). Each 5-point increase in journal impact factor was associated with a more than 2.5-fold increase in the odds of requiring trial registration (AOR, 2.65; 95% CI, 1.45-5.98) and over 50% increase in the odds of requiring protocol submission (AOR, 1.56; 95% CI, 1.26-2.03). Impact factor was not significantly associated with requiring data sharing. No significant associations were found for journal scope or volume of trials published.</p>

![](https://assets.underline.io/markdown_image/1/image/1f3e3ae7eedf2cd146453708cac3c6fc.png)

<h4>Conclusions</h4>
<p> Journal policy requirements vary substantially in supporting best practices for clinical trial transparency. Journals with a purely open access publishing model and higher impact factor were more likely to adopt transparency policies. These findings highlight the need for improved editorial standards across the publishing landscape to promote transparency and reduce research waste.</p>

<sup>1</sup>Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada, anwen.chan@utoronto.ca; <sup>2</sup>Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada; <sup>3</sup>Women’s College Research Institute, Toronto, Ontario, Canada; <sup>4</sup>Faculty of Health Science, University of Western Ontario, London, Ontario, Canada; <sup>5</sup>Division of Dermatology, Department of Medicine, University of Toronto, Toronto, Ontario, Canada.
<h4>Conflict of Interest Disclosures</h4> An-Wen Chan is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.


Medical Journal Policies on Requirements for Clinical Trial Registration, Reporting Guidelines, and Data Sharing: A Systematic Review

<p>M. Janina Sarol,<sup>1</sup> Jodi Schneider,<sup>1,2</sup> Halil Kilicoglu<sup>1</sup></p>
<h4>Objective </h4>
<p>An estimated 25% of citations in medical journals are inaccurate.<sup>1</sup> If not corrected, these citation quotation errors can propagate misinformation and establish unfounded claims.<sup>2</sup> Correcting them before publication is vital, yet citation accuracy is overlooked in the peer review process. The prevalence of citation quotation errors in the biomedical literature also remains unclear, because existing studies rely on manual evaluation on a limited set of articles.</p>
<h4>Design </h4>
<p>We explored the use of large language models, specifically Gemini 1.5 Pro, GPT-4o (gpt-4o-2024-08-06), and LLaMA-3.1 (8B and 70B), to automatically identify citation quotation errors. We compared 2 approaches. In the first, we supplied the prompt with the citation in question, the paragraph containing the citation, and the full reference article. The second approach consists of a 3-step procedure and mimicked how a human reviewer might perform the task manually by first identifying the citation context, then locating relevant sentences in the reference article, and finally assessing citation accuracy. Specifically, the procedure consists of (1) extracting the citation sentence, (2) retrieving the 50 most relevant sentences using the MedCPT retriever model and reranking them with a fine-tuned MonoT5 model to select the top 5 sentences, and (3) verifying the citation accuracy with the large language model (<b>Box 25-1015</b>). Both approaches were evaluated from November 2024 to January 2025 using a citation quotation error dataset,<sup>3</sup> the first publicly available corpus of its kind (Cohen κ for interrater agreement: 0.96 for citation context and 0.31 for accuracy detection). The better-performing approach was applied to assess 2898 citations from 2000 articles to the 100 most cited PubMed Central Open Access (PMC-OA) articles (24 different journals).</p>

![](https://assets.underline.io/markdown_image/1/image/d650d5b842c02ea3ae5d4d9c28d01a15.png)

![](https://assets.underline.io/markdown_image/1/image/e80718c94a9aa238f09a1d74fc02135d.png)

<h4>Results </h4>
<p>Gemini 1.5 Pro yielded best performance, obtaining an accuracy of 69% with both approaches. It outperformed GPT-4o slightly, while LLaMA-3 failed to generate meaningful results with the first approach and performed poorly with the second. Because the first approach only identified 25% (55 of 220) of the erroneous citations and the 3-step procedure identified 51% (113 of 220), the second approach was applied to the subset of PMC-OA articles. In the PMC-OA subset, 34% (981 of 2898) of the citation instances were assessed as erroneous. A total of 37% (742 of 2000) of citing articles were found to contain citation quotation errors; 98% (98 of 100) of reference articles were cited incorrectly at least once.</p>
<h4>Conclusions </h4>
<p>Our preliminary results confirm the high prevalence of citation quotation errors in the medical literature. We will present a more comprehensive analysis of citation quotation errors in the PMC-OA subset at the conference. Automated verification of citations could help journals and peer reviewers identify questionable citation practices and reduce propagation of misinformation, improving the trustworthiness of scientific evidence.</p>
<h4>References</h4>
<p>1. Jergas H, Baethge C. Quotation accuracy in medical journal articles: a systematic review and meta-analysis. <i>PeerJ</i>. 2015;3:e1364. doi:10.7717/peerj.1364</p>
<p>2. Greenberg SA. How citation distortions create unfounded authority: analysis of a citation network. <i>BMJ</i>. 2009;339:b2680. doi:10.1136/bmj.b2680</p>
<p>3. Sarol MJ, Ming S, Radhakrishna S, Schneider J, Kilicoglu H. Assessing citation integrity in biomedical publications: corpus annotation and NLP models. <i>Bioinformatics</i>. 2024;40(7):btae420. doi:10.1093/bioinformatics/btae420</p>

<sup>1</sup>University of Illinois at Urbana-Champaign, IL, US, janinasarol@gmail.com; <sup>2</sup>Harvard Radcliffe Institute for Advanced Study, Cambridge, MA, US.
<h4>Conflict of Interest</h4> Disclosures Jodi Schneider declares nonfinancial associations with Crossref; Committee on Publication Ethics; International Association of Scientific, Technical and Medical Publishers; the European Association of Science Editors; the International Society of Managing and Technical Editors; the Institute of Electrical and Electronics Engineers; the National Information Standards Organization; and the Center for Scientific Integrity (parent organization of Retraction Watch) and has received data-in-kind from Retraction Watch and Scite and usability testing compensation from the Institute of Electrical and Electronics Engineers. The National Information Standards Organization is a subawardee on her Alfred P. Sloan Foundation grant (G-2022-19409). No other disclosures were reported.
<h4>Funding/Support</h4> This study was supported by the Office of Research Integrity (ORI) of the US Department of Health and Human Services (HHS) (ORIIR220073). The contents are those of the authors and do not necessarily represent the official views of, nor an endorsement by, the ORI, Office of the Assistant Secretary of Health, HHS, or the US government. Jodi Schneider was supported in part as the 2024-2025 Perrin Moorhead Grayson and Bruns Grayson Fellow, Harvard Radcliffe Institute for Advanced Study and by US National Science Foundation’s CAREER grant (2046454).
<h4>Role of the Funder/Sponsor</h4> The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.
<h4>Acknowledgment</h4> We thank Shruthan Radhakrishna for his contributions to fine-tuning the MonoT5 model.
<h4>Additional Information</h4> Halil Kilicoglu is a co–corresponding author (halil@illinois.edu).




Downloads

Next from The 10th Peer Review Congress

Using GPT to Identify Changes in Clinical Trial Outcomes Registered on ClinicalTrials.gov

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from The 10th Peer Review Congress

Using GPT to Identify Changes in Clinical Trial Outcomes Registered on ClinicalTrials.gov

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads