"**Introduction.** Mentoring has been demonstrated to increase science identity, retention and academic performance. Students that develop science identity and sense of belonging are more likely to persist in STEM areas. Mentoring can also provide psychosocial support by normalizing struggles and reinforcing identity in the field. Peer-mentoring is a form of mentorship that involves a more experienced student guiding a less experienced student through social, emotional and/or academic support. The benefits of peer-mentoring have been observed to help Latina/o students find role models, access emotional and academic support, and career help, thereby increasing integration into their university. Peer-mentoring has also been credited as one of the factors which lead to the success of Black undergraduate women students graduating with STEM degrees at Historically Black Colleges and Universities (HBCUs). For Latinas and Black in STEM, peer mentoring has been shown to support  student integration and persistence, and increase academic performance and degree achievement. 

**Goal of Research:** We conducted a 1-year pilot intervention that had the goal of providing female Latina and Black STEM undergraduates, usually underrepresented (URM) in STEM, with a social, professional and academic support structure by inviting them to join a peer-mentoring program. Here we present the findings  

**Methods.** First-year science students attending a 4-year public college were invited to join a peer-mentoring program. Peer-mentors were recruited from upper level cohorts, with junior and senior standing in cell and molecular biology, toxicology, and forensic science majors. A total of 20 first-year female students joined the program in September and 8 upper-level female students joined as mentors. All participants were Latinas and/or Black. Participants formed small groups of on average three first-year students that met with a mentor monthly during the semester. In these meetings that lasted between 30-60 minutes, the mentors and mentees shared ideas, resources, discussed strategies and shared concerns. The whole cohort met three times during the semester, and participated in seminars in which invited speakers shared their professional experience and the work they were currently doing. The invited speakers were also female scientists from underrepresented backgrounds in STEM. We assessed the student’s experiences using a 6-question survey at the end of each mentor-mentee meeting, that evaluated the implementation of the intervention and identified strengths and areas in need of improvement. The survey was voluntary and anonymous. The survey contained Likert-scale items and one opened-ended question. We use Leximancer software to conduct a thematic analysis and a relational (or semantic) analysis of the interview data. The software provides word frequency counts and co-occurrence counts of concepts present in participants' responses.  

**Results.** A total of 28 students participated in this intervention, all of which are URM in STEM (either African American or Hispanic). In the fall semester only 14 students joined the program, but 6 additional students requested to participate in the spring semester for a total of 20 participants. We originally had 8 mentors, but only 7 mentors completed the program. Groups were formed in the fall, and mentor-mentee sets continue to work together through the school year. New participants joined groups that already existed. In the quantitative part of the survey students were asked about their experience at each meeting. 58% and 67% responded that they ‘strongly agree’ to the statements  “this was a productive session” and “I look forward to my next mentoring session”. The qualitative assessment of the mentees’ responses showed that  ‘careers’, ‘internships’ and ‘time’ were among the most frequent terms used in the responses. Compound concepts showed ‘careers & goals’ and ‘internship & study’ to have the highest prominence in the meeting category, followed by “time & study’ and ‘helpful & goals’. These highlight the areas of most concern to the students attending. The concept map showed a large component in which terms like ‘improved’, ‘mentor’, ‘discussed’, ‘needs’, ‘helpful’ and ‘opportunities’ were the most frequent. The analysis identified top strengths of the intervention including 1)The mentorship program is comprehensive, covering a wide range of topics from academics to personal development, 2) Mentors are approachable and friendly, fostering a comfortable environment for mentees, and 3) The program provides a platform for mentees to discuss their goals and aspirations. In addition, some concerns were also highlighted by the software including, 1) The program may not be able to accommodate all the topics and concerns that mentees want to discuss due to time constraints, 2) The program may not be able to provide sufficient guidance and support for mentees who join late or miss meetings, and 3) The program may not be able to maintain its quality and effectiveness if it becomes too flexible and unstructured. This initial analysis provides suggestions on how to implement peer-mentoring programs and which aspects to keep in mind during their organization. 

**Conclusions and Discussions.** Our intervention adds to current evidence supporting the need for programs that aid first-year students in facing challenges associated with the college transition. It particularly highlights the importance of these support structures for URM females in STEM majors.  

**Acknowledgments.** This research is funded by the Andrew Mellon Foundation in support of  the Black, Race, Ethnic Studies Initiative (BRESI) at the City University of New York (CUNY)"

Peer-mentoring intervention supports undergraduate Latinas and African-American students in STEM

This session will provide information regarding publishing in the Understanding Interventions Journal. Attendees will learn of the translational approach to mutually inform practice and scholarship; requirements for the types of publications (letters, communications and full papers) published in the journal; its review processes as well as the supports provided to potential authors including workshops and direct mentoring to facilitate manuscript development. 


UI Journal: Disseminating Interventions Research

**Objective** Due to the pandemic, preprint servers contained
up to 25% more trials, whereas some medical journals
accelerated their editorial processes to ensure the rapid
dissemination of findings.1
Concerns regarding quality and
transparency rose.2
This meta-study assessed the
transparency, completeness, and consistency of COVID-19
reports and whether there was an improvement after journal
peer review.

**Design** The Cochrane COVID-19 Study Register and L·OVE
COVID-19 platform were searched to identify all reports
(preprints or peer-reviewed publications) of randomized
clinical trials (RCTs) assessing pharmacologic interventions
for the treatment of COVID-19, up to May 31, 2021. A
standardized, online data-extraction form was developed.
Data extraction covered general trial characteristics,
transparency indicators (eg, trial registration and data
sharing statement), completeness of reporting (eg, 10 of the
most important Consolidated Standards of Reporting Trials
[CONSORT] 2010 items3
), and changed outcomes between
report and registry (ie, switched outcomes). Two reviewers
were trained and assessed 20 trials separately, with an
agreement of 96.6% and a κ coefficient of 0.87. A third person
resolved disagreements, when needed. A single reviewer
assessed all remaining trials. For all trials published first as a
preprint, a systematic search was performed for a subsequent
publication in peer-reviewed journals, up to October 7, 2021.
In a second step, the peer-reviewed journal publication was
assessed and compared with the matched preprint.

**Results** A total of 251 trial reports were identified: 121
peer-reviewed journal publications (48%) and 130 preprints
(52%). Approximately half of the trials were prospectively
registered (140 [56%]); 38% (n = 95) made their full protocols
or statistical analysis plan available, and 29% (n = 72)
provided access to their statistical analysis plan report. A data
sharing statement was available in 68% of the reports (n =
170); 91% stated their willingness to share. Only 32% of the
trials (n = 81) completely defined the prespecified primary
outcome measures; 57% (n = 143) reported the process of
allocation concealment. Overall, 51% (n = 127) adequately
reported results for the primary outcomes, whereas only 14%
of the trials (n = 36) adequately described harms. Primary
outcome(s) reported in trial registries and published reports
were inconsistent in 49% of the trials (n = 104); only 15% (n =
16) disclosed outcome switching in the report. There were no
major differences between preprints and peer-reviewed
publications. Of the 130 RCTs published as a preprint, 78
were then published in a peer-reviewed journal (median delay to journal publication, 94 days; IQR, 55-168 days). There was
no major improvement after the peer review process.

**Conclusions** Lack of transparency, completeness, and
consistency of reporting are important barriers to trust,
interpretation, and synthesis in COVID-19 clinical trials.
Peer-reviewed publications did not report the items assessed
better than preprints. The comparison of paired reports
published as a preprint and as a peer-reviewed publication
did not indicate major improvements.

**References**

1. Else H. How a torrent of COVID science changed research
publishing—in 7 charts. Nature. 2020;588(7839):553.
doi:10.1038/d41586-020-03564-y
2. Besançon L, Peiffer-Smadja N, Segalas C, et al. Open
science saves lives: lessons from the COVID-19 pandemic.
BMC Med Res Methodol. 2021;221(1):117. doi:10.1186/
s12874-021-01304-y
3. Chauvin A, Ravaud P, Moher D, et al. Accuracy in detecting
inadequate research reporting by early career peer reviewers
using an online CONSORT-based peer-review tool (COBPeer)
versus the usual peer-review process: a cross-sectional
diagnostic study. BMC Med. 2019;17(1):205. doi:10.1186/
s12916-019-1436-0

**Conflict of Interest Disclosures** Isabelle Boutron is a member
of the Peer Review Congress Advisory Board but was not involved
in the editorial review or decision for this abstract. No other
disclosures were reported.

**Funding/Support** This study was part of the COVID-NMA
initiative, which received funding from Université de Paris,
Assistance Publique Hôpitaux de Paris, INSERM, Cochrane France
(Ministry of Health), the French Ministry of Higher Education and
Research, Agence Nationale de la Recherche, and the World Health
Organization.

![](https://assets.underline.io/uploads/markdown_image/1/image/5f2462693a970f13400e34d4d146a2a0.png)

Assessment and Comparison of Preprints and Peer-Reviewed Publications of Reporting Characteristics of Randomized Clinical Trials of Pharmacologic Treatment for COVID-19

**Objective** The peer review system used by academic journals
to assess the quality of submitted papers is constantly in need
of new technologies that can help reviewers and increase the
objectivity of the process. The aim of the present study was to
develop a comprehensive list of words related to research
methods and statistics, which could then be used to
automatically study the content and the quantity and types of
methodological and statistical issues identified in peer review
reports.

**Design** The terms for the list were identified from different
glossaries.1,2 The terms were organized into subcategories,
some of which were more related to methodological aspects
(eg, study design, sampling, or procedures) or statistical
aspects (eg, parametric descriptive parameters or data
presentation) of research. The list was refined and enriched
with terms using a computational language model. The final
list and the preregistration can be found at the Open Science
Framework website (https://osf.io/d34b9/). The occurrence
of terms in the PEERE database3
of peer review reports, for
which data on reviewer gender, continental region, review
recommendation, journal impact factor, and subject area (N
= 496,928) were available, was counted. Regression models
were created to determine the frequency of the terms used
based on peer review characteristics.

**Results** The mean length of a peer review was 196 words
(95% CI, 108-330 words). A total of 26.8% of review reports
in the sample contained at least 1 methodological term, while
70.7% contained at least 1 statistical term. It was more likely
that the review contained methodological terms if the
reviewer was female and resided in North America or Oceania
compared with other continents, if the recommendation was
to reject the article or send it for major revisions, if the
journal was from the social sciences and humanities, if the
journal was in the top impact factor quartile, and if the article
was a longer peer review. More words related to statistics
were observed when the reviewer was female, for reviews
recommending any revisions, for journals from health and
medical sciences, in journals in the top impact factor quartile,
and in longer reviews. Mixed regression models indicated
that the scientific area was more strongly associated with
words related to statistics than were reviewer
recommendations.

**Conclusions** The application of this newly developed list
showed that methodological and statistical terms are not a
common topic in peer review and that they are not always
present in review reports from different fields. The list
successfully covered research terms because the most-
identified terms were related to categories relevant for most
types of research. Future analysis should try to understand
the context of the occurrence of words from the list to confirm
the validity of the list of terms.

**References**

1. Stark PB. Glossary of Statistical Terms. Accessed March 23,
2021. https://www.stat.berkeley.edu/~stark/SticiGui/Text/
gloss.htm
2. Framework for Open and Reproducible Research Training.
The Glossary of Terms. Accessed July 12, 2021. https://docs.
google.com/document/d/1oNsHSRmgAsDGUKvq5joqBpv
HeRmDJKldH2kSyJovHhk/edit#heading=h.2l8ssl89e1ey
3. Squazzoni F, Ahrweiler P, Barros T, et al. Unlock ways to
share data on peer review. Nature. 2020;578(7796):512-514.
doi:10.1038/d41586-020-00500-y

**Conflict of Interest Disclosures** Ana Marušić is a member of
the Peer Review Congress Advisory Board by was not involved in the
editorial review or decision for this abstract. No other disclosures
reported.

**Funding/Support** This study was funded by the Croatian Science
Foundation “Professionalism in Health–Decision making in
practice and research” under grant agreement IP-2019-04-4882.
Richard A. Klein received funding from a Replication Studies grant
(401.18.053) from the Netherlands Organization for Scientific
Research, and a Consolidator Grant (IMPROVE) from the European
Research Council (grant 726361). Francisco Grimaldo and Daniel
Garcia-Costa are partially supported by the Spanish Ministry of
Science, Innovation and Universities, the Spanish State Research
Agency, and the European Regional Development Fund under
project RTI2018-095820-B-I00.

**Role of the Funder/Sponsor** The funders had no role in the
design and conduct of the study; collection, management, analysis,
and interpretation of the data; preparation, review, or approval of
the abstract; and decision to submit the abstract for presentation.

![](https://assets.underline.io/uploads/markdown_image/1/image/eba25e8a44f7081b702973e5ddf9a854.png)

Development of a List to Detect Statistical and Methodological Terms in Peer Reviews: A Cross-Sectional Study

**Objective** Existing literature reviews about retraction do not
analyze postretraction citation. This research synthesized a
subgroup of empirical studies about retraction and reports
what is known about postretraction citation.

**Design** This was a subanalysis of a previously reported
scoping review.1
A total of 386 items about retraction were
found by double screening a systematic search of PubMed,
Scopus, and Web of Science up to February 10, 2021; Scopus-
cited reference searching in January 2021; and hand
searching up to July 2021.1
Items comprised published and
unpublished (work in progress) research reports. Subsequent
to the previous report, a custom taxonomy2
of methods,
research goals, and research was iteratively created and made
searchable via an online bibliography.3
For the subanalysis,
the scope was narrowed to postretraction citation after
excluding items on citation-related implications for review
literature and authors’ careers, publicity, and altmetrics. A
codebook to guide data extraction was developed and piloted.
Data extraction and analysis is ongoing.

**Results** This subanalysis included 92 items up to July 2021
on postretraction citation of retracted papers. Items were
classified into 7 topics: database-focused analyses (n = 33)
(eg, PubMed and Web of Science); field-based case studies (n
= 20) (eg, genetics, radiology, and thoracic surgery); paper-
focused case studies of 1 to 125 selected papers (n = 15);
author-focused case studies of 1 or several authors with many
retracted publications (n = 15) (eg, Bruening, Darsee, and
Reuben); studies of retracted publications cited in review
literature (n = 8); geographic case studies (n = 4) (focusing on
Brazil, the European Union, Italy, and South Korea); and
studies selecting retracted publications by method (n = 2) (eg,
human subjects and randomized clinical trials). Five items
were classified as belonging to 2 topics each. Empirical
research about postretraction citation has been published in a
diffuse set of journals, primarily in journals of ethics,
information science, meta-science and scientometrics, and domain sciences, especially medical specialties. The earliest 2
studies identified were both published in 1990; 1 study was an
author-focused case study of citations to Stephen E.
Breuning’s publications and the other studied 82 Index
Medicus articles retracted, with retraction notices in Index
Medicus as of 1990 and citations from SCISEARCH. From
1990 to 2017, a total of 1 to 3 items were found per year
except in 2016 (5 items), and from January 2018 to July 2021,
11 to 18 items per year were found on postretraction citation.
Almost all of these items focused on health sciences (eg,
medicine, dentistry, nursing, psychology, and pharmacy),
with 1 item focused on arts and humanities and 2 items
focused on engineering.

**Conclusions** Postretraction citation has been studied
consistently since 1990, with increasing attention since 2018.
This analysis found an increasing number of items on
postretraction citation from January 2018 to July 2021.
However, relevant work published after July 2021 was not
included, and items published earlier may have been missed
from the scoping review. Inclusion of work in progress may
have increased publication counts for the most recent years.

**References**

1. Schneider J, Woods ND, Proescholdt R, Fu Y; RISRS Team.
Recommendations from the Reducing the Inadvertent Spread
of Retracted Science: shaping a research and implementation
agenda project. MetaArXiv Preprints. Preprint posted online
July 28, 2021. doi:10.31222/osf.io/ms579 2. Proescholdt R, Schneider J. Comparison set guide for
empirical retraction lit. University of Illinois IDEALS
Repository. Deposited September 10, 2021. https://hdl.handle.net/2142/110389
3. Proescholdt R, Schneider J; RISRS Team. Empirical
retraction lit bibliography. Version 2.20. Zenodo. September
9, 2021. doi:10.5281/zenodo.5498500

**Conflict of Interest Disclosures** Jodi Schneider has been an
invited speaker for scholarly publishing organizations Committee on
Publication Ethics, CrossRef, the European Association of Science
Editors, the International Society of Managing and Technical
Editors, the Institute of Electrical and Electronics Engineers,
the National Information Standards Organization, and STM;
has received data-in-kind from Retraction Watch and scite; and
has received usability testing compensation from the Institute of
Electrical and Electronics Engineers. No other disclosures were
reported.

**Funding/Support** This project was supported by the Alfred P.
Sloan Foundation (G-2020-12623).

**Role of the Funder/Sponsor** The study sponsor had no role
in the design and conduct of the study; collection, management,
analysis, and interpretation of the data; preparation, review, or
approval of the abstract; and decision to submit the abstract for
presentation.

![](https://assets.underline.io/uploads/markdown_image/1/image/750e15b2579ba642752748359dd97985.png)

Characterization of Publications on Post-Retraction Citation of Retracted Articles

**Objective** Over the past decade, there has been an
exponential increase in systematic reviews and meta-analyses
(SR/MAs) published in the medical literature, and some
concerns have arisen that many of these are duplicate studies
with redundant results or new studies of unclear quality.1-3
The aim of this study was to determine whether there was any
change in the methodological or reporting quality of SRs in
dermatology over the past decade.

**Design** This was a cross-sectional study of all SR/MAs
published in 2010 and 2019 in the 10 highest-ranked
dermatology journals by SCImago Journal. Methodological
quality was assessed through duplicate and independent
adjudication by 2 reviewers (A.M., L.K.L., J.K., A.L.) using the
Risk of Bias in Systematic reviews (ROBIS) tool and, for SR/
MA of interventions, the A Measurement Tool to Assess
Systematic Reviews (AMSTAR) 2 tool. Reporting quality was
assessed by 1 investigator (A.M.) using the Preferred
Reporting Items for Systematic Reviews and Meta-Analyses
(PRISMA) and PRISMA for Abstracts (PRISMA-A) tools, and study, author, and journal characteristics were abstracted.
Statistical analysis was performed with Jamovi, version
1.6.23. Methodological quality was analyzed with the χ2 test
for independent proportions, or the Fisher exact test if any of
the expected frequencies were less than 5, and difference in
proportions with 95% CIs. Reporting quality was analyzed
with the independent samples t test and difference in means
with 95% CIs.

**Results** In total, 21 SR/MAs from 2010 and 127 from 2019
were included There was little to no difference between 2010
and 2019 in the proportion of SR/MAs at high or unclear
overall risk of bias with ROBIS or with critically low
methodological quality using AMSTAR 2. The only
subdomain of ROBIS with a difference in proportion of SR/
MAs at high or unclear risk of bias between 2010 and 2019
was eligibility bias, with 27.3% more (95% CI, 5.42%-49.2%)
in 2010 (66.7%) than 2019 (39.4%) (P = .02). There was a
strong difference in proportion of PRISMA (t146 = 3.15;
P = .002) and PRISMA-A (t146 = 2.46; P = .02) checklist items
adequately reported between 2010 and 2019. The difference
in mean proportion of PRISMA checklist items adequately
reported was 3.8 items more (95% CI, 1.4-6.2 items more) in
2019 (mean [SD], 15.7 [5.1] items) than in 2010 (mean [SD],
11.9 [5.5]), and of PRISMA-A checklist items adequately
reported was 0.9 items more (95% CI, 0.2-1.6 items more) in
2019 (mean [SD], 5.9 [1.5] items) than in 2010 (mean [SD], 5
[1.9] items).

**Conclusions** While there was an improvement in the overall
reporting quality of SR/MAs between 2010 and 2019, there
was no improvement in the overall methodological quality as
assessed by the ROBIS and AMSTAR 2 tools. This suggests a
persistent potential for forming unreliable conclusions in SR/
MAs through a decade of dermatological research. These
results therefore highlight the urgency with which efforts to
improve SR/MA methodology should be undertaken.

**References**

1. Siontis KC, Ioannidis JPA. Replication, duplication, and
waste in a quarter million systematic reviews and meta-
analyses. Circ Cardiovasc Qual Outcomes.
2018;11(12):e005212.
2. Katsura M, Kuriyama A, Tada M, et al. High variability in
results and methodological quality among overlapping
systematic reviews on the same topics in surgery: a meta-
epidemiological study. Br J Surg. 2021;108(12):1521-1529.
3. Gómez-García F, Ruano J, Gay-Mimbrera J, et al. Most
systematic reviews of high methodological quality on
psoriasis interventions are classified as high risk of bias using
ROBIS tool. J Clin Epidemiol. 2017;92:79-88.

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/8b3c3da9a1f0f9cbc94a7a79a4532217.png)

Methodological and Reporting Quality of Systematic Reviews in Dermatology

**Objective** To assess the publication status and factors
associated with subsequent publication of all pediatric
intervention trials registered in the Clinical Trial Registry of
India (CTRI) over a period of 5 years.

**Design** This cross-sectional study was conducted from
December 2021 to February 2022 and included the first 100
pediatric intervention trials registered in the CTRI from 2008
to 2012. Registry records were identified from the CTRI
website using the keywords pediatric, paediatric, children,
adolescent, infant, newborn, neonate, kids, and school. Trial
characteristics (eg, blinding, type of intervention and
comparator, setting, funding, source of funding, single or
multicentric, and postgraduate thesis) were abstracted from
each registered trial. A list of all the randomized clinical trials
registered on the website was made and their subsequent
publication was systematically searched on PubMed and
Google Scholar using their registered CTRI number up to
December 2021. For trials that were not found, repeat
searches were performed, searching by the first author’s
name, second author’s name, and title of the registered trial.
The proportion of trials subsequently published and the time
to publication from the date of registration were analyzed.
Factors associated with publication were compared between
trials that were published and those not published using χ2
test and univariate analysis by calculating the odds ratio and
95% CI. Multivariable logistic regression was conducted for
factors with P < .50.

**Results** The first 100 pediatric intervention trials registered
from 2008 to 2012 were retrieved from the CTRI. The overall
proportion of trials published was 71%; 87% had
randomization with a comparator arm, 78% examined
intervention for treatment, 22% examined intervention for
process of care change, 62% had retrospective registration,
51% had funding, and 39% had government funding. The
proportion of trials that were performed at a single center was
75%, 92% of trials were hospital based, 84% were conducted
at teaching hospitals, and 43% were postgraduate thesis
based. The median (range) time to publication was 4 (1-9)
years. A non–statistically significant higher proportion of
postgraduate thesis–based trials vs non–thesis-based trials
(79% vs 65%; OR, 2.04; 95% CI, 0.81-5.09) and single-center
trials vs multicenter trials (76% vs 56%; OR, 2.48; 95% CI,
0.96-6.44) were published. On multivariable logistic
regression, none of the factors were associated with higher
odds of publication (**Table 85**).

![](https://assets.underline.io/uploads/markdown_image/1/image/b8780e831d5fc2d23c3c10d09f844e28.png)

**Conclusions** Seventy-one percent of pediatric intervention
trials registered in the CTRI were subsequently published
regardless of their blinding, funding status, type of
intervention, comparator, or setting. A non–statistically
significant higher proportion of postgraduate thesis–based
trials and single-center trials were published. Further studies
with a larger sample size are needed to demonstrate any
statistical significance. Journal editors, funding agencies, and
ethics committees could implement mandatory registration of
intervention trials, as registration seems to be associated with
better quality and a good chance of subsequent publication.
Following guidelines for preparing data sets for submission to
data repositories could help achieve more trial registrations.

**References**

1. Rao MVV, Maulik M, Juneja A, et al. Clinical Trials
Registry—India: a decadal perspective. Indian J Pharmacol.
2020;52(4):272-282. doi:10.4103/ijp.IJP_24_20
2. DePasse JM, Park S, Eltorai AEM, Daniels AH. Factors
predicting publication of spinal cord injury trials registered
on www.ClinicalTrials.gov. J Back Musculoskelet Rehabil.
2018;31(1):45-48. doi:10.3233/BMR-169628
3. National Heart, Lung, and Blood Institute. Guidelines for
preparing clinical study data sets for submission to the
NHLBI data repository. Accessed February 20, 2022. https://
www.nhlbi.nih.gov/grants-and-training/policies-and-
guidelines/guidelines-for-preparing-clinical-study-data-sets-
for-submission-to-the-nhlbi-data-repository

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/067a7989d5d81ff19bd6b13bc8fbf9c2.png)

Factors Affecting Publication of Pediatric Intervention Trials

**Objective** Problems with reporting a scientific study can
affect research in different manners. For example, it is known
that study methods are frequently not described in adequate detail and that results are presented ambiguously,
incompletely, or selectively. A previous study1
confirmed that
most of the authors (approximately 67%) publishing
systematic reviews (SRs) in high-impact rehabilitation
journals did not mention the use of a reporting guideline.
Therefore, the objectives of this study were (1) to evaluate the
completeness of reporting of SRs published in rehabilitation
journals by evaluating their adherence to the Preferred
Reporting Items for Systematic Reviews and Meta-Analyses
(PRISMA) 2009 checklist, (2) to investigate the relationship
between completeness of reporting and risk of bias (ROB),
and (3) to study the association between completeness of
reporting and the characteristics of SRs and journals.

**Design** A random sample of 200 SRs published between
2011 and 2020 in the 68 rehabilitation journals was indexed
under the rehabilitation category in the InCites Journal
Citation Report. Two independent reviewers evaluated the
completeness of reporting as the adherence to the PRISMA
2009 checklist and assessed ROB using the ROBIS tool.
Overall adherence and adherence to each PRISMA 2009 item
and section were calculated as a percentage of the total
number of items described and reported of the total number
of applicable items. Regression analyses investigating the
association between completeness of reporting, ROB, and
other characteristics (impact factor, publication options, year
of publication, and study protocol registration) were
performed.

**Results** The mean overall PRISMA 2009 adherence was
61.4% (**Figure 34**). Studies with high ROB in ROBIS domain
1 had an overall adherence to the PRISMA 2009 checklist,
which was 5.70% (95% CI, −10.07% to −1.34%) lower than
those with low ROB. In domain 2, studies with high ROB had
5.41% (95% CI, −9.74% to −1.07%) lower adherence than
those with low ROB. Studies with high overall ROB had
7.06% (95% CI, −12.10% to −2.01%) lower adherence than
those with low overall ROB. Studies published in fourth-
quartile journals displayed an overall adherence of 7.24%
(95% CI, −13.19% to −1.29%) lower than those published in
the first quartile; there was an 11.95% (95% CI, 5.94%-
17.96%) increase in overall adherence if the SR protocol was
registered.

![](https://assets.underline.io/uploads/markdown_image/1/image/eadc9d58246daba8b9456d19c7356564.png)

**Conclusions** The completeness of reporting in SRs
published in rehabilitation journals is still suboptimal. High
ROB is associated with poorer completeness of reporting, indicating that if a study has better reporting, this could
positively affect the transparency of the ROB evaluation. The
registration of study protocols and journal ranking are also
associated with a more complete reporting. Authors of SRs
should improve adherence to the reporting standards, and
journal editors could implement strategies to optimize the
completeness of reporting.

**Reference**

1. Innocenti T, Salvioli S, Giagio S, Feller D, Cartabellotta N,
Chiarotto A. Declaration of use and appropriate use of
reporting guidelines in high-impact rehabilitation journals is
limited: a meta-research study. J Clin Epidemiol.
2021;131:43-50. doi:10.1016/j.jclinepi.2020.11.010

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/c486ab2b5159af9894c4233312feafc4.png)



Completeness of Reporting and Its Association With Risk of Bias in Systematic Reviews Published in Rehabilitation Journals: A Meta-research Study

**Objective** It is generally believed that evidence from a
weaker body of evidence (eg, poorly designed and executed
and potentially biased studies, sparse studies and/or
heterogenous results) will generate inaccurate estimates
about treatment effects more often than evidence from a
stronger body of evidence. As a result, estimates of effects of
health interventions initially based on high certainty (quality)
of evidence (CoE) are expected to change less frequently than
the effects estimated by lower CoE, and the estimates of
magnitude of effect size are expected to differ between high
and low CoE. Empirical assessment of these foundational
principles of evidence-based medicine has been lacking.

**Design** The Cochrane Database of Systematic Reviews was
reviewed from January 2016 through May 2021 for pairs of
original and updated reviews for change in CoE assessments
based on the Grading of Recommendations Assessment,
Development and Evaluation (GRADE) method. The
difference in effect sizes between the original and updated
reviews were assessed as a function of change in CoE, which
was reported as a ratio of odds ratios (RORs). The RORs
generated in the studies that changed CoE from very low/low
(VL/L) to moderate/high (M/H) vs MH/H to VL/L were
compared. Heterogeneity and inconsistency were assessed
using the τ and I2
statistic. The change in precision of effect
size estimates was assessed by calculating the ratio of
standard errors (seR), and the absolute deviation in estimates
of treatment effects was assessed with adjusted RORs.

**Results** Overall, 419 pairs of reviews were included, of which
414 (207 × 2) informed the CoE appraisal and 384 (192 × 2)
the assessment of effect size. Certainty of evidence originally
appraised as VL/L had 2.1 (95% CI, 1.19-4.12; P = .01) times
higher odds to be changed in the future studies than those
with M/H CoE. However, the pooled effect size was not
different when the CoE changed from VL/L to M/H (ROR,
1.02; 95% CI, 0.74-1.39) compared with M/H CoE changing to VL/L (ROR,1.02; 95% CI, 0.44-2.37). Similarly, the overlap
in aROR between the VL/L CoE to M/H vs the M/H to VL/L
subgroups was observed (median [IQR], 1.12; 95% CI,
1.07-1.57 vs 1.21; 95% CI, 1.12-2.43). There was a large
inconsistency across ROR estimates (I2
= 99%). There was
larger imprecision in treatment effects when the CoE changed
from VL/L to M/H (seR = 1.46) than when it changed from
M/H to VL/L (seR = 0.72).

**Conclusions** This study found that low-quality evidence
changed more often than high CoE. However, the effect size
was not systematically different between studies with low vs
high CoE, indicating the need for improving contemporary
critical appraisal methods.

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/050ec883e962e972a46b1f81946c6345.png)


Comparison of Changes in High-Quality vs Low-Quality Evidence in Original and Updated Systematic Reviews

**Objective** To systematically assess the characteristics and
current practice of developing methods guidance for health
researchers and explore opportunities for improvement.

**Design** A systematic survey of methods-guidance articles
published in general and methodology-focused, high-impact
medical journals indexed in MEDLINE in 2020 was
performed. Articles that explicitly stated the objective to
provide methods guidance for health research were eligible.
Characteristics related to findability, methods for guidance
development, and transparency were extracted.

**Results** A total of 105 guidance articles published in 12
different journals were included. Few articles had a
structured abstract (44 [42%]) or were indexed with Medical
Subject Headings (40 [38%]) or author keywords (18 [17%])
related to guidance. Of the 105 guidance articles, less than
half reported any methods for development (44 [42%]), most
frequently stakeholder involvement (30 [28%]), systematic
review of the methodological literature (21 [20%]), or
consensus process (21 [20%]). Use of explicit methods for
development differed between reporting guidelines (13 of 13
[100%] reported a development process) and other types of
methods guidance (ie, guidance for planning, conduct,
analysis, interpretation, or quality assessment: 31 of 92
articles [34%] reported a development process).
Transparency was limited, with few guidance articles
describing the authors’ expertise (23 [22%]). Conflicts of
interest, if reported (36 [34%]), were frequently unclear.

**Conclusions** Most methods-guidance articles published in
2020 were difficult to find in MEDLINE, were developed with
unclear methods, and lacked transparency regarding the
authors’ expertise and conflicts of interest. For health
researchers, those limitations implied important barriers to
the uptake of methods guidance. To improve findability, we
developed the new open-access Library of Guidance for
Health Scientists (LIGHTS; https://lights.science). More
research is required to inform methods for guidance
development and transparency considerations.

**Conflict of Interest Disclosures** None reported.

**Funding/Support** This work was supported by grant 190566 from
the Swiss National Science Foundation and grant 3MS1043 from the
University of Basel.

**Role of the Funder/Sponsor** The funders had no role in the
design and conduct of the study; collection, management, analysis,
and interpretation of the data; preparation, review, or approval of
the abstract; and decision to submit the abstract for presentation.

**Additional Information** Julian Hirt is a co–corresponding
author.

![](https://assets.underline.io/uploads/markdown_image/1/image/dbaccba0abde9f482b8175b3df1998be.png)

Characteristics and Opportunities for Improvement of Methods Guidance Published in General and Methodology-Focused Medical Journals

**Objective** To provide an up-to-date perspective of
biomedical researchers’ knowledge and perceptions of and
engagement with peer review training. Previous studies,
many done in consultation with editors and publishers, have
reported researchers’ attitudes on peer review.1,2 This survey
study is descriptive and does not have any a priori
hypotheses.

**Design** A cross-sectional online survey of biomedical
researchers around the world was conducted. The CHERRIES
(Checklist for Reporting Results of Internet E-Surveys)
reporting guideline was used to inform the reporting of
findings. A random sample of 2000 corresponding authors
was collected from a Scopus source list of recently published
articles in peer-reviewed biomedical journals. Authors from
journals that exclusively published non-English articles were
excluded. The closed survey was purpose built for this study,
was administered using SurveyMonkey, and was available
only to participants identified via the random sampling framework. Participants were invited through email;
participation was voluntary, and all data were collected
anonymously. Collected data included participants’
demographic characteristics as well as their experience with
and opinions about peer review, with additional open-ended
questions allowing participants to elaborate their responses.
Data were analyzed from all surveys in which participants
responded to 80% or more of the questions.

**Results** Of the 2000 invited researchers, 186 (9.3%)
responded. The average survey completion rate among these
participants was 91%. Most participants (142 [76.3%])
reported having 6 or more years of experience in scholarly
publishing. One hundred two of 180 participants (56.7%)
reported being active as a manuscript peer reviewer for more
than 6 years, and 171 of 185 participants (92.4%) reported
having peer reviewed at least 1 article in the last 12 months.
Despite the robust experience and activity in manuscript peer
review reported by the participants, only 28 of 185
participants (15.1%) completed formal training in peer review.
Twelve of 64 participants (18.8%) received training through
in-person lecture, and 11 of 64 participants (17.2%) received
training through online lecture. Thirteen of 36 participants
(52.8%) received training in peer review provided by a
university or college.

**Conclusions** This study will provide a current international
perspective on biomedical researchers’ knowledge,
perceptions, and engagement regarding peer review training.
The results of the survey may help identify gaps in peer
review training experience and knowledge. Subsequently, the
findings may guide the creation of future training options,
inform the development of preferred training methods, and
increase comprehensiveness of peer review training for
biomedical researchers.

**References**

1. Mulligan A, Hall L, Raphael E. Peer review in a changing
world: an international study measuring the attitudes of
researchers. J Am Soc Inf Sci Tec. 2013;64:132-161.
doi:10.1002/asi.22798
2. Publons. 2018 Global state of peer reviewy. 2018. Accessed
July 2022. https://publons.com/static/Publons-Global-State-
Of-Peer-Review-2018.pdf

**Conflict of Interest Disclosures** David Moher is an associate
director of the International Congress on Peer Review and Scientific
Publication but was not involved in the review or decision for this
abstract. No other disclosures reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/49348bfde8ee658b65c0fa925d323933.png)

An International Survey of Biomedical Researchers Knowledge, Perceptions, and Training on Peer Review

**Objective** The inclusion of duplicate publications in
systematic reviews (SRs) has led to repeated patient
information (RPI).1,2 Repeated patient information in SRs is
the inclusion of a patient’s information multiple times, with
the assumption that they are different participants. This could
result in the overestimation or underestimation of results,
which can lead to substantial clinical implications through
misleading estimates.1
A proportion of studies with shared
timing and location was identified in SRs by evaluating an early-stage sample of COVID-19 SRs. According to the
International Committee of Medical Journal Editors, a
duplicate publication overlaps substantially with a published
article without clear reference to the initial publication.3
The proportion of studies with shared characteristics that could
suggest RPI was assessed in this study.

**Design** This study was an umbrella review of SRs with
clinical data for patients with COVID-19. An experienced
librarian performed a comprehensive search strategy for
peer-reviewed articles in the English language published
between December 1, 2019, and April 6, 2020, in databases
that included Ovid MEDLINE In-Process & Other Non-
Indexed Citations, Ovid MEDLINE, Ovid Embase, Ovid
Cochrane Central Register of Controlled Trials, Ovid
Cochrane Database of Systematic Reviews, and Scopus.
Studies included in SRs were grouped into clusters depending
on characteristics including time frame and location. The
frequency of studies that met the definition of at risk of
including RPI was measured according to shared timing and
location in the same SR.

**Results** Fifteen SRs were included, with a total population of
172,558 participants. A median (IQR) of 9 (6-20) studies were
included in each SR, and 6 (2-10) studies included by each SR
were considered at risk. Of these, 14 (93.3%) had risk of RPI.
Subsequently, 103 clusters were generated. A median (IQR)
of 3 (1-6) at-risk clusters were included in each SR. Eleven
SRs (73.3%) included articles with RPI from a single hospital.

**Conclusions** Risk of RPI was prevalent in COVID-19 SRs
published early in the pandemic, and RPI may also be
common in SRs of topics outside COVID-19. The impact of
RPI on SRs could dilute their validity. Statements about effect
sizes should be made carefully, ensuring studies have
carefully selected their population to include unique
participants. The following are suggestions to improve the
often-complex process of identifying RPI: (1) journals should
ask authors to state if any data have been published and, if so,
to provide a reference; (2) reporting guidelines (CONSORT,
STARD, CARE, STROBE, and PRISMA) should include a
domain asking the authors if any of the data has been
published elsewhere; and (3) the quality assessment tool for
SRs (AMSTAR) should include a domain that evaluates
whether the authors evaluate the inclusion of RPI.

**References**

1. Choi WS, Song SW, Ock SM, et al. Duplicate publication of
articles used in meta-analysis in Korea. SpringerPlus.
2014;3(1):182.
2. von Elm E, Poglia G, Walder B, Tramèr MR. Different
patterns of duplicate publication: an analysis of articles used
in systematic reviews. JAMA. 2004;291(8):974-980.
3. Recommendations for the Conduct, Reporting, Editing,
and Publication of Scholarly work in Medical Journals.
International Committee of Medical Journal Editors. Updated
May 2022. Accessed January 29, 2022. http://icmje.org/
recommendations/

**Conflict of Interest Disclosures** None reported.

**Funding/Support** This study was funded in part by the
intramural research program of the National Institutes of Health.

**Role of the Funder/Sponsor** The sponsor was not involved in
the design or conduct of the study, the preparation of the abstract, or
the decision to submit the abstract.

**Additional Information** Pablo J. Moreno-Peña and Miguel
Zambrano-Lucio share equal credit as co–first authors. Fady
Hannah-Shmouni is a co–corresponding author.

![](https://assets.underline.io/uploads/markdown_image/1/image/120905f0e370a4d658db58c247084fbf.png)

Underline digital video library

Peer-mentoring intervention supports undergraduate Latinas and African-American students in STEM

UI Journal: Disseminating Interventions Research

Assessment and Comparison of Preprints and Peer-Reviewed Publications of Reporting Characteristics of Randomized Clinical Trials of Pharmacologic Treatment for COVID-19

Development of a List to Detect Statistical and Methodological Terms in Peer Reviews: A Cross-Sectional Study

Characterization of Publications on Post-Retraction Citation of Retracted Articles

Methodological and Reporting Quality of Systematic Reviews in Dermatology

Factors Affecting Publication of Pediatric Intervention Trials

Completeness of Reporting and Its Association With Risk of Bias in Systematic Reviews Published in Rehabilitation Journals: A Meta-research Study

Comparison of Changes in High-Quality vs Low-Quality Evidence in Original and Updated Systematic Reviews

Characteristics and Opportunities for Improvement of Methods Guidance Published in General and Methodology-Focused Medical Journals

An International Survey of Biomedical Researchers Knowledge, Perceptions, and Training on Peer Review

Assessing Repeated Patient Information in Systematic Reviews Published Early in the COVID-19 Pandemic

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES