United States

**Objective** To analyze the proportion of spin strategies
present in randomized clinical trials (RCTs) published in
high-impact journals appraised at low risk of bias.

**Design** For this cross-sectional study, a comprehensive
search was made in Ovid MEDLINE, retrieving all RCTs
published in the 10 highest-impact medical journals in
medicine and surgery (by 2018 Journal Citation Reports
impact factor) from January 1, 2018, to February 28, 2020.
Stratified sampling was then performed of 150 articles,
adjusted by journal. Then, the risk of bias of each RCT was
assessed independently and in duplicate using the Cochrane
risk-of-bias tool. For each included article, an adaptation of
the classification scheme created by Boutron et al1
for RCTs
was used to identify spin strategies in every section of the
article.

**Results** A total of 46 RCTs at low risk of bias were appraised
using the spin classification scheme. From those included in
the analysis, 25 studies were published in internal or general
medicine journals and 21 in surgery journals. The number of
patients included in the RCTs ranged from 20 to 12,000. The
most spin was identified in the discussion and conclusion
sections, with the most common strategies being focusing on
statistically significant secondary outcomes (4 [8.7%]), ruling
out adverse events (4 [8.7%]), acknowledging statistically
nonsignificant results for the primary outcome but
emphasizing the beneficial effect of treatment (4 [8.7%]), and
focusing only on statistically significant results (5 [10.9%]). In
subgroup analyses, a total of 38 spin strategies were identified
in studies with nonsignificant primary outcomes vs 11 in
studies with significant results. Assessment according to
specialty identified 35 spin strategies in surgery journals and
14 in internal medicine journals.

**Conclusions** Overall, the use of spin strategies was found to
be more common in studies with a statistically nonsignificant
primary outcome vs those with a significant primary outcome.
This study highlights the need to raise awareness about the
use of spin, even in studies published in high-impact journals
and at low risk of bias, as the use of these strategies can affect
readers’ decision-making.

**Reference**

1. Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and
interpretation of randomized controlled trials with
statistically nonsignificant results for primary outcomes.
JAMA. 2010;303(20):2058-2064. doi:10.1001/jama.2010.651

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/5451b89b7d6c71966bd2b23f3e7d34c2.png)

Peer Review Congress 2022

Spin in Randomized Clinical Trials of Top Medical Journals

publication

bias

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

IJCNLP-AACL 2021

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP).

poster

Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.
 
 **Welcome!**
 
 The JAMA Network, The BMJ, and METRICS welcome you to the Ninth International Congress on Peer Review and Scientific Publication. We have continued our efforts to broaden the scope of the Congress to all aspects of peer review and publication—from funding to postpublication—and to all sciences.
 
 We will have 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination. There are 50 plenary session research presentations and 5 plenary invited talks. Each plenary session research presentation will be followed by equal time for discussion and questions from the audience. In addition, there are 125 posters. All posters, including virtual posters, are available to view throughout the meeting. In-person posters will also be presented at designated times on September 9 and 10. If you are attending virtually, please visit the individual poster pages and ask your questions in the chat box. See the link to Posters on the left.
 
 We hope you will take an active part in the program, as we depend on your participation in the discussion sessions to make the Congress a success.

 [Peer Review Congress Organizers, Directors, and Advisory Board](https://peerreviewcongress.org/organizers-and-advisory-board/)
 
 [Health and Safety Policies](https://web.cvent.com/event/1b505794-6a0b-428c-a44c-6c46c4c50e99/websitePage:152962bb-4567-4561-9d81-21ebfa97f487)

![](https://assets.underline.io/uploads/markdown_image/1/image/7c9f0143dee383780cc4670269ade132.png)

A PDF of the schedule is available [here](https://drive.google.com/file/d/1-Jd191W-nxbWcN1prdX5o9nlSfeo9_-H/view?usp=sharing) and a flipbook version of the complete program and abstracts is included below. See also the link to the interactive schedule to the left.

<iframe width="700px" height="425px" src="https://online.fliphtml5.com/ebtyf/nlfn/#p=1" frameborder="0" allowfullscreen allowtransparency></iframe>

Registration is required to attend this event, please use the URL linked below to secure your ticket. Registration will close on Friday, September 2 at 5:00 PM CT. 

[**Register Here**]( https://web.cvent.com/event/1b505794-6a0b-428c-a44c-6c46c4c50e99/summary)

Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.



**Objective** According to meta-analyses and practice
guidelines, benzodiazepines are effective in the treatment of
panic disorder.1,2 However, to date, no meta-analyses have
incorporated data from unpublished trials. Among all
benzodiazepines, alprazolam is the most widely prescribed
and has the highest frequency of nonmedical use, abuse, and
related harms in the US.3
This study examined reporting bias
with the extended-release (XR) formulation of alprazolam by
comparing its efficacy for panic disorder using trial results
from the published literature and the US Food and Drug
Administration (FDA).

**Design** There was no protocol for this study, and it was not
registered. Medical and statistical reviews for alprazolam XR
were downloaded from Drugs@FDA (https://www.
accessdata.fda.gov/scripts/cder/daf/index.cfm); all phase 2
and 3 randomized, double-blind, placebo-controlled efficacy
trials were identified; summary statistics on 5 primary
outcome measures were extracted; and the FDA’s regulatory
decision as to whether, for purposes of approval, the trial
provided evidence of efficacy (statistical superiority to placebo on all primary outcomes) was also extracted. For each
FDA-registered trial, the published literature was searched
for matching publications using PubMed, bibliographies of
review articles, and Google Scholar. The best match between
FDA-registered trials and publications was based on drug
name, comparator, dosage groups, sample size, duration, and
investigator name. Summary data on the drug-placebo
comparison and whether the publication conveyed that the
drug was effective were extracted. Two meta-analyses were
conducted—one based on the FDA review and the other based
on the published literature—and their effect sizes were
compared. Reporting bias was examined by comparing the
following: (1) overall trial results (positive or not) according
to the FDA vs corresponding publications and (2) effect size
(Hedges’ g) using FDA data vs published data. Risk of bias
was not assessed because the objective was not to assess bias
in trial methods (internal validity) but rather bias in results
reporting.

**Results** The FDA review showed that 5 trials were
conducted, only 1 of which (20%) was positive and published
(as positive). The remaining 4 studies failed to demonstrate
efficacy. Of those, 2 were not published; for the other 2, the
articles selectively reported positive, nonprimary, or post hoc
outcomes. Thus, according to the published literature, 3 of 3
trials (100%) appeared to show positive results. Alprazolam’s
overall effect size calculated using FDA data was 0.33 (95%
CI, 0.07-0.59), while that based on published trial data was
0.47 (95% CI, 0.30-0.65), an increase of 0.14, or 42%
(**Figure 29**).

![](https://assets.underline.io/uploads/markdown_image/1/image/ccb76e5bb705b4f15684a9968911e67e.png)

**Conclusions** According to the results of this analysis,
reporting bias has inflated the apparent efficacy of alprazolam
XR, as previously found with other drug classes. Because this
inflation alters the risk-benefit ratio, clinicians may wish to
reconsider their prescribing practices with respect to this
benzodiazepine. This study highlights the value of regulatory
data to public health.

**References**

1. Stein MB, Goin MK, Pollack MH, et al. Practice Guideline
for the Treatment of Patients With Panic Disorder. American
Psychiatric Association; 2010.
2. Wilkinson G, Balestrieri M, Ruggeri M, Bellantuono C.
Meta-analysis of double-blind placebo-controlled trials of
antidepressants and benzodiazepines for patients with panic
disorders. Psychol Med. 1991;21(4):991-998. doi:10.1017/
s0033291700029986
3. Ait-Daoud N, Hamby AS, Sharma S, Blevins D. A review of
alprazolam use, misuse, and withdrawal. J Addict Med.
2018;12(1):4-10. doi:10.1097/adm.0000000000000350

**Conflict of Interest Disclosures** Rosa Y. Ahn-Horst reported
no conflicts of interest. Erick H. Turner previously worked as a
medical officer for the US Food and Drug Administration and
reviewed applications submitted by pharmaceutical companies to
determine whether they should be approved for US marketing. He
has no financial interest in any pharmaceutical products, approved
or otherwise.

![](https://assets.underline.io/uploads/markdown_image/1/image/2a22e0a53db11370664d0a9941bdb0c9.png)



Analysis of Reporting Bias in Published and Unpublished Trials of Extended-Release Alprazolam for Panic Disorder

**Objective** This research aimed to perform a comparison of
basic bibliometrics of leading Chinese open access (OA)
journals from the Excellence Action Plan for Chinese Science,
Technology, and Medicine Journals. All journals used English
as the publication language.

**Design** This was a cross-sectional investigation reported
following the STROBE checklist for conference abstracts.
Journal data were extracted from the Web of Science (WoS)
and Directory of Open Access Journals databases. All
searches were performed in November 2021, and the Journal
Citation Reports 2020 impact factor release was considered
the latest available data set that allowed for retrieval of
relative data. The top 2 international OA journals in the first
quartile of the journal impact factor rank based on WoS
categories for each leading Chinese OA journal were chosen
as control journals. Bibliometric indicators, such as the
volume of published documents, citations, and Eigenfactor
score (based on items published between 2018 and 2020), were compared between leading Chinese OA journals and
control journals with nonparametric tests.

**Results** A total of 14 of 22 (63.64%) leading Chinese journals
were OA journals. For 2 OA journals, there was no
corresponding international OA journal; for another 2 OA
journals, there was only 1 corresponding OA journal.
Therefore, there were 22 control journals. The median (IQR)
number of published documents (articles and reviews) per
leading Chinese OA journal was less than that of the control
journals (302 [243-352] vs 715 [390-2010]; P = .004); the
median (IQR) citations per document (13 [9-26] vs 14 [9-25];
P = .99) and percentage of cited documents (94.81% [93.63%-
98.24%] vs 96.96% [92.64%-98.14%]; P = .76) were not
statistically different. The median (IQR) normalized citation
impact of the 2 groups was not significantly different (2.008
[1.221-2.933] vs 2.003 [1.353-3.093]; P = .97), but the median
(IQR) Eigenfactor score, which took into account both the
number of citations and the academic influence of citing
journals, of leading Chinese OA journals was lower than that
of control journals (0.005 [0.003-0.007] vs 0.013 [0.005-
0.052]; P = .02]. There were fewer top 1% documents in
leading Chinese OA journals (median [IQR], 10 [5-19] vs 34
[11-65]; P = .02); however, the percentage of top 1%
documents was not statistically different (median [IQR],
4.54% [1.18%-5.82%] vs 3.84% [1.86%-6.86%]; P = .94). The
average citations per top 1% document of leading Chinese OA
journals were more than that of control journals, but the
difference was not statistically significant (median [IQR], 103
[52-134] vs 72 [58-105]; P = .26).

**Conclusions** Compared with top international OA journals,
Chinese leading OA journals published fewer articles with
similar average citations, but they were less cited by highly
cited journals.

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/b465371259175e41a935b95b43682d2e.png)

Comparison of Bibliometrics of Leading Open Access Chinese Journals with Leading Non-Chinese Journals in Science, Technology, and Medicine

**Objective** This systematic review aimed to describe the
characteristics of clinical and biomedical research in
Malaysia.

**Design** A search was conducted on PubMed, Embase,
CINAHL, PsycINFO, and MyMedR (http://mymedr.afpm.
org.my/) for published clinical and biomedical research in primary care settings from 1962 to 2017 by Malaysian authors
in a Malaysian institution. Studies found were independently
screened by a team of reviewers and information was
extracted.1,2 In phase 1, the characteristics of the research and
profiles of the researchers and journals in which they were
published were reported descriptively. In phase 2, the quality
of studies included in phase 1 will be assessed using a newly
developed tool to ascertain risk of bias. Longitudinal trends of
the research characteristics, health conditions studied
(International Classification of Primary Care), and settings,
among other characteristics, were explored. No synthesis of
results was conducted as no effect estimates were available to
be pooled.

**Results** Of 4513 articles, 1078 were included in this
qualitative synthesis and 790 with complete data were
analyzed. Clinical studies (81.9%), primary research (81.1%),
and quantitative studies (74.2%), consisting mostly of
prevalence studies (67.7%) by cross-sectional sampling
(70.4%), were predominant. The number of studies increased
(**Figure 30**) and the number of characteristics also increased
after year 2000. Researchers from family medicine (39.3%)
and public health (15.2%) specialties were the main
contributors to the articles (**Figure 30**, A). Most of the
corresponding authors had a master of medicine degree
(46.5%) compared with a doctor of philosophy (PhD) (24.4%)
or doctor of medicine (MD) (5.2%) degree. Researchers with
PhD and MD degrees were more likely to conduct
interventional studies compared with those with master’s
degrees (8.0% vs 6.4% of studies; χ2 = 54.26; P = .03).
Publications were mainly original research (82.8%) in
international (48.6%) or local (35.0%) journals and were
evenly distributed between multidisciplinary (51.6%) and
discipline-specific (46.6%) journals. The number of authors
per article was most often fewer than 5 (73.6%), and the
number of collaborating institutions was predominantly
fewer than 3 (82.5%). Incidences of coauthorship and collaboration with overseas researchers were few but showed
a significant increasing trend in the last decade (Figure 30,
B). The top 5 conditions studied were general and unspecified
(37.9%); endocrine, metabolic, and nutritional (15.2%);
circulatory (7.8%); psychological (5.9%); and respiratory
(4.9%).

![](https://assets.underline.io/uploads/markdown_image/1/image/46f7fc54671d85ca7536705d5864ba71.png)

**Conclusions** The longitudinal and prospective trends of the
research characteristics assessed in this analysis provided
suggestions of improvement initiatives needed for primary
care research enterprise in Malaysia. This includes training
on the proper use of different study designs, on developing a
better supportive ecosystem for interventional clinical trials,
on skills for international research collaboration, and for
strategizing research topics that meet the issues of primary
medical care. Similar works in other disciplines could be
initiated and better conducted after this first experience. The
aim of phase 2 will be to validate a research-quality screening
tool based on domains of relevance, credibility of the
methods, and usefulness of the results.

**References**

1. Chew BH, Lim PY, Lee SWH, et al. A systematic review of
medical and clinical research landscapes and quality in
Malaysia and Indonesia [REALQUAMI]: the review protocol.
PROSPERO identifier: CRD42020152907. Updated April 28,
2020. Accessed July 14, 2022. https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=152907
2. Chew BH, Lim PY, Lee SWH, et al. A systematic review of
medical and clinical research landscapes and quality in
Malaysia and Indonesia [REALQUAMI]. OSF Registry for
Research on the Responsible Conduct of Research. May 1,
2019. Accessed July 14, 2022. https://osf.io/w85ce

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/37b0ecbcea3db3dda80fcaec6d325d90.png)

A Systematic Review of Medical and Clinical Research Landscapes in Primary Medical Care in Malaysia

**Objective** To compare financial conflict of interest (COI) as
declared by authors of systematic reviews (SRs) of
methylphenidate for attention-deficit disorder with publicly
available information and in relation to risk of bias (RoB).

**Design** SRs on the outcomes associated with
methylphenidate for attention-deficit disorder in all ages were
searched in Medline, Cochrane, Embase, and PsycInfo. The
Preferred Reporting Items for Systematic Reviews and
Meta-analyses (PRISMA) reporting guideline was followed
for the selection of relevant SRs. Two reviewers (A.S. and
S.C.) independently screened open websites and recent
publications for all authors of each SR, and data on financial
COI were extracted. All searches followed a preplanned and
similar routine. A time limit of 3 years from publication of the
index SR was applied. Findings were discussed between the
reviewers and repeated until consensus was reached. If no
data were found for any of the authors, the SR was
categorized as no COI. Two reviewers (A.S. and L.O.)
independently judged RoB of the SR using the Risk of Bias in
Systematic Reviews (ROBIS) tool. Any disagreement was
resolved in consensus. In addition, data were retrieved on
COI as declared by authors.

**Results** Of 651 unique publications, 44 relevant SRs
published between 2008 and 2021 were included. In all, 32
SRs (73%) were based on randomized clinical trials only, 18
(41%) reported positive effects only, 15 (34%) reported both
positive and adverse effects, and 11 (25%) reported adverse
effects only. Eleven SRs (25%) included only studies using
placebo for comparison. A meta-analysis was conducted in 26
of the SRs (59%). COI disclosure was missing for 2 of 44 SRs
(5%). For 15 SRs (34%), authors declared COI, and this
declaration was confirmed by open sources in all cases. For 27
SRs (61%), the authors declared no COI, but discordant
information was publicly available for 8 of 27 (30%). The
direction of COI for most SRs was not able to be assessed. The
RoB was high in 37 of 44 SRs (84%). Of the 7 SRs with low
RoB, 1 had no COI identified in open sources (**Table 78**).

![](https://assets.underline.io/uploads/markdown_image/1/image/df8e29168dd1f7fe42b80257f1d7c32c.png)

**Conclusions** The findings indicated an underreporting of
COI in SRs in studies on attention-deficit disorder, and most
SRs were compromised by high RoB. Owing to small
numbers, no firm conclusion on the association between COI
and RoB was possible.

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/43fce7efe9e39c5aa2156e3ba3ad6038.png)



Conflicts of Interest in Systematic Reviews on Methylphenidate for Attention-Deficit Disorder

**Objective** As gatekeepers for knowledge production, editors
use their backgrounds to determine what counts as high-
quality research, how research should be done, and whether it
offers novel insights. Individuals who come from racialized
backgrounds are often excluded from these conversations,
ensuring the publishing world continues to be led by the
perspectives of the dominant racial group.1
To address this
issue, the journal Teaching and Learning in Medicine (TLM)
created an internship program for early career scholars who
self-identified as Black, Latinx, or Indigenous to gain
experience in the publishing/editorial process.2
The purpose
of this study was to critically examine the outcomes of the
first 6 months and investigate its implications for other
journals interested in similar efforts.

**Design** A collaborative autoethnographical study was
designed to collectively analyze the program. Editorial staff
and interns reflected on their lived experiences participating
in the internship to understand the current culture of
academic publishing and explore how to improve.3
Data
sources included archival emails and program planning
documents, focus group data, and group exercises. After
transcribing the focus groups, data were analyzed using thematic analysis by both editorial staff and interns.
Institutional review board approval was not required for this
study.

**Results** Focus group data indicated that while TLM designed
the program with antiracist intent, mentors did not
foreground their race in the editorial/publishing process
outside of the intern’s minoritized status as a program
selection criterion. Early on, mentors viewed the internship
more as an opportunity to improve the journal’s editorial
work, for participants to gain experience working in a
historically guarded space, and to contribute to achieving
racial equity in medical education. Therefore, despite the
program’s specific antiracist focus, 6 months in, none of the
mentors (who identified as White individuals) had explicitly
discussed topics of race with their interns. Mentors’ reasons
for not discussing race varied, including uncertainty about
how to invite interns into such a discussion and not seeing
interns as racialized individuals. However, at the end of the
6 months, researchers realized the need to discuss this topic,
thus moving the program into explicit conversations about
race and the role it plays in publishing.

**Conclusions** Although the program met some antiracist
goals, stakeholders did not explicitly discuss the role that race
plays in the review process, therefore limiting the program’s
initial impact. Through this collaborative autoethnography,
TLM stakeholders critically reflected on the program in real
time and addressed this gap. In doing so, they engaged in the
ongoing critical action needed to support equity within the
editorial process. Editorial staff at TLM now have the
opportunity to address ongoing power dynamics between
interns and staff members that will advance the journal’s
efforts at antiracism. Plans are currently underway to ensure
interns’ experiences are incorporated.

**References**

1. Salazar JW, Claytor JD, Habib AR, Guduguntla V, Redberg
RF. Gender, race, ethnicity, and sexual orientation of editors
at leading medical and scientific journals: a cross-sectional
survey. JAMA Intern Med. 2021;181(9):1248-1251.
doi:10.1001/jamainternmed.2021.2363
2. Cianciolo AT. Letter from the editor—Teaching & Learning
in Medicine’s anti-racism strategy. Teach Learn Med.
2020;32(5):457-458. doi:10.1080/10401334.2020.1836461
3. Chang H, Ngunjiri FW, Hernandez KAC. Collaborative
Autoethnography. Left Coast Press; 2013.

**Conflict of Interest Disclosures** None reported.

**Disclaimer** This work was prepared by a military or civilian
employee of the US government as part of the individual’s official
duties and therefore is in the public domain. The opinions and
assertions expressed herein are those of the author(s) and do not
necessarily reflect the official policy or position of the Uniformed
Services University or the US Department of Defense.

**Additional Information** Anabelle Andon is a co–corresponding
author.

![](https://assets.underline.io/uploads/markdown_image/1/image/a8d384158802bc2cfad5c9d7e26fb85b.png)

Qualitative Assessment of an Antiracism Editorial Internship Program for Early Career Underrepresented Scholars at Teaching and Learning in Medicine

**Objective** Editorial decisions can depend on factors, like the
timing of submissions or the matching of editors and
reviewers, that are independent of the quality of the work.
This analysis investigates associations of these and other
external factors with editorial outcomes at eLife, a major
biomedical journal.

**Design** This study analyzed whether timing of submission
(weekend vs weekday) was associated with the decision to
send manuscripts for external review among senior editors,
who can desk-reject submissions; compared peer review
manuscript ratings by reviewer volume; and assessed whether
submission time of month or year, preceding decisions to
reject or review a manuscript, and matching of reviewer to
manuscript specialty influenced editorial decisions. Data were
analyzed by single variable regression.

**Results** Between January 2016 and December 2018 eLife
received 23,190 total submissions, 6498 of which were sent
for review. Among senior editors (n = 65), proportions of
manuscripts sent for external review ranged from 9.6% to
49.3% and were statistically significantly lower on weekends
(mean, 24% [SD, 1.3%]) than on weekdays (mean, 29% [SD,
0.6%]; P < .001), an association observed for most senior
editors. Average peer reviewer rating (range, 0-1) increased
with volume category: mean of 0.453 (SD, 0.003) for 1 to 5
submission reviews; 0.463 (SD, 0.008) for 6 to 10
submissions; and 0.472 (SD, 0.007) for 11 or more
submissions, and reviewers’ ratings increased with successive
reviews. In a nonquantitative inspection of submission and
decision trends, submission time of month or year, preceding
decisions to reject or review a manuscript, and matching of
reviewer to manuscript specialty did not appear to influence
editorial decisions.

**Conclusions** This study found a statistically significant
association between timing of submission during the week
and editorial decisions. Peer reviewer ratings increased with
review volume. Submission time of month or year, preceding
decisions to reject or review a manuscript, and matching of reviewer to manuscript specialty did not appear to influence
editorial decisions.

**Conflict of Interest Disclosures** None reported.

**Funding/Support** This research is supported by National Science
Foundation SCISIPBIO (A Science of Science Policy Approach to
Analyzing and Innovating the Biomedical Research Enterprise)
project 2022435.

**Additional Information** We thank Andy Collings and James
Gilbert from eLife for sharing the data and providing explanations
and eLife’s editorial leadership team for input and suggestions.

![](https://assets.underline.io/uploads/markdown_image/1/image/7aad5dd3d2641b2ae252900ca6469e10.png)

Analysis of Timing of Manuscript Submissions and Assignment of Editors and Reviewers on Editorial Decisions at eLife

**Objective** Many journals encounter difficulties obtaining
peer reviewers. However, it is unknown if this is associated
with publication outcomes.1,2 The purpose of this research
was to evaluate whether, for initial manuscript submissions
that were eventually peer reviewed, there was a significant
association between the number of external peer reviewers
sought, unsuccessful invites, and declined invites for external
peer review and the publication outcome.

**Design** This was a retrospective study of anonymized,
unique, original research submissions to the Emergency
Medicine Journal (EMJ) that received at least 1 external peer
review over a 5-year period (January 2016 to December
2020). A database of deidentified original manuscripts
submitted to EMJ during the study period was interrogated to determine if there were significant associations between the
number of unsuccessful external peer review invitations, the
number of total invitations needed, and the acceptance or
rejection of a research manuscript. Original submissions
without any external peer review were excluded. Statistical
review invitations were excluded from the data. Mann-
Whitney U test was used to assess differences between the
variables and their publication outcomes. Odds ratios (OR),
likelihood ratios (LR), and positive predictive value (PPV)
were used as measures of association for potential thresholds
for variables and publication outcomes.

**Results** There were 806 deidentified peer-reviewed original
submissions included with 85 manuscripts (10.6%) accepted
for publication during the study period (**Table 79**). The ORs
for a peer-reviewed original research submission eventually
being rejected, according to number of invitations, were 77.5
(95% CI, 24.2-248.2; P < .001; (LR, 21.0; 95% CI, 6.9-63.7;
PPV, 99.4%; 95% CI, 98.3-99.8) for submissions with 4 or
more invitations, 36.8 (95% CI, 9.0-150.9; P < .001; LR, 20.0;
95% CI, 5.1-78.8; PPV, 99.4%; 95% CI, 97.7-99.9) for
submissions with 3 or more unsuccessful external peer review
invitations, and 22.6 (95% CI, 3.1-163.8; LR, 18.0; 95% CI,
2.6-127.2; PPV, 99.4%; 95% CI, 95.6-99.9) for submissions
with 2 or more peer reviewers who declined review
invitations.

![](https://assets.underline.io/uploads/markdown_image/1/image/c65a6b7606fb2ba5ad006d5543dce3f1.png)

**Conclusions** The number of declined peer review invitations
and total review invitations prior to a decision were
associated with rejection of a manuscript. The wide 95% CIs
in these results could be due to the high variability of
underlying factors that could have influenced the difficulty in
getting peer reviews and their interplay with the decision to
publish. The findings of this study may also be potentially
different for different journals.3
Further research should be
done to provide further insights on specific factors that may
be associated with difficulties in getting peer reviewers.1-3

**References**

1. Tvina A, Spellecy R, Palatnik A. Bias in the peer review
process: can we do better? Obstet Gynecol. 2019;133(6):1081-
1083. doi:10.1097/AOG.0000000000003260
2. Tennant JP, Dugan JM, Graziotin D, et al. A multi-
disciplinary perspective on emergent and future innovations
in peer review. F1000Res. 2017;6:1151. doi:10.12688/
f1000research.12037.3
3. Stamm T, Meyer U, Wiesmann HP, Kleinheinz J, Cehreli
M, Cehreli ZC. A retrospective analysis of submissions,
acceptance rate, open peer review operations, and
prepublication bias of the multidisciplinary open access
journal. Head Face Med. 2007;3:27. doi:10.1186/1746-
160X-3-27

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/fe4b8ba267367c8f45db9c17a01bae41.png)

Association Between Number of External Peer Review Invites, Unsuccessful Invites, and Declined Reviews With Rejection of Manuscripts

**Objective** Over the past decade, several journals in Africa
have transitioned from email submission systems to web-
based platforms for manuscript submission and editorial
processes.1
Previous reports1,2 assessing experience with these
systems have revealed conflicting feedback from authors and
editorial teams, with varying reports of user friendliness.
Notably, a letter to the editor published in Nature3
stated that
such systems “make authors do all the work.” Over time,
online submission systems have sought to have in-built
programs to enable automation of the submission process to
ease the authors’ experience. The Annals of African Surgery
uses the submission prefill feature in its submission system
for the aforementioned purpose.

**Design** This study was cross-sectional. To establish efficacy
and gather authors’ experience of the online submission
system and the submission prefill (ie, the system that
automatically extracts and populates submission fields for
authors from an uploaded manuscript text file), authors
submitting articles to the Annals of African Surgery were
surveyed between September and December 2021. This
survey was delivered to authors using the manuscript
submission system, during the submission process, and at the
end of the editorial workflow after the final decision was
made.

**Results** A total of 44 authors completed the survey on
submission prefill, and 42 completed the survey on the
manuscript submission system. Because the surveys were embedded as part of the submission processes, the response
rate was 100%. Overall, the responses to the question of
whether the submission prefill was effective in reducing the
workload were divided evenly (50.0% agree vs 50.0%
neutral), but there was overwhelmingly positive feedback on
the effectiveness of manuscript submission system in terms of
its ease of use (88.1%) and in obtaining the status of the
manuscript (92.9%). The survey questions and their
respective responses are as outlined in **Table 80**.

![](https://assets.underline.io/uploads/markdown_image/1/image/e973f191539223b5f0e1324aee8ab981.png)

**Conclusions** There was overall positive feedback regarding
authors’ interaction with the web-based online submission
interface. Feedback from authors’ experience on whether
submission prefills reduce submission labor, however, still
remains divided. Measures taken to improve the prefill’s
capacity to extract manuscript details with better accuracy
would, hence, enhance authors’ experience.

**References**

1. Ware M. Online submission and peer-review systems.
Learned Publishing. 2005;18(4):245-250.
doi:10.1087/095315105774648771
2. Ware M. Authors say that they prefer online submission.
Nature. 2005;434(7033):559. doi:10.1038/434559d
3. Moore JP. Online submission makes authors do all the
work. Nature. 2005;433(7028):800. doi:10.1038/433800d

**Conflict of Interest Disclosures** James Kigera is a member of
the Peer Review Congress Advisory Board but was not involved in
the review or decision for this abstract. No other disclosures were
reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/90eea1861b5ade470345f3fae96381f8.png)



Authors' General Experiences With Submitting Manuscripts and With Submission Prefill to a Manuscript Submission System for the Annals of African Surgery

**Objective** The objective of this research was to assess
whether an editorial workflow wherein submissions are
handled solely by dedicated protocol editors reduces the
average time from initial submission to a final decision
compared with standard peer review and whether this
reduction differs by protocol formatting (structured or
unstructured).

**Design** A cross-sectional study of the workflow timing for
protocol submissions to Trials was conducted. Type A
protocols demonstrate prior peer review as part of their
funding process by providing previous comments and
self-declaring this prior review in the submission process.
Type A protocols are handled by protocol editors as the sole
reviewers, whereas type B protocols are handled by other
editors and require full peer review (ie, 2 to 3 reviewers). The
workflow data stored in the Trials manuscript submission
and review system were extracted to compare the timing from
initial submission to first and final decisions for all protocols,
separated by type, before and after protocol editors were
implemented (January to December 2019 and January 2020
to November 2021, respectively). The timing was also
extracted for submissions that were formatted as
unstructured protocols with an accompanying SPIRIT
checklist or following the Structured Protocol Template
recommended by Trials. The workflow timing results are
descriptively summarized.

**Results** From January 2020 to November 2021, 1114 type A
(360 [32%]) and type B (754 [68%]) trial protocols were
submitted to Trials. Compared with type B protocols, 143 of
which (19%) used a structured template, 137 type A protocols
(38%) followed a structured format, possibly owing to self-selection or prior reviews leading to a more structured
approach in submission. A timeline of the workflow for each
protocol type by format as well as overall for each period is
shown in **Figure 31**. Overall, type A protocols had a mean of
118 days to final decision (from initial submission), whereas
type B protocols had a mean of 165 days to final decision.
Overall, across protocols in the period after protocol editors
were implemented, the time to final decision was reduced 30
days from the preceding year: 146 days (January 2020 to
November 2021) versus 176 days (January to December
2019). Among type A protocols, a structured template format
reduced the time to final decision compared with
unstructured—an association that was not seen in type B
protocols, possibly because multiple reviewers were required.

![](https://assets.underline.io/uploads/markdown_image/1/image/41a674ef8c75bec03869985bec7d5311.png)

**Conclusions** The use of dedicated editors to handle trial
protocols that had already undergone peer review was
associated with fewer days to final decision compared with
protocols that were handled by other editors and required full
peer review. Future work should examine the quality of peer
review in both workflows to determine whether use of
dedicated reviewers improves the quality of published works
as well as the time to publication.

**Conflict of Interest Disclosure**s Riaz Qureshi, Kirsty Loudon,
and Alexander Gough are protocol editors of Trials, BMC. Shaun
Treweek and Tianjing Li are editors in chief of Trials, BMC.

**Acknowledgments** We would like to thank Eleanor Cox, Bex
Chang, and Krishna Vairamani for their work as editorial staff and
for managing the workflow of peer review for Trials.

![](https://assets.underline.io/uploads/markdown_image/1/image/ce8b2b25aa2173d0019073c47efcd780.png)



Assessment of Use of Dedicated Editors for Handling and Reviewing Manuscripts With Previously Obtained Peer Reviews

**Objective** Peer review plays a fundamental role in scholarly
publishing, but its legitimacy has been increasingly
questioned. A growing literature discusses how reviewers’
demographic characteristics and biases might lead to
disparities in research dissemination.1,2 Because the extent to
which reviewers are able to determine the outcomes for
papers may vary, it is important to look at the relationship
between reviewers’ recommendations and editors’ decision-
making. However, reviewer recommendations are often
embedded in the text of the review. This work proposes a
method for automatic detection of recommendations based
on review text.

**Design** The automatic classification used a rule-based
algorithm that searched for the presence of 1 or more phrases
that signal the reviewer’s recommendation: accept, minor
revision, major revision, or reject categories, as defined on the
hand-coding process. The algorithm considered the different
combinations of signal phrases to define the outcome. The list
of signal phrases was iteratively built on 3 rounds of hand-
coding and fuzzy matching sentences, while the combinations
were defined to maximize the precision, on the hand-coded
cases. This study used Publons’ data set, which contained
3,310,791 reviews from 25,934 journals; while 600 cases were
hand-coded, a subset of 200 reviews was used to evaluate the
performance. The gender of reviewers was inferred by
matching first and last names to curated lists of country-
specific gendered names, including the US Census.3

**Results** The overall accuracy on the test was that 81% of
assigned recommendations were correct according to hand
coding (n = 149). Since the inclusion of additional phrases is
associated with lowered accuracy, this might indicate an
upper bound in our experiment, given the limits of the
current data and the idiosyncrasies of peer review language.
Nonetheless, the algorithm’s accuracy was comparable to the
rate of agreement between human hand coders (n = 60
[88%]). Over the full data, 14.3% of reviews were assigned a
recommendation by this method (n = 473,443). This was
comparable to the hand-coded identification of 18.3% of
reviews containing an explicit recommendation (n = 399).
From these results, we concluded that the inclusion of an
explicit recommendation remains relatively uncommon in
peer review, with the majority of peer reviewers leaving a final
decision on the manuscript as the responsibility of the editor,
but there was large variation between journals. Initial results
nonetheless showed gender differences in reviewing behavior,
with higher retrieval rates associated with reviewers who
identified as men.

**Conclusions** This work is among the first benchmarks for
automatic classification of review recommendations on a
large-scale, cross-domain database. Though preliminary, it
paves the way for future developments, including studies of potential biases and inequalities in scholarly publishing
through examination of the relationship between reviewer
characteristics and review outcomes.

**References**

1. Lee CJ, Sugimoto CR, Zhang G, Cronin B. Bias in peer
review. J Am Soc Inf Sci Tec. 2013;64(1):2-17. doi:10.1002/
asi.22784
2. Sun M, Barry Danfa J, Teplitskiy M. Does double-blind
peer review reduce bias? evidence from a top computer
science conference. J Am Soc Inf Sci Tec. 2022;73(6):811-819.
doi:10.1002/asi.24582
3. Larivière V, Ni C, Gingras Y, Cronin B, Sugimoto CR.
Bibliometrics: global gender disparities in science. Nature.
2013;504:211-213. doi:10.1038/504211a

**Conflict of Interest Disclosures** None reported.

**Funding/Support** This work was supported by the Doctoral
Training Unit on Data-Driven Computational Modelling and
Applications (DRIVEN), which is funded by the Luxembourg
National Research Fund under the PRIDE programme
(PRIDE17/12252781).

**Acknowledgment** We thank Brad Demarest and Chaoqun Ni for
contributions to an earlier phase of the project. We also thank the
team at Publons who implemented the gender inference algorithm
on their data before giving us access.

**AdditionalInformation** Diego Kozlowski is a co–corresponding
author.

![](https://assets.underline.io/uploads/markdown_image/1/image/3aed057350f3812bef4e49ec6d5137b7.png)

Spin in Randomized Clinical Trials of Top Medical Journals

Next from Peer Review Congress 2022

Analysis of Reporting Bias in Published and Unpublished Trials of Extended-Release Alprazolam for Panic Disorder

Similar lecture

Characterizing News Portrayal of Civil Unrest in Hong Kong, 1998–2020

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES