United States

**Objective** To compare financial conflict of interest (COI) as
declared by authors of systematic reviews (SRs) of
methylphenidate for attention-deficit disorder with publicly
available information and in relation to risk of bias (RoB).

**Design** SRs on the outcomes associated with
methylphenidate for attention-deficit disorder in all ages were
searched in Medline, Cochrane, Embase, and PsycInfo. The
Preferred Reporting Items for Systematic Reviews and
Meta-analyses (PRISMA) reporting guideline was followed
for the selection of relevant SRs. Two reviewers (A.S. and
S.C.) independently screened open websites and recent
publications for all authors of each SR, and data on financial
COI were extracted. All searches followed a preplanned and
similar routine. A time limit of 3 years from publication of the
index SR was applied. Findings were discussed between the
reviewers and repeated until consensus was reached. If no
data were found for any of the authors, the SR was
categorized as no COI. Two reviewers (A.S. and L.O.)
independently judged RoB of the SR using the Risk of Bias in
Systematic Reviews (ROBIS) tool. Any disagreement was
resolved in consensus. In addition, data were retrieved on
COI as declared by authors.

**Results** Of 651 unique publications, 44 relevant SRs
published between 2008 and 2021 were included. In all, 32
SRs (73%) were based on randomized clinical trials only, 18
(41%) reported positive effects only, 15 (34%) reported both
positive and adverse effects, and 11 (25%) reported adverse
effects only. Eleven SRs (25%) included only studies using
placebo for comparison. A meta-analysis was conducted in 26
of the SRs (59%). COI disclosure was missing for 2 of 44 SRs
(5%). For 15 SRs (34%), authors declared COI, and this
declaration was confirmed by open sources in all cases. For 27
SRs (61%), the authors declared no COI, but discordant
information was publicly available for 8 of 27 (30%). The
direction of COI for most SRs was not able to be assessed. The
RoB was high in 37 of 44 SRs (84%). Of the 7 SRs with low
RoB, 1 had no COI identified in open sources (**Table 78**).

![](https://assets.underline.io/uploads/markdown_image/1/image/df8e29168dd1f7fe42b80257f1d7c32c.png)

**Conclusions** The findings indicated an underreporting of
COI in SRs in studies on attention-deficit disorder, and most
SRs were compromised by high RoB. Owing to small
numbers, no firm conclusion on the association between COI
and RoB was possible.

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/43fce7efe9e39c5aa2156e3ba3ad6038.png)



Peer Review Congress 2022

Conflicts of Interest in Systematic Reviews on Methylphenidate for Attention-Deficit Disorder

conflict of interest

funding/sponsorship

research methods

World ENT Congress 2022 is an international conference that encourages effective knowledge transfer and best practices experts from around all the world. ENT 2022 is organized by the International Community and hosted by the SunText Reviews.

The World ENT Congress aims to strengthen the participants’ action and impact on national, regional, and international scales through a multidisciplinary program that features the integration of new technologies in the field of scientific improvements in ENT and Otorhinolaryngology.

World ENT Congress 2022 is scheduled in the Disney World and Universal Orlando. Orlando, being one of the centers for research, education, and science in the USA, we will try to bring the maximum number of Academicians, Researchers and Innovators, Experts, Industrialists, Professors, Researchers, Students, and Technical Staff, Engineers and Delegates, Product Developers, Solution Providers, Sales & Marketing Professionals, Associations, Societies, & Professional Bodies, Directors/Co-Directors of Research-based companies, and workshop teams to provide a better understanding of the field. 

World ENT Congress 2022

The World ENT Congress aims to strengthen the participants' action and impact on national, regional, and international scales through a multidisciplinary program that features the integration of new technologies in the field of scientific improvements in ENT and Otorhinolaryngology.

poster

Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.
 
 **Welcome!**
 
 The JAMA Network, The BMJ, and METRICS welcome you to the Ninth International Congress on Peer Review and Scientific Publication. We have continued our efforts to broaden the scope of the Congress to all aspects of peer review and publication—from funding to postpublication—and to all sciences.
 
 We will have 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination. There are 50 plenary session research presentations and 5 plenary invited talks. Each plenary session research presentation will be followed by equal time for discussion and questions from the audience. In addition, there are 125 posters. All posters, including virtual posters, are available to view throughout the meeting. In-person posters will also be presented at designated times on September 9 and 10. If you are attending virtually, please visit the individual poster pages and ask your questions in the chat box. See the link to Posters on the left.
 
 We hope you will take an active part in the program, as we depend on your participation in the discussion sessions to make the Congress a success.

 [Peer Review Congress Organizers, Directors, and Advisory Board](https://peerreviewcongress.org/organizers-and-advisory-board/)
 
 [Health and Safety Policies](https://web.cvent.com/event/1b505794-6a0b-428c-a44c-6c46c4c50e99/websitePage:152962bb-4567-4561-9d81-21ebfa97f487)

![](https://assets.underline.io/uploads/markdown_image/1/image/7c9f0143dee383780cc4670269ade132.png)

A PDF of the schedule is available [here](https://drive.google.com/file/d/1-Jd191W-nxbWcN1prdX5o9nlSfeo9_-H/view?usp=sharing) and a flipbook version of the complete program and abstracts is included below. See also the link to the interactive schedule to the left.

<iframe width="700px" height="425px" src="https://online.fliphtml5.com/ebtyf/nlfn/#p=1" frameborder="0" allowfullscreen allowtransparency></iframe>

Registration is required to attend this event, please use the URL linked below to secure your ticket. Registration will close on Friday, September 2 at 5:00 PM CT. 

[**Register Here**]( https://web.cvent.com/event/1b505794-6a0b-428c-a44c-6c46c4c50e99/summary)

Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.



**Objective** As gatekeepers for knowledge production, editors
use their backgrounds to determine what counts as high-
quality research, how research should be done, and whether it
offers novel insights. Individuals who come from racialized
backgrounds are often excluded from these conversations,
ensuring the publishing world continues to be led by the
perspectives of the dominant racial group.1
To address this
issue, the journal Teaching and Learning in Medicine (TLM)
created an internship program for early career scholars who
self-identified as Black, Latinx, or Indigenous to gain
experience in the publishing/editorial process.2
The purpose
of this study was to critically examine the outcomes of the
first 6 months and investigate its implications for other
journals interested in similar efforts.

**Design** A collaborative autoethnographical study was
designed to collectively analyze the program. Editorial staff
and interns reflected on their lived experiences participating
in the internship to understand the current culture of
academic publishing and explore how to improve.3
Data
sources included archival emails and program planning
documents, focus group data, and group exercises. After
transcribing the focus groups, data were analyzed using thematic analysis by both editorial staff and interns.
Institutional review board approval was not required for this
study.

**Results** Focus group data indicated that while TLM designed
the program with antiracist intent, mentors did not
foreground their race in the editorial/publishing process
outside of the intern’s minoritized status as a program
selection criterion. Early on, mentors viewed the internship
more as an opportunity to improve the journal’s editorial
work, for participants to gain experience working in a
historically guarded space, and to contribute to achieving
racial equity in medical education. Therefore, despite the
program’s specific antiracist focus, 6 months in, none of the
mentors (who identified as White individuals) had explicitly
discussed topics of race with their interns. Mentors’ reasons
for not discussing race varied, including uncertainty about
how to invite interns into such a discussion and not seeing
interns as racialized individuals. However, at the end of the
6 months, researchers realized the need to discuss this topic,
thus moving the program into explicit conversations about
race and the role it plays in publishing.

**Conclusions** Although the program met some antiracist
goals, stakeholders did not explicitly discuss the role that race
plays in the review process, therefore limiting the program’s
initial impact. Through this collaborative autoethnography,
TLM stakeholders critically reflected on the program in real
time and addressed this gap. In doing so, they engaged in the
ongoing critical action needed to support equity within the
editorial process. Editorial staff at TLM now have the
opportunity to address ongoing power dynamics between
interns and staff members that will advance the journal’s
efforts at antiracism. Plans are currently underway to ensure
interns’ experiences are incorporated.

**References**

1. Salazar JW, Claytor JD, Habib AR, Guduguntla V, Redberg
RF. Gender, race, ethnicity, and sexual orientation of editors
at leading medical and scientific journals: a cross-sectional
survey. JAMA Intern Med. 2021;181(9):1248-1251.
doi:10.1001/jamainternmed.2021.2363
2. Cianciolo AT. Letter from the editor—Teaching & Learning
in Medicine’s anti-racism strategy. Teach Learn Med.
2020;32(5):457-458. doi:10.1080/10401334.2020.1836461
3. Chang H, Ngunjiri FW, Hernandez KAC. Collaborative
Autoethnography. Left Coast Press; 2013.

**Conflict of Interest Disclosures** None reported.

**Disclaimer** This work was prepared by a military or civilian
employee of the US government as part of the individual’s official
duties and therefore is in the public domain. The opinions and
assertions expressed herein are those of the author(s) and do not
necessarily reflect the official policy or position of the Uniformed
Services University or the US Department of Defense.

**Additional Information** Anabelle Andon is a co–corresponding
author.

![](https://assets.underline.io/uploads/markdown_image/1/image/a8d384158802bc2cfad5c9d7e26fb85b.png)

Qualitative Assessment of an Antiracism Editorial Internship Program for Early Career Underrepresented Scholars at Teaching and Learning in Medicine

**Objective** Editorial decisions can depend on factors, like the
timing of submissions or the matching of editors and
reviewers, that are independent of the quality of the work.
This analysis investigates associations of these and other
external factors with editorial outcomes at eLife, a major
biomedical journal.

**Design** This study analyzed whether timing of submission
(weekend vs weekday) was associated with the decision to
send manuscripts for external review among senior editors,
who can desk-reject submissions; compared peer review
manuscript ratings by reviewer volume; and assessed whether
submission time of month or year, preceding decisions to
reject or review a manuscript, and matching of reviewer to
manuscript specialty influenced editorial decisions. Data were
analyzed by single variable regression.

**Results** Between January 2016 and December 2018 eLife
received 23,190 total submissions, 6498 of which were sent
for review. Among senior editors (n = 65), proportions of
manuscripts sent for external review ranged from 9.6% to
49.3% and were statistically significantly lower on weekends
(mean, 24% [SD, 1.3%]) than on weekdays (mean, 29% [SD,
0.6%]; P < .001), an association observed for most senior
editors. Average peer reviewer rating (range, 0-1) increased
with volume category: mean of 0.453 (SD, 0.003) for 1 to 5
submission reviews; 0.463 (SD, 0.008) for 6 to 10
submissions; and 0.472 (SD, 0.007) for 11 or more
submissions, and reviewers’ ratings increased with successive
reviews. In a nonquantitative inspection of submission and
decision trends, submission time of month or year, preceding
decisions to reject or review a manuscript, and matching of
reviewer to manuscript specialty did not appear to influence
editorial decisions.

**Conclusions** This study found a statistically significant
association between timing of submission during the week
and editorial decisions. Peer reviewer ratings increased with
review volume. Submission time of month or year, preceding
decisions to reject or review a manuscript, and matching of reviewer to manuscript specialty did not appear to influence
editorial decisions.

**Conflict of Interest Disclosures** None reported.

**Funding/Support** This research is supported by National Science
Foundation SCISIPBIO (A Science of Science Policy Approach to
Analyzing and Innovating the Biomedical Research Enterprise)
project 2022435.

**Additional Information** We thank Andy Collings and James
Gilbert from eLife for sharing the data and providing explanations
and eLife’s editorial leadership team for input and suggestions.

![](https://assets.underline.io/uploads/markdown_image/1/image/7aad5dd3d2641b2ae252900ca6469e10.png)

Analysis of Timing of Manuscript Submissions and Assignment of Editors and Reviewers on Editorial Decisions at eLife

**Objective** Many journals encounter difficulties obtaining
peer reviewers. However, it is unknown if this is associated
with publication outcomes.1,2 The purpose of this research
was to evaluate whether, for initial manuscript submissions
that were eventually peer reviewed, there was a significant
association between the number of external peer reviewers
sought, unsuccessful invites, and declined invites for external
peer review and the publication outcome.

**Design** This was a retrospective study of anonymized,
unique, original research submissions to the Emergency
Medicine Journal (EMJ) that received at least 1 external peer
review over a 5-year period (January 2016 to December
2020). A database of deidentified original manuscripts
submitted to EMJ during the study period was interrogated to determine if there were significant associations between the
number of unsuccessful external peer review invitations, the
number of total invitations needed, and the acceptance or
rejection of a research manuscript. Original submissions
without any external peer review were excluded. Statistical
review invitations were excluded from the data. Mann-
Whitney U test was used to assess differences between the
variables and their publication outcomes. Odds ratios (OR),
likelihood ratios (LR), and positive predictive value (PPV)
were used as measures of association for potential thresholds
for variables and publication outcomes.

**Results** There were 806 deidentified peer-reviewed original
submissions included with 85 manuscripts (10.6%) accepted
for publication during the study period (**Table 79**). The ORs
for a peer-reviewed original research submission eventually
being rejected, according to number of invitations, were 77.5
(95% CI, 24.2-248.2; P < .001; (LR, 21.0; 95% CI, 6.9-63.7;
PPV, 99.4%; 95% CI, 98.3-99.8) for submissions with 4 or
more invitations, 36.8 (95% CI, 9.0-150.9; P < .001; LR, 20.0;
95% CI, 5.1-78.8; PPV, 99.4%; 95% CI, 97.7-99.9) for
submissions with 3 or more unsuccessful external peer review
invitations, and 22.6 (95% CI, 3.1-163.8; LR, 18.0; 95% CI,
2.6-127.2; PPV, 99.4%; 95% CI, 95.6-99.9) for submissions
with 2 or more peer reviewers who declined review
invitations.

![](https://assets.underline.io/uploads/markdown_image/1/image/c65a6b7606fb2ba5ad006d5543dce3f1.png)

**Conclusions** The number of declined peer review invitations
and total review invitations prior to a decision were
associated with rejection of a manuscript. The wide 95% CIs
in these results could be due to the high variability of
underlying factors that could have influenced the difficulty in
getting peer reviews and their interplay with the decision to
publish. The findings of this study may also be potentially
different for different journals.3
Further research should be
done to provide further insights on specific factors that may
be associated with difficulties in getting peer reviewers.1-3

**References**

1. Tvina A, Spellecy R, Palatnik A. Bias in the peer review
process: can we do better? Obstet Gynecol. 2019;133(6):1081-
1083. doi:10.1097/AOG.0000000000003260
2. Tennant JP, Dugan JM, Graziotin D, et al. A multi-
disciplinary perspective on emergent and future innovations
in peer review. F1000Res. 2017;6:1151. doi:10.12688/
f1000research.12037.3
3. Stamm T, Meyer U, Wiesmann HP, Kleinheinz J, Cehreli
M, Cehreli ZC. A retrospective analysis of submissions,
acceptance rate, open peer review operations, and
prepublication bias of the multidisciplinary open access
journal. Head Face Med. 2007;3:27. doi:10.1186/1746-
160X-3-27

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/fe4b8ba267367c8f45db9c17a01bae41.png)

Association Between Number of External Peer Review Invites, Unsuccessful Invites, and Declined Reviews With Rejection of Manuscripts

**Objective** Over the past decade, several journals in Africa
have transitioned from email submission systems to web-
based platforms for manuscript submission and editorial
processes.1
Previous reports1,2 assessing experience with these
systems have revealed conflicting feedback from authors and
editorial teams, with varying reports of user friendliness.
Notably, a letter to the editor published in Nature3
stated that
such systems “make authors do all the work.” Over time,
online submission systems have sought to have in-built
programs to enable automation of the submission process to
ease the authors’ experience. The Annals of African Surgery
uses the submission prefill feature in its submission system
for the aforementioned purpose.

**Design** This study was cross-sectional. To establish efficacy
and gather authors’ experience of the online submission
system and the submission prefill (ie, the system that
automatically extracts and populates submission fields for
authors from an uploaded manuscript text file), authors
submitting articles to the Annals of African Surgery were
surveyed between September and December 2021. This
survey was delivered to authors using the manuscript
submission system, during the submission process, and at the
end of the editorial workflow after the final decision was
made.

**Results** A total of 44 authors completed the survey on
submission prefill, and 42 completed the survey on the
manuscript submission system. Because the surveys were embedded as part of the submission processes, the response
rate was 100%. Overall, the responses to the question of
whether the submission prefill was effective in reducing the
workload were divided evenly (50.0% agree vs 50.0%
neutral), but there was overwhelmingly positive feedback on
the effectiveness of manuscript submission system in terms of
its ease of use (88.1%) and in obtaining the status of the
manuscript (92.9%). The survey questions and their
respective responses are as outlined in **Table 80**.

![](https://assets.underline.io/uploads/markdown_image/1/image/e973f191539223b5f0e1324aee8ab981.png)

**Conclusions** There was overall positive feedback regarding
authors’ interaction with the web-based online submission
interface. Feedback from authors’ experience on whether
submission prefills reduce submission labor, however, still
remains divided. Measures taken to improve the prefill’s
capacity to extract manuscript details with better accuracy
would, hence, enhance authors’ experience.

**References**

1. Ware M. Online submission and peer-review systems.
Learned Publishing. 2005;18(4):245-250.
doi:10.1087/095315105774648771
2. Ware M. Authors say that they prefer online submission.
Nature. 2005;434(7033):559. doi:10.1038/434559d
3. Moore JP. Online submission makes authors do all the
work. Nature. 2005;433(7028):800. doi:10.1038/433800d

**Conflict of Interest Disclosures** James Kigera is a member of
the Peer Review Congress Advisory Board but was not involved in
the review or decision for this abstract. No other disclosures were
reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/90eea1861b5ade470345f3fae96381f8.png)



Authors' General Experiences With Submitting Manuscripts and With Submission Prefill to a Manuscript Submission System for the Annals of African Surgery

**Objective** The objective of this research was to assess
whether an editorial workflow wherein submissions are
handled solely by dedicated protocol editors reduces the
average time from initial submission to a final decision
compared with standard peer review and whether this
reduction differs by protocol formatting (structured or
unstructured).

**Design** A cross-sectional study of the workflow timing for
protocol submissions to Trials was conducted. Type A
protocols demonstrate prior peer review as part of their
funding process by providing previous comments and
self-declaring this prior review in the submission process.
Type A protocols are handled by protocol editors as the sole
reviewers, whereas type B protocols are handled by other
editors and require full peer review (ie, 2 to 3 reviewers). The
workflow data stored in the Trials manuscript submission
and review system were extracted to compare the timing from
initial submission to first and final decisions for all protocols,
separated by type, before and after protocol editors were
implemented (January to December 2019 and January 2020
to November 2021, respectively). The timing was also
extracted for submissions that were formatted as
unstructured protocols with an accompanying SPIRIT
checklist or following the Structured Protocol Template
recommended by Trials. The workflow timing results are
descriptively summarized.

**Results** From January 2020 to November 2021, 1114 type A
(360 [32%]) and type B (754 [68%]) trial protocols were
submitted to Trials. Compared with type B protocols, 143 of
which (19%) used a structured template, 137 type A protocols
(38%) followed a structured format, possibly owing to self-selection or prior reviews leading to a more structured
approach in submission. A timeline of the workflow for each
protocol type by format as well as overall for each period is
shown in **Figure 31**. Overall, type A protocols had a mean of
118 days to final decision (from initial submission), whereas
type B protocols had a mean of 165 days to final decision.
Overall, across protocols in the period after protocol editors
were implemented, the time to final decision was reduced 30
days from the preceding year: 146 days (January 2020 to
November 2021) versus 176 days (January to December
2019). Among type A protocols, a structured template format
reduced the time to final decision compared with
unstructured—an association that was not seen in type B
protocols, possibly because multiple reviewers were required.

![](https://assets.underline.io/uploads/markdown_image/1/image/41a674ef8c75bec03869985bec7d5311.png)

**Conclusions** The use of dedicated editors to handle trial
protocols that had already undergone peer review was
associated with fewer days to final decision compared with
protocols that were handled by other editors and required full
peer review. Future work should examine the quality of peer
review in both workflows to determine whether use of
dedicated reviewers improves the quality of published works
as well as the time to publication.

**Conflict of Interest Disclosure**s Riaz Qureshi, Kirsty Loudon,
and Alexander Gough are protocol editors of Trials, BMC. Shaun
Treweek and Tianjing Li are editors in chief of Trials, BMC.

**Acknowledgments** We would like to thank Eleanor Cox, Bex
Chang, and Krishna Vairamani for their work as editorial staff and
for managing the workflow of peer review for Trials.

![](https://assets.underline.io/uploads/markdown_image/1/image/ce8b2b25aa2173d0019073c47efcd780.png)



Assessment of Use of Dedicated Editors for Handling and Reviewing Manuscripts With Previously Obtained Peer Reviews

**Objective** Peer review plays a fundamental role in scholarly
publishing, but its legitimacy has been increasingly
questioned. A growing literature discusses how reviewers’
demographic characteristics and biases might lead to
disparities in research dissemination.1,2 Because the extent to
which reviewers are able to determine the outcomes for
papers may vary, it is important to look at the relationship
between reviewers’ recommendations and editors’ decision-
making. However, reviewer recommendations are often
embedded in the text of the review. This work proposes a
method for automatic detection of recommendations based
on review text.

**Design** The automatic classification used a rule-based
algorithm that searched for the presence of 1 or more phrases
that signal the reviewer’s recommendation: accept, minor
revision, major revision, or reject categories, as defined on the
hand-coding process. The algorithm considered the different
combinations of signal phrases to define the outcome. The list
of signal phrases was iteratively built on 3 rounds of hand-
coding and fuzzy matching sentences, while the combinations
were defined to maximize the precision, on the hand-coded
cases. This study used Publons’ data set, which contained
3,310,791 reviews from 25,934 journals; while 600 cases were
hand-coded, a subset of 200 reviews was used to evaluate the
performance. The gender of reviewers was inferred by
matching first and last names to curated lists of country-
specific gendered names, including the US Census.3

**Results** The overall accuracy on the test was that 81% of
assigned recommendations were correct according to hand
coding (n = 149). Since the inclusion of additional phrases is
associated with lowered accuracy, this might indicate an
upper bound in our experiment, given the limits of the
current data and the idiosyncrasies of peer review language.
Nonetheless, the algorithm’s accuracy was comparable to the
rate of agreement between human hand coders (n = 60
[88%]). Over the full data, 14.3% of reviews were assigned a
recommendation by this method (n = 473,443). This was
comparable to the hand-coded identification of 18.3% of
reviews containing an explicit recommendation (n = 399).
From these results, we concluded that the inclusion of an
explicit recommendation remains relatively uncommon in
peer review, with the majority of peer reviewers leaving a final
decision on the manuscript as the responsibility of the editor,
but there was large variation between journals. Initial results
nonetheless showed gender differences in reviewing behavior,
with higher retrieval rates associated with reviewers who
identified as men.

**Conclusions** This work is among the first benchmarks for
automatic classification of review recommendations on a
large-scale, cross-domain database. Though preliminary, it
paves the way for future developments, including studies of potential biases and inequalities in scholarly publishing
through examination of the relationship between reviewer
characteristics and review outcomes.

**References**

1. Lee CJ, Sugimoto CR, Zhang G, Cronin B. Bias in peer
review. J Am Soc Inf Sci Tec. 2013;64(1):2-17. doi:10.1002/
asi.22784
2. Sun M, Barry Danfa J, Teplitskiy M. Does double-blind
peer review reduce bias? evidence from a top computer
science conference. J Am Soc Inf Sci Tec. 2022;73(6):811-819.
doi:10.1002/asi.24582
3. Larivière V, Ni C, Gingras Y, Cronin B, Sugimoto CR.
Bibliometrics: global gender disparities in science. Nature.
2013;504:211-213. doi:10.1038/504211a

**Conflict of Interest Disclosures** None reported.

**Funding/Support** This work was supported by the Doctoral
Training Unit on Data-Driven Computational Modelling and
Applications (DRIVEN), which is funded by the Luxembourg
National Research Fund under the PRIDE programme
(PRIDE17/12252781).

**Acknowledgment** We thank Brad Demarest and Chaoqun Ni for
contributions to an earlier phase of the project. We also thank the
team at Publons who implemented the gender inference algorithm
on their data before giving us access.

**AdditionalInformation** Diego Kozlowski is a co–corresponding
author.

![](https://assets.underline.io/uploads/markdown_image/1/image/3aed057350f3812bef4e49ec6d5137b7.png)

Automatic Classification of Peer Review Recommendation

**Objective** It is possible that most published research
findings are false.1
Effective peer review should detect the
study error and the mental model error. The study error can
be detected by traditional epidemiological/statistical analysis
and exists at the level of the study. The mental model error
exists at the level of underlying structural intellectual
knowledge and beliefs,2
in which accuracy provides the prior
probability and thus substantially affects the posterior
probability of the study.

**Design** The peer review process of The BMJ was assessed for
an inclusive sample of articles published with peer review
cycles (ie, editor and reviewer and author comments and
decision letters) in the research section in July 2021. Likert 5-point scales were used. The peer-review process for each
article was rated for the awareness within the process to
detect and manage study error and mental model error, with
5 representing high estimated awareness and efficacy. For
mental model error detection, higher Likert scores were
accorded in peer review cycles containing the following 4
elements: (1) active criticism of the underlying mental model
(possible errors in diagnostic categories, underlying beliefs of
therapy mechanisms, and the pathological appropriateness of
outcome measures) to estimate the final study posterior
probability; (2) awareness that studies produce contingent
fact claims and that intellectual validity requires analysis of
this contingency; (3) awareness that key patient outcomes
must exist outside the mental model to minimize outcome
contamination by the model and circular reasoning within the
model (eg, use of all-cause morbidity and mortality); and (4)
awareness that clinicians are most interested in the
probability of the mental model because they use the model
daily to inform numerous data decisions made in clinical
practice (eg, decisions regarding history, physical
examination, differential diagnosis, result, treatment
planning, and outcome assessments data). A single ranking
was determined for the totality of the review process for each
article provided by editors and reviewers and author
responses. Result populations of study error and mental
model error scores were compared to assess the separation of
the distributions using the Mann-Whitney test.

**Results** Forty-five peer reviewers produced 21 rounds of
review for 12 articles (**Table 81**). The study errormedian
score was 4.5, and the mental model error median score was 2
with the Mann-Whitney rejection of equal distributions at
P < .05.

![](https://assets.underline.io/uploads/markdown_image/1/image/b78047c705cc4da503421f1f0479a5de.png)

**Conclusions** This preliminary study found The BMJ peer
review process to be effective at detecting and managing
study errors but much less aware of mental model error
problems. Given that the underlying mental model probability may affect each study’s posterior probability,
excellent study internal validity may not secure a strong
warrant for clinical applicability. Peer review that detects and
manages both study error and mental model error may
improve clinical study applicability. Preliminary discussions
and later guideline formation may help correct this deficiency.

**References**

1. Ioannidis JP. Why most published research findings are
false. PLoS Med. 2005;2(8):e124.
2. Holland JH, Holyoak KJ, Nisbett RE, Thagard PR.
Induction: Processes of Inference, Learning, and Discovery.
Massachusetts Institute of Technology; 1986.

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/3b47aecf1afe0d03ad7af9ca6f7ba06b.png)

Assessment of Errors in Peer Reviews Published With Articles in The BMJ | VIDEO

**Objective** To study the manuscripts submitted to the
Journal of Clinical and Diagnostic Research (JCDR) that
were withdrawn or remained unattended by the authors after
receiving the external peer reviewer’s decision, and to further
evaluate the authors’ characteristics and reasons for
withdrawal stated in the correspondence.

**Design** JCDR is a broad-specialty, English-language,
peer-reviewed, monthly journal with no article processing
charges. This cross-sectional study was conducted on
manuscripts that were withdrawn or not followed up (ie,
abandoned) by authors between January 1, 2018, and
December 31, 2021, after posting the external peer review
report requesting revision. The manuscripts reported clinical
trials, observational studies, analytical studies, case reports,
literature reviews, systemic reviews, and meta-analyses. The
following data were recorded: dates of submission and the
email asking for a withdrawal, designation of the
corresponding author, whether a department head or
professor was among the authors, and the reason specified in
the emails. As a policy at JCDR, all the manuscripts that are
under process are considered withdrawn if there is no reply
from the author for more than 4 months since the last
correspondence.

**Results** There were 1080 manuscripts that were withdrawn
(n = 150) or abandoned (n = 930) during the 4-year study
period. Among these 1080 manuscripts, only 150 (13.8%) that
had an email from the corresponding author requesting a
withdrawal were further analyzed in the study. Of these 150
manuscripts, one-third (51 manuscripts [34.0%]) had a
professor as the corresponding author, whereas 44 (29.3%) had a department head or professor among the coauthors.
Thirteen manuscripts (8.6%) were withdrawn within a month
of the submission despite a quick response from the journal,
whereas 79 (52.6%) were withdrawn after more than 3
months with 1 or more revisions. Sixty-five emails (43.3%)
did not mention any specific reason for withdrawal, 35
(23.3%) stated that the authors were not willing to or unable
to revise the manuscripts as per the reviewers’ feedback, and
17 emails (11.3%) stated financial constraints as the reason for
withdrawal.

**Conclusions** It is a waste of journal resources and peer
reviewers’ labor when manuscripts are abandoned or
withdrawn. As a mark of good conduct, authors are expected
to have forthright communication with journals. If strong
reasons exist, authors may withdraw their manuscripts, but
clear communication is welcome. Notably, experienced
academicians (professors) also share the burden of such
misconduct. The effect of such withdrawals is harder for
journals that do not charge up-front processing fees. Along
with rising inflation, such withdrawals add to the economic
burden on a stand-alone journal.

**Conflict of Interest Disclosures** None reported

![](https://assets.underline.io/uploads/markdown_image/1/image/4b3514917c9ea492117196a7564af985.png)

Assessment of Withdrawal of Manuscripts Submitted to the Journal of Clinical and Diagnostic Research | VIDEO

**Objective** Grant peer review relies on scientists’ ability to
evaluate research proposals’ quality. Such judgments are
sometimes beyond reviewers’ discriminatory power, leading
to reliance on subjective biases, including preferences for
lower-risk, incremental projects.1,2 However, reviewers’
research risk tolerance during grant peer review has not been
well studied.

**Design** In late 2020 to early 2021, participants were recruited
through email invitation using the American Institute of
Biological Sciences’ reviewer databases and the National
Institutes of Health’s panel rosters. A cross-sectional,
prospective study was conducted of peer reviewers’ numeric
evaluations of mock primary reviewers’ comments of a
theoretical research proposal in which the level and sources of
risks and weaknesses were manipulated. Specifically, there
were 4 scenarios, with risks associated with either the
investigator (limited experience in leading independent
research; n = 199), the approach (lacking pilot studies; n =
205), both investigator and approach (n = 201), or neither
(control; n = 605). Each participant evaluated the control and
1 randomly chosen manipulated risk scenario (the order of
presentation was randomized). Risk tolerance was measured
by the 13-item Openness to Experience scale from the NEO
Five-Factor Inventory 3.3

**Results** Violin plots of overall and criteria scoring data from
605 participants (lower scores are more positive) showed that
the experimental manipulation was effective: Scenarios with
manipulated risk received more mediocre overall scores and
more dispersed scoring distributions than the control
scenario (**Figure 32**). Ordinal logistic regression models
(Nagelkerke R2
= 0.78) suggest that the manipulated risks in
these scenarios (compared with control) were strongly
associated with participants’ overall scores, with odds ratios
of 15.0 (95% CI, 3.4-66.0) for the investigator risk scenario,
21.7 (95% CI, 5.5-85.4) for the approach risk scenario, and
36.2 (95% CI, 8.5-154.4) for the scenario with both risks. The
approach criterion score was associated with the overall score
for all scenarios; significance and innovation criteria scores
had stronger associations with overall scores in the control
scenario. Openness to experience was not associated with
scores, but differing levels of reviewer-reported scoring
leniency were observed.

![](https://assets.underline.io/uploads/markdown_image/1/image/67d3d55bc212644eb34c166fd051a126.png)

**Conclusions** These data suggest that the evaluation of risks
dominates reviewers’ evaluation of research proposals and is
an important source of interreviewer variability. A weakness
is that this study did not involve actual grant proposals.
Training peer reviewers to consider risks as potential assets
(eg, innovation) may represent an area of useful intervention
for science advancement.

**References**
1. Lamont M. How Professors Think: Inside the Curious
World of Academic Judgment. Harvard University Press;
2009.
2. Luukkonen T. Conservatism and risk-taking in peer review:
emerging ERC practices. Res Eval. 2012;21(1):48-60.
doi:10.1093/reseval/rvs001
3. McCrae RR, Costa PT. NEO Inventories: Professional
Manual. PAR, Inc; 2010.

**Conflict of Interest Disclosures** None reported.

**Funding/Support** This work was supported by the Social,
Behavioral, and Economic Sciences Office of Multidisciplinary
Activities at the National Science Foundation under the standard
grant award numbers 1951132 and 1951251.

![](https://assets.underline.io/uploads/markdown_image/1/image/b3a5deef3c47fc2752770a69e77ae2bd.png)


Assessment of Grant Peer Reviewers' Tolerance for Risk in Research Proposals

**Objective** PubMed is a popular platform for accessing
biomedical research.1
When tagged correctly, text-based and
concise plain language summaries (PLSs) hosted on PubMed
can maximize discoverability by a broader audience. This
function was introduced in 2019 and allows retrospective
tagging of pre-2019 records. Open access (OA) publishing
also enhances discoverability, which increases publication
accessibility and usage.2
The aim was to (1) determine the
proportion of PubMed records correctly using the PLS tag
and reasons for incorrect usage and (2) establish the OA
status of journals publishing PLSs on PubMed.

**Design** The entire PubMed database was downloaded
(February 9, 2022) and searched for PLSs indexed with an
XML <plain-language-summary> tag in the Other Abstract
field. Records were deduplicated, and incorrectly tagged PLSs
were programmatically excluded for incorrect tag usage (ie,
non-PLS content) and confirmed with manual spot checks.
Remaining PLSs were categorized by journal and assessed for
overall OA status using Journal Selector (Sylogent LLC) or
information on journal websites for those PLSs not indexed
on Journal Selector.
	
**Results** There were 3217 records identified with an XML
<plain-language-summary> tag in the Other Abstract field, of
which just over half (1644 [51.10%]) were published in 2021
(annual prevalence of 0.09% [1644 of 1,769,389]). Of the 3217
records, there were 470 (14.61%) with incorrect tag usage.
Categories of incorrect usage included non-English scientific
abstract (137 records [4.26%]); duplication of or a greater
than 90% similarity score with the scientific abstract in the
Abstract field (32 [0.99%]); absence of a scientific abstract in
the Abstract field (99 [3.08%]); other non-PLS content
(including URLs, novelty statements, and article highlights)
(197 [6.12%]); and no content (5 [0.16%]). In addition to
these 470 excluded records, there were 124 records using the
<plain-language-summary> tag to index both non-English
scientific abstracts and English PLSs. Of 105 journals correctly using the <plain-language-summary> tag for PLSs,
30 (28.57%) were full or gold OA journals, 75 journals
(71.43%) offered OA options, and none were closed or
subscription-only journals.

**Conclusions** Despite the use of the <plain-language-
summary> tag increasing over time,3
records using this tag
represent a minority of all PubMed records, and the tag is
used incorrectly for several reasons. There is an unmet need
for explicit guidance on both the processes of indexing and
the correct usage of the <plain-language-summary> tag,
which could help improve correct tagging and uptake. Of
note, all PLSs published on PubMed to date come from
journals allowing for OA publishing, which aids
discoverability and accessibility. Limitations of this analysis
include lack of PLS quality assessment, small sample size
largely owing to low publisher uptake and tagging, and
confounding effects of OA on discoverability. Ultimately,
these findings highlight an opportunity for journals to
increase the impact of their content and reach a broader
audience.
	
**References**
	
1. Williamson PO, Minter CIJ. Exploring PubMed as a reliable
resource for scholarly communications services. J Med Libr
Assoc. 2019;107(1):16-29. doi:10.5195/jmla.2019.433
2. Li H, Liu L, Wang X. The open access effect in social media
exposure of scholarly articles: a matched-pair analysis. J
Informetrics. 2021;15(3):101154. doi:10.1016/j.
joi.2021.101154
3. Gattrell W, Wager K, Sheikh N, Chisholm A. Prevalence
and characteristics of plain language summaries indexed in
PubMed: original abstracts from the 2022 European Meeting
of ISMPP. Curr Med Res Opin. 2022;38(suppl 1):25-45.
	
**Conflict of Interest Disclosures** At the time of abstract
development, William Gattrell was an employee and shareholder
of or held stock or stock options in Ipsen, Milton Park, UK, and is
now an employee of Bristol Myers Squibb, Uxbridge, UK. Valérie
Philippon was an employee and shareholder of or held stock or stock
options in Takeda Development Center Americas Inc, Cambridge,
MA, USA, and is now an employee of UCB, Cambridge, MA, USA.
All other authors are employees of their respective affiliations.
Slávka Baróniková, Namit Ghildyal, Radha Narayan, and Catherine
Skobe may be shareholders or hold stock or stock options for their
respective affiliations.
	
**Funding/Support ** This work was produced by representatives
of Open Pharma. Open Pharma is a multisponsor collaboration
facilitated by Oxford PharmaGenesis Ltd and received sponsorship
funding from Alexion Pharmaceuticals Inc, AstraZeneca
Pharmaceuticals LP, Boehringer Ingelheim International GmbH,
ER Squibb & Sons LLC, F. Hoffmann-La Roche AG, Galápagos
NV, Gilead Sciences Inc, GlaxoSmithKline Biologicals SA, Ipsen
Biopharm Ltd, Janssen Global Services LLC, John Wiley & Sons Ltd, Novartis Pharma AG, Novo Nordisk A/S, Pfizer Inc, Takeda Development Center Americas Inc, and UCB Biopharma SRL.
	
**Role of the Funder/Sponsor** This abstract represents the work
of the individual authors and not of their respective affiliations.
The organizations providing sponsorship funding to Open Pharma
were not involved in the design and conduct of the study; collection,
management, analysis, and interpretation of the data; preparation,
review, or approval of the abstract; or decision to submit the
abstract for presentation.
	
**Acknowledgment** Editorial support was provided by Velissaria
Vanna of Oxford PharmaGenesis, Oxford, UK.

![](https://assets.underline.io/uploads/markdown_image/1/image/9de3546ad33257bcb15eb46613cbd887.png)

Conflicts of Interest in Systematic Reviews on Methylphenidate for Attention-Deficit Disorder

Next from Peer Review Congress 2022

Qualitative Assessment of an Antiracism Editorial Internship Program for Early Career Underrepresented Scholars at Teaching and Learning in Medicine

Similar lecture

Physiology of human voice production revealed by in-vivo experiments

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES