United States

**Objective** Editorial decisions can depend on factors, like the
timing of submissions or the matching of editors and
reviewers, that are independent of the quality of the work.
This analysis investigates associations of these and other
external factors with editorial outcomes at eLife, a major
biomedical journal.

**Design** This study analyzed whether timing of submission
(weekend vs weekday) was associated with the decision to
send manuscripts for external review among senior editors,
who can desk-reject submissions; compared peer review
manuscript ratings by reviewer volume; and assessed whether
submission time of month or year, preceding decisions to
reject or review a manuscript, and matching of reviewer to
manuscript specialty influenced editorial decisions. Data were
analyzed by single variable regression.

**Results** Between January 2016 and December 2018 eLife
received 23,190 total submissions, 6498 of which were sent
for review. Among senior editors (n = 65), proportions of
manuscripts sent for external review ranged from 9.6% to
49.3% and were statistically significantly lower on weekends
(mean, 24% [SD, 1.3%]) than on weekdays (mean, 29% [SD,
0.6%]; P &lt; .001), an association observed for most senior
editors. Average peer reviewer rating (range, 0-1) increased
with volume category: mean of 0.453 (SD, 0.003) for 1 to 5
submission reviews; 0.463 (SD, 0.008) for 6 to 10
submissions; and 0.472 (SD, 0.007) for 11 or more
submissions, and reviewers’ ratings increased with successive
reviews. In a nonquantitative inspection of submission and
decision trends, submission time of month or year, preceding
decisions to reject or review a manuscript, and matching of
reviewer to manuscript specialty did not appear to influence
editorial decisions.

**Conclusions** This study found a statistically significant
association between timing of submission during the week
and editorial decisions. Peer reviewer ratings increased with
review volume. Submission time of month or year, preceding
decisions to reject or review a manuscript, and matching of reviewer to manuscript specialty did not appear to influence
editorial decisions.

**Conflict of Interest Disclosures** None reported.

**Funding/Support** This research is supported by National Science
Foundation SCISIPBIO (A Science of Science Policy Approach to
Analyzing and Innovating the Biomedical Research Enterprise)
project 2022435.

**Additional Information** We thank Andy Collings and James
Gilbert from eLife for sharing the data and providing explanations
and eLife’s editorial leadership team for input and suggestions.

![](https://assets.underline.io/uploads/markdown_image/1/image/7aad5dd3d2641b2ae252900ca6469e10.png)

Peer Review Congress 2022

Analysis of Timing of Manuscript Submissions and Assignment of Editors and Reviewers on Editorial Decisions at eLife

peer review process and models

editorial and peer review process

artificial intelligence


![](https://assets.underline.io/uploads/markdown_image/1/image/2e10c515564e9ec3287ee4c664011c20.png)

The energy cost of storing and moving data is enormous, and is growing at such a rapid pace that within a few years the power demand may overwhelm some developed nations. Non-volatile memory (NVM) and neuromorphic computing are enabling critical advances in this area, but there are many scientific and technological challenges. This conference will cover the materials challenges of different memory technologies, from the fundamental processes and mechanisms, to industry perspectives on materials specifications and manufacturing.

Materials Challenges for Memory

**Objective** Editorial decisions can depend on factors, like the
timing of submissions or the matching of editors and
reviewers, that are independent of the quality of the work.
This analysis investigates associations of these and other
external factors with editorial outcomes at eLife, a major
biomedical journal.

**Design** This study analyzed whether timing of submission
(weekend vs weekday) was associated with the decision to
send manuscripts for external review among senior editors,
who can desk-reject submissions; compared peer review
manuscript ratings by reviewer volume; and assessed whether
submission time of month or year, preceding decisions to
reject or review a manuscript, and matching of reviewer to
manuscript specialty influenced editorial decisions. Data were
analyzed by single variable regression.

**Results** Between January 2016 and December 2018 eLife
received 23,190 total submissions, 6498 of which were sent
for review. Among senior editors (n = 65), proportions of
manuscripts sent for external review ranged from 9.6% to
49.3% and were statistically significantly lower on weekends
(mean, 24% [SD, 1.3%]) than on weekdays (mean, 29% [SD,
0.6%]; P < .001), an association observed for most senior
editors. Average peer reviewer rating (range, 0-1) increased
with volume category: mean of 0.453 (SD, 0.003) for 1 to 5
submission reviews; 0.463 (SD, 0.008) for 6 to 10
submissions; and 0.472 (SD, 0.007) for 11 or more
submissions, and reviewers’ ratings increased with successive
reviews. In a nonquantitative inspection of submission and
decision trends, submission time of month or year, preceding
decisions to reject or review a manuscript, and matching of
reviewer to manuscript specialty did not appear to influence
editorial decisions.

**Conclusions** This study found a statistically significant
association between timing of submission during the week
and editorial decisions. Peer reviewer ratings increased with
review volume. Submission time of month or year, preceding
decisions to reject or review a manuscript, and matching of reviewer to manuscript specialty did not appear to influence
editorial decisions.

**Conflict of Interest Disclosures** None reported.

**Funding/Support** This research is supported by National Science
Foundation SCISIPBIO (A Science of Science Policy Approach to
Analyzing and Innovating the Biomedical Research Enterprise)
project 2022435.

**Additional Information** We thank Andy Collings and James
Gilbert from eLife for sharing the data and providing explanations
and eLife’s editorial leadership team for input and suggestions.

![](https://assets.underline.io/uploads/markdown_image/1/image/7aad5dd3d2641b2ae252900ca6469e10.png)

poster

Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.
 
 **Welcome!**
 
 The JAMA Network, The BMJ, and METRICS welcome you to the Ninth International Congress on Peer Review and Scientific Publication. We have continued our efforts to broaden the scope of the Congress to all aspects of peer review and publication—from funding to postpublication—and to all sciences.
 
 We will have 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination. There are 50 plenary session research presentations and 5 plenary invited talks. Each plenary session research presentation will be followed by equal time for discussion and questions from the audience. In addition, there are 125 posters. All posters, including virtual posters, are available to view throughout the meeting. In-person posters will also be presented at designated times on September 9 and 10. If you are attending virtually, please visit the individual poster pages and ask your questions in the chat box. See the link to Posters on the left.
 
 We hope you will take an active part in the program, as we depend on your participation in the discussion sessions to make the Congress a success.

 [Peer Review Congress Organizers, Directors, and Advisory Board](https://peerreviewcongress.org/organizers-and-advisory-board/)
 
 [Health and Safety Policies](https://web.cvent.com/event/1b505794-6a0b-428c-a44c-6c46c4c50e99/websitePage:152962bb-4567-4561-9d81-21ebfa97f487)

![](https://assets.underline.io/uploads/markdown_image/1/image/7c9f0143dee383780cc4670269ade132.png)

A PDF of the schedule is available [here](https://drive.google.com/file/d/1-Jd191W-nxbWcN1prdX5o9nlSfeo9_-H/view?usp=sharing) and a flipbook version of the complete program and abstracts is included below. See also the link to the interactive schedule to the left.

<iframe width="700px" height="425px" src="https://online.fliphtml5.com/ebtyf/nlfn/#p=1" frameborder="0" allowfullscreen allowtransparency></iframe>

Registration is required to attend this event, please use the URL linked below to secure your ticket. Registration will close on Friday, September 2 at 5:00 PM CT. 

[**Register Here**]( https://web.cvent.com/event/1b505794-6a0b-428c-a44c-6c46c4c50e99/summary)

Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.



**Objective** Many journals encounter difficulties obtaining
peer reviewers. However, it is unknown if this is associated
with publication outcomes.1,2 The purpose of this research
was to evaluate whether, for initial manuscript submissions
that were eventually peer reviewed, there was a significant
association between the number of external peer reviewers
sought, unsuccessful invites, and declined invites for external
peer review and the publication outcome.

**Design** This was a retrospective study of anonymized,
unique, original research submissions to the Emergency
Medicine Journal (EMJ) that received at least 1 external peer
review over a 5-year period (January 2016 to December
2020). A database of deidentified original manuscripts
submitted to EMJ during the study period was interrogated to determine if there were significant associations between the
number of unsuccessful external peer review invitations, the
number of total invitations needed, and the acceptance or
rejection of a research manuscript. Original submissions
without any external peer review were excluded. Statistical
review invitations were excluded from the data. Mann-
Whitney U test was used to assess differences between the
variables and their publication outcomes. Odds ratios (OR),
likelihood ratios (LR), and positive predictive value (PPV)
were used as measures of association for potential thresholds
for variables and publication outcomes.

**Results** There were 806 deidentified peer-reviewed original
submissions included with 85 manuscripts (10.6%) accepted
for publication during the study period (**Table 79**). The ORs
for a peer-reviewed original research submission eventually
being rejected, according to number of invitations, were 77.5
(95% CI, 24.2-248.2; P < .001; (LR, 21.0; 95% CI, 6.9-63.7;
PPV, 99.4%; 95% CI, 98.3-99.8) for submissions with 4 or
more invitations, 36.8 (95% CI, 9.0-150.9; P < .001; LR, 20.0;
95% CI, 5.1-78.8; PPV, 99.4%; 95% CI, 97.7-99.9) for
submissions with 3 or more unsuccessful external peer review
invitations, and 22.6 (95% CI, 3.1-163.8; LR, 18.0; 95% CI,
2.6-127.2; PPV, 99.4%; 95% CI, 95.6-99.9) for submissions
with 2 or more peer reviewers who declined review
invitations.

![](https://assets.underline.io/uploads/markdown_image/1/image/c65a6b7606fb2ba5ad006d5543dce3f1.png)

**Conclusions** The number of declined peer review invitations
and total review invitations prior to a decision were
associated with rejection of a manuscript. The wide 95% CIs
in these results could be due to the high variability of
underlying factors that could have influenced the difficulty in
getting peer reviews and their interplay with the decision to
publish. The findings of this study may also be potentially
different for different journals.3
Further research should be
done to provide further insights on specific factors that may
be associated with difficulties in getting peer reviewers.1-3

**References**

1. Tvina A, Spellecy R, Palatnik A. Bias in the peer review
process: can we do better? Obstet Gynecol. 2019;133(6):1081-
1083. doi:10.1097/AOG.0000000000003260
2. Tennant JP, Dugan JM, Graziotin D, et al. A multi-
disciplinary perspective on emergent and future innovations
in peer review. F1000Res. 2017;6:1151. doi:10.12688/
f1000research.12037.3
3. Stamm T, Meyer U, Wiesmann HP, Kleinheinz J, Cehreli
M, Cehreli ZC. A retrospective analysis of submissions,
acceptance rate, open peer review operations, and
prepublication bias of the multidisciplinary open access
journal. Head Face Med. 2007;3:27. doi:10.1186/1746-
160X-3-27

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/fe4b8ba267367c8f45db9c17a01bae41.png)

Association Between Number of External Peer Review Invites, Unsuccessful Invites, and Declined Reviews With Rejection of Manuscripts

**Objective** Over the past decade, several journals in Africa
have transitioned from email submission systems to web-
based platforms for manuscript submission and editorial
processes.1
Previous reports1,2 assessing experience with these
systems have revealed conflicting feedback from authors and
editorial teams, with varying reports of user friendliness.
Notably, a letter to the editor published in Nature3
stated that
such systems “make authors do all the work.” Over time,
online submission systems have sought to have in-built
programs to enable automation of the submission process to
ease the authors’ experience. The Annals of African Surgery
uses the submission prefill feature in its submission system
for the aforementioned purpose.

**Design** This study was cross-sectional. To establish efficacy
and gather authors’ experience of the online submission
system and the submission prefill (ie, the system that
automatically extracts and populates submission fields for
authors from an uploaded manuscript text file), authors
submitting articles to the Annals of African Surgery were
surveyed between September and December 2021. This
survey was delivered to authors using the manuscript
submission system, during the submission process, and at the
end of the editorial workflow after the final decision was
made.

**Results** A total of 44 authors completed the survey on
submission prefill, and 42 completed the survey on the
manuscript submission system. Because the surveys were embedded as part of the submission processes, the response
rate was 100%. Overall, the responses to the question of
whether the submission prefill was effective in reducing the
workload were divided evenly (50.0% agree vs 50.0%
neutral), but there was overwhelmingly positive feedback on
the effectiveness of manuscript submission system in terms of
its ease of use (88.1%) and in obtaining the status of the
manuscript (92.9%). The survey questions and their
respective responses are as outlined in **Table 80**.

![](https://assets.underline.io/uploads/markdown_image/1/image/e973f191539223b5f0e1324aee8ab981.png)

**Conclusions** There was overall positive feedback regarding
authors’ interaction with the web-based online submission
interface. Feedback from authors’ experience on whether
submission prefills reduce submission labor, however, still
remains divided. Measures taken to improve the prefill’s
capacity to extract manuscript details with better accuracy
would, hence, enhance authors’ experience.

**References**

1. Ware M. Online submission and peer-review systems.
Learned Publishing. 2005;18(4):245-250.
doi:10.1087/095315105774648771
2. Ware M. Authors say that they prefer online submission.
Nature. 2005;434(7033):559. doi:10.1038/434559d
3. Moore JP. Online submission makes authors do all the
work. Nature. 2005;433(7028):800. doi:10.1038/433800d

**Conflict of Interest Disclosures** James Kigera is a member of
the Peer Review Congress Advisory Board but was not involved in
the review or decision for this abstract. No other disclosures were
reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/90eea1861b5ade470345f3fae96381f8.png)



Authors' General Experiences With Submitting Manuscripts and With Submission Prefill to a Manuscript Submission System for the Annals of African Surgery

**Objective** The objective of this research was to assess
whether an editorial workflow wherein submissions are
handled solely by dedicated protocol editors reduces the
average time from initial submission to a final decision
compared with standard peer review and whether this
reduction differs by protocol formatting (structured or
unstructured).

**Design** A cross-sectional study of the workflow timing for
protocol submissions to Trials was conducted. Type A
protocols demonstrate prior peer review as part of their
funding process by providing previous comments and
self-declaring this prior review in the submission process.
Type A protocols are handled by protocol editors as the sole
reviewers, whereas type B protocols are handled by other
editors and require full peer review (ie, 2 to 3 reviewers). The
workflow data stored in the Trials manuscript submission
and review system were extracted to compare the timing from
initial submission to first and final decisions for all protocols,
separated by type, before and after protocol editors were
implemented (January to December 2019 and January 2020
to November 2021, respectively). The timing was also
extracted for submissions that were formatted as
unstructured protocols with an accompanying SPIRIT
checklist or following the Structured Protocol Template
recommended by Trials. The workflow timing results are
descriptively summarized.

**Results** From January 2020 to November 2021, 1114 type A
(360 [32%]) and type B (754 [68%]) trial protocols were
submitted to Trials. Compared with type B protocols, 143 of
which (19%) used a structured template, 137 type A protocols
(38%) followed a structured format, possibly owing to self-selection or prior reviews leading to a more structured
approach in submission. A timeline of the workflow for each
protocol type by format as well as overall for each period is
shown in **Figure 31**. Overall, type A protocols had a mean of
118 days to final decision (from initial submission), whereas
type B protocols had a mean of 165 days to final decision.
Overall, across protocols in the period after protocol editors
were implemented, the time to final decision was reduced 30
days from the preceding year: 146 days (January 2020 to
November 2021) versus 176 days (January to December
2019). Among type A protocols, a structured template format
reduced the time to final decision compared with
unstructured—an association that was not seen in type B
protocols, possibly because multiple reviewers were required.

![](https://assets.underline.io/uploads/markdown_image/1/image/41a674ef8c75bec03869985bec7d5311.png)

**Conclusions** The use of dedicated editors to handle trial
protocols that had already undergone peer review was
associated with fewer days to final decision compared with
protocols that were handled by other editors and required full
peer review. Future work should examine the quality of peer
review in both workflows to determine whether use of
dedicated reviewers improves the quality of published works
as well as the time to publication.

**Conflict of Interest Disclosure**s Riaz Qureshi, Kirsty Loudon,
and Alexander Gough are protocol editors of Trials, BMC. Shaun
Treweek and Tianjing Li are editors in chief of Trials, BMC.

**Acknowledgments** We would like to thank Eleanor Cox, Bex
Chang, and Krishna Vairamani for their work as editorial staff and
for managing the workflow of peer review for Trials.

![](https://assets.underline.io/uploads/markdown_image/1/image/ce8b2b25aa2173d0019073c47efcd780.png)



Assessment of Use of Dedicated Editors for Handling and Reviewing Manuscripts With Previously Obtained Peer Reviews

**Objective** Peer review plays a fundamental role in scholarly
publishing, but its legitimacy has been increasingly
questioned. A growing literature discusses how reviewers’
demographic characteristics and biases might lead to
disparities in research dissemination.1,2 Because the extent to
which reviewers are able to determine the outcomes for
papers may vary, it is important to look at the relationship
between reviewers’ recommendations and editors’ decision-
making. However, reviewer recommendations are often
embedded in the text of the review. This work proposes a
method for automatic detection of recommendations based
on review text.

**Design** The automatic classification used a rule-based
algorithm that searched for the presence of 1 or more phrases
that signal the reviewer’s recommendation: accept, minor
revision, major revision, or reject categories, as defined on the
hand-coding process. The algorithm considered the different
combinations of signal phrases to define the outcome. The list
of signal phrases was iteratively built on 3 rounds of hand-
coding and fuzzy matching sentences, while the combinations
were defined to maximize the precision, on the hand-coded
cases. This study used Publons’ data set, which contained
3,310,791 reviews from 25,934 journals; while 600 cases were
hand-coded, a subset of 200 reviews was used to evaluate the
performance. The gender of reviewers was inferred by
matching first and last names to curated lists of country-
specific gendered names, including the US Census.3

**Results** The overall accuracy on the test was that 81% of
assigned recommendations were correct according to hand
coding (n = 149). Since the inclusion of additional phrases is
associated with lowered accuracy, this might indicate an
upper bound in our experiment, given the limits of the
current data and the idiosyncrasies of peer review language.
Nonetheless, the algorithm’s accuracy was comparable to the
rate of agreement between human hand coders (n = 60
[88%]). Over the full data, 14.3% of reviews were assigned a
recommendation by this method (n = 473,443). This was
comparable to the hand-coded identification of 18.3% of
reviews containing an explicit recommendation (n = 399).
From these results, we concluded that the inclusion of an
explicit recommendation remains relatively uncommon in
peer review, with the majority of peer reviewers leaving a final
decision on the manuscript as the responsibility of the editor,
but there was large variation between journals. Initial results
nonetheless showed gender differences in reviewing behavior,
with higher retrieval rates associated with reviewers who
identified as men.

**Conclusions** This work is among the first benchmarks for
automatic classification of review recommendations on a
large-scale, cross-domain database. Though preliminary, it
paves the way for future developments, including studies of potential biases and inequalities in scholarly publishing
through examination of the relationship between reviewer
characteristics and review outcomes.

**References**

1. Lee CJ, Sugimoto CR, Zhang G, Cronin B. Bias in peer
review. J Am Soc Inf Sci Tec. 2013;64(1):2-17. doi:10.1002/
asi.22784
2. Sun M, Barry Danfa J, Teplitskiy M. Does double-blind
peer review reduce bias? evidence from a top computer
science conference. J Am Soc Inf Sci Tec. 2022;73(6):811-819.
doi:10.1002/asi.24582
3. Larivière V, Ni C, Gingras Y, Cronin B, Sugimoto CR.
Bibliometrics: global gender disparities in science. Nature.
2013;504:211-213. doi:10.1038/504211a

**Conflict of Interest Disclosures** None reported.

**Funding/Support** This work was supported by the Doctoral
Training Unit on Data-Driven Computational Modelling and
Applications (DRIVEN), which is funded by the Luxembourg
National Research Fund under the PRIDE programme
(PRIDE17/12252781).

**Acknowledgment** We thank Brad Demarest and Chaoqun Ni for
contributions to an earlier phase of the project. We also thank the
team at Publons who implemented the gender inference algorithm
on their data before giving us access.

**AdditionalInformation** Diego Kozlowski is a co–corresponding
author.

![](https://assets.underline.io/uploads/markdown_image/1/image/3aed057350f3812bef4e49ec6d5137b7.png)

Automatic Classification of Peer Review Recommendation

**Objective** It is possible that most published research
findings are false.1
Effective peer review should detect the
study error and the mental model error. The study error can
be detected by traditional epidemiological/statistical analysis
and exists at the level of the study. The mental model error
exists at the level of underlying structural intellectual
knowledge and beliefs,2
in which accuracy provides the prior
probability and thus substantially affects the posterior
probability of the study.

**Design** The peer review process of The BMJ was assessed for
an inclusive sample of articles published with peer review
cycles (ie, editor and reviewer and author comments and
decision letters) in the research section in July 2021. Likert 5-point scales were used. The peer-review process for each
article was rated for the awareness within the process to
detect and manage study error and mental model error, with
5 representing high estimated awareness and efficacy. For
mental model error detection, higher Likert scores were
accorded in peer review cycles containing the following 4
elements: (1) active criticism of the underlying mental model
(possible errors in diagnostic categories, underlying beliefs of
therapy mechanisms, and the pathological appropriateness of
outcome measures) to estimate the final study posterior
probability; (2) awareness that studies produce contingent
fact claims and that intellectual validity requires analysis of
this contingency; (3) awareness that key patient outcomes
must exist outside the mental model to minimize outcome
contamination by the model and circular reasoning within the
model (eg, use of all-cause morbidity and mortality); and (4)
awareness that clinicians are most interested in the
probability of the mental model because they use the model
daily to inform numerous data decisions made in clinical
practice (eg, decisions regarding history, physical
examination, differential diagnosis, result, treatment
planning, and outcome assessments data). A single ranking
was determined for the totality of the review process for each
article provided by editors and reviewers and author
responses. Result populations of study error and mental
model error scores were compared to assess the separation of
the distributions using the Mann-Whitney test.

**Results** Forty-five peer reviewers produced 21 rounds of
review for 12 articles (**Table 81**). The study errormedian
score was 4.5, and the mental model error median score was 2
with the Mann-Whitney rejection of equal distributions at
P < .05.

![](https://assets.underline.io/uploads/markdown_image/1/image/b78047c705cc4da503421f1f0479a5de.png)

**Conclusions** This preliminary study found The BMJ peer
review process to be effective at detecting and managing
study errors but much less aware of mental model error
problems. Given that the underlying mental model probability may affect each study’s posterior probability,
excellent study internal validity may not secure a strong
warrant for clinical applicability. Peer review that detects and
manages both study error and mental model error may
improve clinical study applicability. Preliminary discussions
and later guideline formation may help correct this deficiency.

**References**

1. Ioannidis JP. Why most published research findings are
false. PLoS Med. 2005;2(8):e124.
2. Holland JH, Holyoak KJ, Nisbett RE, Thagard PR.
Induction: Processes of Inference, Learning, and Discovery.
Massachusetts Institute of Technology; 1986.

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/3b47aecf1afe0d03ad7af9ca6f7ba06b.png)

Assessment of Errors in Peer Reviews Published With Articles in The BMJ | VIDEO

**Objective** To study the manuscripts submitted to the
Journal of Clinical and Diagnostic Research (JCDR) that
were withdrawn or remained unattended by the authors after
receiving the external peer reviewer’s decision, and to further
evaluate the authors’ characteristics and reasons for
withdrawal stated in the correspondence.

**Design** JCDR is a broad-specialty, English-language,
peer-reviewed, monthly journal with no article processing
charges. This cross-sectional study was conducted on
manuscripts that were withdrawn or not followed up (ie,
abandoned) by authors between January 1, 2018, and
December 31, 2021, after posting the external peer review
report requesting revision. The manuscripts reported clinical
trials, observational studies, analytical studies, case reports,
literature reviews, systemic reviews, and meta-analyses. The
following data were recorded: dates of submission and the
email asking for a withdrawal, designation of the
corresponding author, whether a department head or
professor was among the authors, and the reason specified in
the emails. As a policy at JCDR, all the manuscripts that are
under process are considered withdrawn if there is no reply
from the author for more than 4 months since the last
correspondence.

**Results** There were 1080 manuscripts that were withdrawn
(n = 150) or abandoned (n = 930) during the 4-year study
period. Among these 1080 manuscripts, only 150 (13.8%) that
had an email from the corresponding author requesting a
withdrawal were further analyzed in the study. Of these 150
manuscripts, one-third (51 manuscripts [34.0%]) had a
professor as the corresponding author, whereas 44 (29.3%) had a department head or professor among the coauthors.
Thirteen manuscripts (8.6%) were withdrawn within a month
of the submission despite a quick response from the journal,
whereas 79 (52.6%) were withdrawn after more than 3
months with 1 or more revisions. Sixty-five emails (43.3%)
did not mention any specific reason for withdrawal, 35
(23.3%) stated that the authors were not willing to or unable
to revise the manuscripts as per the reviewers’ feedback, and
17 emails (11.3%) stated financial constraints as the reason for
withdrawal.

**Conclusions** It is a waste of journal resources and peer
reviewers’ labor when manuscripts are abandoned or
withdrawn. As a mark of good conduct, authors are expected
to have forthright communication with journals. If strong
reasons exist, authors may withdraw their manuscripts, but
clear communication is welcome. Notably, experienced
academicians (professors) also share the burden of such
misconduct. The effect of such withdrawals is harder for
journals that do not charge up-front processing fees. Along
with rising inflation, such withdrawals add to the economic
burden on a stand-alone journal.

**Conflict of Interest Disclosures** None reported

![](https://assets.underline.io/uploads/markdown_image/1/image/4b3514917c9ea492117196a7564af985.png)

Assessment of Withdrawal of Manuscripts Submitted to the Journal of Clinical and Diagnostic Research | VIDEO

**Objective** Grant peer review relies on scientists’ ability to
evaluate research proposals’ quality. Such judgments are
sometimes beyond reviewers’ discriminatory power, leading
to reliance on subjective biases, including preferences for
lower-risk, incremental projects.1,2 However, reviewers’
research risk tolerance during grant peer review has not been
well studied.

**Design** In late 2020 to early 2021, participants were recruited
through email invitation using the American Institute of
Biological Sciences’ reviewer databases and the National
Institutes of Health’s panel rosters. A cross-sectional,
prospective study was conducted of peer reviewers’ numeric
evaluations of mock primary reviewers’ comments of a
theoretical research proposal in which the level and sources of
risks and weaknesses were manipulated. Specifically, there
were 4 scenarios, with risks associated with either the
investigator (limited experience in leading independent
research; n = 199), the approach (lacking pilot studies; n =
205), both investigator and approach (n = 201), or neither
(control; n = 605). Each participant evaluated the control and
1 randomly chosen manipulated risk scenario (the order of
presentation was randomized). Risk tolerance was measured
by the 13-item Openness to Experience scale from the NEO
Five-Factor Inventory 3.3

**Results** Violin plots of overall and criteria scoring data from
605 participants (lower scores are more positive) showed that
the experimental manipulation was effective: Scenarios with
manipulated risk received more mediocre overall scores and
more dispersed scoring distributions than the control
scenario (**Figure 32**). Ordinal logistic regression models
(Nagelkerke R2
= 0.78) suggest that the manipulated risks in
these scenarios (compared with control) were strongly
associated with participants’ overall scores, with odds ratios
of 15.0 (95% CI, 3.4-66.0) for the investigator risk scenario,
21.7 (95% CI, 5.5-85.4) for the approach risk scenario, and
36.2 (95% CI, 8.5-154.4) for the scenario with both risks. The
approach criterion score was associated with the overall score
for all scenarios; significance and innovation criteria scores
had stronger associations with overall scores in the control
scenario. Openness to experience was not associated with
scores, but differing levels of reviewer-reported scoring
leniency were observed.

![](https://assets.underline.io/uploads/markdown_image/1/image/67d3d55bc212644eb34c166fd051a126.png)

**Conclusions** These data suggest that the evaluation of risks
dominates reviewers’ evaluation of research proposals and is
an important source of interreviewer variability. A weakness
is that this study did not involve actual grant proposals.
Training peer reviewers to consider risks as potential assets
(eg, innovation) may represent an area of useful intervention
for science advancement.

**References**
1. Lamont M. How Professors Think: Inside the Curious
World of Academic Judgment. Harvard University Press;
2009.
2. Luukkonen T. Conservatism and risk-taking in peer review:
emerging ERC practices. Res Eval. 2012;21(1):48-60.
doi:10.1093/reseval/rvs001
3. McCrae RR, Costa PT. NEO Inventories: Professional
Manual. PAR, Inc; 2010.

**Conflict of Interest Disclosures** None reported.

**Funding/Support** This work was supported by the Social,
Behavioral, and Economic Sciences Office of Multidisciplinary
Activities at the National Science Foundation under the standard
grant award numbers 1951132 and 1951251.

![](https://assets.underline.io/uploads/markdown_image/1/image/b3a5deef3c47fc2752770a69e77ae2bd.png)


Assessment of Grant Peer Reviewers' Tolerance for Risk in Research Proposals

**Objective** PubMed is a popular platform for accessing
biomedical research.1
When tagged correctly, text-based and
concise plain language summaries (PLSs) hosted on PubMed
can maximize discoverability by a broader audience. This
function was introduced in 2019 and allows retrospective
tagging of pre-2019 records. Open access (OA) publishing
also enhances discoverability, which increases publication
accessibility and usage.2
The aim was to (1) determine the
proportion of PubMed records correctly using the PLS tag
and reasons for incorrect usage and (2) establish the OA
status of journals publishing PLSs on PubMed.

**Design** The entire PubMed database was downloaded
(February 9, 2022) and searched for PLSs indexed with an
XML <plain-language-summary> tag in the Other Abstract
field. Records were deduplicated, and incorrectly tagged PLSs
were programmatically excluded for incorrect tag usage (ie,
non-PLS content) and confirmed with manual spot checks.
Remaining PLSs were categorized by journal and assessed for
overall OA status using Journal Selector (Sylogent LLC) or
information on journal websites for those PLSs not indexed
on Journal Selector.
	
**Results** There were 3217 records identified with an XML
<plain-language-summary> tag in the Other Abstract field, of
which just over half (1644 [51.10%]) were published in 2021
(annual prevalence of 0.09% [1644 of 1,769,389]). Of the 3217
records, there were 470 (14.61%) with incorrect tag usage.
Categories of incorrect usage included non-English scientific
abstract (137 records [4.26%]); duplication of or a greater
than 90% similarity score with the scientific abstract in the
Abstract field (32 [0.99%]); absence of a scientific abstract in
the Abstract field (99 [3.08%]); other non-PLS content
(including URLs, novelty statements, and article highlights)
(197 [6.12%]); and no content (5 [0.16%]). In addition to
these 470 excluded records, there were 124 records using the
<plain-language-summary> tag to index both non-English
scientific abstracts and English PLSs. Of 105 journals correctly using the <plain-language-summary> tag for PLSs,
30 (28.57%) were full or gold OA journals, 75 journals
(71.43%) offered OA options, and none were closed or
subscription-only journals.

**Conclusions** Despite the use of the <plain-language-
summary> tag increasing over time,3
records using this tag
represent a minority of all PubMed records, and the tag is
used incorrectly for several reasons. There is an unmet need
for explicit guidance on both the processes of indexing and
the correct usage of the <plain-language-summary> tag,
which could help improve correct tagging and uptake. Of
note, all PLSs published on PubMed to date come from
journals allowing for OA publishing, which aids
discoverability and accessibility. Limitations of this analysis
include lack of PLS quality assessment, small sample size
largely owing to low publisher uptake and tagging, and
confounding effects of OA on discoverability. Ultimately,
these findings highlight an opportunity for journals to
increase the impact of their content and reach a broader
audience.
	
**References**
	
1. Williamson PO, Minter CIJ. Exploring PubMed as a reliable
resource for scholarly communications services. J Med Libr
Assoc. 2019;107(1):16-29. doi:10.5195/jmla.2019.433
2. Li H, Liu L, Wang X. The open access effect in social media
exposure of scholarly articles: a matched-pair analysis. J
Informetrics. 2021;15(3):101154. doi:10.1016/j.
joi.2021.101154
3. Gattrell W, Wager K, Sheikh N, Chisholm A. Prevalence
and characteristics of plain language summaries indexed in
PubMed: original abstracts from the 2022 European Meeting
of ISMPP. Curr Med Res Opin. 2022;38(suppl 1):25-45.
	
**Conflict of Interest Disclosures** At the time of abstract
development, William Gattrell was an employee and shareholder
of or held stock or stock options in Ipsen, Milton Park, UK, and is
now an employee of Bristol Myers Squibb, Uxbridge, UK. Valérie
Philippon was an employee and shareholder of or held stock or stock
options in Takeda Development Center Americas Inc, Cambridge,
MA, USA, and is now an employee of UCB, Cambridge, MA, USA.
All other authors are employees of their respective affiliations.
Slávka Baróniková, Namit Ghildyal, Radha Narayan, and Catherine
Skobe may be shareholders or hold stock or stock options for their
respective affiliations.
	
**Funding/Support ** This work was produced by representatives
of Open Pharma. Open Pharma is a multisponsor collaboration
facilitated by Oxford PharmaGenesis Ltd and received sponsorship
funding from Alexion Pharmaceuticals Inc, AstraZeneca
Pharmaceuticals LP, Boehringer Ingelheim International GmbH,
ER Squibb & Sons LLC, F. Hoffmann-La Roche AG, Galápagos
NV, Gilead Sciences Inc, GlaxoSmithKline Biologicals SA, Ipsen
Biopharm Ltd, Janssen Global Services LLC, John Wiley & Sons Ltd, Novartis Pharma AG, Novo Nordisk A/S, Pfizer Inc, Takeda Development Center Americas Inc, and UCB Biopharma SRL.
	
**Role of the Funder/Sponsor** This abstract represents the work
of the individual authors and not of their respective affiliations.
The organizations providing sponsorship funding to Open Pharma
were not involved in the design and conduct of the study; collection,
management, analysis, and interpretation of the data; preparation,
review, or approval of the abstract; or decision to submit the
abstract for presentation.
	
**Acknowledgment** Editorial support was provided by Velissaria
Vanna of Oxford PharmaGenesis, Oxford, UK.

![](https://assets.underline.io/uploads/markdown_image/1/image/9de3546ad33257bcb15eb46613cbd887.png)

Assessing PubMed Metatag Usage for Plain Language Summary Discoverability

**Objective** Plagiarism is a main concern regarding
misconduct in scientific publication. Therefore, it is
interesting to gauge and characterize research interests on the
topic, particularly in countries with a less established research
tradition. The present study updates information on
plagiarism research directions among Brazilian graduate-
level researchers.

**Design** Two databases were used for this study. The first
(CAPES database) is maintained by the country’s Ministry of
Education, registering information on all master’s/doctoral
studies in the country.1
The second (IBICT/BDTD database),
a similar database, is maintained by the country’s Ministry of
Science.2
Approximately 200,000 manuscripts are included
in these databases. The following keywords were used to first
identify those manuscripts concerning plagiarism:
plagiarism, science integrity, ethics, and scientific
misconduct (in Portuguese and in English). After manuscripts
were identified, abstracts (or the full manuscripts, when
available) were read to select those studies specifically dealing
with academic plagiarism. Thus, studies on the arts (eg,
theater, music, painting), advertisements, en passant
mentions, and historical commentaries were not included in
the analysis. Items were then classified as the following3
:conceptual studies (eg, discussions about the concept or the
implications of plagiarism), development of detection
software and methods, teaching/implementation of
prevention methods, legal aspects, and case or quantitative
studies.

**Results** A total of 206 theses/dissertations were identified,
the first of which was from 1993. Of these, 119 were selected
according to the criteria above, and 1 was discarded because it
could not be located. Studies pertained to all main areas of
academia, with a large number from engineering/computer
sciences (eg, “detection”). Numbers for the past 15 years were
as follows: 2006 to 2010: 18; 2011 to 2015: 38; 2016 to 2020:
56. The most common research area was development of
detection software, with 40 of the 118 identified manuscripts.
Conceptual studies followed second (30 of 118), and prevention methods comprised 18 studies. This tendency was
still evident in the last analyzed 5 years, when detection
comprised 18 of 56 studies.
Conclusions Interest in plagiarism research seems to be
increasing in Brazil, a trend that is compatible with the
growing concern of its funding agencies with the misconduct
topic. In the past decades, the main focus moved away from
the discussion of historical events to more applied areas of
research. The development of detection software was the
most active of these areas, a trend that has been noticed
before3 and continues to this day.

**References**

1. CAPES. CAPES Dissertations Repository (in Portuguese).
CAPES/Ministry of Education. Accessed January 2022.
https://catalogodeteses.capes.gov.br
2. IBCIT/BDTD. Brazilian Dissertations Library (in
Portuguese). IBICT/BDTD/Ministry of Science. Accessed
January 2022. https://bdtd.ibict.br
3. Almeida MVR, Fontes-Pereira J. Graduate research on
plagiarism in Brazil: characterization of current status. In: 5th
World Congress on Research Integrity Abstract Book. World
Conferences on Research Integrity; 2017:94. Accessed
January 2022. https://wcrif.org/documents/41-abstract-book-5th-wcri-2017

**Conflict of Interest Disclosures** None reported.
Funding/Support This study was funded by the Coordination for
the Improvement of Higher Education Personnel–CAPES/Ministry
of Education, Brazil, whom we thank.

![](https://assets.underline.io/uploads/markdown_image/1/image/0e50d91aee4802b9055efdceae208aad.png)

Trends in Research on Plagiarism Among Brazilian Graduate-Level Studies

**Objective** The objective was to test and compare various
methods to detect text duplication in peer reviews submitted
by 2 or more reviewers.

**Design** Peer review fraud is a significant concern.1,2 A data
set of peer review comments submitted to SAGE Publishing
was analyzed to search for duplicate text, a possible sign of
fake peer review.3
Peer review comments for each article peer
reviewed by 19 SAGE Publishing journals were downloaded
from the ScholarOne peer review management system and
loaded into a Pandas DataFrame. Journals were chosen based
on the availability of data; therefore, the data set should be
considered biased. Similar comments were found using a
number of search methods, including MinHash Locality
Sensitive Hashing (MinHash LSH) for detecting near-
duplicate text strings, and Elasticsearch, a scalable graph
database combined with RapidFuzz, a fast string-comparison
library, for distinguishing similar from dissimilar comments.

**Results** Of 62,974 peer reviewer accounts used to evaluate
66,815 articles, 357 accounts (0.05%) were identified that
produced reviews with partial or fully duplicate comments.
One large cluster of 47 accounts that shared a number of
reports included a number of articles rejected because of
suspected paper mill activity. This number suggests that the
cluster of 47 accounts represented 47 fake reviewer accounts
administered by a paper mill. In total, 972 articles (1.5%) had
reviews from reviewer accounts associated with duplicate
commenting activity, and 77 articles had reviews from the 47
suspected paper mill accounts (**Figure 33**). Different search
methods identified different suspect accounts and clusters.
These searches included (1) a search for exact duplicates,
which took 16 seconds to load data into memory and less than
1 second to execute; this search found 29 accounts that had
produced similar comments, and (2) a search for similar
comments using Elasticsearch, which took 18 minutes and 29
seconds to index and 9 hours, 19 minutes, and 2 seconds to
execute; this search found 204 accounts that had produced
similar comments.

![](https://assets.underline.io/uploads/markdown_image/1/image/a6900709a7eb5867a41a4314d71c7dde.png)

**Conclusions** Efficient methods for identifying possible peer
review fraud and paper mill activity were described. The
methods should be tested on broader peer review sets and
settings. When duplication is found, the findings must be
considered in context before a judgment can be made about
whether there is misconduct.

**References**

1. Misra DP, Ravindran V, Agarwal V. Integrity of authorship
and peer review practices: challenges and opportunities for
improvement. J Korean Med Sci. 2018;33(46):e287.
doi:10.3346/jkms.2018.33.e287
2. Cohen A, Pattanaik S, Kumar P, et al. Organised crime
against the academic peer review system. Br J Clin
Pharmacol. 2016;81(6):1012-1017. doi:10.1111/bcp.12992
3. Dadkhah M, Kahani M, Borchardt G. A method for
improving the integrity of peer review. Sci Eng Ethics.
2018;24(5):1603-1610. doi:10.1007/s11948-017-9960-9

**Conflict of Interest Disclosures** None reported.

![](https://assets.underline.io/uploads/markdown_image/1/image/1f42e04c0b732163a3c84190af9419ed.png)


Analysis of Timing of Manuscript Submissions and Assignment of Editors and Reviewers on Editorial Decisions at eLife

Next from Peer Review Congress 2022

Association Between Number of External Peer Review Invites, Unsuccessful Invites, and Declined Reviews With Rejection of Manuscripts

Similar lecture

Neuromorphic computing architecture based on serially connected magnetic tunnel junctions

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES