United States

Peer Review Congress 2022

Morning Welcome: Michael Berkwits | VIDEO

Dear Delegates, 

On behalf of the organizing committee at UCF Rosen College of Hospitality Management and Elsevier, I have the greatest pleasure to welcome you to the **6th World Research Summit for Tourism and Hospitality: Resilience, Adaptability, and Innovation.** 

Though we cannot host you this year owing to Covid-19 at our beautiful resort-like campus in Orlando, Florida, we believe this virtual and live-streamed conference will provide an interactive, dynamic, and shared international platform for global academics and practitioners to discuss how to enhance research applications and knowledge management through resilience, adaptability, digital intelligence, innovation, and smart knowledge in bringing our industry and research back to normal. This summit particularly offers unique opportunities to unfold and embrace various repercussions of COVID-19 pandemic with various stakeholders, including academic researchers, industry professionals, and government officials, through dialogues on how to better utilize research outputs for the benefits of the industry, in a resilient, intelligent and innovative way. 

I hope you will agree that we have put together a wonderful program including keynote speeches, panel discussions, concurrent presentations, poster sessions, and best paper recognition and award ceremony. We hope you will take advantage of these activities and engage yourself with great learning and sharing opportunities during the two days of the Summit. Thank you for joining us and we look forward to our discussions! 

Sincerely, 
Dr. Youcheng Wang 
Dean and Preeminent Chair Professor

Please register with Elsevier to be able to enter the conference on Underline.io. If you have already registered, please login to Underline.io to be able to see full conference content. In case you do not have an Underline account, please create one. Thank you

Please login to your Underline account

TOUR 2021

This conference provides an interactive, dynamic, and shared international platform for global academics and practitioners to discuss how to enhance research applications and knowledge management through resilience, adaptability, digital intelligence, innovation, and smart knowledge in bringing our industry and research back to normal. This summit particularly offers unique opportunities to unfold and embrace various repercussions of COVID-19 pandemic with various stakeholders, including academic researchers, industry professionals, and government officials, through dialogues on how to better utilize research outputs for the benefits of the industry, in a resilient, intelligent and innovative way.

Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.
 
 **Welcome!**
 
 The JAMA Network, The BMJ, and METRICS welcome you to the Ninth International Congress on Peer Review and Scientific Publication. We have continued our efforts to broaden the scope of the Congress to all aspects of peer review and publication—from funding to postpublication—and to all sciences.
 
 We will have 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination. There are 50 plenary session research presentations and 5 plenary invited talks. Each plenary session research presentation will be followed by equal time for discussion and questions from the audience. In addition, there are 125 posters. All posters, including virtual posters, are available to view throughout the meeting. In-person posters will also be presented at designated times on September 9 and 10. If you are attending virtually, please visit the individual poster pages and ask your questions in the chat box. See the link to Posters on the left.
 
 We hope you will take an active part in the program, as we depend on your participation in the discussion sessions to make the Congress a success.

 [Peer Review Congress Organizers, Directors, and Advisory Board](https://peerreviewcongress.org/organizers-and-advisory-board/)
 
 [Health and Safety Policies](https://web.cvent.com/event/1b505794-6a0b-428c-a44c-6c46c4c50e99/websitePage:152962bb-4567-4561-9d81-21ebfa97f487)

![](https://assets.underline.io/uploads/markdown_image/1/image/7c9f0143dee383780cc4670269ade132.png)

A PDF of the schedule is available [here](https://drive.google.com/file/d/1-Jd191W-nxbWcN1prdX5o9nlSfeo9_-H/view?usp=sharing) and a flipbook version of the complete program and abstracts is included below. See also the link to the interactive schedule to the left.

<iframe width="700px" height="425px" src="https://online.fliphtml5.com/ebtyf/nlfn/#p=1" frameborder="0" allowfullscreen allowtransparency></iframe>

Registration is required to attend this event, please use the URL linked below to secure your ticket. Registration will close on Friday, September 2 at 5:00 PM CT. 

[**Register Here**]( https://web.cvent.com/event/1b505794-6a0b-428c-a44c-6c46c4c50e99/summary)

Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.



**Importance** Improving openness, rigor, and reproducibility
in research is less a technical challenge and more a social
challenge. Current practice is sustained by dysfunctional
incentives that prioritize publication over accuracy and
transparency. The consequence is unnecessary friction in
research progress. Successful culture change requires
coordinated policy, incentive, and normative changes across
stakeholders to improve research credibility and accelerate
progress.

**Observations** Rates of preregistration; sharing of data,
materials, and code; and practices that improve
reproducibility of findings are increasing over time. However,
some stakeholder groups and disciplines are making more
progress than others. There are a variety of factors that
account for why some activities are accelerating in some
disciplines and are stalled in others.

**Conclusions** A substantial challenge for effective culture
change is addressing the coordination problem in which there
are many independent stakeholders driving research rewards
and practice. These actors could be doing more
collaboratively to align incentives and rewards with core
scholarly values to accelerate discovery and advancement of
knowledge, solutions, and treatments.

**Conflict of Interest Disclosures** Brian A. Nosek is an employee
of the nonprofit Center for Open Science, which has a mission to
increase openness, integrity, and reproducibility of research.

**Funding/Support** Brian A. Nosek has received funding related
to this abstract from Arnold Ventures, Templeton Religion Trust,
Templeton World Charity Foundation, John Templeton Foundation,
Flu Lab, National Science Foundation, National Institutes of Health,
and the Defense Advanced Research Project Agency.

**Role of the Funder/Sponsor** The funders played no role in the
preparation or approval of this abstract.

![](https://assets.underline.io/uploads/markdown_image/1/image/fadef285ff2177372363abc8566014ca.png)

Improving the Research Culture to Increase Credibility of Research Findings, Brian Nosek

**Objective** Scientific journals may counter the misuse,
misreporting, and misinterpretation of statistical methods by
offering statistical guidance to authors. 1,2,3 This study assessed
the nature and prevalence of statistical guidance in topranked
journals across 22 scientific disciplines. 
 **Design** Statistical guidance from journal websites of 15
journals (top-ranked by impact factor) in each of 22 scientific
disciplines (330 journals) was extracted and classified (in
duplicate). Disagreements were resolved through discussion.
Information was recorded on whether journals had dedicated
statistical guidance sections and/or referred to guidance in
external sources. For journals that provided their own
statistical guidance, advice on each of 20 prespecified topics
was recorded. For 6 topics that were considered in advance
(based on author intuition) to be hotly debated in the
statistical literature (P values, statistical significance,
confidence intervals, effect sizes, sample size justification,
and bayesian statistics), 1 investigator classified whether the
journal indicated opposition or endorsement and whether
this was implicit or explicit. 
 **Results** Of 330 journals, 160 (48%) provided statistical
guidance and 93 (28%) had a dedicated statistical guidance
section in their author instructions (**Figure 9**, A). Statistical
guidance was most common in health and life sciences
journals. Notably, all 15 clinical medicine journals offered
some statistical guidance. In 2 disciplines (computer science
and mathematics), no journals offered any statistical
guidance. Some journals shared the same publisher-level
guidance, including 31 Nature Research journals (9%), 12 Cell
Press journals (4%), and 2 Frontiers Media journals (0.6%). A
total of 137 journals (42%) referred authors to statistical
guidance in 80 individual external sources, 49 of which were
reporting guidelines (the remainder were primarily journal
articles). Among 20 prespecified statistical topics (**Figure 9**,
B), only 2 were mentioned in more than a quarter of the
journals: confidence intervals (90 [27%]) and P values (88
[27%]). Guidance on these topics was inconsistent across
journals. For 6 hotly debated topics, only 3 journals explicitly
opposed the use of statistical significance; more commonly,
journals implicitly endorsed the use of P values (77 [23%]),
statistical significance (35 [11%]), and bayesian statistics (39
[12%]) and explicitly endorsed reporting of effect sizes (62
[19%]), confidence intervals (85 [26%]), and sample size
justifications (67 [20%]). 
 **Figure 9. Statistical Guidance**
![](https://assets.underline.io/uploads/markdown_image/1/image/98f8f2c8e2f944dd86e384f9080f970b.png) 
 **Conclusions** The results of this study suggest that there are
large gaps and inconsistent coverage in the statistical
guidance provided by top-ranked journals across scientific
disciplines. Future studies should investigate whether journal
statistical guidance to authors is associated with improved
selection, use, reporting, or interpretation of statistical
analyses. 
 **References**
1. Strasak AM, Zaman Q, Marinell G, Pfeiffer K, Ulmer
H. The use of statistics in medical research: a comparison of
the New England Journal of Medicine and Nature Medicine.
Am
Statistician. 2007;61:47-55. doi:10.1198/000313007X170242 
 2. Altman DG, Gore SM, Gardner MJ, Pocock SJ.
Statistical guidelines for contributors to medical journals. Br
Med J (Clin Res Ed). 1983;286:1489-1493. 
 3. Schriger DL, Arora S, Altman DG. The content of
medical journal instructions for authors. Ann Emerg Med.
2006;48:743-749. doi:10.1016/j.annemergmed.2006.03.028 
 **Conflict of Interest Disclosures** Tom E. Hardwicke receives
funding from the European Union’s Horizon 2020 research and
innovation program under Marie Skłodowska-Curie grant 841188.
Maia Salholz-Hillel is employed as a researcher under research grants
from the German Bundesministerium für Bildung und Forschung.
Mario Malički is a co–editor in chief of Research Integrity and Peer
Review. John P. A. Ioannidis is a codirector of the Peer Review
Congress but was not involved in the review or decision of this
abstract. The Meta-Research Innovation Center at Stanford is
supported by a grant from the Laura and John Arnold Foundation,
and the Meta-Research Innovation Center Berlin is supported by a
grant from the Einstein Foundation and Stiftung Charité. No other
disclosures were reported. 
 
![](https://assets.underline.io/uploads/markdown_image/1/image/c5a3eb72752d4fda63a155ba269bdce7.png)

Statistical Guidance to Authors at Top-Ranked Journals Across 22 Scientific Disciplines

**Objective** Reporting guidelines have been available since
1994. Numerous studies have shown that adherence to
reporting guidelines is suboptimal, 1,2 raising the question of
whether a specific targeted intervention for peer reviewers
might improve reporting. The aim of this study was to
evaluate whether asking peer reviewers, via email, to check if
specific reporting guideline items were adequately reported in
the submitted manuscripts they were reviewing would
improve adherence to reporting guidelines in published
articles. 
 **Design** Two parallel-group superiority randomized
controlled trials (RCT-1 and RCT-2) using submitted
manuscripts as the unit of randomization. RCT-1 focused on
RCT protocols and how well they were reported considering
the SPIRIT (Standard Protocol Items: Recommendations for
Interventional Trials) guidelines, and RCT-2 focused on RCT
results publications and the reporting of CONSORT
(Consolidated Standards of Reporting Trials) items.
Manuscripts in both RCTs were randomized (1:1) to
intervention or control; the control group received usual
journal practice. RCT-1 included manuscripts containing RCT
protocols submitted from June 2020 to May 2021 to BMJ
Open that were sent for peer review (https://osf.io/z2hm9).
The RCT-2 trial 3 included manuscripts describing RCT
primary results, submitted from July 2019 to July 2021 to 1 of
7 journals (5 BMJ Publishing Group; 2 Public Library of
Science [PLOS]). In the intervention group (both trials), peer
reviewers received an email from the journal reminding them
to check if items were adequately reported in the manuscript.
In RCT-1, these were the 10 most important and poorly
reported SPIRIT items and for RCT-2, the 10 most important
and poorly reported CONSORT items. In both RCTs, peer
reviewers and authors were not informed of the purpose of
the study and outcome assessors were blinded. The primary
outcome was the difference in the mean proportion of
adequately reported 10 SPIRIT and 10 CONSORT items
between intervention and control in the final published
article. 
 **Results** In RCT-1, 245 manuscripts were randomized. Of
those, 178 were published (90 intervention; 88 control). A
mean proportion of 46.1% (95% CI, 41.8%-50.4%) of the 10
SPIRIT items were adequately reported in the intervention
group and 45.6% (95% CI, 41.7%-49.4%) in the control group
(mean difference, 0.5%; 95% CI, −5.2% to 6.3%) (**Figure
10**). In RCT-2, of the 511 randomized manuscripts, 243 were
published (121 intervention; 122 control). A total of 67.4%
(95% CI, 63.8%-71.1%) of the 10 CONSORT items were
adequately reported in the intervention group and 65.9%
(95% CI, 61.9%-69.9%) in the control group (mean difference,
1.5%; 95% CI, −3.8% to 6.9%) (**Figure 10**). 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/df83d52dae4829504213376704550996.png) 
 **Conclusions** Journals asking peer reviewers, via email, to check if the most important and poorly reported items are adequately reported in submitted manuscripts did not improve the reporting completeness of the final published article. 
 **References**
1.	Samaan Z, Mbuagbaw L, Kosa D, et al. A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013;6:169-188. 
 2.	Jin Y, Sanger N, Shams I, et al. Does the medical literature remain inadequately described despite having reporting guidelines for 21 years? a systematic review of reviews: an update. J Multidiscip Healthc. 2018;11:495-510. 
 3.	Speich B, Schroter S, Briel M, et al. Impact of a short version of the CONSORT checklist for peer reviewers to improve the reporting of randomised controlled trials published in biomedical journals: study protocol for a randomised controlled trial. BMJ Open. 2020;10(3):e035114. 
 **Conflict of Interest Disclosures** Erika Mann and Alejandra 
Clark are employed by the Public Library of Science. Paula 
Dhiman, Michael M. Schlussel, Philippe Ravaud, and David Moher are members of the EQUATOR (Enhancing the Quality and Transparency of Research) network. During the design and initial implementation, Iratxe Puebla was an employee by the Public Library of Science. An-Wen Chan and David Moher are authors of the SPIRIT 2013 Statement (Standard Protocol Items: Recommendations for Interventional Trials). David Moher, Sally 
Hopewell, and Isabelle Boutron are members of the Consolidated Standards for Reporting Trials (CONSORT) group and authors of the CONSORT 2010 statement. Sara Schroter is employed by The BMJ. David Moher is an associate director and An-Wen Chan and 
Isabelle Boutron are advisory board members of the International Congress on Peer Review and Scientific Publication but were not involved in the review or decision for this abstract. No other disclosures were reported. 
 **Funding/Support** This study was supported in part by the Swiss National Science Foundation. 
 **Additional Information** The study was registered on the Open Science Framework (https://osf.io/c4hn8). 
 
![](https://assets.underline.io/uploads/markdown_image/1/image/7c461293a2cecd0da9c6e8af6e728d3a.png)

Reminding Peer Reviewers of the Most Important Reporting Guideline Items to Improve Completeness in Published Articles: Primary Results of 2 Randomized Controlled Trials | VIDEO

**Objective** The Resident & Fellow Section (RFS) of Neurology expanded and formalized a biannual virtual, mentored peer review training program in 2020. Similar to prior studies,1 qualitative data demonstrated that mentored peer review improved understanding of and confidence with independent review.2 The objective of the current study was to quantitatively evaluate review quality before and after a mentored peer review program for neurology residents and 
fellows. 
 **Design** In this pre-post intervention study, faculty mentors chosen from a national pool of experienced reviewers with an interest in mentoring were paired with trainee peer reviewers. Mentees were selected from residents and fellows who responded to a call for program participants via American Academy of Neurology social media channels or who applied to the RFS editorial board. Mentees first completed unassisted reviews of a standardized manuscript. Participants then received structured resources on how to review a manuscript and construct written peer review. Mentormentee dyads completed 2 reviews over 6 months. Mentees also completed unassisted postprogram reviews of a standardized manuscript. The association between program participation and peer review quality was quantitatively assessed by 3 independent evaluators scoring the preprogram and postprogram unassisted reviews using a modified version of the Review Quality Instrument (RQI)3 comprising 4 items scored present or absent (1 or 0) and 10 items scored on a 5-point Likert scale for a maximum score of 55. Higher scores indicated better review quality. Evaluator scores were averaged for each participant, and a paired t test was used to compare preintervention and postintervention scores. P < .05 was considered significant. 
 **Results** A total of 20 mentor-mentee pairs, including 8 incoming members of the RFS editorial board and 12 neurology trainees, were enrolled over 2 sessions in 2021. Two mentees failed to complete the postprogram unassisted review and were excluded. Total modified RQI scores ranged from 15.67 to 39.00 before the program and 19.67 to 43.67 after the program. The mean total score for all participants increased following completion of the program (preprogram, 26.38; postprogram, 31.70; P < .001). Postprogram reviews were more likely to include separate comments for editors and authors (mean preprogram score, 0.30; mean postprogram score, 0.91; P < .001), more likely to identify the strengths and weaknesses of the research methods (mean preprogram score, 2.75; mean postprogram score, 3.18; P = .02), and more likely to comment on the interpretation of results (mean preprogram score, 2.33; mean postprogram score, 2.82; P = .009). Preprogram and postprogram scores for all 14 measures of the modified RQI are shown in **Table 10**. 
 **Table 10. Modified RQI Scores Before and After Program Participation**
![](https://assets.underline.io/uploads/markdown_image/1/image/aa84fda7cb5950600e37aea35f54b820.png) 
 **Conclusions** Peer review quality improved for residents and trainees following completion of a structured, mentored peer review curriculum. Formal mentoring to teach the proper approach to peer review is one tool to expand the bench of available quality peer reviewers. 
 **References**
1.	Wong VSS, Strowd RE 3rd, Aragón-García R, et al. Mentored peer review of standardized manuscripts as a teaching tool for residents: a pilot randomized controlled multi-center study. Res Integr Peer Rev. 2017;2:6. doi:10.1186/s41073-017-0032-0 
 2.	Lyons-Warren AM, Aamodt WW, Pieper KM, Strowd R. A sustainable, comprehensive, mentored program for teaching peer review to neurology residents. Presented at: American Academy of Neurology Annual Meeting, April 17-22, 2021. 
 3.	van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol. 1999;52(7):625-629. doi:10.1016/s0895-4356(99)00047-5 
 **Conflict of Interest Disclosures** Ariel M. Lyons-Warren reported serving as the chair of the Resident and Fellow Section Mentored Review Program. Whitley W. Aamodt reported receiving an editorial stipend from Neurology and serving as the deputy section editor for the Resident and Fellow section of Neurology. Roy Strowd reported serving as a consultant for Monteris Medical Inc and Novocure; receiving an editorial stipend as section editor for the Resident and Fellow section of Neurology; having received stipends as an educational lecturer for Lecturio and Kaplan; and having received research and grant support from the American Academy of Neurology (AAN), the American Board of Psychiatry and Neurology, American Society for Clinical Oncology, Southeastern Brain Tumor Foundation, and Jazz Pharmaceuticals. Kathleen M. Pieper reported receiving salary support from AAN and serving as the senior managing editor of Neurology. José G. Merino reported receiving salary support from the AAN for his editorial role and serving as editor in chief of Neurology. 
 **Acknowledgements** We thank the residents and fellows who participated in the program and completed the preprogram and postprogram assessments as well as the senior reviewers who served as mentors. In addition, we thank Seth Retzlaff in the Neurology editorial office for the handling and tracking of manuscripts. We greatly appreciate their time and commitment to improving the next generation of peer reviewers.

![](https://assets.underline.io/uploads/markdown_image/1/image/35f60d76cb1531607f5aefaedb5c1e8e.png)

Assessment of a Structured and Mentored Peer Review Curriculum on Quality of Peer Review | VIDEO

**Objective** To perform a systematic review of available online
training for scholarly peer review of biomedical journal
articles. 
 **Design** A search strategy was developed and reviewed using
the PRESS (Peer Review of Electronic Search Strategies)
checklist by a medical librarian. A database search of
MEDLINE, PsycINFO, Embase, ERIC, and Web of Science
was conducted. Additional supplementary searches were
done of preprint servers, Google, YouTube, university library
websites, publisher websites, and peer review–related events
and groups. All English or French training documents for
scholarly peer review of biomedical manuscripts freely
accessible online between January 1, 2012, and the date of the
search (September 13, 2021) were included. January 1, 2012,
was used as the earliest cutoff because this was the year
Publons was launched. A Preferred Reporting Items for
Systematic Reviews and Meta-Analyses (PRISMA) flow
diagram with full exclusion criteria is shown in **Figure 11**.
Screening was done in duplicate by 2 independent reviewers
in 2 stages, with conflicts resolved by a third party. Data
extraction and risk of bias were done by 1 reviewer and then
verified by a second. As no current risk of bias tool could be
found for evaluating training material, one was created,
which was pilot tested for feasibility. 
 **Figure 11. PRISMA Flow Diagram**
![](https://assets.underline.io/uploads/markdown_image/1/image/b9eb9c92bb533c83bdd0f4764fe80cfa.png)
**Results** Of 1244 records screened, 45 online training
documents were identified for peer review. Barriers such as
paywalls and membership requirements limited access to just
more than half of these documents (23 of 45 [51%]); thus,
they were excluded from data analysis because they were not
freely available online. The included documents were mostly
websites (13 of 22 [59%]) and videos (6 of 22 [27%]) offered
exclusively in English (19 of 22 [86%]). Many of the
documents did not report a creation year (10 of 22 [45%]),
author information (10 of 22 [45%]), or funding sources (19
of 22 [86%]). Countries that developed the greatest amount
of training were the US (8 of 22 [36%]), United Kingdom (4
of 22 [18%]), and Germany (3 of 22 [14%]). The main training
formats were online modules (13 of 22 [59%]) and webinars
(5 of 22 [23%]) and took less than 1 hour to complete (15 of
22 [68%]). Topics that were frequently included were an
overview of the peer review process (18 of 22 [82%]),
synthesis of a peer review report (20 of 22 [91%]), and critical
appraisal of data (18 of 22 [82%]). Conversely, critical
appraisal of clinical trials (4 of 22 [18%]), statistics (4 of 18
[3%]), and reporting guidelines (9 of 22 [41%]) were less
commonly included. 
 **Conclusions** This systematic review identified a
comprehensive list of available online training material for
scholarly peer review of biomedical journals and an analysis
of their characteristics. 
 **Conflict of Interest Disclosures** David Moher is an associate
director of the Peer Review Congress but was not involved in the
review or decision of this abstract. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/96fb46c2427ca7ddd293c8c9e026d696.png)

Online Training in Scholarly Peer Review: A Systematic Review | VIDEO

**Objective** The great astronomical observatories accept
thousands of proposals per year from astronomers hoping to
receive telescope time. Specifically, the Space Telescope
Science Institute receives approximately 1000 proposals per
year for the Hubble Space Telescope, with this number
projected to double as the James Webb Space Telescope has
safely launched. 2 In astronomy, a Time Allocation Committee
(TAC) reviews all proposals submitted for the use of a
telescope and identifies the proper expert to review the
proposal. The goal of the study was to develop a database of
all active astronomers and their publications that assists in
the identification of experts for the peer review of observing
proposals, expanding on work done by Kerzendorf et al 1 and
Strolger et al. 2 
 **Design** The database creation and modeling study has
expanded the reviewer pool to all around the world, instead of
simply relying on the TAC’s personal networks. The Semantic
Scholar Open Research Corpus (S2ORC) data set allowed for
the creation of a preliminary database consisting of authors,
their full-text publications, and associated metadata. The
identification of experts for peer review was systematically
done by leveraging an astronomer’s body of work (ie, scientific
publications). An author’s publications and the observing
proposal were numerically represented using machine
learning models to identify which astronomer’s expertise is
similar for review of the proposal. Various methods were
compared to disambiguate author names using name-based
techniques. However, authors with full names having more
than 3 words were excluded owing to formatting issues
(currently investigating methods to address the issue). A
preliminary prototype using machine learning and natural
language processing models was tested using 918 proposals
from the European Southern Observatory (significant metrics
to evaluate expertise are being researched). 
 **Results** The S2ORC data set, which consists of 12 million
full-text publications, was filtered to only astronomy
publications using publication arXiv identifiers. The database
contains 212,839 publications and a total of 1,801,916
nonunique authors from 1991 to 2020. Three author name
disambiguation algorithms were compared: first initial, all
initials, and hybrid method. 3 The 3 methods were validated
using an initial subset of 1538 ORCID identifiers matched to
astronomers. A contamination rate is the percentage of
validated astronomers whose identity became compromised
due to merging or splitting of names. The contamination
rates of the 3 methods were 1.77%, 15.52%, and 2.02%,
respectively. 
 **Conclusions** The developed database has expanded the
possible reviewer pool from several hundreds known to the
TAC to all active astronomers worldwide. A larger pool of
reviewers allows for more accurate expertise matching. 
 **References**
1. Kerzendorf WE, Patat F, Bordelon D, van de Ven G,
Pritchard TA. Distributed peer review enhanced with natural
language processing and machine learning. Nat Astron.
2020;4(7):711-717. doi:10.1038/s41550-020-1038-y 
 2. Strolger LG, Porter S, Lagerstrom J, Weissman S,
Reid IN, Garcia M. The Proposal Auto-Categorizer and
Manager for time allocation review at Space Telescope
Science Institute. AJ. 2017;153(4):181. doi:10.3847/1538-
3881/aa6112 
 3. Milojević S. Accuracy of simple, initials-based
methods for author name disambiguation. J Informetrics.
2013;7(4):767773. doi:10.1016/j.joi.2013.06.006 
 **Conflict of Interest Disclosures** None reported. 
 **Funding/Support** Funding was received from the Space Telescope
Science Institute–Hubble Space Telescope Science Policies Group. 
 **Role of Funder/Sponsor** The funder collaborated with members
from the Space Telescope Science Institute to oversee the
development of the database. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/38d9f26971a153c5a4bd679ecfae6086.png)

Development of a Global Data Set for Peer Review in Astronomy

**Objective** Many anecdotes suggest that including citations to
the works of potential reviewers is a good (albeit unethical)
way to increase the acceptance chances of a paper. However,
previous attempts 1,2 to quantify this effect (citation bias) had
low sample sizes and unaccounted confounding factors, such
as paper quality (stronger papers had longer bibliographies)
or reviewer expertise (cited reviewers had higher expertise).
In this work, the question of whether positive comments from
reviewers are associated with their work being cited in the
papers that they review was investigated. 
 **Design** The study used data from 2 top-tier computer science
conferences: the 2021 Association for Computing Machinery
Conference on Economics and Computation (EC) and 2020
International Conference on Machine Learning (ICML). Both
conferences received full-length papers that underwent
rigorous review (similar to top journals in other areas). The
study analyzed anonymized observational data, and consent
collection was not required. The dependent variable of the
analysis was the overall score given by a reviewer to a paper
(between 1 and 5 in EC and 1 and 6 in ICML; higher meant
better). To investigate the association between the citation of
a reviewer and their score, parametric (linear regression for
EC and ICML) and nonparametric (permutation test with
covariate matching for ICML) tests at significance level α =
.05 were combined, circumventing various confounding
factors, such as paper quality, genuinely missing citations,
reviewer expertise, reviewer seniority, and reviewers’
preferences in which papers to review. The approach
comprised matching cited and uncited reviewers within each
paper and then carefully analyzing the differences in their
scores. In this way, the aforementioned paper quality
confounder was alleviated as matched cited and uncited
reviewers reviewed the same paper. Additionally, various
attributes of reviewers (eg, their expertise in the paper’s
research area) were used to account for confounders
associated with the reviewer identity (eg, reviewer expertise).
Finally, the genuinely missing citation confounder was
accounted for by excluding papers in which an uncited
reviewer genuinely decreased their evaluation of a paper
because it failed to cite their own relevant past work. 
 **Results** Overall, 3 analyses were conducted, with sample
sizes ranging from 60 to 1031 papers and from 120 to 2757
reviewers’ evaluations. These analyses detected citation bias
in both venues and indicated that citation of a reviewer was
associated with an increase in their score (approximately 0.23
point on a 5-point scale). For reference, a 1-point increase of a
score by a single reviewer would improve the position of a
paper by 11% on average. 
 **Conclusions** To improve peer review, it is important to
understand the biases present and their magnitude. This
work 3 studied citation bias and raised an important open
problem of mitigating the bias. The reader should be aware of
the observational nature of this study when interpreting the
results. 
 **References**
1. Beverly R, Allman M. Findings and implications
from data mining the IMC review process. ACM SIGCOMM
Computer Communication Review. 2012;43(1):22-29.
doi:10.1145/2427036.2427040 
 2. Sugimoto CR, Cronin B. Citations gamesmanship:
testing for evidence of ego bias in peer review.
Scientometrics. 2013;95(3):851-862. doi:10.1007/s11192-012-
0845-z 
 3. Stelmakh I, Rastogi C, Liu R, Echenique F, Chawla S,
Shah NB. Cite-seeing and reviewing: a study on citation bias
in peer review. arXiv. Preprint posted online March 31, 2022.
doi:10.48550/arXiv.2203.17239 
 **Conflict of Interest Disclosures** Ivan Stehlmakh reported
interning for Google. Charvi Rastogi reported interning for IBM. No
other disclosures were reported. 
 **Funding/Support** Ivan Stehlmakh, Charvi Rastogi, Ryan Liu, and
Nihar B. Shah were supported by a National Science Foundation
(NSF) CAREER award (1942124), which supports research on the
fundamentals of learning from people with applications to peer
review. Federico Echenique was supported by NSF grants SES
1558757 and CNS 1518941. Charvi Rastogi was partially supported by
a J. P. Morgan artificial intelligence research fellowship. 
 **Role of the Funder/Sponsor** The funders did not play a role in any of the following: design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation. 
 **Acknowledgement** We appreciate the efforts of all reviewers involved in the review process of the 2021 Association for 
Computing Machinery Conference on Economics and Computation and 2020 International Conference on Machine Learning. We thank Valerie Ventura, PhD, Carnegie Mellon University, for useful comments on the design of the analysis procedure. 
 **Additional Information** Nihar B. Shah is a co–corresponding author.

![](https://assets.underline.io/uploads/markdown_image/1/image/f8f70e34240d7ce6028668bf52422552.png)

Comparison of Review Scores of Computer Science Conference Submissions With Cited and Uncited Reviewers

**Objective** Merton 1 argued that “eminent scientists get
disproportionately great credit for their contribution to
science while relatively unknown scientists tend to get
disproportionately little credit for comparable contributions.”
In this context, this study asked (1) is there a status bias in
reviewers’ propensity to accept review invitations? and (2) is
there a status bias in their evaluation of the paper? 
 **Design** A manuscript written by Vernon Smith (Nobel
laureate, high prominence) and Sabiou Inoua (young
researcher, low prominence) was submitted to the Journal of
Behavioral and Experimental Finance for peer review. The
paper was assigned to 3299 reviewers randomized into 5
conditions: (1) no author name in the invitation email or on
the manuscript’s title page (treatment: AA; 576 reviewers);
(2) high-prominence author name on manuscript only (AH;
696); (3) low-prominence author name on manuscript only
(AL; 739); (4) high-prominence author name on both email
and manuscript (HH; 507); and (5) low-prominence author
name on both email and manuscript (LL; 781). To avoid
confounding, only 1 name was shown in the email and on the
manuscript, and the author was always designated as the
corresponding author. Reviewers gave consent to being part
of the study prior to accessing the paper. Those who
submitted a report were debriefed after the study. Reviewers’
decisions to accept the invitation in response to anonymized
(AA, AH, AL) vs nonanonymized (LL, HH) emails were
compared using Fisher exact tests. The distribution of
publication recommendations (eg, reject, major revision,
minor revision, or accept) was compared for manuscripts that
showed the author’s name (AL, AH) vs those that did not
(AA) using Mann-Whitney tests. 
 **Results** A total of 2611 researchers (79.1%) responded to the
invitation, 821 of whom agreed to review (31.4%). The
invitation showing Vernon Smith was accepted statistically
significantly more often than those showing no author name
or Sabiou Inoua (acceptances: HH, 158 of 410 [38.5%] vs LL,
174 of 610 [28.5%]; P = .001; HH, 158 of 410 [38.5%] vs
anonymized, 489 of 1591 [30.7%]; P = .003). Of the 821
reviewers who accepted the invitation, 534 (65.0%) submitted
reports (AA, 110; AL, 101; AH, 102; LL, 114; and HH, 107).
The manuscript showing the prominent author received
53.3% less reject recommendations and more than 10 times
as many accept recommendations as the anonymized version
(test on the distribution of recommendations: AH vs AA, P &lt;
.001) (**Table 11**, A). The manuscript showing the name of the
less prominent author got 35.5% more reject
recommendations and 63.7% less minor revision
recommendations than the anonymized version (test on the
distribution of recommendations: AL vs AA, P = .005) (**Table
11**, B). Author prominence affected the willingness to review
and reviewers’ recommendations. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/35cb4b32b379646b2f28921dca61b31e.png)**Conclusions** Although double-anonymized peer review is
not a panacea, 2 this study’s results still support its use in the
field of finance. 
 **References**
1. Merton, RK. The Matthew effect in science. Science.
1968;159(3810):56-63. doi:10.1126/science.159.3810.56 
 2. Snodgrass, R. Single- versus double-blind reviewing: an
analysis of the literature. ACM SIGMOD Record.
2006;35(3):8-21. doi:10.1145/1168092.1168094 
 **Conflict of Interest Disclosures** None reported. 
 **Funding/Support** Financial support from the Austrian Science
Fund through SFB F63 is gratefully acknowledged. 
 **Role of the Funder/Sponsor** The funding organization did not
play any role in the design and conduct of the study; collection,
management, analysis, and interpretation of the data; preparation,
review, or approval of the abstract; and the decision to submit the
abstract for presentation. 
 **Additional Information** Jürgen Huber is a co–corresponding
author. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/632941d449c7be540c67c5346fde6ef7.png)

Association Between Author Prominence and Peer Reviewers' Willingness to Review and Their Evaluations of Manuscripts Submitted to a Finance Journal

**Objective** High scientific quality produces knowledge and is continuously assessed by the scientific community in multiple ways (eg, reviews, seminars, committees that control funding, publication of research results, and selection of individuals for awards, employment, and other positions). The quality of these assessments affects the quality of the research produced.1,2 So how valid and reliable are they, and what weight do academics attach to different features of the research in their assessment of scientific quality? 
 **Design** A rating instrument was devised with 36 concrete and 2 overall quality indicators (**Table 12**), partly based on the evaluation criteria in scientific journals’ instructions for reviewers. Fifteen academics with a PhD in the social sciences were recruited through snowball sampling. They rated 60 randomly selected journal articles from the Swedish Gender Studies List of more than 12,000 publications about sex and gender across many disciplines (eg, gender studies [GS], sociology, history, medicine, and psychology) published between 2000 and 2010. Factor analysis was applied to identify the main dimensions that reviewers consider, and these were compared across research with different levels of GS perspective.3 A high level of GS perspective was assigned to articles authored by those who self-identified as gender scholars or explicitly endorsed a gender perspective, a medium level to articles that reflected these values and beliefs, and a low level to the remaining articles. Statistical tests included Cronbach α for interrater reliability, R2 for factor analysis, and Cohen d and P values for differences between GS groups. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/face4de9825e4d15726c834490017978.png) 
 **Results** Interrater reliability was high (Cronbach α = 0.75-0.87). Factor analysis suggested 3 dimensions: logic and clarity (eg, “there is a clear connection between theory and empirics, such that the data logically correspond to what is being claimed, hypothesized, or tested”; R2 = 0.44); causality (eg, “a causal relation is mentioned or can be inferred, and the design and methods in general enable this to be evaluated”; R2 = 0.08); and scope (eg, more data, units of analysis, generalizability, and better sampling; R2 = 0.10). The analysis could discriminate between 3 samples of 20 randomly selected articles from each of the populations with (1) a high level of GS perspective, (2) a medium level, and (3) no GS perspective on these 3 dimensions (logic and clarity, d = 0.62, P < .001; causality, d = 0.35, P < .005; and scope, d = 
0.44, P < .005). 
 **Conclusions** The scientific quality instrument seems promising in terms of high reliability and convergent validity and can be used in any type of design (eg, experimental, cross-sectional, correlational, or descriptive) to assess overall and specific dimensions of scientific quality. Additional testing of the instrument is needed to compare different bodies of research, funding applications, project plans, and 
papers before and after peer review; assess group differences across topics, institutions, countries, and time; and compare mainstream vs controversial fields of research to further assess the role of research quality. 
 **References**
1.	Finkel EJ, Eastwick PW, Reis HT. Replicability and other features of a high-quality science: toward a balanced and empirical approach. J Pers Soc Psychol. 2017;113(2):244-253. 
doi:10.1037/pspi0000075 
 2.	Tuffaha HW, El Saifi N, Chambers SK, Scuffham PA. Directing research funds to the right research projects: a review of criteria used by research organisations in Australia in prioritising health research projects for funding. BMJ Open. 2018;8(12):e026207. doi:10.1136/ bmjopen-2018-026207 
 3.	Söderlund T, Madison G. Objectivity and realms of explanation in academic journal articles concerning sex/ gender: a comparison of gender studies and the other social sciences. Scientometrics. 2017;112(2):1093-1109. doi:10.1007/s11192-017-2407-x 
 **Conflict of Interest Disclosures** None reported. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/5c40d8638b8069d3c57f909b01c9a9f0.png)

Factor Analysis of Academic Reviewers Ratings of Journal Articles on a 38-Item Scientific Quality Instrument | VIDEO

**Objective** The ability of peer review to improve the scientific
endeavor (eg, the conduct, reporting, and validity of study
findings) has been questioned,1 and calls have been made to
showcase changes that occurred to each study due to peer
review.2 Until such transparency is achieved, identification
and synthesis of studies that analyzed differences between
preprints or manuscript versions submitted to journals and
peer-reviewed publications is being undertaken. 
 **Design** In this stage of the living systematic review, studies
were identified based on authors’ knowledge of the field and
by checking all research at peer review conferences
(presented as podium presentations or posters in the
European Union and USA). References and citations of
identified studies were then checked. For all studies, the
following was extracted: year of publication, sampling
method, conflict of interest, funding, data and protocol
sharing, number of analyzed version pairs, sample size
calculation, scholarly discipline, method used to compare
versions, variables (ie, manuscript sections) analyzed for
changes, and metric with which the changes were quantified
or qualitatively classified. 
 **Results** Of 25 studies published from 1990 through the end
of 2021, 16 (64%) analyzed changes between submitted and
published papers and 9 (36%) between preprints and
published papers. Most commonly, changes were analyzed by
filling out questionnaires or scales separately for each of the 2
manuscript versions (11 [44%]) or by manual comparison of
the 2 manuscript versions (6 [24%]). The median number of
analyzed version pairs was 59 (IQR, 41-122). Most studies
analyzed changes that occurred in health (18 [72%]) or social
sciences (4 [16%]) manuscripts. Overall, studies’ conclusions
indicated very high similarity between version pairs, with the
largest changes occurring in introduction and discussion
sections. Examples of items for which most changes were
found are presented in **Table 13**. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/8c97dc5a11f6a5f830a8e977dad133da.png) 
 **Conclusions** The current results indicate that submitted or
preprinted manuscript versions and their peer-reviewed
journal version are very similar, with main (analysis) methods
and main findings rarely changing. Quantification of these
results is pending. Large differences between studies, type of
manuscript changes, and methods with which they were
measured indicate greater need for collaboration in the peer
review field and creation of the core outcomes measures for
manuscript version changes. 
 **References**
1. Tennant JP, Ross-Hellauer T. The limitations to our
understanding of peer review. Res Integr Peer Rev.
2020;5(1):6. doi:10.1186/s41073-020-00092-1 
 2. Limbu S. Building trust in peer review: a Q&A with Dr
Mario Malički. BioMed Central. September 18, 2020.
Accessed June 24, 2022. http://blogs.biomedcentral.com/on-medicine/2020/09/18/building-trust-in-peer-review-a-qa-with-dr-mario-malicki/ 
 **Conflict of Interest Disclosures** IJsbrand Jan Aalbersberg is
senior vice president of research integrity at Elsevier. Mario Malički
is a co–editor in chief of Research Integrity and Peer Review. Lex
Bouter, John P. A. Ioannidis, and Steven N. Goodman are members
of the Peer Review Congress Advisory Board but were not involved
in the review or decision for this abstract. 
 **Funding/Support** Elsevier funding was awarded to Stanford
University for a METRICS postdoctoral position that supported
Mario Malički’s work on the project. 
 **Role of the Funder/Sponsor** IJsbrand Jan Aalbersberg is an
employee of Elsevier and had a role in the design and conduct of
the study; management and interpretation of the data; review and
approval of the abstract; and decision to submit the abstract for
presentation. 
 ![](https://assets.underline.io/uploads/markdown_image/1/image/285fdcf7b8c697b89f8022f686600131.png)

Downloads

Next from Peer Review Congress 2022

Improving the Research Culture to Increase Credibility of Research Findings, Brian Nosek

Similar lecture

Travel influencers under lockdown: The impact of COVID-19 on Instagram

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from Peer Review Congress 2022

Improving the Research Culture to Increase Credibility of Research Findings, Brian Nosek

Similar lecture

Travel influencers under lockdown: The impact of COVID-19 on Instagram

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads