poster

Peer Review Congress 2022

September 11, 2022

Chicago, United States

Development of a List to Detect Statistical and Methodological Terms in Peer Reviews: A Cross-Sectional Study

keywords:

editorial and peer review process

peer review

statistics

Objective The peer review system used by academic journals to assess the quality of submitted papers is constantly in need of new technologies that can help reviewers and increase the objectivity of the process. The aim of the present study was to develop a comprehensive list of words related to research methods and statistics, which could then be used to automatically study the content and the quantity and types of methodological and statistical issues identified in peer review reports.

Design The terms for the list were identified from different glossaries.1,2 The terms were organized into subcategories, some of which were more related to methodological aspects (eg, study design, sampling, or procedures) or statistical aspects (eg, parametric descriptive parameters or data presentation) of research. The list was refined and enriched with terms using a computational language model. The final list and the preregistration can be found at the Open Science Framework website (https://osf.io/d34b9/). The occurrence of terms in the PEERE database3 of peer review reports, for which data on reviewer gender, continental region, review recommendation, journal impact factor, and subject area (N = 496,928) were available, was counted. Regression models were created to determine the frequency of the terms used based on peer review characteristics.

Results The mean length of a peer review was 196 words (95% CI, 108-330 words). A total of 26.8% of review reports in the sample contained at least 1 methodological term, while 70.7% contained at least 1 statistical term. It was more likely that the review contained methodological terms if the reviewer was female and resided in North America or Oceania compared with other continents, if the recommendation was to reject the article or send it for major revisions, if the journal was from the social sciences and humanities, if the journal was in the top impact factor quartile, and if the article was a longer peer review. More words related to statistics were observed when the reviewer was female, for reviews recommending any revisions, for journals from health and medical sciences, in journals in the top impact factor quartile, and in longer reviews. Mixed regression models indicated that the scientific area was more strongly associated with words related to statistics than were reviewer recommendations.

Conclusions The application of this newly developed list showed that methodological and statistical terms are not a common topic in peer review and that they are not always present in review reports from different fields. The list successfully covered research terms because the most- identified terms were related to categories relevant for most types of research. Future analysis should try to understand the context of the occurrence of words from the list to confirm the validity of the list of terms.

References

  1. Stark PB. Glossary of Statistical Terms. Accessed March 23,
  2. https://www.stat.berkeley.edu/~stark/SticiGui/Text/ gloss.htm
  3. Framework for Open and Reproducible Research Training. The Glossary of Terms. Accessed July 12, 2021. https://docs. google.com/document/d/1oNsHSRmgAsDGUKvq5joqBpv HeRmDJKldH2kSyJovHhk/edit#heading=h.2l8ssl89e1ey
  4. Squazzoni F, Ahrweiler P, Barros T, et al. Unlock ways to share data on peer review. Nature. 2020;578(7796):512-514. doi:10.1038/d41586-020-00500-y

Conflict of Interest Disclosures Ana Marušić is a member of the Peer Review Congress Advisory Board by was not involved in the editorial review or decision for this abstract. No other disclosures reported.

Funding/Support This study was funded by the Croatian Science Foundation “Professionalism in Health–Decision making in practice and research” under grant agreement IP-2019-04-4882. Richard A. Klein received funding from a Replication Studies grant (401.18.053) from the Netherlands Organization for Scientific Research, and a Consolidator Grant (IMPROVE) from the European Research Council (grant 726361). Francisco Grimaldo and Daniel Garcia-Costa are partially supported by the Spanish Ministry of Science, Innovation and Universities, the Spanish State Research Agency, and the European Regional Development Fund under project RTI2018-095820-B-I00.

Role of the Funder/Sponsor The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.

Next from Peer Review Congress 2022

Inaugural Drummond Rennie Lecture: Bias, Spin, and Problems With Transparency of Research, Isabelle Boutron
lecture

Inaugural Drummond Rennie Lecture: Bias, Spin, and Problems With Transparency of Research, Isabelle Boutron

Peer Review Congress 2022

Isabelle Boutron

08 September 2022

Similar lecture

Change in statistical terms in peer-reviewed journals
technical paper

Change in statistical terms in peer-reviewed journals

PEERE

+3Emilia López-Iñesta
Emilia López-Iñesta and 5 other authors

30 September 2020

Stay up to date with the latest Underline news!

Select topic of interest (you can select more than one)

PRESENTATIONS

  • All Lectures
  • For Librarians
  • Resource Center
  • Free Trial
Underline Science, Inc.
1216 Broadway, 2nd Floor, New York, NY 10001, USA

© 2023 Underline - All rights reserved