technical paper

Peer Review Congress 2022

September 09, 2022

Chicago, United States

Factor Analysis of Academic Reviewers Ratings of Journal Articles on a 38-Item Scientific Quality Instrument | VIDEO


funding/grant peer review

quality of the literature

editorial and peer review process

Objective High scientific quality produces knowledge and is continuously assessed by the scientific community in multiple ways (eg, reviews, seminars, committees that control funding, publication of research results, and selection of individuals for awards, employment, and other positions). The quality of these assessments affects the quality of the research produced.1,2 So how valid and reliable are they, and what weight do academics attach to different features of the research in their assessment of scientific quality?

Design A rating instrument was devised with 36 concrete and 2 overall quality indicators (Table 12), partly based on the evaluation criteria in scientific journals’ instructions for reviewers. Fifteen academics with a PhD in the social sciences were recruited through snowball sampling. They rated 60 randomly selected journal articles from the Swedish Gender Studies List of more than 12,000 publications about sex and gender across many disciplines (eg, gender studies GS, sociology, history, medicine, and psychology) published between 2000 and 2010. Factor analysis was applied to identify the main dimensions that reviewers consider, and these were compared across research with different levels of GS perspective.3 A high level of GS perspective was assigned to articles authored by those who self-identified as gender scholars or explicitly endorsed a gender perspective, a medium level to articles that reflected these values and beliefs, and a low level to the remaining articles. Statistical tests included Cronbach α for interrater reliability, R2 for factor analysis, and Cohen d and P values for differences between GS groups.

Results Interrater reliability was high (Cronbach α = 0.75-0.87). Factor analysis suggested 3 dimensions: logic and clarity (eg, “there is a clear connection between theory and empirics, such that the data logically correspond to what is being claimed, hypothesized, or tested”; R2 = 0.44); causality (eg, “a causal relation is mentioned or can be inferred, and the design and methods in general enable this to be evaluated”; R2 = 0.08); and scope (eg, more data, units of analysis, generalizability, and better sampling; R2 = 0.10). The analysis could discriminate between 3 samples of 20 randomly selected articles from each of the populations with (1) a high level of GS perspective, (2) a medium level, and (3) no GS perspective on these 3 dimensions (logic and clarity, d = 0.62, P < .001; causality, d = 0.35, P < .005; and scope, d = 0.44, P < .005).

Conclusions The scientific quality instrument seems promising in terms of high reliability and convergent validity and can be used in any type of design (eg, experimental, cross-sectional, correlational, or descriptive) to assess overall and specific dimensions of scientific quality. Additional testing of the instrument is needed to compare different bodies of research, funding applications, project plans, and papers before and after peer review; assess group differences across topics, institutions, countries, and time; and compare mainstream vs controversial fields of research to further assess the role of research quality.

References 1. Finkel EJ, Eastwick PW, Reis HT. Replicability and other features of a high-quality science: toward a balanced and empirical approach. J Pers Soc Psychol. 2017;113(2):244-253. doi:10.1037/pspi0000075

2. Tuffaha HW, El Saifi N, Chambers SK, Scuffham PA. Directing research funds to the right research projects: a review of criteria used by research organisations in Australia in prioritising health research projects for funding. BMJ Open. 2018;8(12):e026207. doi:10.1136/ bmjopen-2018-026207

3. Söderlund T, Madison G. Objectivity and realms of explanation in academic journal articles concerning sex/ gender: a comparison of gender studies and the other social sciences. Scientometrics. 2017;112(2):1093-1109. doi:10.1007/s11192-017-2407-x

Conflict of Interest Disclosures None reported.


SlidesTranscript English (automatic)

Next from Peer Review Congress 2022

A Synthesis of Studies on Changes Manuscripts Underwent Between Submission or Preprint Posting and Peer-Reviewed Journal Publication
technical paper

A Synthesis of Studies on Changes Manuscripts Underwent Between Submission or Preprint Posting and Peer-Reviewed Journal Publication

Peer Review Congress 2022

Mario Malicki
Mario Malicki

09 September 2022

Similar lecture

International research collaboration in the post-COVID era
technical paper

International research collaboration in the post-COVID era

Enago See the Future 2021 (Global - English)

Richard de Grijs
Richard de Grijs

18 November 2021

Stay up to date with the latest Underline news!

Select topic of interest (you can select more than one)


  • All Lectures
  • For Librarians
  • Resource Center
  • Free Trial
Underline Science, Inc.
1216 Broadway, 2nd Floor, New York, NY 10001, USA

© 2023 Underline - All rights reserved