Objective Peer-reviewed journals ask reviewers to rate the perceived impact or priority of a manuscript. Previous research has suggested an association between reviewer priority scores and citations.1 Altmetrics (alternative metrics) provide an alternative view on social impact (ie, uptake on factor, >5). This journal asks peer reviewers to rate the priority (defined as potential impact) of a manuscript on an ordinal rating scale with possible scores of 1, 2, 5, and 10 (highest priority). Manuscripts are typically reviewed by 2 reviewers. The mean priority score of all reviewers for a manuscript in the first review round constitutes the Manuscript Average Priority Score (MAPS). For this analysis, manuscripts were categorized into 4 quartiles (Qs), with the groups labeled as Q4 (MAPS score, ≤3) to Q1 (MAPS score, >5). The dependent variables, citations, and altmetric scores were obtained from the Dimensions database in February 2022; manuscripts and published articles were similarly stratified into quartiles, with the citation (or altmetrics) quartile Q1 containing the group of articles with the highest citation count (or altmetric score). The association between independent variables (MAPS scores) and citation or altmetric scores was measured using χ² tests for 4 × 4 contingency tables for the quartiles and using Spearman rank correlation between MAPS score ranks and citation or altmetric rank, respectively.

Results The MAPS scores for 451 published articles ranged from 1.5 to 10; citations, from 0 to 253; and altmetric scores, from 1 to 849. Although both mean and median citations as well as altmetric scores were higher in the higher MAPS quartiles (Table 46), the results of χ² tests were not statistically significant for citations (P = .46) but were statistically significant for altmetric scores (P = .03). The Spearman rank correlation between citation ranks and MAPS score ranks was statistically significant but weak (ρ = .0955; r2 = .009; P = .03). In contrast, altmetric score ranks had a stronger correlation with MAPS score ranks (ρ = .1313; r2 = .017; P = .002).

Conclusions This longitudinal bibliometric cohort study found that in the Journal of Medical Internet Research, a journal whose subject matter lends itself to the type of attention measured by altmetrics, altmetric scores seemed to be better correlated than citations with a manuscript’s potential impact as assessed by reviewers. Peer reviewers may interpret priority and impact in terms of social impact, rather than citations, raising further questions about the appropriateness of citation-based metrics to measure impact as understood by reviewers.

References

1. Opthof T, Coronel R, Janse MJ. The significance of the peer review process against the background of bias: priority ratings of reviewers and editors and the prediction of citation, the role of geographical bias. Cardiovasc Res. 2002;56(3):339-346. doi:10.1016/S0008-6363(02)00712-5

2. Eysenbach G. Can tweets predict citations? metrics of social impact based on twitter and correlation with traditional metrics of scientific impact. J Med Internet Res. 2011;13(4):e123. doi:10.2196/jmir.2012

3. Araujo AC, Vanin AA, Nascimento DP, et al. What are the variables associated with altmetric scores? Syst Rev. 2021;10:193. doi:10.1186/s13643-021-01735-0

Conflict of Interest Disclosures Gunther Eysenbach reported equity in JMIR Publications.