Objective A previous study revealed that patient-reported outcome measures (PROMs) with poor or unknown psychometric properties were associated with higher estimates of treatment effect in clinical trials of rotator cuff diseases.1 This study assessed the variations in meta-analysis estimates in orthopedics associated with varying quality of PROMs and hypothesized an average higher estimate in PROMs with unknown and poor psychometric properties.

Design Meta-analyses were identified from 5 databases from inception through October 16, 2017. PROM scores were derived from a prior publication that comprehensively assessed the quality of these instruments (higher scores were better quality).2 Standardized mean difference (effect size) or mean difference of change in PROM scores (from before treatment to after treatment) between different treatment types were extracted or calculated for each study. For those studies that did not report standardized results, change scores were divided by the SD for standardization. The SD was imputed in some cases from SEs and CIs. A mixed-effects regression analysis was done, with all standardized change scores as dependent variables and other data as independent variables (PROM overall quality score, number of studies, total sample size across included studies, and average follow-up), controlling for the grouping variable meta- analysis (in which multiple estimates were calculated for several PROMs from within the same meta-analysis). A sensitivity analysis was done excluding meta-analytic estimates for mixed interventions. Increases in β coefficients indicate effect size change for each unit increase in PROM quality.

Results A total of 249 unduplicated meta-analyses on rotator cuff disease were reviewed with 47 being included, with 6 different PROMs included, and several meta-analyses included mixed outcomes with several PROMs being combined. Reviews were excluded (202) primarily because one of the PROMs of interest was not used. The β coefficient for PROM quality and the pooled effect size estimates was −0.012 (95% CI, −0.049 to 0.025; P = .53) before and after controlling for several covariates (Table 77). In the sensitivity analysis, after removing meta-analyses with mixed PROMs in the pooled effect size estimates, the β coefficient for PROM quality and pooled effect estimates was −0.013 (95% CI, −0.034 to 0.007; P = .19); this finding was not statistically significant.

Conclusions In estimating the percentage of bias, this study found that pooled effect size estimates across PROMs of poor quality inflate effect estimates by approximately 10% (the ratio of the β estimate and meta-analytic estimate). This magnitude of effect size is not statistically significant, but larger methodologic studies may be warranted to confirm clinical significance.

References 1. Gagnier JJ, Johnston BC. Poor quality patient reported outcome measures bias effect estimates in orthopaedic randomized studies. J Clin Epidemiol. 2019;116:36-38. 2. Huang S, Grant J, Miller B, Mirza FM, Gagnier JJ. A systematic review of psychometric properties of patient reported outcome instruments for use in patients with rotator cuff disease. Am J Sports Med. 2015;43(10):2572-2582.

Conflict of Interest Disclosures None reported.

Additional Information Jianyu Lai is a co–corresponding author.