Peer Review Congress 2022

September 11, 2022

Chicago, United States

Bias in Meta-analysis Estimates Associated With Varying Quality of Patient-Reported Outcome Measures in Orthopedics


quality of trials



Objective A previous study revealed that patient-reported outcome measures (PROMs) with poor or unknown psychometric properties were associated with higher estimates of treatment effect in clinical trials of rotator cuff diseases.1 This study assessed the variations in meta-analysis estimates in orthopedics associated with varying quality of PROMs and hypothesized an average higher estimate in PROMs with unknown and poor psychometric properties.

Design Meta-analyses were identified from 5 databases from inception through October 16, 2017. PROM scores were derived from a prior publication that comprehensively assessed the quality of these instruments (higher scores were better quality).2 Standardized mean difference (effect size) or mean difference of change in PROM scores (from before treatment to after treatment) between different treatment types were extracted or calculated for each study. For those studies that did not report standardized results, change scores were divided by the SD for standardization. The SD was imputed in some cases from SEs and CIs. A mixed-effects regression analysis was done, with all standardized change scores as dependent variables and other data as independent variables (PROM overall quality score, number of studies, total sample size across included studies, and average follow-up), controlling for the grouping variable meta- analysis (in which multiple estimates were calculated for several PROMs from within the same meta-analysis). A sensitivity analysis was done excluding meta-analytic estimates for mixed interventions. Increases in β coefficients indicate effect size change for each unit increase in PROM quality.

Results A total of 249 unduplicated meta-analyses on rotator cuff disease were reviewed with 47 being included, with 6 different PROMs included, and several meta-analyses included mixed outcomes with several PROMs being combined. Reviews were excluded (202) primarily because one of the PROMs of interest was not used. The β coefficient for PROM quality and the pooled effect size estimates was −0.012 (95% CI, −0.049 to 0.025; P = .53) before and after controlling for several covariates (Table 77). In the sensitivity analysis, after removing meta-analyses with mixed PROMs in the pooled effect size estimates, the β coefficient for PROM quality and pooled effect estimates was −0.013 (95% CI, −0.034 to 0.007; P = .19); this finding was not statistically significant.

Conclusions In estimating the percentage of bias, this study found that pooled effect size estimates across PROMs of poor quality inflate effect estimates by approximately 10% (the ratio of the β estimate and meta-analytic estimate). This magnitude of effect size is not statistically significant, but larger methodologic studies may be warranted to confirm clinical significance.

References 1. Gagnier JJ, Johnston BC. Poor quality patient reported outcome measures bias effect estimates in orthopaedic randomized studies. J Clin Epidemiol. 2019;116:36-38. 2. Huang S, Grant J, Miller B, Mirza FM, Gagnier JJ. A systematic review of psychometric properties of patient reported outcome instruments for use in patients with rotator cuff disease. Am J Sports Med. 2015;43(10):2572-2582.

Conflict of Interest Disclosures None reported.

Additional Information Jianyu Lai is a co–corresponding author.

Next from Peer Review Congress 2022

Development and Pilot Test of Risk of Bias Assessment Tool for Use in Peer Review | VIDEO

Development and Pilot Test of Risk of Bias Assessment Tool for Use in Peer Review | VIDEO

Peer Review Congress 2022

Brian S Alper

11 September 2022

Similar lecture

HeteroCorpus: A Corpus for Heteronormative Language Detection
workshop paper

HeteroCorpus: A Corpus for Heteronormative Language Detection

NAACL 2022

Juan Vaquez
Juan Vaquez

14 July 2022

Stay up to date with the latest Underline news!

Select topic of interest (you can select more than one)


  • All Lectures
  • For Librarians
  • Resource Center
  • Free Trial
Underline Science, Inc.
1216 Broadway, 2nd Floor, New York, NY 10001, USA

© 2023 Underline - All rights reserved