Peer Review Congress 2022

September 11, 2022

Chicago, United States

Methodological and Reporting Quality of Systematic Reviews in Dermatology


quality of reporting

quality assurance


Objective Over the past decade, there has been an exponential increase in systematic reviews and meta-analyses (SR/MAs) published in the medical literature, and some concerns have arisen that many of these are duplicate studies with redundant results or new studies of unclear quality.1-3 The aim of this study was to determine whether there was any change in the methodological or reporting quality of SRs in dermatology over the past decade.

Design This was a cross-sectional study of all SR/MAs published in 2010 and 2019 in the 10 highest-ranked dermatology journals by SCImago Journal. Methodological quality was assessed through duplicate and independent adjudication by 2 reviewers (A.M., L.K.L., J.K., A.L.) using the Risk of Bias in Systematic reviews (ROBIS) tool and, for SR/ MA of interventions, the A Measurement Tool to Assess Systematic Reviews (AMSTAR) 2 tool. Reporting quality was assessed by 1 investigator (A.M.) using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) and PRISMA for Abstracts (PRISMA-A) tools, and study, author, and journal characteristics were abstracted. Statistical analysis was performed with Jamovi, version 1.6.23. Methodological quality was analyzed with the χ2 test for independent proportions, or the Fisher exact test if any of the expected frequencies were less than 5, and difference in proportions with 95% CIs. Reporting quality was analyzed with the independent samples t test and difference in means with 95% CIs.

Results In total, 21 SR/MAs from 2010 and 127 from 2019 were included There was little to no difference between 2010 and 2019 in the proportion of SR/MAs at high or unclear overall risk of bias with ROBIS or with critically low methodological quality using AMSTAR 2. The only subdomain of ROBIS with a difference in proportion of SR/ MAs at high or unclear risk of bias between 2010 and 2019 was eligibility bias, with 27.3% more (95% CI, 5.42%-49.2%) in 2010 (66.7%) than 2019 (39.4%) (P = .02). There was a strong difference in proportion of PRISMA (t146 = 3.15; P = .002) and PRISMA-A (t146 = 2.46; P = .02) checklist items adequately reported between 2010 and 2019. The difference in mean proportion of PRISMA checklist items adequately reported was 3.8 items more (95% CI, 1.4-6.2 items more) in 2019 (mean SD, 15.7 5.1 items) than in 2010 (mean SD, 11.9 5.5), and of PRISMA-A checklist items adequately reported was 0.9 items more (95% CI, 0.2-1.6 items more) in 2019 (mean SD, 5.9 1.5 items) than in 2010 (mean SD, 5 1.9 items).

Conclusions While there was an improvement in the overall reporting quality of SR/MAs between 2010 and 2019, there was no improvement in the overall methodological quality as assessed by the ROBIS and AMSTAR 2 tools. This suggests a persistent potential for forming unreliable conclusions in SR/ MAs through a decade of dermatological research. These results therefore highlight the urgency with which efforts to improve SR/MA methodology should be undertaken.


  1. Siontis KC, Ioannidis JPA. Replication, duplication, and waste in a quarter million systematic reviews and meta- analyses. Circ Cardiovasc Qual Outcomes. 2018;11(12):e005212.
  2. Katsura M, Kuriyama A, Tada M, et al. High variability in results and methodological quality among overlapping systematic reviews on the same topics in surgery: a meta- epidemiological study. Br J Surg. 2021;108(12):1521-1529.
  3. Gómez-García F, Ruano J, Gay-Mimbrera J, et al. Most systematic reviews of high methodological quality on psoriasis interventions are classified as high risk of bias using ROBIS tool. J Clin Epidemiol. 2017;92:79-88.

Conflict of Interest Disclosures None reported.

Next from Peer Review Congress 2022

Inaugural Drummond Rennie Lecture: Bias, Spin, and Problems With Transparency of Research, Isabelle Boutron

Inaugural Drummond Rennie Lecture: Bias, Spin, and Problems With Transparency of Research, Isabelle Boutron

Peer Review Congress 2022

Isabelle Boutron

08 September 2022

Similar lecture

Quantifying Bias from Decoding Techniques in Natural Language Generation

Quantifying Bias from Decoding Techniques in Natural Language Generation


Mayukh Das
Mayukh Das and 1 other author

12 October 2022

Stay up to date with the latest Underline news!

Select topic of interest (you can select more than one)


  • All Lectures
  • For Librarians
  • Resource Center
  • Free Trial
Underline Science, Inc.
1216 Broadway, 2nd Floor, New York, NY 10001, USA

© 2023 Underline - All rights reserved