Objective Researcher and stakeholder interaction in the development of new tools to inform evidence-based medicine is a key factor associated with the impact such tools can have. Currently, there no risk of bias (RoB) tools to assess reviews including network meta-analyses (NMAs; Table 76). The objectives of this research were to identify items for potential inclusion in the tool through a methodological systematic review, conduct a Delphi survey, and conduct a stakeholder survey.

Design An international steering committee developed a protocol following the methods by Whiting et al1 for tool development and made conceptual decisions about the tool’s structure. Tools, articles, and editorial standards presenting items related to bias, reporting, or quality in NMAs were included. General systematic review items were excluded. Experts for the Delphi survey were identified using a purposive sampling. Respondents were asked to rate whether items should be included. All agreed-upon items (defined as 70% agreement) and additional or aggregated items were included in a second round of the survey. The stakeholder survey contained 22 questions and was disseminated anonymously through social media and professional networks.

Results The search returned 3599 citations, from which 59 articles were included, yielding 99 items.2 Of these, 22 items were deemed eligible and were entered into a Delphi survey in which 26 respondents completed round 1 and 22 completed round 2.3 Seven items did not reach consensus in round 2 of the Delphi survey. After further refinement by the committee, 16 items were worded as signaling questions and categorized into 3 domains in the tool. An elaboration and explanation document was drafted. A total of 298 stakeholders participated in the survey; 75% indicated that their organization produced NMAs, and 78% showed high interest in the tool.3 Most stakeholders (84%) who responded to the survey reported they would use the tool to assess an NMA if they had received adequate training. Most stakeholders and Delphi panelists preferred a tool to assess both bias in NMA results and authors’ conclusions. After examining the results of these studies, the committee recommended that the tool be used with the ROBIS tool for assessing biases in systematic reviews using a domain-based structure and to assess both NMA results and authors’ conclusions. Response bias in this sample was a major limitation, as stakeholders and Delphi panelists working in higher-income countries were more represented.

Conclusions These studies inform the development of the first tool to assess RoB in NMAs. In the future, the tool will be pilot tested in different user groups.

References 1. Whiting P, Wolff R, Mallett S, Simera I, Savović J. A proposed framework for developing quality assessment tools. Syst Rev. 2017;6(1):1-9. doi:10.1186/s13643-017-0604-6 2. Lunny C, Veroniki AA, Hutton B, et al. Methodological review to develop a list of bias items used to assess reviews incorporating network meta-analysis. Submitted to Research Synthesis Methods, 2022. 3. Lunny C, Tricco AC, Veroniki AA, et al. Stakeholder opinions on the structure and development of a new risk of bias tool to assess systematic reviews with network meta- analysis (RoB NMA tool): a cross-sectional survey. Research Square. Preprint posted online February 22, 2022. doi:10.21203/rs.3.rs-1324758/v1

Conflict of Interest Disclosures Brian Hutton has previously received honoraria from Eversana Incorporated for the provision of methodologic advice related to the conduct of systematic reviews and meta-analyses. Ian R. White was supported by the Medical Research Council (programme MC_UU_00004/06). Julian P. T. Higgins is a National Institute for Health Research (NIHR) Senior Investigator (NF-SI-0617-10145), is supported by NIHR Bristol Biomedical Research Centre at University Hospitals Bristol and Weston National Health Service (NHS) Foundation Trust and the University of Bristol, is supported by the NIHR Applied Research Collaboration West at University Hospitals Bristol and Weston NHS Foundation Trust, and is a member of the Medical Research Council (MRC) Integrative Epidemiology Unit at the University of Bristol. Andrea C. Tricco holds a Tier 2 Canada Research Chair in Knowledge Synthesis. No other disclosures were reported. Funding/Support This study was supported through a 2020 Canadian Institutes of Health Research Project Grant (2021-2024; ID 174998).

Role of the Funder/Sponsor The funder had no role in the submitted work.

Acknowledgments We acknowledge the panelists in the Delphi study: Simon Turner, Dimitris Mavridis, Virginia Chiocchia, Ian Shrier, Adriani Nikolakopoulou, Dan Jackson, Richard Riley, Becky Turner, Bruno Roza da Costa, Petros Pechlivanoglou, Steve Kanters, Ferran Catala-Lopez, Tianjing Li, Matt Page, Chris Cameron, Anna Chaimani, Kerry Dwan, Audrey Beliveau, Kristian Thorlund, Jonathan Sterne, Cinzia del Giovane, Guido Schwarzer, Jo McKenzie, Lehana Thabane, Theodoros Papakonstantinou, Isabelle Boutron, Sharon Straus, and Jenn Watt. We also thank the participants of the stakeholder survey, who were anonymous.

Additional Information The views expressed in this article are those of the authors and do not necessarily represent those of the NHS, the NIHR, the MRC, or the Department of Health and Social Care.