Objective Reproducibility is a central tenet of research. Explicit reproducibility checks are made across different disciplines trying to assess the replicability of previously published studies. This scoping review aimed to synthesize the literature on reproducibility; describe its epidemiological characteristics, including how reproducibility is defined and assessed; and determine and compare estimates for reproducibility across different fields.

Design All English-language quantitative replication studies within the fields of economics, education, psychology, health sciences, and biomedicine published in 2018 or 2019 were included, as were studies that were explicitly self-described as a replication or a reproducibility study in which a previously published quantitative study is referred to and conducted again. Conference proceedings, commentaries, narrative reviews, systematic reviews, and clinical case studies were excluded. MEDLINE, Embase, PsycINFO, Cumulative Index of Nursing and Allied Health Literature (CINAHL), Education Source via EBSCOHost, ERIC, EconPapers, International Bibliography of the Social Sciences, and EconLit were searched. Retrieved documents were screened in duplicate against our inclusion criteria. The year of publication, number of authors, country of affiliation of the corresponding author, and funding were extracted. For individual replication studies, whether a registered protocol was used, whether there was contact between the replicating team and the original authors, what study design was used, and what the primary outcome was were extracted from each replication study. Finally, how replication was defined by the authors and whether the assessed study or studies successfully replicated a previous study based on this definition were also extracted. Extraction was done by 2 reviewers.

Results The search identified 11,224 unique documents, of which 47 were included (Figure 14). Most studies were related to either psychology (48.6%) or health sciences (23.7%). Among these 47 documents, 36 described a single replication study, while the remaining 11 reported at least 2 replications in the same paper. Less than half of the studies referred to a registered protocol. There was variability in the definitions of replication success, with studies related to psychology and health sciences tending to use a comparison of effect sizes to define replication success. Based on the definition used by the authors of each study, 95 of 177 studies (53.7%) achieved replication success.



Conclusions This study gives an overview of research across 5 disciplines that explicitly set out to replicate previous research. While estimates of reproducibility vary across fields in this modest sample, so too do norms in definitions used to define reproducibility.

Conflict of Interest Disclosures David Moher is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision of this abstract. No other disclosures were reported.

Additional Information Alixe Ménard is a patient author.