technical paper

Peer Review Congress 2022

September 10, 2022

Chicago, United States

Characteristics of Studies of Research Reproducibility in Economics, Education, Psychology, Health Sciences, and Biomedicine: A Scoping Review


reproducible research

quality of the literature

open science

Objective Reproducibility is a central tenet of research. Explicit reproducibility checks are made across different disciplines trying to assess the replicability of previously published studies. This scoping review aimed to synthesize the literature on reproducibility; describe its epidemiological characteristics, including how reproducibility is defined and assessed; and determine and compare estimates for reproducibility across different fields.

Design All English-language quantitative replication studies within the fields of economics, education, psychology, health sciences, and biomedicine published in 2018 or 2019 were included, as were studies that were explicitly self-described as a replication or a reproducibility study in which a previously published quantitative study is referred to and conducted again. Conference proceedings, commentaries, narrative reviews, systematic reviews, and clinical case studies were excluded. MEDLINE, Embase, PsycINFO, Cumulative Index of Nursing and Allied Health Literature (CINAHL), Education Source via EBSCOHost, ERIC, EconPapers, International Bibliography of the Social Sciences, and EconLit were searched. Retrieved documents were screened in duplicate against our inclusion criteria. The year of publication, number of authors, country of affiliation of the corresponding author, and funding were extracted. For individual replication studies, whether a registered protocol was used, whether there was contact between the replicating team and the original authors, what study design was used, and what the primary outcome was were extracted from each replication study. Finally, how replication was defined by the authors and whether the assessed study or studies successfully replicated a previous study based on this definition were also extracted. Extraction was done by 2 reviewers.

Results The search identified 11,224 unique documents, of which 47 were included (Figure 14). Most studies were related to either psychology (48.6%) or health sciences (23.7%). Among these 47 documents, 36 described a single replication study, while the remaining 11 reported at least 2 replications in the same paper. Less than half of the studies referred to a registered protocol. There was variability in the definitions of replication success, with studies related to psychology and health sciences tending to use a comparison of effect sizes to define replication success. Based on the definition used by the authors of each study, 95 of 177 studies (53.7%) achieved replication success.

Conclusions This study gives an overview of research across 5 disciplines that explicitly set out to replicate previous research. While estimates of reproducibility vary across fields in this modest sample, so too do norms in definitions used to define reproducibility.

Conflict of Interest Disclosures David Moher is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision of this abstract. No other disclosures were reported.

Additional Information Alixe Ménard is a patient author.


SlidesTranscript English (automatic)

Next from Peer Review Congress 2022

Data Sharing and Reanalysis for Main Studies Assessed by the European Medicines Agency | VIDEO
technical paper

Data Sharing and Reanalysis for Main Studies Assessed by the European Medicines Agency | VIDEO

Peer Review Congress 2022

Maximilian Siebert

10 September 2022

Similar lecture

From creation to consumable knowledge: supporting research workflows in an open infrastructure
technical paper

From creation to consumable knowledge: supporting research workflows in an open infrastructure

UKSG Annual Conference 2021

Tamir Borensztajn
Tamir Borensztajn

13 April 2021

Stay up to date with the latest Underline news!

Select topic of interest (you can select more than one)


  • All Lectures
  • For Librarians
  • Resource Center
  • Free Trial
Underline Science, Inc.
1216 Broadway, 2nd Floor, New York, NY 10001, USA

© 2023 Underline - All rights reserved