Error rates of human reviewers during abstract screening in systematic reviews

Zhen Wang, Tarek Nayfeh, Jennifer Tetzlaff, Peter O’Blenis, Mohammad Hassan Murad

Research output: Contribution to journalReview articlepeer-review

6 Scopus citations


Background Automated approaches to improve the efficiency of systematic reviews are greatly needed. When testing any of these approaches, the criterion standard of comparison (gold standard) is usually human reviewers. Yet, human reviewers make errors in inclusion and exclusion of references. Objectives To determine citation false inclusion and false exclusion rates during abstract screening by pairs of independent reviewers. These rates can help in designing, testing and implementing automated approaches. Methods We identified all systematic reviews conducted between 2010 and 2017 by an evidence-based practice center in the United States. Eligible reviews had to follow standard systematic review procedures with dual independent screening of abstracts and full texts, in which citation inclusion by one reviewer prompted automatic inclusion through the next level of screening. Disagreements between reviewers during full text screening were reconciled via consensus or arbitration by a third reviewer. A false inclusion or exclusion was defined as a decision made by a single reviewer that was inconsistent with the final included list of studies. Results We analyzed a total of 139,467 citations that underwent 329,332 inclusion and exclusion decisions from 86 unique reviewers. The final systematic reviews included 5.48% of the potential references identified through bibliographic database search (95% confidence interval (CI): 2.38% to 8.58%). After abstract screening, the total error rate (false inclusion and false exclusion) was 10.76% (95% CI: 7.43% to 14.09%). Conclusions This study suggests important false inclusion and exclusion rates by human reviewers. When deciding the validity of a future automated study selection algorithm, it is important to keep in mind that the gold standard is not perfect and that achieving error rates similar to humans may be adequate and can save resources and time.

Original languageEnglish (US)
Article numbere0227742
JournalPloS one
Issue number1
StatePublished - Jan 1 2020

ASJC Scopus subject areas

  • General Biochemistry, Genetics and Molecular Biology
  • General Agricultural and Biological Sciences
  • General


Dive into the research topics of 'Error rates of human reviewers during abstract screening in systematic reviews'. Together they form a unique fingerprint.

Cite this