Multireader evaluation of radiologist performance for COVID-19 detection on emergency department chest radiographs

Judy W. Gichoya, Priyanshu Sinha, Melissa Davis, Jeffrey W. Dunkle, Scott A. Hamlin, Keith D. Herr, Carrie N. Hoff, Haley P. Letter, Christopher R. McAdams, Gregory D. Puthoff, Kevin L. Smith, Scott D. Steenburg, Imon Banerjee, Hari Trivedi

Research output: Contribution to journalArticlepeer-review


Background: Chest radiographs (CXR) are frequently used as a screening tool for patients with suspected COVID-19 infection pending reverse transcriptase polymerase chain reaction (RT-PCR) results, despite recommendations against this. We evaluated radiologist performance for COVID-19 diagnosis on CXR at the time of patient presentation in the Emergency Department (ED). Materials and methods: We extracted RT-PCR results, clinical history, and CXRs of all patients from a single institution between March and June 2020. 984 RT-PCR positive and 1043 RT-PCR negative radiographs were reviewed by 10 emergency radiologists from 4 academic centers. 100 cases were read by all radiologists and 1927 cases by 2 radiologists. Each radiologist chose the single best label per case: Normal, COVID-19, Other – Infectious, Other – Noninfectious, Non-diagnostic, and Endotracheal Tube. Cases labeled with endotracheal tube (246) or non-diagnostic (54) were excluded. Remaining cases were analyzed for label distribution, clinical history, and inter-reader agreement. Results: 1727 radiographs (732 RT-PCR positive, 995 RT-PCR negative) were included from 1594 patients (51.2% male, 48.8% female, age 59 ± 19 years). For 89 cases read by all readers, there was poor agreement for RT-PCR positive (Fleiss Score 0.36) and negative (Fleiss Score 0.46) exams. Agreement between two readers on 1638 cases was 54.2% (373/688) for RT-PCR positive cases and 71.4% (679/950) for negative cases. Agreement was highest for RT-PCR negative cases labeled as Normal (50.4%, n = 479). Reader performance did not improve with clinical history or time between CXR and RT-PCR result. Conclusion: At the time of presentation to the emergency department, emergency radiologist performance is non-specific for diagnosing COVID-19.

Original languageEnglish (US)
Pages (from-to)77-82
Number of pages6
JournalClinical Imaging
StatePublished - Feb 2022


  • COVID-19
  • CXR
  • Chest X-ray
  • Coronavirus

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging


Dive into the research topics of 'Multireader evaluation of radiologist performance for COVID-19 detection on emergency department chest radiographs'. Together they form a unique fingerprint.

Cite this