Much ado about differences: why expert-novice comparisons add little to the validity argument

Research output: Contribution to journalArticlepeer-review

42 Scopus citations


One approach to validating assessment scores involves evaluating the ability of scores to discriminate among groups who differ in a specific characteristic, such as training status (in education) or disease state (in clinical applications). Such known-groups comparison studies provide validity evidence of “relationships with other variables.” The typical education research study might compare scores between staff physicians and postgraduate trainees with the hypothesis that those with more advanced training (the “experts”) will have higher scores than those less advanced (the “novices”). However, such comparisons are too nonspecific to support clear conclusions, and expert-novice comparisons (and known-groups comparisons in general) thus contribute little to the validity argument. The major flaw is the problem of confounding: there are multiple plausible explanations for any observed between-group differences. The absence of hypothesized differences would suggest a serious flaw in the validity argument, but the confirmation of such differences adds little. As such, accurate known-groups discrimination may be necessary, but will never be sufficient, to support the validity of scores. This article elaborates on this and other problems with the known-groups comparison that limit its utility as a source of validity evidence.

Original languageEnglish (US)
Pages (from-to)829-834
Number of pages6
JournalAdvances in Health Sciences Education
Issue number3
StatePublished - Aug 22 2015


  • Assessment
  • Data collection
  • Data interpretation, statistical
  • Evaluation
  • Medical education
  • Reliability
  • Validation Studies

ASJC Scopus subject areas

  • Education


Dive into the research topics of 'Much ado about differences: why expert-novice comparisons add little to the validity argument'. Together they form a unique fingerprint.

Cite this