Level of intrateam agreement on candidate scores in a structured interview process

This study examined the intrateam reliability of a structured interview process used to help select students to a professional education program in physical therapy. Two hundred and twenty academically qualified applicants were interviewed by one of six two-member teams using a standardized format. Each member of a team independently rated the applicant on each of four performance skills using a 5-point ordinal scale; the maximal total score possible was 16 points. A weighted kappa (K_w) statistic was used to estimate intrateam agreement for both 1993 and 1994 on each performance skill and overall interview score. We observed wide variation in intrateam K_w scores for each of the four performance skills between the 1993 and 1994 admission cycles. According to K_w values for overall score agreement, five of the six interview teams either remained the same or increased at least one level between 1993 and 1994. Intrateam variability may be reduced by having team members review specific criteria used to rate a skill before the interview process commences each year.