Predicting Penicillin Allergy: A United States Multicenter Retrospective Study

Background: Using the reaction history in logistic regression and machine learning (ML) models to predict penicillin allergy has been reported based on non-US data. Objective: We developed ML positive penicillin allergy testing prediction models from multisite US data. Methods: Retrospective data from 4 US-based hospitals were grouped into 4 datasets: enriched training (1:3 case-control matched cohort), enriched testing, nonenriched internal testing, and nonenriched external testing. ML algorithms were used for model development. We determined area under the curve (AUC) and applied the Shapley Additive exPlanations (SHAP) framework to interpret risk drivers. Results: Of 4777 patients (mean age 60 [standard deviation: 17] years; 68% women, 91% White, and 86% non-Hispanic) evaluated for penicillin allergy labels, 513 (11%) had positive penicillin allergy testing. Model input variables were frequently missing: immediate or delayed onset (71%), signs or symptoms (13%), and treatment (31%). The gradient-boosted model was the strongest model with an AUC of 0.67 (95% confidence interval [CI]: 0.57-0.77), which improved to 0.87 (95% CI: 0.73-1) when only cases with complete data were used. Top SHAP drivers for positive testing were reactions within the last year and reactions requiring medical attention; female sex and reaction of hives/urticaria were also positive drivers. Conclusions: An ML prediction model for positive penicillin allergy skin testing using US-based retrospective data did not achieve performance strong enough for acceptance and adoption. The optimal ML prediction model for positive penicillin allergy testing was driven by time since reaction, seek medical attention, female sex, and hives/urticaria.