Using machine learning to improve the accuracy of patient deterioration predictions: Mayo Clinic Early Warning Score (MC-EWS)

Santiago Romero-Brufau, Daniel Whitford, Matthew G. Johnson, Joel Hickman, Bruce W. Morlan, Terry Therneau, James Naessens, Jeanne M. Huddleston

Research output: Contribution to journalArticlepeer-review


Objective: We aimed to develop a model for accurate prediction of general care inpatient deterioration. Materials and Methods: Training and internal validation datasets were built using 2-year data from a quaternary hospital in the Midwest. Model training used gradient boosting and feature engineering (clinically relevant interactions, time-series information) to predict general care inpatient deterioration (resuscitation call, intensive care unit transfer, or rapid response team call) in 24 hours. Data from a tertiary care hospital in the Southwest were used for external validation. C-statistic, sensitivity, positive predictive value, and alert rate were calculated for different cutoffs and compared with the National Early Warning Score. Sensitivity analysis evaluated prediction of intensive care unit transfer or resuscitation call. Results: Training, internal validation, and external validation datasets included 24 500, 25 784 and 53 956 hospitalizations, respectively. The Mayo Clinic Early Warning Score (MC-EWS) demonstrated excellent discrimination in both the internal and external validation datasets (C-statistic = 0.913, 0.937, respectively), and results were consistent in the sensitivity analysis (C-statistic = 0.932 in external validation). At a sensitivity of 73%, MC-EWS would generate 0.7 alerts per day per 10 patients, 45% less than the National Early Warning Score. Discussion: Low alert rates are important for implementation of an alert system. Other early warning scores developed for the general care ward have achieved lower discrimination overall compared with MC-EWS, likely because MC-EWS includes both nursing assessments and extensive feature engineering. Conclusions: MC-EWS achieved superior prediction of general care inpatient deterioration using sophisticated feature engineering and a machine learning approach, reducing alert rate.

Original languageEnglish (US)
Pages (from-to)1207-1215
Number of pages9
JournalJournal of the American Medical Informatics Association
Issue number6
StatePublished - Jun 1 2021


  • clinical deterioration
  • early warning score
  • machine learning

ASJC Scopus subject areas

  • Health Informatics


Dive into the research topics of 'Using machine learning to improve the accuracy of patient deterioration predictions: Mayo Clinic Early Warning Score (MC-EWS)'. Together they form a unique fingerprint.

Cite this