TY - JOUR
T1 - Using machine learning to improve the accuracy of patient deterioration predictions
T2 - Mayo Clinic Early Warning Score (MC-EWS)
AU - Romero-Brufau, Santiago
AU - Whitford, Daniel
AU - Johnson, Matthew G.
AU - Hickman, Joel
AU - Morlan, Bruce W.
AU - Therneau, Terry
AU - Naessens, James
AU - Huddleston, Jeanne M.
N1 - Publisher Copyright:
© 2021 The Author(s). Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved.
PY - 2021/6/1
Y1 - 2021/6/1
N2 - Objective: We aimed to develop a model for accurate prediction of general care inpatient deterioration. Materials and Methods: Training and internal validation datasets were built using 2-year data from a quaternary hospital in the Midwest. Model training used gradient boosting and feature engineering (clinically relevant interactions, time-series information) to predict general care inpatient deterioration (resuscitation call, intensive care unit transfer, or rapid response team call) in 24 hours. Data from a tertiary care hospital in the Southwest were used for external validation. C-statistic, sensitivity, positive predictive value, and alert rate were calculated for different cutoffs and compared with the National Early Warning Score. Sensitivity analysis evaluated prediction of intensive care unit transfer or resuscitation call. Results: Training, internal validation, and external validation datasets included 24 500, 25 784 and 53 956 hospitalizations, respectively. The Mayo Clinic Early Warning Score (MC-EWS) demonstrated excellent discrimination in both the internal and external validation datasets (C-statistic = 0.913, 0.937, respectively), and results were consistent in the sensitivity analysis (C-statistic = 0.932 in external validation). At a sensitivity of 73%, MC-EWS would generate 0.7 alerts per day per 10 patients, 45% less than the National Early Warning Score. Discussion: Low alert rates are important for implementation of an alert system. Other early warning scores developed for the general care ward have achieved lower discrimination overall compared with MC-EWS, likely because MC-EWS includes both nursing assessments and extensive feature engineering. Conclusions: MC-EWS achieved superior prediction of general care inpatient deterioration using sophisticated feature engineering and a machine learning approach, reducing alert rate.
AB - Objective: We aimed to develop a model for accurate prediction of general care inpatient deterioration. Materials and Methods: Training and internal validation datasets were built using 2-year data from a quaternary hospital in the Midwest. Model training used gradient boosting and feature engineering (clinically relevant interactions, time-series information) to predict general care inpatient deterioration (resuscitation call, intensive care unit transfer, or rapid response team call) in 24 hours. Data from a tertiary care hospital in the Southwest were used for external validation. C-statistic, sensitivity, positive predictive value, and alert rate were calculated for different cutoffs and compared with the National Early Warning Score. Sensitivity analysis evaluated prediction of intensive care unit transfer or resuscitation call. Results: Training, internal validation, and external validation datasets included 24 500, 25 784 and 53 956 hospitalizations, respectively. The Mayo Clinic Early Warning Score (MC-EWS) demonstrated excellent discrimination in both the internal and external validation datasets (C-statistic = 0.913, 0.937, respectively), and results were consistent in the sensitivity analysis (C-statistic = 0.932 in external validation). At a sensitivity of 73%, MC-EWS would generate 0.7 alerts per day per 10 patients, 45% less than the National Early Warning Score. Discussion: Low alert rates are important for implementation of an alert system. Other early warning scores developed for the general care ward have achieved lower discrimination overall compared with MC-EWS, likely because MC-EWS includes both nursing assessments and extensive feature engineering. Conclusions: MC-EWS achieved superior prediction of general care inpatient deterioration using sophisticated feature engineering and a machine learning approach, reducing alert rate.
KW - clinical deterioration
KW - early warning score
KW - machine learning
UR - http://www.scopus.com/inward/record.url?scp=85108303519&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85108303519&partnerID=8YFLogxK
U2 - 10.1093/jamia/ocaa347
DO - 10.1093/jamia/ocaa347
M3 - Article
C2 - 33638343
AN - SCOPUS:85108303519
SN - 1067-5027
VL - 28
SP - 1207
EP - 1215
JO - Journal of the American Medical Informatics Association
JF - Journal of the American Medical Informatics Association
IS - 6
ER -