Over the past several years, across the globe, there has been an increase in people seeking care in emergency departments (EDs). ED resources, including nurse staffing, are strained by such increases in patient volume. Accurate forecasting of incoming patient volume in emergency departments (ED) is crucial for efficient utilization and allocation of ED resources. Working with a suburban ED in the Pacific Northwest, we developed a tool powered by machine learning models, to forecast ED arrivals and ED patient volume to assist end-users, such as ED nurses, in resource allocation. In this paper, we discuss the results from our predictive models, the challenges, and the learnings from users' experiences with the tool in active clinical deployment in a real world setting.
Emergency departments (EDs) are a critical component of
the healthcare infrastructure and ED crowding is a global
problem. In 2016 there were over 140 million ED visits
in the US (NCHS 2009). The number of ED patients is
growing and, according to US data, this increase has
outpaced population growth for the last 20 years (Weiss et al.
2006). As a result, EDs are increasingly crowded
(McCarthy et al. 2008) and ED overcrowding has been linked
to decreased quality of care (Schull et al. 2003) (Hwang
et al. 2006), increased costs
If ED load could be accurately predicted, staffing could
be adjusted to optimize patient care. The ability to predict
the number of patients seeking ED care on a given day is
essential to optimizing nurse staffing
The availability of accessible data and computational resources has enabled the application of machine learning (ML) to healthcare at an unprecedented scale (Krumholz 2014). While several research groups have developed ML predictions on retrospective and static ED data, operationalized ML solutions in the ED are rare. Chase et al. developed a novel indicator of a busy ED: a care utilization ratio (Chase et al. 2012). The authors report that the prediction of this ratio, which incorporates new ED arrivals, number of patients triaged, and physician capacity, provides a robust indicator of ED crowding. McCarthy et al. utilized a Poisson regression model to predict demand for ED services (McCarthy et al. 2008). They determined that after accounting for temporal, weather, and patient-related factors (hour of day is most important), ED arrivals during one hour had little to no association with the number of ED arrivals the following hour. Jones et al. (Jones et al. 2008) explored seasonal autoregressive integrated moving average (SARIMA), time series regression, exponential smoothing, and artificial neural network models to forecast daily patient volumes and also identified seasonal and weekly patterns in ED utilization.
The goal of our work was to optimize ED operations by accurately predicting ED arrivals and ED patient census to facilitate staffing optimization to better manage the influxes and patterns of ED patients to provide safe and timely care. Here we describe our approach to building the prediction models and we describe the metrics we used to evaluate the model accuracy.
There are two distinct yet related ED load optimization problems that we address in this work, as described below: ED Census ED census is defined as the total number of patients in the ED at a specified time. ED census includes patients in the waiting room, in triage, those receiving care, and those awaiting ED disposition: hospital admission, discharge, or transfer. ED census is a ”snapshot” of ED utilization and includes elements related to ED arrivals as well as ED throughput. Predicting ED census can serve to inform both short-term (minutes to hours) operations, such as reassigning staff or diverting ambulance arrivals and longerterm (hours or longer) administrative decisions, such as calling in additional staff or sending staff members home early. We formulated this problem as a prediction of ED census at t + 2 hours, t + 4 hours, and t + 8 hours, where t is the prediction time. In production, these predictions are
ED Arrivals and Acuity ED arrivals reflect the number of individual patients who are arriving at the ED over a period of time. Arrivals can be described by the acuity level of the individual patient, an indicator of illness or injury severity assessed by nursing staff at the time of patient triage (Gilboy et al. 2012). Predictions of patient volume by acuity level can further inform staffing needs - higher acuity patients tend to have greater intensity of staff and resource needs. Similar to the Census prediction, we framed the Arrivals prediction by acuity for 2, 4, and 8 hour forecasting. To accommodate different patterns in the acuity of patients, we built models for each individual acuity level.
For both Census and Arrivals we include temporal features such as hour of day, day of week, month of year, and quarter of year. To include the unique variations in census and arrival patterns in the evening compared to the morning as well as weekend versus weekday patterns, we included corresponding binary variables.
While ED census or ED arrival may be independent from one hour to the next, we use the current ED census trend to inform future ED census. To include signals for the current census trends in ED in our predictive models, we determine the slope from the census values in the previous 1 hour for every 15 minute intervals. In addition, we weighted values from these 15 minute intervals to that more recent values had higher weights. The census at t 15 minutes, t 30 minutes, t 45 minutes, and t 60 minutes is weighted with 2, 0.5, 0.25, and 0.05 respectively. The weights were chosen empirically based on the performance metrics of the model. Similar to Census, the arrivals for the Arrival prediction are weighted in the same way. The final set of features is shown in Table 1.
The data for the experiments came from a suburban level three trauma center at a hospital in the Pacific Northwest with > 60; 000 annual ED visits. The ED comprises multiple treatment spaces including 40 acute treatment rooms and 4 trauma rooms for the resuscitation of critically ill patients. Individuals are registered at the time of entry to the ED and all registered ED patients were included in this analysis. ED encounters occurring between January 2014 through January 2018 were included in the experiments. The dataset included electronic health record (EHR) data elements such as time, date, location, chief complaint, acuity score, vital signs, and others. This included 205; 929 ED encounters, of which 199; 957 encounters documented patient acuity. ESI is a categorical variable representing patient acuity (based on vital signs and symptoms) where ESI 1 connotes highest urgency and ESI 5 the lowest urgency (Gilboy et al. 2012). We grouped these into three categories reflecting emergent (ESI 1 or 2), urgent (ESI 3), and non urgent (ESI 4 or 5). The distribution of of the encounters split by ESI groups is shown in Table 2.
Multiple regression models were evaluated for both Census and Arrivals predictions. We choose to use a Generalized Linear Model with Poisson Regression (GLM) for its simplicity and capability to model count data (Gardner, Mulvey, and Shaw 1995). We included regularization variants of GLM that include Lasso, Ridge, and Elastic Net for validation. We also included linear Gradient Boosting Machine (GBM) due to its robustness to missing data and predictive power (Friedman 2001). We used the average arrivals and census values at that same time point from the prior two years as our baseline. We used scikit-learn package available in Python 3.6 to implement all models.
We evaluate the performance of our models using root mean squared error (RMSE) and mean absolute error (MAE) (Verbiest, Vermeulen, and Teredesai 2014) which are suitable metrics for regression. However, the real utility of ED load prediction is in staffing optimization. Most common midsize US ED departments have an ED patient to nurse ratio of 4:1. Based on this, we devised an additional metric: we determined the percentage of times the model prediction is within a threshold of 4 (Absolute Error <= 4). Furthermore, we also calculate the percentage of times that the model is accurate to within 70% of the actual value (Accuracy>70%). These additional metrics frame the models performances in terms of their effects on user workflows and provide a simple understanding of the model performance under the system constraints while ensuring interpretability to end users.
Furthermore, combining these models with a model management process to detect changes in model performance or shifts in underlying patient distributions, prevails as novel work. Model management is an iterative process that includes monitoring and evaluating model performance to detect subtle (or unsubtle) changes in the underlying distribution of the data, permitting investigation and, if necessary, model re-training. We have implemented a workflow for automatic model monitoring; the overview of this is represented in Figure 2. As part of this workflow we created a user friendly dashboard to track the model performance and distributions, an example visual can be seen in Figure 3.
Data from January 2014 to October 2017 was used to train the models and data from November 2017 to January 2018 was used to test the models. The performance metrics of the census models for 2 hour prediction are shown in Table 4. The Gradient Boosting Method (GBM) performed the better among the set for all metrics which we believe is due to its robustness to the sparsity in the data. The 4 and 8 hours GBM census model MAEs are 4:0739 and 4:2960 respectively. The metric (Accuracy > 70%) shows that GBM is accurate 81.52% of times for a prediction within 70% of actual census. And, the GBM is accurate 72.90% time for a prediction within a value of 4 of actual census.
For arrival models, we built 9 models, one for each acuity level and for each 2, 4 and 8 hours prediction. We observed that the gradient boosting model performed better than other models and the baseline for Emergent acuity encounters, where as for Urgent, Non-urgent acuity GLM models performed better. The results are shown in Table 3. The absolute error and accuracy were only available for a subset of models. We observe that the MAE and RMSE for all models across different levels of acuity is similar if we consider 2 hour and 4 hour windows. However, the performance goes down if we consider 8 hour windows. This is not unexpected since trends can greatly vary across longer time spans e.g., compare ED trends at 2 am vs. 10 am.
A key differentiator of the work that we present here is that our prediction models were fully operationalized into the clinical workflow, that of the ED charge nurse. Through collaborative design and planning sessions with ED nurses and other health system stakeholders, we developed an ED dashboard to surface the results of our predictions. Prediction based tools are often beset by difficulties in end-user understanding of probability based results (Jeffery et al. 2017). Part of the solution to this problem is the early incorporation of end-user feedback and open discussions around tool utility.
Our dashboard was deployed for 6 months as part of pilot in a large suburban ED. As part of this pilot, data quality was monitored continuously and multiple ML models were scored at 15 minute intervals. End-user training was conducted during the pilot period. During this period charge nurses completed forms at the conclusion of each shift documenting their use of the dashboard and any actions the dashboard prompted (such as calling in additional staff for projected high load or sending staff home early for projected low load). In addition to the potential impact on nurse staffing, accurately forecasting ED arrivals and census may optimize care delivery in other ways - such as reducing waiting times, ED length of stay, and rates of patients leaving without being seen. These additional key performance indicators (KPIs) were also be evaluated to determine the clinical utility of the deployed predictions. The iterative nature of this approach speaks to the engagement needs of the clinical end-users and the imperative of operationalizing machine learning in healthcare. While accurate predictions are key to implementation success and end-user adoption, simple metrics such as prevalence of accuracy above a threshold (Accuracy > 70%) will help health system stakeholders evaluate the impact and maintenance cost over a period of time.
Our work demonstrates that subtle patterns in exogenous and endogenous variability in patient flow can be utilized to predict, with high accuracy, ED patient arrivals and census. Deployment of ML-based predictive models into a complex clinical workflow is challenging. However, predicting ED census is an ideal ML healthcare problem to study for several reasons. First, predicting ED census every 15 minutes across 12 different models allows for 1; 152 predictions daily. Each prediction is clearly falsifiable with a measurable outcome (the actual number of arrivals and patient census), and the follow-up interval is short (e.g., one must only wait 8 hours to determine the accuracy of all predictions). Second, many healthcare ML models are degraded by data censoring; for example, when predicting 30-day hospital readmissions, patients may avoid readmission, they may be readmitted at another facility. Additionally, according to the work of Jeffery and colleagues, prediction based tools are most useful when prompt decision and action are warranted by the endusers (Jeffery et al. 2017), however in some cases, such as predicting hospital readmissions, the action of the clinician can alter the outcome, thus making the prediction appear erroneous. In predicting ED load, there are no actions that the users can take (other than the ED going on diversion status, which is done only seldom) that will alter the number of arrivals or census. The large number of predictions, the short follow-up interval, and the availability of ’perfect information’ about outcomes (akin to ’perfect information’ games like chess) makes ED load prediction an ideal place to optimize model management processes.
We are continuing to improve the performance and
clinical utility of these models by integrating additional data
sources into our predictions. These sources can include
events or include: local weather data, local sporting events,
local traffic, local emergency medical services (EMS)
activity, and Google Trends searches. We plan to further improve
this solution by providing interpretability for the predictions
to help ED staff make informed decisions.
Carter, E. J.; Pouch, S. M.; and Larson, E. L. 2014. The relationship between emergency department crowding and patient outcomes: a systematic review. volume 46, 106–115. Chase, V. J.; Cohn, A. E.; Peterson, T. A.; and Lavieri, M. S. 2012. Predicting emergency department volume using forecasting methods to create a “surge response” for noncrisis events. volume 19, 569–576. Wiley Online Library. Friedman, J. H. 2001. Greedy function approximation: a gradient boosting machine. 1189–1232. JSTOR. Gardner, W.; Mulvey, E. P.; and Shaw, E. C. 1995. Regression analyses of counts and rates: Poisson, overdispersed Poisson, and negative binomial models. volume 118, 392. American Psychological Association.
Gilboy, N.; Tanabe, P.; Travers, D.; Rosenau, A. M.; et al. 2012. Emergency Severity Index (ESI): a triage tool for emergency department care, version 4. 12–0014. Hoot, N. R.; LeBlanc, L. J.; Jones, I.; Levin, S. R.; Zhou, C.; Gadd, C. S.; and Aronsky, D. 2009. Forecasting Emergency Department Crowding: A Prospective, Real-time Evaluation. volume 16, 338–345. ISSN 1067-5027, 1527-974X. doi:10.1197/jamia.M2772.
Hwang, U.; Richardson, L. D.; Sonuyi, T. O.; and Morrison, R. S. 2006. The effect of emergency department crowding on the management of pain in older adults with hip fracture. volume 54, 270–275. Wiley Online Library.
Jeffery, A. D.; Novak, L. L.; Kennedy, B.; Dietrich, M. S.; and Mion, L. C. 2017. Participatory design of probabilitybased decision support tools for in-hospital nurses. volume 24, 1102–1110. Oxford University Press.
Jenkins, M. G.; Rocke, L. G.; McNicholl, B. P.; and Hughes, D. M. 1998. Violence and verbal abuse against staff in accident and emergency departments: a survey of consultants in the UK and the Republic of Ireland. volume 15, 262–265. British Association for Accident and Emergency Medicine. Jones, S. S.; Evans, R. S.; Allen, T. L.; Thomas, A.; Haug, P. J.; Welch, S. J.; and Snow, G. L. 2009. A multivariate time series approach to modeling and forecasting demand in the emergency department. volume 42, 123–139. Elsevier. Jones, S. S.; Thomas, A.; Evans, R. S.; Welch, S. J.; Haug, P. J.; and Snow, G. L. 2008. Forecasting daily patient volumes in the emergency department. volume 15, 159–170. Wiley Online Library.
Krumholz, H. M. 2014. Big Data And New Knowledge In Medicine: The Thinking, Training, And Tools Needed For A Learning Health System. volume 33, 1163–1170. ISSN 0278-2715, 1544-5208. doi:10.1377/hlthaff.2014.0053. McCarthy, M. L.; Zeger, S. L.; Ding, R.; Aronsky, D.; Hoot, N. R.; and Kelen, G. D. 2008. The challenge of predicting demand for emergency department services. volume 15, 337–346. Wiley Online Library.
NCHS. 2009. National hospital Ambulatory Medical Care Survey: 2015 emergency department summary tables. URL https://www.cdc.gov/nchs/data/nhamcs/web tables/2015 ed web tables.pdf.
Peck, J. S.; Benneyan, J. C.; Nightingale, D. J.; and Gaehde, S. A. 2012. Predicting Emergency Department Inpatient Admissions to Improve Same-day Patient Flow: PREDICTING ED INPATIENT ADMISSIONS. volume 19, E1045–E1054. ISSN 10696563. doi:10.1111/j.1553-2712.2012.01435.x. Pham, J. C.; Ho, G. K.; Hill, P. M.; McCarthy, M. L.; and Pronovost, P. J. 2009. National study of patient, visit, and hospital characteristics associated with leaving an emergency department without being seen: predicting LWBS. volume 16, 949–955. Wiley Online Library.
Schull, M. J.; Morrison, L. J.; Vermeulen, M.; and Redelmeier, D. A. 2003. Emergency department overcrowding and ambulance transport delays for patients with chest pain. volume 168, 277–283. Can Med Assoc.
Weiss, A. J.; Wier, L. M.; Stocks, C.; and Blanchard, J. 2006. Overview of emergency department visits in the United States, 2011: statistical brief# 174. Agency for Healthcare Research and Quality (US), Rockville (MD).