=Paper=
{{Paper
|id=Vol-2884/paper_114
|storemode=property
|title=Emergency Department Optimization and Load Prediction in Hospitals
|pdfUrl=https://ceur-ws.org/Vol-2884/paper_114.pdf
|volume=Vol-2884
|authors=Karthik K. Padthe,Vikas Kumar,Carly M. Eckert,Nicholas M. Mark,Anam Zahid,Muhammad Aurangzeb Ahmad,Ankur Teredesai
|dblpUrl=https://dblp.org/rec/conf/aaaifs/PadtheKEMZAT20
}}
==Emergency Department Optimization and Load Prediction in Hospitals==
https://ceur-ws.org/Vol-2884/paper_114.pdf
Emergency Department Optimization and Load Prediction in Hospitals
Karthik K. Padthe,1 Vikas Kumar,1 Carly M. Eckert MD MPH,1,2 Nicholas M. Mark MD,1,3
Anam Zahid,1 Muhammad Aurangzeb Ahmad,1,4 Ankur Teredesai,1,5
1
KenSci Inc, Seattle, WA
2
Department of Epidemiology, University of Washington
3
Swedish Medical Center, Seattle, WA
4
Department of Computer Science, University of Washington - Bothell
5
Department of Computer Science, University of Washington - Tacoma
{karthik, vikas, carly, drnick, anam, muhammad, ankur}@kensci.com
Abstract
Over the past several years, across the globe, there has been
an increase in people seeking care in emergency departments
(EDs). ED resources, including nurse staffing, are strained
by such increases in patient volume. Accurate forecasting of
incoming patient volume in emergency departments (ED) is
crucial for efficient utilization and allocation of ED resources.
Working with a suburban ED in the Pacific Northwest, we de-
veloped a tool powered by machine learning models, to fore-
cast ED arrivals and ED patient volume to assist end-users,
such as ED nurses, in resource allocation. In this paper, we
discuss the results from our predictive models, the challenges,
and the learnings from users’ experiences with the tool in ac-
Figure 1: Overview of set of prediction models that can help
tive clinical deployment in a real world setting. optimize Emergency Department efficiency.
Introduction
Emergency departments (EDs) are a critical component of
2014). Multiple factors influence ED crowding including the
the healthcare infrastructure and ED crowding is a global
number of new patients coming to the ED (arrivals), how
problem. In 2016 there were over 140 million ED visits
severely sick or injured patients are (acuity), and the total
in the US (NCHS 2009). The number of ED patients is
number of patients in the ED (census). Each of these factors
growing and, according to US data, this increase has out-
have both stochastic and deterministic components (Jones
paced population growth for the last 20 years (Weiss et al.
et al. 2009) (Jones et al. 2008) and are influenced by both ex-
2006). As a result, EDs are increasingly crowded (Mc-
ogenous (e.g., vehicle crashes) and endogenous factors (e.g.,
Carthy et al. 2008) and ED overcrowding has been linked
hospital processes). In order to optimize ED flow, it is there-
to decreased quality of care (Schull et al. 2003) (Hwang
fore necessary to integrate multiple predictions as shown in
et al. 2006), increased costs (Bayley et al. 2005), and in-
Figure 1.
creased patient dissatisfaction (Jenkins et al. 1998). Using
machine learning models to predict ED load could amelio- If ED load could be accurately predicted, staffing could
rate the adverse effects of crowding, and multiple strategies be adjusted to optimize patient care. The ability to predict
have been proposed, including forecasting future crowding the number of patients seeking ED care on a given day is
(Hoot et al. 2009), predicting the likelihood of inpatient ad- essential to optimizing nurse staffing (Batal et al. 2001).
mission (Peck et al. 2012), and predicting the likelihood that Currently, ED nurse staffing is assigned using heuristics
a patient will leave the ED without being seen (Pham et al. and anecdotes such as higher census on Mondays, on days
2009). These solutions use a variety of administrative and following federal holidays, and with other factors such as
patient level data to attempt to mitigate common ED bot- changes in weather, traffic, and local sporting events. Inac-
tlenecks, bottlenecks that uncorrected may lead to delays, curate prediction can lead to inappropriate nurse to patient
inefficiencies, and even deaths (Carter, Pouch, and Larson ratios which can lead to dangerous under-staffing, poor clin-
AAAI Fall 2020 Symposium on AI for Social Good. ical outcomes, nursing dissatisfaction, and burnout (Aiken
Copyright © 2020 for this paper by its authors. Use permitted under et al. 2002). Matching staffing levels to the variation in daily
Creative Commons License Attribution 4.0 International (CC BY patient demand can improve the quality of care and lead to
4.0). cost savings.
Related Work Table 1: ED Arrivals and Census Features. The prior census
and slope Census are used only in census prediction model
In this paper, we present our work with a busy suburban
and prior arrival and slope arrival only in arrivals model.
ED in the Pacific Northwest that services a rapidly grow-
ing metropolitan area. We describe the development of novel Feature Description
models to predict ED arrivals and census, the design of Prior Census/Arrival 4 features; census/arrival at 4 time
an easily consumable dashboard integrated into the clinical events (at 15 min intervals) prior to pre-
workflow, and deployment of the dashboard using a live data diction time, i.e. 15 min, 30 min, 45
feed. The current work also addresses a gap in the literature min, 60 min.
where there is a dearth of published work related to ED op- Month of year January - December (12 features)
timization in a real world setting and in production. Hour of day Hour of the day (24 features)
The availability of accessible data and computational re- Day of Week Day of the week (7 features)
sources has enabled the application of machine learning Quarter of Year Season: Q1 Winter, Q2 Spring, Q3
(ML) to healthcare at an unprecedented scale (Krumholz Summer, Q4 Autumn
Weekend Flag Flag if prediction on Saturday or Sun-
2014). While several research groups have developed ML
day
predictions on retrospective and static ED data, operational- Evening Flag Flag if prediction time between 20:00
ized ML solutions in the ED are rare. Chase et al. developed and 08:00
a novel indicator of a busy ED: a care utilization ratio (Chase Slope census/arrivals Slope of change from prior census or ar-
et al. 2012). The authors report that the prediction of this ra- rival
tio, which incorporates new ED arrivals, number of patients
triaged, and physician capacity, provides a robust indicator
of ED crowding. McCarthy et al. utilized a Poisson regres- made every 15 minutes, resulting in near real-time predic-
sion model to predict demand for ED services (McCarthy tions. For instance, at 3:15 PM (t), we predict census for
et al. 2008). They determined that after accounting for tem- 5:15 PM (t + 2 hours), 7:15 PM (t + 4 hours), and 11:15
poral, weather, and patient-related factors (hour of day is PM (t + 8 hours). Then at 3:30 PM (t), we predict 5:30
most important), ED arrivals during one hour had little to PM (t + 2 hours), 7:30 PM (t + 4 hours), and 11:30 PM
no association with the number of ED arrivals the following (t + 8 hours).
hour. Jones et al. (Jones et al. 2008) explored seasonal au-
toregressive integrated moving average (SARIMA), time se- ED Arrivals and Acuity ED arrivals reflect the number of
ries regression, exponential smoothing, and artificial neural individual patients who are arriving at the ED over a period
network models to forecast daily patient volumes and also of time. Arrivals can be described by the acuity level of the
identified seasonal and weekly patterns in ED utilization. individual patient, an indicator of illness or injury severity
assessed by nursing staff at the time of patient triage (Gilboy
ED Predictions et al. 2012). Predictions of patient volume by acuity level
can further inform staffing needs - higher acuity patients
The goal of our work was to optimize ED operations by ac- tend to have greater intensity of staff and resource needs.
curately predicting ED arrivals and ED patient census to fa- Similar to the Census prediction, we framed the Arrivals
cilitate staffing optimization to better manage the influxes prediction by acuity for 2, 4, and 8 hour forecasting. To ac-
and patterns of ED patients to provide safe and timely care. commodate different patterns in the acuity of patients, we
Here we describe our approach to building the prediction built models for each individual acuity level.
models and we describe the metrics we used to evaluate the
model accuracy. Methods
Problem Description For both Census and Arrivals we include temporal features
such as hour of day, day of week, month of year, and quar-
There are two distinct yet related ED load optimization prob-
ter of year. To include the unique variations in census and
lems that we address in this work, as described below:
arrival patterns in the evening compared to the morning as
ED Census ED census is defined as the total number of well as weekend versus weekday patterns, we included cor-
patients in the ED at a specified time. ED census includes responding binary variables.
patients in the waiting room, in triage, those receiving care, While ED census or ED arrival may be independent from
and those awaiting ED disposition: hospital admission, dis- one hour to the next, we use the current ED census trend
charge, or transfer. ED census is a ”snapshot” of ED utiliza- to inform future ED census. To include signals for the cur-
tion and includes elements related to ED arrivals as well as rent census trends in ED in our predictive models, we de-
ED throughput. Predicting ED census can serve to inform termine the slope from the census values in the previous 1
both short-term (minutes to hours) operations, such as re- hour for every 15 minute intervals. In addition, we weighted
assigning staff or diverting ambulance arrivals and longer- values from these 15 minute intervals to that more recent
term (hours or longer) administrative decisions, such as call- values had higher weights. The census at t − 15 minutes,
ing in additional staff or sending staff members home early. t − 30 minutes, t − 45 minutes, and t − 60 minutes
We formulated this problem as a prediction of ED census is weighted with 2, 0.5, 0.25, and 0.05 respectively. The
at t + 2 hours, t + 4 hours, and t + 8 hours, where t weights were chosen empirically based on the performance
is the prediction time. In production, these predictions are metrics of the model. Similar to Census, the arrivals for the
Table 2: Distribution of ED Encounters by Acuity
Acuity ESI Number of Encounters
Emergent 1 1,435
2 46,436
Urgent 3 116,808
Non-Urgent 4 33,023
5 2,315
Total 199,957
Arrival prediction are weighted in the same way. The final
set of features is shown in Table 1.
Dataset Description Figure 2: Schematic showing the data sources, models, and
resulting User and Model Health Dashboards. The actual
The data for the experiments came from a suburban level
dashboard image is hidden due to privacy and data compli-
three trauma center at a hospital in the Pacific Northwest
ance.
with > 60, 000 annual ED visits. The ED comprises multi-
ple treatment spaces including 40 acute treatment rooms and
4 trauma rooms for the resuscitation of critically ill patients.
is within a threshold of ±4 (Absolute Error <= 4). Fur-
Individuals are registered at the time of entry to the ED
thermore, we also calculate the percentage of times that the
and all registered ED patients were included in this analy-
model is accurate to within 70% of the actual value (Accu-
sis. ED encounters occurring between January 2014 through
racy>70%). These additional metrics frame the models per-
January 2018 were included in the experiments. The dataset
formances in terms of their effects on user workflows and
included electronic health record (EHR) data elements such
provide a simple understanding of the model performance
as time, date, location, chief complaint, acuity score, vital
under the system constraints while ensuring interpretability
signs, and others. This included 205, 929 ED encounters, of
to end users.
which 199, 957 encounters documented patient acuity. ESI
Furthermore, combining these models with a model man-
is a categorical variable representing patient acuity (based
agement process to detect changes in model performance or
on vital signs and symptoms) where ESI 1 connotes highest
shifts in underlying patient distributions, prevails as novel
urgency and ESI 5 the lowest urgency (Gilboy et al. 2012).
work. Model management is an iterative process that in-
We grouped these into three categories reflecting emergent
cludes monitoring and evaluating model performance to de-
(ESI 1 or 2), urgent (ESI 3), and non urgent (ESI 4 or 5).
tect subtle (or unsubtle) changes in the underlying distri-
The distribution of of the encounters split by ESI groups is
bution of the data, permitting investigation and, if neces-
shown in Table 2.
sary, model re-training. We have implemented a workflow
for automatic model monitoring; the overview of this is rep-
Models
resented in Figure 2. As part of this workflow we created a
Multiple regression models were evaluated for both Census user friendly dashboard to track the model performance and
and Arrivals predictions. We choose to use a Generalized distributions, an example visual can be seen in Figure 3.
Linear Model with Poisson Regression (GLM) for its sim-
plicity and capability to model count data (Gardner, Mul- Results
vey, and Shaw 1995). We included regularization variants Data from January 2014 to October 2017 was used to train
of GLM that include Lasso, Ridge, and Elastic Net for vali- the models and data from November 2017 to January 2018
dation. We also included linear Gradient Boosting Machine was used to test the models. The performance metrics of the
(GBM) due to its robustness to missing data and predictive census models for 2 hour prediction are shown in Table 4.
power (Friedman 2001). We used the average arrivals and The Gradient Boosting Method (GBM) performed the bet-
census values at that same time point from the prior two ter among the set for all metrics which we believe is due to
years as our baseline. We used scikit-learn package avail- its robustness to the sparsity in the data. The 4 and 8 hours
able in Python 3.6 to implement all models. GBM census model MAEs are 4.0739 and 4.2960 respec-
tively. The metric (Accuracy > 70%) shows that GBM is
Evaluation metrics accurate 81.52% of times for a prediction within 70% of ac-
We evaluate the performance of our models using root mean tual census. And, the GBM is accurate 72.90% time for a
squared error (RMSE) and mean absolute error (MAE) (Ver- prediction within a value of ±4 of actual census.
biest, Vermeulen, and Teredesai 2014) which are suitable For arrival models, we built 9 models, one for each acuity
metrics for regression. However, the real utility of ED load level and for each 2, 4 and 8 hours prediction. We observed
prediction is in staffing optimization. Most common mid- that the gradient boosting model performed better than other
size US ED departments have an ED patient to nurse ra- models and the baseline for Emergent acuity encounters,
tio of 4:1. Based on this, we devised an additional metric: where as for Urgent, Non-urgent acuity GLM models per-
we determined the percentage of times the model prediction formed better. The results are shown in Table 3. The abso-
Table 3: Results of 2, 4 and 8 hour Arrival prediction for GLM variants, GBM, and Baseline model for Emergent, Urgent and
Non-urgent patients
Acuity Time window Model RMSE MAE Absolute Error<4 Accuracy >70%
17*Emergent 6*2 hour GLM 1.9747 1.4267 96.33 32.80
GLM-Lasso 2.1492 1.5400 94.96 31.96
GLM-Ridge 2.0039 1.4451 96.05 32.45
GLM-Elastic Net 2.1494 1.5396 95.08 31.91
GBM 1.9768 1.4283 96.38 32.97
Baseline 2.0278 1.5174 96.59 21.63
6*4 hour GLM 1.9749 1.4272 NA NA
GLM-Lasso 2.1913 1.5642 NA NA
GLM-Ridge 2.0318 1.4615 NA NA
GLM-Elastic Net 2.1700 1.5467 NA NA
GBM 1.9786 1.4276 NA NA
5*8 hour GLM 2.9998 2.1928 NA NA
GLM-Lasso 3.5831 2.6451 NA NA
GLM-Ridge 3.1154 2.2680 NA NA
GLM-Elastic Net 3.4307 2.5060 NA NA
GBM 3.0391 2.2217 NA NA
17*Urgent 6*2 hour GLM 1.5022 1.1088 NA NA
GLM-Lasso 1.5837 1.1576 NA NA
GLM-Ridge 1.5116 1.1168 NA NA
GLM-Elastic Net 1.5630 1.1451 NA NA
GBM 2.4042 1.6891 NA NA
6*4 hour GLM 1.5010 1.1082 NA NA
GLM-Lasso 1.5883 1.1511 NA NA
GLM-Ridge 1.5065 1.1102 NA NA
GLM-Elastic Net 1.5514 1.1311 NA NA
GBM 2.4066 1.6914 NA NA
5*8 hour GLM 2.1792 1.6305 NA NA
GLM-Lasso 2.4764 1.8841 NA NA
GLM-Ridge 2.2214 1.6917 NA NA
GLM-Elastic Net 2.3984 1.8109 NA NA
GBM 3.9960 2.8792 NA NA
17*Non-Urgent 6*2 hour GLM 2.5903 2.0017 NA NA
GLM-Lasso 2.6859 2.0874 NA NA
GLM-Ridge 2.5911 1.9996 NA NA
GLM-Elastic Net 2.6572 2.0412 NA NA
GBM 4.0945 3.5052 NA NA
6*4 hour GLM 2.5987 2.0069 NA NA
GLM-Lasso 2.7321 2.0956 NA NA
GLM-Ridge 2.5989 2.0038 NA NA
GLM-Elastic Net 2.7022 2.0918 NA NA
GBM 4.1064 3.5214 NA NA
5*8 hour GLM 3.7944 2.9291 NA NA
GLM-Lasso 4.2303 3.2661 NA NA
GLM-Ridge 3.8033 2.9463 NA NA
GLM-Elastic Net 4.3284 3.3386 NA NA
GBM 7.6992 6.8340 NA NA
the impact and maintenance cost over a period of time.
Discussion
Our work demonstrates that subtle patterns in exogenous
and endogenous variability in patient flow can be utilized
to predict, with high accuracy, ED patient arrivals and cen-
sus. Deployment of ML-based predictive models into a com-
plex clinical workflow is challenging. However, predicting
ED census is an ideal ML healthcare problem to study for
several reasons. First, predicting ED census every 15 min-
utes across 12 different models allows for 1, 152 predictions
Figure 3: Monitoring model performance in deployment - daily. Each prediction is clearly falsifiable with a measurable
Example of predicted vs actual census for the 2 hour census outcome (the actual number of arrivals and patient census),
prediction model over the course of one day and the follow-up interval is short (e.g., one must only wait 8
hours to determine the accuracy of all predictions). Second,
many healthcare ML models are degraded by data censoring;
lute error and accuracy were only available for a subset of for example, when predicting 30-day hospital readmissions,
models. We observe that the MAE and RMSE for all models patients may avoid readmission, they may be readmitted at
across different levels of acuity is similar if we consider 2 another facility. Additionally, according to the work of Jef-
hour and 4 hour windows. However, the performance goes fery and colleagues, prediction based tools are most useful
down if we consider 8 hour windows. This is not unexpected when prompt decision and action are warranted by the end-
since trends can greatly vary across longer time spans e.g., users (Jeffery et al. 2017), however in some cases, such as
compare ED trends at 2 am vs. 10 am. predicting hospital readmissions, the action of the clinician
can alter the outcome, thus making the prediction appear er-
ED Experience roneous. In predicting ED load, there are no actions that the
users can take (other than the ED going on diversion status,
A key differentiator of the work that we present here is that which is done only seldom) that will alter the number of ar-
our prediction models were fully operationalized into the rivals or census. The large number of predictions, the short
clinical workflow, that of the ED charge nurse. Through col- follow-up interval, and the availability of ’perfect informa-
laborative design and planning sessions with ED nurses and tion’ about outcomes (akin to ’perfect information’ games
other health system stakeholders, we developed an ED dash- like chess) makes ED load prediction an ideal place to opti-
board to surface the results of our predictions. Prediction mize model management processes.
based tools are often beset by difficulties in end-user un- We are continuing to improve the performance and clin-
derstanding of probability based results (Jeffery et al. 2017). ical utility of these models by integrating additional data
Part of the solution to this problem is the early incorporation sources into our predictions. These sources can include
of end-user feedback and open discussions around tool util- events or include: local weather data, local sporting events,
ity. local traffic, local emergency medical services (EMS) activ-
Our dashboard was deployed for 6 months as part of pilot ity, and Google Trends searches. We plan to further improve
in a large suburban ED. As part of this pilot, data quality this solution by providing interpretability for the predictions
was monitored continuously and multiple ML models were to help ED staff make informed decisions. (Ahmad, Eckert,
scored at 15 minute intervals. End-user training was con- and Teredesai 2018)
ducted during the pilot period. During this period charge
nurses completed forms at the conclusion of each shift doc-
umenting their use of the dashboard and any actions the
References
dashboard prompted (such as calling in additional staff for Ahmad, M. A.; Eckert, C.; and Teredesai, A. 2018. Inter-
projected high load or sending staff home early for pro- pretable machine learning in healthcare. In Proceedings of
jected low load). In addition to the potential impact on nurse the 2018 ACM international conference on bioinformatics,
staffing, accurately forecasting ED arrivals and census may computational biology, and health informatics, 559–560.
optimize care delivery in other ways - such as reducing wait- Aiken, L. H.; Clarke, S. P.; Sloane, D. M.; Sochalski, J.; and
ing times, ED length of stay, and rates of patients leaving Silber, J. H. 2002. Hospital nurse staffing and patient mor-
without being seen. These additional key performance indi- tality, nurse burnout, and job dissatisfaction. volume 288,
cators (KPIs) were also be evaluated to determine the clini- 1987–1993. American Medical Association.
cal utility of the deployed predictions. The iterative nature of
this approach speaks to the engagement needs of the clinical Batal, H.; Tench, J.; McMillan, S.; Adams, J.; and Mehler,
end-users and the imperative of operationalizing machine P. S. 2001. Predicting patient visits to an urgent care clinic
learning in healthcare. While accurate predictions are key to using calendar variables. volume 8, 48–53. Wiley Online
implementation success and end-user adoption, simple met- Library.
rics such as prevalence of accuracy above a threshold (Accu- Bayley, M. D.; Schwartz, J. S.; Shofer, F. S.; Weiner, M.;
racy > 70%) will help health system stakeholders evaluate Sites, F. D.; Traber, K. B.; and Hollander, J. E. 2005. The
Table 4: Results of 2, 4 and 8 hour Census prediction for GLM variants, GBM, and Baseline model. The baseline is average of
census at that hour in previous 2 years.
Time window Model RMSE MAE Absolute Error<4 Accuracy >70%
6*2 hour GLM 4.3343 3.3812 71.80 80.42
GLM-Lasso 4.6816 3.6642 68.41 77.83
GLM-Ridge 4.5305 3.4975 70.79 72.80
GLM-Elastic Net 4.6550 3.6395 69.45 78.47
GBM 4.2013 3.2790 72.90 81.52
Baseline 6.9026 5.3926 51.19 60.19
5*4 hour GLM(4 hour) 5.1491 4.0173 64.66 74.95
GLM-Lasso(4 hour) 6.0410 4.6643 56.92 68.78
GLM-Ridge(4 hour) 5.2855 4.1111 63.85 74.44
GLM-Elastic Net(4 hour) 6.1962 4.7468 57.32 67.59
GBM(4 hour) 5.1784 4.0241 64.35 75.05
5*8 hour GLM(8 hour) 5.5026 4.2960 61.42 72.51
GLM-Lasso(8 hour) 6.1726 4.8158 55.62 67.26
GLM-Ridge(8 hour) 5.5829 4.3632 60.68 72.25
GLM-Elastic Net(8 hour) 6.6123 5.1085 53.79 64.54
GBM(8 hour) 5.6013 4.3693 60.50 71.99
financial burden of emergency department congestion and dent and emergency departments: a survey of consultants in
hospital crowding for chest pain patients awaiting admis- the UK and the Republic of Ireland. volume 15, 262–265.
sion. volume 45, 110–117. Elsevier. British Association for Accident and Emergency Medicine.
Carter, E. J.; Pouch, S. M.; and Larson, E. L. 2014. The Jones, S. S.; Evans, R. S.; Allen, T. L.; Thomas, A.; Haug,
relationship between emergency department crowding and P. J.; Welch, S. J.; and Snow, G. L. 2009. A multivariate time
patient outcomes: a systematic review. volume 46, 106–115. series approach to modeling and forecasting demand in the
Chase, V. J.; Cohn, A. E.; Peterson, T. A.; and Lavieri, M. S. emergency department. volume 42, 123–139. Elsevier.
2012. Predicting emergency department volume using fore- Jones, S. S.; Thomas, A.; Evans, R. S.; Welch, S. J.; Haug,
casting methods to create a “surge response” for noncrisis P. J.; and Snow, G. L. 2008. Forecasting daily patient vol-
events. volume 19, 569–576. Wiley Online Library. umes in the emergency department. volume 15, 159–170.
Wiley Online Library.
Friedman, J. H. 2001. Greedy function approximation: a
gradient boosting machine. 1189–1232. JSTOR. Krumholz, H. M. 2014. Big Data And New Knowledge In
Medicine: The Thinking, Training, And Tools Needed For
Gardner, W.; Mulvey, E. P.; and Shaw, E. C. 1995. Regres- A Learning Health System. volume 33, 1163–1170. ISSN
sion analyses of counts and rates: Poisson, overdispersed 0278-2715, 1544-5208. doi:10.1377/hlthaff.2014.0053.
Poisson, and negative binomial models. volume 118, 392.
American Psychological Association. McCarthy, M. L.; Zeger, S. L.; Ding, R.; Aronsky, D.; Hoot,
N. R.; and Kelen, G. D. 2008. The challenge of predict-
Gilboy, N.; Tanabe, P.; Travers, D.; Rosenau, A. M.; et al. ing demand for emergency department services. volume 15,
2012. Emergency Severity Index (ESI): a triage tool for 337–346. Wiley Online Library.
emergency department care, version 4. 12–0014.
NCHS. 2009. National hospital Ambulatory Medical
Hoot, N. R.; LeBlanc, L. J.; Jones, I.; Levin, S. R.; Zhou, C.; Care Survey: 2015 emergency department summary ta-
Gadd, C. S.; and Aronsky, D. 2009. Forecasting Emergency bles. URL https://www.cdc.gov/nchs/data/nhamcs/web
Department Crowding: A Prospective, Real-time Evalua- tables/2015 ed web tables.pdf.
tion. volume 16, 338–345. ISSN 1067-5027, 1527-974X.
Peck, J. S.; Benneyan, J. C.; Nightingale, D. J.; and Gaehde,
doi:10.1197/jamia.M2772.
S. A. 2012. Predicting Emergency Department Inpatient Ad-
Hwang, U.; Richardson, L. D.; Sonuyi, T. O.; and Morrison, missions to Improve Same-day Patient Flow: PREDICTING
R. S. 2006. The effect of emergency department crowding ED INPATIENT ADMISSIONS. volume 19, E1045–E1054.
on the management of pain in older adults with hip fracture. ISSN 10696563. doi:10.1111/j.1553-2712.2012.01435.x.
volume 54, 270–275. Wiley Online Library. Pham, J. C.; Ho, G. K.; Hill, P. M.; McCarthy, M. L.; and
Jeffery, A. D.; Novak, L. L.; Kennedy, B.; Dietrich, M. S.; Pronovost, P. J. 2009. National study of patient, visit, and
and Mion, L. C. 2017. Participatory design of probability- hospital characteristics associated with leaving an emer-
based decision support tools for in-hospital nurses. vol- gency department without being seen: predicting LWBS.
ume 24, 1102–1110. Oxford University Press. volume 16, 949–955. Wiley Online Library.
Jenkins, M. G.; Rocke, L. G.; McNicholl, B. P.; and Hughes, Schull, M. J.; Morrison, L. J.; Vermeulen, M.; and Re-
D. M. 1998. Violence and verbal abuse against staff in acci- delmeier, D. A. 2003. Emergency department overcrowding
and ambulance transport delays for patients with chest pain.
volume 168, 277–283. Can Med Assoc.
Verbiest, N.; Vermeulen, K.; and Teredesai, A. 2014. Eval-
uation of classification methods. In Data classification: al-
gorithms and applications. CRC Press.
Weiss, A. J.; Wier, L. M.; Stocks, C.; and Blanchard, J. 2006.
Overview of emergency department visits in the United
States, 2011: statistical brief# 174. Agency for Healthcare
Research and Quality (US), Rockville (MD).