The number of unplanned readmissions is an important quality improvement indicator for Australian hospitals, as it has a direct impact on costs and patient outcomes. It is a useful indicator for patient multi-morbidity to identify patients that should potentially have different treatment or discharge plans. The objective of the Health Insights Challenge project was to develop a predictive model to determine the likelihood of an unplanned hospital readmission. The outcome was a validated analytical model that predicts the likelihood of an unplanned hospital readmission occurring within 30 days after a given in-patient episode. An 'all-cause' model does not pay attention to any specific disease or diagnosis groups. The predicted likelihood of an unplanned readmission occurring can assist care workers and clinicians in planning treatment and discharge processes.
Austin Health is a public provider of tertiary health services, education and research in Melbourne.
It operates 980 beds across acute, sub-acute and mental health facilities [
Austin Health’s objectives for predicting the likelihood of 30-day unplanned readmissions were to: identify the most likely patient groups for 30-day unplanned readmissions; identify the drivers of 30-day unplanned hospital readmission; redesign the admission, treatment and discharge planning processes; and predict the impact of avoidable bed days on costs, throughput and waiting lists.
Austin Health uses the concept of a spell, which is a continuous series of inpatient episodes, where episodes may be separated by statistical separations to account for different cost centres.
Unplanned readmissions are measured in Victoria as a hospital outcome indicator by the
Australian Commission on Safety and Quality in Health Care [
Preliminary research identified 294 relevant papers on 30-day all-cause readmissions. Five
relevant papers, clearly identifying the variables, the type of model and the model validation
approach, were used as reference case studies, as summarised in table 1 [
The variables used in these models were compared to the data from Austin Health and a large correspondence gave confidence that Austin Health’s dataset was suitable for this project.
The project followed the SEMMA (Sample, Explore, Modify, Model and Assess) framework developed by SAS Institute [11].
Austin Health provided de-identified data for a 5-year period (2009 to 2014) depicted in Table Table 2: Data Extracts (1 July 2009 to 31 October 2014):
No.
1 2 3 4
Activity Data Tables Emergency Department Inpatient Spells Inpatient Episodes Inpatient Comorbidities
Episodes and spells were joined to allocate the spell attributes across the episodes. Episodes resulting from unplanned readmissions were linked to the indexed episodes, to enable the calculation of impact in terms of bed days and direct hospitalisation costs. Lookup tables were joined in to enable analysis and reporting on descriptive attributes. The number of Emergency Department presentations since the previous inpatient episode were determined and added as a new field on each inpatient episode record.
A binary target variable for model training was added to indicate the episodes resulting in
30-day unplanned readmissions according to the Dr Foster’s business rules [
This resulted in a dataset comprising 141 variables and 280,078 episodes, with the proportion of unplanned readmissions (4.27%) to non-readmitting episodes (95.73%) highly skewed.
An extensive data exploration exercise was performed, the results of which will be published elsewhere. The most important findings were:
A majority of the patients having unplanned readmissions were between 50 – 89 years old, were diagnosed with chronic obstructive pulmonary disease (COPD), heart failure, ascites and other diagnoses affecting the heart, lungs and liver, and they had many and frequent inpatient episodes as well as many presentations at the emergency department. Many of the patients having unplanned readmissions had high Charlson scores, which is a measure of comorbidity [10].
Many anecdotal indicators for unplanned readmissions were investigated, such as care at home, hospital in the home days, method of transport, interpreter required and so on, but the data exploration did not surface any positive evidence that these were relevant.
SAS Enterprise Miner was used to develop the models [11]. In order to provide opportunities for intervention, the models were trained and evaluated on the data available at admission. Stepwise, Forward and Main Effects regression, and one Decision Tree models were built. Using lift and the Receiver Operator Characteristic (ROC) index of the validation dataset as selection criteria, the Forward Regression model was the best performing model. It was named the Predictive model for All-Cause unplanned Readmissions (PACR).
Twenty variables were significant in the model, as illustrated in Table 3, calculated using a decision tree algorithm [12]. The top three predictors were counts of emergency presentations, inhospital spells per lifetime and the accumulated length of stay over 5 years. Other important predictors include the admission ward, the Charlson Score and the admission unit.
Age Renal Disease Flag at Admission Mild Liver Disease Flag at Admission Malignancy Disease Flag at Admission COPD Disease Flag at Admission Moderate – Severe Liver Disease Flag at Admission No of Spells Admission Source Care Type Cerebrovascular Disease Flag at Admission Non Chronic Disease Flag at Admission
Dementia Disease Flag at Admission
Episode Episode Episode Episode Episode Episode 5 years Episode Episode Episode Episode Episode The gains chart shows the percentiles of the patient episodes, ranked by their likelihood to have an unplanned readmission. As depicted in Figure 2, the Cumulative Captured Response Rate depicts the number of unplanned readmissions that were correctly identified at each percentile. At the 35th percentile, almost 75% of predicted readmissions were identified. Comparatively, a model with no predictive power (i.e. a random model) would be expected to predict approximately 5% of events in each percentile. The cumulative lift at the 5th percentile shows the model is four times more likely to correctly predict unplanned readmissions, as compared to random chance.
For the models identified in the literature review studies, only the ROC statistic was published
to give an indication of model performance [
PACR model A Public-Private Partnership Develops and Externally Validates a 30-Day Hospital Readmission Risk Predictive Model [8] Potentially Avoidable 30-Day Hospital Readmissions in Medical Patients [7] Risk Factors for All-Cause Hospital Readmission Within 30 Days of Hospital Discharge [9] Development of a predictive model to identify inpatients at risk of readmission within 30 days of discharge (PARR-30) [6]
ROC 0.80 0.77 0.76 0.71 0.70
High Moderate Moderate
To avoid bias and to ascertain the model’s discriminative ability in an actual operational setting, PACR was tested on an out-of-sample dataset, which consisted of data for the first eight months of the 2015 financial year. It consisted of 41,666 inpatient episode records, with their corresponding emergency department presentations. As depicted in Table 5, the model had a marginally higher specificity (74%), as compared to its sensitivity (70%) for this dataset.
Description Probability of identifying patients at risk of unplanned readmissions within 30 days of discharge (event) Probability of identifying patients not at risk of unplanned readmissions within 30 days of discharge (non-event) Accurate classification of ‘events’ and ‘non-events’ as a % of total episodes
Misclassification rate Patients predicted to have unplanned readmissions were categorised using a risk stratification indicator. The top 1/6 were scored as highly likely, the next 1/3 as medium likely and the remaining 1/2 with a low likelihood, based on their predicted likelihood, as illustrated in Figure 3.
Using this stratification, clinicians can prioritise their attention on the predicted episodes. Error! Reference source not found. illustrates the impact of paying attention to each episode predicted to be succeeded by an unplanned readmission in the 8-month validation dataset, using the likelihood stratification introduced in Figure 3. Between 955 patients highly likely to have unplanned readmissions, they had a total of 1747 episodes over the 8 month period. By focusing on an average of 7 cases per day, the highly likely to readmit patients can all be considered for specialised treatment or planning. Should these interventions be successful, 5,509 bed-days of resulting readmissions could potentially be saved (shown as the “readmission LOS days” in figure 4). This represents a theoretical potential saving as not all such interventions would necessarily be successful.
A sensitivity score of 70% indicates that the model has a relatively good discriminative ability for predicting unplanned readmissions – even when only using administrative data.
The inclusion of Charlson comorbidity score at admission is an interesting point. As an indicator of disease complexity, it significantly contributes to the model.
The most likely patients for 30-day unplanned readmissions were identified through data exploration.
The most likely to readmit episodes were identified through the significant variables of the model. The model outputs indicate which values or value bands are most significant [12].
Discussions are underway with Austin Health regarding field trials to utilise the model outputs to redesign admission, treatment and discharge planning processes.
The number of potentially avoidable bed days and their direct hospital costs were shown. These form the basis of the business case to justify the redesign of treatment and discharge processes. A quick simulation study also showed the impact on throughput and waiting lists.
Unplanned readmission is a key quality improvement indicator. Managing highly likely unplanned readmissions can reduce their related costs; increase the capacity for new patients needing care; streamline processes to improve operations; provide higher and more targeted quality of care for patients requiring care; and indirectly improve the hospital’s reputation in the industry.
Validating the model on an out-of-sample dataset has given good insight on the model’s predictive ability and its potential benefits in an operational setting. The next step would be to validate the effectiveness of the model through an on-site field trial at the Austin Health hospitals.
Prediction models that use the Charlson comorbidity index need to be compared to models that do not, as applied to the same datasets. Many Australian hospitals do not use Charlson scores.
Although 30 days is an accepted metric in Victoria, some hospitals and health insurers elsewhere use 28 days. The impact of the length of the interval before an unplanned readmission on the model’s outcomes should also be investigated. Readmissions that occurred just outside the 30 day cut-off are also “medically and financially speaking” just as worthy of intervention.
A thorough simulation study will better show the impact that reducing unplanned readmissions have on patient throughput and waiting times.
Additional improvements could further boost the model’s discriminative ability, for example, by incorporating additional clinical and pharmacy variables.
Thank you to executive sponsors Mark Petty, Fiona Webster, and analysts Ray Robbins and Ronald Ma of Austin Health. Also thanking PBT Australia, Deakin University (Ali Tammadoni), the Australian Centre for Health innovation and EntityThree for collaborating on this project.
Special mention is due to the review board: Dr. Chris Bain, currently Chief Health Information Services Manager at Mercy Public Hospitals; Shmuley Goldberg, former Director of Corporate Development for Victorian Comprehensive Cancer Centre (VCCC), Robert Heyes, Workforce Analyst at the Victorian Department of Health and Professor Damminda Alahakoon, currently Professor in Business Analytics at La Trobe University. [6] J. Billings, I. Blunt, A. Steventon, T. Georghiou, G. Lewis and M. Bardsley, Development of a predictive model to identify inpatients at risk of re-admission within 30 days of discharge (PARR-30), BMJ Open, 2,4 (2012). [7] J. Donzé, D. Aujesky, D, Williams and J.L. Schnipper, Potentially avoidable 30-day hospital readmissions in medical patients: derivation and validation of a prediction model, JAMA Internal Medicine, 173,8 (2013) 632-638. [8] S.A. Choudhry, J. Li, D. Davi, C. Erdmann, R. Sikka and B. Sutariya, A public-private partnership develops and externally validates a 30-day hospital readmission risk prediction model, Online Journal Of Public Health Informatics, 5,2 (2013) 219. [9] L.K. Robin, D.H. Harlen, W.M. Richard, F.E. Matthew, S.W. Douglas and R.M. David, Risk Factors for All-Cause Hospital Readmission Within 30 Days of Hospital Discharge, Journal of Clinical Outcomes Management, 20,5 (2013) 203-214. [10] M.E. Charlson, P. Pompei, K.L. Ales and C.R. MacKenzie, A new method of classifying prognostic comorbidity in longitudinal studies: Development and validation, Journal of Chronic Diseases, 40,5 (1987) 373–383. [11] SAS Institute Inc., Data Mining Using SAS® Enterprise MinerTM: A Case Study Approach, Third Edition, SAS
Institute Inc. Cary NC, 2013. [12] M. Maldonado, Tip: How to Interpret SAS Rapid Predictive Modeler Results, [Online] Available at: https://communities.sas.com/docs/DOC-6813, 2014.