Classification models (“machine learners” or “learners”) were developed using machine learning techniques to predict mortality at discharge from an intensive care unit (ICU) and evaluated based on a large training data set from a single ICU. The best models were tested on data on subsequent patient admissions. Excellent model performance (AUCROC (area under the receiver operating curve) =0.896 on a test set), possibly superior to a widely used existing model based on conventional logistic regression models was obtained, with fewer perpatient data than that model.
Intensive care clinicians use explicit judgement and heuristics to formulate
prognoses as soon as reasonable after patient referral and admission to an intensive care
unit [
Models to predict outcome in such patients have been in use for over 30 years [
There are further challenges for model development. Any model used should be able to deal with the problem of class imbalance, which refers in this case to the fact that mortality should be much less common than survival. Many patient data are probably only loosely or indeed not related to outcome and many are highly correlated. For example, elevated measurements of serum urea, creatinine, urine output, diagnosis of renal failure and use of dialysis will all be closely correlated.
Nevertheless, models are used to risk adjust for comparison within an institution over time or between institutions, and model performance is obviously important if this is to be meaningful. It is also likely that a model with excellent performance could augment clinical assessment of prognosis. Furthermore, a model that performs well while requiring fewer data would be helpful as accurate data acquisition is an expensive task.
The APACHE III-J (Acute Physiology and Chronic Health Evaluation revision
IIIJ [
This study aims to explore machine learning methods that may outperform the logistic regression models that have previously been used.
The reader may like to consult a useful introduction to the concepts and practice of
machine learning [
The study is comprised of three parts: 1. An empirical exploration of raw and processed admission data with a variety of attribute selection methods, filters, base classifiers and metalearning techniques (which are overarching models that have other methods nested within them) that were felt to be suitable to develop the best classification models. Metamodels and base classifiers may be nested within other metamodels and learning schemes can be varied in very many ways .These experiments are represented below in Figure 1 where we used up to two metaclassifiers with up to two base classifiers nested within a metaclassifier.
2. Further testing with the best performing data set (full unimputed training set) and learners with manual hyperparameter setting. A hyperparameter is a particular model configuration that is selected by the user, either manually or following an automatic tuning process. This is represented in a schematic below: 3. Testing of the best models from phase 2 above on a new set of test data to better understand generalizability of the models. This is depicted in Figure 3 below. Matching Test Set
Four Best Models based on 4 Evaluation
Measures
The training data for adult patients (8122 patients over 16 years of age) were obtained from the database of a multidisciplinary ICU in a tertiary referral centre from a period between July 2004 and July 2012.Data extracted were comprised of a demographic variable (age), diagnostic category (with diagnostic coefficient from the APACHE III-J scoring system, including ANZICS modifications), and an extensive list of numeric variables relating to patient physiology and composite scores based on these, along with the classification variable: either survival, or alternatively, death at ICU discharge (as opposed to death at hospital discharge as in the APACHE models). Much of the data collected is used in APACHE III-J model mentioned above, and represents a subset of the data used in that model. Training data, prior to the imputation process, but following discretization of selected variables are represented in Table 1. Test data for the identical variable set were obtained from the same database for the period July 2012 to March 2013.
Of particular interest is that the data is clearly class imbalanced with mortality during ICU stay of approximately 12%. This has important implications for modelling the data.
There were many strongly correlated attributes within the data sets. Many of the model variables are collected as highest and lowest measures within twenty four hours of admission to the ICU. Correlated variables may bring special problems with conventional modelling including logistic regression. The extent of correlation is demonstrated in Figure 4.
Patterns of missing data are indicated in Table 1 and represented graphically in Figure 5.
Fig. 5. Patterns of missing data in the raw training set. Missing data is represented by red colouration.
Missing numeric data in the training set was imputed using multiple imputation
with the R program [
Using the last imputed set was also trialled, as it may be expected to be the most accurate based on the iterative nature of the Amelia algorithm. No categorical data were missing. Date of admission was discretized to the year of admission, age was converted to months of age, and the diagnostic categories were converted to five to eight (depending on study phase) ordinal risk categories by using coefficients from the existing APACHE III-J risk model.
A summary of data is presented below in Table 1.
Phase 1 consisted of an exploration of machine learning techniques thought suitable to this classification problem, and in particular those thought to be appropriate to a class imbalanced data set. Attribute selection, examining the effect of using imputed and unimputed data sets and application of a variety of base learners and metaclassifiers without major hyperparameter variation occurred in this phase. The importance of attributes was examined in multiple ways including using random forest methodology for variable selection, using improvement in Gini index using particular attributes. This information is displayed in figure 6. riables used in the study are ranked by their contribution to Gini index.
A comprehensive evaluation of all techniques is nearly impossible given the
enormous variety of techniques and the ability to combine up to several of these at a
time in any particular model. Techniques were chosen based on the likely success of
their application. WEKA [
Phase 2 of our study involved training and evaluation on the same data sets with
learners that had performed well in phase 1. Hyperparameters were mostly selected
manually, as automatic hyperparameter selection in any software is limited and
hampered by a lack of explicitness. Class imbalance issues were addressed with
appropriate WEKA filters (spread subsample and SMOTE, a filter which generates a synthetic
data set to balance the classes [
Phase 3 involved evaluating the accuracy of the best classification models from phase 2 on a new test set of 813 patient admissions. Missing data in the test set were not imputed. Results are shown in Table 3. 3
Base classifier 2 NA NA
ROC
PRC Preprocess best of all models on any of the four classification methods is shaded in red to emphasise that no one performance measure dominates a classifier’s overall utility. NA J 48 NA NA NA NA NA J48 NA NA NA
NA 0.901 (0.881,0.921) NA Spread subsample uniform Spread subsample Spread subsample Spread subsample Spread subsample uniform Spread subsample uniform 0.888 (0.864,0.912)
Normalizing or standardizing the data did not improve model performance and indeed tended to moderately worsen it.
Table 4 presents the results of applying four of the best models from phase 2 on a test data set of 813 patient admissions which should be from the same population distribution (if date of admission is not a relevant attribute). Evaluation is based on AUC ROC, AUC PRC, Matthews’s correlation coefficient and F-measure. These evaluations were obtained by WEKA’s knowledge flow interface. NA
ROC 0.896 0.893 ROC-area under receiver operating characteristic curve CI-confidence interval PRC-area under precision-recall curve MCC-Matthews correlation coefficient
F-meas-F-measure 4
It is unrealistic to expect models to perfectly represent such a complex reality as that of survival from critical illness. Perfect classification is impossible because of the limitations of any combination of currently available measurements made on such patients to accurately reflect survival potential. Patient factors such as attitudes towards artificial support and presumably health practitioner and institution related factors are important. Additionally non-patient related factors which may be purely logistical will continue to thwart perfect prediction by any future model. For instance, a patient may die soon after discharge from the ICU if a ward bed is available and conversely will die within the ICU if a ward bed is not available and transfer cannot proceed. Models currently employed generally consider death at hospital discharge, but new factors that increase randomness can enter in the hospital stay following ICU discharge, so problems are not necessarily decreased with this approach.
The best models we have studied have excellent performance when evaluated following tenfold cross validation in the single ICU setting with use of fewer data points than the current gold standard model. Machine learning techniques usually make few distributional assumptions about the data when compared with the traditional logistic regression model. Missing data are often dealt with effectively with machine learning techniques while complete cases are generally used in traditional general linear modelling such as logistic regression. Clinical data will never be complete, as some data will not be required for a given patient, while some patients may die prior to collection of data which cannot subsequently be obtained. Imputation may be performed on data prior to modelling but has limitations. It is interesting that models trained on unimputed data tend to perform better than imputed data, both in phase 2 and with the test set in phase 3.
The best comparison we can make in the published literature is the work of Paul et
al [
The techniques associated with the best performance involve addressing class
imbalance (i.e. pre-processing data to create a dataset with similar numbers of those who
survive and those that die). This class imbalance is a well-known problem in
classification. Mortality data from any healthcare setting tend to be class imbalanced. Our
study shows that any approach to class imbalance in the data greatly enhance model
performance. Cost sensitive metalearners [
A cost sensitive learner indicates a technique that reweights cases according to a
cost matrix that the user sets to reflect differing “cost” of misclassification of positive
and negative cases. This intuitively lends itself to the intensive care treatment process
where such a framework is likely implemented at least subconsciously by the
intensive care clinician. For instance the cost of clinically “misclassifying” a patient may
be substantial and clinicians would likely try hard to avoid this situation.
In our study, the ensemble learner random forests [
Similarly, no single evaluation measure can summarize a classifier’s performance and different model strengths and weaknesses may be more or less tolerable depending on the circumstances of model use and hence a range of measures are usually presented as we have done.
There are several weaknesses to our study. It is clearly from a single centre and may not generalize to other ICUs in other healthcare systems. Mortality remains a crude measure of ICU performance but remains simple to measure and of great relevance nevertheless. The existing gold standard models usually measure classification of survival or death at hospital discharge, so are not necessarily directly comparable to our models which measures survival or death at ICU discharge.
We are unable to directly compare our models with what may be considered gold
standards as some of these (e.g. APACHE IV) are only commercially available, and
as mentioned before, even the details of APACHE-III-J are not in the public domain.
The best comparison involving Australasian data using APACHE-III-J comes from
the paper of Paul et al. [
Exploration of the available hyperparameters of the many models examined has
been relatively limited. The ability to do this automatically, and explicitly or in a
reproducible way in WEKA and indeed any available software is limited although this
may be changing [
Likewise, the ability to combine models with the best characteristics [
We have not examined the calibration of our models. Good calibration is not required for accurate classification. Accurate performance across all risk categories is highly desirable in a model. Similarly, performance including calibration for different diagnostic categories that may become more significant in an ICU’s case mix is not accounted for.
Modelling using imputed data in every phase of our study tends to show inconsistent or suboptimal performance. It may be that imputation could be applied more accurately by another approach that would improve model performance.
The major current use of these scores is in quality improvement activities. Once a
score is developed which accurately quantitates risk, the expected number of deaths
may be compared to those observed [
We have presented alternative approaches to the classification problem involving prediction of mortality at ICU discharge using machine learning techniques. Such techniques may hold substantial advantage over traditional logistic regression approaches and should be considered to replace these. Complete clinical data may be unnecessary when using machine learning techniques, and in any case are frequently not available. Out of the techniques studied, random forests seems to be the modelling approach with the best performance and has an advantage that it is relatively easy to conceptualise and implement with open source software. During model training a method to address class imbalance should be used. 6