With continuous glucose monitoring (CGM), data-driven models on blood glucose prediction have been shown to be effective in related work. However, such (CGM) systems are not always available, e.g., for a patient at home. In this work, we conduct a study on 9 patients and examine the predictability of data-driven (aka. machine learning) based models on patient-level blood glucose prediction; with measurements are taken only periodically (i.e., after several hours). To this end, we propose several post-prediction methods to account for the noise nature of these data, that marginally improves the performance of the end system.
Diabetes mellitus has been a major and global problem for
a long time, as it is report that there are over 400 million
patients over the world 1. The knowledge of glucose
concentration in blood is a key aspect in the diagnosis and
treatment of diabetes. The use of signal processing
techniques on glucose data started a long time ago, when
glucose time-series in a given individual could be obtained
in lab study from samples drawn in the blood at a
sufficiently high rate. In particular, related work employed not
only linear (e.g., correlation and spectrum analysis, peak
detection), but also nonlinear (e.g., approximate entropy)
methods to investigate oscillations present in glucose (and
insulin) time-series obtained, during hospital monitoring,
Copyright © CIKM 2018 for the individual papers by the papers'
authors. Copyright © CIKM 2018 for the volume as a collection
by its editors. This volume and its papers are published under
the Creative Commons License Attribution 4.0 International (CC
BY 4.0).
by drawing blood samples every 10-15 min for up to 48
h [
With the support of continuous glucose monitoring
(CGM) sensors, the development of new strategies for the
treatment of diabetes has been accelerated in recent years.
In particular, CGM sensors can be injected into ‘online‘
recommender systems that are able to generate alerts when
glucose concentration is predicted to exceed the normal
range thresholds. Recently, there has been a lot of
complex data-driven prediction models [
2http://time.com/4703099/continuous-glucose-monitorblood-sugar-diabetes/
Dataset Overview The data collection study was conducted from end of February to beginning of April 2017 and includes 9 patients who were given specially prepared smartphones. Measurements on carbohydrate consumption, blood glucose levels, and insulin intake were made with the Emperras Esysta system 3. Measurements on physical activities were obtained using the Google Fit app. We use only steps information (number of steps) for our study.
We describe briefly here some basic patient information. Half of the patients are female and ages range from 17 to 66, with a mean age of 41.8 years. Body weight, according to BMI (Body mass index), is normal for half of the patients, four are overweight and one is obese. The mean BMI value is 26.9. Only one of the patients suffers from diabetes type 2 and all are in ICT therapy 4. In terms of time since being diagnosed with diabetes, patients vary from inexperienced (2 years) to very experienced (35 years), with a mean value of 13.9 years. We anonymize the patients and identify them by IDs (from 8 to 17, we do not have information for patient 9).
We give an overview of the number of different measurements that are available for each patient. The study duration varies among the patients, ranging from 18 days, for patient 8, to 33 days, for patient 14. Likewise, the daily number of measurements taken for carbohydrate intake, blood glucose level and insulin units vary across the patients. The median number of carbohydrate log entries vary between 2 per day for patient 10 and 5 per day for patient 14. Median number of blood glucose measurements per day varies between 2 and 7. Similarly, insulin is used on average between 3 and 6 times per day. In terms of physical activity, we measure the 10 minute intervals with at least 10 steps tracked by the google fit app. This very low threshold for now serves to measure very basic movements and to check for validity of the data. Patients 11 and 14 are the most active, both having a median of more than 50 active intervals per day (corresponding to more than 8 hours of activity). Patient 10 on the other hand has a surprisingly low median of 0 active 10 minutes intervals per day, indicating missing values due to, for instance, not carrying the smartphone at all times.
3https://www.emperra.com/en/esysta-product-system/ 4describes as a model of an insulin therapy for the diabetics with two different types of insulin. particular, for most patients the number of glucose measurements roughly matches or exceeds the number of rapid insulin applications throughout the days. Notable exceptions are patients 14, 15, and 17 (figures excluded). For patient 14, in the evening the number of meals and rapid insulin applications match but exceed the number of blood glucose measurements by far. Patient 17 has more rapid insulin applications than glucose measurements in the morning and particularly in the late evening. For patient 15, rapid insulin again slightly exceeds the number of glucose measurements in the morning. Curiously, the number of glucose measurements match the number carbohydrate entries – it is possible the discrepancy is a result of missing (glucose and carbohydrate) measurements. We further show the blood glucose distribution of each patient in Figure 1. The different lengths of the interquartile range for each distribution also reflects the difficulty of prediction problem on different patients. 1214 12 15 4 6 6 7 5 6 6 6 8 8 11 8 0 0 11213 4 5 6 7 8 9 1011121314151617181920212223 0 1
Hour of day (d) P14 Carbohydrates
basal rapid Our first approach to blood glucose prediction is based on a regression type form of time series prediction. Given historical blood glucose data, we learn a model that predicts future glucose values based on a representation of the current situation (including the recent past), using information on patient context, recent insulin applications, carbohydrate intake, and physical activity levels. 3.1
Our prediction task is a time series prediction of blood glucose values (in mmol/L) with a prediction horizon of 1 hour. Consequently, we can construct a data instance for each glucose measurement found in the dataset and use all information available up until 1 hour before the measurement for predicting the glucose value (c.f., Figure 2).
Performance is evaluated on a per patient basis. In addition, we average performance over patients to get an overview. For each patient, we consider the first 66% of blood glucose measurements as training data to learn the models and the last 34% as test data to evaluate prediction performance.
Prediction performance is measured in terms of median absolute error (MdAE), root mean squared error (RMSE) and symmetric mean absolute percentage error (SMAPE). Given are ground truth values yi and predictions yˆi, with i 2 [1; n]. Median absolute error measures the median error made and is defined as
MdAE = median(jyˆi i yij):
Root mean squared error weighs larger errors more heavily and is defined as
r åin=1(yˆi yi)2 RMSE = : n
Symmetric mean absolute percentage error relates prediction errors to predicted values. It is defined as SMAPE = 100% n ån jyˆi yij i=1 (jyij + jyˆij)=2 : Note that this gives a result between 0% and 200%. Further, the measure penalizes a) deviating for low values and b) over-forecasting. 3.2
As standard simple baselines, we use the last value observed one hour before the value that is being predicted (Last) and the arithmetic mean of glucose values in the training set.
As a more advanced baseline, we use a (temporal) context weighted average of previous glucose values. As our analysis showed differences in glucose values according to time of the day, we weigh previous glucose values base on temporal proximity, weighted exponentially decreasing in the difference of time of day.
LSTM is a recurrent neural network model that effectively accounts for the long-term sequence dependence among glucose inputs.
The Random Forest Regressor (RF) is a meta estimator that
learns an ensemble of regression trees [
The Extra-Trees Regressor (ET) is a variation on
RandomForest that uses a different base learner: Extremely
randomized trees [
AVG Context-AVG ARIMA LSTM (10 iter) LSTM (100 iter) RandomForest Extremely (randomized) Trees
MdAE RMSE SMAPE In this section, we study the confidence of our best performed prediction tree-based models, RandomForest and ExtraTrees. This would, to an extent, facilitate us to answer an important question, when the system is reliable enough to give out predictions. Thus, we study the variability of predictions and estimate standard errors for the prediction model.
When looking at two regression models, while the model predictions could be similar, confidence in them would vary if we look at the training data, a less and more spread out data could bring a low confidence. Hence, a prediction returning a single value (typically meant to minimize the squared error) likewise does not relay any information about the underlying distribution of the data or the range of response values. We hence, leverage the concept of prediction intervals to supplement for the noisy data and enhance the end model, in the sense that it can refuse to give prediction at certain time when the confidence is low.
A prediction interval or confidence interval is an
estimate of an interval into which the future observations will
fall with a given probability. In other words, it can quantify
our confidence or certainty in the prediction. Unlike
confidence intervals from classical statistics, which are about a
parameter of population (such as the mean), prediction
intervals are about individual predictions [
We report here the variablity evaluation across all patients for the regression task. Figure 4 show the error bars using unbiased variance for all patients. We then show in Figures 5 the error bar graphs for patient 8 in an incremental training size setting – meaning that we keep the same actual test set, but training on only part of the training data. E.g., 1=4 training data indicates that we ‘look back’ on only 1=4 of the available past data. The more dots that near the diagonal show the more ‘accurate’ is our prediction model. And the error bars show the ‘confidence’ interval. Figure 5(a) indicates the high ‘confidence’ in the predictions with little training data, yet the dots are far away from the diagonal.
when to predict: on the training size evaluation. To answer this question, we set up an evaluation setting with increasing size of number of instances, order by time. Each training point is evaluated by leave-one-out validation. We show in Figure 6 the results for patient 8. The general conclusion is the that the more training data, the better the performance is, as witness for patient 13, 15 or 17. However, the results for such patients e.,g patient 8, 11 or 16 show that the training size increment could also bring more noise and decrease the results. We envision that it could because the learned model is not stabilized yet with the limited number of instances in our experiment. In addition, training size is not the only factor to decide when to predict. We hence move on to examine the other two factors: (1) model stability - via std. dev. and (2) prediction confidence toward coming instances. 4.1.2
when to predict: on the model stability.
To answer this question, we measure the stability of the model by the standard deviation of the k-fold cross validation with incremental training size. Figure 10 indicate on MAE and RMSE metrics, the model seems to be more stabilized with the more number of training data. This is a good indicator for the when to predict questions. 4.1.3
when to predict: on the prediction confidence. We show in Figure 8 and Figure 9 the confidence distribution at each run of the 5-fold CV for different patients based on bias and no-bias confidences respectively. The results show the confidence distributions are rather similar across different run, indicating that the temporal order of the instances does not impact much on the model performance. Base on the distribution, we move on the the threshold parameter tuning for the data filtering using confidence interval. The idea is to answer the question, ”if we filter low confidence instances (high confident interval), will the model perform better?”
The answer somehow is depicted in Figure 11. For some patients i.e., patient 10, 13, the filtering technique substantially enhance the model performances on MAE and RMSE (not shown) metrics. It is witness that the biased confidence measure somewhat works better than non-biased one across patients. However, for some patients i.e., patient 8 it seems does not bring any effects.
We move on to experiment with filtering instances that we empirically witness that it seems lacking of preditable context within the training data. They are the BG measurements at night. We then attempt to filter those out for prediction. Even though slightly improving for some (c.f., Figure 12), overall the filtering attempt does not make significant difference, indicating that our model learns it better than we expect. 4.1.4
when to predict: combined factors.
Figure 7 show some highlighted combined filtering techniques. In general, combining the aforementioned factors together does improve the model performance. However, the combination is not straightforward, e.g., confidence interval filtering lower the performance at the starting time when the model is unstable aka. cold start. Hence, there is not enough evidence for us to make a hard decision. The more trial-and-error attempts on the fly or a bigger dataset however will be at ease to be built on these as a foundation. 4.2
We show in Table 2 the overall results of our models with different filtering approaches for all patients. We use 2 different filtering approaches: (1) Sanity filter, heuristics (e.g., remove out wrongly input measurement or moments when the last glucose level input is too far) that remove noise and (2) Stability filter: prediction confidence (std. dev is not needed when the training size is large enough). The results show that the stability filter (based on bias and bias-corrected) achieve the best performance, without the need of human efforts on sanity filter. Sole stability filter also provide more predictions (avg. 24) than other filtering combination.
(a) Patient 15 (b) Patient 17
Model # predictions MAE MdAE RMSE SMAPE rf 42 2.58 2.27 12.05 29.98 et 42 2.55 2.16 12.15 29.56 rf + sanity filter 16 2.22 2.01 8.80 28.10 et + sanity filter 16 2.29 2.06 9.01 29.36 rf + sanity + stability filter 15 2.22 1.92 8.71 27.82 rf + stability filter 24 1.92 1.77 7.57 22.65 ● ●● 2 1 0 run1 run2 run3 run4 run5 0 run1 run2 run3 run4 run5 3 12 ●● ● ● ● ● 0 ● 6 345 ●● 12 10 8 46 ● 02 ● ● ● run1 run2 run3 run4 run5 2.0 5 ● 342 ●● ● ● ●●● ●●●
● 01 −1 run1 run2 run3 run4 run5 2.0 1.5 1.0 0.5 0.0 −0.5 We studied the predictability of machine-learning models in the scenarios of non-continuous blood glucose tracking. Additionally, we studied the stability and robustness of the learned model over time. We show that Random Forest and Extra Tree ensemble-based models are the most suitable models for this case, as they can account for the outliers as well as overfitting problems when the data are limited. Our further study on the prediction confidence show that the model can give reliable predictions after acquiring 2530 instances.
Acknowledgements. This work was partially funded by the German Federal Ministry of Education and Research (BMBF) under project GlycoRec (16SV7172). (a) Patient 8 - bias
(b) Patient 8 - no bias (c) Patient 10 - bias
(d) Patient 10 - no bias (e) Patient 13 - bias
(f) Patient 13 - no bias (g) Patient 14 - bias (h) Patient 14 - no bias