In this study we investigate the need for training future blood glucose level prediction models at the individual level (i.e. per patient). Specifically, we train various model classes: linear models, feed-forward neural networks, recurrent neural networks, and recurrent neural networks incorporating attention mechanisms, to predict future blood glucose levels using varying time series history lengths and data sources. We also compare methods of handling missing time series data during training. We found that relatively short history lengths provided the best results: a 30 minute history length proved optimal in our experiments. We observed long short-term memory (LSTM) networks performed better than linear and feedforward neural networks, and that including an attention mechanism in the LSTM model further improved performance, even when processing sequences with relatively short length. We observed models trained using all of the available data outperformed those trained at the individual level. We also observed models trained using all of the available data, except for the data contributed by a given patient, were as effective at predicting the patient's future blood glucose levels as models trained using all of the available data. These models also significantly outperformed models trained using the patient's data only. Finally, we found that including sequences with missing values during training produced models that were more robust to missing values.
Accurate future blood glucose level prediction systems could play
an important role in future type-I diabetes condition management
practices. Such a system could prove particularly useful in avoiding
hypo/hyper-glycemic events. Future blood glucose level prediction
is difficult - blood glucose levels are influenced by many variables,
including food consumption, physical activity, mental stress, and
fatigue. The Blood Glucose Level Prediction Challenge 2020 tasked
entrants with building systems to predict future blood glucose levels
at 30 minutes, and 60 minutes into the future. Challenge participants
were given access to the OhioT1DM dataset [
In the previous iteration of the challenge, several researchers
demonstrated both that it is possible to predict future blood glucose
levels using previous blood glucose levels only [
In this work, we investigate the possibility of building a single
model that is able to predict future blood glucose levels for all 12
patients in the OhioT1DM data set, and the effectiveness of
applying such a model to completely unseen data (i.e. blood glucose series
from an unseen patient). We also investigate the impact of history
length on future blood glucose level prediction. We experiment with
various model types: linear models, feed-forward neural networks,
and recurrent neural networks. Furthermore, inspired by advances in
leveraging long distance temporal patterns for time series prediction
[
As stated above, one of our primary aims was to build a single model that is able to predict future blood glucose levels for each patient in the data set. To this end, we constructed a combined data set containing data provided by each of the patients. Specifically, we created a data set composed of all of the data points contributed by the six patients included in the previous iteration of the challenge (both training and test sets), as well as the training data points provided by the new cohort of patients. This combined data set was split into training and validation sets: the final 20% of the data points provided by each patient were chosen for the validation set. The test data sets for the 6 new patients were ignored during development to avoid bias in the result. For experiments in building patient specific models, training and validation sets were constructed using the patient in question’s data only (again with an 80/20 split). 2.2
Prior to model training, the data were standardized according to:
Model Feed-forward
Hyper-parameters # hidden units 2 f16; 32; 64; 128; 256; 512; 1024g where train, and train, are the mean and standard deviation of the training data, respectively. There is a non-negligible amount of data missing from the training set, which needed to be considered when preprocessing the data. We investigated two approaches to handling missing data: discarding any training sequences with one or more missing data points; and replacing missing values with zeros following standardization. It was hypothesized that the second approach may help the system learn to be robust to missing data. 2.3
We experimented with linear models, feed-forward neural networks, and recurrent neural networks. Each model was trained to minimize the root mean square error (RMSE): v u n RM SE = tuX i=1 y^i
yi 2
n
(2)
where y^i is the predicted value, yi is the true value, and n is
the total number of points in the evaluation set. Various
hyperparameters were tuned when training the feed-forward and
recurrent neural networks; Table 1 provides a summary. During
development, each model was trained for 50 epochs using the Adam
optimizer ( =0.001, 1=0.9, 2=0.999) [
Model selection and hyper-parameter tuning were performed for the 30 minute prediction horizon task. The best performing model was then trained for 60 minute prediction. Experiments were repeated for blood glucose history lengths of 30 minutes, 1 hour, 2 hours, and 24 hours. 2.4
It can be difficult for recurrent neural networks to learn long distance patterns. The LSTM network was introduced to address this problem PH=60 37.37 (0.09) 28.4 (0.14) 27.6 (0.15) 34.28 (0.18) 32.97 (0.09) 25.99 (0.12)
PH=30 16.64 (0.1) 12.85 (0.11) 12.43 (0.12) 15.9 (0.12) 15.99 (0.03) 12.4 (0.04)
PH=60
30.8 (0.13)
23.57 (0.16)
22.78 (0.16)
28.95 (0.13)
27.04 (0.07)
21.33 (0.13)
[
htT W hi ti ct = = = =
exp(score(ht; hi) t j=1exp(score(ht; hj ))
) X
tihi i tanh(Wc[ct; ht]) (3) (4) (5) (6) where score(ht; hi) is an alignment model, ct is the weighted context vector, and at is the attention vector, which is fed into the classification layer (without an attention mechanism, the hidden state ht is fed into the classification layer). The dimensionality of t was chosen using the validation set - see Table 1 for details. We also experimented with attention mechanisms in LSTM networks designed to process shorter sequence lengths: we chose the optimal LSTM model architecture for each history length (30 minutes, 60 minutes, 2 hours), added an attention mechanism, and re-trained the model (again, the optimal t dimensionality was chosen using the validation set). 2.5
In order to investigate the need for an individual’s data when training a model to predict their future blood glucose levels, we trained 6 different models - each with one patient’s training data excluded from
Patient excluded 540 544 552 567 584 596 the training set - using the optimal LSTM architecture determined in previous experiments. The models were then evaluated using the test data for the patient that was excluded from the training set. We also trained 6 patient specific models, each trained using the patient’s training data only. We again used the optimal architecture determined in previous experiments, but tuned the number of hidden units using the validation set in order to avoid over-fitting due to the significantly reduced size of the training set (compared with the set with which the optimal architecture was chosen). Each model was trained using the early-stopping procedure outlined in 2.3. 3
Our evaluation showed that recurrent models performed significantly
better than both linear and feed-forward neural network models, for
each history length we experimented with (p=0.05, corrected paired
t-test [
Table 3 compares LSTM models (with the same architecture as above) trained with the following data sources: the individual patient’s data only, data from all of the available patients, and data from every other patient (excluding data contributed by the patient in question). We observed that models trained using a large amount of data, but excluding the patient’s data, outperformed models trained using the patient’s data only (p=0.05). We also found no significant difference in performance between models trained using all of the available training data (i.e. including the patient’s data) and those tr
Comparison of validation set scores for linear, feed-forward, and LSTM neural networks as a function of history length. ained excluding the patient’s data, highlighting the general nature of the models. We found that including sequences with values in the training set produced models that were more robust to missing data, as evidenced by the improved RMSE scores listed in Table 4: RMSE scores were significantly improved for each patient using this approach to training (p=0.05).
Incorporating an attention mechanism further improved performance in most instances: we observed significant improvements for history lengths of 30 minutes, 60 minutes, and 2 hours (p=0.05), but not for history lengths of 24 hours. Figure 3 compares the regular LSTM and LSTM with attention performance as a function of history length. Figure 2 shows partial auto-correlation plots for 4 different patients. Interestingly, two of the patients’ blood glucose data patient 540, and patient 544 - don’t show any significant long term correlation, whereas the other two - patient 552, and patient 567 both exhibit significant correlation at time lags of approximately 6 and 12 hours. We observed this behaviour in half of the patients. We Patient ID
Exclude missing data
Include missing data 540 544 552 567 584 596 AVG 21.45 (0.06) 16.79 (0.06) 16.27 (0.13) 21.19 (0.1) 21.16 (0.06) 16.08 (0.07) 18.82 (2.45) also observed correlations at even greater time lags, corresponding to multiples of 6 hours. The difference in the patients’ partial autocorrelation plots suggests it may be sub-optimal to train an attention mechanism using each patient’s data at once, and that training at the patient-level may enable the model to learn very long range patterns. Furthermore, while we were able to train an LSTM model with a history length of 30 minutes that generalized across all patients, it may be the case that short range blood glucose patterns are quite general, and long range patterns are more personalized, and tuning the history length per patient could improve prediction performance. All of the results presented in this section can be reproduced using publicly available code 3. 4
In this study we showed that it is possible to train a single LSTM model that is able to predict future blood glucose levels for each of the different patients whose data are included in the OhioT1DM data set. We also demonstrated that an individual patient’s data is not required during the training process in order for our model to effectively predict the patient’s future blood glucose levels. Furthermore we showed that incorporating an attention mechanism in the LSTM improved performance, and that including sequences with missing values during training produced models that were more robust to missing data. 3 https://github.com/robert-bevan/bglp