Individuals with Type I diabetes (T1D) must frequently monitor their blood glucose (BG) and deliver insulin to regulate it. New devices like continuous glucose monitors (CGMs) and insulin pumps have helped reduce this burden by facilitating closed-loop technologies like the artifical pancreas (AP) for delivering insulin automatically. As more people use AP systems, which rely on a CGM and insulin pump, there has been a dramatic increase in the availability of large scale patient-generated health data (PGHD) in T1D. This data can potentially be used to train robust, generalizable models for accurate BG forecasting which can then be used to make forecasts for smaller datasets like OhioT1DM in real-time. In this work, we investigate the potential and pitfalls of using knowledge distillation to transfer knowledge from a model learned from one dataset to another and compare it with the baseline case of using either dataset alone. We show that using a pre-trained model to do BG forecasting for OhioT1DM from CGM data only (univariate setting) has comparable performance to training on OhioT1DM itself. Using a single-step, univariate recurrent neural network (RNN) trained on OhioT1DM data alone, we achieve an overall RMSE of 19.21 and 31.77 mg/dl for a prediction horizon (PH) of 30 and 60 minutes respectively.
Type 1 diabetes (T1D) is a chronic lifelong disease that requires dozens of daily decisions to manage blood glucose (BG). While keeping BG in a healthy range is critical for avoiding complications, it is challenging, as meals and many other factors like exercise and stress can affect BG and insulin sensitivity. Closed-loop technologies, which connect a continuous glucose monitor (CGM) and insulin pump with a control algorithm, could relieve this burden by automatically dosing insulin. This requires an accurate forecast of where glucose is headed so the right amount of insulin can be delivered to keep BG within a target range dynamically.
Prior works include using system identification techniques to
model glucose-insulin interactions [
We propose that large public datasets like OAPS can be used to pretrain models, allowing deep learning to be used on smaller curated datasets for forecasing BG. In particular, we show by augmenting and distilling knowledge across models trained on data obtained from different sources using RNN, we achieve an accuracy comparable to that achieved by using OhioT1DM dataset alone for univariate setting. We also compare the performance with multi-output setting in which multiple BG values are estimated in the prediction horizon simultaneously. The code is available at https://github.com/health-ailab/BGLP BG forcasting. 2
The task here is to forecast future values for BG. We compare singlestep and multi-output forecasting. In the single-step setting, a single glucose value is estimated several minutes into the future, whereas in multi-output forecasting several future values are estimated simultaneously to model the signal trajectory over the prediction horizon. We begin by describing our time series forecasting approach, and later discuss the dataset specific preprocessing. 2.1
We define the feature vector X0:t = fx0; x1; :::; xtg 2 Rn with n being the number of variables. We use only raw CGM values and do not incorporate additional features like carbohydrate intake and insulin dosage. We also have a corresponding output time series Xt0+1:t+h = fx0t+1; x0t+2; : : : ; x0t+hg 2 R representing multiple future glucose values across a given prediction horizon (PH) of 30 and 60 minutes. As CGM data is recorded at a frequency of 5 minutes, a PH of 30 and 60 minutes will lead to h = 6 and h = 12 samples, respectively. For the single step setting, this target vector becomes Xt0+h = fx0t+hg estimating only a single value h time instances in the future. Multi-output forecasting, on the other hand, aims to estimate the joint probability p(Xt0+1:t+hjX0:t) simultaneously. However, root mean square error (RMSE) was calculated by comparing the actual future glucose level and the last future value in the estimated multi-output sequence, to accurately measure the performance of the forecasting model across the two output settings.
Final predictions
Yn Pathway II Teacher Model Pathway III re-trained on
OhioT1DM Student
Model
Soft predictions Yn
Pathway IV
Teacher Model trained on OpenAPS Ohio T1DM
Open
APS
Pathway I
We use a vanilla RNN with a single hidden layer, H(t) with 32 units, followed by a fully-connected output layer O(t). This was used as the teacher model in Approaches II, III and IV and trained on the source patient-generated data. In Approach I where no teacher model was involved, the RNN was used as a student model trained only on the target OhioT1DM dataset to observe the effects of using datasets of different sizes and from different sources with the same network architecture. In Approach IV, a teacher RNN model trained on OAPS was used to teach a student ANN model using OhioT1DM data to study the effects of knowledge distillation between different kinds of networks. We use a simple, fully-connected ANN with a single hidden layer with 32 units. The number of units for both RNN and ANN were chosen after trying and testing [28, 32, 64, 128] and optimizing for the least RMSE. The output layer O(t) predicts the glucose 2.2
Our proposed approach is to make glucose estimations for a small
dataset by pre-training an RNN on a larger dataset and then
retraining it using a smaller dataset. We compare four learning
approaches for glucose forecasting, as shown in Fig.1: I) training and
testing an RNN on OhioT1DM only (red path), II) training an RNN
on OAPS dataset and testing on OhioT1DM without any re-training
(blue path), III) training an RNN on OAPS dataset, training again
OhioT1DM, and then testing on the OhioT1DM (purple path), and
IV) the pre-trained RNN model makes intermediate estimates called
soft predictions, which are given as target estimates to a student
artificial neural network (ANN) model instead of the actual ground truth,
as done for a classification task in [
Approach I: Student model only Approach II: Teacher model without re-training Approach III: Teacher model with re-training Approach IV: Mimic learning (teacher + student model) value(s) 30 or 60 minutes into the future depending on the PH and the output setting (single-step or multi-output). 2.4
The teacher model was trained on OAPS dataset which was
preprocessed the same way as OhioT1DM, as discussed in Section
4. Early stopping was used to halt the training process if
validation loss was not improving significantly, with the maximum
number of epochs being 1000 with a batch size of 248 and 128 for
OAPS and OhioT1DM, proportional to size of each dataset.
Glorot normal initialization [
We aim to evaluate the impact of using a large noisy dataset for
improving forecasting in a smaller more controlled dataset. The larger
(source) dataset from OAPS [
The collection of OAPS data started in 2015 as part of an initiative
to make APS technology more accessible and transparent for people
with T1D and to enable them to create their own customized AP
systems. Participants can voluntarily donate their data, including
glucose levels recorded via CGM, insulin basal and bolus rates, carbs
intake, physical activity, and other physiological data. Researchers
can gain access to this dataset free of charge, provided they share
their insights and research findings with the public within a
reasonable frame of time [
The training data consists of 12 subjects: six from the OhioT1DM
dataset shared in 2018 for the First BGLP Challenge (Group I)[
For both OAPS and OhioT1DM, we use four recorded variables and one attribute derived from the raw glucose values. The list of features used in the experiments includes raw CGM values (glucose level, insulin basal rate (basal and temp basal), bolus amount (bolus), carbs intake (meal), and difference between consecutive glucose values calculated during data pre-processing (glucose diff ). The first step in data pre-processing was to synchronize the multi-modality data by generating a single timestamp data field based on the timestamps for each of the four fields, generating an irregularly sampled multivariate time series.
In OAPS dataset, there were two types of gaps present in the data, first where both timestamp and glucose values were missing, and second where the timestamp was recorded but the corresponding glucose value was missing. In OhioT1DM, missing glucose values were identified once the multi-modality data was synchronized since basal, bolus, and meals are not recorded at the same 5-minute frequency as glucose levels. When there was missing glucose data for more than 25 consecutive minutes, these times were not used during training. Each data segment (series of points not separated by gap longer than 25 minutes) was then imputed and windowed separately to maintain temporal continuity in the data.
For the rest of the data, which may contain shorter gaps, we used
linear interpolation to impute missing glucose values in training data.
Missing values in test data were imputed by extrapolation to avoid
using data from the future. Basal rates were imputed with forward
filling, meaning replacing missing values with the last recorded basal
rate, since the value is only recorded when it changes and thus
missing values mean the last recorded one is still active. However, if
the field “temp basal”, recording temporary basal infusion rate, was
present for a given set of timestamps, it was used to replace the
recorded basal rate [
In addition to missing data, the sensors are also noisy, leading to
sudden changes in glucose levels, which can cause high variance in
the learned model. To remove these spikes, the signal was passed
through a median filter with a window size of 5 samples, as in [
A sliding window was used to split the data into fixed sized sequences for further downstream analysis. There are three parameters for the moving window configuration: history window size (number of past samples to use for forecasting), prediction horizon (PH) and output window (how far into the future and how many future values to predict), and stride (number of samples to skip while sliding the window). An hour (12 samples) of past values were used to predict the glucose levels 30 and 60 minutes into the future (PH = 30, 60) with a unit stride, which means overlapping windows were used to partition the data.
In OhioT1DM train and test data, the raw CGM values range from 70 – 275 mg/dl and 75 – 290 mg/dl on average, respectively. To ensure that values of all the features were in the same range, insulin basal, bolus rates and carbs intake were normalized based on the minimum and maximum value of glucose levels using Min-Max Normalization. 5.1
The last ten days of data for subjects with ID 559, 563, 570, 575, 588 and 591 were used as validation set and test set was sampled from data for subjects 540, 544, 552, 567, 584, 596. The processing steps for the test data included linear extrapolation for imputing missing values and normalization. The test data was not passed through a median filter like the training set to see how robust the trained models were to unseen, noisy data. We use root mean square error (RMSE) and mean absolute error (MAE) to compare the predicted values with the actual ground truth to evaluate the model. MAE and RMSE can be expressed as,
M AE RM SE = = 1 Xn n
n=1 v uu 1 t n jyi
y^ij n X(yi n=1 y^i)2 (1) (2) where yi is true glucose level and y^i is estimated glucose level, both measured in mg/dl. We repeated the experiments 10 times and calculated the average RMSE and MAE for each subject across the ten trials. We also report the best, worst and mean RMSE (MAE) across all the subjects for each of the four pipelines using both singlestep and multi-output models. 5.2
Overall, approach I achieved the lowest RMSE (MAE) with 19.21
(13.07) for a PH of 30 minutes and 31.77 (23.09) for PH = 60
minutes. In this approach an RNN was trained only using the OhioT1DM
data, using raw CGM values. The worst performance was from
approach IV, where estimations made by a teacher model pre-trained on
OpenAPS dataset were given as ground truth to student ANN model
for training on OhioT1DM, as shown in Tables 1a and 2a. This
approach did not improve the forecast accuracy as it did in [
For BG forecasting using multi-output model, all approaches performed equally well, with approach I, II, and IV (student model, teacher and teacher student model) giving the same RMSE on average. For approach II, the error did not worsen significantly, showing that pre-trained models can be used for making forecasts for OhioT1DM data in real-time, without having to set aside a portion of the dataset for retraining the model, an important consideration for smaller datasets. However, the RMSE improved slightly for approach II when the teacher model was retrained.
Approach IV In this work we have compared four different learning strategies for BG forecasting using two different datasets. We have shown that an RNN model pre-trained on a bigger dataset such as OpenAPS can be used directly to do BG forecasting for a smaller dataset like OhioT1DM when using CGM data only. We predicted BG levels 30 and 60 minutes into the future using single-step and multi-output models, using univariate BG data. Overall, a single-step RNN trained only on univariate data from OhioT1DM dataset achieved the least RMSE of 19.21 and 31.77 mg/dl for a PH of 30 and 60 minutes, respectively.
We would like to thank the referees for their comments, which helped improve this paper considerably. This work was supported in part by the NSF under award number 1915182, NIH under award number R01LM011826, and Fulbright Scholarship.