Diabetes mellitus is a metabolic disorder and a significant cause of morbidity and mortality worldwide [1]. As yet, there is no cure developed for diabetes; and management of the corresponding lifeimpeding conditions is recommended as the most successful way to control the disease [6]. In fact, the occurrence of the associated complications can be suspended or even prevented by effective management of the disease [11].
Among different types of diabetes, the importance of the selfmanagement for type 1 diabetes mellitus (T1DM) is accentuated [8,19]. The key factor in T1DM management is to control the blood glucose level (BGL) within the normal range [2]. BGL predictive models could contribute to achieving this goal. They can help avert adverse glycaemic events by forecasting them and giving patients the chance to take corrective actions ahead of time [2].
The importance of the development of BGL predictive models in T1DM management has spurred research into this field [16,22]. According to the knowledge requirement, predictive models can be classified as; physiological, data-driven, and hybrid models [21]. Datadriven models interpret trends in sequences of data to make estimations of future BGLs. Machine learning approaches are broadly adopted in this area [21].
Mirshekarian et al. [17] developed a model to predict blood glucose in 30-minute and 60-minute horizons using a recursive neural network (RNN) with long short-term memory (LSTM) units. The model explored BGL, insulin, food, and activity information as inputs. For the same prediction horizons, Bertachi et al. [4] and Georga et al. [9], in separate studies, proposed predictive models. Bertachi et al. applied an artificial neural network contemplating glucose, insulin, carbohydrate and physical activity as inputs for their system. BGL profile, insulin, carbohydrate intake and physical activity were inputs for a support vector regression (SVR) in the model developed by Georga et al. Investigating continuous glucose monitoring (CGM) data by recursive and direct deep learning approaches, Xie et al. [22] recommended a model for BGL prediction. Martinsson et al. [15] proposed an automatic forecast model for a prediction horizon of up to 60 minutes using RNN. The model used only the information from past BGLs as input. Bunescu et al. [7] created descriptive features to train a SVR using a physiological model of blood glucose dynamics. Carbohydrate intake, insulin administration, and the current and past BGLs were inputs of their model. Despite extensive research devoted to the development of predictive models, the performance of the proposed models remains a challenge [3].
In this work, we contributed to the improvement of BGL prediction for T1DM by applying a multi-lag stacking methodology. Initially, three conventional regression tools-partial least squares, multilayer perceptron, and long-short term memory-were applied to forecast BGLs in horizons of 30 and 60 minutes. Each tool was trained twice; once on a lag of 30 minutes and once on a lag of 60 minutes of CGM data. Therefore, six basic models were created for each prediction horizon. For each horizon, three stacking systems were then developed where predictions from a selection of the basic models were used as features to train a new regression. The first two stacking systems followed a uni-lag approach. They used predictions from the three base models trained on a history of 30 minutes and 60 minutes, respectively. The third system was multi-lag and used predictions from all six base models. The stacking systems resulted in appreciable improvements in predictive accuracy as compared to the basic predictive models. The third stacking system showed a predictive performance better than the other systems. This is the first paper, to our knowledge, that has combined models with different time-lags to generate a multi-lag BGL prediction system.
The Ohio T1DM dataset comprises several features collected from 12 individuals with type 1 diabetes in 8 weeks [14,13]. The last ten days' worth of data for each contributor was considered as the test set. Data for a cohort of six subjects was released in 2018 for the first BGL prediction challenge [14]; data for another six subjects was released in 2020 for the second challenge [13].
In this work, the 2020's data was investigated for developing and evaluating predictive models. Among the collected features were CGM data every 5 minutes, which was the only feature explored in this work. A brief description of the CGM data in the Ohio T1DM dataset released for 2020 BGL prediction challenge is displayed in Table 1.
As mentioned earlier, this work proposes methodologies to predict BGL in horizons of 30 and 60 minutes. The detail of the pursued methodologies is presented in this section
The first pre-processing task was taking care of missing data. Missing data in the training set was imputed applying a simple linear interpolation. Alternatively, for the test set, a linear extrapolation was employed. This was to ensure the model is not contaminated by observing future data in its pre-processing stage.
The next pre-processing step was transferring the time series forecasting problem to a supervised learning task. To this end, a rolling window consisting of a lag and future data was used as explanatory and dependent variables respectively. To give an illustration, for forecasting BGL of 30 minutes later using a history of 60 minutes, for example, we used a window with the length of 18. As a consequence of the 5-minute interval between data points, it therefore follows that the first 12 data points in the window were explanatory variables, and the rest were dependent variables.
First, six basic predictive models were created by means of three conventional regression tools. Subsequently, employing stacking learning, three more advanced predictive systems were developed where a collection of the basic models were considered as base-learners and a partial least squares regression as meta-learner. All proposed models/systems were personalised to individuals.
Initially, for each prediction horizon of 30 and 60 minutes, the following three conventional regressions tools were employed to generate six basic predictive models-two models by each tool. For this purpose, these tools were trained once on a history of 30 and once on a history of 60 minutes.
PLSR, as a basic linear regression, holds substantial popularity in different applications due to its easy-to-apply nature and minimal computation time requirement. In a previous work, we applied PLSR for glucose quantification which provided promising results [12]. In this work, PLSR was used as one of the regression tools. For the number of components, different values ranging from 1 to the length of the input variable were tried. Each time, the predicted residual sum of squares (P RESS) was calculated as follows. The number of components (A) resulting in the minimum value for P RESS/(N − A − 1) was then selected [20].
where, N is the size of the evaluation set , yi is reference value, and ŷi is predicted value.
• Multilayer perceptron (MLP)
An MLP [18] with an architecture of one hidden layer including 100 nodes and an output layer was implemented. ReLU was used as the activation function for he hidden layer, Adam as the optimiser, and mean absolute error as the loss function. Learning rate was 0.01, and the training process was based on 100 epochs.
• Long short-term memory (LSTM)
We used a Vanila LSTM [10] composed of a single hidden LSTM layer with 200 nodes, a fully connected layer with 100 nodes, and an output layer. ReLU was the activation function for both hidden layers, mean squared error was the loss function, and Adam was the optimizer. The model trained on 100 epochs with a learning rate of 0.01.
Ensemble learning is a machine learning technique that combines decisions from several models to create a new model. Stacking (Figure 1) is an ensemble approach that uses predictions from multiple base-learners (first level models) as features to train a meta-learner (second level model). This meta-learner then makes the final predictions on the test set [23]. In this paper, for each prediction horizon of 30 and 60 minutes, three stacking systems comprised of two uni-lag and one multi-lag were developed.
The three basic models trained on a history of 30 minutes were the base-learners of this uni-lag system and a PLSR was its metalearner.
• System 2 This system was also uni-lag. It was similar to system 1, except it used the three basic models trained on a history of 60 minutes in place of 30 minutes as base-learners.
In this multi-lag system, all the six basic models were considered as the base-learners and again a PLSR was the meta-learner. By performing a multi-lag approach the idea was to help capture a broader frequency range of BGL dynamics.
The test set was held out, and the train set was used to create the predictive models/systems. The developed models/systems were then utilised to predict the test data. The set of evaluation points starts 60 minutes after the beginning of the test set. First evaluation points would be otherwise similar to the training data, and it can affect the reliability of the results. Hence, the number of evaluated points for each patient is 12 less than the number of test examples mentioned in Table 1. Root mean square error (RMSE) and mean absolute error (MAE) were calculated as follows and then used as evaluation metrics.
where, N , yi , and ŷi carry the same definition as in (1).
This section presents the evaluation results for both the basic models and stacking systems. Models/systems with a performance depended on random initialization ran five times, and corresponding results have been reported in the form of mean and standard deviation. Extrapolated points were excluded when calculating the evaluation metrics. All models were built to predict future BGLs up to the end of the intended prediction horizon, but only the evaluation results for the horizon of interest are reported.
The results of the RMSE and MAE of the basic predictive models for the prediction horizon of 30 minutes are displayed in Table 2.
Based on the average of RMSE and MAE for all patients, LSTM trained on a history of 30 minutes showed the best performance among the basic models. PLSR with 60-minute lag was the secondbest model. All models had satisfactory standard deviations.
LSTM yielded the best overall predictive accuracy among the three regression tools. However, the results of the other two tools were also comparable to that of LSTM. It is worth remarking that PLSR, as a linear regression tool, was able to generate results comparable to that of LSTM and even better than that of MLP.
Among all patients, patient 552 had the best overall evaluation results. The worst results, on the other hand, belonged to patients 584 and 540.
Table 3 shows the evaluation results of the stacking systems for a prediction horizon of 30 minutes. For all patients, the performance of the stacking systems surpassed that of the basic models. System 3 proposed the best predictions overall based on average RMSE and MAE values. This system resulted in the best predictive accuracy for all patients except patient 544 and 584. All systems possessed small standard deviation values. The best result among all patients belonged to patient 552. The worst results, on the other hand, were those of patients 584, 540, and 567.
BGL prediction improved using stacking learning concepts. Initially, a time series problem was translated into a supervised learning task. Three conventional regression tools were trained with on different history length of 30 and 60 minutes, resulting in six basic predictive models. Predictions from the basic models trained with a history of 30 minutes were fed as features to a regression to build a combined learner. The learner was then used to make final predictions on the test set. The same scenario was repeated using the basic models trained on 60-minute lag observations. In both cases, the combined learner was able to make more accurate predictions on the test set. The overall performance further improved when predictions from all basic models-trained on both histories of 30 and 60 minutes-were considered as features to train a new learner.
For data analysis we used Python 3.6, TensorFlow 1.15.0 and Keras 2.2.5. Pandas, NumPy and Sklearn packages of python were used. The codes are available at: https://gitlab.com/ Heydar-Khadem/multi-lag-stacking.git