Online learning platforms also known as Computer Aided Education Systems have recently grown in importance owing to their ability to personalize study plans in accordance with individual student requirements. Learning platforms have modeled student knowledge state using student responses with the recently popular Deep Knowledge Tracing (DKT) technique. Using context information has also proven efective in various predictive problems prompting learning platforms to store a variety of context features about a student's performance history. An example context may be response time, where shorter times to answer questions may indicate higher mastery of a skill. Therefore, it is crucial to incorporate context features in the most efective way possible. Most of the research in DKT either use no context features, or use a set of context features that span only a narrow set of student characteristics. To overcome this, we identify a wide set of context features and incorporate their interactions into the DKT model. We then observe the efects of incorporating these additional context feature interactions and also propose an adaptive weighting technique that learns the appropriate context feature interaction weights. These techniques are compared with state-of-the-art baselines and their performances were evaluated using AUC scores.
Computer Aided Education (CAE) systems aim to personalize the study plan of a user to best
suit his/her needs. This is achieved through the process of Knowledge Tracing where the
current knowledge state of the user is estimated using the history of their interactions with the
system, and this estimated knowledge state is used to predict the future performance of the user.
Accurately predicted future student performances are then used as cues to better personalize
the study plan of each user. In addition to a history of user responses, CAE systems usually
also store additional metadata related to user performance history, like response time, type of
question, number of attempts, etc. Adomavicius et al. [
A popular approach to knowledge tracing over the past few years had been Deep
Knowledge Tracing [
A drawback of BIDKT was using only a small set of additional context features (narrow), in this case, student forgetting behavior. While including only a few features led to a reasonable improvement in performance, it would be of interest to know if the trend would have continued to rise as more related contexts were identified and included in the model. Additionally, the bi-interaction technique used, assigned the same weight to all feature interactions. This might be a problem if a larger set of context features are used, as important interactions might get diluted along with unimportant ones. This may result in either saturation or even a drop in model performance.
In this paper we posit that using additional context features (wide) should lead to improved
performance of knowledge tracing models. We further hypothesize that existing models may
not be well suited to efectively use additional contexts since they do not weigh contexts by
their importances. To verify these ideas, we first identify additional contexts which may
provide important cues for future student performance prediction. We then analyze how current
best model performance changes with wider contexts. Finally, we propose a new technique
by modifying BIDKT which adaptively learns weights for contexts via an Attention network
similar to [
Knowledge Tracing: Since the emergence of Long Short-term Memory(LSTM) Networks,
the Deep Knowledge Tracing model has been the most popular knowledge tracing technique
[
Using Context Features in Knowledge Tracing: Given the success of using context
feature and their interactions in other domains [
There have also been eforts to incorporate context features in the form of interactions for
the task of knowledge tracing. Vie et al. [24] uses Factorization Machines to model the
interaction between a wide variety of features. Nagatani et al. [
In this section, we will provide some background to the domain of knowledge tracing and also describe the architectures of the DKT and BIDKT models. These models are the basis for our proposed architecture.
Knowledge tracing is the process of estimating a student’s current knowledge state and using it to predict future performance. Given a sequence of past learning attempts 0 ⋯ , we need to predict the student’s performance for attempt +1. In general, an attempt = ( , ) is defined as a tuple that contains the skill set id ( ) of a question at time step and whether the student response ( ) to the question is correct or not. In this case, is identified as a skill set id from a set of skills and is a binary variable. We need to predict +1 for +1.
Deep Knowledge Tracing (DKT) shown in Figure 1(a), models students’ knowledge state
transition using an LSTM, which is a modified version of the RNN. The architecture of the DKT
model is from [
In the case of DKT, the input is a one-hot vector, which is the Cartesian product of and . is then embedded into a dense real-valued vector . During the knowledge state estimation process, for a given input = ( , ) at each time step , the knowledge state is updated. The knowledge state is estimated using the embedded vector and previous knowledge state −1 using the LSTM module. For the prediction process, the output layer is implemented as a linear layer with sigmoid activation. The predicted probability of correct responses to all skill sets ∈ ℝ| | formed the model output.
Bi-Interaction Deep Knowledge Tracing (BIDKT) [
The input to the RNN module is computed using an integration technique called biinteraction. is the product of the interactions between vector of the input , and , the embedded dense real-valued vector of each context feature the embedded dense real-valued =1 = ∑ ⊙
previous knowledge state −1 and the product of integration as: Here, is the number of context features. The current knowledge state is computed using the To predict the student’s performance at the next attempt, the interaction between the current knowledge state
and context at the next attempt +1 parameters are shared between the current knowledge state estimation step and the future is computed. The context embedding performance prediction step. (1) (2) (3) (4) And finally the probability of answering correctly ∈ ℝ| | is computed as: where (⋅) is the sigmoid function, ∈ ℝ
| |× is the weight matrix, and bias vector of the output. The implementation of the output layer is similar to that of DKT. ∈ ℝ | | is the
Our proposed model Attentional Bi-Interaction Deep Knowledge Tracing (ABIDKT), shown in ifgure 1(c) is an extension to the BIDKT model, which weights interactions between the skill id and context features in the BIDKT model. The original BIDKT model uses a narrow set of context features related only to the long term trait of forgetting. But our goal is to use a wider set of features so as to better estimate the knowledge state and accurately predict future performance. The additional context features are wins, fails, question type, previous attempt response time and diference in previous attempt response time which have been described in detail in Section 5.2.
The wins and fails context features have been picked up from [
However, using this wider set of features could lead to the issue of the important interactions being averaged out. Therefore, the ABIDKT model uses an attention network in a modified =1 = ∑ ( ⊙ )
= ( , −1) =1 = ∑ ( ⊙ +1) integration technique to weight the important interactions and ensure that they do not get averaged out. embedded dense real-valued vector of each context feature.
In this case the input to the RNN module is computed using a modified integration technique called attentional bi-interaction. In this integration method, is the product of the interactions between
the embedded dense real-valued vector of the input , and , the weight normalized by the Softmax function are computed as: interaction calculated using the attention layer. The attention weight Here is the number of context features and ∈ ℝ is the normalized attention weight of the ′ and the attention ′ = ℎ ( ( ⊙ ) + ) = ∑ =1
′ ( )
′ ( )
Similar to BIDKT, the current knowledge state is computed using the previous knowledge state −1 and the product of integration as: To predict the student’s performance at the next attempt, the weighted interaction between the current knowledge state and context at the next attempt is computed as: (5) (6) (7) (8) (9) The probability of a correct answer ∈ ℝ implementation of the output layer is also the same as the DKT and BIDKT models. Similar to the architecture of BIDKT, the parameters of context embedding are shared between the current knowledge state estimation step and the future performance prediction step in the ABIDKT model as well. In the case of the attention network parameters, 2 variations were experimented with, one where the attention network parameters are shared and the other | | is computed in the same way it is for BIDKT. The where the parameters are not shared.
The training parameters for BIDKT are the skill id ( ) embedding matrix , weights of the RNN, weight and bias
for prediction and embedding matrix for the context information. In the case of ABIDKT we additionally have to train weight , bias parameter of the attention layer. These parameters are jointly learned by minimizing a standard cross entropy loss between the predicted probability of correctly answering the next and question for the skill id +1 and the true label +1: = − ∑( +1log( ( +1)) + (1 − +1)log(1 − ( +1)))
where ( +1) is the one-hot encoding for which skill id is answered in the next time step + 1.
The training process for the ABIDKT model is the same as the training process for BIDKT and DKT. The main diference between the models lies in the set of trainable parameters.
Experiments were conducted to compare the performances of the proposed architecture ABIDKT
with BIDKT and DKT with diferent combinations of context features. The experiments were
conducted to verify the following 2 hypotheses:
1. The bi-interaction technique used in the BIDKT architecture cannot efectively leverage
a wider set of context features than those used in [
The datasets chosen for the experiments are the Assistments 2012-2013 [25] dataset which contains information about students studying school level Mathematics with multiple question types, and the Slepemapy.cz [26] dataset which contains data from an online platform that teaches primary school Geography mainly consisting of 2 question types.
Dataset Assistments 2012-2013 slepemapy.cz #records 5,818,868 10,087,305 #users 45,675 87,952 #items
266 1,458
Records where user made only a single attempt of a single skill set item were removed as
in [
• binary discrete value 0,1 for the Slepemapy.cz dataset. 7. previous attempt response time: time taken for the response of the previous attempt of the same skill, in seconds. 8. diference in previous attempt response time: calculated as the diference in response times of the last 2 attempts in seconds, of the same skill.
All features except for question type were discretized using the 2 scale. The repeated time
gap, sequence time gap and past trial counts features are same as the context features used
in BIDKT [
The set of hyperparameters which maximized averaged AUC over 5-fold cross validation were chosen for final model implementations. Final results on corresponding test sets were also reported using the AUC metric.
The hyper-parameters were set as follows:
1. learning rate: varying the learning rate did not have a significant efect on the maximum
value of AUC. Various learning rate values between 0.001 and 1 were tried, at values that
were approximate multiples of 3 i.e. 0.001, 0.003, 0.01, 0.03, etc. The value was further
ifne-tuned around the best performing value. Finally, the learning rate was set at 0.7 for
the Assistments 2012-2013 dataset, except for the DKT architecture which was fixed at
0.5. For the Slepemapy.cz dataset, the learning rate was fixed at 0.9 for all architectures.
2. hidden layer dimensions: Diferent values of hidden layer dimension between 10 and 100
were tried at diferences of 10, and the value was empirically set at 30 for all variations
of architectures and datasets.
3. dropout: The value of dropout had been set using the best value of dropout from the
experiments in [
The results shown are for the DKT, BIDKT and 2 variations of the ABIDKT architecture respectively. The results for the BIDKT and ABIDKT architectures are shown for diferent number of features. The feature combinations are: (a) (b) • forgetting: repeated time gap + sequence time gap + past trial counts • forgetting+3: forgetting features + wins + fails + question type • forgetting+5: forgetting+3 + previous attempt response time + diference in previous attempt response time The variations of the ABIDKT architecture are as follows: • ABIDKT-SP: The parameters of the attention network and bi-interaction layer are shared between the knowledge state estimation step and the future performance prediction step similar to the BIDKT architecture • ABIDKT: The parameters of the bi-interaction layer are shared between the knowledge state estimation step and the future performance prediction step, while the parameters of the attention network are trained independently
Figures 2(a) and 2(b) show the average test AUC results across 5 folds, for diferent combinations of features and being incorporated in diferent architectures. From the results we can observe that sharing trainable parameters between the knowledge state estimation step and future performance prediction step (ABIDKT-SP) does not have any significant impact on the performance, although not sharing parameters (ABIDKT) does perform marginally better when the number of features are increased for both datasets. The main takeaways from the results are as follows:
Hyperparameter Tuning and Reproducibility. All baselines were reproduced and their
hyperparameters were tuned using the same methodology as for the proposed model. We found that
our tuning method led to an AUC improvement of 0.7% for both models for the Assistments
2012-2013 dataset compared to values stated in [
Efect of Additional Context Features. Including wide context features led to improvements in AUC for both BIDKT and ABIDKT models and on both datasets, suggesting that identified features encapsulated information indicative of future student performance. Also, in support of hypothesis 1 stated in Section 5, the extent of improvement in BIDKT tapered of and there was negligible change when number of features was increased from 5 to 8.
Efect of Attention Layers. Contrary to hypothesis 2, adaptive learned context weights in the form of attention layers did not provide a substantial improvement in model performance, instead consistently achieving an AUC 0.1% less than its BIDKT counterpart. This may be because the added context features are not large in number, and attention layers involve more trainable parameters. Trained on the same amount of data, the benefit from fewer trainable parameters in BIDKT outweighs the adapative weight assignments learned by attention layers.
We conducted further analysis by computing the micro-AUC by binning the predictions based on past trial counts as shown in Figure 3 and computing the percentage improvement of the ABIDKT architecture over the BIDKT architecture for each bin. The bin sizes were chosen so as to balance the number of samples in each bin. This analysis was performed on the Assistments 2012-2013 dataset as this is a Mathematics tutor dataset where each user is bound to have a large number of trials. From these results we can observe that for low trial counts, ABIDKT does not show an improvement over BIDKT, but as the number of trials increase, the percentage improvement of ABIDKT over BIDKT also steadily increases for all sets of features. This shows that ABIDKT may be useful in databases where each student has a large number of trials.
The focus of this paper was to observe the efect of using a wider range of context features in the BIDKT model and to propose techniques to efectively incorporate them. We first identified a wider set of context features and incorporated them in the BIDKT model. Experimental results on 2 datasets showed that increasing the number of context features improves the performance of BIDKT significantly, but the performance begins to taper of as the number of features are increased from 5 to 8. We postulated that this could be because of important feature interactions being diluted with other unimportant ones. To overcome this drawback, we proposed a technique that adaptively learns the weights of feature interactions and incorporated this as an attention layer in the BIDKT model. Experimental results of these models on 2 datasets show that this weighting technique was not suficient to improve the performance of our models. This could be because we were trying to learn additional parameters using the same amount of data. We therefore analyzed the performance of our model across diferent trial counts and found that our model does outperform BIDKT when the number of past trial counts are high.
In future work, we first plan to implement our models on datasets that have a higher number of trials counts per student. We also plan to modify the architecture of Attention and see if this can perform better than the ABIDKT model. Additionally, we plan to try these approaches in an architecture where skill is modeled separately as in Dynamic key-value Memory Networks for Knowledge Tracing.