We propose a workflow to be implemented in a new workflows architecture of the LearnSphere We present the motivation and initial validation of the value of regularizing student-level parameters in an individualized Bayesian Knowledge Tracing model. Theoretically grounded, regularization of iBKT models, as we show, leads to increased model accuracy. Bayesian Knowledge Tracing (BKT) model individualization has been an active research topic recently. Individualization is accounting for student population variance that may come in different forms. One of the ways to individualize BKT is to introduce student-specific parameters. The new individualized BKT (iBKT) has been shown to fit the data better and even some promise, if deployed, to be able to save students time by better accounting for personal learning paths [7]. One of the toolkits that support iBKT models and is available via the LearnSphere workflow is hmm-scalable1. iBKT via hmmscalable, as compared to regular BKT [5], splits every BKT parameter into per-skill and per-student components and allows a subset or all BKT parameters to be individualized. Both studentand skill-level parameter are fit with the help one of the gradientbased approaches including stochastic or conjugate gradient descent, or a Lagrange updates method [6].
The solvers implemented in the core hmm-scalable utility that
the LearnSphere iBKT wrapper workflow uses are fitting both
skill and student parameters in fixed-factor fashion. However, it is
often advised to treat student-related variables as random factors.
The logic behind this is that students at hand are sampled from a
larger population of students [
In this paper, we propose to take advantage of the regularization feature in hmm-scalable. This feature adds a penalty to the objective log-likelihood function in the form of the L2 norm. That is an approximation of treating the regularized parameter as a random factor. The influence of the L2 term is weighted by the λ coefficient. The larger the λ, the stronger the effect of the regularization. In simpler terms, regularization penalizes deviation of parameter values from the centroid value. In regularized
hmm-scalable toolkit, https://iedms.github.io/standard-bkt the public version regression, the centroid is usually 0, and regularization takes a form of scarcity-inducing mechanism. In the case of iBKT, as it is implemented in hmm-scalable, an appropriate centroid is 0.5. If student-level parameters have a value of 0.5, this effectively means no student-level effect on the relevant still parameter. Making the right choice of the λ that would lead to an improved fit (and, potentially, the generalizability) is a problem in an of itself. The hmm-scalable does not support the search for an optimal λ directly. We propose to perform this search as part of the wrapping LearnSphere workflow. 2. 2.1
The student-parameter-regularizing workflow takes the same data inputs with regular BKT workflow. These are integer correctness variable (correct encoded as 1, incorrect as 2), student as a character factor, the problem as a character factor, and delimited skill(s) as character factor. 2.2
The workflow performs a sequence of runs of a chosen iBKT model with a set of λ weights for the regularizing L2 parameter penalty, where the first value is 0 (the default), and the rest could be defined by the user. For example, from 0.1 to 4, with a 0.1 increment. In this model, the skill parameters are not regularized. 2.3
The output of the workflow is built on the standard output of the BKT workflow. An array of statistical fit metrics (Log-likelihood, AIC, BIC, RMSE, Accuracy) is presented along with the λ weight used for the regularization. The main outcome is whether the regularization improves the fit of the model, as compared to the non-regularized version, and at what λ value. A graphical webfriendly plot of a selected via drop-down fit metric across all tested λ values could help better visualize the effect of the regularization. Skill and student level parameters could also be printed for the winning model.
As part of pilot-testing the method we propose to be implemented as a LearnSphere workflow, we have run a series of iBKT models with student-level parameter regularizations for a set of λ weights. Initially, we used values from 0 to 6 with a step of 1. Later, we added values 0.1 to 0.9 with a step of 0.1, 1.5, 2.5, and 3.5. We used the 2010 KDD Cup Challenge Set B (available via PSLC DataShop2), the largest available dataset of student learning data. 2 2010 KDD Cup https://pslcdatashop.web.cmu.edu/KDDCup The overall number of student-step transactions in that dataset is over 20 million. The data in that dataset was collected by the application called Cognitive Tutor. It was donated by Carnegie Learning Inc. and contained data of students working on a Bridge to Algebra course.
For each setting, we performed 5 runs of 2-fold student-stratified
cross-validation. As identified in [
For the sake of selecting the best λ, we used simple average across
5x2=10 statistical goodness of fit values. We picked RMSE and
Accuracy as our metrics. Figures 1 and 2 show the plots of the
RMSE and Accuracy (respectively) for the set of tested λ weights.
Although RMSE and Accuracy plots are not entirely in sync: the
best (lowest) RMSE value corresponds to λ=2.0, and the best
(highest) Accuracy value corresponds to λ=4.0, it is, arguably,
appropriate to choose λ=2.0, thus trusting RMSE metric.
To underline the usefulness of the regularizing student parameters
in iBKT models, we have performed the model ranking F-test
described in [
If we look at RMSE, BKT models, again, prevail. There is now the statistical difference between iBKT models: regularized iBKT model has an edge. Standard BKT model fit using EM is not statistically different from either iBKT model in terms of RMSE.