We use the Additive Factors Model to drive the evaluation of the student model of an Intelligent Tutoring System. Using data from the Andes Physics Tutor, applying the simple location heuristic and implementing the Additive Factors Model tool in the Pittsburgh's Science of Learning Center's DataShop, we discover possible ways to improve the student model of the Andes Intelligent Tutor.
Learning curves derived from student models drive evaluation,
revision and improvement of the Intelligent Tutor. The AFM is a
statistical algorithm which models learning and performance by
using logistical regression performed over the “error rate”
learning curve data [
While use of learning curves is now a standard technique for
assessing the cognitive models of Intelligent Tutors, the technique
requires that a method is instated for attributing blame to skills or
KCs. This simply means that each error a student makes must be
blamed on a skill or set of skills. Four different heuristics for error
attribution have been proposed and tested. These heuristics are
guided by whether the method is driven by location – the simple
location heuristic (LH), the model-based location heuristic
(MLH); or by the temporal order of events – the temporal
heuristic (TH), the model-based temporal heuristic (MTH); and
whether the choice of the student model is leveraged (MLH,
MTH) [
Prior to using the AFM tool on the dataset, the simple location heuristic (LH) was applied to error transactions in the Andes log data which had missing KCs. That is, when the Andes failed to assign blame to a KC on an error transaction, the LH will select the first correctly implanted KC in the same location as the error. The LH was applied to about 44% of the original data. Table 1 depicts a summary of the LH data. 2.2 Generating Model Values using AFM The Datashop’s AFM algorithm was used to compute statistical measures of goodness of fit for the model - Akaike Information Criterion (AIC) and Bayesian Information criterion (BIC), as well as to generate learning curves for the Andes log data. 3 RESULTS AND DISCUSSION We found that there were 5 groups of KCs – “Low and Flat”, “No learning”, “Still high”, “Too Little data” and “Good”. The “Low and Flat” group indicated KCs where students likely received too much practice. It appears that although students mastered the KCs they continued to receive tasks for them. It may be better to reduce the required number of tasks or change Andes’ knowledge tracing parameters so that students get fewer opportunities with these KCs. The “Still high” group suggests KCs, which students continued to struggle with. Increasing opportunities for practice for these KCs might improve the student model. The “No learning” group indicated KCs where the slope of the predicted learning curve showed no apparent learning. A step towards improving the student model could be to explore whether each of these KCs can be split into multiple KCs. The new KCs may better reflect the variation in difficulty and transfer of learning that may be happening across problem steps, which are currently labeled by each KC. The KCs in the “Too Little data” group seem to be KCs for which students were exposed to insufficient practice opportunities for the data to be meaningful. For these KCs, adding more tasks or merging similar KCs might provide data that is interpretable. The KCs that appeared “Good” may reflect those in which there was substantial student learning. Table 2 shows the different group of KCs, their frequencies and AIC and BIC scores. Figures 1a – 1d show the different groups of KCs. Intercept (logit) and intercept (probability) both indicate KC difficulty. Higher intercept values indicate more difficult KCs. The slope parameter indicates the KC learning rate. Higher values suggest students will learn such KCs faster.
Flat 2 AIC BIC # of Knowledge Components 2 5,857 71,300 107.02 34
Intercept (probability) 0.85 0.65 0.120 0.037
4 CONCLUSION AND FUTURE WORK
This paper presented how the AFM can be used to evaluate the
student model of the Andes Physics Tutor. Refining four of the
five groups of KCs identified, might improve the Andes student
model. A further approach would to use Learning Factors
Analysis [
Our thanks to the Pittsburgh Science of Learning Center for providing the analysis tool for this work, to Bob Hausmann and Kurt VanLehn for dataset access.