Evaluating Student Models Adaeze Nwaigwe University of Maryland University College 3501 Unversity Blvd East Adelphi, MD 207831 412 608 8747 adaeze.nwaigwe@faculty.umuc .edu ABSTRACT 2 EVALUATING THE STUDENT MODEL We use the Additive Factors Model to drive the evaluation of the student model of an Intelligent Tutoring System. Using data from 2.1Adapting the Andes Log data for the AFM the Andes Physics Tutor, applying the simple location heuristic Algorithm and implementing the Additive Factors Model tool in the The log data used for this work was obtained from the Andes Pittsburgh’s Science of Learning Center’s DataShop, we discover Intelligent Tutor [4] and encompassed four problems in the area possible ways to improve the student model of the Andes of electric field, across 102 students. The data was collected in Intelligent Tutor. Spring 2005 at the US Naval Academy during its regular physics class and as part of the PSLC’s LearnLab facility that provides Keywords researchers, access to run experiments in or perform secondary Student modeling, learning curves, additive factors model. analyzes of data collected from one of seven available technology- enhanced courses running at multiple high school and college sites (see http://learnlab.org). 1. INTRODUCTION Prior to using the AFM tool on the dataset, the simple location The quality of student models drive many of the instructional heuristic (LH) was applied to error transactions in the Andes log decisions that automated tutoring systems make, whether it is data which had missing KCs. That is, when the Andes failed to what feedback to provide, when and how to sequence topics and assign blame to a KC on an error transaction, the LH will select problems in a curriculum, how to adapt pacing to the needs of the first correctly implanted KC in the same location as the error. students and even what problems and instructional materials are The LH was applied to about 44% of the original data. Table 1 necessary [1]. We used the Additive Factors Model (AFM) tool in depicts a summary of the LH data. the Pittsburgh’s Science of Learning Center’s (PSLC) DataShop to identify areas for improvement in the curriculum for the ANDES Intelligent Tutoring System. 2.2 Generating Model Values using AFM The Datashop’s AFM algorithm was used to compute statistical 1.1 BACKGROUND measures of goodness of fit for the model - Akaike Information Learning curves derived from student models drive evaluation, Criterion (AIC) and Bayesian Information criterion (BIC), as well revision and improvement of the Intelligent Tutor. The AFM is a as to generate learning curves for the Andes log data. statistical algorithm which models learning and performance by using logistical regression performed over the “error rate” 3 RESULTS AND DISCUSSION learning curve data [1]. If a student is learning the knowledge We found that there were 5 groups of KCs – “Low and Flat”, “No component (KC) or skill being measured, the learning curve is learning”, “Still high”, “Too Little data” and “Good”. The “Low expected to follow a so-called “power law of practice” [2]. If such and Flat” group indicated KCs where students likely received too a curve exists, it presents evidence that the student is learning the much practice. It appears that although students mastered the KCs skill being measured or conversely, that the skill represents what they continued to receive tasks for them. It may be better to the student is learning. reduce the required number of tasks or change Andes’ knowledge While use of learning curves is now a standard technique for tracing parameters so that students get fewer opportunities with assessing the cognitive models of Intelligent Tutors, the technique these KCs. The “Still high” group suggests KCs, which students requires that a method is instated for attributing blame to skills or continued to struggle with. Increasing opportunities for practice KCs. This simply means that each error a student makes must be for these KCs might improve the student model. The “No blamed on a skill or set of skills. Four different heuristics for error learning” group indicated KCs where the slope of the predicted attribution have been proposed and tested. These heuristics are learning curve showed no apparent learning. A step towards guided by whether the method is driven by location – the simple improving the student model could be to explore whether each of location heuristic (LH), the model-based location heuristic these KCs can be split into multiple KCs. The new KCs may (MLH); or by the temporal order of events – the temporal better reflect the variation in difficulty and transfer of learning heuristic (TH), the model-based temporal heuristic (MTH); and that may be happening across problem steps, which are currently whether the choice of the student model is leveraged (MLH, labeled by each KC. The KCs in the “Too Little data” group seem MTH) [3]. to be KCs for which students were exposed to insufficient practice opportunities for the data to be meaningful. For these KCs, adding more tasks or merging similar KCs might provide data that is interpretable. The KCs that appeared “Good” may reflect those in which there was substantial student learning. Table 2 shows the different group of KCs, their frequencies and AIC and BIC scores. Figures 1a – 1d show the different groups of KCs. Intercept (logit) and intercept (probability) both indicate KC difficulty. Higher KC Name Intercept Intercept Slope intercept values indicate more difficult KCs. The slope parameter (logit) (probability) indicates the KC learning rate. Higher values suggest students will learn such KCs faster. draw-efield-vector 0.06 0.52 0.000 Table 1. LH Data Summary Figure 1c – “No Learning” Number of Students 102 Number of Unique Steps 125 Total Number of Steps 5,857 Total Number of Transactions 71,300 Total Student Hours 107.02 KC Name Intercept Intercept Slope # of Knowledge Component Model 34 (logit) (probability) compo-parallel-axis -0.28 0.43 0.000 Table 2. KC Groups and Statistical Scores draw-electric-force- -0.01 0.50 0.000 Low Good given-field-dir No Still Too and Learning High Little data Flat 2 2 4 24 2 Figure 1d – “Still High” # of Knowledge Components 34 AIC 6532.75 4 CONCLUSION AND FUTURE WORK This paper presented how the AFM can be used to evaluate the BIC 7668.14 student model of the Andes Physics Tutor. Refining four of the five groups of KCs identified, might improve the Andes student model. A further approach would to use Learning Factors Analysis [1] algorithm to automatically find better student models by searching through a space of KC models. The next step is to explore these options and measure their effect. 5 ACKNOWLEDGMENTS Our thanks to the Pittsburgh Science of Learning Center for providing the analysis tool for this work, to Bob Hausmann and KC Name Intercept Intercept Slope Kurt VanLehn for dataset access. (logit) (probability) 6 REFERENCES define-constant- 1.77 0.85 0.120 [1] Koedinger, K.R., McLaughlin, E.A., Stamper, J.C. 2012 charge-on-obj-var Automated Student Model Improvement. Proceedings of the write-known-value-eqn 0.63 0.65 0.037 5th International Conference on Educational Data Mining, Chania, Greece, pp. 17–24. [2] Mathan S. & Koedinger K. 2005. Fostering the Intelligent Actual data predicted Novice: Learning From Errors With Metacognitive Tutoring. Educational Psychologist. 40(4), pps. 257–265. Figure 1a – “Good” [3] Nwaigwe, A. & Koedinger, K.R. 2011. The Simple Location Heuristic is Better at Predicting Students’ Changes in Error Rate Over Time Compared to the Simple Temporal Heuristic. Proceedings of the 4th International Conference on Educational Data Mining. Eindhoven, Netherlands. [4] VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R. H., Taylor, L., et al. 2005. The Andes physics tutoring system: Lessons learned. International Journal of Artificial Figure 1b – “Low and Flat” Intelligence and Education, 15(3), 147-204.