=Paper=
{{Paper
|id=Vol-1183/bkt20y_paper11
|storemode=property
|title= Evaluating Student Models
|pdfUrl=https://ceur-ws.org/Vol-1183/bkt20y_paper11.pdf
|volume=Vol-1183
|dblpUrl=https://dblp.org/rec/conf/edm/Nwaigwe14
}}
== Evaluating Student Models==
https://ceur-ws.org/Vol-1183/bkt20y_paper11.pdf
Evaluating Student Models
Adaeze Nwaigwe
University of Maryland University
College
3501 Unversity Blvd East
Adelphi, MD 207831 412 608 8747
adaeze.nwaigwe@faculty.umuc
.edu
ABSTRACT 2 EVALUATING THE STUDENT MODEL
We use the Additive Factors Model to drive the evaluation of the
student model of an Intelligent Tutoring System. Using data from 2.1Adapting the Andes Log data for the AFM
the Andes Physics Tutor, applying the simple location heuristic Algorithm
and implementing the Additive Factors Model tool in the The log data used for this work was obtained from the Andes
Pittsburgh’s Science of Learning Center’s DataShop, we discover Intelligent Tutor [4] and encompassed four problems in the area
possible ways to improve the student model of the Andes of electric field, across 102 students. The data was collected in
Intelligent Tutor. Spring 2005 at the US Naval Academy during its regular physics
class and as part of the PSLC’s LearnLab facility that provides
Keywords researchers, access to run experiments in or perform secondary
Student modeling, learning curves, additive factors model. analyzes of data collected from one of seven available technology-
enhanced courses running at multiple high school and college
sites (see http://learnlab.org).
1. INTRODUCTION Prior to using the AFM tool on the dataset, the simple location
The quality of student models drive many of the instructional
heuristic (LH) was applied to error transactions in the Andes log
decisions that automated tutoring systems make, whether it is
data which had missing KCs. That is, when the Andes failed to
what feedback to provide, when and how to sequence topics and
assign blame to a KC on an error transaction, the LH will select
problems in a curriculum, how to adapt pacing to the needs of
the first correctly implanted KC in the same location as the error.
students and even what problems and instructional materials are
The LH was applied to about 44% of the original data. Table 1
necessary [1]. We used the Additive Factors Model (AFM) tool in
depicts a summary of the LH data.
the Pittsburgh’s Science of Learning Center’s (PSLC) DataShop
to identify areas for improvement in the curriculum for the
ANDES Intelligent Tutoring System. 2.2 Generating Model Values using AFM
The Datashop’s AFM algorithm was used to compute statistical
1.1 BACKGROUND measures of goodness of fit for the model - Akaike Information
Learning curves derived from student models drive evaluation, Criterion (AIC) and Bayesian Information criterion (BIC), as well
revision and improvement of the Intelligent Tutor. The AFM is a as to generate learning curves for the Andes log data.
statistical algorithm which models learning and performance by
using logistical regression performed over the “error rate” 3 RESULTS AND DISCUSSION
learning curve data [1]. If a student is learning the knowledge We found that there were 5 groups of KCs – “Low and Flat”, “No
component (KC) or skill being measured, the learning curve is learning”, “Still high”, “Too Little data” and “Good”. The “Low
expected to follow a so-called “power law of practice” [2]. If such and Flat” group indicated KCs where students likely received too
a curve exists, it presents evidence that the student is learning the much practice. It appears that although students mastered the KCs
skill being measured or conversely, that the skill represents what they continued to receive tasks for them. It may be better to
the student is learning. reduce the required number of tasks or change Andes’ knowledge
While use of learning curves is now a standard technique for tracing parameters so that students get fewer opportunities with
assessing the cognitive models of Intelligent Tutors, the technique these KCs. The “Still high” group suggests KCs, which students
requires that a method is instated for attributing blame to skills or continued to struggle with. Increasing opportunities for practice
KCs. This simply means that each error a student makes must be for these KCs might improve the student model. The “No
blamed on a skill or set of skills. Four different heuristics for error learning” group indicated KCs where the slope of the predicted
attribution have been proposed and tested. These heuristics are learning curve showed no apparent learning. A step towards
guided by whether the method is driven by location – the simple improving the student model could be to explore whether each of
location heuristic (LH), the model-based location heuristic these KCs can be split into multiple KCs. The new KCs may
(MLH); or by the temporal order of events – the temporal better reflect the variation in difficulty and transfer of learning
heuristic (TH), the model-based temporal heuristic (MTH); and that may be happening across problem steps, which are currently
whether the choice of the student model is leveraged (MLH, labeled by each KC. The KCs in the “Too Little data” group seem
MTH) [3]. to be KCs for which students were exposed to insufficient practice
opportunities for the data to be meaningful. For these KCs, adding
more tasks or merging similar KCs might provide data that is
interpretable. The KCs that appeared “Good” may reflect those in
which there was substantial student learning. Table 2 shows the
different group of KCs, their frequencies and AIC and BIC scores.
Figures 1a – 1d show the different groups of KCs. Intercept (logit)
and intercept (probability) both indicate KC difficulty. Higher KC Name Intercept Intercept Slope
intercept values indicate more difficult KCs. The slope parameter (logit) (probability)
indicates the KC learning rate. Higher values suggest students will
learn such KCs faster. draw-efield-vector 0.06 0.52 0.000
Table 1. LH Data Summary Figure 1c – “No Learning”
Number of Students 102
Number of Unique Steps 125
Total Number of Steps 5,857
Total Number of Transactions 71,300
Total Student Hours 107.02
KC Name Intercept Intercept Slope
# of Knowledge Component Model 34 (logit) (probability)
compo-parallel-axis -0.28 0.43 0.000
Table 2. KC Groups and Statistical Scores draw-electric-force- -0.01 0.50 0.000
Low Good given-field-dir
No Still Too
and
Learning High Little data
Flat
2 2 4 24 2 Figure 1d – “Still High”
# of Knowledge Components 34
AIC 6532.75 4 CONCLUSION AND FUTURE WORK
This paper presented how the AFM can be used to evaluate the
BIC 7668.14 student model of the Andes Physics Tutor. Refining four of the
five groups of KCs identified, might improve the Andes student
model. A further approach would to use Learning Factors
Analysis [1] algorithm to automatically find better student models
by searching through a space of KC models. The next step is to
explore these options and measure their effect.
5 ACKNOWLEDGMENTS
Our thanks to the Pittsburgh Science of Learning Center for
providing the analysis tool for this work, to Bob Hausmann and
KC Name Intercept Intercept Slope Kurt VanLehn for dataset access.
(logit) (probability)
6 REFERENCES
define-constant- 1.77 0.85 0.120 [1] Koedinger, K.R., McLaughlin, E.A., Stamper, J.C. 2012
charge-on-obj-var Automated Student Model Improvement. Proceedings of the
write-known-value-eqn 0.63 0.65 0.037 5th International Conference on Educational Data Mining,
Chania, Greece, pp. 17–24.
[2] Mathan S. & Koedinger K. 2005. Fostering the Intelligent
Actual data predicted Novice: Learning From Errors With Metacognitive Tutoring.
Educational Psychologist. 40(4), pps. 257–265.
Figure 1a – “Good” [3] Nwaigwe, A. & Koedinger, K.R. 2011. The Simple Location
Heuristic is Better at Predicting Students’ Changes in Error
Rate Over Time Compared to the Simple Temporal
Heuristic. Proceedings of the 4th International Conference on
Educational Data Mining. Eindhoven, Netherlands.
[4] VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby,
R. H., Taylor, L., et al. 2005. The Andes physics tutoring
system: Lessons learned. International Journal of Artificial
Figure 1b – “Low and Flat” Intelligence and Education, 15(3), 147-204.