Many di erent metrics are used to evaluate and compare performance of student models. The aim of this paper is to provide an overview of commonly used metrics, to discuss properties, advantages, and disadvantages of di erent metrics, and to summarize current practice in research papers. The paper should serve as a starting point for workshop discussion about the use of metrics in student modeling.
However, despite the fact that the choice of metric is important and that there is no clear consensus on the usage of performance metrics, the topic gets very little attention in most research papers. Most authors do not provide any rationale for their choice of metric. Sometimes it is not even clear what metric is exactly used, so it may be even di cult to use the same metric as previous authors. The main aim of this paper is to give an overview of performance metrics relevant for evaluation of student models and to explicitly discuss points that are in most papers omitted.
To attain clear focus we discuss only models that predict probability of a correct answer. We assume that we have data about n answers, numbered i 2 f1; : : : ; ng, correctness of answers is given by ci 2 f0; 1g, a student models provides predictions pi 2 [0; 1]. A model performance metric is a function f (p~; ~c). Note that the word \metric" is here used in a sense \any function that is used to make comparisons", not in the mathematical sense of a distance function. Since we are interested in using the metrics for comparison, monotone transformations (square root, logarithm, multiplication by constant) are inconsequential and are used mainly for better interpretability (or sometimes rather for traditional reasons).
This basic metric consider the absolute di erences between predictions and answers: MAE = n1 Pin=1 jci pij. This is not a suitable performance metric, because it prefers models which are biased towards the majority results. As a simple illustration, consider a simulated student which answers correctly with constant probability 0.7. If we compare di erent constant predictors with respect to this metric, we get that the best model is the one which predicts probability of correct answer to be 1. This is clearly not a desirable result. As this example illustrates, the use of MAE can lead to rather misleading conclusions. Despite this clear disadvantage, MAE is sometimes used for evaluation (although mostly in combination with other metrics, which reduces the risk of misleading conclusions in published papers).
A similar metric is obtained by using squared values instead of absolute values: RMSE = q n1 Pin=1(ci pi)2. Note that from the perspective of model comparison, the important part is only the sum of square errors (SSE). The square root in RMSE is traditionally used to get the result in the same units of as the original \measurements" and thus to improve interpretability of the resulting number. In the particular context of student modeling and evaluation of probabilities, this is not particularly useful, since the resulting numbers are hard to interpret anyway. In order to get better interpretability researchers sometimes use R2 metric: R2 = 1 Pin=1(ci pi)2= Pin=1(ci c)2. With respect to comparison of models, R2 is equivalent to RMSE since here again the only model dependent part is the sum of square errors. In the context of the standard linear regression (where it is most commonly used) R2 has a nice interpretation as \explained variability". In the case of logistic regression (which is more similar to student models) this interpretation does not hold and di erent \pseudo R2" metrics are used (e.g., Cox and Snell, McFadden, Nagelkerke). Thus a disadvantage of R2 is that unless the authors are explicit about which version of R2 they use (usually they are not), a reader cannot know for sure which metric is reported.
In educational data mining the use of RMSE metric is very
common (it was also used as a metric in KDD Cup 2010
focused on student performance evaluation). In other
areas, particularly in meteorology, mean square error (RMSE
without the square root) is called the Brier score [
The likelihood of data (the answers) given a model
(predicted probabilities) is L = Qin=1 pici (1 pi)(1 ci). Since
we are indi erent to monotonic transformations we typically
work with the numerically more stable logarithm of the
likelihood LL = Pin=1 ci log(pi)+(1 ci) log(1 pi). This metric
can also be interpreted from information theoretic
perspective as measure of data compression provided by a model [
Another popular metric is based on the receiver operating
characteristics (ROC) curve. If we want to classify
predictions into just two discrete classes (correct, incorrect),
we need to select a threshold for the classi cation. For a
xed threshold we can compute standard metrics like
precision, recall, and accuracy. If we do not want to use a xed
threshold, we can use the ROC curve, which summarises the
behaviour of the prediction model over all possible
thresholds. The curve has \false positive rate" on x-axis and \true
positive rate" on the y-axis, each point of the curve
corresponds to a choice of a threshold. Area under the ROC curve
(AUC) provides a summary performance measure across all
possible thresholds. It is equal to the probability that a
randomly selected correct answer has higher predicted score
than a randomly selected incorrect answer. The area under
the curve can be approximated using a A' metric, which is
equivalent to the well-studied Wilcoxon statistics [
The ROC curve and AUC metric are successfully used in
many di erent research areas, but their use is sometimes
also criticised [
Nevertheless, despite this disadvantage, AUC is widely used for evaluation of student models, often as the only metric. It seems that in some cases AUC is used as the only metric for nal evaluation, but the parameter tting procedure uses (implicitly) di erent metric (RMSE or LL). Particularly in cases of brute force tting this approach seems strange and should be at least explicitly mentioned.