Learning analytics should go beyond data analysis and include approaches and algorithms that are meaningful for learner performance and that can be interpreted by teacher and related to learning outcomes. Assessment analytics has been lagging behind other research in learning analytics. This holds true especially for peer-assessment analytics. In this paper we present a mathematical model for peerassessment based on the use of scoring rubrics for criteriabased assessment. We propose methods for the calculation of the nal grade along with reliability measurues of peerassessment. Modeling is motivated and driven by the identied peer-assessment scenarios.
in [
Aim of this paper is to model peer-assessment and to
discuss issues of nal grade calculation and reliability of raters'
judgments. Jonsson and Svingby note that variations in
raters' judgments can occur either across raters, known as
inter-rater reliability, or in the consistency of one single rater,
called intra-rater reliability. Referring to [
In the following sections we describe two major peer-assessment scenarios we have recognized and for which we have developed a mathematical model. After that we present and analyze a model for these scenarios.
Reliability of peer-assessment depends on many factors but
consistency of individual evaluator was very early recognized
as the most important (see [
We analyze scenarios with respect to the experience of evaluator as is shown in scenario grid (Fig. 1). We have placed a continuum of possible scenarios in a grid with four quadrants. Within four quadrants we recognize two interesting scenarios for peer-assessment and discard the other two as either unrealistic or inappropriate.
In the rst scenario, let us call it Scenario A, participants are inexperienced evaluators (for example undergraduate students with introductory domain knowledge and no experience in peer-assessment) whereas in the scenario B evaluators have higher expertize in the evaluated domain (i.e. teachers, graduate students or senior undergraduates) and prior training in assessment. In scenario A, the lack of experience in evaluation must be compensated with a quantity of peerassessments, i.e. having a larger group size in peer-assessment. On the other hand, setting a group size too large in scenario B is a needless waste of expert's time. inexperienced evaluators (Scenario A) experienced evaluators (Scenario B)
t appropriate no small size of a peer assessment group large
Peer-assessment activity starts after the work on the assignment task has completed. In a general case peer-assessment consists of two phases. We identify following activities in the whole process.
i. Learners assess a (prede ned) number of assigned assignments ii. Analysis of peer-assessments
(grouped by assignment) iii. Calculation of the assignment grade
i. Analysis of peer-assessments
(grouped by grader) ii. Calculation of the assessment grade First phase starts with learners assessing the assignment work of their peers. We assume that each participant grades several assignments (at least 2). At the end of the rst phase a reliability check has to be performed and the nal grade has to be calculated. Second phase is concerned with the quality of assessments relative to the evaluator. As on outcome of the second phase graders can receive a grade (points) for the quality of their assessments.
We recognized three challenges: (1) calculation of the nal grade based on di erent assessment scenarios, (2) measurement of the assessment's reliability and (3) measurement of reliability of each grader (for grading of the graders).
A grading G from the scoring rubric with n criteria is a
tuple of numbers G = (g1; : : : ; gn). We consider gradings as
points in an n-dimensional space endowed with a metric d,
i.e. a function that measures the distance between points
(i.e. gradings) and satis es the axioms of a metric space.
In [
An assignment graded through peer-assessment will receive several peer gradings. These will have to be analyzed. If estimated as reliable these gradings will be use as input for the calculation of the nal grade.
A simplest approach is to calculate the nal grade of assignment as the mean value of received assessments.
Let S = fSk1; : : : ; Skmg denote a set of peer gradings for assignment k, then the mean grade is
M (S) = (af1 ; : : : ; afn); where aif = 1 m m X c(kj;)i j=1 ! M (S) is a center of mass of the set S. This method for grade calculation is suitable for scenario A. We can say that M (S) is sensitive to quantity, and less sensitive to outliers (it \respects the decision of the majority ").
For scenario B, we propose an alternative grade calculation
method (see [
O(S) = (of1 ; : : : ; ofn); where oif =
W (S) + B(S) :
W (S) and B(S) represent amalgamations of worst and best
received gradings respectively, de ned by:
This approach is inspired by Hwang and Yoon's TOPSIS
(Technique for Order of Preference by Similarity to Ideal
Solution) method of multi-criteria decision making in [
A summary of our recommendations for two scenarios A and B is given in the Table 2.
A prerequisite for the calculation of the assignment's nal grade is the determination whether a received set of peerassessments is (su ciently) reliable, i.e. acceptable. For reasoning about reliability it is necessary to have granular data. The importance of granular scoring data is illustrated in the example in Table 3. Gradings S1 and S2 agree on the summative level, but seem very distinct at the granular level. This is an example of an unreliable peer-grading set where this incoherence is not visible on the summative level. We consider a set S of peer gradings as reliable if diam S (maximal pairwise distance between gradings) is less than 2e where e is acceptable error given in advance.
Note that the diameter of the set S is also a diameter of an encompassing sphere. So, we can say that a reliable peer-grading set ts within an encompassing e-sphere. If a set of peer-assessments is estimated as not acceptable (un-reliable) on the granulated level then the nal grade cannot be calculated. A recommendation about acceptability of particular peer-assessment set can be given to teacher or course designer by LA. This can be implemented in the learning management system (LMS, for example Moodle). Practical related issues will be discussed in the section 5.
Assessment set can turn out as unacceptable because of a single outlier grading. As an attempt to eliminate the outlier grading we propose to search for a maximal acceptable subset of the received peer-assessments. If such subset can be found, it is then used as input for the nal grade calculation. As a measure of nal resort, an supervisor's intervention is asked for. In a course with a large student enrollment (thousands for a MOOC) this will be avoided as much as possible. However, if present, instructor's assessment becomes a nal grade (no need for calculation). This is described in Algorithm 1.
Metric d can be linearly scaled to obtain a normalized metric d0 with values within the interval [0; 1]. Distance of d0 = 1 corresponds to the maximal distance between worst and best possible gradings.
This would facilitate having general recommendations for setting acceptable error e on a normalized scale (setting Algorithm 1: Semi-autonomous Grading Process input : Set of gradings S = fS(1); : : : ; S(m)g, acceptable error e 0 grading calculation method g critical size N (i.e. N = 3) output : Final grade or indicate gradings S as invalid 1 nd a maximal S0 S with acceptable error 2 if #(S0) N then 3 nd S00 of size #(S00) = #(S0) of minimal diameter 4 return g(S00) as a proper grade for assignment k else 5
Ask for teacher intervention (grading) e0 = 0:2 for example). Additionally, this could facilitate comparison of data from di erent tasks (within a course, or from di erent courses).
Goal of the second phase of the peer-assessment process is to reward the graders for their e ort. Graders (peers) who have graded consistently and accurately (near the nal grade) should be rewarded more than inconsistent and inaccurate graders.
Let us assume that a maximum of A points is awarded for the peer-assessment task. Then grader k can be awarded Ai points for each of the m gradings Gi that he/she was assigned, where Ai is calculated by the following formula Ai(di) := 8 A < me
A support for peer-assessment LA is lacking in assessment analytics in general. We analyze the current implementation in the Moodle LMS where peer-assessment activity is implemented with the Workshop plugin.
In a Workshop activity, students receive a grade for their work and another grade for the quality of their assessment of other student's assignments.
Each participant in Workshop gets a grade for his submission and a grade for her assessments. These grades are visible as separate grade items in student's gradebook.
Current implementation of Workshop calculates the
assignment grade as a weighted mean of received assessment
gradings. Received gradings are not analyzed for reliability. If the
teacher wishes to override or in uence the calculated
assignment grade, he can (a) additional provide his own assessment
and set its corresponding weight to a higher value or (b) even
completely override the nal grade. As we have argued here
and in [
Assessment grade calculation is more complex. The goal is to estimate the quality of each assessment. One assessment is singled out as the best one { it is the assessment closest to the mean value of all assessments. This selected assessment is assigned with highest grade. Other assessments receive grades based on the distance from the selected assessment. Teacher can in uence in this process by setting the parameter which determines how quickly a grade should decreased relative to the distance.
We are currently developing a new Moodle plugin for peerassessment. This plugin will address the identi ed problems of the current implementation according to our model.
Peer-assessment has many advantages for students (for example development of metacognitive skills) and for teachers (for example saves teacher's time) but there are several challenges related to their implementation such as calculation of nal grade, reliability check and awarding an evaluator for peer-assessment.
In this paper we propose new methods for calculation of
the grades in peer-assessment. We propose a measure for
reliability and a method for grading peer-evaluations in a
peer-assessment exercise. These metrics are based on two
distinguished scenario analysis that takes into account a
number of possible evaluators and evaluator expertize (domain
knowledge and evaluation skills). We pursue an approach to
model assessment LA analytics with a geometric model.
In [
Also, we intend to implement our model (algorithms and the supporting recommendation system) as a peer-assessment plug-in for the Moodle LMS.
Finally, we conclude that a well founded mathematical modeling, based on not just descriptive statistics, should be used more often in learning analytics. [11] Moodle LMS (https://moodle.org/)
Plugins available on January 10th, 2016. at https://moodle.org/plugins/