For peer assessments to be helpful, student reviewers need to submit reviews of good quality. This requires certain training or guidance from teaching staff, lest reviewers read each other's work uncritically, and assign good scores but offer few suggestions. One approach to improving the review quality is calibration. Calibration refers to comparing students' individual reviews to a standard-usually a review done by teaching staff on the same reviewed artifact. In this paper, we categorize two modes of calibration for peer assessment and discuss our experience with both of them in a pilot study with Expertiza system. Educational peer review; peer assessment; calibration.
In educational peer-review systems, students submit their artifacts
and other students rate and/or give comments on artifacts
submitted by their peers. Previous research has shown that this
process benefits both reviewers and reviewees. The reviewers
benefit by seeing others’ work and thinking metacognitively about
how they can improve their own work. The reviewees profit from
receiving comments and advice from their classmates. That
feedback is both more timely and more copious than feedback
from teaching staff [
The efficacy of peer assessment depends heavily on the quality of
the reviewing. Left to their own devices, students tend to examine
peers’ work uncritically, and make few suggestions on how to
The Peerlogic project is funded by the National Science Foundation under grants
1432347, 1431856, 1432580, 1432690, and 1431975.
improve it. When asked to rate it on a Likert scale, they gravitate
to the upper end of the scale, making little distinction between the
various artifacts that they review [
One approach to improving the quality of peer review is to
interpose a calibration phase before the actual peer-review task.
“Calibration” refers to having students evaluate sample artifacts
that have already been rated by teaching staff. Then the online
peer-review system can use the comparison between students’
reviews and those of the teaching staff to calculate review
proficiency values for students. This approach was pioneered in
Calibrated Peer Review ™ [
We can divide calibrations into two modes. The first mode
separates the calibration from actual peer-review assignments, in
which students rate and on comment each other's work. We call
this stand-alone calibration. An example is Calibrated Peer
Review™. A calibrated assignment has a separate calibration
phase in which students need to rate three sample artifacts, one of
which is exemplary, and the other two of which have known
defects. The system uses their ratings to calculate the Reviewer
Competency Index, which is a measure of the student’s review
proficiency [
The other mode of calibration combines the calibration with
ordinary peer-review activity. In the peer-review phase, students
review both sample artifacts and artifacts submitted by their peers.
Usually, they are not aware of whether the artifact is a sample for
calibration or an actual peer submission. We call this approach
mixed calibration. An example is the Coursera system [
Mixed calibration does not emphasize training — to make students better peer-reviewers — but score aggregation — how to identify the good reviewers and use their peer-review responses to aggregate grades for each artifact. Therefore, students who did poorly on the peer-review do not receive any pedagogical intervention, though their identities are known. So the mixed calibration is used more often by classes of massive sizes, e.g. some courses in the Coursera system.
Beginning in 2016, the Expertiza system has included a calibration feature, which supports both stand-alone calibration and mixed calibration. In setting up an assignment, an instructor can designate an assignment as a calibrated assignment, and submit sample artifacts and “expert” reviews. The instructor can give students the right to do reviews, but not submit work. This makes the assignment a stand-alone calibration assignment. (Ordinarily, students are permitted both to submit and to review.) The review was done in double-blind style in Expertiza. In neither calibration mode did student reviewers see the expert review before they finished reviewing an artifact. But, after a student finishes reviewing an artifact that is a calibration sample that has been reviewed by the instructor, Expertiza shows a comparison between the student’s review and the expert review (see Figure 1 for an example). No update is allowed after the expert review is displayed.
Three instructors at two universities set up a total of four calibration assignments using Expertiza. Those assignments used calibration feature in Spring 2016 but did not have calibration in Fall 2015. Other than the calibration, those four assignments were of the same settings including review rubrics.
Assignment 1: Course: Foundations and Introduction to Assessment of Education; Assignment: Grade Sample Lessons. This assignment was a precursor to engage students in evaluating peers’ writing before they assessed each other’s work. Pre-service teachers were asked to grade two different example lesson plans with a five-item rubric by ranking the (1) importance, (2) interest, (3) credibility, (4) effectiveness, and (5) writing quality of the lesson. They were asked to consider what was effective and ineffective in each lesson based on the strengths and weakness they identified from the rubric. The artifacts were lessons created by students of prior semesters whose lessons exemplified both noteworthy achievements and pitfalls. By evaluating these two lessons, students gain valuable insight into the act of evaluating peers’ writing and are provided with a model to guide their own submissions. The students’ completed the calibration assignment, ranking each of the rubric categories on a 1-5 scale. Their results were then compared with the “expert” review completed by the course instructor.
Assignment 2: Course: Project Design and Management I; Assignment: Practice Introduction to Peer Review. This assignment was designed to expose students to writing an introduction for their senior project, to orient them to the peer review process, and to understand the instructor’s expectations for the peer review assignment. The calibration exercise had the students peer review two introductions from a previous class, one with a good grade and one that received a poor grade. The calibration exercise was performed before the introduction was drafted. The general introduction assignment included a draft with an in class peer review, a second draft peer review using Expertiza and the submission of a final draft.
Assignment 3: Course: Object-Oriented Design and Development; Assignment: Calibration for reviewing Wikipedia pages. This assignment was to get the students ready to write and peer-review Wikipedia entries. The instructor provided a list of topics on recent softwaredevelopment techniques, frameworks, and products. Some of these topics had pre-existing Wikipedia pages; some did not. Where the pages existed, they were stubs or otherwise in need of improvement. Students could choose one topic and create the corresponding page. Then students were required to review at least two others’ artifacts and provide both textual feedback and ratings.
We created a separate assignment for calibration. The sample artifacts were chosen from a previous semester. The instructor took two reviews done by good reviewers and made further changes in an effort to make the review of exemplary quality.
Assignment 4: Course: Object-Oriented Design and Development; Assignment: create and review CRC (Classresponsibility-collaborator) cards. CRC cards are an approach to designing object-oriented software. The instructor’s students tended to make the same mistakes, semester after semester. The goals of this calibration assignment were to (1) allow students to submit their own CRC-card design and (2) review some CRC-card designs that contained common mistakes. In this assignment, each student reviewed one of their peers’ designs, and two designs arranged by the instructor to contain common mistakes. These designs were created by merging the errors made by previous students on an exam.
Unlike the other three calibration assignments, this assignment did not precede another assignment where the students submitted their own work. Rather, it was done as practice for the next exam. We asked the instructors to identify a few good reviewers in the actual peer-review assignments of exemplary quality to compare the student performance on the calibration assignment and the actual assignments for which they received training. To test student performance on different assignments, we used the metrics below:
Percentage of exact agreement on each criterion. All the rubrics used in our experiments were scored on either a 0to-5 or a 1-to-5 scale. On each criterion, exact agreement was when instructor and student gave exactly the same score. Percentage of adjacent agreement on each criterion. On each criterion, adjacent agreement means that the score assigned by the student is within ±1 of the instructor’s score.
Percentage of empty comment boxes. Some criteria asked students to give both a score and textual feedback. In the calibration, the instructors tried to give textual feedback on all these criteria. If the sample artifact was in good shape, the instructors commented why it was good; otherwise, if the sample artifact needed improvement, the instructors suggested changes for the author to consider. We hoped this would encourage students to comment on more of the criteria.
Average non-empty comment length. We counted the words in the non-empty responses. In calibration, the expert reviews were usually longer than the average of students’ review (see Figure 1 for example).
Average of number constructive comments. We tried to
measure how much constructive content was provided in the
non-empty responses. We used the same constructive
lexicon used by Hsiao and Naveed [
The first three calibration assignments (Assignment 1, 2 and 3) were followed by an actual assignment where the students carried out the same kind of review on which they were calibrated. We measured the percentage of empty comments, average comment length, and number of constructive comments in the response to each criterion, and the overall readability. In the following actual assignment, we also measured the students’ agreement on exemplary reviews (done by students). The results are shown in Table 1.
In all three classes, we found there was a similar amount of exact agreement on calibration assignments and following assignment. But we observed increases in the adjacent agreement on the following assignment. The reason for that could be that the calibration phase led students to become more skilled and more polite as reviewers. The instructor of assignment 1 observed that her students were critical or even bullying, in their peer reviews at the very beginning of the semester. In the calibration phase, students were able to see how the instructor reacted to various issues and what the instructor grades were. This gave students guidance on how to rate artifacts that still needed improvement. The comment length between the calibration assignment and the following assignment were almost the same. Two out of three classes had a higher average comment length after they did calibration, compared with corresponding assignments last semester.
From the amount of constructive content per response to each criterion, we found that the students tended to give as many or more constructive comments in the peer-review after the calibration. Two out of three classes made more constructive comments after calibration compared with corresponding assignments last semester.
In this study, we found that students tended to write more complicated sentences in calibration tasks, but in the assignments right after the calibration, their comments were a little easier to read but close to college level, which was acceptable to instructors.
Assignment 4 was our only experiment with the mixed calibration mode: each student reviewed two calibration submissions and one submission from their classmates. Unlike Assignments 1–3, which aimed to train students to become better reviewers on the actual peer assessment, Assignment 4 was not followed with an “actual” assignment on the same topic. Instead, Assignment 4 was designed to give students the opportunity to see common mistakes that others had made on a certain kind of question (on CRC-card design) on exams in earlier semesters.
On Assignment 4, the percentage of exact agreement was 52.2% and percentage of adjacent agreement was 91.3%, which were both very high. This was partially due to a review rubric that asked students to count the number of errors of certain types (e.g. the number of class names that are not singular nouns), instead of ordinary rubric criteria that ask students to rate the artifact on some aspect (e.g., the language usage of an article). This rubric design reduces ambiguity and thereby increased the agreements. The percentage of the empty comment was 77.0%, the average of non-empty comment length was 5.4 and average of number constructive comments was 0.13, which are all lower than Assignment 1-3. The ostensible reason was that the review rubric was not designed to encourage students to give textual comments, but simply to count the errors. The review readability index was 60.1, which indicates that for those reviewers who gave textual feedback, the feedback was not short and simple as we expected. We hypothesized that after this calibration, student's’ average score on related questions on the exam would be higher. We compared the student performance on CRC-card related questions in exams of this semester (with calibration as training) and last semester (without training). However, we found that the students’ average grade was 85.3% on those questions in this semester, and 85.4% on last semester. We did not find any significant change between this semester and last semester. Upon seeing those results, we surmised this calibration assignment was done several weeks before the next exam, and, without follow-up practice, students forgot the training they received.
After students finish the calibration, the instructor can see the calibration reports for each artifact, as shown in Figure 2. Each table shows the students’ grades on each question on a sample artifact. The green color highlights the expert grade, and the bolded number was the plurality of students’ grades.
Figure 2 shows a sample artifact where the calibration was quite successful, with exact agreement of more than 40% and adjacent agreement of almost 80%. However, it is still not clear that if it was related to the quality of the artifact. When we calculate the percentages of agreements for each sample artifacts, we found that the level of agreement is related to the quality of the artifact: the higher grade that a sample had, the higher agreement that students might achieve. This raises another question: what kind of artifacts work better as samples in calibration? We put the percentages of agreement and grades for the artifacts together to compare the relationship between the agreement and the grades that the sample artifacts received. We used both the sample artifacts and the artifacts reviewed by the exemplary reviewers. The distribution and fit line are shown below. We find that the samples that received higher grades usually have higher levels of agreement (on both exact agreement and adjacent agreement). The lower quality a sample is, the lower agreement we observed between teaching staff and students.
We looked into the samples used in each assignment, and we found that usually it is harder for students to make the same judgment as teaching staff on an artifact of low quality. There could be multiple reasons. The first reason is that teaching staff has seen more artifacts, therefore they know the distribution of the quality of the artifacts and thereby they made better judgments. For student reviewers, they may be able to tell an artifact is of low quality based on one criterion, but they could be more critical than warranted since they have not seen even worse examples. From this perspective, it is important for instructors to use at least one or two low-quality sample artifact as a sample artifact to show students how to rate poor work.
Another factor that may lower the agreement between teaching
staff and students is the reliability of the criterion: some of the
criteria are not specific enough for the reviewers to make reliable
judgments [
In this paper, we have described our experience with the calibration in peer assessment in Expertiza. We first introduced two modes of calibration that have been used in online peer assessment systems, which are stand-alone calibration and mixed calibration. Stand-alone calibration trains students to become better reviewers, while mixed calibration finds credible reviewers in the course of performing peer assessment. We also discussed the pedagogical scenario in which each mode is suitable. We calculated the agreement between students’ rating and teaching staff’s rating on the sample artifacts. We found that students in our assignments, on average agreed exactly with teaching staff on more than 40% of ratings. This means that on more than 40% of the ratings done by students during calibration gave exactly the same scores given by teaching staff. In addition, more than 70% of the ratings done by students gave the score within the ±1 range to the scores given by teaching staff. To test if students still perform as well on the actual peer assessment after training, we asked the teaching staff to identify some good reviewers in each course. Using their reviews as exemplars, we found that, in the actual peer assessment phases, the agreement was similar to that on the calibration assignments, sometime even a little higher.
We compared the volume of textual feedback from the semester with calibration and the previous semester without calibration. We found that after calibration, students tend to give more extensive textual feedback, fill in more text boxes with comments, and give more constructive feedback.
We also found that the level of rating agreement between students and teaching staff is related to the quality of the artifact; namely students tended to agree less with teaching staff on artifacts of low quality. To improve agreement, we suggested: (1) on the calibration, an instructor can use both median-quality artifacts and low-quality artifacts as samples and (2) the instructor can provide “advice” for each level of each criterion.
One future study we are interested in is to calibrate the textual feedback. In this paper, we have only calibrated the numerical scores. It is possible that both a student and the teaching staff gave a ⅘ on one criterion on a sample artifact, but may not see the same issue. This kind of agreement can only be measured by calibration of textual feedback.