In educational peer-review activities, one challenge is to tell which peer reviews are credible. Reputation systems are one approach. However, they work best in a scenario where (i) most of the peer reviewers can do a decent job, or (ii) reviewers tend to do peer-reviews in fixed styles─e.g., negative reviewers are negative on all artifacts they review; accurate reviewers grade all work accurately. We argue that, those hypotheses may not hold for many cases for educational peer review. For this reason, we invited student authors in one class to create quiz questions based on their own artifacts. After their reviewers finished peer-review responses, they could take the corresponding quizzes. The mere existence of those quizzes may encourage reviewers to take the peer-review activity more seriously. In addition, their quiz scores can be used as another source of reputation. Since those quizzes are generated by authors to “test” the reviewers, the quiz scores give teaching staff another input on deciding whether a peer review is credible. Our experiments show that a reputation generated from quiz scores provided more accurate estimates of final grades for students' artifacts than some existing reputation systems.
Educational peer assessment has proven to be a useful approach
that can provide students timely feedback and opportunities to
help and learn from each other. Reviewers are often expected to
provide both formative feedback─textual feedback telling the
authors where and how to improve the artifacts─and summative
review─peer grading telling authors how good their artifacts
are─at the same time. Formative feedback is important for the
authors when the assignments are still ongoing because timely and
insightful feedback can help authors improve their artifacts. In a
large class or MOOC, when the teaching staff is stretched thin in
providing help, formative feedback from peers is the best help that
the course participants may receive. Summative feedback can also
aid the teaching staff, as it provides more input to help determine
final grades. In a large class or MOOC, the teaching staff is
almost forced to base final grades on the summative reviews that
students have received [
After peer reviews have been completed, they can simply be
shown to the student authors. However, they may not tell the
authors much because there is no guarantee that the peer reviews
are insightful or helpful. A peer-review done without dedication
or conscientiousness may not have much impact on the author’s
learning process [
Are the scores assigned by the reviewer close to the scores assigned by the instructor, on work that they both review? Are the scores assigned by the reviewer close to the scores assigned by the other reviewers, on work they both review? Is the reviewer habitually stingy, or generous, to different kinds of work?
How competent has the reviewer been on other work in class? Any or all of the above information may be factored into a reviewer's reputation. These reputation scores can help teaching staff come up with the final grades for each artifact. If students are allowed to see their reputation scores, they can also learn how effective their reviewing has been.
There is a common assumption in some existing reputation
systems that a reviewer's reputation on one task carries over to
other review tasks [
Though unrelated to peer review, student-generated questions
have long been considered a helpful process which triggers
critical thinking [
In this section we provide an overview of our experimental design, class policies for our later experiments and our approach to checking and validating the data set we collected.
Our data is collected from students using Expertiza, a web-based
system developed to facilitate peer-review activities on
studentgenerated course content [
The assignment on which we applied our experiments was a wikiwriting assignment over two semesters in a graduate-level course in the College of Engineering in NC State University. Students were required to write or edit a Wikipedia page on a recent technique, product, programming framework, etc., related to object-oriented design or development.1 Students were allowed to work in teams with maximum of two members per team. The peer-review task for this assignment asked about a variety of aspects which related to the quality of the artifacts, including originality, structure, use of language and examples, etc. Most of the criteria in the review rubric allowed reviewers to give both textual feedback (as formative feedback) and Likert-scale scores (as summative feedback). According to the syllabus, each student was required to do only two peer reviews, but could do extra reviews for extra credit. The assignment consisted of two rounds of submission and review. After the first round, authors could change their artifacts based on the reviews they received. After the second review phase, the reviewers could take the quizzes on the artifacts they had reviewed.
Almost 200 students completed those assignments. We collected 74 sets of quiz questions. On average, each artifact was reviewed 9.1 times, and each quiz was taken 4.8 times. Quiz-authoring and quiz-taking was not required in the syllabus but students were given extra credit for them.2 Quiz takers needed to score 80% or higher to pass the quiz. The overall passing rate was 81.6%.
A member of the teaching staff reviewed the student-generated quizzes to make sure that they were well formed (e.g., each choice of a multiple-choice question should have some content) before the quizzes were taken by the reviewers. We found three cases of incomplete questions and asked the authors to change them. After this assignment was finished, we did further checks on the quality of student-generated questions. Figure 1 shows the distribution of passing rates of student-generated quizzes against the average quiz scores from quiz takers. Most of the quizzes had a passing rate of almost 80% (x-axis of Figure 1). The average quiz score on each set of the quiz was also high; most of them were higher than 80% (y-axis of Figure 1).
We suspected that the quizzes which had both a high average score and high pass rate might be too easy, and the ones which had both low average score and low passing rate might be poorly designed. Therefore, we examined all the quizzes from the upperright corner and lower-left corner of the graph within a [10%, 10] range. We also examined some of the quizzes not in those 2 ranges. Among 14 samples, we found that two of them were so easy that the quiz takers could answer most of the questions correctly without fully understanding the artifacts. The quizzes in the lower left-hand corner are harder to answer because the authors created more checkbox questions. We did not identify any question that was invalid, irrelevant or given the wrong answer. 1 The Wikipedia did not have that page, or only had very limited content when students started to add the content. The writing was done in the Wikipedia sandbox. 2 We provided a online instruction document to students: goo.gl/7Eud42
Inputs of reputation systems vary, but a common way to represent peer-review scores is to use an adjacency matrix. In this matrix, each row stands for an artifact and each column represents a reviewer. The values in the matrix are the scores given by each reviewer to each artifact (if the reviewer reviewed that artifact). Reputation systems use the matrix to generate two quantities: the reputation scores for each reviewer and the weighted grades for each artifact. Since we used scores of student-generated quizzes as reputations, we have another adjacency matrix for quiz scores. In this matrix, again, each row stands for an artifact and each column stands for a reviewer. The values in the matrix are the quiz scores obtained by each reviewer on each quiz from the artifact author. Please note that a reviewer could only take a quiz if that reviewer had reviewed the artifact created by the same author.
The reputation can be calculated by comparing the peer-review
scores and instructor-assigned grades. Since in practice, the
reputation systems runs before the grading is done by teaching
staff, we made use of two existing reputation algorithms which do
not use instructor-assigned grades [
To give precise definitions of the algorithms in this paper, let a be an artifact; A be the set of all the artifacts; r be a reviewer; R be the set of all the reviewers; gar be the grade that r assigned to a; qar be the quiz score that r earned from the quiz associated with a; Rr be the set of artifacts reviewed by r; Aa be the set of reviewers who have reviewed a;Wr be the weight for reviewer r (the weight could be temporary depending on which algorithm is used); Gaaggregated be the grade that the reputation algorithm aggregated for artifact a based on current reputations; and Ga be the temporal grade for artifact a in the algorithm.
We assume that the quiz score qar can be used as reputation: qar Wr qar NULL qar = NULL (1) in which is an small constant for the reviewers who did not take the quiz. The idea behind this is that, those reviewers who did not take the quiz still did the peer reviews, so they should have a reputation, though modest.
Gaaggregated is the grade aggregated from the weights of the reviewers who have reviewed a:
rAa
rAa Wr (2) We compare our algorithm with Hamer’s algorithm and Lauw’s algorithm, which are both reputation algorithms based on only peer-review scores (without quiz scores).
Both Hamer's [
In Hamer’s algorithm the grading variance of r is calculated from the difference between the reviewer's grading and the aggregated grades from last round: r (Ga gar )2 aRr
Rr (3) After calculating the variances from all the reviewers, the weight is defined as: Wr ' r r (4) So the higher Wr ' is for r R , the more “reliable” reviewer r is. The range for Wr ' is (0, ) .
Since some of the reviewers may have very high weights and thereby "dominate" the aggregated grades, Hamer proposed a “log-damping” process:
2 log(Wr '1) Wr Wr '
Wr ' 2
Wr ' 2 (5) Lauw’s algorithm is similar to Hamer’s algorithm. The difference is that Lauw’s algorithm keeps track of the leniency for each reviewer. The leniency can either be positive or negative, depending on whether the reviewer tends to grade artifacts higher or lower compared with aggregated grades.
Given lr for all r R , Gaaggregated can be calculated by:
a ( [
a
| Rr | (6) gar (1 lr ) | Aa | (7) Wr 1 lr (8)
We used students’ quiz scores as reputations to aggregate the grades for all the artifacts and compare them with the final grades assigned by teaching staff (“expert” grades). Similar to our approach, Hamer’s algorithm and Lauw’s algorithm are also used to calculate the reputation-aggregated grades. The comparison results are shown in Table 1 (please note that all the grades are 100 based in this paper). The bias range is the range of difference between aggregated grades and expert grades. The quiz-based reputation approach has the smallest bias range, which means that even in the extreme case, the aggregated grades by quiz-based reputation algorithm may make smaller mistakes than other reputation algorithms. The rootmean-square error on quiz-based reputation approach was 4.4 and average of absolute bias is 3.4, which means that if teaching staff applies a similar approach and do not grade all by themselves, the average absolute bias between aggregated grades and expert grades is 3.4, which is still acceptable in most cases.
Not all the peer-reviewers took the quiz since it is not a required activity. We next tested if our quiz-based reputation approach works for the artifacts with smaller number of quiz takers. We divided the artifacts into two groups with roughly the same sizes: Group 1 is formed by the artifacts with higher numbers of quiz takers and Group 2 is formed by the artifacts with lower numbers of quiz takers. We did the comparison again, and the results are shown in Table 2.
From Table 2 we find that, for the artifacts in Group 2 which all have less than 4 quiz takers, the aggregated grades are still quite close on average to the expert grades. There is also no significant difference on the biases between Group 1 and Group 2 (the pvalue between the biases from Group 1 and Group 2 from the quiz-based algorithm is 0.81). This indicates that, even in the extreme cases that some artifacts only have a few quiz takers, our approach still works. However, comparing the biases from Group 1 and Group 2, quiz-based algorithm works better when there are more quiz takers.
Another perspective we considered was whether our approach still works on the artifacts that received limited numbers of peer reviews. For the reputations calculated from reputation systems, the reputation of a reviewer was not based on whether his/her peer-review grades agree with others' on one artifact, but on all the artifacts that (s)he has reviewed. Therefore, even if an artifact is reviewed by a limited number of reviewers, the reputations and 3.7 2.7 5.0 4.0 4.4 3.3 From Table 3 we find that, all three approaches work better on Group 3 which had more reviewers, and our approach performed the best. On Group 4, our approach performed slightly worse than Lauw's algorithm, but the difference was small (0.3 on average bias and 0.1 on RMSE). We also found that the results are similar to Table 2 because in our experiment the quiz takers have to be reviewers of an artifact first. Therefore, the artifacts from Group 1 and Group 3 are mostly the same, similarly, a large portion of the artifacts in Group 2 and Group 4 are also the same. aggregated grades should still be reliable since the reviewers may also review other artifacts. However, if the reputations are the reviewers' quiz scores on the artifacts, will smaller numbers of reviewers lead to lower validities of aggregated grades using our approach? We again divided the artifacts into Group 3 and Group 4 which almost had the same size. Group 3 contains the artifacts which have higher numbers of reviewers; Group 4 contains the artifacts with fewer reviewers. We did the comparison again and the results are shown in Table 3.
From the results above, we found that using quiz scores as reputations worked as well as, or sometimes even better than some existing reputation systems.
To look more closely into the reason for this, we used one artifact from our data set as an example. This artifact received 4 peer reviews, and all those 4 reviewers took the quiz created by the author.
Figure 2 shows the bubble graph of the peer-review scores and the reputations calculated by Hamer’s algorithm. The x-axis is for different reviewers; the y-axis shows the peer-review scores; the size of bubbles demonstrates the reputation of each reviewer from Hamer’s algorithm. The bubbles are bigger for the reviewers with higher reputations. We found that Reviewer 1 earned a very high reputation score (2.6) while the rest of the reviewers had low reputations (0.5, 0.1 and 0.1). Considering that the range of reputations from Hamer’s algorithm is (0,) , and a reviewer with average review skill should receive a 1 for reputation, Reviewer 2, Reviewer 3 and Reviewer 4 are considered to be very poor in peer-reviewing. Since Reviewer 1 gave this artifact a 100, the aggregated grade for this artifact from Hamer’s algorithm was also very close to 100, which is much higher than the expert grade (by 11.4 points).
Figure 3 shows another bubble graph of the peer-review scores
and reputations calculated by Lauw’s algorithm on the same
artifact. Lauw’s algorithm usually assigns reputations more
leniently than Hamer’s algorithm. The x-axis is for different
reviewers; the y-axis shows the peer-review scores; the size of
bubbles demonstrates the reputation of each reviewer from
Lauw’s algorithm. We found that Reviewer 1 and Reviewer 2
both received higher reputation scores (0.93 and 0.96) from
Lauw’s algorithm and Reviewer 3 and Reviewer 4 received low
reputations (0.51 and 0.65) (the reputation range of Lauw’s
algorithm is [
Comparing Figure 2 with Figure 3, there is a disagreement about
Reviewer 2’s reputation between Hamer’s algorithm and Lauw’s
algorithm. We sometimes observe this kind of disagreement, but it
does not result in a large performance difference between the two
reputation algorithms [
The lesson of this paper is that reputation systems for peer review, and, by extension, the reliability of peer reviewing and peer grading, can be enhanced by having student authors write quizzes on the artifacts that they have submitted, and using reviewers’ scores on these quizzes as evidence for the validity of their reviews.
In our experimental design, after authors submit their artifacts, they can create a set of quiz questions for their peer reviewers. The reputations of peer reviewers are based on how well they did on quizzes written by the authors of the artifacts they have reviewed. The philosophy behind this is, the reviewer's formative feedback (suggestions) and summative feedback (peer-grading) are credible if they are capable of passing the quiz created by the authors.
We compared our approach with two existing reputation algorithms. We found that using the quiz scores as reputations can help teaching staff to aggregate final grades and this approach provide similar or even better estimates of expert grades than the reputation algorithms. In section 4.2 we used examples to explain the reason for this finding.
We found that using quiz scores as reputations has several advantages. First, this approach is not based on “holistic reputation” hypothesis since in the assignment that we used for experiment students worked on different topics separately. We argue that, if a student peer reviewer is qualified to evaluate some artifacts, then this reviewer also has higher chances of passing the quizzes created by the authors. On the contrary, for the topics that this reviewer is not competent to review, the reviewer has a lower chance of passing the corresponding quizzes. If so, the peer reviews may not be qualified to help the authors to improve the artifacts, nor to the teaching staff aggregating the final grades. Another advantage for using student-generated quizzes in the peer-review task is that we do not need to display the reputation scores calculated from reputation algorithms to students. We have been hesitating to show those reputation scores because they come from “black boxes” from the students’ perspective. But students have no trouble understanding their own quiz scores.
Despite our success, there are still more perspectives we can
investigate. Our work suggests that the “holistic reputation”
hypothesis (which assumes that the credibility of a reviewer is
stable) of some existing reputation algorithms may not hold for an
assignment where students are working on different topics.
However, it remains unclear whether this hypothesis holds for
assignments where students are working on the same topic or very
similar topics. Some other recent findings of our team also show
that the reliability of peer grading is related to the design of
peerreview rubrics [
The Peerlogic project is funded by the National Science Foundation under grants 1432347, 1431856, 1432580, 1432690, and 1431975.