Peer assessment continues to be a topic of interest in the educational arena for decades, especially with the popularity of MOOC in recent years. However, the reviewer assignment moment, which may affect the process efficiency and learning proactiveness reward, lacks appropriate consideration. In this study, we propose three reviewer assignment algorithms, postassignment, pre-assignment, and submission-queue, integrated with the educational peer code review system developed for the purpose of programming language learning. We compare and analyze the algorithms performance qualitatively according to processing efficiency and proactiveness reward to learners. In order to measure the three algorithms quantitatively, we carry out one preliminary investigation involving students in a programming course at a Chinese university. The results indicate that submission-queue has predominant advantages in both proactiveness reward and process efficiency relative to the other two. Moreover, data reveals that active students are more sensitive to reviewer assignment moment algorithms than the inactive ones. peer assessment; peer code review (PCR); reviewer assignment moment (RAM); submission queue; proactiveness reward; processing efficiency
Peer assessment has become more relevant and has gathered
increasingly more attention in education, as it fits the self-directed
and collaborative learning processes [1]. Newman and Taylor
defined the reviewer assignment time as the time taken to assign a
reviewer to a paper and to record the assignment by conference
organizers [2]. However, the reviewer assignment moment (RAM),
which may affect the process efficiency and proactiveness reward
to learners, has not been concerned yet. In previous researches on
academic peer review, only a few studies have gone beyond
performing an optimized assignment of manuscripts (or proposals)
to reviewers in consideration of the quality assurance. In the
studies of peer code review (PCR), scholars are mainly focusing
on its quality assurance, learning outcomes, or sharing of
successful instruction experiences. However, in our experience,
we find that choosing an appropriate RAM does not only improve
the efficiency of peer review process, but also stimulates the
students' learning. Thus, in this study, setting a PCR software as
the context, we concentrate our attention in answering two
important questions: (
During the last few years, we have analyzed the output data from EduPCR to study the students' learning behavior. The system has inspired us to explore the students' e-learning data in different areas: their competence in coding, learning attitude, compliance with coding standards and the capability of following schedules [3]. Along with the previous studies, the RAM addresses the main concern of this study. In order to improve both students' activeness and efficiency in learning a programming language, we have tried a series of reviewer assignment algorithms relevant to RAM. This study seeks to find scientific evidence in different reviewer assignment algorithms, from perspectives such as process efficiency and proactiveness reward to learners. The paper outlines as follows. Three RAM algorithms are proposed in section 2. Section 3 analyses their process efficiency and proactiveness reward qualitatively. These algorithms are studied comparatively in section 4. Discussions are made at last.
So far, we have applied three practical reviewer assignment algorithms in the EduPCR system. Since every student plays the roles of reviewer and author, the final reviewer assignment result will build up one or several circles, namely, review rings.
Post-assignment is a matured algorithm defined in peer review literature many years ago [4], for assigning resources or tasks after a specific phase. A networked peer assessment system for secondary science education utilized it [5]. In this study, we define post-assignment as "assigning reviewers after all students have submitted their source code".
We develop our own post-assignment strategy in EduPCR. With it, the assignment action is completed when all participants have submitted their source code or the stage deadline is reached. Postassignment is the easiest approach of reviewer assignment for both, teachers' implementing and students' understanding. However, the disadvantage of post-assignment is that all the students who have submitted the source code prior to the deadline have to wait for the ones behind; if a single student fails to finish his/her source code, the rest of the students cannot start their review. Undoubtedly, post-assignment has a low efficiency rate. It is important to point out that there is no need to consider exceptional situations since all the students who are assigned as reviewers have finished and submitted their source code.
When the low efficiency of post-assignment was realized, preassignment became an alternative assignment method. It is a common algorithm widely applied to solve schedule problems; for instance, Tànfani & Testi proposed a pre-assignment algorithm to solve the Master Surgical Schedule Problem [6]. Pre-assignment is also used in educational peer review to achieve a certain goal in specific research [7].
Following these studies, we created our own pre-assignment algorithm in EduPCR. We define this algorithm as "reviewer assignment is finished the moment a programming task is released by teacher". Since every student can begin the review work as soon as the source code is submitted, this algorithm seems to be more efficient than post-assignment.
However, the pre-assignment algorithm is not without flaws. Due to the random review assignment in advance, eventually a reviewer has to experiment long waiting times for the manuscript of the assigned author. Additionally, one situation that often occurs is when one or more students miss the deadline of submitting their source code. For an instance in 1-to-1 assignment, see (a) of Figure 1, the student #2 who fails to submit source code on time will affect his/her two peers: the assigned reviewer #1 has no code to review and the author #3 has no reviewer.
In order to remedy this, the ideas of compact algorithm (see Figure 1) and merge algorithm (see Figure 2) are utilized so that the computer compacts the review ring by eliminating the students who do not submit their source code. When there is only one student left in a review ring, that student is labeled as a "lost" one, and will be merged to another review ring. Note that the merging position will be carefully chosen to make sure that the previous reviewer has not yet started his/her review work at the merging moment. In Figure 2, before merging, student #4 is the reviewer of student #5. If student #4 has not started the review work when the computer tries to merge, the merging can address this position, i.e. inserting student #3 between student #4 and student #5. 2
2 (a) student #2 misses (b) student #2 is removed
the first deadline from the ring Figure 1. Illustration of the compact algorithm (a) only student #3 (b) another (c) student #3 is is remained after reviewer merged into that
compacting assignment ring ring
Figure 2. Illustration of the merge algorithm When the system cannot find a position to merge a "lost" student into because all assigned reviewers have started (or finished) their review work, the system will collect all the "lost" students in all of the review rings and build up a new one. If there is only one "lost" student incapable of being merged, the system will notify with the message "You can revise your program upon your own idea since system fails to assign a reviewer for your work".
Making use of the well-known scheduling strategy "first come, first served" in service policy or queuing law [8, 9], an "first submit, first assigned" algorithm (namely, submission queue), was implemented to overcome the problems brought by the postassignment and pre-assignment algorithms. With the timer function, the computer checks who has submitted the source code at a predefined time interval (i.e. every 30 minutes). When the time is reached, the computer scans the number of students who have submitted their source code in the previous interval (let this variable be X), and the number of students who have not submitted their source code (let this variable be Y). Then there are three possible cases, called assignment conditions, as follows: Case 1: X<3. If the source code deadline has not been reached, no action is needed. Otherwise, if X=2, the two students build up the smallest possible review ring (they are assigned as reciprocal reviewers); and if X=1, the merge algorithm is invoked (see Figure 2).
Case 2: X≥3 and Y≥2. The reviewer assignment will be performed among those X students.
Case 3: X≥3 and Y=1. In order to give a chance to the last student to submit the source code (we call him/her student L), we will let the penultimate student (we call him/her student W) who has submitted the source code, wait for L. In this case, our reviewer assignment will be performed among those X students, except for W. Even if X=3, the two students except for W can build up the smallest possible review ring. Finally, student W will not be assigned until student L submits his/her source code. It is possible for only one student being unassigned by the first deadline. In Case 2, only one among all Y students submits his/her work before the deadline which will make the single student dangling. In Case 3, L has not submitted the source code when the 1st deadline is reached, student W will be dangling. To deal with dangling students, the merge algorithm (Figure 2), would be necessary. Theoretically, there is an additional scenario, in which all students in the last-built review ring have started their review work when the computer tries to merge. Despite having a very low probability, the system will notify the dangling student revise the source code according to his/her own understanding. Certainly, we can improve this algorithm by searching for an appropriate position in all review rings in the future.
Generally, most students can finish the 3 stages on time, but exceptional cases may occur occasionally, not only in source code stage, but in the reviewing and revising stage. The possible exceptional cases and their solutions are listed in Table 1.
From a managerial viewpoint, different algorithms may have different performances. The following section provides a capability analysis of the three RAM algorithms from two different perspectives: processing efficiency and proactiveness reward. 1 5 2 1 3 3 4 3 4 7 5 6 3 4 4 7 1 5 5 6
The author who fails to submit source code is considered to quit the current program task. The assigned reviewer gets review marks automatically.
Compact a review ring by eliminating the students who fail to submit the source code. When the number of students in one review ring minimizes to one, the single student will be merged into another review ring.
Only one student is still waiting to be assigned when the first deadline is reached. The student is merged into the last-built review ring.
When the review deadline is reached, computer prompts the author to revise the program by himself/herself.
The corresponding marks of the author will be deduced.
Whether a reviewer is active or passive, the possible starting point
to review is critical because it bottlenecks the entire process. The
assignment algorithm plays an important role in facilitating an
active reviewer to start a review job as early as possible. The
necessary conditions for starting a review work are:
(
The starting time available for review is of great significance; among the four conditions, the 3rd might bottleneck the entire review process, hence it becomes fundamental to optimize the assignment moment.
It is easy to understand that the students can start the reviewing and revising process much earlier in pre-assignment and submission-queue than in post-assignment, which implies that the first two may be more efficient. Considering the individual activity, some students may submit their source code very quickly while others might submit their source code just before the first deadline. In pre-assignment, if the fastest and slowest students are assigned to a group to review each other, the fastest student will waste time waiting for the slowest student's source code. This issue is resolved in submission-queue, allotting for faster students to be assigned to fellow fast student.
Therefore, from the processing efficiency standpoint, we predict that submission-queue is the most effective and post-assignment is the least effective.
In post-assignment, all students have to wait until the last student
submits his/her source code or the deadline, and then the
assignment begins. Therefore, many students may choose to
submit their source code until the deadline, since an early
submission cannot trigger an early reviewer assignment.
In pre-assignment, the reviewer assignment is finished when the
project (programming task) is released. A reviewer can start the
review process only after his/her peer student (previously assigned)
has submitted the source code. Students may submit their source
code at their convenience, since an early or late submission will
not accelerate or slow down the entire process much. If a student
demands his/her source code to be reviewed, he/she must have
submitted his/her own source code. On the other hand, if a student
wants to review his/her peer's code, his/her peer must have
submitted his/her own source code as well. The time interval
between submission and review may be large but since there is no
difference between submitting it early or late, the majority of
students might submit their own source code as late as their peers
do; which in turn becomes a submission close to the deadline.
However, in submission-queue we find two obvious advantages
related to proactiveness rewards, relative to the other two
assignment methods:
(
Hence, in terms of proactiveness reward, we predict that submission-queue may have predominant positive influence on students' leaning than the other two.
The EduPCR system has been applied in the pedagogy of two
courses: C Programming and Object Oriented Programming in
Java for students majoring in Information Systems at a Chinese
university. To understand the three assignment algorithms’ effect,
we conducted a comparative investigation on the course Object
Oriented Programming in Java in one 23-student class. Because
of lack of experience, we stored the review submission time in
12hour (not 24-hour) format, losing some information and accuracy;
the results might be affected to some extent.
4.1 Investigation Design
(
Since the time when students submit their source code is not greatly affected by the reviewer assignment algorithm and the revision submission time is indirectly affected by the assignment algorithm, we just focus on the time when the students submit their reviews. As previously mentioned, post-assignment algorithm starts the review process after the source code deadline, while pre-assignment and submission-queue algorithms can start the review process before the source code deadline. To compare the review efficiency of the three assignment algorithms, we take the difference between the review submission time of a reviewer and the source code deadline as the central statistic measure. This measure is named wait time (WT) of the reviewer; different reviewers have different WT values. Obviously, the smaller the WT value is, the more active the reviewer is. Then with preassignment and submission-queue algorithms, some reviewers' WT value is negative, which means that the reviewers finish their second stage task (i.e., review work) before the deadline of the first stage. During the implementation of EduPCR for years, we find that active students often finish their own source code and begin review work as soon as they get the "new review" notification, while some tardy students will not start each step until the deadline is approaching. Thus, the value of average wait time (AWT) can be used to measure the activeness of students in each assignment algorithm. AWT is the average value of several WTs, which has two cases: 1) individual AWT means the average value of one student’s WTs within all tasks, which is used for clustering students into groups by their WTs; 2) group AWT is the average value of a group of students' WTs within all tasks. Each group has three group AWTs because three RAM algorithms are deployed.
To eliminate the effect of different personalities, we cluster all students according to individual AWT within all tasks. Usually clusters generated from the clustering algorithm are not comparable. However, in this study, students are clustered by a single variable: individual AWT. Each cluster has a group AWT that is comparable with another group AWT of a different cluster so that we can order the generated clusters.
With K-Means in IBM SPSS Modeler, we cluster all students into four clusters according to individual AWT in ascending order, which means that the students in the first cluster spend the least amount of time finishing the review, while the ones in the fourth cluster spend the largest amount of time to do so. Moreover, we classify all the students as active and inactive; the students in the first two clusters of both investigations are considered active students, while those in the last two clusters are considered inactive. Finally, the clustering result is obtained, i.e. 5, 5, 7, 6 students in cluster 1 through 4 accordingly. Within each cluster, we compute the AWT values according to the algorithm group. The AWT of the four clusters in the three groups is shown in Figure 3. For each algorithm, the group AWT is increasing from cluster 1 to 4, meaning the activeness of students is decreasing correspondingly (see Figure 3).
Even though the database design defect might affect the accuracy
of this study, the results are as follows:
(
With the increasing requirement of peer assessment research, especially in the age of MOOC, the reviewer assignment becomes a pressing concern. The reviewer assignment research in the educational context, especially on the reviewer assignment moment, gets little focus.
From the perspective of RAM in the PCR context, we propose
three algorithms comprising post-assignment, pre-assignment,
and submission-queue. The preliminary investigation among one
classes in an academic years reveal that submission-queue has a
much higher processing efficiency and proactiveness reward than
the other two algorithms. Moreover, we found that active students
are much more sensitive to the application of reviewer assignment
algorithms. Based on these results we discuss two main concerns:
(
This work was partially funded by China Scholarship Council [201506125055], National Natural Science Foundation of China [71573065], and Online Education Research Foundation (QTone Education) of China's MOE [2016YB130].
Thanks to some students at School of Management, Harbin Institute of Technology, such as Miss Hang Li and Miss Xiaolei Wang, for their assistance of collecting data and analyzing the results.