Choosing a Better Moment to Assign Reviewers in Peer Assessment: The Earlier the Better, or the Later the Better? Yanqing Wang Haoran Wang Christian Schunn, Emily Baehr School of Management Northeast Yucai Foreign Language Learning Research Development Center Harbin Institute of Technology School University of Pittsburgh Harbin, Heilongjiang 150001, China Shenyang, Liaoning 110179, China Pittsburgh, PA 15260, USA yanqing@hit.edu.cn 728999801@qq.com {schunn, ecb42}@pitt.edu ABSTRACT courses: C Programming and Object Oriented Programming in Peer assessment continues to be a topic of interest in the Java. An e-learning information system dedicated to PCR in educational arena for decades, especially with the popularity of programming language learning, EduPCR, was developed and MOOC in recent years. However, the reviewer assignment implemented by our team from 2007 for programming language moment, which may affect the process efficiency and learning learning and for e-learning research. proactiveness reward, lacks appropriate consideration. In this During the last few years, we have analyzed the output data from study, we propose three reviewer assignment algorithms, post- EduPCR to study the students' learning behavior. The system has assignment, pre-assignment, and submission-queue, integrated inspired us to explore the students' e-learning data in different with the educational peer code review system developed for the areas: their competence in coding, learning attitude, compliance purpose of programming language learning. We compare and with coding standards and the capability of following schedules analyze the algorithms performance qualitatively according to [3]. Along with the previous studies, the RAM addresses the main processing efficiency and proactiveness reward to learners. In concern of this study. In order to improve both students' order to measure the three algorithms quantitatively, we carry out activeness and efficiency in learning a programming language, we one preliminary investigation involving students in a have tried a series of reviewer assignment algorithms relevant to programming course at a Chinese university. The results indicate RAM. This study seeks to find scientific evidence in different that submission-queue has predominant advantages in both reviewer assignment algorithms, from perspectives such as proactiveness reward and process efficiency relative to the other process efficiency and proactiveness reward to learners. two. Moreover, data reveals that active students are more sensitive to reviewer assignment moment algorithms than the inactive ones. The paper outlines as follows. Three RAM algorithms are proposed in section 2. Section 3 analyses their process efficiency Keywords and proactiveness reward qualitatively. These algorithms are studied comparatively in section 4. Discussions are made at last. peer assessment; peer code review (PCR); reviewer assignment moment (RAM); submission queue; proactiveness reward; processing efficiency 2. THREE RAM ALGORITHMS So far, we have applied three practical reviewer assignment 1. INTRODUCTION algorithms in the EduPCR system. Since every student plays the roles of reviewer and author, the final reviewer assignment result Peer assessment has become more relevant and has gathered will build up one or several circles, namely, review rings. increasingly more attention in education, as it fits the self-directed and collaborative learning processes [1]. Newman and Taylor 2.1 Post-assignment defined the reviewer assignment time as the time taken to assign a Post-assignment is a matured algorithm defined in peer review reviewer to a paper and to record the assignment by conference literature many years ago [4], for assigning resources or tasks after organizers [2]. However, the reviewer assignment moment (RAM), a specific phase. A networked peer assessment system for which may affect the process efficiency and proactiveness reward secondary science education utilized it [5]. In this study, we to learners, has not been concerned yet. In previous researches on define post-assignment as "assigning reviewers after all students academic peer review, only a few studies have gone beyond have submitted their source code". performing an optimized assignment of manuscripts (or proposals) to reviewers in consideration of the quality assurance. In the We develop our own post-assignment strategy in EduPCR. With studies of peer code review (PCR), scholars are mainly focusing it, the assignment action is completed when all participants have on its quality assurance, learning outcomes, or sharing of submitted their source code or the stage deadline is reached. Post- successful instruction experiences. However, in our experience, assignment is the easiest approach of reviewer assignment for we find that choosing an appropriate RAM does not only improve both, teachers' implementing and students' understanding. the efficiency of peer review process, but also stimulates the However, the disadvantage of post-assignment is that all the students' learning. Thus, in this study, setting a PCR software as students who have submitted the source code prior to the deadline the context, we concentrate our attention in answering two have to wait for the ones behind; if a single student fails to finish important questions: (1) what is the most appropriate moment to his/her source code, the rest of the students cannot start their assign reviewer? and (2) what benefits can we obtain from review. Undoubtedly, post-assignment has a low efficiency rate. It applying the most appropriate reviewer assignment algorithm? is important to point out that there is no need to consider exceptional situations since all the students who are assigned as Since 2004, our team has been researching on PCR and we have reviewers have finished and submitted their source code. been applying techniques for this end in two formal university 2.2 Pre-assignment student incapable of being merged, the system will notify with the When the low efficiency of post-assignment was realized, pre- message "You can revise your program upon your own idea since assignment became an alternative assignment method. It is a system fails to assign a reviewer for your work". common algorithm widely applied to solve schedule problems; for instance, Tànfani & Testi proposed a pre-assignment algorithm to 2.3 Submission Queue Making use of the well-known scheduling strategy "first come, solve the Master Surgical Schedule Problem [6]. Pre-assignment first served" in service policy or queuing law [8, 9], an "first is also used in educational peer review to achieve a certain goal in submit, first assigned" algorithm (namely, submission queue), was specific research [7]. implemented to overcome the problems brought by the post- Following these studies, we created our own pre-assignment assignment and pre-assignment algorithms. With the timer algorithm in EduPCR. We define this algorithm as "reviewer function, the computer checks who has submitted the source code assignment is finished the moment a programming task is released at a predefined time interval (i.e. every 30 minutes). When the by teacher". Since every student can begin the review work as time is reached, the computer scans the number of students who soon as the source code is submitted, this algorithm seems to be have submitted their source code in the previous interval (let this more efficient than post-assignment. variable be X), and the number of students who have not submitted their source code (let this variable be Y). Then there are However, the pre-assignment algorithm is not without flaws. Due three possible cases, called assignment conditions, as follows: to the random review assignment in advance, eventually a reviewer has to experiment long waiting times for the manuscript Case 1: X<3. If the source code deadline has not been reached, no of the assigned author. Additionally, one situation that often action is needed. Otherwise, if X=2, the two students build up the occurs is when one or more students miss the deadline of smallest possible review ring (they are assigned as reciprocal submitting their source code. For an instance in 1-to-1 assignment, reviewers); and if X=1, the merge algorithm is invoked (see see (a) of Figure 1, the student #2 who fails to submit source code Figure 2). on time will affect his/her two peers: the assigned reviewer #1 has Case 2: X≥3 and Y≥2. The reviewer assignment will be no code to review and the author #3 has no reviewer. performed among those X students. In order to remedy this, the ideas of compact algorithm (see Case 3: X≥3 and Y=1. In order to give a chance to the last Figure 1) and merge algorithm (see Figure 2) are utilized so that student to submit the source code (we call him/her student L), we the computer compacts the review ring by eliminating the students will let the penultimate student (we call him/her student W) who who do not submit their source code. When there is only one has submitted the source code, wait for L. In this case, our student left in a review ring, that student is labeled as a "lost" one, reviewer assignment will be performed among those X students, and will be merged to another review ring. Note that the merging except for W. Even if X=3, the two students except for W can position will be carefully chosen to make sure that the previous build up the smallest possible review ring. Finally, student W will reviewer has not yet started his/her review work at the merging not be assigned until student L submits his/her source code. moment. In Figure 2, before merging, student #4 is the reviewer of student #5. If student #4 has not started the review work when It is possible for only one student being unassigned by the first the computer tries to merge, the merging can address this position, deadline. In Case 2, only one among all Y students submits his/her i.e. inserting student #3 between student #4 and student #5. work before the deadline which will make the single student dangling. In Case 3, L has not submitted the source code when the 2 2 1st deadline is reached, student W will be dangling. To deal with dangling students, the merge algorithm (Figure 2), would be 1 3 1 3 necessary. Theoretically, there is an additional scenario, in which all students in the last-built review ring have started their review work when the computer tries to merge. Despite having a very low 5 4 5 4 probability, the system will notify the dangling student revise the source code according to his/her own understanding. Certainly, (a) student #2 misses (b) student #2 is removed the first deadline from the ring we can improve this algorithm by searching for an appropriate Figure 1. Illustration of the compact algorithm position in all review rings in the future. 3 2.4 Exceptional Cases and Solutions Generally, most students can finish the 3 stages on time, but 2 4 5 4 5 exceptional cases may occur occasionally, not only in source code stage, but in the reviewing and revising stage. The possible exceptional cases and their solutions are listed in Table 1. 1 3 7 6 7 6 (a) only student #3 (b) another (c) student #3 is 3. EFFICIENCY AND PROACTIVENESS is remained after compacting reviewer assignment ring merged into that ring REWARD Figure 2. Illustration of the merge algorithm From a managerial viewpoint, different algorithms may have different performances. The following section provides a When the system cannot find a position to merge a "lost" student capability analysis of the three RAM algorithms from two into because all assigned reviewers have started (or finished) their different perspectives: processing efficiency and proactiveness review work, the system will collect all the "lost" students in all of reward. the review rings and build up a new one. If there is only one "lost" Table 1. The exceptional situations and corresponding solutions of three algorithms Situations Has influence on peers? Solutions The author who fails to submit source code is considered to quit the current program task. post-assignment No The assigned reviewer gets review marks automatically. Compact a review ring by eliminating the students who fail to submit the source code. fail to submit pre-assignment Yes When the number of students in one review ring minimizes to one, the single student will source code be merged into another review ring. Only one student is still waiting to be assigned when the first deadline is reached. The submission-queue Yes student is merged into the last-built review ring. fail to finish When the review deadline is reached, computer prompts the author to revise the program Yes review by himself/herself. fail to revise No The corresponding marks of the author will be deduced. 3.1 Processing Efficiency In pre-assignment, the reviewer assignment is finished when the Whether a reviewer is active or passive, the possible starting point project (programming task) is released. A reviewer can start the to review is critical because it bottlenecks the entire process. The review process only after his/her peer student (previously assigned) assignment algorithm plays an important role in facilitating an has submitted the source code. Students may submit their source active reviewer to start a review job as early as possible. The code at their convenience, since an early or late submission will necessary conditions for starting a review work are: not accelerate or slow down the entire process much. If a student demands his/her source code to be reviewed, he/she must have (1) The author has submitted the source code; submitted his/her own source code. On the other hand, if a student (2) The reviewer has finished and submitted his/her own source wants to review his/her peer's code, his/her peer must have code. To reduce plagiarism, the process constrains that a reviewer submitted his/her own source code as well. The time interval cannot review other's code before he/she submits his/her own between submission and review may be large but since there is no source code; difference between submitting it early or late, the majority of (3) The reviewer assignment has been completed, that is to say students might submit their own source code as late as their peers that an author has an assigned reviewer who will review the do; which in turn becomes a submission close to the deadline. source code. The assignment of a reviewer to an author depends on which of the three possible reviewer assignment algorithms is However, in submission-queue we find two obvious advantages applied; related to proactiveness rewards, relative to the other two (4) EduPCR automatically informs the reviewer that the review assignment methods: process may start by short messages. (1) Active students can maximize their learning pace as they wish, The starting time available for review is of great significance; since whoever submits the source code first is assigned a reviewer among the four conditions, the 3rd might bottleneck the entire therefore having their work reviewed first. review process, hence it becomes fundamental to optimize the (2) submission-queue attracts students to join the active student assignment moment. circle. Normally, students who have good programming skills It is easy to understand that the students can start the reviewing often submit their source code very early, and submission-queue and revising process much earlier in pre-assignment and can meet these students' time saving requirement. In addition, submission-queue than in post-assignment, which implies that the some students whose programming skills are limited may wish to first two may be more efficient. Considering the individual be assigned in one review ring consisting of skilled students, in activity, some students may submit their source code very quickly order to improve their abilities. Therefore, submission-queue can while others might submit their source code just before the first stimulate both skilled and limited students to submit their source deadline. In pre-assignment, if the fastest and slowest students are code as soon as possible. assigned to a group to review each other, the fastest student will Hence, in terms of proactiveness reward, we predict that waste time waiting for the slowest student's source code. This submission-queue may have predominant positive influence on issue is resolved in submission-queue, allotting for faster students students' leaning than the other two. to be assigned to fellow fast student. Therefore, from the processing efficiency standpoint, we predict 4. PRELIMINARY STUDY that submission-queue is the most effective and post-assignment is The EduPCR system has been applied in the pedagogy of two the least effective. courses: C Programming and Object Oriented Programming in 3.2 Incentive Effect to Learners Java for students majoring in Information Systems at a Chinese university. To understand the three assignment algorithms’ effect, In post-assignment, all students have to wait until the last student we conducted a comparative investigation on the course Object submits his/her source code or the deadline, and then the Oriented Programming in Java in one 23-student class. Because assignment begins. Therefore, many students may choose to of lack of experience, we stored the review submission time in 12- submit their source code until the deadline, since an early hour (not 24-hour) format, losing some information and accuracy; submission cannot trigger an early reviewer assignment. the results might be affected to some extent. 4.1 Investigation Design With K-Means in IBM SPSS Modeler, we cluster all students into (1) There are ten tasks in this course. At an average interval of four clusters according to individual AWT in ascending order, seven days, the teacher sets one task according to the lecture's which means that the students in the first cluster spend the least amount of time finishing the review, while the ones in the fourth content. Each task has three phases: submitting source code, cluster spend the largest amount of time to do so. Moreover, we reviewing peer's code, and revising source code. The average classify all the students as active and inactive; the students in the duration of a phase is about 2 days. For every phase, a deadline is first two clusters of both investigations are considered active set to control the process so that any activity that misses the students, while those in the last two clusters are considered deadline will be blocked. inactive. Finally, the clustering result is obtained, i.e. 5, 5, 7, 6 (2) With each new task, all students are randomly divided into students in cluster 1 through 4 accordingly. Within each cluster, three groups to assure objectivity, thus, the sizes of the groups are we compute the AWT values according to the algorithm group. about 8, 8, and 7 people. For every group, an assignment The AWT of the four clusters in the three groups is shown in algorithm is randomly chosen among post-assignment, pre- Figure 3. assignment and submission-queue. We name each random group an algorithm group since each group is deployed with a different assignment algorithm. All students may start their review work only after they have received the review notifications sent by EduPCR. (3) After all the ten tasks are completed, the time consumption is analyzed so that the efficiency of the three algorithms can be studied. Since the time when students submit their source code is not greatly affected by the reviewer assignment algorithm and the revision submission time is indirectly affected by the assignment algorithm, we just focus on the time when the students submit their reviews. As previously mentioned, post-assignment algorithm starts the review process after the source code deadline, while pre-assignment and submission-queue algorithms can start the review process before the source code deadline. To compare the review efficiency of the three assignment algorithms, we take the difference between the review submission time of a reviewer and the source code deadline as the central statistic measure. This Figure 3. AWT distribution of the students measure is named wait time (WT) of the reviewer; different reviewers have different WT values. Obviously, the smaller the For each algorithm, the group AWT is increasing from cluster 1 to WT value is, the more active the reviewer is. Then with pre- 4, meaning the activeness of students is decreasing assignment and submission-queue algorithms, some reviewers' correspondingly (see Figure 3). WT value is negative, which means that the reviewers finish their Even though the database design defect might affect the accuracy second stage task (i.e., review work) before the deadline of the of this study, the results are as follows: first stage. During the implementation of EduPCR for years, we find that active students often finish their own source code and (1) The processing efficiency prediction for the three algorithms is begin review work as soon as they get the "new review" validated. Except for the submission-queue's value of the third notification, while some tardy students will not start each step cluster in Figure 3 is a bit higher than their corresponding pre- until the deadline is approaching. Thus, the value of average wait assignment value, all the other values meet our prediction time (AWT) can be used to measure the activeness of students in (1=submission-queue 2=pre-assignment 3=post-assignment). each assignment algorithm. AWT is the average value of several (2) The proactiveness reward prediction for the three algorithms is WTs, which has two cases: 1) individual AWT means the average validated. The time windows available to review with pre- value of one student’s WTs within all tasks, which is used for assignment and submission-queue are the same. That is to say that, clustering students into groups by their WTs; 2) group AWT is the without the effect of behavioral factors, the submission-queue and average value of a group of students' WTs within all tasks. Each pre-assignment AWT values of each cluster should not be greatly group has three group AWTs because three RAM algorithms are different. However, from the data shown in Figure 3, we can find deployed. that almost all the AWT values with submission-queue are much 4.2 Results from Data Clustered by more efficient than those with pre-assignment because students do not need to wait the peer's source code submission for a long Individual AWT time after he/she submits his/her own source code. Moreover, the To eliminate the effect of different personalities, we cluster all earlier a student submits his/her source code, the more chance students according to individual AWT within all tasks. Usually he/she has of being assigned to a review circle with similarly clusters generated from the clustering algorithm are not active students. We do not mean that all active students are comparable. However, in this study, students are clustered by a excellent, but the active students may like to study together with single variable: individual AWT. Each cluster has a group AWT students of great learning enthusiasm. Thus, the result of the that is comparable with another group AWT of a different cluster investigation demonstrates that the submission-queue does play so that we can order the generated clusters. the function of stimulating. (3) The active students are more sensitive to the reviewer assessment greatly improves the grading efficiency in MOOCs, assignment algorithm. Within each cluster, the submission-queue such assessment of open-ended assignments is still much complex and pre-assignment AWT values have difference, so do the AWT and time-consuming with massive students in courses. If value of pre-assignment and that of post-assignment. We use the reviewers could start their review process as soon as possible, the AWTs difference value to measure the students' sensitivity to the peer assessment process will be largely advanced. Therefore, the reviewer assignment algorithms. From Figure 3, we prove that the processing efficiency and proactiveness reward of submission- AWT differences in the first two clusters are much greater than queue will be more prominent in MOOCs. those in the last two clusters, which indicates that the more active the students are, the more sensitive they are to the reviewer ACKNOWLEDGEMENT assignment algorithms. This work was partially funded by China Scholarship Council [201506125055], National Natural Science Foundation of China 5. CONCLUSION AND DISCUSSIONS [71573065], and Online Education Research Foundation (QTone With the increasing requirement of peer assessment research, Education) of China's MOE [2016YB130]. especially in the age of MOOC, the reviewer assignment becomes Thanks to some students at School of Management, Harbin a pressing concern. The reviewer assignment research in the Institute of Technology, such as Miss Hang Li and Miss Xiaolei educational context, especially on the reviewer assignment Wang, for their assistance of collecting data and analyzing the moment, gets little focus. results. From the perspective of RAM in the PCR context, we propose three algorithms comprising post-assignment, pre-assignment, REFERENCES and submission-queue. The preliminary investigation among one [1] Van Zundert, M., D. Sluijsmans and J. Van Merriënboer classes in an academic years reveal that submission-queue has a (2010). Effective peer assessment processes: Research much higher processing efficiency and proactiveness reward than findings and future directions. Learning and Instruction, the other two algorithms. Moreover, we found that active students 20(4), 270-279. doi:10.1016/j.learninstruc.2009.08.004 are much more sensitive to the application of reviewer assignment algorithms. Based on these results we discuss two main concerns: [2] Newman, W., & Taylor, A. (1999). Towards a methodology employing critical parameters to deliver performance (1) Do assignment pairs created with submission-queue algorithm improvements in interactive systems. In Proceedings of promote fair feedback? This is an interesting issue addressed by INTERACT (Vol. 99, pp. 605-612). some scholars with academic rigor. Actually we have to admit that [3] Wang, Y., Li, H., Feng, Y., Jiang, Y., & Liu, Y. (2012). proactiveness is the base of active learning, however, for the Assessment of programming language learning based on peer participants in peer assessment, being active is not the whole story. code review model: Implementation and experience report. Fair feedback depends on the competence matching between peers, Computers & Education, 59(2): 412-422. morality level of participants, adequate and appropriate training, doi:10.1016/j.compedu.2012.01.007 quality assurance strategy by instructors, and so on. These are all challenging topics in the future research. [4] Mahoney, M. J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review (2) Since submission-queue has predominant advantages relative system. Cognitive therapy and research,1(2),161-175. to the other two, can it be taken as a universal algorithm in all doi:10.1007/BF01173636 scenarios? In our opinion, the choice of RAM algorithm depends on the context of each specific application. Even though [5] Tsai, C. C., Lin, S. S., & Yuan, S. M. (2002). Developing submission-queue has more advantage over post-assignment and science activities through a networked peer assessment pre-assignment, these two algorithms also have practical values. system. Computers & Education, 38(1), 241-252. doi: For example, if the proactiveness reward and late submissions are 10.1016/S0360-1315(01)00069-0 not important, pre-assignment is easy to be applied and the entire [6] Tànfani, E., & Testi, A. (2010). A pre-assignment heuristic process is simple to be managed. Similarly, if the proactiveness algorithm for the Master Surgical Schedule Problem (MSSP). reward and completion time are not critical, post-assignment Annals of Operations Research, 178(1), 105-119. becomes a very suitable algorithm. doi:10.1007/s10479-009-0568-6 (3) Does submission-queue suit for MOOC or traditional [7] Cho, K., & MacArthur, C. (2010). Student revision with peer classroom? We consider that the reviewer assignment moment in and expert reviewing. Learning and Instruction, 20(4), 328- submission-queue always plays its role effectively in peer 338. doi:10.1016/j.learninstruc.2009.08.006 assessment no matter what the education context is. We find that [8] Chen, Y. (1999). Banking panics: The role of the first-come, submission-queue can stimulate students’ learning enthusiasm and first-served rule and information externalities. Journal of advance the task completion process especially when the students Political Economy, 107(5), 946-968. are inherently active. In traditional offline courses, the peer assessment process is the same as online process like in EduPCR [9] Wang, J., Cao, J., & Li, Q. (2001). Reliability analysis of the so that submission-queue will suit for traditional classroom. More retrial queue with server breakdowns and repairs. Queueing importantly, submission-queue tends to be more useful for Systems, 38(4), 363-380. doi:10.1023/A:1010918926884 assessing the performance of students in MOOCs. Although peer