In recent years, peer assessment has established itself as one of the most promising methods of assessment in open learning environments typically in massive open online courses (MOOCs). However, researchers are still working on ways to address a number of challenges in peer assessment. In this paper, we propose some solutions from the field of learning analytics which have the potential to provide significant value in the peer assessment process. In recent years, Massive Open Online Courses (MOOCs) have changed the ideology and direction of technology enhanced learning (TEL). A number of challenges have been identified for MOOCs which include (but not limited to) the teacher's role, completion rates and most importantly assessment. Peer assessment offers a scalable and cost effective way in MOOCs where fellow learners are asked to evaluate student assignments and provide feedback to their peers encouraging them to take part in the assessment process [WCS16]. Despite its benefits, peer assessment faces issues like validity and quality of feedback, which require innovative solutions to make it an effective assessment method. In this study, we highlight the challenges of peer assessment and propose some possible solutions to these challenges from the emerging field of learning analytics. Peer assessment not only lessens the teacher's workload; it brings many potential benefits to student learning as well, including a sense of ownership and autonomy, increasing the motivation for learning and high level cognitive and discursive reasoning [WCS16]. But despite these potential benefits, peer assessment faces a number of challenges like transparency, credibility, accuracy, reliability, validity, diversity, scalability, and effeciency. In order to address the concern of accuracy of peer assessment in MOOCs, a number of approaches used by different peer assessment tools at various stages are discussed in [Su14]. They include connectivist MOOCs, reviewer calibration (Calibrated Peer Review, CPR), Bayesian post hoc statistical correction methods and creation of a credibility index [Su14].
Despite the importance of the work done in this area, the peer assessment challenges outlined above are far from being effectively addressed. In this paper, we argue that the emerging field of learning analytics can provide innovative ways to overcome these challenges. 3
In this section, we investigate how the learning analytics methods can be leveraged to improve the peer assessment process. A recent systematic comparison of peer assessment tools provides a number of dimensions including review loop, rubrics, validation, reviewer calibration, reverse reviews and scalability to judge effectiveness of any peer assessment system [WCS16]. These dimensions provide general research areas which could benefit from the use of certain learning analytics techniques. 3.1
Grading rubrics are an easy and effective way of defining a transparent criteria on which the learners can rate the work of their peers [Yo15].
Feedback Classification: The answers to rubrics from the reviewer would normally fall in short answers category. Applying classification techniques on these short answers could really help speed up the process of identifying better reviews and reviewers based on different features, such as feedback content length, reviewer’s credibility, content type (e.g. text, image), feedback creation duration etc.
Analyzing Rubric Answers: Another variation could be to use text analysis, word count and string matching techniques to make sure that the reviewer provides some meaningful answers to the rubric questions, instead of just writing ”nice” or ”good work”. 3.2
The validation dimension in peer assessment is a broader category that co-relates to the validity, reliability, accuracy and credibility challenges in peer assessment. Combining Peer and Automated Assessment: Validation could profit from the use of automated assessment using the classification and natural language processing techniques. However, to rely solely on automated assessment does not ensure a valid assessment. Thus, there is a need to follow a human-in-the-loop (peers) approach by blending automated assessment with peer assessment.
Accuracy Indicators: Accuracy indicators refer to the assigned weight to the peer raters according to their relative degree of accuracy. The higher the accuracy indicator, the more weight is given to that rater’s rating of peer submission [Su14]. Predictive analytics methods can be applied to predict the accuracy indicators of peer raters based on e.g. knowledge in the subject area, received ratings, and feedback history etc. 3.3
The number of learners in open learning environments tends to be greater than in traditional learning scenarios. It is thus essential for the assessment and feedback methods in these environments to be scalable.
Visual Assessment: A peer assement system could make use of clustering and visualization techniques to scale up the feedback process. It could either cluster similar submissions together or cluster them based on the reviews provided by peers and the answers to the rubric questions. Once the clustering is done, then visualization techniques (e.g. dashboards, word clouds etc.) could be applied to show clusters of submissions to the instructor, who could decide whether to grade all the related submissions at once or take a closer look at any of them. Another variation could be to show each submission with a visual aid to the teacher with features extracted from the submission itself as well as the reviews from peers. Important aspects of the submission could be highlighted using word clouds etc., to help speed up the teacher feedback as they could rely on visual aids to make a quick decision instead of going through all the submissions in detail. 4
Peer assessment is a cost effective and powerful assessment method which could be used in open learning environments such as MOOCs to provide effective assessment and feedback to a large number of learners. In this paper, we presented a number of open challenges in peer assessment including transparency, credibility, accuracy, reliability, validity, diversity, scalability, and effeciency. Further, we discussed how learning analytics can help to overcome these challenges by using classification, text mining, prediction, and visualization techniques at various stages of the peer assessment process to ensure its effectiveness. [Su14]
Suen, Hoi: Peer assessment for massive open online courses (MOOCs). The International Review of Research in Open and Distributed Learning, 15(3), 2014. [Yo15]