A review's quality can be evaluated through metric-based automated metareview. But not all the metrics should be weighted the same when it comes to evaluating the overall quality of reviews. For instance, if a review identifies specific problems about the reviewed artifact, then even with a low score for other metrics it should be evaluated as a helpful review. To evaluate the usefulness of a review, we propose a use of decision-tree based classifier models computed from the raw score of metareview metrics, instead of using all the metrics, we can use a subset of them.
MOOC-based education platforms as well as face-to-face
classrooms are increasingly adopting peer assessment. Peer
reviewing increases students’ participation and fosters
collaborative learning. Students are encouraged to review their
peers’ work and provide formative feedback. High-quality
feedback can help the reviewee improvise his/her work.
Reviewing (or evaluating) a review is known as metareviewing.
For best results, a review should be metareviewed before being
presented to the reviewee. Usually this is a manual task [
As mentioned above, a metareview evaluates a review based on certain textual properties, otherwise known as metrics. Below are the metrics used by our metareview evaluator.
Review relevance: A relevant review should discuss the work reviewed and try to identify problems/issues in author’s work. Review content: This metric is further divided into three metrics, such as: Summative, Problem Detection and Advisory. Summative: A summative review provides either positive feedback or a summary of the author’s work.
Problem detection: A review can detect one or more specific problems in the reviewed artifact.
Advisory: A reviewer can provide specific advice to the author, which can be used by the author to improve the artifact. Coverage: Coverage is a measure of review’s ability to cover the main points of the artifact.
Tone: Tone refers to the semantic orientation of a text. Tone is divided into three categories: positive, negative and neutral. A single review can contain various measures of positive, negative and neutral tone.
Volume: Volume measures the quantity of textual feedback provided by the reviewer.
Plagiarism: This metric is based on the originality of a review. If a review is copied, then it is marked as plagiarized. A review is compared against artifact, rubrics used and the internet search results based on the review text.
Our automated metareview system is a Ruby on Rails-based
web service [
Participants were former and current TAs from different departments of Engineering, Science and Business. We trained them by explaining the essence of each metareview metric used in automated metareviewing. Multiple participants were asked to rate the same reviews to generate a holistic model. We created an anonymous system to prevent the reviewers from knowing the identity of the authors of the artifact and the reviews.
The artifacts selected for this experiment were taken from the articles created by Spring 2016 students in CSC 517 course at NC State University. As a part of this course, students wrote Wikipedia articles which were then given to other students in class for reviewing. Each student was required to review two articles. They were given an option to review two more articles to receive extra points.
Table 1: Questionnaire for the survey
reviewer to the author to improve the artifact?
reviewer? (1: strongly negative, 2: negative, 3: neutral, 4: positive, 5: strongly positive
comments provided by reviewer?
metareview metric (scale of 1– 5)
A total of 119 reviews were surveyed in this experiment. Since more than one survey participant reviewed the same artifact, each review was assigned the average of the scores it received from all the participants.
All the questions were answered on a scale of 1–5, with 5 being the “best” score. For the tone metric, we found that only two surveys assigned a score of 1 (highly negative) to a review, whereas about 60% reviews received a score of 4 (positive). About 10% received a score of 5 (highly positive). We normalized the survey score for tone and grouped them into three categories. A score less than 3 (<3) was translated to –1 (Negative), whereas 3 was translated to 0 (Neutral) and a score greater than 3 (>3) was converted to 1 (Positive). The survey question associated with the overall quality of the review was normalized as well. A score higher than 3 was translated to good review (1), otherwise it was marked as bad review (0). We used this metric as class identifier for our data modeling. This was done to create a holistic model.
Figure 1 shows the distribution of surveys scores for each metric individually. We can see from this figure that not all the metrics are dispersed equally, which correlates with the idea that each metric is not equally important for evaluating the overall quality of the review.
Each review used in the survey was evaluated using the automated metareviewer, which generated metareview score for each review. The metareview web service evaluates each sentence of a review and tries to identify positive or negative words used in it from a collection of word list. If the count is same, then it is marked as neutral. An aggregated score of all the sentences is calculated for the review. So if a review contains positive and negative sentences, then the overall score can have a score for positive metric as well as negative metric. But for our experiment, we scaled the overall tone score. If the overall positive score for a review was higher than the negative score, then it was translated to 1 (overall positive review). If overall negative score was higher than positive score, then it was translated to –1 (overall negative review), else it was converted to 0 (overall neutral review).
The survey participants had an absolute agreement (zero
tolerance) of 38.8% with inter-rater reliability, calculated using
weighted kappa [
Sixty-five percent of reviews were rated as good whereas others
were marked as bad by the survey experts. Table 2 lists the
Pearson Correlation matrix between the score of the questions
based on metareview metrics to the overall quality of the review
as rated by survey participants. It can be easily inferred from
Table 2, that each metric is highly correlated with the overall
quality of the review, except tone. As per figure 1, volume and
advisory are two most dispersed metrics and they also show
greater correlation with the overall grade of a review, which
makes them two most important metrics for data modelling.
review as per surveys experts. This translates to similar results,
which we derived from Table 2. Based on the experiment and the
data collected from automated metareviewing, volume,
summative, and advisory are better suited metrics on which to
create a model to categorize the quality of a review. Other
metrics like tone, and problem identification should be used in
modeling as well. But metrics such as relevance, and coverage
are not performing well, so these metrics cannot be used for data
modeling.
While selecting the model that can be used to differentiate
between a good and a bad review, we investigated various
modeling methodologies. We wanted a model that is inexpensive
to construct, which can be retrained, and is extremely fast in
classifying unknown reviews. Also since, we are ignoring two
metrics in this modeling, we wanted a model that is flexible to
incorporate these variables at a later stage. One modeling
technique that looks ideal for these cases is a decision tree.
To create a decision tree, we started with Classification and
Regression Trees (CART) modeling using the rpart [
To find an optimal tree, a first attempt was made with volume, summative, advisory, problem identification and tone metrics. The summary function in rpart library shows that volume is a very important metric when generating the classification tree. Table 5 shows the result of the summary function, which states that the tone and problem identification were the least preferred metrics for classification From table 2, 3 and 4, Volume shows a stronger correlation with the class identifier (overall quality). Figure 2 shows that the volume metric alone can construct a classification tree to identify review quality. This decision tree can be used to identify whether the review is good or bad on the basis of the volume score received from the automated metareview metric. For instance, if the volume metric score is greater than 68, then it is a good review, or if score is less than 26, that is a bad review. This is not pruned at the moment. Another algorithm discussed later generates a more pruned tree.
Node 1 divided the sample space into two sets containing 42 and 77 observations respectively. A review with a score of 68.5 for metareview metric volume is used as the first split criterion. Each node number is marked in Figure 2, with split criteria and class probabilities.
Though volume can be a good classifier, volume alone should not
be used to identify the quality of a review. We found in another
study [
Majority class probability 64.7% 72% 88% 80.7% 87.4%
Metareviewing is an essential tool, which can improve the quality of reviewing. A reviewer can write a good review if timely feedback can be provided on the review before he/she submits it to the author.
As part of this work, we created a decision-tree data classifier based on the score a review receives from the automated metareviewer. Decision trees are fast and efficient classifiers for peer review metrics. We found that certain metrics, such as volume, dominate the decision trees. But reliance on the volume metric alone can generate false positives. We also created a decision tree excluding the volume metric. That decision tree uses content advisory, content summative, tone and problem detection metrics. We suggest the use of a hybrid model that includes use of both the trees. Each review is rated on both trees from Figure 2 and Figure 3. A good review should score well on both.
We used Wikipedia artifacts and reviews written for them in this experiment. To make the model more robust, more similar experiments can be done to include artifacts from other educational domains. We used supervised learning to create this model. Natural language processing (NLP) is becoming more and more efficient in determining the semantics of a text. The relevance metric generator should be updated to make it more robust, so that it can also be used in the classification decision tree.