Learnersourcing presents an eficient and economical pathway to producing more learning content, whilst engaging students more actively and deeply in their learning. However, it also presents new challenges to solve. Most of all, how best can we manage the variance in quality of content produced. In this work, we focus on the extent to which students and teachers agree on quality of learnersourced multiple choice questions. Students (n=30) were tasked with producing six questions over three weeks of an introductory programming course as part of their assessment. They also had to review 12 questions authored by their peers over the same period using a set of principles for good questions. After this period, four teaching staf involved with the course reviewed the student questions using the same process and principles. Inter-rater reliability statistics found overall positive agreement across principles, however this dropped to weaker agreement for principles aimed at more subjective and higher order concerns of question quality and quality of question feedback.
Learnersourcing can be succinctly defined as crowdsourcing content from students in a learning
context [
Creating learning content is a higher-order activity for students [
Here, we approach this question by studying the agreement gaps that emerge when students
and teachers are asked to give their opinions about learnersourced MCQs. To give structure
to these opinions, we use a set of principles for producing better MCQs that were presented
during the production and review of questions [
In response to the 2015 European Refugee Crisis, academics at KTH Royal Institute of Technology,
Stockholm, Sweden developed an intensive three month training to integrate newly arrived
into the local IT workforce [
Early iterations of this training found that, whilst students could produce good MCQs there
was a cold-start problem in MCQ quality - students gradually got better over time. Furthermore,
students were good at writing questions and answering alternatives, however when it came to
the explanation or feedback that accompanied the MCQ, students struggled to provide similar
quality [
In the most recent iteration, to help students understand and apply the principles, students worked as a group to learn about writing good questions before they undertook the task individually. Principles were used to create a question and then review a question created by another group. In this way, students had a chance to discuss their interpretation of the principles and apply them both in creating and reviewing MCQs. For each week of the three week course, students were tasked with creating two questions and reviewing four questions. Once the course had ended, four teachers reviewed 96 questions independently using the principles and review process used by the students. For each question there were three student reviews and four teacher reviews.
First, taking student and teacher IRR scores separately, teachers had a higher average agreement over all principles (0.82) versus students (0.65). Also, students had a greater distribution of scores (from 0.20 to 0.97) versus teachers (from 0.54 to 1.00). These findings can be partly explained by the teachers both generating the principles and also discussing their possible interpretations. Students on the other hand had much less opportunities to discuss the interpretation of the principles, other than within the scheduled group creation and review sessions, which only occurred twice in the three week course.
Second, looking at the largest diferences in IRR scores by principle, “ Only the feedback to the correct alternative reveals the answer ” had a magnitude of 0.53. This suggests a diference in attitude with teachers seeing more value for MCQ feedback that helps correct student misconception without taking away the chance of a second attempt. Another large diference relating to the feedback (0.37), “Feedback is unique and provided for each answer alternative”, continues this theme, whereas students did not have similar strong agreement. This is interesting as it is an objective measure - each answering alternative should have its own unique feedback and it only requires a cursory glance to confirm. Much like the smallest diference (0.03) for “Three or more answer alternatives are provided”, where it is quite easy to count the number of answering alternatives. One limitation of PeerWise is that there is only a single text area for ‘explanation’ and this may have contributed to this anomaly.
Third, looking at the smallest diferences that are not clearly objective or easy to determine (like “Question is from the course domain”), both principles: “All answer alternatives are plausible and related to a misconception” and “Feedback is suficient for you to understand why each alternative was incorrect or correct” are interesting as these are quite challenging aspects of quality to agree upon, even for teachers who understand the intent of each principle. This suggests a limitation of the depth of quality one can hope to measure when rating learnersourced content, however the IRR results here are still promising with medium agreement between students (0.59 and 0.48) and between teachers (0.69 and 0.54).
Finally, when combining both student and teacher reviews together, creating a pool of seven reviewers, the most agreement can be found in perhaps the most objectively answerable principles. This is not surprising and acts as a nice control for agreement on the basics of MCQs. Of more concern is that where there is weak agreement, two principles are concerned with feedback (“Feedback is suficient for you to understand why each alternative was incorrect or correct” and “Only the feedback to the correct alternative reveals the answer ”) and the other concerning if the question targets higher order thinking. This represents a challenge that warrants deeper investigation as the value of feedback is well known and accepted in efective education, but what if our diferent points of view on it never actually meet and the feedback that teachers feel is suficient is not exactly (or even close to) what students need?
Learnersourcing generates more content, but comes with the challenge of how to determine quality. Part of solving the challenge is finding out where students and teachers agree (or not) about quality. Use of agreement statistics, such as inter-rater reliability, are a potentially useful metric, but as mentioned here, caution is advised as there are many to choose from and not all work as expected. The work presented here shows that there are areas we can find agreement, however we need to find better ways to solicit impressions of quality at deeper levels and then ifnd ways to integrate and automate them within learnersourcing platforms.