Learnersourcing Modular and Dynamic Multiple Choice Questions Haesoo Kim 1, Inhwa Song1, Juho Kim1 1 University - Korea Advanced Institute of Science & Technology (KAIST), Daejeon, Republic of Korea Abstract Multiple choice questions (MCQs) are widely used for evaluating learning outcomes. In particular, student-generated questions have shown promise in promoting active learning and higher-order thinking. However, the process of generating quality MCQs can be challenging and time-consuming for students. Additionally, the existing crowdsourcing approaches for MCQ generation lack scalability and quality control. To address these issues, we introduce a system concept called Kuiz that implements a modularized and dynamic method for generating MCQs, allowing students to contribute at various levels and personalize their learning experience. The questions are modularized into question stems, answer sets, and distractor sets, enabling students to refine and improve them collaboratively. By dynamically altering question stems and answer sets, we enhance the quality and difficulty of the MCQs, providing personalized learning opportunities. Through Kuiz, we aim to reduce students' burden in question generation tasks, increase engagement, and create scalable learning materials. By combining learnersourcing with dynamic question generation, Kuiz offers a framework for creating engaging and personalized learning experiences. Keywords 1 Learnersourcing, Student generated questions, Dynamic question generation 1. Introduction Multiple choice questions (MCQs) are a commonly used resource in learning and are known to be an effective way of evaluation and testing for various learning goals [3]. In particular, student generated questions have been noted as an effective way to promote active learning as it encourages higher level thinking in students [9]. Systems such as Peerwise have evaluated the effect of using student generated multiple choice questions (SGMCQs) in the classroom, suggesting that it is capable of increasing engagement as well as learning effect across various subjects [2]. Interventions based on motivational theories [9] can come to the rescue, encouraging students to pursue these optional activities. We propose applying the ideas behind educational psychological interventions to create just-in-time interventions that are triggered straight at the decision points with a focus on stimulating students’ engagement with extra activities. However, the issue still remains that generating questions is a challenging task for most. Question generation often requires high level thinking and understanding of the subject, which can be a discouraging factor for students [5]. There is also the issue of quality control, inherent in many crowdsourcing tasks. The average quality of SGMCQs can be high when students are provided with proper scaffolding activities [1], but there is still room for improvement especially regarding large- scale, open question repositories outside the classroom. Khashaba et al. have also noted that users of SGMCQ systems also preferred answering questions to creating them, due to the larger perceived efficacy in learning [6]. Here, we also recognize the need to effectively utilize the generated question set in a way that is scalable and beneficial to the students. In light of this, we propose a method of learnersourcing multiple choice questions such that the questions are modularized and dynamic. We also introduce Kuiz, a system concept that utilizes the The first annual workshop on Learnersourcing: Student-generated Content @ Scale, June 01, 2022, NYC, NY EMAIL: haesookim@kaist.ac.kr (H. Kim); igreen0485@kaist.ac.kr (I. Song) ; juhokim@kaist.ac.kr (J. Kim) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) aforementioned method. Here, the questions are modularized in that each question can be subdivided into question stems and options, both of which are subject to refinement through learnersourcing. This has the effect of allowing modular participation from the students, reducing their burden and cognitive load. Furthermore, the questions are also dynamic in that a given question stem and answer set could be utilized to create multiple versions of varying quality and difficulty. Through this approach, we aim to reduce the students’ burden in question generation tasks by allowing students to contribute in various levels and forms, and ultimately facilitating engagement. We also provide increased flexibility and variability in the question creation process, allowing for more personalized and effective methods of self-testing for learning. Finally, by incorporating these two tasks, we propose a framework where learnersourcing tasks can directly contribute to creating scalable learning materials. 2. Dynamic Generation of MCQs To dynamically construct a multiple choice question, we first divide them into smaller units. Each question contains three components: (1) the question stem, (2) the answer set, and (3) the distractor set. The structure is illustrated in Figure 1. Figure 1: Structure of a dynamically generated MCQ. (A) shows the construction of the question stem and (B) shows how options are determined from the distractor and answer sets. 2.1. Question Stem Each MCQ is built on a question stem, or the question part of the MCQ. Question stems often follow a set convention of formats that are representative of the answer type [4]. Through multiple different approaches to the question stem, it is also possible that the question-maker can inquire the same knowledge through multiple different approaches. We propose that, given a concept that the question is trying to verify, there can be multiple different versions of the question stem. For example, if there is more than one element in the answer set, the question stem may be modified to a multiple answer question. Similarly, given an MCQ that asks for the correct option, the answer set and distractor sets could be substituted so that the question stem asks for the incorrect option (e.g. “Which of the following is NOT an instance of...”). This method of altering the question can be used to modify the quality and difficulty of the given MCQ. For example, multiple answer MCQs are known to be more challenging than typical MCQs, as well as having more pedagogical value due to the expanded solution space and reduced efficiency of random guessing [7]. Following such methods, questions could be personalized to the student to further increase learning gains. 2.2. Answer and Distractor Sets The answer set and distractor set refer to the collection of possible answers and non-answers to a given question stem, respectively. Each answer or distractor should be paired with an explanation that explains why or why not this is the answer for a given question stem. Since the quality of multiple choice questions relies heavily on the quality of the options [8], we aim to provide a scalable way to generate and evaluate options through learnersourcing. Each answer or distractor can be evaluated based on metrics such as the selection ratio: how many times it has been chosen with reference to how many times it appeared. For distractors in particular, the system can use this data to determine ‘effective distractors’. If a distractor is chosen many times in lieu of the actual answer, such data may suggest that the option is an effective distractor, swaying the student from the true answer. Conversely, if a certain answer is not chosen often, it might mean it is a ‘harder’ answer, more difficult to guess. This approach can be used in tandem with the variable question stems, since questions with effective distractors would be more difficult than questions with distractors that are ‘obviously wrong’. 3. Learnersourcing System Design: Kuiz Kuiz is a system that utilizes the dynamic SGMCQ concept to promote efficient learnersourcing at a larger scale. There are two main stages to the system: Question Creation and Self Testing. 3.1. Question Creation and Refinement In the question creation stage, students focus on creating questions, augmenting questions that others have made through quantitative and qualitative feedback, and adding their own options. This process will build the question stems as well as the answer and distractor sets to be used in future phases. We further modularize the question creation process by eliminating the need to create full questions in the initial phase. Students may first create a question based on the given set of possible question stem types (denoted in Figure 1 (A)). Without options, this functions as a simple answer question. Students are then encouraged to present their own answers or distractors, as well as their level of confidence. Here, there can be two desired effects. First, the open-endedness of the question format reduces the impact of guessing, encouraging students to think more deeply about the concept. Second, even if the student submits a wrong answer, such answers can still contribute to the system as a distractor. Thus, the system can encourage students to try and answer even if they are not very confident about their knowledge level. As the options are accumulated, they can be grouped by similarity and evaluated by other students to ensure correctness. Even if a student submitted a wrong answer in the first stage, the feedback process can rectify this mistake and transfer the option to the distractor set instead. Finally, the system constructs answer and distractor sets based on the collected options, and can begin generating the MCQs. Students can continuously contribute to the generated questions, by leaving feedback on the question stem or options, as well as creating new options. 3.2. Self Testing In the testing stage, the collected set of questions can be used to dynamically generate ‘test exams’. Through this, students can evaluate their level of understanding of a subject. Here, dynamic MCQs can be utilized to create non-identical variations of the same question Thus, students will be less affected by learning effect by solving the same question repeatedly. This approach improves the scalability of the testing process and allows students to have a bigger learning effect. Moreover, by using multiple versions of the question stem, the system can account for varying difficulty per the level of student even with the same question. Finally, the testing stage can provide data on such as answer rate, user ratings on questions, and so forth. This can be further used to evaluate metrics such as question quality, the difficulty of the question, and even the quality of options; allowing further development of the question set. 4. Conclusion In conclusion, our research introduces Kuiz, a learnersourcing system that addresses the challenges of generating high-quality MCQs in education. By modularizing the question components and incorporating dynamic variations, Kuiz allows students to contribute and refine questions collaboratively, reducing their burden and promoting engagement. The system offers personalized learning opportunities by modifying question stems and answer sets, catering to individual students' needs and enhancing learning effectiveness. Moreover, Kuiz integrates feedback mechanisms and evaluation metrics, ensuring quality control and improvement of the question set. Overall, Kuiz represents a promising system that empowers students, improves question quality, and facilitates the creation of scalable and engaging learning materials. 5. References [1] Simon P Bates, Ross K Galloway, Jonathan Riise, and Danny Homer. 2014. Assessing the quality of a student-generated question repository. Physical Review Special Topics-Physics Education Research 10, 2 (2014), 020105. [2] Paul Denny, John Hamer, Andrew Luxton-Reilly, and Helen Purchase. 2008. PeerWise: students sharing their multiple choice questions. In Proceedings of the fourth international workshop on computing education research. 51–58. [3] Mercedes Douglas, Juliette Wilson, and Sean Ennis. 2012. Multiple-choice question tests: a convenient, flexible and effective learning tool? A case study. Innovations in Education and Teaching International 49, 2 (2012), 111–121. [4] Thomas M Haladyna, Steven M Downing, and Michael C Rodriguez. 2002. A review of multiple- choice item-writing guidelines for classroom assessment. Applied measurement in education 15, 3 (2002), 309–333. [5] Vincent Hoogerheide, Justine Staal, Lydia Schaap, and Tamara van Gog. 2019. Effects of study intention and generating multiple choice questions on expository text retention. Learning and Instruction 60 (2019), 191–198. [6] Ahmed Sayed Khashaba. 2020. Evaluation of the Effectiveness of Online Peer-Based Formative Assessments (PeerWise) to Enhance Student Learning in Physiology: A Systematic Review Using PRISMA Guidelines. International Journal of Research in Education and Science 6, 4 (2020), 613– 628. [7] Andrew Petersen, Michelle Craig, and Paul Denny. 2016. Employing multipleanswer multiple choice questions. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education. 252–253. [8] Michael C Rodriguez. 2005. Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational measurement: issues and practice 24, 2 (2005), 3–13. [9] Anjali Singh, Christopher Brooks, Yiwen Lin, and Warren Li. 2021. What’s In It for the Learners? Evidence from a Randomized Field Experiment on Learnersourcing Questions in a MOOC. In Proceedings of the Eighth ACM Conference on Learning@ Scale. 221–233.