Participatory Design of Feedback Mechanism in a Physics Blended-Learning Environment Elad Yacobson1 , Armando M.Toda2 , Alexandra I. Cristea2 and Giora Alexandron1 1 Weizmann Institute of Science, 234 Herzl St., Rehovot, Israel 2 Durham University, Stockton Road, Durham, UK Abstract This paper describes a series of participatory-design experiments conducted with physics teachers who use a blended learning environment, with the goal of developing mechanisms for collecting teachers’ feedback and opinions on the resources that they have used, which can assist peer teachers in search & discovery of appropriate learning materials. The main challenge that this study addresses is how to incentivise teachers to provide such feedback, as originally the response rate for feedback requests was very low. The novel approach that it proposes to address this challenge is by incorporating the idea of gamification into this crowd-sourcing challenge. First, we present a successful two-phase experiment conducted during the previous school-year, in which two gamification elements were implemented into the learning environment, resulting in a 𝑥2.6 increase in the response rate to feedback requests. Second, we discuss additional changes we are planning to implement in the feedback mechanism during the next school year. These are inspired by incentive frameworks, adjusted to the domain and context through interviews, questionnaires, and group discussions conducted with teachers who use the platform. Keywords Participatory-Design, Social-Recommendations, Blended Learning, crowd-Sourcing, Gamification 1. Introduction Blended-Learning environments usually provide educational users with large repositories of learning-resources (LR) [1]. Teachers can use this repositories to find learning materials that are most suitable to their students’ needs and to their own preferences, and administer them to their students [2, 3]. However, the vast amount of LRs that blended learning environments typically include make it difficult for teachers to find the appropriate learning materials [1]. Interviews we conducted with physics teachers regarding the process of searching and choosing LRs in a blended learning environment emphasized the importance that teachers ascribe to social recommendations, meaning that teachers highly value the opinion of their peers, and a recommendation about a LR given by a fellow teacher can substantially shorten and simplify the process of finding appropriate learning materials [4]. Alas, despite the fact that social recommendations are conducive to the process of searching LRs, when teachers are asked ECTEL 2021: AI for Blended Learning: Empowering Teachers in Real Classrooms, September 20, 2021, Bozen-Bolzano, Italy Envelope-Open elad.yacobson@weizmann.ac.il (E. Yacobson); armando.toda@usp.br (A. M.Toda); alexandra.i.cristea@durham.ac.uk (A. I. Cristea); giora.alexandron@weizmann.ac.il (G. Alexandron) Orcid 0000-0002-9816-4235 (E. Yacobson); 0000-0003-2681-8698 (A. M.Toda); 0000-0002-1454-8822 (A. I. Cristea); 0000-0003-2676-6912 (G. Alexandron) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) to provide feedback about LRs that they have used, they are often reluctant to do so [5, 6]. Therefore, our goal in this research is to explore the factors that influence teachers’ motivation to participate in teacher-sourcing activities, such as recommending LRs to other teachers, and to design elements and mechanisms that could be implemented into blended learning environments with the purpose of enhancing teacher participation in crowd-sourcing and collective wisdom processes.We believe our goal can be facilitated with the use of gamification, the use of game-like elements outside of a game, to stimulate the social interactions and recommendations [7]. To the best of our knowledge this line of research is the first to study gamification as means to incentivise teachers in crowdsourcing activities, with a practical contribution to an online learning system that is in use nation-wide. The rest of the paper is organized as follows: first, we present the learning environment in which we are conducting our research; next we present an experiment conducted during the previous school year, in which two gamification elements were implemented into the learning environment with the goal of encouraging teachers to provide feedback regarding LRs they used; and finally we discuss our plans for additional experiments we intend to perform next year. 2. Learning Environment - PeTeL PeTeL (Personalised Teaching and Learning) is a blended-learning environment that is developed at the Department of Science Teaching at the Weizmann Institute of Science. It is implemented on top of a Moodle learning management system (LMS), and consists of three main components: a shared repository of open educational resources (OER), an LMS that offers teachers learning analytics tools, and social network features. The main purpose of PeTeL is to assist STEM teachers in providing personalised instruction to their students. PeTeL is divided into separate modules for each subject matter: Biology, Chemistry and Physics. To assist teachers in searching and choosing learning materials that best suit their students’ needs, PeTeL provides common search filters, such as subject matter, level of difficulty, duration, technical requirements (e.g. projector or mobile devices), nature of the activity (e.g. diagnostic questionnaire, interactive task, home assignment, exercise etc.), and in addition, social-based search and discovery features. For example, teachers can follow other teachers within a social network style collaborative environment (referred to as the ‘peer network’), receive recommendations from them, copy their teaching sequences, and more. Teachers can also search and rank materials based on reviews provided by their peers. After using an activity in their class, the teachers are presented with a ‘pop-up’ window requesting them to provide feedback concerning the resource they used. The teachers can either fill the pop-up survey, postpone filling out the form to a later date, or cancel it. This feedback mechanism was initially activated in PeTeL during the 2019 – 2020 school year. However, teachers’ cooperation was relatively low, and their response rate to the feedback requests during this first year was below 3%. Since the reviews were identified by the teachers as very influential on their decision on which activities to use, and also provide the basis for an automatic ranking algorithm that is currently under design, we marked the issue of increasing the response rate as a major challenge that should be addressed. 3. Gamification Experiment In the first experiment, we decided to explore the idea of using gamification as means to enhance teachers’ motivation to provide feedback. In recent studies, gamification has proven to be an effective incentive mechanism for crowdsourcing methods [8, 9]. Our research questions were: (a) What gamification elements do teachers believe will encourage them to provide feedback?, and (b) Does implementing these elements actually enhance teachers willingness to provide feedback? 3.1. Methodology This experiment took place during the 2020-2021 school year, and consisted of two stages: in the first stage, we tried to understand which gamification elements are most likely to influence teachers’ behavior. In order to do so, we presented 17 physics teachers with five mock-ups of gamification elements that were based on a new and cutting-edge gamification taxonomy, designed specifically for learning environments [10], and asked them to rank these elements according to how much they believed each element would contribute to teachers’ willingness to give feedback. In addition, we gave teachers a few open ended questions and held a group discussion in which they were asked to express their opinion about the feedback mechanism in PeTeL and raise ideas on how to improve it. Our conclusions from this first stage of the experiment were: (a) Teachers need to have clear goals and to know their status with respect to these goals. (b) Teachers need to have a feeling of ‘impact’, i.e. they need to feel that their feedback is meaningful, that it is taken seriously, and that it is contributing to the rest of the teacher community and to the learning environment. (c) Social recognition matters, meaning that teachers not only want to feel that they are contributing, but also to be recognized by their peers for their effort and contribution. In light of these findings, in the second stage of the experiment, we implemented two gamification elements into PeTeL and measured the impact on the amount of feedback we received from teachers. The first element was a progress-bar: we defined a goal of five reviews per teacher per year, and each time a teacher would review an activity he or she used in class, the progress-bar would indicate the progress of that teacher toward that goal (see Figure 1). This element addressed the finding from the previous stage, regarding teachers’ need to have clear goals and to know their status with respect to these goals. The second element that was implemented into PeTeL was a recommendation bulletin board, appearing in the main page of the learning environment, and presenting teachers’ reviews about LRs they used in their classrooms (see Figure 2). Each time a teacher reviewed a LR he/she used in class, the bulletin board would be updated, presenting the new feedback at the top of the bulletin board. Each new entry to the bulletin board contained both the name of the recommending teacher, and the title of the LR that has been recommended. When hovering with the mouse over a recommendation, a mouse-over text would appear with the details of the recommendation. All of the recommendations appearing in the bulletin board were ‘linkable’, meaning that teachers could click on a recommendation and be moved to the LR repository so they could download the recommended LR to their own course or class. This element addressed Figure 1: Progress Bar Figure 2: Recommendations Bulletin Board both the teachers’ need to have a feeling of impact, that their feedback is actually helpful to the entire teacher community, and their need for social recognition. 3.2. Results After the implementation of these two elements, we monitored teachers’ feedback and compared them to the amount of feedback received during the previous school year (before the implemen- tation). The results showed a 𝑥2.65 increase in the amount of feedback we got from the teachers, and an equal increase in the response rate (i.e. the percentage of feedback requests that were actually answered by the teachers). For detailed information regarding this experiment, see [6]. 4. Planned Experiments 4.1. Improvements in the Design of the Feedback Mechanism Throughout the 2020-2021 year, we interviewed individual teachers, held group discussions with teachers attending a teacher-training program, and conducted several surveys, with the goal of better understanding teachers’ attitudes and preferences towards the feedback and recommendation mechanism, especially regarding the new gamification elements introduced in the system. Our intention was to build on the findings we obtained, in order to further improve the design of the feedback mechanism in PeTeL, hopefully bringing an even larger increase in the number of feedback items we will receive next year. A few findings emerged during this process: Importance of recommendation above all other aspects of the review: we found that teachers are most interested in whether the LR is recommended or not by other teachers. Questions that appear in the existing feedback form, such as whether the teacher used the LR in class or as homework, or what was the timing in which the teacher used the activity, were identified as second order. Progress bar does not influence teachers’ motivation: both in the interviews and in the surveys, a majority of the teachers indicated that they did not even notice the progress bar. The few that did see it, said it had no impact on their willingness to fill in reviews. Therefore, in the spirit of simplicity (see Occam’s razor), we decided to remove the progress-bar element from the new feedback design. We note that even though the participatory design process marked this issue – having an individual goal defined in terms of expected number of responses, and an indication on the progress towards it – as important, empirical evidence show its effect is minor. While this may be an indication of the (low) importance of this feature, it may also be that its design and implementation were unsuccessful in achieving this goal. Shorter feedback: teachers repeatedly emphasized that the length of the feedback form (i.e. the large amount of questions it contains) deters them from filling it. Following from these findings, in the new design of the feedback survey the teachers will be presented with only two questions: the first is whether they recommend this LR to other teachers; the second is an open-ended question, asking teachers for insights or suggestions concerning the use of the LR (see Figure 3). Another change to the feedback mechanism design will be performed due to the understanding of the importance of default choices on user behavior. According to the status-quo bias theory [11], when making decisions, people are usually more inclined to stick with the existing situation rather than making an active choice to change it. Therefore, the choice of the default options in the design of a system can have a crucial influence on users’ behavior, as was shown for instance in [12]. In the existing feedback mechanism in PeTeL, teachers are first presented with a pop-up window asking them if they would be willing to fill in the feedback form, and only if they click on the ‘write your response’ button, they are moved to the feedback form. Therefore, Figure 3: New Pop-Up Window Design the default choice is not to give feedback, as they are first required to actively choose to fill it. In the new design, the two questions in the feedback form are immediately presented to the teacher in the pop-window, making filling in the form the default choice. In this design, the teacher has to actively choose the ‘cancel’ button in order to avoid filling a review. 4.2. Planned Experiments and Conclusions In the coming school year, we are planning to continue our experiments, in order to examine both the effect of the new design on teachers’ behavior, and the effect of the aforementioned gamification elements on a larger number of teachers. To achieve these goals, we intend to implement the new feedback mechanism design we described previously into all three modules in PeTeL: Physics, Biology and Chemistry. In the physics module, we will compare the amount of feedback we will collect from teachers in the coming year (2021 - 2022) to the amount of feedback we received during the current year (2020 - 2021), in order to see whether the new design has an influence on teachers’ motivation. In the chemistry and biology modules, during the previous school year, there was no feedback mechanism available for teachers, to review and recommend learning resources they used. In the coming year, the new feedback mechanism will be also implemented in these two modules, and we will run an A/B test experiment: 50% of the teachers will be able to see the recommendation bulletin board (experimental group), and 50% will not (control group). This way, we shall be able to see whether there is a (statistically significant) difference in the amount of feedback we will get from each group, and thus understand if the bulletin board affects teachers’ motivation. 5. Acknowledgements This research is supported by a Making Connections Grant funded by Weizmann UK. The work of GA is also supported by the Willner Family Leadership Institute for the Weizmann Institute of Science. References [1] T. T. Dien, N. Thai-Nghe, et al., An approach for semantic-based searching in learning resources, in: 2020 12th International Conference on Knowledge and Systems Engineering (KSE), IEEE, 2020, pp. 183–188. [2] S. Downes, Models for sustainable open educational resources, Interdisciplinary Journal of E-Learning and Learning Objects 3 (2007) 29–44. [3] D. Porcello, S. Hsi, Crowdsourcing and curating online education resources, Science 341 (2013) 240–241. [4] K. I. Clements, J. M. Pawlowski, User-oriented quality for oer: Understanding teachers’ views on re-use, quality, and trust, Journal of Computer Assisted Learning 28 (2012) 4–14. [5] N. T. Heffernan, K. S. Ostrow, K. Kelly, D. Selent, E. G. Van Inwegen, X. Xiong, J. J. Williams, The future of adaptive learning: Does the crowd hold the key?, International Journal of Artificial Intelligence in Education 26 (2016) 615–644. [6] E. Yacobson, A. Toda, A. I. Cristea, G. Alexandron, Encouraging teacher-sourcing of social recommendations through participatory gamification design, in: A. I. Cristea, C. Troussas (Eds.), Intelligent Tutoring Systems, Springer International Publishing, Cham, 2021, pp. 418–429. [7] S. Deterding, M. Sicart, L. Nacke, K. O’Hara, D. Dixon, From Game Design Elements to Gamefulness: Defining ”Gamification”, Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems - CHI EA ’11 (2011) 2425. URL: http://www.scopus.com/inward/record.url?eid=2-s2.0-79957930613{&}partnerID= tZOtx3y1. doi:1 0 . 1 1 4 5 / 1 9 7 9 7 4 2 . 1 9 7 9 5 7 5 . a r X i v : 1 1 / 0 9 . [8] Y. Feng, H. J. Ye, Y. Yu, C. Yang, T. Cui, Gamification artifacts and crowdsourcing partic- ipation: Examining the mediating role of intrinsic motivations, Computers in Human Behavior 81 (2018) 124–136. [9] B. Morschheuser, J. Hamari, J. Koivisto, Gamification in crowdsourcing: a review, in: 2016 49th Hawaii International Conference on System Sciences (HICSS), IEEE, 2016, pp. 4375–4384. [10] A. M. Toda, W. Oliveira, A. C. Klock, P. T. Palomino, M. Pimenta, I. Gasparini, L. Shi, I. Bittencourt, S. Isotani, A. I. Cristea, A taxonomy of game elements for gamification in educational contexts: Proposal and evaluation, in: 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), volume 2161, IEEE, 2019, pp. 84–88. [11] W. Samuelson, R. Zeckhauser, Status quo bias in decision making, Journal of risk and uncertainty 1 (1988) 7–59. [12] C. Ansher, D. Ariely, A. Nagler, M. Rudd, J. Schwartz, A. Shah, Better medicine by default, Medical Decision Making 34 (2014) 147–158.