A Pilot Study of a Digital Skill Tree in Gameful Education Gustavo F. Tondello Lennart E. Nacke HCI Games Group, Games Institute and School HCI Games Group, Games Institute and Stratford of Computer Science, School of Interaction Design and Business, University of Waterloo, ON, Canada University of Waterloo, ON, Canada gustavo@tondello.com lennart.nacke@acm.org ABSTRACT regulate aspects of their thinking, motivation, and behaviour Gameful digital applications have been adopted in higher ed- during learning, such as the setting of learning goals, the strate- ucation to help increase student engagement and improve gies to achieve these goals, the management of resources, the learning. However, many studies have only evaluated educa- amount of effort exerted to study, and the reaction to external tional applications that combine some common game design feedback. Two related mechanisms that can be used to foster elements—such as points, leaderboards, or levels. Conse- self-regulation are open learner models (OLM) [4, 5, 6, 19] quently, we still lack studies exploring different ways of de- and self-assessment [22, 25, 27]. In intelligent tutoring sys- signing gameful learning experiences. Therefore, we introduce tems, the learner model is the data about the student’s current the design and implementation of a digital system employing a competence in the skills being taught [6]. When these models skill tree to mediate instructor feedback and assignment grad- are opened to the students, they can better understand and ing in a university course, Additionally, we present the results self-regulate their learning journey. In turn, independent open of a pilot evaluation with 16 students in which we summa- learner models (IOLM) are similar representations, which are rized the positive and negative aspects of the experience to not connected to a specific tutoring system [5]. On the other derive lessons learned for the use of digital skill trees in similar hand, self-assessment refers to the student’s judgement or rat- contexts. Finally, we suggest topics for further investigation. ing of their own work [25, 27]. When the two concepts are combined, the IOLM supports self-assessment [19, 22] and Author Keywords represents a common tool that learners and instructors can Gamification; Education; Skill Tree. use to discuss and plan the student’s learning journey. Self- assessment and self-regulation have been shown to improve INTRODUCTION the learning outcomes [23, 25, 27]. Gamification is being adopted in education to improve learn- ing and increase students’ motivation and engagement. This In this work, we introduce a novel use of a digital skill tree as tendency has been identified in systematic reviews focusing a gameful implementation of an independent learner model to specifically on gamification for education [3, 9, 15] and gami- support self-assessment and instructor assessment of student’s fication in general [16, 26]. Gamification is the use of game work in a university-level computer science course. Skill design elements in non-game contexts [8] or the use of af- trees are representations of progressive learning paths [29]. fordances for gameful experiences to support users’ value They have been used to organize lecture content (e.g., [1, 11, creation [12]. In education, elements of games can be used 18]), to provide structure and motivate students to complete to make the content more interesting, to motivate students to additional learning tasks (e.g., [2, 10]), or as a visualization complete more learning tasks, or to modify the way students option for open learner models (e.g., [5, 7, 14, 20]). Uniquely, are assessed and graded. Nonetheless, the majority of stud- we describe a digital implementation of a skill tree to structure ies focus on a small subset of gamification elements, such as and mediate student self-evaluation and instructor assessment points, badges, leaderboards, levels, and avatars [9]. There- of the programming assignments completed by the students fore, we still lack studies exploring different ways of applying over the course of a four-month term. Additionally, we present gamification elements to the learning experience. a pilot evaluation of this design idea through a descriptive study with 16 students to understand their experiences with At the same time, education scholars have been proposing the skill tree. To conclude, we then summarize the positive and new ways to improve students’ motivation and performance in negative aspects of the students’ experience with the gameful higher education, for example, by empowering them as self- aspects of the course and derive general lessons learned. regulated learners [23, 24]. Self-regulated students are able to Therefore, our work contributes to gamification and education by proposing a new way of implementing independent open learner models, self-assessment, and instructor assessment of students’ work with a gameful approach using digital skill trees. This design concept can be further combined in future work with additional gameful design elements, such as badges Copyright © 2019 for this paper by its authors. Use permitted under Creative Com- or unlockable content, to provide a comprehensive gameful mons License Attribution 4.0 International (CC BY 4.0). In: J. Arnedo-Moreno, C.S. González, A. Mora (eds.): Proceedings of the 3rd In- solution for higher education classrooms. ternational Symposium on Gamification and Games for Learning (GamiLearn’19), Barcelona, Spain, 22-10-2019, published at http://ceur-ws.org RELATED WORKS dents to complete additional tasks, or to display a hierarchical structure of the learning topics. Instead, we use it to struc- Gamification Applied to Education ture assessment and feedback regarding the skills needed to Studies of gamification in education comprise a considerable complete the programming assignments in a university course. portion of the existing gamification literature [16, 26]. Gam- ification has been used in educational contexts with positive Self-Assessment and Open Learner Models or mixed results to support a learning activity, improve an Research has shown that self-assessment improves student existing tutorial system, encourage participation, increase stu- learning [25] and “is considered one of the most important dent motivation and engagement, and encourage students to skills that students require for effective learning and for future do homework [26]. professional development and life-long learning” [27]. According to Kapp [13], gamification can be applied to ed- There are many ways to implement self-assessment of student ucation in two distinct ways. In structural gamification, the practical work. The approach that we propose shares some content is not altered and does not become game-like, but the similarities with the combined use of self-assessment and open structure around the content does. An example is using points, learner models. For example, Long and Aleven [19] allowed badges, and levels to track student progress. In content gam- student to self-assess their skills before displaying the tutoring ification on the other hand, the content itself is altered using system’s OLM. Mitrovic and Martin [22] allowed students to game elements. An example is adding gameful story elements inspect their OLM so they could self-assess their progress and to modify the way the content is presented to learners. It is choose the next tasks to solve. Another approach is that of also important to note that the gamification of education is persuadable OLMs [4, 21]. In this case, if the student does different from a serious game. A serious game is a full-fledged not agree with the assessment provided by the OLM, they can game with an instructional purpose, whereas gamification con- request a modification. sists of inserting elements of games without turning the whole instruction into a full game [3, 9, 17]. Our approach is similar to these prior works in the sense that it allows students to modify their self-assessed grades in Landers [17] proposed a theory of gamified learning, which the learner model (represented by the skill tree in our work). indicates that gamification can affect learning via moderation However, while previous works focused on letting students when the instructor makes pre-existing content better in some negotiate the values provided by intelligent tutoring systems way. An example is incorporating a gameful narrative into or other classroom assessments, our work is focused on let- an existing learning plan. On the other hand, gamification ting students self-assess their programming assignments on a affects learning via mediation when the instructor encourages computer science course. a behaviour or attitude that itself should improve learning outcomes. An example is using gamification to increase the SKILL TREE DESIGN AND IMPLEMENTATION amount of time that students spend with the course material, We implemented our gameful design in a third-year User Inter- which should cause greater learning. faces course of the Computer Science undergraduate program at the University of Waterloo during the Spring 2017 term Digital Skill Trees Applied to Education (May–August 2017). Students spent the majority of the course Skill trees are used in games and gameful applications as a learning how to implement user interfaces, with some course representation of progression [28, 29]. They have been used in time dedicated to issues of design and usability. They were gameful education with two different purposes: as a means to tasked with completing two major programming assignments organize lecture content or to provide structure and motivate and one small programming exercise: students to complete additional learning tasks. For example, A1: Implementing an interactive side scrolling game with a Lee and Doh [18] suggest a design for a digital e-learning level editor in Java. This was the largest assignment, which system that uses a skill tree to inform the user about what lec- consisted of three parts to be delivered on separate dates: tures they have already completed and what are the subsequent (1) user interface design wireframes; (2) basic user interface learning goals to pursue. Similarly, Anderson et al. [1], Turner and gameplay; and (3) level editor. et al. [30], and Hee et al. [11] describe gameful platforms for data science education that use skill trees to organize the A2: Implementing a small animation with a timer in Java. lessons into a logical progression. Following the approach of This assignment was introduced after A1, but students were using skill trees to organize task completion, Fuß et al. [10] de- expected to work on it in parallel and complete it before scribe a gameful system that groups related tasks with similar finishing A1. topics into lessons, then combines lessons into skills, which A3: Implementing a web client to retrieve information from are then organized as a skill tree. Likewise, Barata et al. [2] an open data API using jQuery and AJAX. used a skill tree to organize thematic tasks, which would earn We introduced gamification into the course by using a skill tree students experience points (XP) upon completion. Regarding the use of skill (or competence) trees as open learner models, to mediate assignment assessment instead of plain numerical a few works [5, 7, 14, 20] describe hierarchical displays or grades as is the common practice at our university. The skill pre-requisites views, which resemble skill trees. tree was implemented as a small web application that was used by students, teaching assistants, and the instructor. It Our approach differs from these prior works because we do not represented the skills that students were supposed to develop use the skill tree to organize lecture content, to motivate stu- while working on the three programming assignments. Figure 1. Skill tree used in the course (partly filled example). Figure 1 presents an example of a partly filled skill tree used in ation, then reflect on the feedback and fix their mistakes for the course. The precedence relationship between skills repre- improved learning and grades. Likewise, graders could update sented a suggested path for students to take while studying and their evaluation of the students’ skills at any time by providing working on the assignments. The colours represented different a proficiency level and a free-text comment. Grading was types of skills: grey for basic programming skills, green for carried out by six graduate students appointed by the course design skills, red for Java Swing skills, orange for Java draw- administrators as teaching assistants (TAs). The students’ final ing skills, blue for the model-view-controller pattern, and aqua assignments grade at the end of the course was calculated as for web programming skills. The numbers in the circles repre- the percentage of the skill tree that they had completed in sented the student’s completed proficiency on each skill and the graders’ evaluations. There were 23 skills in total with were updated separately by the student (self-evaluation) and by four levels each, thus adding up to 92 potential levels to be the graders throughout the term. The first number represented completed. Together, the three assignments accounted for 40 the student’s self-evaluated proficiency, whereas the second per cent of their final grade in the course; the remaining 60 per number represented the grader’s evaluation of their work. For cent of the grade were distributed between two written exams. example, a “3/2” meant that students evaluated themselves at Situating our gameful course design in the classifications pro- level 3 of proficiency, but the grader had evaluated them at level 2 out of 4 levels. Proficiency levels ranged from 0 (the posed by the literature, our approach is an example of struc- student has not demonstrated any skill yet) to 4 (the student tural gamification according to Kapp [13] because we did has achieved the top skill level expected for the end of the not modify the content of the assignments with gamification, course). The numbers in the mail icons (top-right corner of we merely provided a structure around it to improve grad- each skill box) showed how many new evaluations from the ing and feedback. Considering Landers’s theory of gamified graders were available for each skill and were updated as the learning [17], our gameful skill tree is supposed to affect graders registered new assessments. learning via mediation because it was designed to encour- age behaviours that could potentially improve learning: self- By clicking on each skill, students and graders had access evaluation and continuous improvement. Moreover, our skill to a detailed information page. Students could update their tree implemented the following design principles identified by proficiency for each skill at any time, by providing their cur- Dicheva et al. [9]: progress, feedback, accrual grading, visible rent proficiency level and a free-text comment. They were status, and freedom to fail. Finally, according to Taras’s classi- also allowed to resubmit previously submitted work at any fication of self-assessment models [27], our implementation time. The goal of this practice was to allow students to focus is an example of a standard model, in which learners use cri- on learning new skills more than on having to do everything teria to judge and grade their work prior to submission, then perfectly the first time to receive good grades. Therefore, they graders mark their work in the usual way while also providing could do their best work the first time and submit it for evalu- feedback regarding the students’ self-assessments. PILOT EVALUATION RESULTS Participants Table 1 shows how many participants answered with each The course had 159 registered students split into two sections. rating for the questions with Likert scales in the survey. Fur- While all of them used the skill tree to submit and receive thermore, all participants reported having completed at least feedback for their assignments, participation in the study was 70% of the skill tree in response to Q6: 70-79%: 2; 80-89%: voluntary and involved only a feedback survey. We sent an 8; 90-99%: 2; 100%: 2; N/A: 2; but we had no means of invitation by e-mail on the week after all the assignments had checking if their self-report was accurate because participation been submitted inviting all registered students to participate was anonymous. in the study by filling out an online form. Because the first author was the course instructor, the invitation was sent by a Experience Q2 Q3 Q4 (overall) (self-evaluation) (TA feedback) third party who was not involved in the course or the research; nevertheless, the invitation clearly stated the name of the re- Very positive 2 3 3 searchers responsible for the study. Furthermore, students Positive 9 6 7 were assured that their participation would be anonymous and Neutral 2 4 5 Negative 2 2 1 that the researchers would only access their responses after Very negative 1 1 0 the course grades had been finalized. These measures were re- quired by the ethical guidelines adopted by our institution and Q5 Preference (general preference) were intended to assure students that neither their decision to participate (or not), nor the answers they would provide, could Strongly prefer a skill tree 5 affect their grades in any way. Participants did not receive any Slightly prefer a skill tree 7 compensation. The study was reviewed and approved by the Strongly prefer numerical grades 3 University of Waterloo Office of Research Ethics. N/A 1 Table 1. Number of participant responses for each rating. In total, 16 students answered the online survey (14 men), aged between 20 and 24 years old. The low response rate is not unexpected because students did not receive any incentive Additionally, we read participants’ responses to the free-text to participate and the invitation was sent only by e-mail from a questions to understand their general impressions and the rea- third party unknown to them, without any mention or incentive son for their ratings. We summarize their answers in the for participation during presential lectures (as explained above, following subsections. Due to the small sample size, we were to avoid students feeling that their decision to participate could able to include at least a partial quote from all meaningful re- affect their grades). sponses (not all students provided a free-text follow-up to their quantitative answers). When quoting participants’ free-text Procedure responses, we use the letter “P” followed by the participant’s After following the link provided in the recruitment e-mail, par- order in the dataset (e.g., P1). Moreover, we classified the ticipants were asked to complete an online informed consent free-text responses as positive, neutral, or negative, according form. The research was presented as a study to understand stu- to the participant’s response to each question in the Likert dents’ impressions of the gameful elements used in the course. scale (because each free-text question was a follow-up to a The course included one lecture about gamification; thus, we quantitative question, as described in the previous section). can assume that the students were familiar with the term. In addition to demographic information (gender and age), the General Impression of the Skill Tree survey included the following questions: In answering Q1, 10 participants reported a positive impres- Q1: What was your general impression regarding the skill sion, noting that the skill tree was “an innovative way to do tree system used in the course? (free-text) grading” (P1), “a very unique way to evaluate my own skills Q2: How would you rate your overall experience with the and see what skills apply to which assignment” (P4), and a skill tree system? (5-point Likert scale with a free-text “very useful, transparent marking” (P16). P5 said “I liked the comment) Skill Tree system a lot, since when I update my skills and write Q3: How would you rate the experience of self-evaluating how I have achieved a rating for a skill, I actually think what your skills? (5-point Likert scale with a free-text comment) I have done for that skill. It also helps me in thinking what Q4: How would you rate the experience of receiving feedback else I can do to make sure I learn as much as possible about from the markers via the skill tree? (5-point Likert scale a skill. And of course, the feedback from TA’s also helped with a free-text comment) a lot.” Similarly, P15 said “Good and unique. It helped me Q5: In comparison with other courses which you have taken clearly understand where my strengths and weaknesses were at the University of Waterloo, which used a numeric grade regarding the course content.” system for assignments, how would you rate your prefer- On the other hand, four participants reported a negative im- ence? (5-point Likert scale only) pression. P6 said it was an “Interesting idea but seemed a Q6: How much of the skill tree have you completed? (selec- bit vague at times. There is a disconnect between assignment tion list with options corresponding to 10% ranges) requirements and the point evaluation system of the skill tree. Q7: Would you like to make any additional comments or Felt like a separate element rather than something directly suggestions regarding the skill tree? (free-text) connected to the assignments / progression in the course.” P11 did not like the fact that “requiring the student to go beyond in clear requirements for each level made this easier.” Also, P6 order to receive 25% of the marks is a terrible system to mark explained that “I just looked at the outline for the points (i.e., with,” because achieving level 4 of proficiency for many skills 0 - did nothing, 1 - submitted something, 2 - implemented it required students to implement an enhancement that went be- incorrectly, 3 - implemented it correctly once, 4 - implemented yond the minimal requirements. P12 felt that “it created more it correctly twice) and submitted the appropriate evaluation.” work for myself,” and P14 felt that “the skill tree to me, was confusing, and I think not needed.” On the other hand, participants who reported a neutral experi- ence said that “Self-evaluating made it more obvious what I It is also noteworthy that P3 reported liking the skill tree, but needed to do for each skill. However, since there were require- added that “sometimes the expectations between 3/4 were too ments listed on every page, it felt unnecessary at times,” (P13) strict and trying for a 4 and almost getting it but not quite and that “It would have been easier to simply check off boxes would make you get a 3 and not a 4 or anything in between,” for features that we did or did not implement” (P14). To the meaning that sometimes students would try to get the full 4 contrary, P11 reported a negative experience and asked “Why marks for a particular skill, but the grader thought it was not is it my responsibility to do my assignments, and mark them? enough and only rated the student at level 3 for that skill. I have no incentive to give myself anything but the highest mark.” Additionally, P7 stated “Don’t make 4/4 = going over Overall Experience with the Skill Tree and above. If you do the bare minimum you should still get Participants who reported positive overall experiences when 100%, if you do extra it should give you extra.” answering Q2 said that they “enjoyed seeing my progress and being able to visualize by learning” (P1), that it “motivated Experience of Receiving Feedback me to actively seek and meet the requirements” (P3), that it Participants who reported positive experiences in response to was an “interesting and helpful visualization of what is being Q4 said that it was “nice to know exactly what I needed to learned” (P7), and that it was “good for enticing me to add work on” (P3), “The TA responsible for giving me feedback features” (P13). was very good, gave good advice on how to improve my skill as well as why I deserve a particular rating. That really helped Contrarily, P14 had a negative experience and noted that: “I in my improvements” (P5), and “comments were always de- had a difficult time understanding the connection between tailed enough to act on” (P13). On the other hand, P4 stated what we were asked to do and how it was going to be marked – that the experience was “Generally positive but had some is- the communication of that information was unclear to me. As sues. I would meet the mastery requirements for some skills well, it reminded me of mobile games where they have deals [...] but the TAs would find an issue [...] and not give me full like ‘1530 coins for $4.22’, where it is difficult to understand marks for the ones I have fully implemented...” the impact of real life money on in-game currency. The fea- tures we were asked to implement did not get us marks, but the For the neutral experiences, participants said that “In my case, ‘skills’ that were expressed were marked, not to mention hav- I never received very meaningful feedback, it was usually of ing a denominator of 23 skills, and 23*4 marks in total made a binary nature, a checkmark of sorts (‘yes you submitted it hard to gauge progress while working on the assignment.” this and it’s working and looks good’)” (P6), “I would rather they just mark my assignments normally” (P11), and “there Even with a positive experience, P1 argued that “sometimes was very little incentive to finish assignments early” (P14). the assignment requirements and the skill tree don’t match Contrarily, P9 seemed to have a negative experience due to an up.” Likewise, P13 had a positive experience, but said that issue with the time that it took to receive feedback from the “skills did not always match up to requirements in the assign- TA: “It also took them 2 weeks to give me feedback”. ment.” P10 reported a neutral experience and said that “there was sort of a disconnect between the assignment expectations Additional Comments and the skill tree itself (some features in the assignment were In response to Q7, some participants made general suggestions not actually graded on the skill tree).” Furthermore, P6 said for the improvement of the skill tree: that the skill tree was a “Neat idea, fun to see how the skills connect. I regret not working on the assignments continuously “As mentioned previously a more clear assignment to skill tree and thus receiving feedback continuously. Instead I just did skill would be appreciated (e.g., setting menu in A1 is not everything at once and submitted it.” P7 had a positive experi- covered in the skill tree).” (P1) ence, but thought that the “course was too structured around “Give room for some almost marks between two levels (i.e. 3 it.” Finally, P10 suggested that “having individual trees for and 4) and have some bonus marks the instructor can give for each assignment would be clearer.” efforts that don’t exactly reflect on the skill tree.” (P3) Experience of Self-Evaluation “Skill tree system needs to somehow be worked in with the Participants who reported a positive self-evaluation experience traditional grading system. Perhaps more gameful elements when answering Q3 stated that it “motivated me to check my would make it more interesting / useful. Maybe something like, work that I met the requirements” (P3) and that “there were ‘receive X/Y points on these skills to unlock advanced starter clear explanations for what was expected, which helped gain code for assignment Z.’ Perhaps I would have been more an understanding of what the mark would look like” (P10). motivated to finish my first assignment at an earlier time if I P1 mentioned that “I felt conflicted about this at first, what had some motivation. [...] I would have definitely appreciated if I was too generous or too harsh on my own grades but the if I could unlock a better starter code for the assignment.” (P6) DISCUSSION levels were more granular (for example, with 10 levels In the previous section, we presented the results of an eval- instead of 4), so that students could be better rewarded. uation of the students’ experience with the skill tree system • Some students disliked that completing the minimum re- implemented in a university-level course on user interfaces, in quirements would grant them level 3 for most skills (a grade which the skill tree was used to provide feedback and replace of 75%), so they needed to implement enhancements of numerical grades for the programming assignments. In this their choice to get 100%. However, this was a common section, we summarize and discuss the findings from this study practice for this course in prior terms, which was only made and the lessons learned. more apparent by the skill tree. Instructors could take this opportunity to help students understand that this gives them Overall Experience some flexibility to implement what they want to get full The results suggest that most students had an overall positive grades instead of giving them only fixed requirements. experience with the skill tree: 11 participants (69%) reported • Having 23 skills with four levels each resulted in a total an overall positive experience, 9 participants (56%) reported of 92 skills points. It would be better to have a number of a positive experience with self-evaluating their skills in the skills and levels that lead to a round number (like 100) so assignments, and 10 participants (62%) reported a positive that students can easily understand how much each point experience with receiving feedback from the TAs through the earned in the skill tree will represent to their final grade. skill tree. Additionally, 12 participants (75%) said they would • Having just one large skill tree for the three assignments prefer a skill tree grading system in their next course instead of made it hard for students to separate the skills related with a numerical grading system. These findings suggest that using each assignment. a skill tree system might be a good idea. However, instructors • Some students misused the possibility of resubmitting their should take additional precautions to mitigate the negative projects, by delivering incomplete code initially and com- experience of the students who might not enjoy the skill tree pleting it later, whereas it was intended for students to use or offer them alternative means of assessment. the feedback to improve their initial work, thus leading to improved learning. Thus, the freedom to resubmit improved Strengths and Weaknesses projects needs to be better discussed at the beginning and By examining participants’ qualitative responses, we can iden- safeguards must be used to avoid students abusing it. tify the strengths and weaknesses of the skill tree design we • Some students did not understand why they had to evalu- employed in this study. The following aspects worked well: ate their own work. This shows that the benefits of self- evaluation must be better discussed at the beginning of the • Students enjoyed that the skill tree was an “innovative” and term to help students understand why it is an important “transparent” way of grading assignments. learning activity and a valuable ability for them to develop. • Students could better understand how they were learning • The TAs gave feedback with different levels of detail, possi- skills that they could use to implementing each assignment. bly due to their different time commitments and availability. • The skill tree helped students grasp their progress and work Some students felt that the TAs’ comments were not detailed to meet all the requirements for a full grade. or timely enough to help them. Thus, it is important to guar- • Students could understand if their work met the require- antee that feedback will be timely and that the amount and ments and could have a good idea of what their grades style of feedback given by the graders will be consistent. would be once they completed their self-evaluation. • The explanations given about how to self-evaluate each Lessons Learned skill seem to have worked well for most participants who Considering the findings from this study, we learned the fol- reported a positive experience. lowing lessons: • When the TAs gave students clear and detailed explanations • The tree must provide clear descriptions of each skill and about what they could do to improve their implementations, what students are expected to accomplish. students appreciated and acted on the feedback. • The tree must provide a clear mapping between assignment To the contrary, the following aspects did not work as intended: requirements and skills. • There was a disconnection between the skills and the assign- • The evaluation must use a sufficient grading granularity to ment requirements. The instructor and TAs had a table in adequately reward students for their efforts. Also, there which they established a relationship between programming must be a way of rewarding students for extra efforts that requirements and the skills they would provide; however, were not covered by the skills. this information was not disclosed to students. It would • It is better to use a number of skills and grading levels that have been better if the skill tree provided a clear informa- allow students to clearly understand how much each skill tion about what programming requirements were associated point will represent to their final grade. with each skill. The requirements can be presented as a checklist within each skill. • It is better to clearly identify each different assignment on • There were no grades in between proficiency levels 3 and the skill tree or use a different tree for each assignment. 4. Thus, students who tried to do extra work to obtain the • Instructors should create ways to mitigate the negative expe- latest level, but failed for any reason, ended up not getting rience or allow students who do not feel comfortable with anything for their effort. It would have been better if the the skill tree to be graded using a traditional method. In addition, some lessons learned from our experiment are ACKNOWLEDGMENTS more related to the experience of self assessment rather than We would like to thank all the students who took the User the skill tree, and echo best practices already identified in Interfaces course at the University of Waterloo on the Spring the education literature. However, we include them here for 2017, especially those that contributed with suggestions and completeness, and to make them easily available for educators the 16 anonymous participants of this study. We also thank the following the design concept proposed in this work. teaching assistants who carried out the work of grading and providing feedback to students. Moreover, we thank Marcela • At the beginning of the course, it is beneficial to discuss Bomfim for helping with participant recruitment. the benefits of self-evaluation and of improving one’s work from the instructor feedback for the students’ learning and This work was supported by the CNPq, Brazil; SSHRC [895- development of important life skills. 2011-1014, IMMERSe]; NSERC Discovery [RGPIN-2018- • If resubmission of improved work is allowed, safeguards 06576]; NSERC CREATE SWaGUR; and CFI [35819]. should be employed to avoid misuse of this freedom and REFERENCES prevent students from submitting incomplete initial work. 1. Paul E. Anderson, Thomas Nash, and Renée McCauley. • It is important to ensure that all graders will provide timely 2015. Facilitating Programming Success in Data Science feedback, which is detailed enough to help students im- Courses Through Gamified Scaffolding and Learn2Mine. prove their work. Moreover, graders should be available to In Proceedings of ITiCSE ’15. ACM, 99–104. DOI: respond to students’ questions as needed. http://dx.doi.org/10.1145/2729094.2742597 2. Gabriel Barata, Sandra Gama, Joaquim Jorge, and Daniel Limitations and Future Work Gonçalves. 2017. Studying student differentiation in Despite the small sample size, the free-text responses pro- gamified education: A long-term study. Computers in vided a rich source of data that allowed us to interrogate, and Human Behavior 71 (June 2017), 550–585. DOI: posit reasons for, the students’ positive or negative experi- http://dx.doi.org/10.1016/j.chb.2016.08.049 ences, congruent to our intentions for this study. Nevertheless, although the majority of participants in the cohort reported 3. Simone de Sousa Borges, Vinicius H. S. Durelli, positive experiences, our sample might have been affected Helena Macedo Reis, and Seiji Isotani. 2014. A by a self-selection bias because students who had a positive Systematic Mapping on Gamification Applied to experience might have been more inclined to participate in the Education. In Proceedings of SAC ’14. ACM, 216–222. study. Furthermore, some of the concerns described by the DOI:http://dx.doi.org/10.1145/2554850.2554956 students represent factors that are not directly related to the 4. Susan Bull, Blandine Ginon, Clelia Boscolo, and skill tree, such as the quality and timeliness of TA feedback. Matthew Johnson. 2016. Introduction of learning Future studies should try to better control for these factors. visualisations and metacognitive support in a persuadable Moreover, the course used in this study was heavily based open learner model. In Proceedings of LAK ’16. ACM, on a set of large programming assignments, which could be 30–39. DOI:http://dx.doi.org/10.1145/2883851.2883853 mapped to a skill tree. Future studies will need to investigate if 5. Susan Bull, Matthew. D. Johnson, Mohammad Alotaibi, this approach can be generalized to different styles of courses Will Byrne, and Gabi Cierniak. 2013. Visualising and courses in different disciplines. Multiple Data Sources in an Independent Open Learner Additionally, we evaluated the skill tree isolated in our course, Model. In Artificial Intelligence in Education. AIED 2013. so we could gather students’ feedback about this element LNCS 7926. Springer, Berlin, Heidelberg, 199–208. DOI: specifically. However, future works could combine it with http://dx.doi.org/10.1007/978-3-642-39112-5_21 other design elements, such as badges to reward the student’s 6. Susan Bull and Judy Kay. 2010. Open Learner Models. progress or unlockable content based on skill tree completion, In Advances in Intelligent Tutoring Systems. Studies in to design a comprehensive gameful learning experience. Computational Intelligence, vol 308. Springer, Berlin, Heidelberg, 301–322. DOI: CONCLUSION http://dx.doi.org/10.1007/978-3-642-14363-2_15 In this paper, we presented a novel design and implementation of a digital skill tree as a mediator of self-evaluation, feedback, 7. Ricardo Conejo, Monica Trella, Ivan Cruces, and Rafael and grading of assignments in higher education. The results Garcia. 2012. INGRID: A Web Service Tool for showed that the experience was generally positive. Therefore, Hierarchical Open Learner Model Visualization. In the digital system we described can be used to mediate the Advances in User Modeling. UMAP 2011. LNCS 7138. communication between graders and students in a gameful Springer, Berlin, Heidelberg, 406–409. DOI: http://dx.doi.org/10.1007/978-3-642-28509-7_38 way, particularly for larger classes, when communication in person with all students is not viable. The lessons learned 8. Sebastian Deterding, Dan Dixon, Rilla Khaled, and that we presented can guide educators who are willing to Lennart E Nacke. 2011. From Game Design Elements to implement this design idea. They can also help researchers Gamefulness: Defining “Gamification”. In Proceedings devise new ways to combine the digital skill tree with addi- of the 15th International Academic MindTrek Conference. tional gameful design elements for a more complete gameful ACM, Tampere, Finland, 9–15. DOI: learning experience. http://dx.doi.org/10.1145/2181037.2181040 9. Darina Dicheva, Christo Dichev, Gennady Agre, and 20. Andrew Mabbott and Susan Bull. 2004. Alternative Galia Angelova. 2015. Gamification in Education: A Views on Knowledge: Presentation of Open Learner Systematic Mapping Study. Journal of Educational Models. In Intelligent Tutoring Systems. ITS 2004. LNCS Technology & Society 18, 3 (2015), 75–88. 3220, J.C. Lester, R.M. Vicari, and F. Paraguaçu (Eds.). http://www.jstor.org/stable/jeductechsoci.18.3.75 Springer, Berlin, Heidelberg, 689–698. DOI: http://dx.doi.org/10.1007/978-3-540-30139-4_65 10. Carsten Fuß, Tim Steuer, Kevin Noll, and André Miede. 2014. Teaching the Achiever, Explorer, Socializer, and 21. Andrew Mabbott and Susan Bull. 2006. Student Killer – Gamification in University Education. In Games Preferences for Editing, Persuading, and Negotiating the for Training, Education, Health and Sports: GameDays Open Learner Model. In Intelligent Tutoring Systems. ITS 2014. Proceedings, Stefan Göbel and Josef Wiemeyer 2006. LNCS 4053, M. Ikeda, K. D. Ashley, and TW. (Eds.). Springer International Publishing, 92–99. DOI: Chan (Eds.). Springer, Berlin, Heidelberg, 481–490. DOI: http://dx.doi.org/10.1007/978-3-319-05972-3_11 http://dx.doi.org/10.1007/11774303_48 11. Kim Hee, Roberto V. Zicari, Karsten Tolle, and Andrea 22. Antonija Mitrovic and Brent Martin. 2007. Evaluating the Manieri. 2016. Tailored Data Science Education Using Effect of Open Student Models on Self-Assessment. Gamification. In Proceedings of CloudCom 2016. IEEE, International Journal of Artificial Intelligence in 627–632. DOI: Education 17, 2 (2007), 121–144. http://dx.doi.org/10.1109/CloudCom.2016.0108 23. David J. Nicol and Debra Macfarlane-Dick. 2006. 12. Kai Huotari and Juho Hamari. 2017. A definition for Formative Assessment and Self-Regulated Learning: A gamification: anchoring gamification in the service Model and Seven Principles of Good Feedback Practice. marketing literature. Electronic Markets 27, 1 (2017), Studies in Higher Education 31, 2 (2006), 199–218. DOI: 21–31. DOI: http://dx.doi.org/10.1080/03075070600572090 http://dx.doi.org/10.1007/s12525-015-0212-z 24. Paul R. Pintrich and Akane Zusho. 2007. Student 13. Karl M. Kapp. 2012. The Gamification of Learning and Motivation and Self-Regulated Learning in the College Instruction: Game-based Methods and Strategies for Classroom. In The Scholarship of Teaching and Learning Training and Education. Pfeiffer, San Francisco, CA. in Higher Education: An Evidence-Based Perspective, Raymond P. Perry and John C. Smart (Eds.). Springer 14. Judy Kay, Z. Halin, T. Ottomann, and Z. Razak. 1997. Netherlands, Dordrecht, 731–810. DOI: Learner know thyself: Student models to give learner http://dx.doi.org/10.1007/1-4020-5742-3_16 control and responsibility. In Proc. of International Conference on Computers in Education. 17–24. 25. Philip M. Sadler and Eddie Good. 2006. The Impact of Self- and Peer-Grading on Student Learning. Educational 15. Ana Carolina Tomé Klock, Aline Nunes Ogawa, Isabela Assessment 11, 1 (2006), 1–31. DOI: Gasparini, and Marcelo Soares Pimenta. 2018. Does http://dx.doi.org/10.1207/s15326977ea1101_1 gamification matter? A systematic mapping about the evaluation of gamification in educational environments. 26. Katie Seaborn and Deborah I. Fels. 2014. Gamification in In Proceedings of SAC 2018: Symposium on Applied theory and action: A survey. International Journal of Computing. ACM, 2006–2012. DOI: Human-Computer Studies 74 (2014), 14–31. DOI: http://dx.doi.org/10.1145/3167132.3167347 http://dx.doi.org/10.1016/j.ijhcs.2014.09.006 16. Jonna Koivisto and Juho Hamari. 2019. The rise of 27. Maddalena Taras. 2010. Student self-assessment: motivational information systems: A review of processes and consequences. Teaching in Higher gamification research. International Journal of Education 15, 2 (2010), 199–209. DOI: Information Management 45 (Apr 2019), 191–210. DOI: http://dx.doi.org/10.1080/13562511003620027 http://dx.doi.org/10.1016/J.IJINFOMGT.2018.10.013 28. Gustavo F. Tondello, Alberto Mora, and Lennart E. 17. Richard N. Landers. 2014. Developing a Theory of Nacke. 2017a. Elements of Gameful Design Emerging Gamified Learning. Simulation & Gaming 45, 6 (2014), from User Preferences. In Proceedings of CHI PLAY ’17. 752–768. DOI: ACM, 129–142. DOI: http://dx.doi.org/10.1177/1046878114563660 http://dx.doi.org/10.1145/3116595.3116627 18. Haksu Lee and Young Yim Doh. 2012. A Study on the 29. Gustavo F. Tondello, Rina R. Wehbe, Rita Orji, Giovanni Relationship between Educational Achievement and Ribeiro, and Lennart E. Nacke. 2017b. A Framework and Emotional Engagement in a Gameful Interface for Video Taxonomy of Videogame Playing Preferences. In Lecture Systems. In 2012 International Symposium on Proceedings of CHI PLAY ’17. ACM, 329–340. DOI: Ubiquitous Virtual Reality. IEEE, 34–37. DOI: http://dx.doi.org/10.1145/3116595.3116629 http://dx.doi.org/10.1109/ISUVR.2012.21 30. Clayton A. Turner, Jacob L. Dierksheide, and Paul E. 19. Yanjin Long and Vincent Aleven. 2017. Enhancing Anderson. 2014. Learn2Mine: Data Science Practice and learning outcomes through self-regulated learning Education through Gameful Experiences. International support with an Open Learner Model. User Modeling and Journal of e-Education, e-Business, e-Management and User-Adapted Interaction 27, 1 (Mar 2017), 55–88. DOI: e-Learning 4, 3 (2014), 243–248. DOI: http://dx.doi.org/10.1007/s11257-016-9186-6 http://dx.doi.org/10.7763/IJEEEE.2014.V4.338