Gamification is a rising research trend but with varying reported results in terms of behavioral and psychological outcomes. We compare the effect of leaderboards and experience points between an online and offline cohort of an identical web design module. We find higher engagement and enjoyment amongst online students and that experience points significantly enhanced their learning effectiveness. Leaderboards were found to be generally ineffective as a motivator for on-time submissions, especially for mid to late semester assignments. Grades for the offline cohort were significantly worse when compared to their non-gamified counterparts from the previous two years.
Students in higher education tend to spend a great deal of leisure time playing computer games (Hainey, Connolly, Stansfield & Boyle, 2011). There has also been an increasing trend in the number of students participating in online courses due to the expanding ways in which Universities deliver their content (Kim & Bonk, 2006). During recent years, efforts to "gamify" online and traditional courses brought the term "gamification" into academia with many reported benefits (Camilleri, Busuttil, & Montebello), (Muntean, 2011). As described in (Deterding, Dixon, Khaled, & Nacke, 2011) gamification is "the use of game design elements in non-game contexts". Since 2011, the area of gamification in academic writings has been increasing rapidly, making the study popular in academic research (Hamari, Jonna & Harri, 2014). Gamification is generally theorized to affect the process of learning by altering the behavior of the student and affecting the relationship between the instructional content and the learning outcomes (Landers & Landers, 2014). Research on gamification in a learning context often measures student perspectives on a gamified module using such metrics as: enjoyment, engagement and impact of learning (Dong, Dontcheva, Joseph, Karahalios, Newman, & Ackerman 2012). As suggested in (Christopher, Cheong & Filippou, 2013) motivation and engagement in particular may be a rising problem for students in higher education with part-time employment and social activities competing with academic studies.
Our study applies two motivational affordances (points and leaderboard) to a cohort of online and offline (traditional classroom) students. Each cohort is partaking in an identical web design module as part of a postgraduate diploma in computer science. For the points affordance, we measure the engagement, enjoyment and impact on learning of both cohorts to explore if the content delivery (online or offline) had any effect on student perceptions. For the leaderboard affordance, we awarded students a place on the leaderboard for punctual submission in an attempt to encourage on-time and early submissions. Our paper is structured as follows, in Section 2 we discuss some of the relevant literature in the field, in Section 3 we detail the research methodology and provide justification for same. In Section 4, we evaluate our findings, in Section 5 we discuss our research in broader terms, reflecting on strengths and limitations and in Section 6 we give considerations for future work.
Studies on gamification generally measure behavioral and psychological outcomes using both quantitative and qualitative research methods. Points, leaderboards and achievement badges were found to be the most commonly investigated motivational affordance. Psychological outcomes (the focus of our own study) often measure aspects such as motivation, enjoyment and attitude (Hamari, 2013). Studies often report a blend of positive and negative results with (Cheong et al., 2013) reporting a majority of students felt engaged enough to earn points but were not necessarily happy while doing so. Other research using similar affordances (Witt, Scheiner, & Robra-Bissantz, 2011), report a majority of participants did enjoy the experience and felt engaged with the task at hand. In a detailed literature review of empirical studies on gamification, (Hamari et al., 2014) suggest several limitations of the research, including a lack of comparison groups, short period of evaluation, singular assessments and a lack of validated measures. Studies such as (Hanus & Fox, 2015) attempt to address these limitations by studying the effects of educational gamification over an entire semester. Using leaderboards, badges and incentive systems, gamified students were found to have lower grades, be less motivated, satisfied and empowered when compared to the control group. The authors suggest that applying gamification mechanics to an already interesting module, can harm intrinsic motivation and appear "controlling". A similar perspective is found in (Mekler, Brühlmann, Tuch & Opwis, 2015) showing that individual gamification elements do not increase intrinsic motivation or satisfaction, however a significant increase in performance does occur in most cases. As demonstrated by (Mollick & Rothbard, 2014) gamification is more effective when participants can choose whether or not to participate in "mandatory fun", although they comment in the context of workplace gamification. In order to account for an increase in performance in gamified tasks, (Gollwitzer & Oettingen, 2012) suggest goal setting as a possible psychological mediator. As discussed in (Landers, Bauer, & Callan 2015) leaderboards may function as an implicit form of goal-setting, inviting participants to set goals towards the top of the leaderboard. Many of the mixed results from studies on gamification stem from the metaphorical distance between good game design and academia. As suggested by (Deterding, Canossa, Harteveld, Cooper, Nacke, & Whitson, 2015), much of the limitations or misconceptions surrounding gamification as a tool for learning, may be due to the design of the game component, which in many cases does not have a grounding in game design. Other research suggests that student learner types evolve from patterns of interaction with gamified components over long periods of time, suggesting a more customized gamification experience may be beneficial for learners (Barata, Gama, Jorge, & Gonçalves 2017). In a recent thoughtful reflection on gamification research, (Nacke & Deterding, 2017) postulate that when it comes to gamification, there is no "one size fits all" approach and careful consideration must be given to game design frameworks as well as individual learning and gaming personas.
In total 54 students signed up for the Web Design Module. Of those, 27 participated in online classes and 27 participated in offline (traditional face-to-face) classes. For the online cohort, 22 were male and 5 were female, 1 was aged 15-24, 23 were aged 25-44 and 3 were aged 45-64. For the offline cohort, 18 were male and 9 were female, 1 was aged 15-24, 22 were aged 25-44 and 4 were aged 45-64.
The application of experience points was facilitated by a third-party online platform called 'Quizizz'. We wanted to apply a computer-game feel to our quiz in order to try and elicit a similar psychological experience as a game generally would. Quizizz is colourful, incorporating game-like graphics, avatars and sounds. In previous years of the module, we had used standard Moodle quizzes.
We applied a leaderboard similar in structure to (Landers & Landers, 2014), however our motivation for using the various goals differed whereby we didn't explicitly measure time-on-task but rather the time at which the student submitted their assignments. The leaderboard (Table 1) consisted of ten goals divided into three sections. The first section awarded students for being the first, second and third to accomplish each task. In order not to weaken the quality of submitted material, we also added a second section where the best submissions were subjectively chosen by the lecturer at each stage. In any case, the Section 1 goals were not related to ability and could be achieved by any group. In addition, the position awarded on the leaderboard had no impact on grades as students could make changes to their submissions before the end of the semester. The third section displayed the top three individuals who had accumulated the most points from the multiple choice quizzes. To earn points, our students completed five quiz's and were awarded a point score depending on the number of questions they answered correctly and the time in which those questions were answered. Sections One and Two on the leaderboard helped to create high-stakes long term conflict, while section three created short-term lowstakes conflict (points awarded weekly).
We measured the enjoyment, engagement and learning of the points affordance using self-evaluation. Specifically, we used questions assessing the dimensions of enjoyment and engagement from (Whitton, 2007) and (Feng, Chan, Brzezinski, & Nair, 2008). We measured learning basing our questions on the work by (Bourgonjon,Valcke, Soetaert, & Schellens, 2010). The structure of our survey was similar to that of (Cheong et al., 2013). With regards to the leaderboard motivational affordance, we thought it might be worthwhile to see if groups would submit their various project components early or on time. In previous years of this particular module, students had a tendency to submit either just-in-time or past the deadline entirely.
We measured the engagement, enjoyment and learning of our experience points using questionnaires for both online and offline cohorts. In total, 23 of the 54 students completed the end of semester questionnaires, with n=9 for the online cohort and n=14 for the offline cohort. We evaluate the engagement (Table 2) and compare the results from both cohorts. We found that online students felt generally more engaged in the earning of points across four of the five dimensions. Although no significant difference was found between cohorts in terms of delivery method; both felt engaged while earning points in the module.
Next we evaluate enjoyment (Table 3) and find that online students were happier and less worried about earning points. In contrast the sentiment of exhaustion was slightly less pronounced in the offline cohort. Again both cohorts tended to enjoy the application of points, but no significant difference was found as a result of content delivery method. Finally, we evaluate the learning dimension (Table 4) and find that the online cohort had a better learning experience in terms of improved performance, productivity and effectiveness. Both cohorts felt similarly about experience points as a tool to enhance grades. A simple unpaired t-test showed a significant difference between the mean responses of the online cohort (M=3.82, SD=.09) and offline cohort (M=3.6, SD=.08); t(6)=3.56, p<0.05). We conclude that the content delivery method caused a significant difference to opinions on experience points in the learning dimension.
By introducing leaderboards into the module, we hoped that students would be more likely to adhere to submission deadlines in contrast to previous years. All 54 students from both cohorts were separated randomly into groups of three or four. We found that offline groups did best at the beginning of semester with 4 of the 7 groups submitting early or on-time for Stage 1. This gradually degraded as the semester went on with only 1 group submitting on time for Stage 3 and 4. In contrast, early or on-time submissions for the online groups increased at the end of the semester with 3 groups submitting on time. In general online students had little interest in the leaderboard at the beginning and middle of semester with only 1 group submitting early or on time for Stages 1 and 3 while 2 groups submitted early or on time for Stage 2.
We were able to compare only the offline cohort with previous non-gamified years as this was the first year that content for the module was delivered online. Quiz grades for gamified offline students (M=66.72, SD=12.22) were significantly weaker than the 2015 cohort (M=80.71, SD=15.31); t(48)=3.49, p<0.05. Quiz Grades were also significantly weaker than the 2016 cohort (M=83.62, SD=20.66); t(49)=4.93,p<0.05. In terms of project grades, the gamified offline cohort (M=69.2, SD=20.66) were also significantly weaker when compared to the 2015 (M=78.88, SD=20.66); t(45)= 2.09, p<0.05) and 2016 (M=78.59, SD= 12.11); t(60)= 2.25, p<0.05) cohorts.
Overall, students from both cohorts enjoyed the gamification experience and were generally positive towards experience points. With regards to engagement, online students felt more engaged but less excited than their offline counterparts, however we can't say this was due to the content delivery method. For enjoyment, online students expressed stronger positive and negative feelings (happier, miserable) but again, no effect from content delivery is found. We find that content delivery had a positive impact on how points were perceived in the context of the learning experience of the online cohort. We can speculate that the online nature of the class made it more difficult for students to interact or have a sense of presence amongst their classmates and the use of a game mechanic such as experience points may help somewhat. As we only measured psychological outcomes of a single motivational affordance, we cannot say that badges/achievements or a narrative wouldn't have done just as well.
The use of leaderboards as a motivator for on-time submissions yielded contrasting results for both cohorts with the effect quickly diminishing for the offline students and only lightly motivating some online groups to submit on-time. The project CA component submissions had varying difficulty, the first and last submission being the easiest to complete, this may have been a factor in both cohorts' submission habits. Another possible contributing factor is the lack of incentive for other groups to submit in a timely manner after the top three positions had been filled. Additionally, weaker groups may have traded-off a position in Section 1 if they felt their work was not up to the standard of achieving a position in Section 2 (though we can only speculate). In any case, we suggest that mid-late semester goals be carefully thought out, particularly for offline cohorts. The application of a leaderboard affordance in an online context needs further work in terms of goal setting. We find similarities with (Cheong et al., 2013) in that students felt engaged and learning was enhanced, but also we find similarity with (Hanus & Fox, 2015) as grades were significantly weaker than previous years, at least for the offline cohort.
Online students showed a significantly improved perception of learning experience from the application of experience points when compared to an offline cohort in an identical module. Further work should test additional motivational affordances on the same cohorts for comparative analysis. As a motivator for on-time submissions, we find that leaderboards are ineffective for mid-late semester goals for offline cohorts and almost completely ineffective for online cohorts during the semester. Careful design considerations should be placed on these goals to avoid loss off student interest. In future it may be beneficial to rethink how students complete the end of semester questionnaires in order to close the completion gap between different groups of participants. Finally, the use of motivational affordances may have significant negative effect on grades and serious thought should be applied to the application of gamification in higher education modules, even in the case where leaderboard positions have no relationship with actual CA grades.
This research is part of the project NEWTON, funded by the European Union's Horizon 2020 Research and Innovation programme under Grant Agreement no. 688503.