Izabella Jedela, Adam Palmquistb, Robin Munkvolda, Ole Goethea, Hrafnhildur Jonasdottira, Erik Olssona a Nord University, Universitetsalléen 11, 8026, Bodø, Norway b University of Gothenburg, Forskningsgången 6, 417 56, Gothenburg, Sweden In experimental study with 1780 adults who participated in distance education, it was shown that gamification did not have an effect on performance, drop-out rate and pass-rate compared to a control group. Although no effect was shown, descriptive survey data suggests that the students were positive towards having similar gamification in other courses and would likely recommend the course due to the design. While previous research on gamification in online education indicates that gamification can have a positive effect, gamification researchers highlight the importance of context and design for successful implementations. The present study supports the need for context considerations and psychological design in gamifying online education. Furthermore, the present study highlights the need for a more nuanced understanding of engagement measures in gamification research and for more practical frameworks regarding the successful application of gamification in online education. Adult education, Distance education, Engagement, Gamification, Game elements, MOOC, Online education [2]. Previous literature has suggested gamification as a means to increase student performance and engagement [3, 4, 5] The production industry is currently going Gamification has been defined as the use of through a transformation in which game elements in a non-game context [6], and digitalization, manufacturing and automation focuses on creating gameful experience to are in focus. Educational institutions are [7]. In expected to provide accurate skills demanded education, the most common game elements by the industry. However, the size of the used are points, badges and leaderboards (PBL) workforce and the rapid transformation renders [3, 8]. However, the success of gamification is traditional campus courses ineffective [1]. One debated. While studies in academic settings problematic aspect with online education is low show positive outcomes of gamification [9], completion rates and as a consequence high many researchers also highlight the need for drop-out rate. The completion rate for most contextual considerations and good design MOOCs (Massive Open Online Courses) is practices [9, 10]. It has been suggested that below 13% [2] and at educational institutions, simply applying game elements in education the drop-out is estimated to be up to seven times should not be expected to lead to desired results higher in online courses than campus courses 5th International GamiFIN Conference 2021 (GamiFIN 2021), April 7-10, 2021, Finland. EMAIL: izabellajedel@hotmail.com (A. 1); adam.palmquist@ait.gu.se (A. 2); robin.munkvold@nord.no (A. 3); ole.goethe@kristiania.no (A.4); hjonasdottir@gmail.com (A. 4); erik.s.olsson@gmail.com. ORCID: 0000-0001-9212-3259 (A. 1); 0000-0003-0943-6022 (A. 2). 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) [12]. Instead careful consideration should be implementations in online education has been made to each game element and its intended subjected to scientific scrutiny. The implication effects on students [10]. of study design, gamification design and This study presents an experimental contextual factors is a common theme industry case in which students in several discussed. Rohan et al. [15] note that some of different online courses for adults were sample sizes in the studies reviewed were small randomly divided into either a group with a and that the studies focused on specific learner gamified learning management system (LMS) types or subjects, making the results less or the same non-gamified LMS to measure the generalizable. Looyestyn et al. [18] state that effect of gamification on drop-out rate, pass- although the majority of studies reviewed rate and performance. concluded that gamification is effective for increasing engagement in online programs, there is a need to determine how to better achieve sustained engagement. An et al. [19] addressed several barriers related to the Students have expressed interest for gamification of MOOCs, specifically gamification in online learning environments. mentioning lack of time, limited knowledge, In a survey conducted with over 500 lack of funding, lack of fit between participants to measure students' expectations gamification and the course content, concerns of MOOCs, approximately a third expressed a , desire for more developed gamification and concerns regarding negative effects of elements within MOOCs [13]. Furthermore, in gamification [19]. previous research it has been suggested that In another literature review of 61 articles gamification can increase engagement and concerning gamification in distance education, performance for students in online education as Antonaci et al. [20] showed that gamification in well as lower drop-out rates. Khalil et al. [14] the different studies is not coherent regarding found that of 10 previous studies that measured design, implementation, or outcome. A vital motivation and engagement effects of aspect of a gamification design seems to be that gamification in MOOCs, nine found a positive the context of the application is determined by effect and one found a partially positive effect. the aim of the gamified intervention, which in In a more recent literature review, Rohan et al. distance education can differ from case to case [15] found that of the 26 studies reviewed, 11 [20]. Therefore, Antonaci et al. [20] suggests reported positive findings, five partially that each game element in a gamification design positive findings and that no tests had been should be carefully chosen for its contextual conducted in the remaining 10 studies. purpose. However, the study concludes that the Similarly, to Khalil et al. [14] no studies were selection, as well as design of the most found that did not demonstrate a complete lack appropriate game elements are difficult to of effects. Aparicio et al. [16] suggests that determine. In addition, the authors conclude using gamification in MOOCs can increase that gamification, and its applications in engagement and participation in learning distance education, is still an undeveloped field contexts, and that the success of gamification lacking rigorous empirical investigations [20]. can be measured through engagement. Aparicio Several studies imply the importance of an et al. [16] further implies that the presence of increased understanding of why users become game elements positively affects MOOC usage, motivated by the gamification design. Khalil et and that gamification is a decisive factor in the al. [14] highlight the need to empirically test success of MOOCs. De Freitas & da Silva [17] the effectiveness of gamification elements in found a general increase in participation and MOOCs based on theoretical and retention on gamified MOOCs in a review of 22 multidisciplinary approaches. Similarly, De papers. In another systematic review of 15 Freitas & da Silva [17] highlight that an studies Looyestyn et al. [18] measured the important premise for successful effect of gamification in online programs. 12 of implementation is the need for a theoretical 15 studies showed that gamification could be model that relates game elements to their useful to increase end-user engagement [18]. specific learning outcomes. In addition, De Despite the positive effects proposed in Freitas & da Silva [17] argue that, based on self- previous research, the effect of gamification determination theory, intrinsic motivation should be a central to the design since elements programs for adults through distance education. that support extrinsic motivation are less The teaching method in the LMS differed from effective, and can, in fact, reduces intrinsic regular MOOCs by being teacher-driven with motivation, leading to negative motivational instructions taking place in the LMS, similar to effects [17]. Borrás-Gené et al. [21] found that an analog classroom. Lectures held by teachers students who had more fun were more likely to ran several times a day and adult learners were complete MOOC classes and that social aspects expected to study in the LMS in which at least seemed to have a positive effect on engagement one teacher was present for student support and completion rates. Romero-Rodríguez et al. during the workday. Course material (e.g., [22] found that incorporating gamification in presentations, recorded video lectures, articles) MOOCs by the means of competition increased was available in the LMS at all times. social interaction through intrinsic motivation. Students who attended courses in the LMS Furthermore, Krause et al. [23] found that a during one semester (n=1780) were social based gamification design in a MOOC automatically and randomly assigned to the was more successful than a less social design. gamification condition (n=909) or a control group (n=871). The students in both conditions ranged in age from 17 to 69 (M=27.29, SD=7.619), 59.7% (n=1062) where women and 40.3% where men (n=718). The courses The present study responds to the consisted of 19 different 5-week distance discrepancy addressed in previous literature courses in English, History, and Mathematics. reviews between empirical findings and All courses had varying educational levels and theoretical discussions on the importance of used both physical and digital learning context and design considerations. By studying material. The gamified condition and control an industry experiment in which gamification condition were identical apart from several was applied, this study aims to understand the game elements being included in the gamified potential effect of gamification in distance condition. For ethical purposes the students had education. Compared to previous studies, in the option to turn off the game elements and which sample sizes have been small and continue the course without the elements gamification has been course specific [15], the present. The students who had switched off present case includes a large sample size gamification were excluded from the analysis. (n=1780) with the same gamification design In the following parts, the gamification applied to several courses. The case involves a condition is first presented, and the data school that implemented gamification in analysis is later described. several courses and in different subjects, with the aim to increase completion rates and decrease drop-out rates. The following research questions are explored: Gamification was implemented through a RQ1: Does gamification increase project done in collaboration between the LMS student performance in an online- provider and a gamification studio. To identify learning environment? gamification design RQ2: Does gamification have an effect an initial workshop was conducted. In the on student drop-out rate in an online- workshop, two teachers, one educational learning environment? leader, the LMS product owner, and three RQ3: Does gamification have an effect developers, partook from the LMS enterprise. on student pass rate in an online- A gamification designer from the gamification learning environment? studio organized the workshop to collect information about the courses, the students and the intended outcome of the gamification implementation. The design workshop took approximately four hours and was outlined as a In the present experimental case study, modified version of the UX-design workshop gamification was applied to an LMS that portrayed by the Nielsen and Norman Group. In provided upper secondary courses and the design workshop, the gamification designer conducted several brainstorming exercises with The progression board contained a timeline for the stakeholders regarding user demographics, each week with stars for the activities to background, and personas. Furthermore, the complete. If the activities for the week were participating stakeholders' perceived notion of completed, part of the timeline was marked as their users' attitudes and perceptions towards green and if the activities for the week were not games in learning, study habits, and the digital fully completed, part of the timeline was experience was explored. The gamification marked as red. Apart from the game elements, designer asked questions concerning the the LMS, course design and learning enterprise existing impediments in their present environment was identical for the control group value proposition and how they visualized and the group with the gamified LMS. gamification could aid their predicaments. The designer also explained why the enterprise desired to implement gamification and investigated how that intention corresponded with the stakeholder's intentions. Lastly, it was decided which performance indicator was relevant to evaluate. The design workshop's outcome was included in a gamification design document functioning as a blueprint of the upcoming gamification design and implementation. The gamification designer who carried out the gamification design workshops and designed the gamification implementation, had experience from operating in several fields such as geosocial networking services, e-health e- commerce, education, retraining, and human resource management. The designer had a background in information systems and human- computer interaction, has executed over 40 gamification implementations to various extents - from smartphone applications to operator workflow in production plants. The designer had lectured on gamification in higher education, private institutions, presented at various gamification industry conferences, and had been ranked in the top 20 most influential thought leaders in gamification in several lists over the last decade. S/he defined gamification as a design practice within the user experience (UX) discipline concerning behavior science and motivational psychology as much as games. The gamified platform included the following game elements: mission, accomplishments, experience points, level, a progression board and notifications about achieving experience points. Experience points were gained for completing specific behaviors included in missions and accomplishments Individual data for each student consisting (Table 1) as well as for in other ways interacting of information on gender, age, final grade, with the platform. The level was study-pace, course, subject and if the student shown in the top left-hand corner and when was in the gamification group or the control pressing on the level icon the experience points group was provided by the LMS provider. The needed for reaching the next level were shown. final grade variable was divided into three variables; performance, measured by the grade achieved at the end of the course; pass-rate, towards the gamification design were derived measured by the students who had achieved a grade; and drop-out rate, measured by the would prefer further courses to have the same number of students who had discontinued the LMS design (using elements such as Levels and course. In the performance variable, the students could get the grade A, B, C, D, E, F or you would recommend the course to a friend II (insufficient information to set a grade), with A being the highest grade (coded as 7) and II -Point response scale being seen as the lowest grade (coded as 1). The was used, where one symbolized not at all students who discontinued the course were not likely and ten symbolized very likely. The included in the grade category. Drop-out rate survey was not the main focus for the study, but was coded as a binary variable of either instead used as complementary descriptive data continuing the course (0) or discontinuing the to understand the students perception of the course (1). The pass-rate variable was defined implementation. as the number of students from the start who received a passing grade (A, B, C, D, or E) and coded as either passing (1) or not passing (0) the course. SPSS version 26 was used for running all the tests. The research questions intended to To answer research question one (RQ1), the determine the effect of the gamification implementation on student performance (RQ1), effect of gamification on student performance, a Mann Whitney test was run with performance drop-out rate (RQ2) and pass-rate (RQ3). In the as a dependent variable. Since the dependent Mann Whitney test, run to determine if gamification had an effect on performance variable was ordinal, the corresponding (RQ1), the student in the gamified condition parametric test was not run. In answering had a higher mean rank (538.69) compared to research question two and three, the effect that gamification has on drop-out rate (RQ2) and the control group (525.9). However, the effect pass- -square tests on performance was not significant, U=144820, z=0.738, p=0.461, r=0.0175. For research were run due to the dependent variables being question two, the chi-square test for measuring categorical. The alpha level to determine significance was set to 0.05 in all of the tests drop-out rate showed that the gamification run. implementation did not have a significant effect on not dropping out of the course 2 (1) = 0.08, p=0.783. Finally, the chi-square test for measuring pass rate showed that the effect of gamification on passing the course was not A voluntary user-survey was prepared and significant 2 (1) = 0.05, p=0.944. distributed throughout the 19 courses that had The gamified and control condition slightly implemented gamification to complement the varied in means regarding performance, drop- quantitative data and to investigate the out-rate and pass-rate. However, none of the tests run produced a significant difference gamified LMS. Only the students assigned to depending on the gamified or control condition the gamified LMS received the survey, of (Table 2). Due to the large sample size in the which 321 answered. The survey was sent to the gamified-condition (n=909) and in the control students at the end of the courses. It included group (n=871) together with the high p-values demographic questions and questions about in each test (Table 3), the results indicate that how the gamification features in the LMS were perceived. Of the students who answered the survey, 63.6% were women, 34.9% were men, and 1.6% defined themselves as other, non- binary, or did not want to disclose gender. The respondents age ranged from 17 to 54 years (M = 27.73, s = 7.83). gamification in the present case did not have an effect on neither performance (RQ1), drop-out rate (RQ2) nor pass-rate (RQ3) for the students participating in the courses. Several follow-up tests were conducted to determine if there were any differences depending on gender, study pace or subject. However, no significant performance, pass-rate or drop-out-indicators could be identified. Despite not seeing significant results for performance, drop-out-rate and pass-rate, the students reported positively toward wanting other courses to incorporate the game elements level and accomplishments (question 1) and to recommending the course to others due to the design (question 2). In both questions there was a clear positive skew. In question one (Figure 1) the average answer was 8.36 with a clear majority of the students (n=145, 45%) reporting the highest answer. Similar to question one, in question two (Figure 2), the average answer was 8.09 with most students reporting the highest answers (n=126, 36%). The results show that gamification did not have a significant effect on performance, drop- out rate or continuation of courses in an online learning environment for adult students partaking in distance education. One limitation The results of the study contradict previous in the research design is that the testing was not findings of positive effects on drop-out rate and performed in a controlled setting and that performance of gamification effects in online education. Compared to the literature reviews could have occurred due to a high technological mentioned above, which found positive effects focus compared to design focus in the of most gamification studies in MOOCs and development process. Instead of asking: how online education [7, 8, 17, 18], the present study can the technology respond to design did not identify similar effects. This despite the considerations, the design seems to have been implementation development being supervised derived from; how can the design respond to the by a designer with senior experience as well as technological barriers? This is in line with the precautionary measures being employed in the time and funding barriers that have previously design process to understand the contextual been identified related to the gamification of factors of the implementation. Compared to MOOCs [19]. Economic barriers, time other studies involving gamification in a more constraints and technological limitations are controlled setting and with more course specific often prevalent in an industry context with design [15], this case involved a design that was limited resources making it difficult to put the not adapted to a specific setting but was instead design efforts in center. Here, the gamification expected to work due to the involvement of the system was limited to what the students game elements in the course per sei. However, reported and did in the LMS, inhibiting linking the results showed that the gamification design the gamification progress to course specific was not sufficient to produce the desired progress. To realize and operationalize progress outcomes, implicating the need for a greater connected to behaviors that could enhance consideration to what makes gamification learning for students, the courses would need to work. In accordance with previous suggestions, comprise of progress related activities that gamification does not work in itself but needs could easily be tracked in the LMS. However, to be designed in a mindful way from the user this would have gone beyond the feasible perspective and involve an iterative design boundaries of the project. process [26]. Instead of assuming that Another important distinction from previous gamification works simply by the application of studies is that the teaching method in the LMS game elements, the inner workings of how the was more similar to analog classrooms than to user interacts with the system needs to be regular MOOCs. I.e. the courses where teacher- central in the design process [27]. The results centered with long lessons instead of including from the present study indicates that this is true interactive digital material. Gamification was for distance education as well. implemented without changing the teaching The reasons for the gamification style and the course format. A potential reason implementation not producing intended effects for the change of course design not being in light of the context and design could be due considered is the belief that the game elements to several factors. To understand the in themselves are motivating and drive results with regards to previous research and behavior, without considering the learning and indications of positive effects in gamification motivation theories behind them. As previous implementations, further implications are studies have suggested, other fields related to derived from analyzing the structure of the gamification should be considered in the design platform, and the gamification design as well as of gamification implementation [14]. Whereas how this differs from other studies. One of the research implementing gamification to study its most prevalent differences was that effect is often embedded in an academic context gamification was not only applied to several with access to researchers from several fields, courses but also adapted to the behaviors that the same is not necessarily true for gamification gamification could respond to on a general level practitioners. This was prevalent in the current instead of a course specific level. Instead of design where important aspects from basing reward systems on the course progress educational and motivational psychology were and positive behaviors that could help a student not included. No underlying theory had been in reaching his or her course goals, activity- used as a basis for the design. Neither could related activities such as logging-in, visiting a specific psychological mechanisms that are site and booking exams were included. No clear commonly used in gamification design be connection could therefore be found between identified in the design. As other research completing accomplishments and missions and suggests, the reason why game elements are progressing in level with the actual course successful in games is due to the motivational progress and performance in the course. This mechanism being triggered. For example, Landers et al. [28] explains the success of results. There is also a need for more rigor in gamification through the application of the research design connected to gamification motivation theories such as self-determination in online education, where the effect of theory, operant conditioning, expectancy gamification can be isolated. In reviewing theory and goal setting theory. Similarly, previous works several studies too hastily Deterding [29] proposes that the success in attribute effects to gamification, i.e. in a study games is derived from achieving autonomy, conducted by Vaibhav and Gupta [30] it was competence and relatedness as proposed by concluded that gamification increased self-determination theory, and that the success completion and pass-rate. However, the control of gamification lies in the application of the condition consisted of students accessing a same motivation mechanisms. paper with the material whereas the gamified condition consisted of students accessing an online platform with game elements, making the comparison not only about gamification but also about analog compared to digital learning. Another reason for the high number of positive When looking at the engagement measures results presented previously, could be in the present study, without considering the publication bias, increasing the number of performance, pass-rate and drop-out measures, studies showing positive effects of one could falsely draw the conclusion that the gamification. To gain a reliable understanding gamification implementation had been of gamification and its effects it is also successful and did not need to be optimized. important to study and understand gamification However, as seen in the results, the engagement implementations that do not reach expected of students measured in their perception of the outcomes. For the field of gamification to move gamification design does not guarantee that the forward and become a successful learning tool students were more likely to continue the in online education, more studies are needed to course nor more likely to learn more. These address and problematize why gamification results challenge the assumption made by does not always work on an empirical level as Aparicio et al. [16] suggesting that the success well. Finally, more practical frameworks for of gamification can be measured through how to successfully apply gamification in an engagement. Instead of viewing gamification as industry setting are needed, to broaden the field a means to increase engagement towards the of gamification beyond academia and create LMS, further research needs to consider successful implementations that lead to engagement in terms of engagement towards intended effects. the course material and the subject being learned. Even though students might appreciate game elements [13], this does not imply that gamification is successful in creating learning engagement that leads to lower drop-out-rate Contrary to previous studies in gamified and higher performance and pass-rate. MOOCs and distance education, the present Therefore, it is important for both researchers study did not find an increased performance, and practitioners to clarify the main objective pass-rate and decrease of drop-rate when with gamification implementations; to make the gamification was implemented into an LMS for course more appealing and make more students adults participating in distance education. The want to join, or to create a gameful design that findings highlight that the design and context engages and motivates students toward course are crucial to consider when implementing completion and higher performance. The gamification in industry settings. This calls for present case suggests that the latter requires more practical frameworks on how to apply more from the design and implementation in gamification and more empirical research that terms of course design, technological flexibility questions the effect of gamification. Moreover, and knowledge in the underlying motivational this study found that students were positive to a mechanisms behind gamification. gamification implementation despite it not In line with Rohan et al. [15], we leading to intended effects, which implies the recommend more studies to include large need for a more nuanced understanding of sample sizes in order to determine reliable engagement measures in gamification research. review of empirical studies on Proceedings of the Annual Hawaii International [1] Conference on System Sciences, 2014, Challenges and Opportunities for pp. 3025 3034, doi: Procedia - 10.1109/HICSS.2014.377. Soc. Behav. Sci., 2012, doi: [10] C. Dichev and D. Dicheva, Gamifying 10.1016/j.sbspro.2012.05.408. education: what is known, what is [2] D. F. O. Onah, J. Sinclair, and R. Boyatt, believed and what remains uncertain: a critical review, vol. 14, no. 1. International Journal of Educational EDULEARN14 Proc., 2014. Technology in Higher Education, 2017. [3] S. Subhash and E. A. Cudney, [11] education: motivational information systems: A Int. J. Comput. Human Behav., vol. 87, no. Inf. Manage., vol. 45, no. June 2017, pp. February, pp. 192 206, 2018, doi: 191 210, 2019, doi: 10.1016/j.chb.2018.05.028. 10.1016/j.ijinfomgt.2018.10.013. [4] N. Limantara, A. Nizar Hidayanto, and [12] C. M. Chee and D. H. Ten Wong, Gamification Learning in Higher Fail Gamification in Education: A Education: a Systematic Literature IETE Tech. Rev. (Institution Int. J. Mech. Eng. Technol., Electron. Telecommun. Eng. India), vol. vol. 10, no. 02, pp. 982 991, 2019. 34, no. 6, pp. 593 597, 2017, doi: [5] 10.1080/02564602.2017.1315965. gamification improve student learning [13] J. Renz, D. Hoffmann, T. Staubitz, and outcome? Evidence from a meta- analysis and synthesis of qualitative ACM data International Conference Proceeding Educational Research Review. 2020, Series, 2016, doi: doi: 10.1016/j.edurev.2020.100322. 10.1145/2883851.2883876. [6] S. Deterding, D. Dixon, R. Khaled, and [14] M. Khalil, J. Wong, B. De Koning, M. to gamefulness: Defining MOOCs: A review of the state of the Proc. 15th Int. Acad. IEEE Global Engineering MindTrek Conf. Envisioning Futur. Education Conference, EDUCON, Media Environ. MindTrek 2011, pp. 9 2018, doi: 15, 2011, doi: 10.1109/EDUCON.2018.8363430. 10.1145/2181037.2181040. [15] R. Rohan, D. Pal, and S. Funilkul, [7] gamification - A service marketing Right Direction?: A Systematic Proceedings of the 16th ACM International Academic MindTrek International Conference Proceeding Conference 20 Series, 2020, doi: , 10.1145/3406601.3406607. 2012, pp. 17 22, doi: [16] M. Aparicio, T. Oliveira, F. Bacao, and 10.1145/2393132.2393137. M. Pai [8] R. Alomari, I; Al-Samarraie, H; Yousef, determinant of massive open online Inf. Manag., in Promoting Student Learning: A 2019, doi: 10.1016/j.im.2018.06.003. J. Inf. Technol. [17] M. Jarnac de Freitas and M. Mira da Educ. Res., vol. 18, pp. 395 417, 2019, doi: 10.28945/4417. Open [9] J. Hamari, J. Koivisto, and H. Sarsa, Learn., 2020, doi: - A literature 10.1080/02680513.2020.1798221. [18] J. Looyestyn, J. Kernot, K. Boshoff, J. Learning Tools? A Review of Source J. Educ. gamification increase engagement with Technol. Soc., vol. 21, no. 2, pp. 74 84, 2018, doi: 10.2307/26388380. PLoS One, 2017, doi: [25] J. Looyestyn, J. Kernot, K. Boshoff, J. 10.1371/journal.pone.0173403. [19] Y. An, M. Zhu, C. J. Bonk, and L. Lin, gamification increase engagement with practices, and perceived support needs 2017, doi: and barriers related to the gamification 10.1371/journal.pone.0173403. J. Comput. High. Educ., [26] B. Morschheuser, J. Hamari, K. Werder, 2020, doi: 10.1007/s12528-020-09256- o Gamify? A Method w. Proc. [20] A. Antonaci, R. Klemke, and M. Specht, 50th Hawaii Int. Conf. Syst. Sci., no. Hicss, pp. 1298 1307, 2017, doi: learning environments: A systematic 10.24251/hicss.2017.155. Informatics, vol. 6, [27] A. Shahri, M. Hosseini, K. Phalp, J. no. 3, pp. 1 22, 2019, doi: 10.3390/informatics6030032. gamification: The consensus, the best [21] O. Borrás-Gené, M. Martínez-Núñez, and L. Martín-Fernández, 10.4018/JOEUC.2019010103. fun through gamification to improve [28] R. N. Landers, K. N. Bauer, R. C. Informatics, Callan, and M. B. Armstrong, 2019, doi: 10.3390/informatics6030028. [22] L. M. Romero-Rodriguez, M. S. Gamification in Education and Ramirez-Montoya, and J. R. V. Business, 2015. [29] Engagement Application Test in Energy IEEE Access, Human-Computer Interact., vol. 30, no. 2019, doi: 3 4, pp. 294 335, 2015, doi: 10.1109/ACCESS.2019.2903230. 10.1080/07370024.2014.993471. [23] M. Krause, M. Mogalle, H. Pohl, and J. [30] of MOOCs for increasing user Fostering student retention in online Proceedings of the 2014 IEEE International Conference on L@S 2015 - 2nd ACM Conference on MOOCs, Innovation and Technology in Learning at Scale, 2015, doi: Education, IEEE MITE 2014, 2014, doi: 10.1145/2724660.2724665. 10.1109/MITE.2014.7020290. [24]