Exploring the connection between gamification and student engagement in computer-supported collabo- ration

Abstract

Collaboration platforms and online forums often employ gamification in order to produce a better end-user experience. However, designing for gamified elements can be challenging, especially in the education context of learning platforms: In order to support online learning activities, one must carefully deliberate which type of activities can be enhanced with gamified design. In this paper, we investigate the effect of gamification in relation to activities on a collaborative online platform. Specifically, we use the partial least squares path modelling method to determine the effect of gamification in relation to three different collaborative learning activities: 1) seeking help, 2) helping other students, and 3) receiving help. According to our analysis, gamification has a positive impact on students engaging in beneficial collaborative behaviour, such as asking questions and helping others by answering. Furthermore, gamification has an indirect impact on performance, which was measured by the students receiving help on the platform.

Keywords

• gamification
• computer-supported collaborative learning
• student engagement
• path modelling

Introduction

Collaborative learning is a learning method where students have a symmetry of action, knowledge and status, and have a low division of labor [ 11 ]. Computer-supported collaborative learning (CSCL) facilitates the interaction with software tools and increases the potential for creative activities and social interaction [ 47 ]. Furthermore, CSCL is an approach used in online environments such as Massive Open Online Courses (MOOCs) to provide mutual learning opportunities, often implemented through a discussion forum as a part of a ‘social learning’ strategy [ 16 ].

However, in order for the students to benefit from the collaboration, they first have to be engaged as users of the platform. In recent studies, it has been shown that students can be guided towards educational goals like collaboration by using gamification [ 35 ], which is the intentional use of game elements for a gameful experience of non-game tasks and contexts [ 9,27,45 ]. Furthermore, gamification approaches have had success in engaging users of online programs [ 30 ] and engaging students in online discussions [ 12 ].

Points, badges, and leaderboards are one of the most often used gamification elements [ 45 ] and are being used in successful online collaboration platforms, such as Stack Overflow and Quora. The elements have faced critique for being rather simple and overused [ 34 ], with authors calling for more holistic design of gamified systems [ 7,8,34 ]. However, according to earlier research [ 32 ] those gamification elements have had an impact on performance, if not on motivation. Our research goal is to further explore the connection between users engaging in collaborative behaviour and the most commonly used gamification features of online collaboration forums.

To realize our goals, we explored a previously collected survey dataset [ 29 ] from an introduction to programming course using the Partial Least Squares Path Modelling (PLS-PM) research method [ 20 ], which has also been used in previous gamification research [ 3,22 ]. We used the method to create a path model to evaluate hypotheses between students engaging with gamification features, other engagement behaviours, and collaborative outcomes.

In the next section, we review theory and present our hypotheses. In Section 3, we present the research context and data analysis. In Section 4, we present our findings. The paper concludes with Section 5. 2 2.1

Theory and hypotheses

Gamification in collaborative learning

Collaborative learning fosters the development of critical thinking through interactions such as discussion, clarification of ideas, and evaluation of others’ ideas in both traditional classroom and online programs [ 14,19 ]. To stimulate engagement and influence the learners’ activities within the collaborative learning, gamification has been considered as one possible approach through the inclusion of game-like features such as points and badges, in non-game contexts [ 30 ]. However, Looyestrn et al. [30, p. 1] point out that ”engagement in online programs is difficult to maintain” and performs a systematic review to understand “Are gamification strategies effective in increasing engagement in online programs?”. In the review, [ 30 ] examined a variety of forms of gamification: leader boards, badges, points and rewards, and commonly in combination. The authors claim that leader boards as a gamification feature are particularly effective compared to extrinsic rewards such as badges and points which tend to wear off after a short period of novelty. The study concludes that gamification can increase engagement and downstream behaviors, and enhance related outcomes (e.g. health behaviour and academic performance) in online programs. Other authors, for example, Ding et al. [ 12 ], applied a gamification approach and conducted two trial studies in the online discussion tool, gEchoLu. The study concludes, gamification is still in its infancy stage and students may have limited experiences with it. For gamification to be successful, i) every game element of gamification should be meaningful to students; ii) instructors should serve as facilitator.

Fundamentally, gamification aims to improve both user engagement and user experience [ 9 ]. The different empirical studies concerning the effect of gamification were examined in a literature review by Hamari et al. [ 21 ], where the authors create a framework for examining the users’ motivational affordances, along with psychological and behavioral outcomes of gamification. The study by Hamari et al. [ 21 ] stresses the role of context in which gamification is being applied, and also users’ qualities. The authors state that many previous studies do not take into account the complexity of the phenomena they observe: Gamification can be effective, but the effects are ”greatly dependent on the context in which gamification is being implemented.” 2.2

Engagement and online discussion

Student engagement is often defined by three components: behavioural, as evidenced through active engagement in learning tasks; cognitive, as evidenced through learning; and emotional, as evidenced through the level of interest [ 1 ]. Of these, the most commonly used within online learning environments (and the easiest to capture at scale) is the behavioural component, as indicated by a broad spectrum of online actions [ 37 ]. More focused research has looked at how students engage in discussion forums. Once seen as an extra to learning activities, forums are increasingly being recognised as an important part of a ‘social learning’ approach, and are used by MOOCs such as Futurelearn to support learning steps [ 16 ]. Participation in forums can be shown to lead to improved learning outcomes [ 40 ], with both student-student and instructor-student communication being important [ 13 ]. 2.3

Research model and hypotheses

In this section, we present our research model and then present our list of hypotheses.

Our research model is based on four constructs, which are all based on behavioural engagement [ 1 ] with different platform features. The constructs are based on engagement with (1) gamification features, (2) asking questions, and (3) answering questions. We define performance as the number of times the forum user has received help. The survey is not based on a pre-published instrument but does have similarities to engagement measures used by Shin [ 46 ]. The research model and the hypotheses are presented visually in Fig. 1 and the hypotheses with their rationale are listed as follows.

Gamification has been shown to increase engagement with online platforms [ 45,30 ] and in MOOCs [ 18,48 ]. From this, we derive hypotheses H1 and H2 that forum users engaging with gamification features has an impact on increasing behavioural engagement with collaborative features, operationalized as initiating and responding to discussions. H1. Engaging with gamification features positively influences asking questions on the platform.

H2. Engaging with gamification features positively influences answering questions on the platform. Participation in course forums has been shown to lead to improved learning outcomes [ 40 ] and increased collaboration [ 29,30 ]. In this study, we evaluate if the increased collaborative activity on the forum increases collaborative outcomes, defined as instances of received help on the forum (H3a, H3b). Furthermore, we evaluate if the impact of gamification on outcomes (H4) is mediated by collaborative activity in the form of asking or answering questions.

H3a. Asking questions and seeking for answers on the platform has a positive influence on receiving help on the platform.

H3b. Answering questions on the platform has a positive influence on receiving help on the platform.

H4. Engaging with gamification features indirectly increases the chances of positive collaborative outcomes, or receiving help on the forum.

Research context and methods

The study was conducted for the duration of a fourteen week long introduction to programming course that had 249 participating students, four assistants helping with the exercises and one lecturer ultimately responsible for the course. Most of the participants were first year university students. The course had weekly two-hour lectures, exercises of the same duration and weekly assignments that had to be returned through an online grading system. The grading depended on the return of the weekly exercises, one larger exercise project, and a final exam. Additionally, lecture or online activity was rewarded with activity bonus points.

The course has online support materials like a programming guide, lecture notes, and video lectures. This year an online asynchronous collaboration system with gamification elements was added to the course 3. The use of the system was taught on the introductory lecture of the course and the assistants also encouraged its use over questions by email, in order to have other participants to the course benefit from the discussion.

The discussion system’s primary feature is the possibility for participants to post questions or issues regarding the course and for others to participate in discussions. The somewhat novel element that separates this discussion system from basic online forums is the aspect of gamification. The system recognizes and rewards people who constructively contribute with answers, comments or just ask good questions. Other users of the systems can upvote or downvote content of the system, further rewarding and encouraging other contributors. The users can also publicly show the points and achievement badges collected in the system. In essence, the system is much like Stack Overflow.

An earlier study analyzed the same course’s student behaviour from a social network analysis perspective [ 29 ] using recorded system interactions. The study also collected a survey about user’s engagement behaviour, which could benefit from further in-depth analysis, which we present in this paper. The students were surveyed about their use of the system, including how often they had engaged with gamification features, asking and answering questions, and from how many other students they had received help. In the survey, 73 users replied, 61 of whom had used the Q2A discussion system and 12 students who hadn’t.

Survey items are presented in Table 1. Survey items in categories GAM, ANS, ASK asked ”how often” and answers were on a five-step Likert scale of ”never” to ”more than once per day.” Survey item REC1 asked ”how many” on a scale of one to five, with one denoting one and six denoting more than five. Engaging with the gamification system’s features is operationalized by asking participants how involved they were with checking up their points status and voting, which is essentially like awarding points to other users. The archival nature of the data imposed some limitations on how the constructs were operationalized, such as measuring some constructs with only one variable and not having more diverse variables for the ”engaging with gamification” construct. 3.1

Data analysis

We analyzed the research model with a path modelling approach. Path modelling allows specifying a visual research model with multiple theory-based constructs and defining paths between them that represent hypotheses of variable or construct relationships [ 20 ]. We selected the PLS-PM statistical modelling method with non-parametric bootstrapping to cope with slight non-normality of the observed variables and low sample size (for a path modelling approach) [ 20,23,24 ]. Furthermore, PLS-PM is an appropriate method to use because of the limited sample size, the research is based on secondary or archival data, and the path model includes formatively measured constructs [ 44 ]. We applied the SmartPLS 3 software package for data analysis [ 39 ]. Our sample size was 61, which is enough for PLS-PM both according to the rule of thumb of having 10 times per largest number of paths from independent variable going into a dependent variable [ 20 ] and 3 http://www.question2answer.org using a power analysis guidelines adapted by Hair et al. [ 20 ] from [ 4 ] at 5% significance level.

Specifying and validating the measurement model

In our model, we used both reflective and formative constructs [ 20 ]. This causes some considerations, as not all validity measures apply to formative constructs [ 20 ]. Assessing convergent and discriminant validity using criteria similar to those associated with reflective measurement models is not meaningful when formative indicators and their weights are involved [ 2,20 ]. For formative and reflective constructs we validated the (1) measurement reliability and (2) model structure. For reflective constructs, we additionally validated the (3) discriminant validity. Furthermore, we assessed nomological validity, according to which the relationships between the constructs in the path model, sufficiently well-known through prior research, should be strong and significant [ 26 ]. The survey was not based on existing measurement scales or instruments, which is its main weakness, but after assessment, we decided that it is of sufficient nomological quality to proceed with analysis.

The measurement reliability was assessed with composite reliability (CR) and average variance extracted (AVE) indicating convergent validity. The measurement reliabilities are reported in Table 2. All of the convergent validity metrics were greater than the thresholds recommended in literature (CR > 0.7; AVE > 0.5) [ 17,24 ]. Some constructs are composed of only one variable because of the archival nature of the data, which prevents the evaluation of their convergent validity. The fields where evaluation was not possible are marked with N/A. However, other quality metrics were high enough that we proceeded with analysis. Furthermore, single variable constructs are possible in PLS-PM, especially in cases where the variable is a concrete, observable item, even though their use decreases construct reliability [ 20 ]. It should be noted that because of this issue the data should be considered only an initial, exploratory investigation instead of full-fledged theory testing.

We analyzed the structure of the measurement model, where applicable, by significance and weight of factor loadings, and for cross-loadings between the latent factors. All loadings in the outer model (measurement model) were significant and varying from acceptable .534 to good .946, indicating valid model structure.

We assessed discriminant validity of the measurement model, firstly, by the square root of AVE (i.e. Fornel-Larcker criterion [ 17 ]), where all of the AVE square roots should be greater than the squared latent variable correlations. Secondly, we verified that all item loadings exceed cross-loadings [ 24 ]. The results indicate good discriminant validity of the measurement model. We did not use the heterotrait-monotrait ratio of correlations (HTMT) method, as the use of some of the validity methods is still unclear for formative or single variable constructs [ 20 ].

Validating the structural model

We used a bootstrapping method to evaluate the coefficients for their significance [ 5 ], which is the recommended method for PLS-PM [ 20 ]. The bootstrap sample size was n = 61, and resampling was performed 5000 times, which are parameters following best practises in literature and are sufficient to evaluate the model [ 20,24 ]. We tested and validated the quality of the structural model representing our hypotheses by evaluating (1) collinearity issues and overall fit, (2) explanatory power, and (3) path significances.

We assessed the collinearity and the model fit in order to validate the structural model and identify misspecification problems. VIF (variance inflation factor) of the latent constructs did not indicate collinearity issues with values clearly between recommended values (0.2 < VIF < 5) [ 20 ]. Because of the hypothesis testing objective of the paper, we further assessed the overall fit of the structural model to the data in order to analyze whether the model was specified correctly. For that purpose, we used the standardized root mean square residual quality metric (SRMR < .08) [ 24 ] to evaluate estimation error and misspecification of the model [ 20 ]. In our case, the model fit of SRMR = .106 indicates that serious misspecification of the structural model does not occur.

The R2 for the latent variables in the path model varies from ”answering” = 0.53 to ”received help” = 0.35, which range from substantial to moderate according to guidelines interpreted by Henseler et al. [ 25 ] from [ 2 ], indicating acceptable explanatory power of the inner path model structure. If a good proportion of the variance is explained, the explanatory power of the model can be high regardless of the relatively low sample size [ 38 ].

We present the path significances and hypotheses testing results in the next section. 4

Findings

We analyzed direct and indirect effects in the model, which are defined by hypothesis one to four and summarized Table 3. The path model could account for 36% of the variance in receiving help, 34% of the variance in asking questions, and 54% in answering questions. The model shows that engaging with gamification features has a strong influence on engaging both in asking and answering behaviour (H1 and H2). Furthermore, engaging in asking behaviour affects receiving help (H3a). Engaging in answering behaviour does not increase the chances of receiving help on the platform (H3b).

In order to test H4, we evaluated the mediation effects following a process by Zhao et al. [ 49 ], which is also proposed by Hair et al. [ 20 ] for use with PLS-PM. First, we established that a minimum of one of the mediating effects is significant, with a path from the gamification (GAM) construct to the received help (REC) through asking questions (ASK), as displayed in Table 3. The second mediating effect through answering questions (ANS) is not significant. Furthermore, the direct effect of GAM on REC is not significant ( = 0:237, p = 0:054). After testing for and excluding direct and alternative mediating effects, we conclude that GAM has an indirect effect on REC through only one construct, ASK, and is fully mediated by it (H4). The path coefficient and significance p presented for H4 is the specific indirect effect through the ASK construct. To summarize, engaging with gamification has an impact on receiving help only when it affects asking questions behaviour. n) not significant, *) Statistically significant at p<0.05, **) Statistically significant at p<0.01, ***) Statistically significant at p<0.001

Discussion and conclusion

In this paper, we explored the connection between students engaging with gamification features and engaging in computer-supported cooperative learning. The results indicate that engaging with gamification is positively associated both with desired behaviour and positive outcomes on the collaborative discussion platform. Previous research has established that gamification can increase engagement with software systems [ 36,45,30 ] and activity in online discussion [ 12 ]. In this study, we additionally demonstrate that gamification increases engagement and this is connected to participants receiving help in the forum. To summarize, students asking for help receive it and students who do not need help are engaged in providing answers, with the activities supported by gamification features.

The connection between gamification and increased collaboration in forum-based CSCL systems suggests that asynchronous discussion systems structured in the manner of Stack Overflow provide a valuable addition to courses on technical subjects. Here our findings match the findings in other contexts on social learning, including the studies by Ferguson et al. [ 16 ] on social learning in MOOCs and Romero et al. [ 40 ] on online discussion forums. Furthermore, our findings imply that gamification is not only a useful feature in supporting engagement in online learning environments [ 30 ], but can also support beneficial outcomes, such as increased collaboration. Our research provides some insight into mechanisms that can foster greater collaboration and social learning opportunities. This is of value because gamification is a desirable feature in MOOC platforms [ 28 ], but one that can be misunderstood or misapplied [ 10,15 ]. By providing new evidence on how gamification affects phenomena present in collaborative and social learning, the findings of this study can assist in the design of future social learning features in MOOCs and other online educational environments.

Our main contribution to state of the art is confirming a connection between engaging with a gamification system and collaborative outcomes. While this study did not explore the underlying causes of effective gamification, the forum system follows design reflections found essential in other studies: 1) intrinsic integration, where the gamification system (e.g. content voting and points) should be an intrinsic part of main activities [ 8 ]; 2) facilitation [ 12 ]; and 3) generally supporting Deci and Ryan’s [ 6 ] self-determination theory’s competence, relatedness, and autonomy factors from an education perspective [ 41 ]. The connection between self-determination theory and the design of the case study are discussed further in [ 29 ].

Another contribution of our study is confirming the findings by Mekler et al. [ 32 ] using a different research method and a new context, providing evidence on the effectiveness of points in the context of CSCL. While not a replication study, it is still valuable, as more authors are calling for important findings in education research and beyond to be replicated [ 31,33 ], and more in the wild studies to be implemented on gamification [ 42 ]. Mekler et al. did take their study to a more advanced level by also exploring the effects of gamification elements at an individual level and including motivation, as defined in Deci and Ryan’s self-determination theory [ 43 ], in addition to performance.

While we provide new additions to the scientific body of knowledge on gamification, engagement and CSCL, we acknowledge some limitations. The main limitation is that the model has minor imperfections because of the use of archival data and this has also limited the number and granularity of constructs in our research model. However, the constructs and our hypotheses were solidly based on earlier literature, and the results supported the hypotheses. As future work, we agree with earlier publications [ 32,42 ] that point out the need for more in the wild studies with detailed, theory-driven models that include motivation-based constructs that provide more understanding to connections between diverse types of gamification, motivation, and outcomes.

Acknowledgements

We thank our colleague Mika Immonen for sharing his knowledge on the PLS-PM research method.

References

1. Axelson , R.D., Flick , A. : Defining student engagement . Change: The magazine of higher learning 43(1) , 38 - 43 ( 2010 )

2. Chin , W.W.: The partial least squares approach to structural equation modeling . Modern methods for business research 295(2) , 295 - 336 ( 1998 )

3. Codish , D. , Ravid , G.: Academic course gamification: The art of perceived playfulness . Interdisciplinary Journal of E-Learning and Learning Objects 10 ( 1 ), 131 - 151 ( 2014 )

4. Cohen , J.: A power primer . Psychological bulletin 112(1) , 155 ( 1992 )

5. Davison , A.C. , Hinkley , D.V. , et al.: Bootstrap methods and their application , vol. 1 . Cambridge university press ( 1997 )

6. Deci , E.L. , Ryan , R.M. : Motivation, personality, and development within embedded social contexts: An overview of self-determination theory . The Oxford handbook of human motivation pp. 85 - 107 ( 2012 )

7. Deterding , S. : Eudaimonic Design, or: Six Invitations to Rethink Gamification . SSRN Scholarly Paper ID 2466374 , Social Science Research Network, Rochester, NY (Jul 2014 )

8. Deterding , S.: The Lens of Intrinsic Skill Atoms: A Method for Gameful Design . Human-Computer Interaction 30 ( 3-4 ), 294 - 335 (May 2015 )

9. Deterding , S. , Dixon , D. , Khaled , R. , Nacke , L. : From Game Design Elements to Gamefulness: Defining ”Gamification” . In: Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments . pp. 9 - 15 . MindTrek '11, ACM , New York, NY, USA ( 2011 )

10. Dichev , C. , Dicheva , D. : Gamifying education: what is known, what is believed and what remains uncertain: a critical review . International Journal of Educational Technology in Higher Education 14 ( 1 ), 9 ( 2017 )

11. Dillenbourg , P. : What do you mean by collaborative learning? Collaborative-learning: Cognitive and computational approaches 1 , 1 - 15 ( 1999 )

12. Ding , L. , Kim , C. , Orey , M. : Studies of student engagement in gamified online discussions . Computers & Education 115 , 126 - 142 ( 2017 )

13. Dixson , M.D.: Creating effective student engagement in online courses: What do students find engaging? Journal of the Scholarship of Teaching and Learning pp. 1 - 13 ( 2010 )

14. Faja , S. : Collaborative learning in online courses: Exploring students perceptions . Information Systems Education Journal 11 ( 3 ), 42 ( 2013 )

15. Falkner , N.J. , Falkner , K.E. : Whither, badges? or wither, badges!: a metastudy of badges in computer science education to clarify effects, significance and influence . In: Proceedings of the 14th Koli Calling International Conference on Computing Education Research . pp. 127 - 135 . ACM ( 2014 )

16. Ferguson , R. , Sharples , M. : Innovative pedagogy at massive scale: teaching and learning in moocs . In: European Conference on Technology Enhanced Learning . pp. 98 - 111 . Springer ( 2014 )

17. Fornell , C. , Larcker , D.F. : Evaluating structural equation models with unobservable variables and measurement error . Journal of marketing research pp. 39 - 50 ( 1981 )

18. Gene´, O.B. , Nu´n˜ez, M.M. , Blanco , A ´ .F.: Gamification in mooc: challenges, opportunities and proposals for advancing mooc model . In: Proceedings of the Second International Conference on Technological Ecosystems for Enhancing Multiculturality . pp. 215 - 220 . ACM ( 2014 )

19. Gokhale , A.A. : Collaborative learning enhances critical thinking . Volume 7 Issue 1 (fall 1995 ) ( 1995 )

20. Hair Jr , J.F. , Hult , G.T.M. , Ringle , C. , Sarstedt , M.: A primer on partial least squares structural equation modeling (PLS-SEM) . Sage Publications ( 2016 )

21. Hamari , J. , Koivisto , J. , Sarsa , H.: Does Gamification Work? - A Literature Review of Empirical Studies on Gamification . In: 2014 47th Hawaii International Conference on System Sciences (HICSS) . pp. 3025 - 3034 ( Jan 2014 )

22. Hamari , J. , Shernoff , D.J. , Rowe , E. , Coller , B. , Asbell-Clarke , J. , Edwards , T. : Challenging games help students learn: An empirical study on engagement, flow and immersion in gamebased learning . Computers in Human Behavior 54 , 170 - 179 ( 2016 )

23. Henseler , J. , Dijkstra , T.K. , Sarstedt , M. , Ringle , C.M. , Diamantopoulos , A. , Straub , D.W. , Ketchen Jr , D.J. , Hair , J.F. , Hult , G.T.M. , Calantone , R.J.: Common beliefs and reality about pls: Comments on r o¨nkko¨ and evermann ( 2013 ). Organizational Research Methods 17 ( 2 ), 182 - 209 ( 2014 )

24. Henseler , J. , Hubona , G. , Ray , P.A. : Using pls path modeling in new technology research: updated guidelines . Industrial management & data systems 116(1) , 2 - 20 ( 2016 )

25. Henseler , J. , Ringle , C.M. , Sinkovics , R.R.: The use of partial least squares path modeling in international marketing . In: New challenges to international marketing, pp. 277 - 319 . Emerald Group Publishing Limited ( 2009 )

26. Henseler , J. , Ringle , C.M. , Sinkovics , R.R.: The use of partial least squares path modeling in international marketing . In: New challenges to international marketing, pp. 277 - 319 . Emerald Group Publishing Limited ( 2009 )

27. Huotari , K. , Hamari , J.: Defining Gamification: A Service Marketing Perspective . In: Proceeding of the 16th International Academic MindTrek Conference . pp. 17 - 22 . MindTrek '12, ACM , New York, NY, USA ( 2012 )

28. Kasurinen , J. , Knutas , A. : Publication trends in gamification: A systematic mapping study . Computer Science Review 27 , 33 - 44 ( 2018 )

29. Knutas , A. , Ikonen , J. , Nikula , U. , Porras , J.: Increasing Collaborative Communications in a Programming Course with Gamification: A Case Study . In: Proceedings of the 15th International Conference on Computer Systems and Technologies . pp. 370 - 377 . ACM ( 2014 )

30. Looyestyn , J. , Kernot , J. , Boshoff , K. , Ryan , J. , Edney , S. , Maher , C. : Does gamification increase engagement with online programs? a systematic review . PloS one 12(3) , e0173403 ( 2017 )

31. Makel , M.C. , Plucker , J.A. : Facts are more important than novelty: Replication in the education sciences . Educational Researcher 43 ( 6 ), 304 - 316 ( 2014 )

32. Mekler , E.D. , Bru¨hlmann, F. , Tuch , A.N. , Opwis , K. : Towards understanding the effects of individual gamification elements on intrinsic motivation and performance . Computers in Human Behavior 71 , 525 - 534 ( Jun 2017 )

33. Moonesinghe , R. , Khoury , M.J. , Janssens , A.C.J.: Most published research findings are false-but a little replication goes a long way . PLoS medicine 4 ( 2 ), e28 ( 2007 )

34. Nacke , L.E. , Deterding , S.: The maturing of gamification research . Computers in Human Behavior 71 , 450 - 454 ( Jun 2017 )

35. Nah , F.F.H. , Zeng , Q. , Telaprolu , V.R. , Ayyappa , A.P. , Eschenbrenner , B. : Gamification of education: a review of literature . In: International Conference on HCI in Business. pp. 401 - 409 . Springer ( 2014 )

36. Nah , F.F.H. , Zeng , Q. , Telaprolu , V.R. , Ayyappa , A.P. , Eschenbrenner , B. : Gamification of education: a review of literature . In: International conference on hci in business. pp. 401 - 409 . Springer ( 2014 )

37. Onah , D.F. , Sinclair , J. , Boyatt , R.: Dropout rates of massive open online courses: behavioural patterns . EDULEARN14 proceedings pp. 5825 - 5834 ( 2014 )

38. Prentice , D.A. , Miller , D.T. : When small effects are impressive . Psychological bulletin 112(1) , 160 ( 1992 )

39. Ringle , C.M. , Wende , S. , Becker , J.M.: Smartpls 3. boenningstedt: Smartpls gmbh ( 2015 )

40. Romero , C. , Lo´pez, M.I. , Luna , J.M. , Ventura , S. : Predicting students' final performance from participation in on-line discussion forums . Computers & Education 68 , 458 - 472 ( 2013 )

41. van Roy, R. , Zaman , B. : Why gamification fails in education and how to make it successful: introducing nine gamification heuristics based on self-determination theory . In: Serious Games and edutainment applications , pp. 485 - 509 . Springer ( 2017 )

42. van Roy, R. , Zaman , B. : Unravelling the ambivalent motivational power of gamification: A basic psychological needs perspective . International Journal of Human-Computer Studies ( 2018 )

43. Ryan , R.M. , Deci , E.L. : Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being . American psychologist 55(1) , 68 ( 2000 )

44. Sarstedt , M. , Ringle , C.M. , Hair , J.F. : Partial least squares structural equation modeling . In: Handbook of market research , pp. 1 - 40 . Springer ( 2017 )

45. Seaborn , K. , Fels , D.I. : Gamification in theory and action: A survey . International Journal of Human-Computer Studies 74 , 14 - 31 ( Feb 2015 )

46. Shin , N.: Online learner's 'flow'experience: an empirical study . British Journal of Educational Technology 37 ( 5 ), 705 - 720 ( 2006 )

47. Stahl , G. , Koschmann , T. , Suthers , D. : Computer-supported collaborative learning: An historical perspective . Cambridge handbook of the learning sciences 2006 , 409 - 426 ( 2006 )

48. Vaibhav , A. , Gupta , P. : Gamification of moocs for increasing user engagement . In: MOOC, Innovation and Technology in Education (MITE) , 2014 IEEE International Conference on. pp. 290 - 295 . IEEE ( 2014 )

49. Zhao , X. , Lynch Jr , J.G. , Chen , Q. : Reconsidering baron and kenny: Myths and truths about mediation analysis . Journal of consumer research 37(2) , 197 - 206 ( 2010 )