Higher-education students' perceptions gamification in a learning management system

Abstract

To understand what makes gamification successful there is a need to study individual game elements over time. The aim of the present longitudinal case study was to investigate the game element points in a higher-education context, in relation to the hedonic and utilitarian aspects of a gamified learning management system and novelty effects. A scale was developed based on utilitarian, hedonic and motivational elements of the point system, uncovering two main components: perceived value of points, and perceived motivation of points. The results showed that the perception of the points was positively related to the hedonic perception of a gamified learning management system, and that novelty effects are present. Based on the results, we suggest that points in higher education should be designed with regards to how students perceive the value and motivational aspects behind them and with considerations of how novelty effects can be mitigated.

Keywords

• 1 Gamification
• hedonic dimension
• utilitarian dimension
• learning management systems
• selfdetermination theory
• higher education
• points

1. Introduction

Student engagement in higher education is associated with a vast set of positive educational outcomes [ 1 ]. One promising approach to enhance engagement is through gamification [ 2 ], the application of game elements in non-game contexts [ 3 ]. Game elements influence behavior by eliciting psychological outcomes such as motivation [ 4 ] and encourages users to reach external, utilitarian goals by engaging in enjoyable, hedonic experiences [ 5 ]. In higher education, gamification implementations have produced positive [ 6 ], non-significant [ 7 ] and negative [ 8 ] student outcomes. The mixed effects could be due to a limited understanding of the impact of individual game elements [ 4, 9 ], and the presence of novelty effects, generating shortterm engagement due to the novelty associated with new technology [ 4 ].

For gamification to promote positive outcomes in higher education, apart from considering the behavioral aspect, students’ user experience should also be taken into consideration [ 10 ]. In education, gamification is most commonly adopted through learning management systems (LMS) [ 11 ]. Previous research indicates that students’ hedonic and utilitarian perceptions of an LMS can predict usage [ 12 ] and acceptance [ 13 ].

Gamification has been found to influence the usage of an LMS [ 14 ]. Less is known about how individual game elements can contribute to the user experience, in terms of hedonic and utilitarian aspects, of an LMS, and how the perceptions of individual game elements changes over time. When considering the perception of individual game elements their nature of motivation drivers and hedonic and utilitarian providers should be considered [ 4, 5 ].

One of the most commonly game elements in higher education are points [ 6 ], utilized to measure and to reward students’ success [ 15 ]. Point systems are a foundational part of gamification since they enable designers to value and track behavior in a specific context [ 16 ], and function as a way to measure and reward behavior [ 17 ]. The present work intends to study how the perception of points relates to the hedonic and utilitarian aspects of an LMS in higher education, and how the perception of points change over time. The following two research questions will be explored:

RQ1: How does higher-education students’ perception of points relate to hedonic and utilitarian perceptions of a gamified LMS?

RQ2: How does higher-education students’ perception of points in a gamified LMS change over time? 1.1.

The perception of points

The use of points to increase motivation dates back long before the emergence of gamification. In the early 1960s, token economies were introduced to reinforce behavior by rewarding standardized and quantifiable tokens for desired behaviors [ 17 ]. With the introduction of technology, the token economy evolved into what Raczkowski [ 17 ] refers to as scoring economies, automated systems that in contrast to the token economy’s goal of increasing institutional efficiency, aimed at increasing individual efficiency by allowing users to measure themselves and improve their performance. Points in a learning context have been found successfully increase behaviors that are rewarded with points [ 18-20 ], while at the same time undermining student behaviors that are not rewarded with points [ 19, 20 ]. Students have reported points as one of the most motivational game elements [ 21 ], and have also described it as a useful game element in creating enjoyable experiences in gamified learning contexts [ 22 ]. Based on qualitative assessments of points in an educational context, students have expressed that they appreciate points being added as a value representation of different activities and selfassessment [ 23 ]. 1.2.

Gamification and motivation

In broad terms motivation involves being moved to do something [ 24 ]. Self-determination theory, a motivational theory often used to describe the motivational drivers behind game play [ 25 ], divides motivation into two subcomponents; internal motivation, referred to behaviors done for their one sake; and external motivation, referred to behaviors done for reasons beyond internal satisfaction [ 26 ]. Gamification based on promoting internal motivation intends to enhance users experience through an understanding of their innate psychological needs [ 27 ]. Basic psychological needs theory, a subtheory of self-determination theory, proposes that people have three innate psychological needs that must be fulfilled to become intrinsically motivated: autonomy (having freedom to choose and act from one’s own interests), competence (having the opportunity to be effective in one’s environment by getting support to practice, develop skills and improve) and relatedness (being able to form relations with others) [ 28 ]. Gamification based on external motivation relies on motivation theories of reinforcing conditioning, i.e., reinforcing behavior by associating stimuli and response [ 29 ], and operant conditioning, i.e., reinforcing behavior by pairing behavior with an expected reward [ 30 ]. To base gamification on external motivation has been criticized for negatively impacting internal motivation [ 31 ]. On the other hand, to base gamification on external motivation has been suggested to be effective for short term immediate effects and for learners who are not intrinsically motivated [ 32 ]. 1.3.

Hedonic and utilitarian benefits

For gamification to be used and accepted utilitarian and hedonic aspects need to be considered, i.e., creating enjoyable, hedonic experiences that support external, utilitarian objectives [ 5 ]. Game elements can provide utilitarian benefits, i.e., using an application to fulfil external objectives such as productivity, and hedonic benefits, i.e., using an application for its own sake [ 5 ]. The utilitarian benefits of gamified services relate to the perceived usefulness and perceived ease of use of the system, as proposed by the technology acceptance model [ 33 ], whereas the hedonic benefits relate to the enjoyment and playfulness of the gamified service [ 5 ]. Investigating the hedonic and utilitarian dimensions of a gamified fitness service, it was found that hedonic and utilitarian aspects both contributed to the attitude and continued use of the service [ 5 ]. On the utilitarian dimension perceived usefulness was positively associated with attitude and continued use of a gamification service and perceived ease of use was associated with continued use but not with attitude towards use. On the hedonic dimension playfulness was associated with continued use as mediated by attitude but did not show a direct relationship to either attitude or continued use, while enjoyment had a positive association to continued use but not to attitude. Previous research supports the notion that the inclusion of hedonic and utilitarian features in gamified services are positively associated with continued use [ 34 ] and enhancement of user experience [ 31, 32 ], which in turn is positively associated with attitude toward the gamified service [ 36 ]. Related to gamified learning, the influence of hedonic and utilitarian dimensions has been studied to a limited extent. In one study concerning the perceptions of a gamified learning platform it was found that both utilitarian and hedonic aspects contributed to use of the platform, with utilitarian aspects being the main motivational driver for students and hedonic aspects amplifying the utilitarian aspects [ 37 ]. Furthermore, it was found that students differed in their motivational drivers; some students commented on being motivated by utilitarian aspects and others by hedonic aspects [ 37 ].

Based on the positive effects of the hedonic and utilitarian dimensions on the service in which gamification is implemented we hypothesize that:

H1: A more positive perception of points will be positively related to the hedonic perception of an LMS.

H2: A more positive perception of points will be positively related to the utilitarian perception of an LMS. 1.4.

Novelty effects

Apart from considering the motivational, hedonic, and utilitarian dimensions of different game elements, it is relevant to consider the effects produced in relation to the time horizon of the implementation. Short term effects from gamification implementations have been attributed to so called novelty effects [ 4 ], wherein short-term engagement arises from the introduction of a new technology. It has been suggested that students tend to increase effort and attention when interacting with new technologies, however once they become familiar with the technology these effects tend to diminish [ 38 ]. In one study gamification had a positive effect on test scores initially, but did not persist for follow up tests, indicating that initial positive effects could be attributed to a novelty effect of the game elements [ 39 ]. Similarly, another study showed that gamification engagement decreased over time after initial introduction, but that engagement increased through the introduction of new gamification features [ 40 ]. Another study showed that gamification novelty effects were prevalent in terms of learning behavior but that novelty effects were counterbalanced over longer time periods when students reached a higher degree of familiarity with the gamification platform [ 41 ]. Apart from introducing new gamification features, another proposed way to overcome the novelty effects and produce long term results is to utilize game elements that satisfy learners’ basic psychological needs and intrinsic motivation [ 42 ].

Based on the presence of novelty effects in previous gamification implementations we hypothesize that:

H3: Students’ perception of points in a gamified LMS will decrease over time.

2. Method

To explore higher education students’ perception of points, a case study [ 43 ] was conducted, studying students’ perception of points over time, and the relationship between the perception of points and the utilitarian and hedonic dimensions of a gamified LMS in which the points system was applied. 2.1.

Study setting

The study was conducted in a soft skills course for first-year students in two bachelor engineering programs (data engineering and electrical engineering) at Linköping University. The course lasted 8-weeks and included three modules. Each module consisted of preparation tasks in terms of videos to watch, texts to read, mandatory assignments, and mandatory essays that were reflections of the mandatory assignments. At the end of each module, the students participated in a mandatory seminar in which essays were read and then discussed with other students. The students could either pass or fail the course.

An in-house developed gamified LMS called E-prof (Figure 1) was used by the students during the course. The LMS was web-based and contained information about the course and each module, preparation videos and texts, assignment instructions, submissions, teacher feedback, and grade results.

Two game elements, a progress bar, and a points system, were included in the LMS and displayed at the top of the menu. The progress bar visualized the amount of the course the students had completed, represented by both the tasks that the students had marked as done and by the assignments that had been graded. The point system consisted of three types of points visualized through different symbols: • Course points were visualized through a star and represented the points that the students needed to collect to pass the course. The students achieved course points for seminars attended and passed assignments. • Order points were visualized through a shield and represented the points that the students achieved for completing activities in an orderly manner. The students achieved order points for submitting assignments on time, for submitting the work in the LMS and for submitting the file in PDF form. • Master points were visualized through a puzzle piece and represented the points that the students achieved for completing activities particularly well. The students achieved master points for submitting assignments of high quality and for submitting assignments the day before the deadline.

When students had not collected any of the points all the point symbols where black; when the students had achieved 1-29% of one of the points, the symbol changed to a bronze color; when the students had achieved 30-69% of the points, the symbol changed to a silver color; and when the students had achieved 70-100% of the points the symbol changed to a gold color (Figure 2).

To pass the course the students needed to get all 35 of the course points and 19 of 28 order points (68%). If students collected all the course points, 18 of 21 (86%) master points and 23 of 28 (82%) order points they achieved a coffee cup, on which three stars, the name of the course, and the University logo were visualized (Figure 3).

Participants and procedure

Ninety-three students participated in the course. The data collection consisted of two main phases. In the first phase, RQ1 was addressed by students filling out a voluntary survey during the first seminar, held during the second course week. Since students had to submit an assignment and an essay in the LMS before the seminar, all students were assumed to have entered the LMS and seen the game elements. However, no points had yet been awarded to the students. Before filling out the survey, students signed a consent form consisting of information about the data that would be collected, that participation was voluntary and could be discontinued at any time, and that anonymized results would be used for publication and further development of the LMS. Fifty-one students submitted answers to the first survey, of which 35 studied data engineering and 16 studied electrical engineering. The students ranged in age from 19 to 35 with a mean age of 22 years (SD = 2.98). Forty-five students identified as men, 5 as women, and 1 person reported being unsure about their gender.

In the second phase, RQ2 was addressed, by students filling out the same survey during the last seminar of the course, six weeks after the initial survey. Again, students were asked to fill out the consent form before filling out the survey if this had not been done previously. No compensation was awarded for answering either of the surveys. Forty-one students answered the second survey. In both surveys, students submitted a personal code to be able to track individual answers anonymously between the two surveys. Students filled out the personal code by writing the first letter of their father’s name, the first letter of their mother’s name, and the day they were born. Thirty-two students in total answered both the first and the second survey and were thus included in the analysis to answer RQ2. 2.3.

Measurements

Two main quantitative measures were included in the first and second survey: the perception of points and pragmatic and hedonic quality from user experience questionnaire [ 44 ].

Perception of points: For the perception of point, we were not able to identify a specific measure for how individual game elements are perceived. In previous studies on the perception of points, measures have been based on a single item [ 18, 14 ]. Since the perception of a game element should be considered from a utilitarian and hedonic perspective [ 5 ] and well as a motivational perspective [ 25 ], we wanted to include several items that could represent the perception of points. The scale was first developed in English (Table 1) and was translated to Swedish to be included in the first and second survey. Related to the hedonic and utilitarian dimensions, one general item was included symbolizing the preference of the point system (PP2), students who perceived the point system as both hedonic and utilitarian were deemed to be more likely to want to see the point system in other courses. In terms of the hedonic dimension of the perception of points, one item was included related to the direct hedonic experience of perceiving the points as fun (PP1). In terms of the utilitarian dimension of the perception of points one item was included that represented seeing the point system as a valuable part of the LMS (PP3). Related to motivation, two items were included related to wanting to collect more points, one for each point type that was not related to passing the course (PP4, PP5). Two items were also included related to the external motivation associated with the points system (PP6, PP7). These items were associated with external motivation due to their association with achieving enough points to get the cup. Finally, two items were added related to the intrinsic motivation of the points system (PP8, PP9). These items were associated with the internal motivation, due to students wanting to pursue the points for their own sake rather than reaching an external objective [ 26 ]. Each item was answered on a five-point Likert scale ranging from I completely disagree (1) to I completely agree (5).

User experience questionnaire: The user experience questionnaire (EEQ) [ 44 ] was included in the survey based on its representation of hedonic and utilitarian dimensions in relation to software products. The scale is based on 26 questions containing opposite statements, e.g., annoying/enjoyable, in which respondents answer on a seven-point scale ranging between the two attributes. EEQ items are divided into six subscales that are combined into two aspects of the user experience: hedonic quality and pragmatic quality. The Swedish version of the EEQ was included in the first and second survey.

A principal factor analysis was run with direct oblimin oblique rotation to uncover the underlying components behind the perceptions of points items. Direct oblimin oblique rotation was chosen due to the theoretical ground of the factors being expected to correlate with each other [ 45 ]. An R-matrix was produced to uncover items with small bi-variate correlations as well as potential multicollinearity problems. Low item correlation was determined through items having several items with a bi-variate correlation <0.3 and multicollinearity problems where determined for bi-variate correlations >0.9 and a R-matrix determinant < 00001 [ 45 ]. The Kaiser-MeyerOlkin measure was used to verify sampling adequacy with an excepted criterion of >0.5 for the combined and individual items [ 45 ]. Finally, Bartlett’s measure was checked to validate that the R-matrix significantly differed from an identity matrix.

For the items in the user experience questionnaire, negative items, i.e., items with a positive attribute before the negative attribute, were reversed. Cronbach alpha was measured to determine internal consistency for the components uncovered through the principal factor analysis of perception of points and for the scale items of hedonic quality and pragmatic quality. To answer RQ1 and explore the relationship between the perception of points and hedonic and utilitarian dimensions of the gamified LMS, two multiple linear regressions were run based on the students’ answers in survey 1; one with hedonic quality as the dependent variable and one with pragmatic quality as the dependent variable. In both cases, the forced entry method was chosen due to no clear order of importance between the independent variables [ 45 ]. Assumptions of the multiple linear regressions were accounted for in terms of outliers and linearity (identified through scatterplots), homoscedasticity and independence (identified through partial regression plots), and normality (identified through residual histograms) [ 45 ]. To answer RQ2 and investigate how the perception of points changed over time, a paired samples ttest was run with perception of points scales as dependent variables based on the students’ answers in survey 1 and survey 2. Before running the analysis, the normal distribution of the difference between scores were analyzed through the Shapiro-Wilk test-statistic for each component.

3. Results

The R-matrix showed no correlations over 0.9 and only one bi-variate correlation below 0.3. The determinant of the R-matrix was 0.000, thus surpassing the value of 0.00001 and not indicating any problems with multicollinearity. The KMO test showed a value of 0.804, which represented an adequate sampling size. The individual values of KMO identified in the anti-image correlation matrix showed that all values were above 0.658. Bartlett’s test of sphericity showed a significant value (p<0.001) thus showing that the matrix significantly differed from an identity matrix. Since all assumptions had been met, all items were included in the analysis.

Item PP1 PP2 PP3 PP4 PP5 PP6 PP7 PP8 PP9

In obtaining the eigenvalues for each factor, two factors had an eigenvalue above Kaiser’s criterion of 1, together explaining 74.03% of the variance (Table 3). The scree plot showed the clearest inflexion point when retaining two factors. Two factors were therefore retained. The factor loading after rotation (Table 4) demonstrated one factor related to the perceived value of points in terms of the hedonic and utilitarian aspects of the point system (Factor 1) and the other factor (Factor 2) related to the motivational perception of points.

Cronbach’s alpha was generated for the scales of perceived value of points and motivational perception of points, perceived ease of use and perceived usefulness, and for hedonic quality and pragmatic quality. The threshold value of an accepted alpha value was set at 0.7 [ 46 ]. All scales showed an accepted alpha value with all items included (Table 5) and individual scale items were thus derived by taking the mean of the items connected to each scale.

The assumptions behind the two linear regression models had been met showing linear tendencies and no clear outliers, evenly distributed values around the linear tendencies for the predicting variables in the partial regression plots, and evenly distributed residuals. The overall regression model for hedonic quality was statistically significant (R2 = 0.16, R2adj = 0.13, F (2, 50) = 4.58, p = 0.02), demonstrating that the perceived motivation of points and perceived value of points in combination had a positive relationship to the hedonic quality of the LMS. However, neither the perceived motivation of points (b = 0.34, p = 0.10), nor the perceived value of points (b = 0.10, p = 0.48), significantly predicted the hedonic quality of the LMS (Table 6.).

The overall regression model for pragmatic quality was not statistically significant (R2 = 0.02, R2adj = -0.02, F (2, 50) = 0.45, p = 0.64), demonstrating that the perceived motivation of points and perceived value of points in combination did not have a positive relationship to the pragmatic quality of the LMS

The Shapiro-Wilk test statistic showed a nonsignificant value for the perceived motivation of points factor (p=0.200) and for the perceived value of points factor (p=0.200). Thus, the data did not deviate from the assumption of normality. The average score for the perceived value of points after two weeks was higher (M=3.64, SD = 0.791) than after eight weeks (M=3.33, SD = 0.900). The results showed a significant mean difference (M=0.304, SD = 0.730) between measures (t (31) = 2.354, p = 0.025). The average score for the perceived motivation of points after two weeks was higher (M=3.39, SD = 0.969) than after eight weeks (M=2.89, SD = 0.973). The results showed a significant mean difference (M=0.496, SD =0.753) between measures (t (31) = 3.723, p =< 0.001). The results indicate that the perception of points had decreased over time in terms of utilitarian and hedonic perception as well as motivational perception.

4. Discussion

The present work intended to investigate how higher-education students’ perception of points relate to hedonic and utilitarian dimension of a gamified LMS as well as how students’ perceptions of points change over time. The factor analysis based on the first survey uncovered two factors related to the perception of points: perceived value of points (Factor 1) and perceived motivation of points (Factor 2).

In relation to RQ1, the present study showed that higher-education students’ perception of points related to the hedonic but not to the utilitarian perception of the gamified LMS. The perceived value of points and the perceived motivation of points predicted the hedonic quality of the LMS, explaining 16% of the variance. Students who experienced the perception of points more positively, in terms of both value and motivation, also perceived the hedonic quality of the LMS more positively, providing support for H1. At the same time, the factors did not show a significant individual prediction on the hedonic quality. The result could indicate that perceived value of points and perceived motivation of points should both be high to support the hedonic quality of the LMS.

The result is aligned with previous research, suggesting that hedonic dimensions of gamified services contribute to a more positive attitude [ 16, 32 ], higher likelihood of continued use [ 16, 30, 33 ], and positive user experience [ 31, 32 ]. However, while previous research focused on the hedonic dimension of the gamified system as a whole, the present work focused on the perception of the individual game element points in relation to the hedonic and utilitarian perception of the LMS. This highlights that focusing on providing value in terms of the utilitarian, hedonic, and motivational benefits to individual game elements can contribute positively to the hedonic perception of the LMS.

The perceived value of points and the perceived motivation of points did not significantly predict the pragmatic quality of the LMS, thus not supporting H2. This differs from previous research, in which the utilitarian dimension of gamified services contributed to a more positive attitude [ 16, 32 ], higher likelihood of continued use [ 16, 30, 33 ], and positive user experience [ 31, 32 ]. In light of the study focusing on the individual game element of points the results could be explained by the fact that the point system had been designed to enhance the pragmatic quality of the LMS. As has been showed in previous studies [ 20, 21, 22 ] point systems encourage the behavior they represent, which might also hold true for the perception of points; that is, points encourage positive perceptions in relation to what the gamification system is designed for but not necessarily other factors.

In relation to RQ2, the present study showed that higher-education students’ perception of points decreased over time There has been a significant decrease in terms of both perceived value of points and perceived motivation of points between the second and the eight week of the course. Similar results are present in previous research, with performance [ 39 ], engagement [ 40 ] and learning behavior [ 41 ] increasing at the introduction of new game elements but decreasing over time. Previously, the initial increase followed by a decrease has been attributed to the novelty effect [ 38 ]. The result of the present study shows support for H2 and the novelty effect being present in relation to the perception of points over time. Since the students had not received any points when answering the first survey the results could also arise from a discrepancy between the students’ expectations of points their actual experience after receiving them. Furthermore, the reason for the students’ perception changing over time could also be attributed to the extrinsic motivational nature of the gamified system. It is possible that when students could no longer achieve the cup, their perception of points as a game element could have decreased, explaining the lower perception over time.

In the present study a positive perception of points was associated with a positive hedonic LMS perception and the perception of points decreased over time. Based on the results, we suggest that points in higher education should be designed with regards to how students perceive the value and motivational aspects behind them and with considerations of how novelty effects can be mitigated.

4.1. Limitations research and further

Limitations include a lack of generalizability due to the case study design and small sample size. To generalize the findings, more studies are needed that investigate higher education students’ perception of points in different settings. The nature of the study did not allow for causal determinates of the effect of the perception of points on the hedonic and utilitarian perception of the LMS. To determine causality, we recommend that further studies manipulate the points systems and study the impact on the hedonic and utilitarian perception of the LMS over time.

Another limitation is the time horizon of the implementation: no longitudinal effects could be determined for a longer extent than the course’s time horizon of 8-weeks. As previous research suggests, initial novelty effects could be counterbalanced with familiarity over time [ 41 ]. Further research should therefore investigate the presence of the novelty effect on students’ perception of points over longer time horizons. While the results showed that perceived usefulness and hedonic quality were higher for students who perceived the points system as more positive, the results did not shed light on individual differences between students that could have contributed to the variation in perceptions. The majority of the students identified as male in the study which could also have impacted the results. Therefore, one avenue for further research is to explore what contributes to students’ differences in perceptions about point systems.

5. References

References

[1] P. Trowler and V. Trowler , “ Student engagement evidence summary,” High . Educ., no. November , 2010 .

[2] J. Looyestyn , J. Kernot , K. Boshoff , J. Ryan , S. Edney , and C. Maher , “ Does gamification increase engagement with online programs? A systematic review , ” PLoS One , 2017 , doi: 10.1371/journal.pone. 0173403 .

[3] S. Deterding , D. Dixon , R. Khaled , and L. Nacke, “ From game design elements to gamefulness: Defining 'gamification ,' ” Proc. 15th Int. Acad. MindTrek Conf. Envisioning Futur. Media Environ. MindTrek 2011 , pp. 9 - 15 , 2011 , doi: 10.1145/2181037.2181040.

[4] J. Hamari , J. Koivisto , and H. Sarsa , “ Does gamification work? - A literature review of empirical studies on gamification,” in Proceedings of the Annual Hawaii International Conference on System Sciences , 2014 , pp. 3025 - 3034 , doi: 10.1109/HICSS. 2014 . 377 .

[5] J. Hamari and J. Koivisto , “ Why do people use gamification services?,” Int. J. Inf . Manage., vol. 35 , no. 4 , 2015 , doi: 10.1016/j.ijinfomgt. 2015 . 04 .006.

[6] S. Subhash and E. A. Cudney , “ Gamified learning in higher education: A systematic review of the literature , ” Comput. Human Behav ., vol. 87 , no. February , pp. 192 - 206 , 2018 , doi: 10.1016/j.chb. 2018 . 05 .028.

[7] I. Jedel , A. Palmquist , R. Munkvold , O. Goethe , H. Jonasdottir , and E. Olsson, “ An industry experiment of academic performance and drop-out in gamified distance education,” in CEUR Workshop Proceedings , 2021 , vol. 2883 , pp. 1 - 10 .

[8] M. D. Hanus and J. Fox , “ Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison , satisfaction, effort, and academic performance,” Comput. Educ., vol. 80 , 2015 , doi: 10.1016/j.compedu. 2014 . 08 .019.

[9] C. Dichev and D. Dicheva , “ Gamifying education: what is known, what is believed and what remains uncertain: a critical review,” Int. J. Educ. Technol. High. Educ. , vol. 14 , p. 9 , 2017 , doi: 10.1186/s41239-017- 0042-5.

[10] M. Urh , G. Vukovic , E. Jereb, and R. Pintar , “ The Model for Introduction of Gamification into E-learning in Higher Education,” Procedia - Soc. Behav. Sci. , vol. 197 , 2015 , doi: 10.1016/j.sbspro. 2015 . 07 .154.

[11] J. Broer and A. Breiter , “ Potentials of gamification in learning management systems: A qualitative evaluation , ” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2015 , vol. 9307 , doi: 10.1007/978-3- 319 -24258-3_ 29 .

[12] M. C. Lee , “ Explaining and predicting users' continuance intention toward e-learning: An extension of the expectation-confirmation model,” Comput . Educ., vol. 54 , no. 2 , 2010 , doi: 10.1016/j.compedu. 2009 . 09 .002.

[13] S. S. Alam , M. Masukujjaman , M. Ahmad , and R. Jaffor , Acceptance of online distance learning (ODL) among students: Mediating role of utilitarian and hedonic value , no. 0123456789 . Springer US , 2022 .

[14] M. Aparicio , T. Oliveira , F. Bacao , and M. Painho , “ Gamification: A key determinant of massive open online course (MOOC) success ,” Inf. Manag., 2019 , doi: 10.1016/j.im. 2018 . 06 .003.

[15] F. F. H. Nah , Q. Zeng , V. R. Telaprolu , A. P. Ayyappa , and B. Eschenbrenner , “ Gamification of education: A review of literature , ” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2014 , vol. 8527 LNCS, doi: 10.1007/978-3- 319 -07293-7_ 39 .

[16] G. Zichermann and C. Cunningham , “Gamification By Design,” Vasa. CA: O'Reilly Media , p. 208 , 2011 .

[17] F. Raczkowski , “ Making Points the Point: Towards a History of Ideas of Gamification,” Rethink . Gamification, no. May , 2014 .

[18] D. Song , P. Ju , and H. Xu , “ Engaged cohorts: Can gamification engage all college students in class?,” Eurasia J. Math. Sci. Technol . Educ., vol. 13 , no. 7 , 2017 , doi: 10.12973/eurasia. 2017 .00755a.

[19] S. E. Koppitsch and J. Meyer, “ DO POINTS MATTER? THE EFFECTS OF GAMIFICATION ACTIVITIES WITH AND WITHOUT POINTS ON STUDENT LEARNING AND ENGAGEMENT,” Mark. Educ. Rev., vol. 32 , no. 1 , 2022 , doi: 10.1080/10528008. 2021 . 1887745 .

[20] Y. Attali and M. Arieli-Attali , “ Gamification in assessment: Do points affect test performance?,” Comput. Educ. , vol. 83 , 2015 , doi: 10.1016/j.compedu. 2014 . 12 .012.

[21] J. R. Chapman and P. J. Rich , “ Does educational gamification improve students' motivation? If so, which game elements work best?,” J. Educ . Bus., vol. 93 , no. 7 , 2018 , doi: 10.1080/08832323. 2018 . 1490687 .

[22] C. Cheong , J. Filippou , and F. Cheong , “ Towards the gamification of learning: Investigating student perceptions of game elements,” J. Inf . Syst. Educ., vol. 25 , no. 3 , 2014 .

[23] T. Aldemir , B. Celik , and G. Kaplan, “ A qualitative investigation of student perceptions of game elements in a gamified course,” Comput. Human Behav ., vol. 78 , 2018 , doi: 10.1016/j.chb. 2017 . 10 .001.

[24] P. R. Kleinginna and A. M. Kleinginna , “ A categorized list of motivation definitions, with a suggestion for a consensual definition,” Motiv . Emot., vol. 5 , no. 3 , pp. 263 - 291 , Sep. 1981 , doi: 10.1007/BF00993889.

[25] S. Deterding , “ The lens of intrinsic skill atoms: A method for gameful design,” Human-Computer Interact ., vol. 30 , no. 3-4 , pp. 294 - 335 , 2015 , doi: 10.1080/07370024. 2014 . 993471 .

[26] R. M. Ryan and E. L. Deci , “ Intrinsic and extrinsic motivation from a selfdetermination theory perspective: Definitions, theory, practices, and future directions,” Contemp. Educ. Psychol. , vol. 61 , 2020 , doi: 10.1016/j.cedpsych. 2020 . 101860 .

[27] S. Conway , “Zombification  ?: Gamification , motivation , and the user ,” vol. 6 , no. 2 , pp. 129 - 141 , 2014 , doi: 10.1386/jgvw.6.2.129.

[28] E. L. Deci and R. M. Ryan , “ The 'what' and 'why' of goal pursuits: Human needs and the self-determination of behavior,” Psychol . Inq., vol. 11 , no. 4 , 2000 , doi: 10.1207/S15327965PLI1104_ 01 .

[29] I. P. Pavlov , “ Excerpts from The Work of the Digestive Glands ., ” Am. Psychol. , vol. 52 , no. 9 , pp. 936 - 940 , 1997 , doi: 10.1037/ 0003 - 066x . 52 .9.936.

[30] B. F. Skinner , “ Operant behavior ., ” Am. Psychol ., 1963 , doi: 10.1037/h0045185.

[31] E. L. Deci , R. M. Ryan , and R. Koestner , “ A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation,” Psychol . Bull., vol. 125 , no. 6 , pp. 627 - 668 , 1999 , doi: 10.1037/ 0033 - 2909 . 125 .6.627.

[32] S. Nicholson , “ A recipe for meaningful gamification,” in Gamification in Education and Business , 1st ed., T. Reiners and L. C. Wood , Eds. Springer, 2015 , pp. 1 - 35 .

[33] F. D. Davis , “ Perceived usefulness, perceived ease of use, and user acceptance of information technology,” MIS Q. Manag . Inf. Syst., 1989 , doi: 10.2307/249008.

[34] D. B. Köse , B. Morschheuser , and J. Hamari , “ Is it a tool or a toy? How user conceptions of a system's purpose affect their experience and use,” Int. J. Inf . Manage., vol. 49 , no. February , pp. 461 - 474 , 2019 , doi: 10.1016/j.ijinfomgt. 2019 . 07 .016.

[35] C. L. Hsu and M. C. Chen , “ How does gamification improve user experience? An empirical investigation on the antecedences and consequences of user experience and its mediating role , ” Technol. Forecast. Soc. Change , vol. 132 , 2018 , doi: 10.1016/j.techfore. 2018 . 01 .023.

[36] C. L. Hsu , Y. C. Chen , T. N. Yang , and W. K. Lin , “ Do website features matter in an online gamification context? Focusing on the mediating roles of user experience and attitude ,” Telemat. Informatics, vol. 34 , no. 4 , pp. 196 - 205 , 2017 , doi: 10.1016/j.tele. 2017 . 01 .009.

[37] D. Dicheva , K. Irwin , and C. Dichev , “ Gamifying with oneup: For learning, grades or fun? , ” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2019 , vol. 11385 LNCS, doi: 10.1007/978-3- 030 -11548-7_ 32 .

[38] R. E. Clark , “ Reconsidering Research on Learning from Media,” Rev. Educ. Res. , vol. 53 , no. 4 , 1983 , doi: 10.3102/00346543053004445.

[39] D. R. Sanchez , M. Langer , and R. Kaur , “ Gamification in the classroom: Examining the impact of gamified quizzes on student learning,” Comput. Educ. , vol. 144 , 2020 , doi: 10.1016/j.compedu. 2019 . 103666 .

[40] C. H. H. Tsay , A. K. Kofinas , S. K. Trivedi , and Y. Yang , “ Overcoming the novelty effect in online gamified learning systems: An empirical evaluation of student engagement and performance,” J. Comput . Assist. Learn., vol. 36 , no. 2 , 2020 , doi: 10.1111/jcal.12385.

[41] L. Rodrigues et al., “ Gamification suffers from the novelty effect but benefits from the familiarization effect: Findings from a longitudinal study,” Int. J. Educ. Technol. High. Educ. , vol. 19 , no. 1 , 2022 , doi: 10.1186/s41239-021-00314-6.

[42] A. H. T. Kam and I. N. Umar , “ Fostering authentic learning motivations through gamification: A self-determination theory (SDT) approach ,” J. Eng. Sci. Technol., vol. 13 , no. Special Issue , 2018 .

[43] J. Gerring , “ What is a case study and what is it good for?,” Am. Polit. Sci. Rev. , vol. 98 , no. 2 , 2004 , doi: 10.1017/S0003055404001182.

[44] B. Laugwitz , T. Held , and M. Schrepp , “ Construction and evaluation of a user experience questionnaire , ” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2008 , vol. 5298 LNCS, doi: 10.1007/978-3- 540 -89350- 9 _ 6 .

[45] A. Field , Discovering statistics using IBM SPSS statistics , vol. 58 . 2013 .

[46] J. M. Bland and D. G. Altman , “Statistics notes: Cronbach's alpha, ” BMJ , vol. 314 , no. 7080 , 1997 , doi: 10.1136/bmj.314.7080.572.