The effect of gamification on students engagement: A preliminary study

Abstract

The use of game elements and mechanics in a non-game context, gamification, is being applied in both academia and industries in recent years. However, the evaluation process to recognize the benefits and limitations of using this concept in education still requires further investigation. This paper aims to investigate the effect of the gamification element on student behavioral engagement in an online course. A leaderboard plugin was developed and deployed in a Learning Management System (LMS). To investigate the effect, a quasi-experimental design was used with 48 students, in the middle school's computer course in Saudi Arabia. The effect of the leaderboard plugin was compared to those without a gamification plugin in the same educational setting. Initial findings indicate that the gamified online course presents better results than the traditional online course.

Keywords

• Gamification
• Student Engagement
• E-Learning
• Game-Element
• Leaderboard
• K-12

Introduction

The application of gamification in a variety of sectors has increased rapidly in recent years, due to its potential benefits in relation to enhancing user motivation and engagement, driving user behavior and increasing productivity [ 1-4 ]. In an educational context, gamification allows the student to receive direct feedback regarding their progress in the classroom and acknowledgment of completed activities [ 5 ]. Also, it offers the potential to increase student motivation in the classroom [ 6 ].

Student engagement is critical for academic success [ 7 ], due to enhance student motivation to learn, increase student satisfaction and improves student performance [ 8 ]. Student engagement defined as the amount of time and energy that the student dedicated to academic experience [ 9, 10 ]. Although extensive studies have been performed, engaging students in activities and the learning process in the classroom is still a challenge for educators [ 4, 11 ]. Students still face problems in developing the required level of engagement to achieve the full benefits of their education [ 4, 11, 12 ]. Gamification has been introduced as one approach to address this challenge, through the introduction of a game element and design, such as using badges and points, in non-game context [ 13 ]. However, there is a still need for more research on the effectiveness of gamification in a real classroom [ 14-16 ]. In the literature, many studies have investigated a combination of elements [ 17-19 ], rather than limiting their focus to one element. Thus, making it difficult to assess the relative impact of each element of student engagement and which has a greater impact. In this study, we aim to empirically demonstrate the effect of a gamification element in the educational environment, in terms of student behavioral engagement, by introducing a leaderboard into an online course. The leaderboard is an element in gamification that represented the rank of participants in the competitive environment that represented in a table based on their achievement [ 20 ]. This element is promoted to a competitive environment which allows the student to compare their performance with others [ 20 ]. Four indices of student behavioral engagement were selected and examined which are login frequency, activity completion rate, number of posts and number of views. These indices have previously been used as measures of student behavioral engagement [ 21-23 ].

This paper is organized as follows: Section 2 presents a literature review on the topic. Section 3 explains the methodology used, the design tool, and the preliminary results obtained. Finally, Section 4 provides a brief discussion and the conclusions of the work. 2

Literature review

Gamification has been defined as the application of game design elements in the nongame context [ 24 ]. A similar definition was given by Landers [ 25 ] who defined gamification as the utilization of features associated with video games in the non-game context. In trying to define gamification from an educational view, Kapp [ 5 ] introduced nine concepts, which are similar to those of Deterding et al. [ 26 ], which describes gamification as a game-based approach that includes rules, feedback, challenges, interactivity, and evoked emotions. In addition, the purpose of gamification is primarily to engage the participants in solving problems and to motivate actions, both socially and individually [ 27 ]. In gamification, the participants directly impact the outcome of the game with immediate effects, generating an environment that is controlled by them [ 27 ].

Gamification consists of various principles, each of which can be applied to create a desired gamified experience. Gamification mechanics refers to the decisions that the designers involved in the gamification of non-game context make to identify the rules, context, goals, and boundaries of the events and activities to be gamified, as well as the interactions types, such as opponents [ 28 ]. Gamification dynamics refers to the participant behavior characteristics that occur as the participants engage in the activity [ 29 ]. Gamification elements are comparatively easy to recognize and use, especially because many of the users have played games previously [ 30 ]. A wide variety of gamification elements exist [ 31 ]. Examples of these include things such as points, badges, leaderboards, levels, challenges, and rewards. A leaderboard is a popular element of gamification that has been successful in different fields due to several reasons [ 20 ].

One of these reasons is that the concept of the leaderboard is familiar to all ages. In addition, it makes the handling of the successes and failures visible to all participants. As well, it motivates the participants by presenting them in social comparison [ 20 ].

Gamification primarily attempts to enhance user motivation towards a particular activity or the use of particular technology, thus leading to an increase in the quality and quantity of certain output activities [ 32 ]. Various studies focused on the educational field have shown that gamification has a positive effect on student learning [ 33, 34 ], student participation [ 34 ], and increase student motivation [ 33 ].

For online courses, student engagement refers to the effort and amount of time that students put into their courses [ 35, 36 ]. Student engagement can be subdivided into three primary domains, i.e., behavioral, emotional and cognitive engagement [ 37 ]. Behavioral engagement relates to the student’s behavior in class, their participation in school-based activities and their interest in academic tasks [ 38 ]. Emotional engagement emphasizes issues relating to the feelings of the students, in terms of their belonging or being of value to their class, school or teacher (e.g. interest, happiness, sadness, and boredom) [ 39 ]. Finally, cognitive engagement focuses on the internal investment of learners in the learning process, including their internal psychological qualities and hidden characteristics that encourage effort in learning [ 37 ]. The engagement is essential for students to obtain the skills and knowledge to succeed, both in their school life and future careers [ 40 ]. When students are not engaged, this leads to various adverse outcomes, such as students dropping out of certain courses or units, achieving lower grades and having reduced employment opportunities [ 41 ]. Gamification has been introduced as a new approach with great promise for enhancing the engagement of the student in online learning platforms [ 42 ]. Various studies have been conducted to measure the impact of gamification on student engagement. Simões et al. [ 6 ] proposed a social gamification framework for K–6 students using a social learning platform, which enables teachers to choose an appropriate social gamification tool to deliver content to their students, depending on the mechanics and dynamics of the social gamification, using a point-based rewards system. However, no empirical study has been completed that shows the effectiveness of this approach. Hew et al. [ 43 ] explored the effect of gamification on student engagement and motivation in an online course. They gamified course using points, leaderboard and badges in relationship to each other. Results showed that the use of points, badges, and leaderboard in an online course motivated the students to choose challenging activities and to produce higher-quality finished results. Song et al. [ 44 ] investigated the influence of points on student engagement, showing that the use of points increased the students’ enjoyment and engagement.

As mention by [ 45 ] although of the increasing amount of literature in the field of gamification in education, Alsawaier [ 46 ] stated that there is still a lack of researches that studies the impact of gamification on student engagement. In addition, the wide range of student background, course type, and learning preferences require more researches on the impact of gamification elements to assess their effectiveness [ 45 ]. Consequently, this study aimed to measure the effect of a single element, i.e., a leaderboard on student behavioral engagement in an online course.

Study

Research design and sample

A quasi-experimental design was used to conduct this study. A leaderboard plugin was build and embedded in the Moodle1 learning management system. The study was conducted on two groups of students, namely the gamified group and the control group, from two different classes. The participants in the study were 48 female students from the third intermediate grade in Saudi Arabia’s public schools. The gamified group contains 23 students, and the control group contains 25 students. The study was run during the first semester of the academic year 2018–2019 on a computer course. Two versions of the course were created in Moodle: one was gamified using the leaderboard plugin and the other without it. The course uses a blended learning approach. The lectures were complemented with activities and resources that were delivered online through the Moodle. The contents, course materials, and the instructor were the same for both groups. The data collected were analyzed using the descriptive statistical method and the T-test method to determine the significant results between the groups. 3.2

Tool design

In order to examine the effect of the gamification element on student behavioral engagement, we design and develop a plugin tool that consists of a leaderboard and deploy it on the Moodle platform that has been selected in this study due to its similarity to another system that the students are familiar with. The leaderboard lists the top five of students based on how many points they have earned. So, as their earned points increase, their rank in the leaderboard increases too. The students' points appear for all students which increase the competition between them [ 21, 47, 48 ]. Leaderboard displays on the main page of the gamified course and shows the student name, rank, points earned, their picture and the percentage of their progress in the course. The tool has customization features that enabled the instructor to define their specific rules for earning points. Students will only earn points based on the instructor’s rules set at the beginning of the class, and the rules were visible to the students. The earning points give the students direct feedback on their achievements. The students start with 0 points. The instructor decides the actions to reward points to the student throughout the course based on their participation in course chat or discussion forum either they view or posts, and their view of the course resources (course slides, video tutorial, PDF, and URL).

1 Moodle: https://moodle.com / 3.3

Data collection and analysis

A quantitative data of both groups was collected in this study in order to examine the effect of a leaderboard on student behavioral engagement. Quantitative data was collected from the log file and activity completion report in Moodle. Logfile contains data related to login frequency, number of views, and number of posts. The activity completion rate report contains data related to student activity in the course activities and resources. The data was preliminarily analyzed using descriptive statistics to assess the student behavioral engagement of both groups throughout the semester. In addition, a T-test was used to determine significant results between the gamified and control groups. The data was evaluated using a one-tailed T-test with assuming unequal variance. The results of the analyses are illustrated in Tables 2-4.

The analysis result of login frequency in Table 2 indicate a significant login frequency for the gamified group (M = 28.64, SD = 56.93) over the control group (M = 6.46, SD = 9.11), t(22) = 1.72, p = .05.

The analysis results of the number of views and the number of posts for both groups illustrated in Table 3. Alike to the login frequency result, the result indicated higher significant result for the gamified group ( M = 23.41, SD =40.73) than the control group ( M = 5.35, SD = 9.15), t(18) = 1.73, p = .05.

The data analysis collected records of the number of the posts by both groups also indicated the same results as the previous analysis i.e., the gamified group ( M = 5.82, SD =7.78) performed significantly higher number of posts compared to the control group ( M = 1.94, SD = 2.88), t(20) = 1.72, p = .05.

Action No. of views No. of posts

The activity completion rate report refers to student activity in the course activities and resources. It contains the status information of the course activities and resources (completed or not completed) for each student. The course activities represented in chat, forum, and assignments. The course resources are consisting of course slides and PDF, which are represented in the following figure as file, as well as videos. The status of course activities will change to completed in chat and forum once the student post, while in the assignment the status will change if the student submit s the assignment. For the course resources, the status will change once the student view course slides, PDF, and videos. Fig. 3 and Table 4 illustrated the analysis results of the activity completion rate report for both groups.

120%

Comparing the results of activity completion rate for both groups are illustrated in Fig.3. They show that the gamified group completed all activities and resources, whereas the control group only completed 53%. The results also reveal a significantly higher activity completion rate for the gamified group (chat = 14%, file 1 = 10%, file 4 = 14%, file 5 = 14%, video 2 = 14%, video 6 = 17%, assignment 1 = 31%) over the control group (chat = 8%, file 1 = 8%, file 4 = 8%, file 5 = 8%, video 2 = 4%, video 6 = 12%, assignment 1 = 12%). The results of the T-test illustrated in Table 4 indicated a higher significant result of activity completion rate to the gamified group ( M = 0.12, SD = 0.06) over the control group ( M = 0.04, SD = 0.05), t(25) = 1.71, p = .05.

Discussion and conclusion

A total of 48 female students of third intermediate grade participated in the study. The gamified group contained 23 students and the control group 25 students, hence the two groups were unequal in number. This also indicates that the records of the total number of logins by both groups lose some significance due to the unequal number of participants in each group, yet even with the lower number of participants in the gamified group, there were a higher number of logins than the control group: the gamified group recorded 630 logins whereas the control group recorded 155. The maximum number of logins was 237 and 35 for the gamified group and control group respectively.

The data analysis for the number of views indicates a significantly higher viewing for the gamified group than the control group. The highest number of views for both groups was 162 and 32 for the gamified group and control group respectively.

The analysis of the number of posts by both groups also indicates a similar result: that the gamified group performed a significantly higher number of posts compared to the control group. The gamified group performed 30 posts as the maximum number of posts, whereas the control group performed just 10.

The results of the activity completion rate report indicate that the gamified group completed all the course activities and resources compared to the control group. A comparison of the completion percentage for each course activity and resource reveal that the gamified group performed higher than the control group, except for assignment 2 where the control group recorded a higher result, with an 8% student completion rate compared to 7% for the gamified group.

The biggest differences from the results were found in login frequency, number of views, and number of posts, and the smallest with the activity completion rate report.

This paper contributes to the leaderboard effect on student behavioral engagement in an online course by providing evidence from a real class. A leaderboard plugin was developed and embedded into Moodle to gamify the online course and examine its effect. The experiment was conducted using a quasi-experimental design. Quantitative data of the gamified group and control group were collected from Moodle. Overall, the results of the four indices of online student engagement—login frequency, activity completion rate, number of views, and number of posts—show that the gamified group recorded the higher score in all indices compared to the control group. The recorded results were compared using the T-test method to determine significance between the groups. The initial findings of the study indicate that the results were significantly higher, i.e., the gamified group performed at a higher engagement level than the control group. In summary, these results support the hypothesis that gamification is valuable in enhancing student behavioral engagement in online courses.

In future work, we will validate these quantitative results. This will be done using the student-self report. In addition, we will test the impact of the leaderboard against the students’ performance by comparing the academic grades of both groups.

References

1. Kankanhalli , A. , Taher , M. , Cavusoglu , H. , Kim , S. : Gamification: A new paradigm for online user engagement . In: Thirty Third International Conference on Information Systems , vol. 4 , pp. 3573 - 3582 . Orlando ( 2012 ).

2. Hamari , J. , Koivisto , J. , Sarsa , H.: Does Gamification Work? -A Literature Review of Empirical Studies on Gamification . In: 47th Hawaii international conference on system sciences , vol. 14 , no. 2014 , pp. 3025 - 3034 . IEEE, Waikoloa, Hawaii ( 2014 ).

3. Association for Project Management: Introduction to Gamification . APM: Princes Risborough ( 2014 ).

4. Gapp , R., Fisher, R.: Undergraduate management students' perceptions of what makes a successful virtual group . Education+ training 54 ( 2 /3), 167 - 179 ( 2012 ).

5. Kapp , K. : The gamification of learning and instruction . John Wiley & Sons, San Francisco ( 2012 ).

6. Simões , J. , Redondo , R. , Vilas , A. : A social gamification framework for a K-6 learning platform . Computers in Human Behavior 29 ( 2 ), 345 - 353 ( 2013 ).

7. Manigault , E.: A Phenomenological Study of Student Engagement in United States History Classrooms ( 2014 ).

8. Martin , F. , Bolliger , D. : Engagement matters: Student perceptions on the importance of engagement strategies in the online learning environment . Online Learning , 22 ( 1 ), 205 - 222 ( 2018 ).

9. Astin , A. : Student involvement: A developmental theory for higher education . Journal of college student personnel 25(4) , 297 - 308 ( 1984 ).

10. Kuh , G.: What we're learning about student engagement from NSSE: Benchmarks for effective educational practices . Change: The Magazine of Higher Learning 35 ( 2 ), 24 - 32 ( 2003 ).

11. Gibbs , R. , Poskitt , J.: Student engagement in the middle years of schooling (Years 7-10): A literature review report to the Ministry of Education ( 2010 ).

12. da Rocha Seixas, L. , Gomes , A. , de Melo Filho , I. : Effectiveness of gamification in the engagement of students . Computers in Human Behavior 58 , 48 - 63 ( 2016 ).

13. Looyestyn , J. , Kernot , J. , Boshoff , K. , Ryan , J. , Edney , S. , Maher , C. : Does gamification increase engagement with online programs? A systematic review . PloS one 12(3) ( 2017 ).

14. Gåsland , M. : Game mechanic based e-learning: A case study . Master's thesis , Norwegian University of Science and Technology ( 2011 ).

15. Connolly , T. , Boyle , E. , MacArthur , E., Hainey , T. , Boyle , J.: A systematic literature review of empirical evidence on computer games and serious games . Computers & education 59(2) , 661 - 686 ( 2012 ).

16. Antonaci , A. , Klemke , R. , Specht , M.: The Effects of Gamification in Online Learning Environments: A Systematic Literature Review . Informatics. Multidisciplinary Digital Publishing Institute 6 ( 3 ), 32 ( 2019 ).

17. De-Marcos , L. , Garcia-Cabot , A. , Garcia-Lopez , E. : Towards the social gamification of e-learning: A practical experiment . International journal of engineering education 33(1) , 66 - 73 ( 2017 ).

18. Sánchez-Martín , J. , Cañada-Cañada , F. , Dávila-Acedo , M. : Just a game? Gamifying a general science class at university: Collaborative and competitive work implications . Thinking Skills and Creativity 26 , 51 - 59 ( 2017 ).

19. Çakıroğlu , Ü., Başıbüyük , B. , Güler , M. , Atabay , M. , Memiş , B. : Gamifying an ICT course: Influences on engagement and academic performance . Computers in human behavior , 69 , 98 - 107 ( 2017 ).

20. Chernbumroong , S. , Sureephong , P. , Muangmoon , O. : The effect of leaderboard in different goal-setting levels . In: 2017 International Conference on Digital Arts, Media and Technology (ICDAMT) , pp. 230 - 234 . IEEE ( 2017 ).

21. Barata , G. , Gama , S. , Jorge , J. , Gonçalves , D. : Engaging engineering students with gamification . In: 2013 5th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES) , pp. 1 - 8 . IEEE ( 2013 ).

22. Mandernach , B. , Donnelli-Sallee , E. , Dailey-Hebert , A. : Assessing course student engagement . Promoting student engagement 1 , 277 - 281 ( 2011 ).

23. Ding , L. , Er , E. , Orey , M.: An exploratory study of student engagement in gamified online discussions . Computers & Education 120 , 213 - 226 ( 2018 ).

24. Gafni , R. , Achituv , D. , Eidelman , S. , Chatsky , T. : The effects of gamification elements in e-learning platforms . Online Journal of Applied Knowledge Management 6 ( 2 ) ( 2018 ).

25. Landers , R.: Developing a theory of gamified learning: Linking serious games and gamification of learning . Simulation & Gaming 45 ( 6 ), 752 - 768 ( 2014 ).

26. Deterding , S. , Dixon , D. , Khaled , R. , Nacke , L. : From game design elements to gamefulness: defining gamification . In: Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments , pp. 9 - 15 . ( 2011 ).

27. Morford , Z. , Witts , B. , Killingsworth , K. , Alavosius , M. : Gamification: the intersection between behavior analysis and game design technologies . The Behavior Analyst 37 ( 1 ), 25 - 40 ( 2014 ).

28. Elverdam , C. , Aarseth , E.: Game classification and game design: Construction through critical analysis . Games and Culture 2 ( 1 ), 3 - 22 ( 2007 ).

29. Camerer , C. , Ho , T. , Chong , J.: Behavioural game theory: thinking, learning and teaching . In: Advances in Understanding Strategic Behaviour , pp. 120 - 180 . Palgrave Macmillan, London ( 2004 ).

30. Kim , B. : Understanding gamification . ALA TechSource ( 2015 ).

31. Dicheva , D. , Dichev , C. , Agre , G. , Angelova , G.: Gamification in education: A systematic mapping study . Educational Technology & Society 18 ( 3 ), 75 - 88 ( 2015 ).

32. Morschheuser , B. , Hamari , J. , Werder , K. , Abe , J.: How to gamify? A method for designing gamification . In: Proceedings of the 50th Hawaii International Conference on System Sciences 2017. University of Hawai'i at Manoa ( 2017 ).

33. Papp , T. : Gamification Effects on Motivation and Learning: Application to Primary and College Students . International Journal for Cross-Disciplinary Subjects in Education (IJCDSE) 8 ( 3 ), 3193 - 3201 ( 2017 ).

34. Buckley , P. , Doyle , E.: Gamification and student motivation . Interactive learning environments 24(6) , 1162 - 1175 ( 2016 ).

35. Li , F. , Qi , J. , Wang , G. , Wang , X. : Traditional classroom vs e-learning in higher education: Difference between students' behavioral engagement . International Journal of Emerging Technologies in Learning (iJET) 9 ( 2 ), 48 - 51 ( 2014 ).

36. Kuh , G.: What student affairs professionals need to know about student engagement . Journal of college student development 50(6) , 683 - 706 ( 2009 ).

37. Cooper , K. : Eliciting engagement in the high school classroom: A mixed-methods examination of teaching practices . American Educational Research Journal 51 ( 2 ), 363 - 402 ( 2014 ).

38. Wang , M. , Holcombe , R.: Adolescents' perceptions of school environment, engagement, and academic achievement in middle school . American educational research journal 47(3) , 633 - 662 ( 2010 ).

39. Renninger , K. , Bachrach , J.: Studying triggers for interest and engagement using observational methods . Educational Psychologist 50 ( 1 ), 58 - 69 ( 2015 ).

40. Wang , M. , Eccles , J.: Social support matters: Longitudinal effects of social support on three dimensions of school engagement from middle to high school . Child development 83( 3) , 877 - 895 ( 2012 ).

41. Bennett , J.: Work-based learning and social support: Relative influences on high school seniors' occupational engagement orientations . Career and Technical Education Research 32 ( 3 ), 187 - 214 ( 2007 ).

42. Silpasuwanchai , C. , Ma , X. , Shigemasu , H. , Ren , X. : Developing a comprehensive engagement framework of gamification for reflective learning . In: Proceedings of the 2016 ACM Conference on Designing Interactive Systems , pp. 459 - 472 . ACM ( 2016 ).

43. Hew , K. , Huang , B. , Chu , K. , Chiu , D. : Engaging Asian students through game mechanics: Findings from two experiment studies . Computers & Education 92 , 221 - 236 ( 2016 ).

44. Song , D. , Ju , P. , Xu , H. : Engaged cohorts: Can gamification engage all college students in class? . Eurasia Journal of Mathematics, Science and Technology Education , 13 ( 7 ), 3723 - 3734 ( 2017 ).

45. Panagiotis , F. , Theodoros , M. , Leinfellner , R. , Yasmine , R.: Climbing up the leaderboard: An empirical study of applying gamification techniques to a computer programming class . Electronic Journal of e-learning 14(2) , 94 - 110 ( 2016 ).

46. Alsawaier , R.: The effect of gamification on motivation and engagement . The International Journal of Information and Learning Technology 35 ( 1 ), 56 - 79 ( 2018 ).

47. Çakıroğlu , Ü., Başıbüyük , B. , Güler , M. , Atabay , M. , Memiş , B. : Gamifying an ICT course: Influences on engagement and academic performance . Computers in human behavior 69 , 98 - 107 ( 2017 ).

48. How to Boost Engagement on your LMS with Leaderboards , https://www.growthengineering.co.uk/gamification-leaderboards-lms/, last accessed 2019 /10/1.