The influence of gamified workshops on students' knowledge retention

Abstract

Educators frequently face serious problems concerning students' engagement and knowledge retention. As a proposed solution, gamification represents a new tool for active learning to increase students' motivation and thus improve their learning results. The goal of this paper is to investigate the effects of gamification on short- and long-term knowledge retention in all-day workshops on sustainable transport. A longitudinal experiment with 334 logistics students was conducted comparing the results of gamified and non-gamified workshops with students as future managers. The results suggest that gamification is an effective measure to increase students' learning outcomes with respect to sustainable transport.

Introduction

Despite continuous efforts by education professionals to seek novel and innovate educational approaches, many students perceive traditional schooling as boring and ineffective (Lee & Hammer, 2011) . Thus, they often lack motivation and engagement which negatively influences their learning performance. In fact, the forgetting curve by Ebbinghaus (1913) is still up to date and subject to intense scientific discussions about knowledge retention (Murre & Dros, 2015) . It states that the vast majority of knowledge has been forgotten two weeks after acquisition. To create highly motivating learning environments that help to overcome the lack of student interest and increase students’ knowledge retention level, researchers and educators increasingly employ gamification in education (Dicheva, Dichev, Agre, & Angelova, 2015) . Gamification is a promising approach to foster intrinsic motivation (Hamari & Keronen, 2017) , make learning more engaging and increase students’ learning performance (Buckley & Doyle, 2017; Kapp, 2012) . By applying game design elements in non-game contexts (Deterding, Dixon, Khaled, & Nacke, 2011) , gamification tries to take advantage of the growing passion for games (Thiebes, Lins, & Basten, 2014) . However, empirical research on the effectiveness of gamification in educational environments and its influence on learning outcomes is still scarce (Dicheva et al., 2015; Hamari & Koivisto, 2015) . This especially pertains to the question whether gamification has the potential to positively influence students’ knowledge retention. In this study, we therefore investigate the suitability and potential of gamification to make sustainable transport education more appealing and effective. In particular, we aim to answer the question whether students memorize more knowledge when they participate in gamified full-day workshops in comparison to traditional (i.e., non-gamified) workshops. Therefore, an experiment was conducted comparing the knowledge retention performance of two groups of workshop participants in the area of sustainable transport.

The remainder of the paper is organized as follows. First, existing literature on the application and effects of gamification in education is reviewed. Next, we briefly outline our hypotheses, followed by a description of the methodology. Finally, the results are presented and the paper ends with a discussion as well as concluding thoughts. 2.

Gamification & Education

The term “gamification” was first used in 2008, but only gained widespread adoption in academia and the industry in 2010 (Thiebes et al., 2014) . The most common definition stems from Deterding et al. (2011, p. 9), who describe gamification as “the use of game design elements in non-game contexts”. Popular game design elements include points, badges, leaderboards, competition, immediate feedback, and time constraints (Deterding et al., 2011; Monu & Ralph, 2013) . It is noteworthy that the concept of gamification differs from educational and serious games. While gamification only employs game elements in a context that is primarily not connected to games or gameful design, the latter ones describe full-fledged games for non-entertainment purposes (e.g., education) (Dicheva et al., 2015) .

Despite its acknowledged positive effects in fields such as health (Schmidt-Kraepelin, Thiebes, Tran, & Sunyaev, 2018; Spil, Sunyaev, Thiebes, & van Baalen, 2017) , crowdsourcing (Morschheuser, Hamari, & Koivisto, 2016) or enterprise systems (Augustin, Thiebes, Lins, Linden, & Basten, 2016) gamification is increasingly being applied for educational purposes. The main objective of educational gamification is to motivate students to participate and engage more intensively in class, thereby improving learning effectiveness (Siemon & Eckardt, 2016) . Game elements need to be deployed in a way such that students are able to retain and apply the educational content in order to succeed in the game and to be able to apply their learning experience outside of the game context (Moore-Russo, Wiss, & Grabowski, 2017) . Existing empirical studies on gamification in education focus mainly on engagement and motivation as learners’ outcomes (Nah, Telaprolu, Ayyappa, & Eschenbrenner, 2014) . Research that answers the question whether gamification can lead to increased learning performance remains scarce until this day. Although gamified teaching techniques have been shown to be suitable in areas such as the military, retail organizations, computer service providers and manufacturing organizations (Kapp, 2012) , little research has been conducted on gamification and knowledge retention (Buckley & Doyle, 2017; Kapp, 2012) .

Hypotheses

An increase in knowledge is a major goal of all educational measures. It is therefore highly desirable that the content is fully understood and retained by the students for as long as possible after the educational event (Ebbinghaus, 1913; Murre & Dros, 2015) . In this paper “short term” refers to a period of about 20 minutes after the workshops and “long term” refers to two weeks after the workshops. Since both types of workshops characterize educational measures, it is assumed that both yield to an increase in knowledge. The learning curve of Ebbinghaus (1913) is a benchmark in learning literature for students’ replicability of learning content. It assumes a maximum amount of 100% of recall directly after a learning event and shows that memory retention is about 58% of the total knowledge after 20 minutes (which corresponds to the second assessment). In fact, after two weeks the retention rate is about 25% (Ebbinghaus, 1913; Murre & Dros, 2015) .

Gamification is frequently applied in marketing and education with the aim to encourage a specific behaviour and to increase engagement and motivation. Gamification has been used for teaching purposes to help educators broaden the variety of teaching methods to motivate students (Huang & Soman, 2013) . Gamification is intended to engage and motivate students in an interactive setting and to support them to remember and learn, leading to better memorization (de Sousa Borges, Durelli, Reis, & Isotani, 2014; Hamari & Keronen, 2017; Kapp, 2012) . Dicheva et al. (2015) conducted a literature review and a systemic mapping study about gamification in education and concluded that the majority of the reviewed papers found positive results from the influence of gamification in teaching. In fact, they found that students’ engagement and motivation was higher when gamification was used. Gamified teaching resulted in a higher use of forums through more active participation, a higher engagement in projects, an increased participation or attendance and a higher number of students who passed the course. Reiners et al. (2012) developed a framework on how gamification can be used for supply chain management education in order to increase students’ level of engagement and enjoyment of the courses. Dias (2017) conducted an experiment in an operations research class, comparing a non-gamified and a gamified group. They found positive results in the gamified class since the percentage of successful students as well as students’ participation in class increased (Dias, 2017) . Moreover, students evaluated the course itself more highly than the non-gamified students did. We therefore argue that gamification is especially suitable for fullday workshops as students need to stay concentrated and focused over a long period of time and face a huge amount of learning content. It is thus hypothesized that students in the gamification group are able to memorize more knowledge about sustainable transport than students in the non-gamified group.

H1: The gamified group achieves higher scores in knowledge than the non-gamified in the short term H2: The gamified group achieves higher scores in knowledge than the non-gamified in the long term

Methodology

An experimental design was used to investigate whether significant differences in knowledge exist between students who participated in a gamified full-day workshop and students who were not exposed to gamification. The questionnaire to measure students’ knowledge was developed together with the industry and experts from the educational sector and the industry (The questionnaire can be found in Table 3). It consisted of two single choice, two multiple choice, and three open ended questions with a maximum score of eleven points. The developed scale was pre-tested in three workshops with 131 students to ensure its understandability. To ensure comparability of the participants in terms of educational level, all participants were recruited from vocational schools in Austria in their second year. In Austria, an increasing number of students visit the vocational school also later in their career, which is reflected by the age distribution of the sample. The pilot study showed that separation of students within the same class into different testing groups led to social interaction threats in form of diffusion/imitation of treatment and resentful demoralization (Trochim et al., 2016) . The subjects of the experiment did not focus on the treatment they received, since they were deflected by their thoughts what happens in the other room e.g. ‘What are the colleagues doing in the other room?’. Thus, to reach a high level of internal validity in this pilot study, it was essential that students and teachers did not know that there is another group which receives a rival treatment (Trochim et al., 2016) . Since a pre-tests showed that the separation of students within classes is problematic, the classes were randomly assigned to either the non-gamified or the gamified group. The study included a gamified and a non-gamified group. Measurements were taken at three points in time: immediately before (observation 1: O1), 20 minutes after (O2), and two weeks after (O3) the workshops using identical questions.

The aim of the workshops was to train logistics students on sustainable transport by combining theoretical and practical knowledge. The gamified and non-gamified workshops had the same length, learning goals and equal supporting educational material. Both workshops were organized as full-day events from 9:45 am to 3:45 pm. The instructors of the workshops were the same for all workshops in order to support a comparability of the workshops. The program and the interactive tasks were the same in both workshops. Whereas the gamified workshops included motivational affordances (i.e., competition, leaderboards, badges, time constraints, storytelling, immediate feedback, rewards, clear goals, social interaction) (Warmelink, Koivisto, Mayer, Vesa, & Hamari, 2018) , the non-gamified did not include any game elements. For example, the students had to do the same calculation in each group, but received points for correct solutions in the gamified workshops. The gamified workshops were designed as a challenge in which groups of students had to complete specific tasks (i.e. the tasks were the same in the non-gamified group). The students received a certain number of points for each of the accomplished tasks such as solving a transport calculation or finding the correct solution in a LEGO simulation. The tasks were embedded in a story to use the motivational advantages of the game element story telling (Kapp, 2012) . Competition between the groups was encouraged by leaderboards. Grouping students into different teams was also intended to reduce the negative effects of competition on an individual level and to support social interactions (Sailer, Hense, Mandl, & Klevers, 2013) .

Results

In total, 334 students participated in the study, with 261 students assigned to the gamified group and 73 to the non-gamified group. The demographic statistics can be found in Table 1. The distribution of gender in the total sample was fairly balanced with 160 female and 174 male students. In the non-gamified group the students were older ( = 19.37,  = 4.151) than in the gamified group ( = 26.86,  = 9.899), since a class for returnees and retaining was randomly assigned to the non-gamified group. However, the differences in age had no effect on the level of knowledge the students had before the workshops as discussed in the next paragraph. A nonparametric Mann-Whitney U test showed that the knowledge level of the gamified and the nongamified group was equal (p = .39, U = 7,383.00) Since the assumption of normal distribution which is necessary for parametric tests was not fulfilled by the data set, non-parametric tests were used for this study. In addition, non-parametric test (e.g. a Mann-Whitney-U-Test) can be conducted with different group sizes, but it should be taken into account that the statistical power might be slightly diminished.

Gamified group Non-gamified group Mean/total group increased from 3.62 ( = 1.48) in O1 to 7.10 ( = 2.19) O2 and decreased to 5.39 ( = 2.13) in O3. In the non-gamified group, the scores were 3.47 ( = 1.77) in O1, increased to 5.49 ( = 2.17) in O2 and dropped to 4.97 ( = 1.77) in O3.

A dependent sample Wilcoxon signed-rank test was used to test whether the knowledge improvements between O1 – O2 (short term) and O1 – O3 (long term) were significant. The results show a significant difference between O1 and O2 (Z = -11.972, p<.01) and between O1 and O3 (Z = -9.127, p<.01) for the gamified group. Results for the non-gamified group were similar (Z = -6.378, p<.01 for O1 to O2 and Z = -3.498, p<.01 for O1 to O3). A non-parametric independent sample Mann-Whitney U test showed that the scores in the first assessment were not significantly different between the gamified and non-gamified group (U = 7,883, p =.390). In the second assessment, the gamified group outperformed the non-gamified group (U = 4,582.50, p < .01, H1 supported). In the third assessment, the mean value in the gamified group (=5.39) was higher than in the non-gamified group (=4.97), but no significant difference between the groups was found (U = 3,3357, p = .114, H2 rejected). An Analysis of frequency shows that 69.7% of the gamified group achieved more than six points in assessment 2, as opposed to 35.5% of the non-gamified group. 9.9% of the gamified group achieved ten or eleven points (out of a maximum of 11 points), as opposed to 1.5% of the non-gamified group. In assessment 3, no student of the non-gamified group achieved even nine points and 6.4% of the students achieved nine or more points in the gamified group. 29.5% of the gamified and 24.3 % of the non-gamified group achieved more than six points in assessment 3.

Discussion, Limitations and Conclusion

In this study, we investigated whether the utilization of game elements in full-day workshops leads to increased knowledge retention. The results of our study show that both types of workshops lead to a significant increase in short- and long-term knowledge as students’ knowledge increased substantially directly after the experiment and remained at a high level after a sustained period. Furthermore, the gamification group clearly outperformed the nongamified group and showed significantly higher scores in short-term knowledge which indicates that gamification can be especially suitable to increase short-term knowledge memorization. In addition, analysis of the frequency and the better mean values of the gamified group in the third assessment indicate that the participants of the gamified workshops also memorized the knowledge slightly better in the long-term even though we did not find any significant difference in long-term knowledge retention.

In summary, our results indicate that gamification can be a suitable approach to increase knowledge retention and thus influence the effects described in the forgetting curve by Ebbinghaus (1913) . However, it is important to mention that we treated gamification in this study as a black box and did not investigate the underlying mechanisms that explain its fostering effects on knowledge retention. Possible explanations can be found in extant research. For example, (Mullins & Rajiv, 2018) strongly argue that gamification can trigger emotions that have positive effects on knowledge retention. With regard to practical implications, we strongly encourage educators that conduct demanding and exhausting all-day workshops to implement meaningful gamification concepts in order to foster students’ knowledge retention levels. When designing gamified workshops, the didactical methods for the contents have to be chosen wisely regarding the level of the target group. Following the flow theory of Csikszentmihalyi (1996) , the information and learning aims must be adopted due to age and educational background of the participants to meet an adequate level of difficulty for the workshops, which is not too easy and not too hard. Moreover, the role of the instructors is highly important since their motivation and knowledge is needed to attract the participants of the workshops for a certain topic. This study has four main limitations which influence its generalizability. First, the study was conducted as a face-to-face course with information system support, so generalizability to online courses or blended courses is limited. Second, the whole classes, instead of individuals, were randomly assigned to the gamified or non-gamified group since the pre-test showed that a separation of groups influences students’ behaviour and increases social threats (Trochim, Donnelly, & Arora, 2016) . Third, the study was conducted in Austria and may not hold across cultures. Fourth, this study was applied in the area of logistics and further research would be necessary to investigate if the application of gamified workshops would also improve knowledge retention in other areas.

Our study has multiple opportunities for future research. First, as stated above we treat gamification as a black box approach since we only take a look at gamification as input and knowledge retention as the educational output. Future studies could delve deeper and aim to find more profound explanations on the positive effects of gamification. A possible approach would be to include and test the effect of gamification on emotion and further constructs, many of which can be found in the existing literature (e.g., enjoyment, intrinsic motivation). In addition, the motivational and learning effects over a longer time span of gamification are another topic of interest. Moreover, investigating differences between gender (Koivisto & Hamari, 2014; Riquelme & Rios, 2010) and school types as well as learning types (e.g. Yanuschik, Pakhomova, & Batbold, 2015) might be another direction for further research. Finally, researchers often suggest that the positive effects of gamification in education are not only limited to its ability to improve knowledge memorization but also enhance skills such as problem solving, collaboration, and communication (Dicheva et al., 2015) . Thus, future research might also have a closer look at how gamification can facilitate social dynamics and thus provides experiences and soft skills that prepare students for their later work life (MooreRusso et al., 2017).

Acknowledgements

This research study is part of the research field ‘sustainable transport systems,’ which was funded by the State of Upper Austria as part of the research program ‘FTI Struktur Land Oberösterreich’. We would like to thank our project partners: the port of Enns and the participating Austrian schools.

Appendix

Knowledge

Question Which is the largest European inland port in terms of total cargo volume? (single choice) What percentage of the modal split was used for inland waterway transport in Europe in 2014? (open question: 0-100%) Which of the following types of goods are appropriate for inland waterway transport? (multiple choice) How much percent of the potential cargo volume of the Danube are currently used for freight transport? (open question: 0-100%) 15% Which of the following key characteristics describe the new logistics concept of ‘synchromodality’? (multiple choice) What was the total cargo volume transported in 2014 in the European Union on inland waterways? (single choice) How many trucks are substituted by one common inland vessel of the Danube? (open question) Duisport 7 %  ores and metal waste  agricultural and forestry

products  petroleum products  fast moving consumer goods a) cooperative network b) real-time switching between

transport modes c) flexibility d) unimodal transport 550 million tons 280 trucks

Grading 1 point for correct answer 1 point for correct answer 1 point = one answer right 2 points = two answers right; 3 points = all answers right (all scores = 3 points) 1 point deduction for every incorrect answer 1 point for correct answer 1 point for correct answer 1 point for correct answer

References

Augustin , K. , Thiebes , S. , Lins , S. , Linden , R. , & Basten , D. ( 2016 ). Are we Playing Yet? A Review of Gamified Enterprise Systems . PACIS 2016 Proceedings, 2.

Buckley , P. , & Doyle , E. ( 2017 ). Individualising gamification: an investigation of the impact of learning styles and personality traits on the efficacy of gamification using a prediction market . Computers & Education , 106 , 43 - 55 .

Csikszentmihalyi , M. ( 1996 ). Flow and the Psychology of Discovery and Invention . New York, NY: Harper Collins.

De Sousa Borges , S. , Durelli , V. H. S. , Reis , H. M. , & Isotani , S. ( 2014 ). A systematic mapping on gamification applied to education . Proceedings of the 29th Annual ACM Symposium on Applied Computing , 216 - 222 .

Deterding , S. , Dixon , D. , Khaled , R. , & Nacke , L. ( 2011 ). From game design elements to gamefulness: defining gamification . 15th International Academic MindTrek Conference: Envisioning Future Media , 9 - 15 .

Dias , J. ( 2017 ). Teaching operations research to undergraduate management students: the role of gamification . International Journal of Management Education , 15 ( 1 ), 98 - 111 .

Dicheva , D. , Dichev , C. , Agre , G. , & Angelova , G. ( 2015 ). Gamification in education: a systematic mapping study . Journal of Educational Technology & Society , 18 ( 3 ), 75 - 88 .

Ebbinghaus , H. ( 1913 ). Memory: a Contribution to Experimental Psychology . New York City: Teachers College, Columbia University.

Hamari , J. , & Keronen , L. ( 2017 ). Why do people play games? A meta-analysis . International Journal of Information Management , 37 ( 3 ), 125 - 141 .

Hamari , J. , & Koivisto , J. ( 2015 ). Why do people use gamification services? International Journal of Information Management , 35 ( 4 ), 419 - 431 .

Huang , W. H.-Y. , & Soman , D. ( 2013 ). Gamification of Education . Toronto, ON: Rotman School of Management.

Kapp , K. M. ( 2012 ). The Gamification of Learning and Instruction: Game-Based Methods and Strategies for Training and Education . San Francisco, CA: Pfeiffer.

Koivisto , J. , & Hamari , J. ( 2014 ). Demographic differences in perceived benefits from gamification . Computers in Human Behavior , 35 , 179 - 188 .

Lee , J. J. , & Hammer , J. ( 2011 ). Gamification in education: what, how, why bother . Academic Exchange Quarterly , 15 ( 2 ), 146 - 151 .

Monu , K. , & Ralph , P. ( 2013 ). Beyond gamification: implications of purposeful games for the information systems discipline . arXiv preprint arXiv:1308 . 1042 .

Moore-Russo , D. , Wiss , A. , & Grabowski , J. ( 2017 ). Integration of gamification into course design: a noble endeavor with potential pitfalls . College Teaching , 66 ( 1 ), 3 - 5 .

Morschheuser , B. , Hamari , J. , & Koivisto , J. ( 2016 ). Gamification in crowdsourcing: a review . Proceedings of the 49th Hawaii International Conference on System Sciences , 4375 - 4384 .

Mullins , J. K. , & Rajiv , S. ( 2018 ). Beyond enjoyment: a cognitive-emotional perspective of gamification . Proceedings of the 51st Hawaii International Conference on System Sciences , 1237 - 1246 .

Murre , J. M. J. , & Dros , J. ( 2015 ). Replication and analysis of Ebbinghaus' forgetting curve . PloS One , 10 ( 7 ), 1 - 23 .

Nah , F. F.-H. , Telaprolu , V. R. , Ayyappa , A. P. , & Eschenbrenner , B. ( 2014 ). Gamification of education: a review of literature . International Conference on HCI in Business , 401 - 409 .

Reiners , T. , Wood , L. C. , Chang , V. , Gütl , C. , Herrington , J. , Teräs , H. , & Gregory , S. ( 2012 ). Operationalising gamification in an educational authentic environment . Proceedings of the IADIS International Conference on Internet Technologies & Society (ITS 2012 ), 93 - 100 .

Riquelme , H. E. , & Rios , R. E. ( 2010 ). The moderating effect of gender in the adoption of mobile banking . International Journal of Bank Marketing , 28 ( 5 ), 328 - 341 .

Sailer , M. , Hense , J. , Mandl , H. , & Klevers , M. ( 2013 ). Psychological perspectives on motivation through gamification . Interaction Design and Architecture(s) Journal , 19 , 28 - 37 .

Schmidt-Kraepelin , M. , Thiebes , S. , Tran , M. C. , & Sunyaev , A. ( 2018 ). What's in the Game? Developing a taxonomy of gamification concepts for health apps . 51st Hawaii International Conference on System Sciences , 1218 - 1226 .

Siemon , D. , & Eckardt , L. ( 2016 ). Gamification of Teaching in Higher Education . In Gamification. New York NY: Springer Berlin Heidelberg.

Spil , T. , Sunyaev , A. , Thiebes , S. , & Van Baalen, R. ( 2017 ). The adoption of wearables for a healthy lifestyle: can gamification help? . Proceedings of the 50th Hawaii International Conference on System Sciences , 3617 - 3626 .

Thiebes , S. , Lins , S. , & Basten , D. ( 2014 ). Gamifying information Systems - a synthesis of gamification mechanics and dynamics . Twenty Second European Conference on Information Systems.

Trochim , W. M. K. , Donnelly , J. P. , & Arora , K. ( 2016 ). Research Methods: The Essential Knowledge Base . Boston, MA: Cengage Learning.

Warmelink , H. , Koivisto , J. , Mayer , I. S. , Vesa , M. , & Hamari , J. ( 2018 ). Gamification of the work floor: a literature review of gamifying production and logistics operations . In 51st Hawaii International Conference on System Sciences 1108-1116.

Yanuschik , O. V. , Pakhomova , E. G. , & Batbold , K. ( 2015 ). E-learning as a way to improve the quality of educational for Iiternational students . Procedia - Social and Behavioral Sciences , 215 , 147 - 155 .