outcomes of virtual vs. in-person gamified workshops: A pre-post survey experiment

Abstract

In response to the COVID-19 pandemic, the use of gamification in education intensified as it allows for higher levels of participation in online environments, which can increase student motivation through greater interaction. However, there is little research on using the same gamification elements in different settings. Therefore, we explore the different cognitive learning outcomes of participants in virtual and in-person gamified workshops. We developed a gamified workshop concept on sustainable transport logistics that was used for virtual and inperson settings in Austria. To ensure the quality of our workshops, we have introduced a prepost experiment where learning effects are measured by a survey. The main findings show that gamification has proven to be a useful pedagogical strategy to achieve cognitive learning outcomes in in-person and virtual workshops. However, the cognitive learning outcome differs significantly between the participants of virtual vs. in-person gamified workshops. Our study shows that more attention needs to be paid to gamification in virtual vs. in-person settings to maximize the cognitive learning outcome of users with different demographic characteristics. Gamified workshops, gamification in education, cognitive learning outcome, pre-post-survey,

Keywords

• educators to ensure quality of the teaching content

-

experiment, virtual vs. in-person

1. Introduction

As a result of the COVID 19 pandemic, education experienced significant interventions and transformation. One of the most influential adjustment was the replacement of in-person teaching environments with virtual environments due to lockdowns and limitations of personal contact around the world [ 1 ]. Millions of students were negatively affected by the restriction of inperson interactions, leading to mental health consequences such as despair, insecurity or anxiety [ 2 ]. According to the United Nations more than 87% of the global student population in 165 countries were influenced by the temporary closure of educational institutions [ 3 ]. The transition to a virtual environment included challenges such as a lack of preparation time for

2023 Copyright for this paper by its authors. Use permitted under Creative and a lack of technical infrastructure [ 4 ]. While technology provides students with extensive access to information and encourages the creation and sharing of knowledge, educators were required to identify

ways to foster students’ motivation and engagement without in-person interaction [ 5 ]. It was necessary to develop attractive teaching strategies that increase students' motivation and engagement as well as maximize their knowledge acquisition [ 6 ]. In recent years, gamification has proven to be an effective way to increase student motivation and knowledge. It involves using game elements such as points, badges, and leaderboards in a nongaming context to

motivate students. [ 7–9 ].

Several studies examined the effectiveness of gamification in virtual environments before the COVID 19 pandemic [ 7, 8, 10 ], which focused on gamification in e-learning environments. During the pandemic, [ 11 ] analyzed gamification in an academic environment focusing on differences between engineering and social science students. Using a mixed methods approach, with qualitative interviews and a quantitative survey, the authors showed that gamification is a valuable pedagogical strategy to increase student engagement in an academic context. An important finding of the study is that in virtual gamified courses, teachers should encourage students to turn on their cameras to foster collaboration. Another study by [ 12 ] used gamification during the pandemic to examine differences between a gamified flipped course, a non-gamified online flipped course, and a gamified traditional online course. Contrary to the expectations on the impact of gamification, students in the non-gamified class performed significantly better than students in the two gamified online classes. The authors found that participation was affected due to poor networking and communication in online classes. In addition, they noted that technical support, professional training for teachers, and strengthening students' sense of belonging to their classes are necessary to ensure quality of gamified learning in virtual environments. Conversely, a study by [ 6 ] using a quantitative survey of 140 primary and secondary school students discovered that virtual gamification had a positive impact on learner motivation and proposed that gamification can be used as a method to achieve the United Nations Sustainable Development Goal 4 of ensuring quality education.

The contrasting results of these papers encourage further research on gamification in virtual environments [ 5, 7, 11 ]. Thus, we set the focus of this paper on investigating differences of cognitive learning outcomes between virtual vs. in-person gamified workshops. We evaluate the performance of participants regarding knowledge retention of facts. This kind of cognitive learning outcome (CLO) is referred to as conceptual knowledge [ 13 ]. This study is guided by the research question: What differences in cognitive learning outcomes exist in virtual vs. in-person gamified workshops?

The structure of this paper is as follows: Section 2 describes the theoretical background that provides our hypotheses. Section 3 describes the research methodology used for this study including the gamified workshop design. The results of the quantitative survey are presented in Section 4. Section 5 provides the conclusions drawn from our results and an outlook for future research.

2. Theoretical background

Previous research has demonstrated that gamification is broadly used to design learning environments that aim at positive experiences such as a higher level of students’ learning and/or motivation [ 14 ] as well as better knowledge retention [ 15, 16 ]. Findings from an experiment that compares non-gamified and gamified groups found that gamification yielded positive results including an increase in class participation and course success [ 17 ]. One of the latest conducted meta-analyses investigated the effects of gamification on CLO and found a stable positive impact in the reviewed studies [ 15 ]. Summarizing the findings of earlier research, we propose that gamification, in the context of a gamified workshop, is an appropriate measure to increase CLO in both, virtual and in-person settings. Hence, we formulate the following hypotheses:

H1: The CLO increased after attending a gamified workshop.

H1a: The CLO increased after attending an inperson gamified workshop.

H1b: The CLO increased after attending a virtual gamified workshop.

Numerous studies have demonstrated that gamification is an effective tool to increase student motivation [ 7–9 ]. Nonetheless, the impact of gamification in online learning environments remains controversial. Multiple studies have identified motivational problems linked to gamification in virtual settings, implying that students may not be as engaged or motivated as they would be in an in-person classroom setting [ 5, 7, 8, 10 ]. As such, it is imperative to examine whether gamification can effectively increase learners' motivation, and therefore knowledge retention, in online learning settings. Based on this premise, we put forward the following hypothesis:

H2: The CLO is higher when attending a gamified workshop in person than virtually.

Our gamified workshop design is not targeted at a certain age group and is used in various settings, predominantly in schools for higher education, vocational schools, and adult education, resulting in a wide range of participant age. Previous research on the effects of age on CLO in gamification is inconsistent. Some researchers have found that there tends to be a gap in the adoption, motivation and learning effects of gamification between different age groups [ 18– 20 ]. As the gamified workshop involves the use of digital gamification applications (e.g., an augmented reality app), it is important to explore whether a virtual setting, in conjunction with digital gamification components, produces agerelated differences in CLO. Prior research has identified digital components as potential barriers for older participants, warranting a focus on agerelated differences between the younger and older generations [ 18, 19 ]. To test the effect of age on the cognitive learning outcome, we differentiate between digital natives referring to individuals born into the digital age, and digital immigrants who learned to use technology later in life [ 19 ]. To delineate the terms digital natives and digital immigrants, we follow existing research and divide participants based on age. Specifically, we define digital natives as individuals aged 17 to 24 belonging to the older cohort of Generation Z and those up to 30 years old belonging to Generation Y (born after 1991). By implication, digital immigrants are participants over the age of 30 [ 19, 21–23 ]. Given the various ages of our workshop participants and the use of digital game elements in both workshop types, we evaluate CLO regarding age as follows:

H3: The level of CLO after attending the gamified workshop differs between digital natives and digital immigrants.

H3a: The level of CLO differs between digital natives and digital immigrants after in-person gamified workshops.

H3b: The level of CLO differs between digital natives and digital immigrants after virtual gamified workshops.

Research studies on gender differences and CLO show conflicting results. On the one hand it is indicated by [ 24 ] that female students outperform male students in terms of knowledge gains. On the other hand [ 25 ] did not find significant differences between genders. Another study argues that discrepancies between genders are converging [ 26 ]. Considering the conflicting results concerning the moderating effect of gender on CLO, we hypothesize no gender bias:

H4: There are no differences in CLO after the gamified workshops regarding genders.

H4a: There are no differences in CLO after attending the gamified workshops in person.

H4a: There are no differences in CLO after attending the gamified workshops virtually.

3. Methodology

3.1. Gamified workshop design

The objective of the gamified workshops is to educate the participants about sustainable transport. This topic was chosen for its appropriateness in evaluating the efficacy of gamification in facilitating learning. Prior research revealed a lack of knowledge among individuals of diverse ages, genders, and educational backgrounds regarding sustainable transport [ 27, 28 ]. Moreover, it is a domain of significance for scientific inquiry, as sustainable transport has the potential to mitigate carbon emissions and promote sustainable modes of transportation according to the European Green Deal goals [ 29 ].

The target audience originates from varying educational levels, i.e., vocational schools or higher educational institutions in Austria, and age groups, ranging from students starting at the age of 14 to participants of adult education programs. To exclude the possibility of research bias, all gamified workshops were held by the same instructor group. The agenda and the gamified elements are identical in both workshop types to allow profound comparability. A detailed schedule can be found in Table 1. Both workshop types last 3.5 hours. The agenda items were supported by various media and digital formats using image-rich presentation slides for the interactive lecture, the online quiz tool "Kahoot", the publicly available augmented reality app "Logistify" [ 19 ] and the adaptation of the game "Activity" for logistics jobs.

Reduced interpersonal interaction is the main difference between the virtual and in-person workshops, e.g., limited eye contact, leading to a reduced possibility of checking the attention of the participants. Since previous studies point out motivational issues within online environments, as stated by [ 5, 7 ], the participants were asked to turn on their cameras during the virtual gamified workshops as recommended by [ 11 ]. 3.2.

Pre-post-survey experiment

We studied the effect of gamified workshops with quantitative methods using an experimental study comparing virtual and in-person gamified workshops. A one-group pre-test-post-test design was used for this study. First, participants completed a pre-test, then the treatment took place through a virtual or in-person gamification workshop, followed by the post-test. For both measurements, we used an online questionnaire. The structured questionnaire contained closed questions on socio-demographic characteristics and knowledge questions to measure performance of the participants regarding CLO shown in Table 2. The answers of the respondents were evaluated using a 7-point Likert scale. The CLO measurement contains two multiple-choice questions (CLO1, CLO2) with four possible options and six single-choice questions with four options (CLO3 to CLO8). Therefore, a maximum score of 14 could be achieved. 3.3.

Data set and analysis

The 17 gamified workshops were conducted between January 2021 and November 2022 in Austria. In total, 428 questionnaires were completed (233 pre-survey, 195 post-survey). For the data analysis, the answers were transferred into a data set. Data cleaning and transformation were done using Microsoft Excel and the statistical evaluation was done using Jamovi. Table 3 shows the sample sizes per workshop type and the drop-out quote. The drop-out in the virtual workshop (29.1%) was remarkably higher than in the in-person setting (8.8%). No categorical differences between respondents and nonrespondents were collected. We checked whether the participants quit the survey at a specific question. It showed that no specific question is responsible for quitting.

To assign pre- and post-answers for a single person, the anonymous questionnaire asked for the first three letters of the first name, the month of birth and the age. In combination with the date of the workshop attended, this results in an ID code for the pairs of answers. Through this procedure, the answers remain anonymous but can be paired. First, a pivot table is set up based on the distinction of pre and post answers. The resulting table contains separate columns (variables) for all CLO items in the pre- and postsurvey (CLO1pre, CLO2post, …, CLO8pre, CLO8post) with the achieved score as data values. A total score was calculated as well resulting in the variables CLOsumpre and CLOsumpost. The total score variables are used to assess hypothesis 1 to 4. Additionally, we analyze the individual items to determine in which parts of the gamified workshop less or no knowledge is retained. As a result, individual parts of the gamified workshop can be specifically adapted in future.

The next step was to group the answers regarding the pair ID. This resulted in a table of 261 pre-post-pairs of answers (rows) and 22 variables (pair ID, gender, age group, workshop type, 18 CLO variables). 167 participants completed pre- and post-test both and 94 participants completed either pre- or post-test which results in 28 missing pre-tests and 66 missing post-tests. As the tests were filled out voluntarily by the participants, the missing values can be attributed to three reasons: Arriving late, leaving the workshop early or not being interested in participating in the evaluation.

To test significance levels, we use a confidence interval of 95% with Student’s t-test for paired samples tests and Welch’s t-test for independent samples due to better reliability in cases of unequal variances and sample sizes of the groups. Another advantage of the Welch test is that a test for the normal distribution of the data is obsolete [ 30, 31 ]. Nevertheless, when there are more than 25 observations per group and no extreme outliers, the t-test is still an appropriate tool to analyze the moderately skewed distributions of the outcome variable [ 32 ].

4. Results

Out of the 261 participants, 101 (38.7%) were part of virtual gamified workshops and 160 (61.3%) took part in the in-person gamified workshops. Figures 1 and 2 show the boxplots for CLOsumpre and CLOsumpost respectively per workshop type. The boxplots show that the two groups have similar medians and means in the pretest, whereas they differ noticeably in the posttest. The following subchapters investigate the CLO further for virtual vs. in-person gamified workshops as well as differences in these two environments regarding gender and age group.

CLO for virtual vs. in-person gamified workshops

In total 167 participants completed both preand post-tests. For this sample, a paired samples t-test was used to measure whether the gamified workshop had a significant effect on the CLO by comparing the mean scores achieved in the prepost-tests. We evaluated the results of every single knowledge question as well as the sum achieved by each person. Student’s t is calculated despite a violation of normality criteria because the sample size is greater than 30 [ 33 ]. As shown in Table 5, all t-tests show significant results except for CLO3. No CLO was measured in CLO3 due to a positive value in the pre-post difference. However, since the total response score CLOsum significantly improved between the two measurements, we can interpret the result in favor of H1 and accept the increase in CLO after participation in the gamified workshops as statistically significant with p<0.001 (t=-17.94).

As a next step, we tested whether there is an increase of CLO in the virtual and in-person setting in order to accept or reject H1a and H1b. We separated the data set by workshop type and conducted paired samples t-tests likewise for H1. The results in Table 6 below show similar patterns as in Table 5. Despite a contradicting learning outcome in CLO3, we consider H1A to be statistically accepted due to a p-value of smaller than 0.001 in the overall CLO result of the paired samples CLOsumpre and CLOsumpost with a mean difference of -3.05 (t = -15.94).

For the virtual group, the results of the paired samples t-test for H1b are shown in Table 7. The sample size consists of 46 complete pairs of pre and post-measurements. No increase in CLO could be detected for CLO3 and CLO4. However, in all other knowledge questions, as well as for CLOsum, a statistically significant difference could be identified. Thus, H1b can also be accepted.

Since the previously calculated mean values of CLOsumpost differ in the paired samples t-test, we suspect that the CLO is higher in the group of inperson participants than in the virtual group, which is addressed in H2. We assume that the two groups started at the same knowledge level since the means and variances of CLOsumpre in the pretest of virtual participants (µ=5.99 σ=1.90) and inperson participants (µ=6.03 σ=2.08) are similar and show no significant difference (p=0.865 t=0.171). Welch’s t of CLOsumpost reveals that after the gamified workshops, there is a significant difference with p<0.001 (t=-3.615) between the virtual and the in-person group. The virtual group achieved a lower mean score (µ=7.98 σ=1.88) than the in-person group (µ=9.12 σ= 2.35).

We additionally aimed at quantifying the gain of cognitive learning achieved by virtual versus in-person participants. We introduced a variable CLgain as follows: = − (1)

CLgain was calculated for pairs where both tests have been completed. This results in a sample size of 167 pairs, 46 virtual cases and 121 in-person cases. The descriptive statistics reveal that the virtual group has a mean CLgain of 2.37 (σ=1.88). The in-person group achieved a mean CLgain score of 3.05 (σ=2.10). We used a onetailed test as we have an effect assumption of CLgain being greater in the in-person group. The results indicate a significant difference in the group means with a p-value of 0.023 (t = -2.02).

Since both variables CLgain and CLOsumpost show significant differences between the groups, H2 is accepted, which stated that the CLO of the in-person group is higher than the CLO of the virtual group. 4.2.

Evaluation regarding age

As previously mentioned, we assume that there is a difference in CLO between varying age ranges, divided into digital natives and digital immigrants, due to the presence of digital gamification elements in our workshops. We evaluate H3, H3a and H3b by the variable CLOsumpost. The calculation of CLgain is not applicable in the case of the two age groups, as the two groups start the workshop with different levels of knowledge, i.e., CLOsumpre. We classified the participants by grouping persons 30 years or younger (n=122) and older than 30 years (n=73). The Welch’s t-test of CLOsumpost shows a significant difference between digital natives (µ=9.14 σ=2.10) and immigrants (µ=8.14 σ=2.41) with p=0.004 (t=2.95). This leads us to accept H3.

Next, we tested the in-person cases only which resulted in Welch’s t being statistically significant with p=0.001 (t=3.29). The digital natives achieved a mean CLOsumpost of 9.62 (n=87 σ=2.07) and digital immigrants 8.19 (n=47 σ=2.56). With these results, we can accept H3a.

The same procedure was carried out for testing H3b. The virtual participants older than 30 years achieved a higher CLOsumpost score (µ=8.04 σ=2.14) than the younger age group (µ=7.94 σ=1.68). We must reject hypothesis H3b due to the difference not being significant with p=0.851 (t=-0.188). 4.3.

Evaluation regarding gender

We test whether there is a difference in group means between female and male participants in their CLO. We assume that women and men started the workshop at the same knowledge level since the difference in CLOsumpre is not significant (p=0.424 t=-0.801). We tested for statistically significant disparity regarding the mean score CLOsumpost (127 women, 68 men) and the gain of knowledge CLgain (112 women, 55 men) between these genders using Welch’s t-test. The female group achieved a CLOsumpost score of µ=8.51 (σ=2.32) below the male group’s results of µ=9.24 (σ=2.10). The t-test shows a significant difference with p=0.029 and t=-2.208. For CLgain women have a mean score of µ=2.76 (σ=2.06) whereas for men it is µ=3.07 (σ=2.06) with the ttest being not significant (p=0.340, t=-0.924). This shows that the difference in CLgain between the two groups is not statistically significant and we conclude that there is no difference between the genders in terms of knowledge increase. However, since CLOsumpost shows a statistically significant difference between women and men, H4 must be rejected.

The evaluation of CLO for in-person gamified workshop cases shows a similar result regarding gender. The female group has a CLOsumpost of µ=8.73 (σ=2.46 n=86) and a CLgain of µ=2.91 (σ=2.11 n=78). The male group achieved in both measures higher mean scores with µ=9.81 (σ=1.96) for CLOsumpost (n=48) and µ=3.30 (σ=2.10) for CLgain (n=43). Again, the difference tested with Welch’s t is only significant for the variable CLOsumpost (p=0.006, t=-2.781) and not for CLgain (p=0.329, t=-0.982). This result leads us to rejecting H4a as well. For the knowledge gain no gender differences are found but for the CLO there is a significant gap in in-person gamified workshops.

Finally, we assessed CLO in virtual gamified workshops in terms of gender, with female sample sizes of 41 and 34 for CLOsumpost and CLgain, respectively, and male sample sizes of 20 and 12. Within the virtual participants, no statistically significant difference between these two genders was found. Although, it can be noted that the female group achieved slightly higher scores for CLOsumpost (µ=8.05 σ=1.94) and CLgain (µ=2.41 σ=1.94) than the male group for the two variables, respectively, with µ=7.85 (σ=1.79) and µ=2.25 (σ=1.76). The difference is not significant with p=0.693, t=0.397 for CLOsumpost and p=0.793, t=0.266 for CLgain. Consequently, we accept H4b which assumed no difference between genders in virtual gamified workshops.

5. Discussion and outlook

We have conducted a pre-post-survey experiment with gamified workshops in virtual and in-person environments to find out whether the setting has an influence on the CLO achieved by the participants. Table 8 gives an overview of the results of the hypothesis tests.

We found significant differences in CLO, which confirms H1, H1a and H1b. Furthermore, our results show that participants in face-to-face workshops have a significantly higher CLO than those in the virtual gamified setting (H2).

We argued that age may be an important variable that could further explain CLO differences between virtual and in-person settings due to the extensive use of digital gamified elements throughout our workshops, e.g., by an augmented reality app. Our results for H3 and H3a suggest that the generation a person grew up in, and thus their digital progress, has an influence on their CLO. Significantly different CLOs were achieved between the digital natives and immigrants in the in-person group. However, H3b provided contradictory results, showing no significant difference between the two age groups. Further research is needed to investigate the influence of age in virtual vs. in-person gamified environments.

Finally, our results show that there are gender differences in CLO in the in-person group, which is contrary to our expectations. We suspect other factors than workshop type that cause this result, such as personal interest in the subject area or outcome expectations. These and other factors were investigated for the subject area of logistics by [ 34 ], for example. On the other hand, no difference between genders was found in the virtual gamified workshop participants. Further research and evaluation of the workshop format need to be done to overcome this potential gender gap for in-person gamified workshops.

Our research sheds light on the differences of cognitive learning outcomes in virtual vs. inperson environments. We need to diminish these differences as digitalization is on the forefront. Strategies need to be developed to overcome the differences in CLO between virtual and in-person gamified workshops to ensure the effectiveness of gamified elements. Nevertheless, our study has limitations which are avenues for future research:

The research was limited to Austria, hence evaluation of CLO in gamified workshops in terms of virtual vs. in-person should be conducted in other countries as well. Furthermore, our sample shows a gender imbalance with more women than men which may could cause bias in our results. For data analysis, we used full prepost-pairs of answers and excluded incomplete responses. The drop-out rate shown in Table 3 in the virtual environment is problematic, urging for collecting information about non-respondents. The reasons for the high drop-out rate in the virtual workshop setting need to be evaluated systematically in future surveys to exclude a bias based on categorical differences between respondents and non-respondents.

We evaluate solely the cognitive learning outcomes without addressing the pedagogical challenges of using gamification in these different environments. The assessment of the measurement items of CLO suggests that parts of the gamified workshop may need further evaluation and improvement. In particular, CLO3 shows contradictory results.

For future research, it would be beneficial to further explore participants' familiarity with the use of digital components or devices to better understand the relationship between age differences and the terms "digital natives" and "digital immigrants." In this way, it would be an opportunity to determine whether or not agerelated differences are directly related to the use of digital devices. Added to this is a lack of guidelines for gamified online workshops, e.g., how to effectively adapt in-person gamified workshops to virtual ones. This would be of particularly value for all types of educational institutions to overcome obstacles in varying situations. The evaluation of our hypotheses revealed a difference between the virtual and inperson settings. It is proposed to further investigate the impact of reduced interpersonal interaction to subsequently define how to improve virtual gamified workshops.

6. Acknowledgements

This research study is part of the research cooperation ‘REWWay’ funded by viadonau.

7. References

References

[1] Barrera G , M.I. , D. Benalcazar Ch ., and D.C. San Lucas S, ” Gamification in the teaching of prevention measures for Covid-19” , IEEE Global Engineering Education Conference (EDUCON) , 2022 , pp. 1512 - 1516 . 10 .1109/EDUCON52537. 2022 . 9766821 .

[2] Gómez-García , G. , M. Ramos-Navas-Parejo , J. -C. de La Cruz-Campos , and C. RodríguezJiménez, ”Impact of COVID-19 on University Students: An Analysis of Its Influence on Psychological and Academic Factors”, International Journal of Environmental Research and Public Health , 19 ( 16 ), 2022 , pp. 1 - 2 . 10 .3390/ijerph191610433.

[3] https://www.unesco.org/en/articles/covid19-educational -disruption-and-response.

[4] Rincon-Flores , E.G. and B.N. SantosGuevara , ”Gamification during Covid- 19 : Promoting active learning and motivation in higher education” , Australasian Journal of Educational Technology , 37 ( 5 ), 2021 , pp. 43 - 60 . 10 .14742/ajet.7157.

[5] Nieto-Escamez , F.A. and M.D. RoldánTapia , ” Gamification as Online Teaching Strategy During COVID-19: A MiniReview” , FRONTIERS IN PSYCHOLOGY, 12 , 2021 , p. 648552 . 10 .3389/fpsyg. 2021 . 648552 .

[6] Park , S. and S. Kim , ” Is Sustainable Online Learning Possible with Gamification?-The Effect of Gamified Online Learning on Student Learning” , Sustainability, 13 ( 8 ), 2021 , p. 4267 . 10 .3390/su13084267.

[7] Jang , J. , J.J.Y. Park , and M.Y. Yi, ” Gamification of Online Learning” , in Artificial Intelligence in Education, C. Conati, N. Heffernan , A. Mitrovic , and M.F. Verdejo , Editors. 2015 . Springer International Publishing: Cham. 10 .1007/978-3- 319 -19773-9_ 82 .

[8] Looyestyn , J. , J. Kernot , K. Boshoff , J. Ryan , S. Edney , and C. Maher , ” Does gamification increase engagement with online programs? A systematic review” , PloS one , 12 ( 3 ), 2017 , e0173403 . 10 .1371/journal.pone. 0173403 .

[9] Acosta-Medina , J.K. , M.L. Torres-Barreto , and A.F. Cárdenas-Parga , ” Students' preference for the use of gamification in virtual learning environments” , Australasian Journal of Educational Technology , 2021 , pp. 145 - 158 . 10 .14742/ajet.6512.

[10] Antonaci , A. , R. Klemke , and M. Specht , ” The Effects of Gamification in Online Learning Environments: A Systematic Literature Review” , Informatics , 6 ( 3 ), 2019 , p. 32 . 10 .3390/informatics6030032.

[11] Rincon-Flores , E.G. , J. Mena , and E. LópezCamacho, ” Gamification as a Teaching Method to Improve Performance and Motivation in Tertiary Education during COVID-19: A Research Study from Mexico” , Education Sciences , 12 ( 1 ), 2022 , p. 49 . 10 .3390/educsci12010049.

[12] Ng , L. - K. and C.-K. Lo , ” Online Flipped and Gamification Classroom: Risks and Opportunities for the Academic Achievement of Adult Sustainable Learning during COVID-19 Pandemic” , Sustainability, 14 ( 19 ), 2022 , p. 12396 . 10 .3390/su141912396.

[13] Jong, T. de and M.G.M. Ferguson-Hessler , ”Types and qualities of knowledge”, Educational psychologist , 31 ( 2 ), 1996 , pp. 105 - 113 . 10 .1207/s15326985ep3102_ 2 .

[14] Oliveira , W. , J. Hamari , L. Shi , A.M. Toda , L. Rodrigues , P.T. Palomino , and S. Isotani , ” Tailored gamification in education: A literature review and future agenda” , Education and Information Technologies , 2022 , pp. 1 - 34 .

[15] Sailer , M. and L. Homner , ” The Gamification of Learning: a Meta-analysis” , Educational Psychology Review , 32 ( 1 ), 2020 , pp. 77 - 112 . 10 .1007/s10648-019- 09498-w.

[16] Putz , L. -M. , F. Hofbauer , and H. Treiblmaier , ” Can gamification help to improve education? Findings from a longitudinal study” , Computers in Human Behavior , 110 ( 106392 ), 2020 , 1 - 12 . 10 .1016/j.chb. 2020 . 106392 .

[17] Dias , J. , ” Teaching operations research to undergraduate management students: The role of gamification” , International Journal of Management Education , 15 ( 1 ), 2017 , pp. 98 - 111 . 10 .1016/j.ijme. 2017 . 01 .002.

[18] White , B.K. , A. Martin , and J. White, ” Gamification and older adults: Opportunities for gamification to support health promotion initiatives for older adults in the context of COVID-19”, The Lancet regional health . Western Pacific , 2022 , p. 100528 . 10 .1016/j.lanwpc. 2022 . 100528 .

[19] Putz-Egger , L. -M. , D. Beil , S. Dopler , and J. Diephuis , ” Combining Gamification and Augmented Reality to Raise Interest in Logistics Careers” , Applied sciences, 12 ( 18 ), 2022 . 10 .3390/app12189066.

[20] Bittner , J.V. and J. Schipper , ” Motivational effects and age differences of gamification in product advertising” , Journal of Consumer Marketing , 31 ( 5 ), 2014 , pp. 391 - 400 . 10 .1108/JCM-04-2014-0945.

[21] Dimock , M. , ” Defining generations: Where Millennials end and Generation Z begins” , Pew Research Center, 17 . January 2019 .

[22] Yamane , T. and S. Kaneko , ” Is the younger generation a driving force toward achieving the sustainable development goals? Survey experiments” , Journal of Cleaner Production , 292 , 2021 , p. 125932 . 10 .1016/j.jclepro. 2021 . 125932 .

[23] Wang , Q. , M.D. Myers , and D. Sundaram , ” Digital Natives and Digital Immigrants” , Business & Information Systems Engineering , 5 ( 6 ), 2013 , pp. 409 - 419 . 10 .1007/s12599-013-0296-y.

[24] van Houtte , M. , ” Why boys achieve less at school than girls: The difference between boys' and girls' academic culture”, Educational Studies , 30 ( 2 ), 2004 , pp. 159 - 173 .

[25] Haruna , H. , Z. Zainuddin , K. Okoye , R.R. Mellecker , X. Hu , S.K.W. Chu , and S. Hosseini , ” Improving instruction and sexual health literacy with serious games and gamification interventions: an outlook to students' learning outcomes and gender differences” , Interactive Learning Environments , 2021 , pp. 1 - 19 .

[26] Koivisto , J. and J. Hamari , ” Demographic differences in perceived benefits from gamification” , Computers in Human Behavior , 35 , 2014 , pp. 179 - 188 . 10 .1016/j.chb. 2014 . 03 .007.

[27] Lisa-Maria Putz , Horst Treiblmaier, and Sarah Pfoser, ” Using gamification for sustainable transport education: results from an empirical study” . 10 .5281/zenodo.1451345.

[28] Putz , L.-M. , M. Schmidt-Kraepelin , H. Treiblmaier , and A. Sunyaev , ” Gamification for a better future: the influence of gamified workshops on students' knowledge gains about sustainable transport” , in GamiFIN, GamiFIN, Tampere. 2018 : Tampere.

[29] European Commission, ”The European Green Deal”, Bruxelles, 2019 .

[30] Rasch , D. , K.D. Kubinger , and K. Moder , ” The two-sample t test: pre-testing its assumptions does not pay off” , Statistical Papers , 52 ( 1 ), 2011 , pp. 219 - 231 . 10 .1007/s00362-009-0224-x.

[31] Ruxton , G.D. , ” The unequal variance t-test is an underused alternative to Student's t-test and the Mann-Whitney U test” , Behavioral Ecology , 17 ( 4 ), 2006 , pp. 688 - 690 . 10 .1093/beheco/ark016.

[32] Le Cessie , S. , J.J. Goeman , and O.M. Dekkers , ” Who is afraid of non-normal data? Choosing between parametric and nonparametric tests” , European journal of endocrinology , 182 ( 2 ), 2020 , E1 - E3 . 10 .1530/EJE-19-0922.

[33] Stone , E.R. , ”t Test, Paired Samples”, in Encyclopedia of research design, N.J. Salkind, Editor. 2010 . SAGE Publications: Thousand Oaks, Calif., London, New Delhi, Singapore.

[34] Putz , L.-M. , V. Doppler , and V. Stockhammer , ” Gamified workshops in career choice: Gamification to reduce the lack of personnel in the logistics sector ”, 2022 .