=Paper= {{Paper |id=Vol-2186/paper5 |storemode=property |title=The influence of gamified workshops on students’ knowledge retention |pdfUrl=https://ceur-ws.org/Vol-2186/paper5.pdf |volume=Vol-2186 |authors=Lisa-Maria Putz,Manuel Schmidt-Kraepelin,Horst Treiblmaier,Ali Sunyaev |dblpUrl=https://dblp.org/rec/conf/gamifin/PutzSTS18 }} ==The influence of gamified workshops on students’ knowledge retention== https://ceur-ws.org/Vol-2186/paper5.pdf
            The influence of gamified workshops on students’ knowledge
                                      retention


                                                 Lisa-Maria Putz
                             University of Applied Sciences Upper Austria, Austria
                                           lisa-maria.putz@fh-steyr.at

                                           Manuel Schmidt-Kraepelin
                                   Karlsruhe Institute of Technology, Germany
                                       manuel.schmidt-kraepelin@kit.edu

                                                 Horst Treiblmaier
                                           MODUL University, Austria
                                          horst.treiblmaier@modul.ac.at

                                                   Ali Sunyaev
                                   Karlsruhe Institute of Technology, Germany
                                                sunyaev@kit.edu



       Abstract: Educators frequently face serious problems concerning students’ engagement and knowledge
       retention. As a proposed solution, gamification represents a new tool for active learning to increase
       students’ motivation and thus improve their learning results. The goal of this paper is to investigate the
       effects of gamification on short- and long-term knowledge retention in all-day workshops on sustainable
       transport. A longitudinal experiment with 334 logistics students was conducted comparing the results of
       gamified and non-gamified workshops with students as future managers. The results suggest that
       gamification is an effective measure to increase students’ learning outcomes with respect to sustainable
       transport.


       1.        Introduction

       Despite continuous efforts by education professionals to seek novel and innovate educational
       approaches, many students perceive traditional schooling as boring and ineffective (Lee &
       Hammer, 2011). Thus, they often lack motivation and engagement which negatively influences
       their learning performance. In fact, the forgetting curve by Ebbinghaus (1913) is still up to date
       and subject to intense scientific discussions about knowledge retention (Murre & Dros, 2015).
       It states that the vast majority of knowledge has been forgotten two weeks after acquisition. To
       create highly motivating learning environments that help to overcome the lack of student
       interest and increase students’ knowledge retention level, researchers and educators
       increasingly employ gamification in education (Dicheva, Dichev, Agre, & Angelova, 2015).

       Gamification is a promising approach to foster intrinsic motivation (Hamari & Keronen, 2017),
       make learning more engaging and increase students’ learning performance (Buckley & Doyle,
       2017; Kapp, 2012). By applying game design elements in non-game contexts (Deterding,
       Dixon, Khaled, & Nacke, 2011), gamification tries to take advantage of the growing passion
       for games (Thiebes, Lins, & Basten, 2014). However, empirical research on the effectiveness
       of gamification in educational environments and its influence on learning outcomes is still
       scarce (Dicheva et al., 2015; Hamari & Koivisto, 2015). This especially pertains to the question
       whether gamification has the potential to positively influence students’ knowledge retention. In


GamiFIN Conference 2018, Pori, Finland, May 21-23, 2018                                                             40
       this study, we therefore investigate the suitability and potential of gamification to make
       sustainable transport education more appealing and effective. In particular, we aim to answer
       the question whether students memorize more knowledge when they participate in gamified
       full-day workshops in comparison to traditional (i.e., non-gamified) workshops. Therefore, an
       experiment was conducted comparing the knowledge retention performance of two groups of
       workshop participants in the area of sustainable transport.

       The remainder of the paper is organized as follows. First, existing literature on the application
       and effects of gamification in education is reviewed. Next, we briefly outline our hypotheses,
       followed by a description of the methodology. Finally, the results are presented and the paper
       ends with a discussion as well as concluding thoughts.

       2.        Gamification & Education

       The term “gamification” was first used in 2008, but only gained widespread adoption in
       academia and the industry in 2010 (Thiebes et al., 2014). The most common definition stems
       from Deterding et al. (2011, p. 9), who describe gamification as “the use of game design
       elements in non-game contexts”. Popular game design elements include points, badges,
       leaderboards, competition, immediate feedback, and time constraints (Deterding et al., 2011;
       Monu & Ralph, 2013). It is noteworthy that the concept of gamification differs from educational
       and serious games. While gamification only employs game elements in a context that is
       primarily not connected to games or gameful design, the latter ones describe full-fledged games
       for non-entertainment purposes (e.g., education) (Dicheva et al., 2015).

       Despite its acknowledged positive effects in fields such as health (Schmidt-Kraepelin, Thiebes,
       Tran, & Sunyaev, 2018; Spil, Sunyaev, Thiebes, & van Baalen, 2017), crowdsourcing
       (Morschheuser, Hamari, & Koivisto, 2016) or enterprise systems (Augustin, Thiebes, Lins,
       Linden, & Basten, 2016) gamification is increasingly being applied for educational purposes.
       The main objective of educational gamification is to motivate students to participate and engage
       more intensively in class, thereby improving learning effectiveness (Siemon & Eckardt, 2016).
       Game elements need to be deployed in a way such that students are able to retain and apply the
       educational content in order to succeed in the game and to be able to apply their learning
       experience outside of the game context (Moore-Russo, Wiss, & Grabowski, 2017).

       Existing empirical studies on gamification in education focus mainly on engagement and
       motivation as learners’ outcomes (Nah, Telaprolu, Ayyappa, & Eschenbrenner, 2014).
       Research that answers the question whether gamification can lead to increased learning
       performance remains scarce until this day. Although gamified teaching techniques have been
       shown to be suitable in areas such as the military, retail organizations, computer service
       providers and manufacturing organizations (Kapp, 2012), little research has been conducted on
       gamification and knowledge retention (Buckley & Doyle, 2017; Kapp, 2012).

       3.        Hypotheses

       An increase in knowledge is a major goal of all educational measures. It is therefore highly
       desirable that the content is fully understood and retained by the students for as long as possible
       after the educational event (Ebbinghaus, 1913; Murre & Dros, 2015). In this paper “short term”
       refers to a period of about 20 minutes after the workshops and “long term” refers to two weeks
       after the workshops. Since both types of workshops characterize educational measures, it is
       assumed that both yield to an increase in knowledge. The learning curve of Ebbinghaus (1913)
       is a benchmark in learning literature for students’ replicability of learning content. It assumes a
       maximum amount of 100% of recall directly after a learning event and shows that memory
       retention is about 58% of the total knowledge after 20 minutes (which corresponds to the second


GamiFIN Conference 2018, Pori, Finland, May 21-23, 2018                                                      41
       assessment). In fact, after two weeks the retention rate is about 25% (Ebbinghaus, 1913; Murre
       & Dros, 2015).

       Gamification is frequently applied in marketing and education with the aim to encourage a
       specific behaviour and to increase engagement and motivation. Gamification has been used for
       teaching purposes to help educators broaden the variety of teaching methods to motivate
       students (Huang & Soman, 2013). Gamification is intended to engage and motivate students in
       an interactive setting and to support them to remember and learn, leading to better memorization
       (de Sousa Borges, Durelli, Reis, & Isotani, 2014; Hamari & Keronen, 2017; Kapp, 2012).
       Dicheva et al. (2015) conducted a literature review and a systemic mapping study about
       gamification in education and concluded that the majority of the reviewed papers found positive
       results from the influence of gamification in teaching. In fact, they found that students’
       engagement and motivation was higher when gamification was used. Gamified teaching
       resulted in a higher use of forums through more active participation, a higher engagement in
       projects, an increased participation or attendance and a higher number of students who passed
       the course. Reiners et al. (2012) developed a framework on how gamification can be used for
       supply chain management education in order to increase students’ level of engagement and
       enjoyment of the courses. Dias (2017) conducted an experiment in an operations research class,
       comparing a non-gamified and a gamified group. They found positive results in the gamified
       class since the percentage of successful students as well as students’ participation in class
       increased (Dias, 2017). Moreover, students evaluated the course itself more highly than the
       non-gamified students did. We therefore argue that gamification is especially suitable for full-
       day workshops as students need to stay concentrated and focused over a long period of time
       and face a huge amount of learning content. It is thus hypothesized that students in the
       gamification group are able to memorize more knowledge about sustainable transport than
       students in the non-gamified group.

       H1: The gamified group achieves higher scores in knowledge than the non-gamified in the short
       term
       H2: The gamified group achieves higher scores in knowledge than the non-gamified in the long
       term

       4.        Methodology

       An experimental design was used to investigate whether significant differences in knowledge
       exist between students who participated in a gamified full-day workshop and students who were
       not exposed to gamification. The questionnaire to measure students’ knowledge was developed
       together with the industry and experts from the educational sector and the industry (The
       questionnaire can be found in Table 3). It consisted of two single choice, two multiple choice,
       and three open ended questions with a maximum score of eleven points. The developed scale
       was pre-tested in three workshops with 131 students to ensure its understandability. To ensure
       comparability of the participants in terms of educational level, all participants were recruited
       from vocational schools in Austria in their second year. In Austria, an increasing number of
       students visit the vocational school also later in their career, which is reflected by the age
       distribution of the sample. The pilot study showed that separation of students within the same
       class into different testing groups led to social interaction threats in form of diffusion/imitation
       of treatment and resentful demoralization (Trochim et al., 2016). The subjects of the experiment
       did not focus on the treatment they received, since they were deflected by their thoughts what
       happens in the other room e.g. ‘What are the colleagues doing in the other room?’. Thus, to
       reach a high level of internal validity in this pilot study, it was essential that students and
       teachers did not know that there is another group which receives a rival treatment (Trochim et
       al., 2016). Since a pre-tests showed that the separation of students within classes is problematic,
       the classes were randomly assigned to either the non-gamified or the gamified group. The study


GamiFIN Conference 2018, Pori, Finland, May 21-23, 2018                                                       42
       included a gamified and a non-gamified group. Measurements were taken at three points in
       time: immediately before (observation 1: O1), 20 minutes after (O2), and two weeks after (O3)
       the workshops using identical questions.

       The aim of the workshops was to train logistics students on sustainable transport by combining
       theoretical and practical knowledge. The gamified and non-gamified workshops had the same
       length, learning goals and equal supporting educational material. Both workshops were
       organized as full-day events from 9:45 am to 3:45 pm. The instructors of the workshops were
       the same for all workshops in order to support a comparability of the workshops. The program
       and the interactive tasks were the same in both workshops. Whereas the gamified workshops
       included motivational affordances (i.e., competition, leaderboards, badges, time constraints,
       storytelling, immediate feedback, rewards, clear goals, social interaction) (Warmelink,
       Koivisto, Mayer, Vesa, & Hamari, 2018), the non-gamified did not include any game elements.
       For example, the students had to do the same calculation in each group, but received points for
       correct solutions in the gamified workshops. The gamified workshops were designed as a
       challenge in which groups of students had to complete specific tasks (i.e. the tasks were the
       same in the non-gamified group). The students received a certain number of points for each of
       the accomplished tasks such as solving a transport calculation or finding the correct solution in
       a LEGO simulation. The tasks were embedded in a story to use the motivational advantages of
       the game element story telling (Kapp, 2012). Competition between the groups was encouraged
       by leaderboards. Grouping students into different teams was also intended to reduce the
       negative effects of competition on an individual level and to support social interactions (Sailer,
       Hense, Mandl, & Klevers, 2013).

       5.        Results

       In total, 334 students participated in the study, with 261 students assigned to the gamified group
       and 73 to the non-gamified group. The demographic statistics can be found in Table 1. The
       distribution of gender in the total sample was fairly balanced with 160 female and 174 male
       students. In the non-gamified group the students were older ( = 19.37,  = 4.151) than in the
       gamified group ( = 26.86,  = 9.899), since a class for returnees and retaining was randomly
       assigned to the non-gamified group. However, the differences in age had no effect on the level
       of knowledge the students had before the workshops as discussed in the next paragraph. A non-
       parametric Mann-Whitney U test showed that the knowledge level of the gamified and the non-
       gamified group was equal (p = .39, U = 7,383.00) Since the assumption of normal distribution
       which is necessary for parametric tests was not fulfilled by the data set, non-parametric tests
       were used for this study. In addition, non-parametric test (e.g. a Mann-Whitney-U-Test) can be
       conducted with different group sizes, but it should be taken into account that the statistical
       power might be slightly diminished.

                                         Table 1: Demographic statistics
                                               Age                  Age                Gender
                                      mean, std. dev., number   mode, median      Female      Male
            Gamified group              18.37 (4.151), 261           17, 18        100        161
            Non-gamified group           26.86 (9.899), 73           16, 26         60         13
            Mean/total                  20.51 (6.780), 334           17, 18        160        174

       Table 2 shows the descriptive results of the knowledge measurements. Given the novelty of the
       topic of sustainable transport in both groups, the total mean values were initially quite low (
       = 3.59,  = 1.55), but improved immediately after the workshops ( = 6.74,  = 2.90). As
       expected, knowledge levels had declined after two weeks, ( = 5.33,  = 2.08), but the scores
       were still significantly better than those at their initial assessment. The values in the gamified


GamiFIN Conference 2018, Pori, Finland, May 21-23, 2018                                                     43
       group increased from 3.62 ( = 1.48) in O1 to 7.10 ( = 2.19) O2 and decreased to 5.39 ( =
       2.13) in O3. In the non-gamified group, the scores were 3.47 ( = 1.77) in O1, increased to 5.49
       ( = 2.17) in O2 and dropped to 4.97 ( = 1.77) in O3.

       A dependent sample Wilcoxon signed-rank test was used to test whether the knowledge
       improvements between O1 – O2 (short term) and O1 – O3 (long term) were significant. The
       results show a significant difference between O1 and O2 (Z = -11.972, p<.01) and between O1
       and O3 (Z = -9.127, p<.01) for the gamified group. Results for the non-gamified group were
       similar (Z = -6.378, p<.01 for O1 to O2 and Z = -3.498, p<.01 for O1 to O3).

            Table 2: Knowledge mean values and standard deviations across groups (max = 11)
                               Gamified group         Non-gamified group           Total
                               mean, std. dev.,        mean, std. dev.,      mean, std. dev.,
                                   number                  number                number
                     O1      3.62 (1.48), 240         3.47 (1.77), 66      3.59 (1.55), 306
                     O2      7.10 (2.19), 234         5.49 (2.17), 68      6.74 (2.9), 302
                     O3      5.39 (2.13), 207         4.97 (1.77), 37      5.33 (2.08), 244
                     Average 5.37                     5.24                 5.30

       A non-parametric independent sample Mann-Whitney U test showed that the scores in the first
       assessment were not significantly different between the gamified and non-gamified group (U =
       7,883, p =.390). In the second assessment, the gamified group outperformed the non-gamified
       group (U = 4,582.50, p < .01, H1 supported). In the third assessment, the mean value in the
       gamified group (=5.39) was higher than in the non-gamified group (=4.97), but no significant
       difference between the groups was found (U = 3,3357, p = .114, H2 rejected). An Analysis of
       frequency shows that 69.7% of the gamified group achieved more than six points in assessment
       2, as opposed to 35.5% of the non-gamified group. 9.9% of the gamified group achieved ten or
       eleven points (out of a maximum of 11 points), as opposed to 1.5% of the non-gamified group.
       In assessment 3, no student of the non-gamified group achieved even nine points and 6.4% of
       the students achieved nine or more points in the gamified group. 29.5% of the gamified and
       24.3 % of the non-gamified group achieved more than six points in assessment 3.

       6.        Discussion, Limitations and Conclusion

       In this study, we investigated whether the utilization of game elements in full-day workshops
       leads to increased knowledge retention. The results of our study show that both types of
       workshops lead to a significant increase in short- and long-term knowledge as students’
       knowledge increased substantially directly after the experiment and remained at a high level
       after a sustained period. Furthermore, the gamification group clearly outperformed the non-
       gamified group and showed significantly higher scores in short-term knowledge which
       indicates that gamification can be especially suitable to increase short-term knowledge
       memorization. In addition, analysis of the frequency and the better mean values of the gamified
       group in the third assessment indicate that the participants of the gamified workshops also
       memorized the knowledge slightly better in the long-term even though we did not find any
       significant difference in long-term knowledge retention.

       In summary, our results indicate that gamification can be a suitable approach to increase
       knowledge retention and thus influence the effects described in the forgetting curve by
       Ebbinghaus (1913). However, it is important to mention that we treated gamification in this
       study as a black box and did not investigate the underlying mechanisms that explain its fostering
       effects on knowledge retention. Possible explanations can be found in extant research. For
       example, (Mullins & Rajiv, 2018) strongly argue that gamification can trigger emotions that


GamiFIN Conference 2018, Pori, Finland, May 21-23, 2018                                                    44
       have positive effects on knowledge retention. With regard to practical implications, we strongly
       encourage educators that conduct demanding and exhausting all-day workshops to implement
       meaningful gamification concepts in order to foster students’ knowledge retention levels. When
       designing gamified workshops, the didactical methods for the contents have to be chosen wisely
       regarding the level of the target group. Following the flow theory of Csikszentmihalyi (1996),
       the information and learning aims must be adopted due to age and educational background of
       the participants to meet an adequate level of difficulty for the workshops, which is not too easy
       and not too hard. Moreover, the role of the instructors is highly important since their motivation
       and knowledge is needed to attract the participants of the workshops for a certain topic.

       This study has four main limitations which influence its generalizability. First, the study was
       conducted as a face-to-face course with information system support, so generalizability to
       online courses or blended courses is limited. Second, the whole classes, instead of individuals,
       were randomly assigned to the gamified or non-gamified group since the pre-test showed that
       a separation of groups influences students’ behaviour and increases social threats (Trochim,
       Donnelly, & Arora, 2016). Third, the study was conducted in Austria and may not hold across
       cultures. Fourth, this study was applied in the area of logistics and further research would be
       necessary to investigate if the application of gamified workshops would also improve
       knowledge retention in other areas.

       Our study has multiple opportunities for future research. First, as stated above we treat
       gamification as a black box approach since we only take a look at gamification as input and
       knowledge retention as the educational output. Future studies could delve deeper and aim to
       find more profound explanations on the positive effects of gamification. A possible approach
       would be to include and test the effect of gamification on emotion and further constructs, many
       of which can be found in the existing literature (e.g., enjoyment, intrinsic motivation). In
       addition, the motivational and learning effects over a longer time span of gamification are
       another topic of interest. Moreover, investigating differences between gender (Koivisto &
       Hamari, 2014; Riquelme & Rios, 2010) and school types as well as learning types (e.g.
       Yanuschik, Pakhomova, & Batbold, 2015) might be another direction for further research.
       Finally, researchers often suggest that the positive effects of gamification in education are not
       only limited to its ability to improve knowledge memorization but also enhance skills such as
       problem solving, collaboration, and communication (Dicheva et al., 2015). Thus, future
       research might also have a closer look at how gamification can facilitate social dynamics and
       thus provides experiences and soft skills that prepare students for their later work life (Moore-
       Russo et al., 2017).

       Acknowledgements

       This research study is part of the research field ‘sustainable transport systems,’ which was
       funded by the State of Upper Austria as part of the research program ‘FTI Struktur Land
       Oberösterreich’. We would like to thank our project partners: the port of Enns and the
       participating Austrian schools.




GamiFIN Conference 2018, Pori, Finland, May 21-23, 2018                                                     45
       References

       Augustin, K., Thiebes, S., Lins, S., Linden, R., & Basten, D. (2016). Are we Playing Yet? A Review
          of Gamified Enterprise Systems. PACIS 2016 Proceedings, 2.
       Buckley, P., & Doyle, E. (2017). Individualising gamification: an investigation of the impact of
          learning styles and personality traits on the efficacy of gamification using a prediction market.
          Computers & Education, 106, 43–55.
       Csikszentmihalyi, M. (1996). Flow and the Psychology of Discovery and Invention. New York, NY:
          Harper Collins.
       De Sousa Borges, S., Durelli, V. H. S., Reis, H. M., & Isotani, S. (2014). A systematic mapping on
          gamification applied to education. Proceedings of the 29th Annual ACM Symposium on Applied
          Computing, 216–222.
       Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to
          gamefulness: defining gamification. 15th International Academic MindTrek Conference:
          Envisioning Future Media, 9–15.
       Dias, J. (2017). Teaching operations research to undergraduate management students: the role of
          gamification. International Journal of Management Education, 15(1), 98–111.
       Dicheva, D., Dichev, C., Agre, G., & Angelova, G. (2015). Gamification in education: a systematic
          mapping study. Journal of Educational Technology & Society, 18(3), 75–88.
       Ebbinghaus, H. (1913). Memory: a Contribution to Experimental Psychology. New York City:
          Teachers College, Columbia University.
       Hamari, J., & Keronen, L. (2017). Why do people play games? A meta-analysis. International Journal
          of Information Management, 37(3), 125–141.
       Hamari, J., & Koivisto, J. (2015). Why do people use gamification services? International Journal of
          Information Management, 35(4), 419–431.
       Huang, W. H.-Y., & Soman, D. (2013). Gamification of Education. Toronto, ON: Rotman School of
          Management.
       Kapp, K. M. (2012). The Gamification of Learning and Instruction: Game-Based Methods and
          Strategies for Training and Education. San Francisco, CA: Pfeiffer.
       Koivisto, J., & Hamari, J. (2014). Demographic differences in perceived benefits from gamification.
          Computers in Human Behavior, 35, 179–188.
       Lee, J. J., & Hammer, J. (2011). Gamification in education: what, how, why bother. Academic
          Exchange Quarterly, 15(2), 146–151.
       Monu, K., & Ralph, P. (2013). Beyond gamification: implications of purposeful games for the
          information systems discipline. arXiv preprint arXiv:1308.1042.
       Moore-Russo, D., Wiss, A., & Grabowski, J. (2017). Integration of gamification into course design: a
          noble endeavor with potential pitfalls. College Teaching, 66(1), 3–5.
       Morschheuser, B., Hamari, J., & Koivisto, J. (2016). Gamification in crowdsourcing: a review.
          Proceedings of the 49th Hawaii International Conference on System Sciences, 4375-4384.
       Mullins, J. K., & Rajiv, S. (2018). Beyond enjoyment: a cognitive-emotional perspective of
          gamification. Proceedings of the 51st Hawaii International Conference on System Sciences, 1237-
          1246.
       Murre, J. M. J., & Dros, J. (2015). Replication and analysis of Ebbinghaus' forgetting curve. PloS One,
          10(7), 1–23.
       Nah, F. F.-H., Telaprolu, V. R., Ayyappa, A. P., & Eschenbrenner, B. (2014). Gamification of
          education: a review of literature. International Conference on HCI in Business, 401–409.
       Reiners, T., Wood, L. C., Chang, V., Gütl, C., Herrington, J., Teräs, H., & Gregory, S. (2012).
          Operationalising gamification in an educational authentic environment. Proceedings of the IADIS
          International Conference on Internet Technologies & Society (ITS 2012), 93 - 100.
       Riquelme, H. E., & Rios, R. E. (2010). The moderating effect of gender in the adoption of mobile
          banking. International Journal of Bank Marketing, 28(5), 328–341.
       Sailer, M., Hense, J., Mandl, H., & Klevers, M. (2013). Psychological perspectives on motivation
          through gamification. Interaction Design and Architecture(s) Journal, 19, 28–37.
       Schmidt-Kraepelin, M., Thiebes, S., Tran, M. C., & Sunyaev, A. (2018). What’s in the Game?


GamiFIN Conference 2018, Pori, Finland, May 21-23, 2018                                                          46
          Developing a taxonomy of gamification concepts for health apps. 51st Hawaii International
          Conference on System Sciences, 1218 - 1226.
       Siemon, D., & Eckardt, L. (2016). Gamification of Teaching in Higher Education. In Gamification.
          New York NY: Springer Berlin Heidelberg.
       Spil, T., Sunyaev, A., Thiebes, S., & Van Baalen, R. (2017). The adoption of wearables for a healthy
          lifestyle: can gamification help?. Proceedings of the 50th Hawaii International Conference on
          System Sciences, 3617-3626.
       Thiebes, S., Lins, S., & Basten, D. (2014). Gamifying information Systems - a synthesis of
          gamification mechanics and dynamics. Twenty Second European Conference on Information
          Systems.
       Trochim, W. M. K., Donnelly, J. P., & Arora, K. (2016). Research Methods: The Essential Knowledge
          Base. Boston, MA: Cengage Learning.
       Warmelink, H., Koivisto, J., Mayer, I. S., Vesa, M., & Hamari, J. (2018). Gamification of the work
          floor: a literature review of gamifying production and logistics operations. In 51st Hawaii
          International Conference on System Sciences 1108–1116.
       Yanuschik, O. V., Pakhomova, E. G., & Batbold, K. (2015). E-learning as a way to improve the
          quality of educational for Iiternational students. Procedia - Social and Behavioral Sciences, 215,
          147–155.


       Appendix

                            Table 3: Questionnaire: Knowledge about sustainable transport
         Knowledge
                             Question                          Correct Answer                           Grading
               Which is the largest European inland port in
         kn1   terms of total cargo volume?                   Duisport                         1 point for correct answer
               (single choice)
               What percentage of the modal split was used
         kn2   for inland waterway transport in Europe in     7%                               1 point for correct answer
               2014? (open question: 0-100%)
                                                                                               1 point = one answer right
                                                               ores and metal waste
                                                                                               2 points = two answers right;
               Which of the following types of goods are       agricultural and forestry
                                                                                               3 points = all answers right
         kn3   appropriate for inland waterway transport?       products
                                                                                               (all scores = 3 points)
               (multiple choice)                               petroleum products
                                                                                               1 point deduction for every
                                                               fast moving consumer goods     incorrect answer
               How much percent of the potential cargo
         kn4   volume of the Danube are currently used for    15%                              1 point for correct answer
               freight transport? (open question: 0-100%)
                                                              a) cooperative network
               Which of the following key characteristics     b) real-time switching between
         kn5   describe the new logistics concept of             transport modes               as kn3
               ‘synchromodality’? (multiple choice)           c) flexibility
                                                              d) unimodal transport
               What was the total cargo volume transported
         kn6   in 2014 in the European Union on inland        550 million tons                 1 point for correct answer
               waterways? (single choice)
               How many trucks are substituted by one
         kn7   common inland vessel of the Danube? (open      280 trucks                       1 point for correct answer
               question)




GamiFIN Conference 2018, Pori, Finland, May 21-23, 2018                                                                     47