F-LauReLxp: A gameful learning experience in forecasting

Abstract

It is beyond question that technology determines various aspects of students' learning process. Gamification, the application of gaming elements in non-gaming environment such as education, taking advantage of information technology, has recently gained perpetual attention as a method to increase motivation and ameliorate learning outcomes. F-LauReLxp is a web-based platform that hosts three gamified applications related to statistical, judgmental forecasting and forecasting accuracy respectively. Additionally, F-LauReLxp aims to enhance educational process around milestones research conclusions of forecasting and promote learning performance through students' engagement. This study presents a quantitative analysis of true experimental design, using treatment and control groups. Our main result is that using gamified applications as a complementary teaching tool in a forecasting course could have a positive impact on students' learning performance. Humans love to play games as a way to escape reality and enjoy themselves (Maican et al., 2016). Given that, a variety of gameful applications has appeared, in order to give a playful character to difficult life tasks. In this respect, there is an increasing interest from both academics and practitioners in using game components in educational process either at university courses, on-line courses or even at business trainings for motivation and amelioration of learning outcomes. Gamification, defined as the integration of game elements in non - gaming context (Deterding et al., 2011) has gained remarkable popularity during the last decade (Hamari et al., 2014), especially in education. The majority of studies opt for the introduction of gamification into learning process due to its fun and attractive tone, putting emphasis on the dearth of empirical evidence (Hamari, 2017; Hanus & Fox, 2015). Since there is no magic potion in the admixture of gamification and education, this study examines the effect of gamification in teaching research conclusions about forecasting principles. Predictive analytics are a new trend and in high demand nowadays, principally with the help of the growing computer's storage and process power. Additionally, the deep-rooted humans' desire to predict future events in order to plan their actions is unquestionable. Forecasting techniques help to predict future trends and estimate future values of variables under examination, based on past and present data. Hereof it has been considered as a vital addition in economic curriculums (Loomis & Cox Jr, 2003), even in undergraduate level (Gavirneni, 2008). However, approximately only half of Business schools offer forecasting courses because of its complexity (Hanke, 1989). This study

Introduction

investigates the impact specific developed gamified applications have in learning outcomes, assessing students’ comprehension of published research conclusions retrieved from fundamental forecasting sections. In our experiment, we focus on examining the impact of different tasks such as: reading, use of gamified applications and their combination in students’ performance along with the respective performance of the control group. The experiment spanned over one and a half years and the total sample is composed of 261 undergraduate and MBA students of Electrical and Computer Engineering School of the National Technical University of Athens. 2.

Literature Review

2.1. Gamification in education

Over the last decade, there has been a tremendous increase in literature about gamification in a variety of sectors, principally in education (Hamari et al., 2014; Kasurinen & Knutas, 2018) . This fact is justified by its proven effectiveness on learning, from elementary school level (da Rocha Seixas et al., 2016) up to higher education and business training. Popularity of gamification in teaching is based on its potential to engage students, as it happens in the case of game users ( Simões et al., 2013 ), and motivate them to participate in courses (Buckley & Doyle, 2016a) . Actually, based on the literature review of Kasurinen & Knutas (2018) , the majority of published papers around education and the new gamified concept aim to trigger students’ motivation, which is affiliated with positive impact on learning. In this regard, a review of gamified projects and webbased platforms with game elements accentuates gamification’s contribution to classical education (Maican et al., 2016) . Kuo & Chuang (2016) proved that gamification is helpful for the dissemination of academic content as well. Game elements most commonly embodied in educational gamified applications are points, levels and badges (Pedreira et al., 2015; Hamari et al., 2014) . Rules, rewards, quick feedback and competitiveness have been used in gamified contexts to induce positive learning outcomes (Buckley & Doyle, 2016a) . Despite the fact that gamification in a serious context, such as education, is a promising trend with great potential in teaching and lecture attendance (Kapp, 2013) , it cannot be used as cure-all. Gamification’s effects are interwoven with the respective target group and environment (Hamari et al., 2014; Buckley & Doyle, 2017) . Hence, the results of gamification vary (Sánchez-Martín et al., 2017) and may have positive or no impact on the educational process in the short run (Hanus & Fox, 2015) . Nevertheless, research, regarding the acceptance of gamification in education, agrees upon the need for more experimental results supported by statistical analysis (Hanus & Fox, 2015; Buckley & Doyle, 2017; Maican et al., 2016; Morschheuser et al., 2017) as there is a lack of empirical data analysis regarding gamification’s implementation in teaching process. 2.2

Teaching forecasting

Gapp & Fisher (2012) emphasize the lack of students’ engagement in their academic activities in management courses that discourage them to reach their full learning potential. In this direction, forecasting courses, usually considered as part of management or economic syllabuses, follow more the rule than the exception regarding students’ reluctance, essentially because of its complexity (Craighead, 2004) . Trying to change this picture, teaching guidelines have been proposed as an effort to ameliorate forecasting teaching and learning (Loomis & Cox Jr, 2003; Love & Hildebrand, 2002) and attract students’ attention. Improving lectures and teaching processes with information technology and real events exercises are some of the teaching guidelines with published positive impact on students’ motivation. Furthermore, virtual environments are a catalyst for students’ participation in management courses (Gapp & Fisher, 2012) . Last but not least, a prediction market has been used as a pedagogical tool during management courses (Buckley et al., 2011; Buckley & Doyle, 2016a) , producing real case decision scenarios. Students were intrigued to search more information about the problem under examination and they were able to apply this gained knowledge more effectively (Buckley et al., 2011) . Hence, active learning and information technology may perform as a force to magnetize students’ interest in management and forecasting courses.

2.3. Gamification in teaching forecasting

In this direction, we reviewed journal articles about forecasting courses that incorporate active learning events or innovative educational methods. Keeping score (Craighead, 2004) , the ad hoc use of spreadsheets (Gardner, 2008) and the adoption of competition between teachers and students (Snider & Eliasson, 2013) are only some examples of effective active learning proposed in forecasting courses. Another in-class exercise with promising results was the forecast of the points scored by the university basketball team (Gavirneni, 2008) . During the lectures, authors explained calculations of forecasts, general trend and time series components using this real-world case study. Thus, active learning is beneficial for teaching statistical forecasting methods. However, forecasting can also be used as a way to attract student’s interest in management courses. Buckley et al. (2011) triggered students’ active participation, using a prediction market to build decision scenarios based on real facts, during an undergraduate course in risk management. Buckley & Doyle (2016a) also proved that the use of a prediction market in a course could be considered as a useful pedagogical tool that gives active character to education. Since the application of a prediction market is accompanied by objective rules, feedback and competition among learners, Buckley & Doyle (2016a) portrayed a gamified learning experience in a taxation course, with positive impact on students’ knowledge level. Forecasting is a kind of art rather than a scientific field (Gavirneni, 2008) , thus it can be considered as fertile ground for applying gamification (Buckley & Doyle, 2016a) , in order to not only motivate students but also increase their learning outcomes.

F-LauReLxp Description

F-LauReLxp is designed as a complementary teaching tool in the context of forecasting techniques course, using gamification as defined by Deterding et al. (2011) : “the use of game design elements in non-game contexts”. F-LauReLxp is named after Forecasting and LauReL, a plant that was used as aliment for an ancient Greek priest in order to say oracles and wise advises. The idea behind this platform has arisen as an effort to engage students into a forecasting techniques course, to improve their learning outcomes, disseminate published research conclusions in this field and consequently improve students’ forecasting skills.

3.1. F-LauReLxp architecture

F-LauReLxp is a web-based modular platform, easily accessible with a browser. Since it is publicly available, a user may navigate through F-LauReLxp and find information about forecasting aspects, recent research findings and the gamified applications with respective instructions. F-LauReLxp is composed of three web-gamified applications, as illustrated in Figure 1. These applications are independent from each other, and they use different interfaces and databases. The platform also has a pivot leader board of participants and statistics about its gamified applications for registered users. 3.2. F-LauReLxp components’ design Guidelines for the design of F-LauReLxp and its components were derived by the literature about gamification effectiveness in learning (da Rocha Seixas et al., 2016; Yildirim, 2017; Hamari et al., 2016; Sánchez-Martín et al., 2017; Kuo & Chuang, 2016; Kyewski & Krämer, 2018 ; Maican et al., 2016; Pedreira et al., 2015; DomíNguez et al., 2013 ) and direction on how to design and develop gamified applications (Zichermann & Cunningham, 2011; Morschheuser et al., 2017; Kapp, 2013) . The most commonly used and assessed game elements in reviewed studies are points, levels, achievements and leader boards (Hamari et al., 2014) . Given that, all three F-LauReLxp’s gamified applications embody them, in order to invoke to students the willingness of reward, status, and competition (Bunchball, 2010) . However, each of the three gamified applications incorporates one or more game mechanisms, such as meaningful storyline, time constraints and challenges (Kapp, 2013; Zichermann & Cunningham, 2011; Bharathi et al., 2016) . More precisely, Table 1 indicates the included game components and mechanisms per gamified application and the respective purpose served in the context of a forecasting course. In addition, user-friendliness and clear player’s guidance (Kapp, 2013) determined our design decisions. All F-LauReLxp’s components have similar user interfaces, in order to keep their aesthetic connection. From a technical perspective, considering the methods and design principles presented in the study of Morschheuser et al. (2017) on engineering gamified software, all applications are implemented by the authors of this study exclusively for the teaching needs of a forecasting course. F-LauReLxp’s gamified applications are fully accessible to registered users, with a browser (a free unity-plugin is required for Metrics to Escape). Each application requires registration with an email and a password of user’s choice.

A brief description of gamified applications can be seen below: Horses for Courses. This application aims to disseminate the method selection protocols for fastmoving and intermittent demand time series (Petropoulos et al., 2014) . Students choose the most appropriate forecasting method based on different conditions and data at each level, getting points according to their choices. Instructions for each level are available to students. A new challenge rises at each level, enforcing the student to apply the knowledge of method selection rules, and improve their performance (Buckley et al., 2011) , in order to conquer a leader board position. JudgeIt. This application targets to communicate heuristics and biases that have great impact on judgmental forecasting (Tversky & Kahneman, 1974) . Students participate in a meaningful story, where they become travelers in order to explore different destinations related to heuristics and biases. Travelers aim to gain points by identifying the respective biases. Useful video and pictures puzzle and challenge them, whilst instructions guide them to collect points and useful elements, which form their score on the final leader board.

Metrics to Escape. Forecasting accuracy is the subject of this application, which aims to point out the advantages and disadvantages of different accuracy metrics (Hyndman & Koehler, 2006) . Students become prisoners who are looking for clues regarding statistical metrics, answer questions and solve riddles about metrics characteristics in order to escape a 3D virtual room. Students’ target should be to both escape on time and collect points to reach a good position in leader board. sceolerrScettciuotdlnyetpnoartpscophtgolaaycliilonnelgnpagotnehidsnetrmsepbelytyhiondg ibdSaetsnuetdidfeyonintnsgvgibadiieanospceoaxitanemtgsopbrlieyess Stuidnaedddniivcstasaadntgitvnaaagignnemtpsaegoateinrnsidctssby

Students are aware of their progress, via suitable labels and feel well guided

Looking for ways to maximize points gained in every level, students are motivated to apply the gained knowledge from the lecture in the most suitable way

Increase competition among students

Student is an explorer who wants to reach a goal, not only learn

Student is a prisoner who

wants to escape not only learn Student is more challenged to find clues and escape

3.3. F-LauReLxp components implementation

Responsive and user-friendly interface was chosen for all applications, based on bootstrap framework. For the implementation, web technologies were used. More precisely, Javascript, ASP.NET and Unity were used in front-end developing, while PHP with MySQL data-base and VB. NET or C# with MS-SQL database were used in the back-end.

Experiment Description and Assessment

4.1. Participants

F-LauReLxp’s gamified applications were launched to students in different semesters. Hence, the experiments for the evaluation of the first gamified application: Horses for courses took place in spring semester 2015 and 2016 to 49 and 60 undergraduate students respectively and fall semester 2015 to 37 MBA students, whilst for the rest applications’ evaluations took place in spring semester 2016 to 58 and 57 undergraduate students. All experiments were conducted in the context of forecasting techniques course, delivered in the Electrical and Computer Engineering School of the National Technical University of Athens in a total sample of 261 students. Table 3 presents in more detail, the number of students who participated in each experiment per gamified application.

4.2. Experimental design

The experimental design was followed strictly, independent of the gamified application, the semester or the level of studies. Students had the same background, without any prior knowledge of the respective field, and their participation in each experiment was optional. However, they were aware of the incentive, which was a 0.5 out of 10 grade for each gamified application, instead of a respective equivalent exercise in final examination of the course. Thus, every student could receive the highest grade. Moreover, there was no difference in incentives among the different groups that the students were randomly assigned to. Table 2, illustrates the experimental setup for the evaluation. Initially, all students attended a lecture for 15 minutes, during which the main conclusions of the respective research were presented. Then, they were randomly assigned to one of the groups, represented in Table 2. Each group had 15 minutes to fulfill each one of the task assigned to them. More precisely, the Group Control did not have any additional tasks to complete, Group Read had to read the paper for 15 minutes, Group Play had 15 minutes available to make a full round in the respective gamified application passing through all the levels and reach the leader board of the respective gamified application (named thenceforth as task play). Group Read&Play had 30 minutes to fulfill the task read and then the task play. Finally, all groups had to complete an on-line evaluation form with 30 equivalent questions about the respective research’s findings within 15 minutes. The evaluation experiment for each gamified application had a different lecture and on-line evaluation form based on the related research. All of them were composed of 30 questions of the same type. Students’ performance was calculated as the sum of right answers (normalized to have 100 as maximum value) for each experiment of each gamified application. During the experiment, every task had a strict duration, clear instructions and no extra advice was given. Students were not allowed to collaborate or look for information online while completing each of the tasks.

4.3. Results of experiment

The analysis of results was conducted in two steps. Firstly, due to the small sample size, we investigated median instead of mean values of students’ performances per group and experiment, received from the assessment of the evaluation forms. Table 3 presents students’ performance results, number of students per experiment and their percentages in each group. In general, Group Play had the best performance and half of the times, Group Read&Play, whose participants read the paper and used the respective gamified application, reached the second position. Group Read, whose participants just read the paper, presented a slightly better performance than Group Control, whose participants received no treatment. Group Control was mostly at the last position. Additionally, pairwise non-parametric tests were conducted, with a confidence interval equal to 95%, concluding that groups populations means rank different in most of the cases. In the second step, we gathered data of students’ performances from all the experiments and then divided it into two major groups: No F-LauReLxp group, composed of 127 students who have not been through F-LauReLxp (Group Control and Group Read) and 134 students who used it (Group Play and Group Play&Read), named F-LauReLxp. We opt for this strategy for a number of reasons, namely the gamified applications were designed under the same guidelines, the evaluation experiments were conducted with exactly the same laboratory settings, and finally the evaluation forms for each experiment had the same number and type of questions. In the case of Horses for Courses evaluation experiment, the same evaluation form was used independently of the semester of application or participants’ level of studies. Figure 2 illustrates the distribution of gathered performances in percentiles with box-plot diagrams. Having larger samples, we conducted paired t-test, with a confidence interval equal to 95%. Null hypothesis of equal differences in means is rejected (t = -9.4146, df = 126, p <0.001), while the use of F-LauReLxp presents an improvement regarding mean values of performances, equal to 34% approximately.

These gamified applications are proposed as a complementary teaching tool to motivate students and consequently ameliorate their performance. Laboratory settings of this study simulate the future use of these gamified applications, without impact on results’ validity. Since F-LauReLxp is publicly available, students could use any application out of lectures or in an e-learning environment in the future. However, playing more or looking for further information and applying the gained knowledge in order to achieve a better position in leader board probably would be beneficial for learning outcomes (Buckley & Doyle, 2016a) , supporting the results of this study.

Conclusions The conclusions of our empirical study are in line with literature findings about the positive impact of gamification on learning performance (da Rocha Seixas et al., 2016; Buckley & Doyle, 2016a; Hamari et al., 2016; Kuo & Chuang, 2016; Maican et al., 2016; Yildirim, 2017) . We designed and implemented F-LauReLxp, which hosts three web gamified applications related to forecasting sections. It aims to improve students’ learning outcomes, increasing their motivation with gamification mechanisms. Results advocate that gamification does improve students’ performance and under certain conditions, it may have a greater impact than reading or even reading and use FLauReLxp, as far as forecasting learning is concerned. It could increase students’ performance by up to 76% compared to merely attending a respective lecture. In these terms, F-LauReLxp can be suggested as a useful complementary educational tool for improving learning outcomes and comprehension.

A detailed quantitative analysis of this data is required to have conclusions that are more robust. Furthermore, a wider sample, composed of students and practitioners, could be an interesting addendum to compare gamification’s impact on different populations. Further extension of FLauReLxp could be the integration of a superforecasters project (Tetlock & Gardner, 2016) , as another evaluation method of students’ performance. Finally, F-LauReLxp should host more applications to teach forecasting aspects. The integration of the “Learning to Forecast Experiment” (Hommes, 2011; Assenza et al., 2014; Bao et al., 2017) could add important value to F-LauReLxp, by helping collect data about students’ interactions to predict the asset price under changeable conditions in an artificial and gamified market. Bao, T., Hommes, C., & Makarewicz, T. (2017). Bubble formation and (in) efficient markets in learningto-forecast and optimise experiments. The Economic Journal, 127.

Sánchez-Martín, J., & Dávila-Acedo, M. A. (2017). Just a game? gamifying a general science class at university: Collaborative and competitive work implications. Thinking Skills and Creativity, 26, 51–59.

References

Assenza , T. , Bao , T. , Hommes , C. , & Massaro , D. ( 2014 ). Experiments on expectations in macroeconomics and finance . In Experiments in macroeconomics (pp. 11 - 70 ). Emerald Group Publishing Limited.

Bharathi , A. K. B. G. , Singh , A. , Tucker , C. S. , & Nembhard , H. B. ( 2016 ). Knowledge discovery of game design features by mining user-generated feedback . Computers in Human Behavior , 60 , 361 - 371 .

Buckley , P. , & Doyle , E. ( 2016a ). Gamification and student motivation . Interactive Learning Environments , 24 , 1162 - 1175 .

Buckley , P. , & Doyle , E. ( 2016b ). Using web-based collaborative forecasting to enhance information literacy and disciplinary knowledge . Interactive Learning Environments , 24 , 1574 - 1589 .

Buckley , P. , & Doyle , E. ( 2017 ). Individualising gamification: An investigation of the impact of learning styles and personality traits on the efficacy of gamification using a prediction market . Computers & Education , 106 , 43 - 55 .

Buckley , P. , Garvey , J. , & McGrath , F. ( 2011 ). A case study on using prediction markets as a rich environment for active learning . Computers & Education , 56 , 418 - 428 .

Bunchball , I. ( 2010 ). Gamification 101: An introduction to the use of game dynamics to influence behavior . White paper, 9.

Craighead , C. W. ( 2004 ). Right on target for time-series forecasting . Decision Sciences Journal of Innovative Education , 2 , 207 - 212 .

Deterding , S. , Dixon , D. , Khaled , R. , & Nacke , L. ( 2011 ). From game design elements to gamefulness: Defining “gamification” . In Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments MindTrek '11 (pp. 9 - 15 ). New York, NY, USA: ACM.

DomíNguez , A. , Saenz-De-Navarrete , J. , De-Marcos , L. , FernáNdez-Sanz , L. , PagéS , C. , & MartíNezHerráIz , J. J. ( 2013 ). Gamifying learning experiences: Practical implications and outcomes . Computers & Education , 63 , 380 - 392 .

Gapp , R. , & Fisher, R. ( 2012 ). Undergraduate management students perceptions of what makes a successful virtual group . Education+ training , 54 , 167 - 179 .

Gardner , L. ( 2008 ). Using a spreadsheet for active learning projects in operations management . INFORMS Transactions on Education , 8 , 75 - 88 .

Gavirneni , S. ( 2008 ). Teaching the subjective aspect of forecasting through the use of basketball scores .

Decision Sciences Journal of Innovative Education , 6 , 187 - 195 .

Hamari , J. ( 2017 ). Do badges increase user activity? a field experiment on the effects of gamification.

Computers in Human Behavior , 71 , 469 - 478 .

Hamari , J. , Koivisto , J. , & Sarsa , H. ( 2014 ). Does gamification work?-a literature review of empirical studies on gamification . In System Sciences (HICSS) , 2014 47th Hawaii International Conference on (pp.

3025- 3034 ). IEEE.

Hamari , J. , Shernoff , D. J. , Rowe , E. , Coller , B. , Asbell-Clarke , J. , & Edwards , T. ( 2016 ). Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning . Computers in Human Behavior , 54 , 170 - 179 .

Hanke , J. ( 1989 ). Forecasting in business schools: A follow-up survey . International Journal of Forecasting , 5 , 259 - 262 .

Hanus , M. D. , & Fox , J. ( 2015 ). Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison , satisfaction, effort, and academic performance.

Computers & Education, 80 , 152 - 161 .

Hommes , C. ( 2011 ). The heterogeneous expectations hypothesis: Some evidence from the lab . Journal of Economic dynamics and control , 35 , 1 - 24 .

Hyndman , R. J. , & Koehler , A. B. ( 2006 ). Another look at measures of forecast accuracy . International journal of forecasting , 22 , 679 - 688 .

Kapp , K. M. ( 2013 ). The gamification of learning and instruction fieldbook: Ideas into practice . John Wiley & Sons.

Kasurinen , J. , & Knutas , A. ( 2018 ). Publication trends in gamification: A systematic mapping study .

Computer Science Review, 27 , 33 - 44 .

Kuo , M.-S. , & Chuang , T.-Y. ( 2016 ). How gamification motivates visits and engagement for online academic dissemination an empirical study . Computers in Human Behavior , 55 , 16 - 27 .

Kyewski , E. , & Krämer , N. C. ( 2018 ). To gamify or not to gamify? an experimental field study of the influence of badges on motivation, activity, and performance in an online learning course . Computers & Education , 118 , 25 - 37 .

Loomis , D. G. , & Cox Jr, J. E. ( 2003 ). Principles for teaching economic forecasting . International Review of Economics Education , 2 , 69 - 79 .

Love , T. E. , & Hildebrand , D. K. ( 2002 ). Statistics education and the making statistics more effective in schools of business conferences . The American Statistician , 56 , 107 - 112 .

Maican , C. , Lixandroiu , R. , & Constantin , C. ( 2016 ). Interactivia.ro a study of a gamification framework using zero-cost tools . Computers in Human Behavior , 61 , 186 - 197 .

Morschheuser , B. , Hassan , L. , Werder , K. , & Hamari , J. ( 2017 ). How to design gamification? A method for engineering gamified software . Information and Software Technology.

Pedreira , O. , Garca , F. , Brisaboa , N. , & Piattini , M. ( 2015 ). Gamification in software engineering a systematic mapping . Information and Software Technology , 57 , 157 - 168 .

Petropoulos , F. , Makridakis , S. , Assimakopoulos , V. , & Nikolopoulos , K. ( 2014 ). horses for courses in demand forecasting . European Journal of Operational Research , 237 , 152 - 163 .

da Rocha Seixas , L. , Gomes , A. S. , & de Melo Filho , I. J. ( 2016 ). Effectiveness of gamification in the engagement of students . Computers in Human Behavior , 58 , 48 - 63 .

Simões , J. , Redondo , R. D. , & Vilas , A. F. ( 2013 ). A social gamification framework for a k-6 learning platform . Computers in Human Behavior , 29 , 345 - 353 .

Snider , B. R. , & Eliasson , J. B. ( 2013 ). Beat the instructor: An introductory forecasting game . Decision Sciences Journal of Innovative Education , 11 , 147 - 157 .

Tetlock , P. E. , & Gardner , D. ( 2016 ). Superforecasting: The art and science of prediction . Random House.

Tversky , A. , & Kahneman , D. ( 1974 ). Judgment under uncertainty: Heuristics and biases . science, 185 , 1124 - 1131 .

Yildirim , I. ( 2017 ). The effects of gamification-based teaching practices on student achievement and students' attitudes toward lessons . The Internet and Higher Education , 33 , 86 - 92 .

Zichermann , G. , & Cunningham , C. ( 2011 ). Gamification by design: Implementing game mechanics in web and mobile apps . ” O'Reilly Media , Inc.”.