Towards reward-based gamification in Collaborative Learning Flow Patterns based on learning analytics

Abstract

Collaborative learning flow patterns (CLFPs) formulate good practices for the orchestration of activity sequences and collaboration mechanisms that can elicit fruitful social interactions. Despite their benefits, it is worth exploring how their implementation can be improved. The use of reward-based gamification strategies accompanied with Learning Analytics is one approach that can potentially. reinforce the participation and collaboration between participants, and at the same type help teachers to make decisions and measure the impact of the activities. This paper presents a first iteration for a model to integrate the use of gamification and learning analytics in collaborative learning activities structured following the Jigsaw and Pyramid CLFP. The applicability of the model is illustrated through two scenarios (face-to-face and MOOC settings).

Keywords

• Collaborative Learning
• CLFP
• Gamification
• Learning Analytics
• Pyramid
• Jigsaw

-

Collaborative Learning Flow Patterns (CLFPs) are topic-independent structures of Collaborative Learning (CL) activity flow that can be adapted to multiple learning scenarios [ 1 ]. These patterns can help teachers design and incorporate adapted collaborative learning scenarios into their teaching practice. However, the application of CL poses certain challenges and drawbacks [ 2 ]: (1) students usually divide the tasks, working individually without collaborating; (2) these activities require extra time for both teachers and students; and (3) can emerge eventual interaction and communication problems among students. Despite the scripting structure proposed by CLFPs to overcome these challenges [ 2 ], additional strategies to reinforce their effects on 1 Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License

Attribution 4.0 International (CC BY 4.0). learning and collaboration skills are worth exploring. In this context, gamification is proposed as a pedagogical strategy to support several underpinning CL mechanics (e.g., positive interdependence, individual accountability, knowledge sharing) based on the positive effects reported in other educational experiences [ 3,4 ].

Gamification is defined as the use of game design elements and structures (e.g., customization, targets, engagement loops) in non-game contexts [ 5 ]. Gamifications involving rewards and completion logics (i.e., reward-based gamification) is the most implemented strategy in both face-to-face and on-line educational settings [ 6 ]. In this type of gamification, task indicators are crucial to understand the student behavior within the CL learning task and the performance level of reward conditions. Therefore, Learning Analytics (LA) plays an important role in this type of gamification and needs to be adapted to the different gamification designs created by teachers. Reward-based strategies have been effectively implemented in educational environments to foster students’ behavior towards concrete individual activities. In a similar way, this type of strategies could be used to foster CL actions expected to happen in CLFP (e.g., promote student discussion). As CLFPs provide teachers with pre-established structures of CL activity flow, gamified CLFPs could also incorporate pre-established reward-based strategies that can encourage CL mechanics based on the positive results already reported in the literature [ 28 ].

Previous studies have addressed the use of reward-based strategies in collaborative learning activities and environments. For instance, [ 7 ] propose an ontology to represent gamification strategies in collaborative learning scenarios. Also, [ 4 ] describe a case study in which an online gamified discussion forum increased student collaboration and reduced response times. However, to the best of our knowledge, none of the previous studies have focused on CLFPs, the pre-established features that can be gamified, and the LA indicators that need to be used to monitor the gamified CLFPs and students’ actions supporting collaborative learning.

The general Research Question (RQ) guiding this work-in-progress is: the extent to which LA can support gamification strategies in CLFP activities to foster student collaboration? This paper presents a work-in-progress analysis of two CLFPs (i.e., jigsaw and pyramid) regarding the LA and reward-based strategies that can be incorporated into pre-established CLFPs to foster student collaboration. Additionally, the paper reports two scenarios to illustrate the applicability and benefits of two gamified CLFP in both face-to-face and online educational settings.

The next section describes the theoretical background of CLFPs, gamification and LA, and similar studies using gamification strategies to promote CL. Section 3 presents the model to integrate gamification and learning analytics in collaborative learning activities following the Jigsaw or Pyramid CLFP. Section 4 presents two scenarios from teachers’ perspective. Finally, some conclusions and lines of future work are outlined in Section 5.

Background

Collaborative Learning Flow Patterns

Collaborative Learning Flow Patterns (CLFPs) represent broadly accepted techniques that are repetitively used by collaborative learning practitioners (e.g., teachers) when structuring the flow of types of learning activities involved in collaborative learning scenarios [ 8 ]. CLFP pre-structure collaboration in such a way that productive interactions are promoted, so that the potential effectiveness of the educational situation is enhanced [ 9 ], fostering individual participation, accountability and balanced positive interdependence. Examples of CLFPs include TPS (Think-Pair-Share), Simulation, TAPPS (Thinking Aloud Pair Problem Solving) and Brainstorming [ 10 ]. This work-inprogress paper focused on the Jigsaw and Pyramid patterns, which are CLFPs with complex scripting structures that cover the key scripting mechanisms of knowledge distribution and changing groups (in terms of members and group size) along a learning flow.

The “Jigsaw” pattern [ 11,12 ] structures the solution of a complex problem into independent sub-problems that each participant in (small) groups (“Jigsaw Group”) studies or works around such particular sub-problems. Participants from different groups working in the same sub-problem meet in an “Expert Group” for knowledge sharing and exchange of ideas. Therefore, these temporary groups become experts in the given sub-problem. At last, participants from the same “Jigsaw group” meet to contribute with their “expertise” in the different sub-problems to provide a solution to the complex problem [ 8 ].

The “Pyramid” pattern is used for complex problems, usually without a specific solution, whose resolution implies the achievement of gradual consensus among all participants [ 8 ]. A Pyramid flow is usually initiated with individual students solving a global task. Then, in a second phase of the Pyramid, such individual solutions are discussed in small groups and agreed upon a common proposal. These small groups then form larger-groups iteratively and large group discussions will continue until a consensus is reached at the global level. Pyramid flows foster individual participation, accountability and balanced positive interdependence [ 10 ]. Furthermore, the Pyramid pattern promotes conversations in incrementally sized groups, clear expectations of reaching consensus and positive reinforcement mechanisms leading to desired positive behaviors in the learning process [ 13,14 ]. Gamification strategies try to replicate the benefits of games (e.g., increase student motivation, promote participation) by applying concrete game design elements in nongame context (e.g., health, workplace, education) [ 15-20 ]. Reward-based strategies usually refer to those gamifications in which rewards are the principal game design element (e.g., badges, points, ribbons). In this type of gamification, rewards are issued when a completion logic (e.g., relevant student actions defined beforehand) is satisfied [ 21 ]. Although these conditions can be as simple as responding to a question, more complex conditions could require the support of "gamification analytics". Heilbrunn et al [ 22 ]. defined gamification analytics as “the data-driven processes of monitoring and adapting gamification designs” showing that the data generated by gamification activities can be monitored and analyzed to obtain valuable insights, this data can come from three sources (1) user behavior in the gamified application (2) user properties like age, gender, location (3) gamification data representing the user progression over time.

Gamification Analytics are important because gamification designs are not rigid artefacts, but subject to change over time. Some of the reasons for changes are for example [ 22 ]: • The gamification goal may not be achieved using the proposed design. • All the users are not influenced in the same way by the same gamification elements. • The original goals of the design may change, and an update to the design and its activities may be needed • The effect of the novelty factor can decrease, so existing gamification elements might be adjusted Freire et al. explored the use of tools and technologies from game Analytics with educational purposes in what they called Game Learning Analytics (GLA) [ 23 ]. This combination can help the design and refinement of serious games providing real time data of the user interactions that can be related with the actual learning. moving to a data-driven design approach [ 23 ]. 2.3

Related Work

Previous studies have analyzed the effects of implementing gamification in educational contexts to foster learners’ collaboration. Some of the most representative examples are presented in this subsection.

Challco et al. [ 7 ] propose an ontological model for the formal systematization and representation of knowledge that describes concepts from gamification and its use as Persuasive Technology (PT) in Collaborative Learning (CL) scenarios. Their approach proposes to formalize the connection of concepts from theories and models to design PT in order to specify gamified CSCL scripts that induce students to willingly follow an intended learning behavior.

Knutas et al. [ 3 ] investigate the impact of a gamified online collaboration system on collaborative behavior and communications efficiency in a case study. The gamification elements of the system were a likely factor in encouraging skilled students to participate and contribute to the online community. The discussion system increased student collaboration, course communication efficiency and reduced response times. Li et al. [ 28 ] develop an online social network based learning environment to attract and engage students in collaborative learning, by introducing game mechanics to promote the students learning by encouraging them to: (1) Participate more in social and learning activities, (2) Develop a stronger sense of community, and (3) be more willing to help each other on academic and social matters.

Additionally, other previous publications also explored the use of LA in gamification environments. For instance, the literature review performed by Calderon et al. shows the lack of tools that can provide gamification experts with real-time analytics from gamified systems, so experts can evaluate, improve and adapt their gamification strategies [ 24 ].

Pérez-Colado et al. advanced in this aspect by proposing a conceptual model for GLA. The Learning Analytics Model (LAM) provides practitioners with information about how games should be tracked, aggregated and reported to a Learning Analytics System (LAS). The LAM isolates learning analytics users from the implementation details of the underlying LAS. This allows both systems to evolve independently as long as the interface between the model and the system is well-defined and represents policy and mechanism respectively [ 25 ].

However, to the best of our knowledge, none of the previous studies have addressed the use of gamification strategies in CLFPs, their pre-established features that can be gamified (e.g., phases, relevant expected student actions), and the gamification analytics required to monitor the students’ actions supporting collaborative learning. 3

Modeling rewards-based gamification for CLFPs based on learning analytics

In this work-in-progress paper, we propose the first iteration of a model (See Fig. 2) for the design of gamification elements in collaborative learning activities that foster the achievement of key objectives in CL: positive interdependence (team members need each other to succeed), individual accountability (students must contribute their fair share), and discussion to construct students’ shared knowledge. The studied CLFP jigsaw and pyramid have in common an individual phase and two or more groupal phases. For these phases, teachers configure course activities, and set gamification conditions under which the rewards will be issued to the students. Learning analytics play a fundamental role by tracking the whole process, delivering both, near-real time and asynchronous analytics. As shown in Table 1, CLFPs target different CL objectives, whose achievement can be tracked with multiple LA indicators. Those indicators are associated with a set of gamification analytics KPIs that are used to issue the configured rewards and to provide information during and after the activity to students (e.g., student progress) and teachers (e.g., activity redesign). In order to identify the KPIs we followed a top-down approach starting from the 3 mentioned objectives, then we studied the selected CLFP (jigsaw and pyramid) to generate a set of LA indicators that are aligned with each of the possible phases. As stated by Hilburn et al. [ 26 ], gamification experts should be able to define KPIs based on the available application log data which is typically available in form of event streams, databases, or log files. Furthermore, in order to improve students’ motivation toward collaborative actions associated with KPIs, teachers could use course privileges as rewards such as choosing a friend to join the same CLFP group or extending the deadline to submit the collaborative task [ 21 ]. Phase Individual Groupal 5. Do video-conference with group members 3.1 Participation in role assignation 3.2 Role assigned 4.1 Participation in the peer assessment activity 4.2 mark obtained in the peer assessment 5.1 Participation in videoconference 5.2 duration of the videoconference

Scenarios

This section introduces two scenarios, the first one using a gamified jigsaw pattern in a face-to-face learning environment, and the second one, using a gamified pyramid pattern in a MOOC. These scenarios are intended to show the applicability of the proposed model in two different learning environments (i.e., face-to-face and online) considering their differences in the synchronicity, time constraints, type of learner, tools, technology and communication mechanisms. 4.1

Face-to-face learning situation

Juan is a high education teacher of history who wants to use collaborative learning for teaching contents that can be dense, difficult and sometimes boring for some students. He decided to use the jigsaw CLFP in order to foster students’ positive interdependence, interaction, and accountability, so students learn course contents collaboratively and feel more engaged with the different course topics. Following the general configuration of the jigsaw pattern [ 11,12 ], Juan configured the activity as follows: Activity structure: • 1st phase: Individual [at home]. Teams of individuals are formed with each person being responsible for a specific chunk of the topic that can be a perspective, section of a topic or a case, subsequently becoming an expert for that chunk of knowledge. • 2nd phase: expert groups [⅓ of class time]. students are gathered in equal sized groups of experts in the same chunk of knowledge. Each group discusses the case from their assigned character or perspective. To motivate the discussion Juan prepares a set of guiding questions for each group. • 3rd phase: jigsaw groups [⅔ of class time]. students are gathered in new groups each having at least one expert for each chunk of knowledge. Juan assigns a set of guiding questions so students in these new "jigsaw" groups can argue while considering and analyzing the perspectives of the others.

Juan also decided to use reward-based gamification strategies to promote desirable CL actions expected to happen within the activity (see Table 2). According to the different KPIs and the characteristics of his students, Juan thought that the rewards can be: additional points, possibility to have more discussion or preparation time in one of the phases, badges for the best groups and students. After each one of the phases, according to their performance (tracked by the KPIs) students can reclaim in-course-privileges like: choosing team members, choosing role distribution, tokens for asking help to teachers and other students For the different phases of the activity students will use shared documents online with Juan and with the teammates, so the results can be saved, and Juan can observe the progress of the activity if he needs. Shared documents can help the development of the activity, but for a deeper integration of the activity with the gamification and the LA a more specialized tool will be needed.

MOOC

MOOCs present some specific features different from other more traditional learning environments (e.g., face-to-face, blended learning settings), which can potentially constraint the attainment of the expected CLFP benefits. Some of these limitations are: • Participants’ massiveness. The high number of enrolled students hinders the manual management of groups and individual needs [ 29 ]. Therefore, these environments require computer-interpretable data models supporting practitioners’ decisions to enable its automation during course run-time (e.g., group creation, artifacts flow, etc.). • Asynchronous interaction among course participants. The time flexibility of MOOCs allows participants to complete course activities without a restricted schedule [ 30 ]. Therefore, the number of connections to the course and the number of interactions between group members are more limited than in synchronous learning environments, thus hindering collaboration among group members. Additionally, the high dropout rates of this kind of courses and the lack of indicators to know whether a student is still active in the course also contribute to this potential problem • Participants’ heterogeneity. The open enrollment enables the participation of worldwide students with different backgrounds [ 30 ]. Although a heterogeneous set of participants can provide multiple opportunities for knowledge sharing, differences among group members in previous knowledge and background (e.g., language) can lead to problematic learning situations not frequently experienced in environments with heterogeneous participants (e.g., SPOCs).

These limitations should be considered by instructors and instructional designers during the design phase of CLFPs in MOOCs to avoid negative counter effects (e.g., student disengagement). Given this context, gamification can serve as twofold: to foster collaboration and to help overcome some of the previous limitations. These benefits can be shown with the following scenario.

Maria is a MOOC practitioner who is preparing the 2nd version of a MOOC about collaborative learning for higher educational settings. The course is configured as instructor-led containing 6 weekly modules with videos, recommended readings, and compulsory and optional individual activities. In this new version, Maria decided to include two collaborative learning activities with the purpose of (1) fostering discussion between course participants in order to construct students’ shared knowledge, and (2) increasing positive interdependence, and (3) individual accountability. To this end, Maria designed two Pyramid CLFP activities about ‘CL design’ (module 2) and ‘CL evaluation’ (module 4) following the structure presented below. In the first phase (1 week), each participant studies individually a shared problem, and provides a solution for such a problem. In the second phase (1 week), participants are gathered in small groups (6-person) to compare and discuss the individual proposals with the purpose of creating a new shared solution. Maria heard about the benefits of using reward-based gamification strategies to promote desirable actions expected to happen within CLFPs while engaged with activities. During the MOOC re-design, Maria used the proposed model to help configure reward strategies (i.e., LA indicators for conditions, in-course privileges) targeting the expected CL objectives. Table 3 shows the resulting gamification design for the Pyramid activities. Gamification Expected Benefit (5-7) Badge Suites: XX = 25% - Bronze Group Medal XX= 50% - Silver Group Medal XX=60% - Gold Group Medal (5-7) Classification of CLFP groups attending to their activity which can be used for the re-design of groups and future group formations. (8) Individual Badge (9) Individual Badge (10) Group Badge + Privilege: Extend the deadline submission for the final course project (8-10) Identification of active students.

Maria used GamiTool to implement the gamification design. GamiTool allows to design and automatically enact gamification learning designs involving individual and group conditions in multiple MOOC platforms [ 21 ], thus avoiding the manual management of reward-based strategies in such a massive environment. This tool also allows students to disable the gamification capabilities, thus avoiding bothering those students not interested in these strategies. Information about which students disable gamification features could be used by Maria together with other learning (e.g., dropout students) and gamification analytics during CLFP enactment (e.g., more active students) to support the monitoring and redesign of the CLFP activity (e.g., group formation).

For instance, during the CLFP individual phase, practitioners can make use of these indicators to create CLFP groups likely to collaborate. From the total number of active students during this week, two clusters of students can be formed: gamification enabled and gamification disabled. Attending the gamification enabled cluster, practitioners can make use of the privilege associated with Reward (3), and form 6-person groups containing at least 2-3 people that already know each other or are likely to collaborate, thus avoiding feelings of isolation and heterogeneity typical from MOOC environments. Additionally, other gamification parameters can be used (e.g., number of individual rewards earned, time to claim the rewards) to understand the level of students’ engagement, and configure balanced groups engaged with course individual and collaborative activities. Finally, the gamification analytics obtained from the first CLFP activity (see Table 3) can be also used to redesign the second CLFP activity (e.g., optimal number of group members). 5

Conclusions

Despite the increasing number of works in gamification and collaborative learning there is still a lack of studies addressing reward-based gamifications for scripted CL, such as those structured according to Collaborative Learning Flows (CLFPs). The use of gamification analytics is essential for a “successful” implementation of the rewardbased mechanics. This work-in-progress paper proposes a first iteration for a model to help educators incorporate gamification strategies in CLFPs to foster collaborative learning objectives, such as positive interdependence and individual accountability, while taking into account the role of game learning analytics. As future work, we plan to carry out co-design activities with teachers interested in using gamification in CLFPs in their teaching practice and evaluate the effects of their implementation with students. This will help us know (1) whether the proposed pre-established gamification features (in the model) actually support practitioners during the design of such gamified activities, and therefore, refine the model accordingly; (2) analyze whether the gamification strategies implemented, actually improved the students’ collaborative learning skills, (3) determine which variables can help practitioners during the development of the activities. 6

Acknowledgements

This work has been partially funded by the Eu. Regional Development Fund and the National Research Agency of the Spanish Ministry of Science, Innovation and Universities, under project grants TIN2017-85179-C3-2-R, TIN2017-85179-C3-3-R. D. Hernández-Leo acknowledges the support by ICREA under the ICREA Academia programme.

References

1. Hernandez-Leo , D. , Villasclaras-Fernandez , E. D. , Asensio-Perez , J. I. , Dimitriadis , Y. A. , Retalis , S.: CSCL scripting patterns: hierarchical relationships and applicability . In Proceedings of the Sixth IEEE International Conference on Advanced Learning Technologies , 388 - 392 ( 2006 ).

2. Radkowitsch , A. , Vogel , F. , Fischer , F. : Good for learning, bad for motivation? A metaanalysis on the effects of computer-supported collaboration scripts , Int. J. Comput. Collab. Learn. , 15 : 5 - 47 ( 2020 ).

3. Knutas , A. , Ikonen , J. , Nikula , U. , Porras , J.: Increasing collaborative communications in a programming course with gamification: a case study . In Proceedings of the 15th International Conference on Computer Systems and Technologies , 370 - 377 ( 2014 ).

4. Moccozet , L. , Tardy , C. , Opprecht , W. , Léonard , M. : Gamification-based assessment of group work . In 2013 International Conference on Interactive Collaborative Learning (ICL) , IEEE, pp. 171 - 179 ( 2013 ).

5. Deterding , S. , Dixon , D. , Khaled , R. , Nacke , L. : From game design elements to gamefulness: defining" gamification" . In Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments , pp. 9 - 15 . ( 2011 ).

6. Dicheva , D. , Dichev , C. , Agre , G. , Angelova , G.: Gamification in education: A systematic mapping study . Journal of Educational Technology & Society , vol. 18 ( 3 ), ( 2015 ).

7. Challco , G. C. , Mizoguchi , R. , Bittencourt , I. I. , Isotani , S. : Gamification of collaborative learning scenarios: Structuring persuasive strategies using game elements and ontologies . In International Workshop on Social Computing in Digital Education , pp. 12 - 28 . Springer, Cham ( 2015 ).

8. Hernández-Leo , D. , Asensio-Pérez , J. I. , Dimitriadis , Y. , Bote-Lorenzo , M. L. , JorrínAbellán , I. M. , Villasclaras-Fernández , E. D. : Reusing IMS-LD Formalized Best Practices in Collaborative Learning Structuring , Adv. Technol. Learn., vol. 2 , no. 4 , pp. 223 - 232 , ( 2005 ).

9. Jermann , P. , Soller , A. , Lesgold , A. : Computer software support for CSCL . In Dillenbourg, P. (Ed.), What we know about CSCL and implementing it in higher education, Norwell , MA, USA: Kluwer Academic Publishers, 141 - 166 ( 2004 ).

10. Hernández‐Leo , D. , Villasclaras‐Fernández , E. D. , Asensio‐Pérez , J. I. , Dimitriadis , Y. , Jorrín‐Abellán , I. M. , Ruiz‐Requies , I. , Rubia‐Avi , B. : COLLAGE: a collaborative learning design editor based on patterns . Journal of Educational Technology & Society , 9, pp. 58 - 71 , ( 2006 ).

11. Aronson , E. , Thibodeau , R.: “ The Jigsaw classroom: a cooperative strategy for an educational psychology course,” Cult . Divers. Sch., pp. 231 - 256 , ( 1992 ).

12. Clarke , J.: “Pieces of the puzzle: The jigsaw method ,” Handb. Coop. Learn. methods, pp. 34 - 50 , ( 1994 ).

13. Fluke , S. M. , Peterson , R. L. : Positive behavior interventions & supports. Strategy brief . Student Engagement Project. Lincoln , NE: University of Nebraska‐Lincoln and the Nebraska Department of Education, ( 2013 ).

14. Manathunga , K. , Hernández-Leo , D. : Authoring and enactment of mobile pyramid-based collaborative learning activities , British Journal of Educational Technology , 49 ( 2 ), pp. 262 - 275 . ( 2018 ).

15. Johnson , D. , Deterding , S. , Kuhn , K. A. , Staneva , A. , Stoyanov , S. , Hides L.: Gamification for health and wellbeing: A systematic review of the literature, Internet Interv ., vol. 6 , pp. 89 - 106 , ( 2016 ).

16. Sardi L. , Idri , A. , Fernández-Alemán , J. L.: A systematic review of gamification in e- Health , J. Biomed. Inform. , vol. 71 , pp. 31 - 48 , ( 2017 ).

17. Warmelink , H. , Koivisto , J. , Mayer , I. , Vesa , M. , Hamari , J.: Gamification of the work floor: A literature review of gamifying production and logistics operations , in Renewable and Sustainable Energy Reviews , vol. 73 ,, pp. 249 - 264 . ( 2018 ).

18. Ferreira , A. T. , Araújo , A. M. , Fernandes , S. , Miguel , I. C. : Gamification in the Workplace: A Systematic Literature Review , in Recent Advances in Information Systems and Technologies , pp. 283 - 292 . ( 2017 ).

19. Goshevski , D. , Veljanoska , J. , Hatziapostolou , H.: A review of gamification platforms for higher education , ACM Int. Conf. Proceeding Ser. , vol. Part F1309 , ( 2017 ).

20. Nah , F. F.-H. , Zeng , Q. , Telaprolu , V. R. , Ayyappa , A. P. , Eschenbrenner , B. : Gamification of Education: A Review of Literature , in HCI in Business, pp. 401 - 409 . ( 2014 ).

21. Ortega-Arranz , A. , Bote-Lorenzo , M. L. , Asensio-Pérez , J. I. , Martínez-Monés , A. , GómezSánchez , E., Dimitriadis , Y. : To reward and beyond: Analyzing the effect of reward-based strategies in a MOOC, Comput . Educ., vol. 142 , p. 103639 , ( 2019 ).

22. Heilbrunn , B. , Herzig , P. , Schill , A. : Gamification Analytics---Methods and Tools for Monitoring and Adapting Gamification Designs, in Gamification: Using Game Elements in Serious Contexts , S. Stieglitz , C. Lattemann , S. Robra-Bissantz , R. Zarnekow , and T. Brockmann, Eds. Cham: Springer International Publishing, pp. 31 - 47 . ( 2017 ).

23. Freire , M. , Serrano-Laguna , Á., Manero , B. , Martínez-Ortiz , I. , Moreno-Ger , P. , FernándezManjón , B.: Game Learning Analytics: Learning Analytics for Serious Games . ( 2016 ).

24. Trinidad , M. , Calderón , A. , Ruiz , M. A.: systematic literature review on the gamification monitoring phase: how SPI standards can contribute to gamification maturity . In International Conference on Software Process Improvement and Capability Determination , pp. 31 - 44 . Springer, Cham. ( 2018 ).

25. Perez-Colado , I. , Alonso-Fernandez , C. , Freire , M. , Martinez-Ortiz , I. , Fernandez-Manjon , B. : Game learning analytics is not informagic! , IEEE Glob. Eng. Educ. Conf. EDUCON , vol. 2018-April , pp. 1729 - 1737 , ( 2018 ).

26 Heilbrunn, B. , Herzig , P. , Schill , A. : Towards gamification analytics-requirements for monitoring and adapting gamification designs , Lect. Notes Informatics (LNI) , Proc. - Ser. Gesellschaft fur Inform. , vol. P- 232 , pp. 333 - 344 , ( 2014 ).

27 Dipace, A. , Fazlagic , B. , Minerva , T. : The design of a learning analytics dashboard: Eduopen MOOC platform redefinition procedures , J. E-Learning Knowl. Soc. , vol. 15 , no. 3 , pp. 29 - 47 , ( 2019 ).

28. Li , C. , Dong , Z. , Untch , R. H. , Chasteen , M. : Engaging computer science students through gamification in an online social network based collaborative learning environment . International Journal of Information and Education Technology , 3 ( 1 ), 72 . ( 2013 ).

29. Dillenbourg , P. , Fox , A. , Kirchner , C., Mitchell, J. , Wirsing , M. : Massive open online courses: current state and perspectives . In Dagstuhl Manifestos (Vol. 4 , No. 1). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik . ( 2014 ).

30. Shah , D. : ( 2018 ). By The Numbers: MOOCs in 2018 . https://www.classcentral. com/report/mooc-stats-2018 , last access: 2020 /04/25.