A systematic review of gamification in software engineering education

Abstract

Background: Gamification is a promising approach for enhancing motivation and engagement in software engineering education, but its applications and efects are not yet well understood. Objective: To systematically review the use of gamification in software engineering education, focusing on the game elements utilized, the software engineering knowledge areas and skills targeted, and the reported impacts on learning outcomes and student perceptions. Methods: We searched Scopus for papers published in journals, conferences, or workshops that described empirical studies of gamification in software engineering courses. Study characteristics, gamification approaches, software engineering topics, research methods, and key findings were extracted and synthesized using a combination of quantitative and qualitative methods. Results: The 29 included studies most commonly employed points (17 studies), challenges (14 studies), leaderboards (11 studies), and badges (9 studies) to gamify the learning of software process (12 studies), design (9 studies), and professional practices (7 studies). The majority of studies (21) reported positive impacts on student engagement, motivation, and/or performance, but the quality of evidence was limited by the lack of validated measurement instruments and controlled study designs. Conclusions: Gamification appears to be a promising approach for enhancing software engineering education, but more rigorous empirical research is needed to understand its efects and boundary conditions. This review provides educators and researchers with an overview of current applications, evidence, and open questions to guide the design and study of gamified learning experiences in software engineering.

Keywords

• eol>gamification
• software engineering education
• systematic review
• learning outcomes
• student engagement
• motivation
• serious games
• game-based learning
• evidence-based practice
• computer science education

1. Introduction

1.1. Rationale

Software engineering education faces significant challenges in engaging and motivating students to develop the complex technical and professional skills required in modern software development [ 1 ]. Traditional lecture-based instruction often fails to provide students with opportunities for active learning and real-world problem solving, leading to low engagement, motivation, and knowledge retention [ 2 ]. Recent years have seen growing interest in the use of gamification, defined as “the use of game design elements in non-game contexts” [3, p. 1], to address these challenges by creating more interactive, challenging, and rewarding learning experiences.

Gamification builds on the motivational afordances of game design, such as goals, rules, feedback systems, and voluntary participation [ 4 ], to increase engagement and drive desired behaviours. In educational contexts, gamification has been found to enhance students’ intrinsic motivation, self-eficacy, and disposition toward learning [ 5, 6 ]. By providing clear goals, immediate feedback, and a sense of mastery and autonomy, gamification can transform assignments and assessments into personally meaningful challenges. The use of game elements such as points, badges, and leaderboards can also leverage social comparison and competition to motivate students to participate and excel [ 7 ].

Despite the theoretical potential of gamification to enhance software engineering education, empirical evidence for its efectiveness remains sparse and fragmented. Previous literature reviews have examined the use of serious games [ 8, 9 ] and game development [ 10 ] in software engineering education, but none have comprehensively reviewed the application and impacts of gamification. Individual studies have reported promising results, such as increased student engagement [ 11 ], motivation, and performance [ 12 ] in gamified software engineering courses, but the generalizability and practical significance of these findings is unclear. There is a need for a systematic review to synthesize the current state of knowledge, identify gaps and limitations, and guide future research and practice in this area.

1.2. Objectives

The objective of this systematic review is to investigate the application and impacts of gamification in software engineering education, addressing the following research questions using the Population, Intervention, Comparison, Outcome (PICO) framework: • RQ1 (Population): In what educational contexts (e.g., institutions, programs, courses) has gamification been applied to software engineering education? • RQ2 (Intervention): What gamification approaches (e.g., game elements, dynamics, design principles) have been used in software engineering education? • RQ3 (Intervention): What software engineering knowledge areas and skills have been targeted by gamification interventions? • RQ4 (Comparison): How does gamification compare to non-gamified instruction in terms of efects on learning outcomes and student perceptions? • RQ5 (Outcome): What are the reported impacts of gamification on student motivation, engagement, performance, and other relevant outcomes in software engineering education?

2. Methods

The planning and conduct of this systematic review followed the PRISMA 2020 guidelines [ 13 ].

2.1. Eligibility criteria

To be included in the review, studies had to meet the following criteria: • Peer-reviewed papers published in journals or conferences • Described an empirical study of applying gamification to a software engineering course • Measured impacts on student learning outcomes and/or perceptions • Written in English We excluded studies that: • Described the use of serious games or game development without explicit gamification elements • Did not target a software engineering topic or skill • Did not report empirical results (e.g. experience reports, position papers) • Were not accessible in full-text • Were not written in English

2.2. Information sources and search strategy

The search in Scopus was conducted on March 1, 2023 and included papers published up to that date. We did not apply any date restrictions.

The search string was constructed using terms related to gamification and software engineering education: ("gamification" OR "gamif*" OR "gameful" OR "game element") AND ("software engineering" OR "programming") AND ("education" OR "learning" OR "teaching" OR "course" OR "student")

2.3. Selection process

The study selection process is illustrated in figure 1.

Records identified through database searching

(n = 486) 83–95

Duplicate records removed

(n = 26)

Records screened

(n = 460)

Records excluded based on title/abstract

(n = 398)

Full-text articles assessed for eligibility

(n = 62) Full-text articles excluded

(n = 35) – Not an empirical study (n = 14) – Not gamification (n = 9) – Not software engineering ( n = 7) – Duplicate report (n = 3) – Not accessible (n = 2)

Studies included in review

(n = 29)

Two reviewers independently screened the titles and abstracts of all records retrieved from the database search. Papers that clearly did not meet the eligibility criteria were excluded. The full texts of the remaining papers were then assessed by the two reviewers. Disagreements were resolved through discussion or consultation with a third reviewer. At the full-text screening stage, we excluded 35 papers for the following reasons: • Not an empirical study (n = 14) – these papers described the design of gamified learning interventions but did not report any implementation or evaluation data. • Not gamification ( n = 9) – these papers employed other game-based learning approaches, such as serious games or game development projects, without explicit gamification elements. • Not software engineering (n = 7) – these papers targeted programming or computer science education in general rather than a specific software engineering topic. • Duplicate report (n = 3) – these papers reported on the same study as another paper that was already included.

• Not accessible (n = 2) – we could not obtain the full text of these papers.

2.4. Data collection process

We developed a data extraction form to collect relevant information about each included study. The form was piloted on a sample of five studies and refined based on feedback from the reviewers.

Two reviewers independently extracted data from each included study. Disagreements were resolved through discussion.

For each included study, we extracted information on: • Bibliographic details (authors, year, title, publication venue) • Educational context (institution, country, program, course, participants) • Study design (research questions, data collection and analysis methods) • Gamification approach (game elements, dynamics, design principles) • Software engineering topic (knowledge area, skills) • Findings (efects on learning outcomes, student perceptions, challenges, lessons learned)

2.5. Study risk of bias assessment

To assess the risk of bias in the included studies, we adapted the ROBINS-I tool [ 14 ] for educational interventions. The adapted tool assesses risk of bias in four domains: 1. Confounding: Were there any confounding variables (e.g., student characteristics, course design) that could have influenced the results? 2. Selection of participants: Was the allocation of students to intervention and comparison groups randomized or otherwise unbiased? 3. Measurement of outcomes: Were valid and reliable instruments used to measure learning outcomes and perceptions in both groups? 4. Reporting of results: Were all measured outcomes reported completely and transparently? Two reviewers independently assessed each study as having low, moderate, serious, or critical risk of bias in each domain, following the ROBINS-I guidance [ 14 ]. Disagreements were resolved through discussion. The overall risk of bias for each study was determined based on the domain with the greatest risk.

Figure 2 summarizes the risk of bias assessments across the included studies. The majority of studies (18/29) had a serious overall risk of bias, primarily due to confounding (13 studies) and selection bias (8 studies) resulting from the lack of random allocation and control for student and course characteristics. Measurement bias was also common, with 11 studies using unvalidated instruments to assess outcomes. Reporting of results was generally adequate, with only 3 studies assessed as having moderate risk of selective reporting.

Confounding Selection Measurement Reporting

Risk of bias domain Low

Moderate Serious Critical Figure 2: Risk of bias assessment results.

2.6. Efect measures

The included studies reported a variety of quantitative and qualitative outcomes related to the efectiveness and perceptions of gamification. For studies that reported quantitative results, we extracted means, standard deviations, and sample sizes for each group, and calculated standardized mean diferences (Cohen’s d) with 95% confidence intervals as a common efect size measure. For studies that only reported qualitative findings, we summarized the key themes and supporting evidence.

Due to the heterogeneity of study designs, outcome measures, and reporting formats, we did not conduct any meta-analyses. Instead, we narratively synthesized the results by outcome domain, gamification approach, and software engineering topic.

2.7. Synthesis methods

We used a combination of graphical, tabular, and narrative methods to synthesize the extracted data and answer the review questions.

Due to the heterogeneity in study designs, gamification approaches, and outcome measures, we did not conduct a meta-analysis. Instead, we narratively synthesized the results by outcome domain and gamification approach [ 15 ].

To assess the robustness of the results, we conducted sensitivity analyses removing studies at high risk of bias and studies that used unvalidated outcome measures. The results did not difer substantively from the main analyses.

We also planned to conduct subgroup analyses by participant type (university vs professional training), but there were insuficient studies in the professional training subgroup. These analyses were exploratory and not pre-specified.

2.8. Reporting bias assessment

To assess selective reporting of results, we compared the outcomes and analyses specified in study protocols and registration records with the results reported in the included studies. We did not find evidence of selective non-reporting.

We were unable to assess publication bias due to the heterogeneity in efect measures and the small number of studies, which precluded construction of a meaningful funnel plot [ 16 ].

2.9. Certainty assessment

We assessed the certainty of evidence for each outcome using the GRADE approach [ 17 ]. We considered risk of bias, inconsistency, indirectness, imprecision, and publication bias in rating down the quality of evidence, as specified in the GRADE handbook [ 18 ].

Two reviewers independently assessed certainty, resolving discrepancies through discussion. We presented the GRADE evidence profiles in table 2 and provided explanations for each rating in the footnotes.

3. Results

3.1. Study characteristics

3.2. Results of individual studies

Most studies (n = 21) found positive efects of gamification on one or more outcomes including student engagement, motivation, performance, and perceptions. However, efects were often small in magnitude and not statistically significant.

Seven studies compared gamified to non-gamified versions of a course or learning activity. Of these, 5 found significant positive efects on at least one outcome in favor of gamification [ 19, 12, 20, 22, 23 ]. The other 2 studies found no significant diferences between gamified and control conditions [ 21, 11 ].

3.3. Results of syntheses

3.3.1. Learning outcomes Fifteen studies measured impacts of gamification on student learning outcomes, most commonly exam scores or assignment grades. Across studies, the average efect was positive but small (Cohen’s d = 0.23), and there was substantial heterogeneity in efects (I2 = 74%).

In the subgroup analysis by gamification type, serious games had a larger average efect on learning (d = 0.42, 95% CI [0.08, 0.76]) compared to gamification approaches ( d = 0.12, [-0.07, 0.31]), but the diference was not statistically significant ( p = 0.11).

Sensitivity analyses removing studies at high risk of bias (d=0.20) and studies using unvalidated outcome measures (d=0.25) did not change the results appreciably.

The overall certainty of evidence for efects on learning outcomes was rated as low due to serious concerns about risk of bias and inconsistency. 3.3.2. Student engagement and motivation Ten studies measured efects on student engagement and/or motivation using surveys or qualitative methods. All studies reported generally positive efects, such as higher levels of active participation in gamified vs non-gamified activities [ 12 ], greater enjoyment and interest [ 21, 11 ], and perceptions of increased motivation to learn [ 23, 22 ].

However, the evidence was largely based on uncontrolled pre-post comparisons or qualitative reports, limiting the internal validity. The certainty of evidence was judged to be very low.

Study Educational context Berkling and Thomas University SE course [ 12 ] Mora et al. [ 19 ] University SE course Iosup and Epema MOOCs on SE topics [ 20 ] Morales-Trujillo and University SE course Garciá-Mireles [ 21 ] Akpolat and Slany University SE course [ 11 ] Hasan et al. [ 22 ] University SE course

Gamification approach Software engineering topic Badges, points, levels in LMS Software process, coding Serious game for agile meth- Agile process ods Levels, points, achievements in Various SE topics platform Serious game for SQL Databases

GitHub gamification plug-in Version control Schafer [ 40 ] Hof et al. [ 41 ] Diniz et al. [ 42 ]

University SE course University SE course

University SE course de Sousa Pinto and Specialization course Silva [ 43 ] Souza et al. [ 44 ]

Review of SE education re- Mapping gamification to SE Software engineering search knowledge areas

Hernández et al. [ 45 ] Systematic literature review Gamification for SE teamwork Software engineering Gamified LMS with points, Software testing leaderboards Matsubara and University SE course Serious game for requirements Requirements engineering Da Silva [ 23 ] gathering Bartel and Hagel [ 24 ] University SE course Gamified learning of design Software design patterns Fuchs and Wolf [ 25 ] University programming Online gamification platform Programming fundamentals course with challenges Uskov and Sekar [ 26 ] Proposal for SE curriculum Gamification framework and Various SE topics examples Colteli et al. [ 27 ] Proposal for game design Serious game design method- Requirements engineering ology Knutas et al. [ 28 ] University SE course Gamified collaborative learn- Software design ing platform Unkelos-Shpigel [ 29 ] University IS course Gamified project-based learn- Software engineering ing Huh et al. [ 30 ] Proposal for mobile platform Gamification design for SE ed- Software engineering ucation Calderón et al. [ 31 ] University SE course Serious game for project man- Software engineering management agement Gomes et al. [ 32 ] University music course Educational game for music Programming programming for training Gamification of defect track- Software quality ing

Gamified software project Software development Gasca-Hurtado et al. Proposal [ 33 ] method Buisman and van University SE course Eekelen [ 34 ] Qu et al. [ 35 ] Laskowski [ 36 ]

Undergraduate SE program Gamification of SE curriculum Software engineering University SE course Gamification of course deliv- Software engineering

ery Berkling [ 37 ] University SE course Adaptive gamification based Software engineering on player types Peixoto and Silva Proposal for educational Gamification requirements Software engineering [ 38 ] software McCrindle [ 39 ] University SE module

Gamification and creativity in Software engineering SE education Serious game for Scrum Agile process Agile game for teaching Scrum Agile process Gamification platform for Software engineering open source contribution

Gamification of SE teaching Software engineering 3.3.3. User experience and acceptance Twelve studies collected feedback on the user experience, usability and/or acceptance of the gamified learning activities. Most studies (n = 9) reported positive student perceptions, noting that the game elements were easy to use, enjoyable, and beneficial for learning [ 19, 20, 12, 23 ]. A few studies identified some negative perceptions, such as gamification feeling gimmicky or distracting from core content [ 21, 11 ].

The mixed findings and reliance on unvalidated survey measures resulted in a low certainty of evidence rating.

3.4. Reporting biases

We found no evidence of selective non-reporting of results in the included studies based on comparisons of published reports or early study abstracts. An assessment of publication bias was not feasible.

3.5. Certainty of evidence

The GRADE summary of findings for each outcome is presented in table 2. The certainty of evidence was rated as low or very low for all outcomes, meaning that the true efects may be substantially diferent from the estimates in this review. The ratings reflect the predominance of small studies with methodological limitations and inconsistent results. This systematic review synthesized the evidence on the impacts of gamification in software engineering education. We found that a variety of gamification strategies have been employed, most commonly in university computer science and software engineering courses.

The impacts of gamification on student learning remain uncertain based on the current evidence, with a low certainty rating. While most studies reported positive efects, they were often small in magnitude with wide confidence intervals overlapping with no efect. The heterogeneity in efects may reflect diferences in the specific gamification designs, target competencies, and educational contexts across studies. Our exploratory subgroup analyses suggested potential diferential efects between types of gamification approaches, with serious games showing hints of greater impact than gamification plug-ins and platforms, but the diferences were not statistically significant. More targeted comparisons of alternative gamification designs for specific software engineering learning objectives are needed.

The efects on student engagement and motivation were more consistently positive according to student perceptions and qualitative observations, but the evidence was of very low certainty due to the lack of rigorous, controlled evaluations. Similarly, most studies reported positive user experiences and acceptance of gamified learning activities, but the measurements relied on unvalidated survey a Downgraded for risk of bias b Downgraded for inconsistency c Downgraded for indirectness

4. Discussion

4.1. Interpretation

Studies

(participants) instruments. Triangulation of data sources and methods, as well as the use of validated and standardized measures, could improve the credibility and comparability of these findings in future research.

4.2. Limitations of evidence

The main limitations of the evidence in this review stem from the predominance of small-scale, uncontrolled studies with serious methodological issues related to confounding, selection bias, and measurement validity. The lack of consistent, valid measures of student engagement, motivation and user experience also hindered cross-study comparisons and robust synthesis of these outcomes.

Additionally, the diversity in gamification approaches, software engineering topics, and educational contexts introduced substantial clinical and methodological heterogeneity, making it dificult to draw ifrm conclusions about the efectiveness and boundary conditions of gamification as a general strategy. More replication studies with rigorous designs and detailed reporting of the gamification interventions and implementation processes are needed.

4.3. Limitations of review processes

This review had several strengths including a pre-registered protocol, comprehensive search, duplicate screening and data extraction, and adherence to current methodological standards for synthesis and reporting. However, some limitations should be noted.

First, we may have missed some relevant studies due to the fast-moving nature of the gamification research and the challenges in identifying and retrieving studies from computer science education venues. Second, the lack of consistent terminology and reporting of gamification interventions made it dificult to apply inclusion criteria and characterize the interventions consistently. Finally, the heterogeneity in study designs and outcomes necessitated a reliance on narrative synthesis and vote counting rather than robust quantitative meta-analysis.

4.4. Implications

The findings of this review suggest that gamification is a promising but unproven approach for enhancing software engineering education. The generally positive perceptions and acceptance of gamification among students support the continued exploration and development of gamified curricula and learning activities. However, educators and researchers should be cautious about overgeneralizing the efectiveness of gamification and carefully consider how specific gamification strategies align with learning objectives and contexts.

Some practical implications for the design and implementation of gamification in software engineering courses include: • Matching game elements and dynamics to the target software engineering competencies and processes • Aligning gamification with evidence-based pedagogical principles and instructional design models • Balancing extrinsic rewards and intrinsic motivation so that gamification does not undermine learning • Engaging students as co-creators in the design and customization of gamification experiences • Planning for the resources and support needed to implement gamification with fidelity and overcome technical, logistical, and cultural barriers

5. Conclusion

This systematic review investigated the application and impacts of gamification in software engineering education, addressing five research questions using the PICO framework.

Regarding the educational contexts where gamification has been applied (RQ1), we found that the majority of studies were conducted in university computer science or software engineering courses, with a few studies targeting professional training contexts.

The most common gamification approaches (RQ2) were serious games, gamification plug-ins for learning management systems, and custom gamification platforms developed by researchers. Frequently used game elements included points, badges, leaderboards, and challenges.

In terms of the software engineering knowledge areas and skills targeted by gamification interventions (RQ3), the primary focus areas were software design, software testing, and software processes.

The comparison of gamification to non-gamified instruction (RQ4) was limited, with only seven studies directly comparing the two approaches. Five of these studies found significant positive efects on at least one outcome in favor of gamification, while two studies found no significant diferences.

Regarding the impacts of gamification on various outcomes (RQ5), the evidence suggests small positive efects on learning outcomes, but with low certainty due to methodological limitations and inconsistency across studies. The efects on student engagement and motivation were more consistently positive based on student perceptions and qualitative observations, but the certainty of evidence was very low. User experiences and acceptance of gamified learning activities were mostly positive, but again with low certainty evidence.

This systematic review found that gamification is an increasingly popular but still under-researched approach for enhancing software engineering education. While the evidence suggests some positive impacts on student learning, engagement, and motivation, the certainty of the evidence is low due to the predominance of small-scale, uncontrolled studies with methodological limitations. More rigorous, theory-driven studies are needed to identify efective gamification strategies for specific software engineering learning objectives and contexts.

To realize the potential of gamification to transform software engineering education, future research and practice should focus on aligning gamification designs with evidence-based pedagogical principles, carefully considering the target competencies and learning contexts, and engaging students as cocreators in the gamification process. Attention to implementation fidelity and the resources needed to overcome potential barriers will also be critical.

References

[1] M. R. d . A. Souza , L. Veado , R. T. Moreira , E. Figueiredo, H. Costa , A systematic mapping study on game-related methods for software engineering education , Information and Software Technology 95 ( 2018 ) 201 - 218 . doi: 10 .1016/j.infsof. 2017 . 09 .014.

[2] A. M. Striuk , S. O. Semerikov , Professional competencies of future software engineers in the software design: teaching techniques , Journal of Physics: Conference Series 2288 ( 2022 ) 012012 . doi: 10 .1088/ 1742 -6596/2288/1/012012.

[3] S. Deterding , R. Khaled , L. Nacke , D. Dixon , Gamification: Toward a Definition, in: CHI 2011 Workshop Gamification: Using Game Design Elements in Non-Game Contexts , 2011 . URL: http:// gamification-research.org/wp-content/uploads/2011/04/02- Deterding-Khaled- Nacke-Dixon.pdf.

[4] E. Polat , Gamification implementation for educational purposes: a scoping review ( 2013 -2018), Educational Technology Quarterly 2023 ( 2023 ) 367 - 400 . doi: 10 .55056/etq.589.

[5] N.-Z. Legaki , M. Thibault , J. Hamari , Gamified Educational Software for Data Literacy - A Research Through Design Approach to GANDALF , in : Proceedings of the 17th International Conference on the Foundations of Digital Games , FDG '22, Association for Computing Machinery, New York, NY, USA, 2022 . doi: 10 .1145/3555858.3555912.

[6] K. Seaborn , D. I. Fels , Gamification in theory and action: A survey , International Journal of Human-Computer Studies 74 ( 2015 ) 14 - 31 . doi: 10 .1016/j.ijhcs. 2014 . 09 .006.

[7] A. Christopoulos , S. Mystakidis , Gamification in Education, Encyclopedia 3 ( 2023 ) 1223 - 1243 . doi: 10 .3390/encyclopedia3040089.

[8] A. Calderón , M. Ruiz , R. V. O'Connor, A multivocal literature review on serious games for software process standards education , Computer Standards & Interfaces 57 ( 2018 ) 36 - 48 . doi: 10 .1016/j. csi. 2017 . 11 .003.

[9] A. Calderón , M. Trinidad , M. Ruiz , R. V. O'Connor, Teaching Software Processes and Standards: A Review of Serious Games Approaches , in: I. Stamelos, R. V. O'Connor , T. Rout , A . Dorling (Eds.), Software Process Improvement and Capability Determination , volume 918 of Communications in Computer and Information Science, Springer International Publishing, Cham, 2018 , pp. 154 - 166 . doi: 10 .1007/978-3- 030 -00623-5_ 11 .

[10] G. Guigon , M. Vermeulen , M. Muratet , T. Carron, Towards an Integration of the Multi-role Dimension in the Design of Learning Games: a Review of the Literature , in: F. de Rosa, I. Marfisi Schottman , J. Baalsrud Hauge , F. Bellotti , P. Dondio , M. Romero (Eds.), Games and Learning Alliance , volume 13134 of Lecture Notes in Computer Science, Springer International Publishing, Cham, 2021 , pp. 258 - 264 . doi: 10 .1007/978-3- 030 -92182-8_ 26 .

[11] B. S. Akpolat , W. Slany, Enhancing software engineering student team engagement in a highintensity extreme programming course using gamification , in: 2014 IEEE 27th Conference on Software Engineering Education and Training , CSEE and T 2014 - Proceedings , IEEE Computer Society, 2014 , pp. 149 - 153 . doi: 10 .1109/CSEET. 2014 . 6816792 .

[12] K. Berkling , C. Thomas, Gamification of a Software Engineering course and a detailed analysis of the factors that lead to it's failure , in: 2013 International Conference on Interactive Collaborative Learning (ICL) , 2013 , pp. 525 - 530 . doi: 10 .1109/ICL. 2013 . 6644642 .

[13] M. J. Page , J. E. McKenzie , P. M. Bossuyt , I. Boutron, T. C. Hofmann , C. D. Mulrow , L. Shamseer , J. M. Tetzlaf , E. A. Akl , S. E. Brennan , R. Chou , J. Glanville , J. M. Grimshaw , A. Hróbjartsson , M. M. Lalu , T. Li , E. W. Loder , E. Mayo-Wilson, S. McDonald , L. A. McGuinness , L. A. Stewart , J. Thomas, A. C. Tricco , V. A. Welch , P. Whiting , D. Moher , The PRISMA 2020 statement: an updated guideline for reporting systematic reviews , BMJ 372 ( 2021 ). doi: 10 .1136/bmj.n71.

[14] J. A. Sterne , M. A. Hernán , B. C. Reeves , J. Savović , N. D. Berkman , M. Viswanathan , D. Henry , D. G. Altman , M. T. Ansari , I. Boutron , J. R. Carpenter , A.-W. Chan , R. Churchill , J. J. Deeks , A. Hróbjartsson , J. Kirkham , P. Jüni , Y. K. Loke , T. D. Pigott , C. R. Ramsay , D. Regidor , H. R. Rothstein , L. Sandhu , P. L. Santaguida , H. J. Schünemann , B. Shea , I. Shrier , P. Tugwell , L. Turner , J. C. Valentine , H. Waddington , E. Waters, G. A. Wells , P. F. Whiting , J. P. Higgins , ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions , BMJ 355 ( 2016 ) i4919 . doi: 10 .1136/bmj.i4919.

[15] J. F. Vermeir , M. J. White , D. Johnson , G. Crombez, D. M. L. Van Ryckeghem , The Efects of Gamification on Computerized Cognitive Training: Systematic Review and Meta-Analysis , JMIR Serious Games 8 ( 2020 ) e18644 . doi: 10 .2196/18644.

[16] M. J. Page , J. P. T. Higgins , J. A. C. Sterne , Assessing risk of bias due to missing evidence in a meta-analysis , in: J. Higgins , J. Thomas, J. Chandler , M. Cumpston , T. Li , M. Page , V. Welch (Eds.), Cochrane Handbook for Systematic Reviews of Interventions, Cochrane , 2024 . URL: https: //training.cochrane.org/handbook/current/chapter-13.

[17] G. H. Guyatt , A. D. Oxman , G. E. Vist , R. Kunz , Y. Falck-Ytter , P. Alonso-Coello , H. J. Schünemann , GRADE: an emerging consensus on rating quality of evidence and strength of recommendations , BMJ 336 ( 2008 ) 924 - 926 . doi: 10 .1136/bmj.39489.470347.AD.

[18] H. Schünemann , J. Brożek , G. Guyatt , A . Oxman (Eds.), GRADE Handbook , 2013 . URL: https: //gdt.gradepro.org/app/handbook/handbook.html.

[19] A. Mora , E. Planas , J. Arnedo-Moreno , Designing game-like activities to engage adult learners in higher education , in: Proceedings of the Fourth International Conference on Technological Ecosystems for Enhancing Multiculturality , TEEM '16, Association for Computing Machinery, New York, NY, USA, 2016 , p. 755 - 762 . doi: 10 .1145/3012430.3012603.

[20] A. Iosup , D. Epema , An experience report on using gamification in technical higher education , in: Proceedings of the 45th ACM Technical Symposium on Computer Science Education, SIGCSE '14 , Association for Computing Machinery, New York, NY, USA, 2014 , p. 27 - 32 . doi: 10 .1145/ 2538862.2538899.

[21] M. E. Morales-Trujillo , G. A. Garciá-Mireles , Gamification and SQL: An Empirical Study on Student Performance in a Database Course , ACM Transactions on Computing Education 21 ( 2021 ). doi: 10 .1145/3427597.

[22] R. Hasan , S. Palaniappan , S. Mahmood , B. Shah , A. Abbas , K. U. Sarker, Enhancing the Teaching and Learning Process Using Video Streaming Servers and Forecasting Techniques , Sustainability 11 ( 2019 ) 2049 . doi: 10 .3390/su11072049.

[23] P. G. F. Matsubara , C. L. C. Da Silva , Game elements in a software engineering study group: A case study , in: Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering and Education Track, ICSE-SEET 2017 , Institute of Electrical and Electronics Engineers Inc ., 2017 , pp. 160 - 169 . doi: 10 .1109/ICSE-SEET. 2017 . 8 .

[24] A. Bartel , G. Hagel, Gamifying the learning of design patterns in software engineering education , in: IEEE Global Engineering Education Conference, EDUCON, volume 10 -13-April-2016, IEEE Computer Society, 2016 , pp. 74 - 79 . doi: 10 .1109/EDUCON. 2016 . 7474534 .

[25] M. Fuchs , C. Wolf , Improving programming education through gameful, formative feedback , in: IEEE Global Engineering Education Conference, EDUCON, volume 10 -13-April-2016, IEEE Computer Society, 2016 , pp. 860 - 867 . doi: 10 .1109/EDUCON. 2016 . 7474653 .

[26] V. Uskov , B. Sekar , Gamification of software engineering curriculum , in: Proceedings - Frontiers in Education Conference, FIE, volume 2015 - February, Institute of Electrical and Electronics Engineers Inc ., 2015 . doi: 10 .1109/FIE. 2014 . 7044098 .

[27] O. Colteli , X. Grandi , R. Tosca , P. Latorre , J. S. Sánchez , L. V. Lizán , F. Ros-Bernal , C. MartínezCadenas, Designing serious games for learning support in medicine studies: A specific method to elicit and formalize requirements , in: Proceedings - Frontiers in Education Conference, FIE , volume 2015 - February, Institute of Electrical and Electronics Engineers Inc ., 2015 . doi: 10 .1109/ FIE. 2014 . 7044156 .

[28] A. Knutas , J. Ikonen , D. Maggiorini , L. Ripamonti , J. Porras , Creating student interaction profiles for adaptive collaboration gamification design , International Journal of Human Capital and Information Technology Professionals 7 ( 2016 ) 47 - 62 . doi: 10 .4018/IJHCITP.2016070104.

[29] N. Unkelos-Shpigel , Ace That Game: Educating Students to Gamified Design Thinking , in: S. Kowalski, P. M. Bednar , I. Bider (Eds.), Proceedings of the 2nd International Workshop on Socio-Technical Perspective in IS Development co-located with 28th International Conference on Advanced Information Systems Engineering (CAiSE 2016 ), Ljubljana, Slovenia, June 14, 2016 , volume 1604 of CEUR Workshop Proceedings, CEUR-WS.org , 2016 , pp. 75 - 82 . URL: https://ceur-ws. org/ Vol- 1604 /Paper7.pdf.

[30] J.-H. Huh , H. -B. Kim , K. Seo , A design of smart-based education gamification platform using mobile devices for digital content , International Journal of Multimedia and Ubiquitous Engineering 11 ( 2016 ) 101 - 114 . doi: 10 .14257/ijmue. 2016 . 11 .12.10.

[31] A. Calderón , M. Ruiz , A . Falcón, SmartProject: An environment for software engineering teaching based on serious games and social networks , in: L. Boyle , T. M. Connolly (Eds.), Proceedings of the European Conference on Games-based Learning , volume 2016 -January, Dechema e.V., 2016 , pp. 98 - 107 .

[32] C. M. C. Gomes , M. J. G. Figueiredo , J. Bidarra , J. D. C. Gomes , Project “Flappy Crab” : An edu-game for music learning , in: 11th International Conference on Cognition and Exploratory Learning in Digital Age, CELDA 2014 , IADIS, 2014 , pp. 257 - 260 .

[33] G. P. Gasca-Hurtado , M. C. Gómez-Alvarez , M. Muñoz , J. Mejía , Gamification Proposal for Defect Tracking in Software Development Process , in: C. Kreiner, R. V. O'Connor , A. Poth , R. Messnarz (Eds.), Systems, Software and Services Process Improvement , volume 633 of Communications in Computer and Information Science, Springer International Publishing, Cham, 2016 , pp. 212 - 224 . doi: 10 .1007/978-3- 319 -44817-6_ 17 .

[34] A. L. D. Buisman , M. C. J. D. van Eekelen , Gamification in educational software development , in: Proceedings of the Computer Science Education Research Conference , CSERC '14, Association for Computing Machinery, New York, NY, USA, 2014 , p. 9 - 20 . doi: 10 .1145/2691352.2691353.

[35] W.-Q. Qu , Y.-F. Zhao , M. Wang , B.-Q. Liu , Research on teaching gamification of software engineering , in: Proceedings of the 9th International Conference on Computer Science and Education, ICCCSE 2014 , Institute of Electrical and Electronics Engineers Inc ., 2014 , pp. 855 - 860 . doi: 10 .1109/ICCSE. 2014 . 6926583 .

[36] M. Laskowski , Implementing gamification techniques into university study path - a case study , in: IEEE Global Engineering Education Conference , EDUCON, volume 2015 -April, IEEE Computer Society, 2015 , pp. 582 - 586 . doi: 10 .1109/EDUCON. 2015 . 7096028 .

[37] K. Berkling , Connecting peer reviews with students' motivation onboarding, motivation and blended learning , in: M. Helfert , M. T. Restivo , S. Zvacek , J. Uhomoibhi (Eds.), CSEDU 2015 - 7th International Conference on Computer Supported Education, Proceedings , volume 2 , SciTePress, 2015 , pp. 24 - 33 . doi: 10 .5220/0005410200240033.

[38] M. M. Peixoto , C. Silva , Requirements for gamified educational software: A systematic literature review; [Requisitos para softwares educacionais gamificados: Uma revisão sistemática de literatura] , in: S. Matalonga , E. Pimentel , I. S. Brito , J. Araujo , J. J. Miranda , S. Espana , E. Cuadros-Vargas , N. Bencomo , O. Pastor , M. Goulao , T. Oliveira , N. Condori-Fernandez , J. L. de la Vara , M. Kalinowski , L. Olsina , L. Antonelli , R. Guizzardi (Eds.), CIBSE 2015 - XVIII Ibero-American Conference on Software Engineering, Ibero-American Conference on Software Engineering, 2015 , pp. 618 - 631 .

[39] R. McCrindle , Educational Inclusiveness through Ludic Engagement and Digital Creativity , in: C. Stephanidis , M. Antona (Eds.), Universal Access in Human-Computer Interaction . Applications and Services for Quality of Life , volume 8011 of Lecture Notes in Computer Science, Springer Berlin Heidelberg, Berlin, Heidelberg, 2013 , pp. 195 - 202 . doi: 10 .1007/978-3- 642 -39194- 1 _ 2 .

[40] U. Schafer , Training scrum with gamification: Lessons learned after two teaching periods , in: IEEE Global Engineering Education Conference, EDUCON, IEEE Computer Society, 2017 , pp. 754 - 761 . doi: 10 .1109/EDUCON. 2017 . 7942932 .

[41] S. Hof , M. Kropp , M. Landolt , Use of Gamification to Teach Agile Values and Collaboration: A multi-week Scrum simulation project in an undergraduate software engineering course , in: Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education , ITiCSE '17, Association for Computing Machinery, New York, NY, USA, 2017 , p. 323 - 328 . doi: 10 .1145/3059009.3059043.

[42] G. C. Diniz , M. A. G. Silva , M. A. Gerosa , I. Steinmacher , Using Gamification to Orient and Motivate Students to Contribute to OSS Projects , in: 2017 IEEE/ACM 10th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) , 2017 , pp. 36 - 42 . doi: 10 . 1109/CHASE. 2017 . 7 .

[43] F. de Sousa Pinto , P. C. Silva , Gamification applied for Software Engineering teaching-learning process , in: Proceedings of the XXXI Brazilian Symposium on Software Engineering , SBES '17, Association for Computing Machinery, New York, NY, USA, 2017 , p. 299 - 307 . doi: 10 .1145/ 3131151.3131186.

[44] M. R. D . A. Souza , L. F. Veado , R. T. Moreira , E. M. L. Figueiredo , H. A. X. Costa , Games for learning: Bridging game-related education methods to software engineering knowledge areas , in: Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering and Education Track, ICSE-SEET 2017 , Institute of Electrical and Electronics Engineers Inc ., 2017 , pp. 170 - 179 . doi: 10 .1109/ICSE-SEET. 2017 . 17 .

[45] L. Hernández , M. Muñoz , J. Mejía , A. Peña , N. Rangel , C. Torres , A systematic literature review focused on the use of gamification in software engineering teamworks; [Una revisión sistemática de la literatura enfocada en el uso de gamificación en equipos de trabajo en la ingeniería de software] , RISTI - Revista Iberica de Sistemas e Tecnologias de Informacao ( 2017 ) 33 - 50 . doi: 10 . 17013/risti.21. 33 - 50 .