First-hand experience of why gamification projects fail and what could be done about it

Abstract

A plethora of services, applications and scholarly research has emerged related to gamification. Regardless of the optimistic onset of this hype around the technology trend, designing gamification has proved to be a challenging endeavor; requiring multidisciplinary work that is often hindered by multiple theoretical and practical challenges. Problem-driven, theory-advancing approaches to gamification research could assist in the addressment of gamification design challenges and accelerate the growth of the gamification field however not all such approaches have been equally utilized or understood. This paper presents the case of MANGO: a project to design a gamified e-participation tool through Action Design Research (ADR). The paper reflects on the challenges of gamification design and development and possible strategies to address them. It additionally reflects on the ADR process; an under-utilized and hence possibly a superficially understood approach to gamification research. The paper is hence a guide for researchers and practitioners as to possible challenges they can face with gamification research and design and how to counteract them.

1. Introduction

During the last years, an increased interest has been observed in gamifying information systems in attempts to positively impact engagement and motivation (Hamari, Koivisto, & Sarsa, 2014; Koivisto & Hamari, 2017; Nicholson, 2012, 2015; Rigby, 2015) . Gamification refers to designing systems, services and processes to provide positive, engaging experiences similar to the ones games provide (Huotari & Hamari, 2017) . What a large stream of this gamification research and practice acknowledges, however, is that gamification is challenging to design and implement (Deterding, 2012; Deterding, 2015; Hamari et al., 2014; Hassan & Nader, 2016; Koivisto & Hamari, 2017; Morschheuser et al., 2017a; Nicholson, 2012; Rigby, 2015) . Gamification design not only require attention to the operational requirements of the gamified system, but also to the psychological needs of its target users (Burke, 2014; Morschheuser et al., 2017a; Nicholson, 2012, 2015) and the organizational and environmental context in which gamification is being introduced (Deterding, 2012; Hamari et al., 2014; Hassan, 2017; Hassan & Nader, 2016) . The theoretical and practical understanding of gamification is however often observed to lag behind with regards to the understanding of gamification design and development (Deterding, 2012; Hassan & Nader, 2016; Morschheuser et al., 2017a; Nicholson, 2015) . Fortunately, problem-driven, theory-advancing research approaches to gamification have the potential to simultaneously increase both our theoretical and practical understanding of gamification design and development, hence accelerating the growth of the gamification field on these two angles.

The case study of MANGO, reported on in this paper focused on contextualized gamification design and development in the area of e-participation through Action Design Research (ADR); a problem-drive, theory-advancing research approach. MANGO involved research work with various participants with different roles (civil servants, citizens, designers, etc.). It aimed to advance our theoretical and practical understanding of the gamification of e-participation and to answer: how to design and develop gamified e-participation? While MANGO contributed a gamification design based on the theoretical framework for the gamification of e-participation by Hassan (2017) and has additionally, theoretically extended previous research on the gamification of e-participation, the empirical research did not fully go as planned and the project was terminated before practical implementation. Hence, the main goal of the present paper is to examine why gamification projects fail and what could be done about it? The paper contributes lessons learned and suggested strategies to mitigate the possible failure of practical gamification projects based on examining the case of MANGO. The paper additionally demonstrates the possible value of problem-driven, theoryadvancing approaches such as through ADR that could ensure a contribution for gamification projects at least on one end: theoretical or practical if not on both. Overall, the paper aims to provide a guide for future gamification projects and research towards increased chances at success. 2. Case study – MANGO (Motivational Affordances iN Governmental Organizations) Project MANGO was to involve the research, design and development of an IT-artefact for gamified e-participation to answer how to design and develop gamified e-participation? It hence had a dual theoretical and practical focus and involved contextualization, iterative and participatory work. Action Research (AR) is a problem-driven, theory advancing, iterative research approach that allows work in context with multi-actors on the development of theory, and practical guidelines (Baskerville & Myers, 2004; Blum, 1955; Brydon-Miller, Greenwood, & Maguire, 2003; Järvinen, 2005) , However, AR employs an emergent research process that lacks guidelines for the design and evaluation of IT-artefacts, which is a core research goal in MANGO. Design Science Research (DSR), however, allows for the systematic and controlled design, development and evaluation of theoretical and practical artefacts (Hevner, March, Park, & S, 2004; Iivari, 2007, 2015; March & Smith, 1995) . Nonetheless, DSR is not always able to introduce artefacts that meet their predefined criteria of utility due to environmental factors often unaccounted for in the DSR controlled research environment (Iivari, 2007; March & Smith, 1995; Markus, Majchrzak, & Gasser, 2002) . On the other hand, the emergent design of AR is often volatile (Järvinen, 2005). It hence appeared at the outset of MANGO that it and possibly projects like it, need to utilize research approaches that possibly combine the strengths of AR and DSR, while attempting to water down their shortcomings. Action Design Research (ADR) is one such approach (Coenen, Donche, & Ballon, 2015; Iivari, 2015; Sein, Henfredsson, Purao, Rossi, & Lindgren, 2011) .

2.1. Case methodology

ADR emphasizes complementarity between design and theory, interaction between research participants, contextualized design, and it provides guidelines as to navigate such a research and design process. While it appears that gamification as a domain and ADR as a method may have a high degree of complementarity, as both recognize the importance of contextualized, inclusive, iterative, and theory advancing design, merely three studies joining them have appeared in the literature (Coenen, 2014; Klapztein & Cipolla, 2016; Schacht & Maedche, 2015) . This dearth in gamification research through ADR regardless its possible merits or demerits emphasizes the need for further work to be conducted through ADR to understand how it can affect the development, introduction and success or failure of gamification. The methodology may prove useful to gamification research if carried out properly as it can rapidly advance contextualized gamification theory and practice, hence, increasing the probability that the gamified artifacts developed through it would meet their design objectives. Caution however is preached (Coenen, 2014) as the result of this union between AR and DSR in the form of ADR may lead to the rapid, rudimentary development of interventions and artefacts, jeopardizing the interests of project stakeholders as happened to be the case of MANGO.

In this research, we adopted an understanding of Action Design Research (ADR) (Figure 1) according to (Sein et al., 2011) who provide the most utilized guide to ADR. ADR is conducted through four stages: Stage (1): problem formulation, Stage (2): building, intervention & Evaluation (BIE), Stage (3): reflection & learning and Stage (4): formulation of learning. The stages are guided through seven research principles.

ADR stage (1): During stage (1), researchers are to identify both a theoretical and a practical problem for their research. The theoretical problem of MANGO was identified through literature reviews and discussions with academics as recommended by (Sein et al., 2011) . The theoretical problem pertained to a lack of theory as a base for understanding and guide gamified e-participation design and development. The theoretical aim hence became to investigate how to design and develop gamified e-participation? Through literature study, concept analysis and discussions with academics, we development a theoretical framework for the gamification of e-participation (Hassan 2017) . The framework employs the self-determination theory (Baard, Deci, & Ryan, 2004) as a psychological base for understanding intrinsic motivation and gamification design, and the deliberations theory (Min, 2007; Perote-Peña & Piggins, 2015) as the normative base for civic participation. The framework additionally employs practical research bridging gamification and civic engagement (e.g. (Bista et al., 2014; Dargan & Evequoz, 2015) ) and research bridging gamification and employee engagement (Alcivar & Abad, 2016; Morschheuser, Maedche, & Walter, 2017b; Schacht & Maedche, 2015) . The peer-reviewed and published framework was next employed to guide the subsequent practical work of MANGO. The practical problem however required more conceptualization i.e. what are the specific practical objectives of this research? Accordingly, we worked closely with a smallsized governmental unit and through discussions and documentations with a middle level manager, we identified that the organization needed a customized IT-artefact to assist civil servants in the creation, filling and discussion of service quality surveys with citizens in order to improve the services provided by the organization. Gamification was imperative in the IT-artefact as it would incentives engagement with the artefact and with the underlying imitative to improve the quality of the services the unit offers, through the intrinsically rewarding experience of gamefulness rather than through extrinsic rewards although these were initially considered. Finally, Burke (2014) was employed as a guiding design process to guide the project designer through how to design gamification.

ADR stage (2): emphasizes the iterative, participative and reciprocated nature of research under ADR. This stage involved cooperative work for approximately more than eighteen months between several stakeholders: the authors – acting as researchers, designers, and project coordinators - a middle level manager from the cooperating unit, a developer, as well as research funders. First, 2 personas (Morschheuser et al., 2017a) and 89 user stories were developed to provide descriptions of the two main target groups of the intervention (civil servants, and citizens) along with a description of their expected level of technological literacy and expected user goals in order to ensure a gamification design-user fit (Hamari, Hassan, & Dias, 2018) . Furthermore, a list of technical requirements outside gamification was developed. Iterative brainstorming and theory examination took place and each participant had a clear role in the process dictated by their title, and expertise and their involvement was sought accordingly. Eight prototype wireframes were developed to communicate finalized designs the developer and the stage ended with the settlement of a Minimum Viable Product (MVP); an artefact with working core features that is ready for evaluation in its intended use context. The outcome intervention also included plans for training sessions and publicity campaigns.

ADR stage (3): This stage is rather a longitudinal one, running in parallel to most of the other stages. Researchers are advised to actively reflect on the research process and to document learnings as they might contain valuable research and design insight (Sein et al., 2011) . Research logs, emails, informal notes, and archives of designs and meetings minutes were maintained throughout all stages of this research for these purposes and were actively reflected on to discern changes that occurred to the IT-artefact and the overall intervention. These documentations were of course beneficial to study the design and development process and discern where challenges occurred as communicated in this paper.

2.2. Case summary results

ADR stage (4): The final stage (4) is intended to formalize and communicate generalizable learning through reflecting on the research problems and their addressment. Examining the actual impact of MANGO was not possible as the project was terminated, and the developed MVP was never evaluated, yet during the process, MANGO and its theory-driven focus contributed 2 peer-reviewed publications (Hassan 2017) and this present paper. Additionally, MANGO contributed a Minimum Viable Product based on a completed design both of which could be implemented in other contexts with a few modifications. The lessons in this paper from the design and development process guided by the ADR approach are valuable as this is one of the few gamification research approaches utilizing ADR additionally this is one of the few gamification papers that reflect on failure and how it can be mitigated.

3. Discussion and lessons learned

The emergent design process of ADR was valuable in that its iterations allowed for the evolution of the gamification design through increments, allowing for quick inexpensive design changes. For example, the initially competitive gamification design that was though appropriate for the artefact was changed to a mix competitive-cooperative design after discussions of personas. Similarly, a mix solitary/multiplayer gameplay was adopted instead of only solitary to widen the appeal of the artefact to users with different preferences. Such changes to the design may not have been as quick through a controlled design process. A summary of these design decisions and later changes is in Table 1. On the other hand, this emergent process led to the introduction of gamification elements or the lack of elements that were later deemed unneeded or needed, thus lengthening development time.

Adversely, the general lack of controlled lab testing of artefacts, increases the likelihood that they enter operation without intensive evaluation, failing to meet real expectations. This risk was addressed in MANGO, although perhaps not effectively enough, through iterations of Proofs of Concepts during which the IT-artefact was evaluated by the participants however it was not possible within the time and budget allocated to MANGO to carryout user evaluations of the artefact as recommended by user-centric approaches to gamification design (Deterding, 2015; Morschheuser et al., 2017a; Nicholson, 2012, 2015) . The iterative and user-centric nature of gamification design, appear to lengthen projects and place needs for multidisciplinarity and resources that should be accounted for from the initial planning phases of a gamification project.

Problem-driven, theory-advancing approaches to research emphasize the importance of identifying and documenting a theoretical and a practice problem to guide research work (BrydonMiller et al., 2003; Iivari, 2015; Sein et al., 2011) . The determined research problems at the start of this project guided the possible gamification design that could be developed in practice. While the initial design evolved through the ADR iterations, no changes were permitted to the scope of the identified problems to reduce conflicts between the research participants, Documentations of the research problems served to control expectations during the various research stages and to resolve conflicts between the research participants. It appears however that an occasional revision of these research objectives may be valuable in light of any significant changes to the available resources or environment. We chose not to revise the objectives of MANGO in light of changing circumstances leading to the lengthening of the project and an overdraft of its budget, however similar research projects may wish to avoid these consequences by revising the project objectives and possibly downsizing the scope and complexity of the work or dropping the gamification design angle in favor of at least delivering a non-gamified but operational artefact within a reasonable time and budget. This tradeoff is however subjective to the researchers and research circumstances.

Conflicts occasionally rose due to differences between participants’ backgrounds, goals, and understanding of gamification. For example, there was a common perception that gamification would merely entail the addition of elements such as badges and points to an application, while, another understanding of gamification is that it is a holistic design process that involve the consideration of how all elements in a system could add to an overall game-like, enjoyable experience Various studies exist on the effectivity or infectivity of these approaches to gamification with merits and criticism attached to each, however, having different intentions for design creates discrepancies between the individuals involved in it, leading to misallocation of time and resources during the design iterations. It is hence important to agree at the start of a project on what gamification is to all the parties involved. Additionally, documentations and having a coordinator between the research participants assists in resolving conflicts and ensuring valuable involvement of all participants when needed. Playjourney Storyline Points Ranks Missions Leaderboards Badges Avatars Rewards

Design Reasoning

Implementation

Reasoning for change (if any) Number of users is small to facilitate multiplayer.

Multiplayer

Options are available if players wish to play in groups.

To communicate a hierarchy, progress, and mastery

Numerical levels

Easier to implement since the theme was abandoned Next to competition to widens play appeal to various player types Design difficulties and fears that the target audience may perceive the theme too playfully rather than serious lead to abandoning themes.

Monthly missions for civil servants as their tasks requires more time Monthly leaderboards for civil servants as their tasks requires time Intra and across groups leaderboards to encourage collaboration.

Game-play

Competitive

A popular design able to drive activity Superhero theme

Civil servants as Heroes on a mission to improve their country. Citizen as Side-kicks who assist the Heroes.

Base tool to facilitate gamification To provide a purpose and reignite engagement To showcase mastery, provide purpose and fuel competition Newsfeed

To facilitate information provision Survey Descriptions

To facilitate information provision on the underlying political matters Sharing Extended profiles

To market the artifact to non-users For the easy identification, specially of civil servants to increase trust in government and accountability

Collaboration No theme Monthly added Monthly added Systematic

Awarded upon milestones to show mastery & purpose

Purchasable

Points as a currency to purchase badges and provide autonomy To increase autonomy, and identification with artefact

Provided list

Programing difficulties led to the adoption of an easier to implement.

Conversion of earned points to redeemable Mobile minutes

Not implemented

Abandoned due to the unsustainable nature, notice boards for “employee of the month” was suggested instead.

Commenting

To facilitate interactivity and reflections

Forums

Allows for a more lengthy and structured deliberations and reflections Internal sharing

Options added to facilitate share of posts on the artefact

ADR (Sein et al., 2011) , and gamification research (Burke, 2014; Deterding, 2012, 2015; Hamari et al., 2014; Nicholson, 2012, 2015) emphasize the importance of contextualized design and development of interventions so that the interventions influence and are influenced by their use context. While it was relatively easy for MANGO to be influenced by its organizational context, it was more difficult for it to influence its environment. Nonetheless, through the iterations, we realized the need for a larger organizational intervention to accompany the gamified IT-artefact through initiating an organizational culture that emphasizes the importance of the IT- artefact and the need for a formal introduction of the artefact through trainings. Additionally, we identified the need to reward frequent users of the IT-artefact with their announcement as “employee of the month” through notice boards in their workplace. Additionally, a publicity plan was thought to be needed to market the artefact specifically to citizens to ensure its diffusion and adoption.

While MANGO intentionally adopted a minimalistic layout for the gamified IT-artefact as the future users of the IT-artefact were thought to possess limited computer literacy skills, aesthetics do play an integral role in the perception and acceptance of gamified application. Figure 2 presents a sample wireframe of the artefact and how it was minimally implemented as part of the developed Minimum Viable Product. As the IT-artefact neared completion, the IT-artefact was, however, perceived as, too minimalistic, and unengaging. Simple aesthetics such as colors and musical chimes could add to the perceived gamefulness of an IT-artefact without demanding higher use skills and are sometimes of intuitive importance to experiencing gamefulness.

5. Conclusion

A dominant way of coming up with best practices and frameworks is through examining successful projects and lessons learned from them. However, equally can be learned from unsuccessful endeavors as they shed the light on what can fail and what should be avoided and how. This should be an especially pertinent learning approach in the realm of gamification where it is projected that most gamification projects will fail. While this research work has struggled, it benefited from utilizing a problem-driven, theory-advancing approach to research that allowed it to contribute an operationalizable design and design reasonings as to the selection of gamification elements in the design of a gamified artefact. The research additionally offers learnings on practical ADR work and gamification design for e-participation which respectively are relatively unexplored method and gamification design area. The afore discussed observations and learnings were actively presented and discussed in academic seminars and conferences and provide techniques as to the operationalization of the ADR principles and the possible positive and negative outcomes at each ADR stage and how they can be reached or mitigated. This operationalization may facilitate the implementation of further theory-drive, problem-oriented gamification research by providing one understanding of its implementation, implications and benefits. Utilizing these learnings in future projects might increase their chances at success. Future research is recommended to continue exploring the utilization of ADR in various research fields to further provide guidelines to ensure its successful utilization, Researchers are also encouraged to evaluation the gamified e-participation design contributed by MANGO and to develop it further as such evaluations were not possible yet.

Acknowledgments References

This work was supported by the Finnish foundation for economic education (grants number 10-5562 and 12-6385).

Burke, B. (2014). Gamify: How gamification motivates people to do extraordinary things. Bibliomotion, Inc. Coenen, T. (2014). The design and evaluation of a pervasive engagement game in a city neighborhood. Proceedings of the 18th International Academic MindTrek Conference on Media Business, Management, Content & Services - AcademicMindTrek (pp. 221–228). ACM.

Coenen, T., Donche, V., & Ballon, P. (2015). LL-ADR: Action design research in living labs. In Proceedings of the Annual Hawaii International Conference on System Sciences (Vol. 2015–March, pp. 4029–4038). Dargan, T., & Evequoz, F. (2015). Designing Engaging e-Government Services by Combining UserCentered Design and Gamification: A Use-Case. Proceedings of the 15th European Conference on eGovernment 2015: ECEG 2015 (p. 70). Academic Conferences Limited.

References

Alcivar , I. , & Abad , A. G. ( 2016 ). Design and evaluation of a gamified system for ERP training . Computers in Human Behavior , 58 ( 2016 ), 109 - 118 .

Baard , P. P. , Deci , E. L. , & Ryan , R. M. ( 2004 ). Intrinsic Need Satisfaction: A Motivational Basis of Performance and Weil-Being in Two Work Settings1 . Journal of Applied Social Psychology , 34 ( 10 ), 2045 - 2068 .

Baskerville , R. , & Myers , M. D. ( 2004 ). Special issue on action research in information systems: Making IS research relevant to practice: Foreword . MIS Quarterly: Management Information Systems , 28 ( 3 ), 329 - 335 .

Bista , S. K. , Nepal , S. , Paris, C. , & Colineau , N. ( 2014 ). Gamification for online communities: A case study for delivering government services . International Journal of Cooperative Information Systems , 23 ( 3 ).

Blum , F. H. ( 1955 ). Action research--A scientific approach? . Philosophy of Science , 22 ( 1 ), 1 - 7 .

Brydon-Miller , M. , Greenwood , D. , & Maguire , P. ( 2003 ). Why action research? Action Research , 1 ( 1 ), 9 - 28 .

Deterding , S. ( 2012 ). Gamification: designing for motivation . Interactions , 19 ( 4 ), 14 - 17 .

Deterding , S. ( 2015 ). The lens of intrinsic skill atoms: A method for gameful design . Human-Computer Interaction , 30 ( 3-4 ), 294 - 335 .

Hamari , J. , Hassan , L. , & Dias , A. ( 2018 ). Gamification, quantified-self or social networking? Matching users' goals with motivational technology. User Modeling and User-Adapted Interaction . 28 ( 1 ), 35 - 74 .

Hamari , J. , Koivisto , J. , & Sarsa , H. ( 2014 ). Does gamification work? - A literature review of empirical studies on gamification . In proceedings of the 47th Annual Hawaii International Conference on System Sciences (pp. 3025 - 3034 ). Hawaii, USA: IEEE.

Hassan , L. ( 2017 ). Governments Should Play Games: Towards a Framework for the Gamification of Civic Engagement Platforms . Simulation & Gaming , 48 ( 2 ), 249 - 267 .

Hassan , L. , & Nader , A. ( 2016 ). Gamification design in action: the practical cases of gamification platforms for employee work motivation and citizens' civic engagement . In Proceedings of the International Conference on ICT Management for Global Competitiveness and Economic Growthin Emerging Economies (ICTM 2016 ) (pp . 67 - 70 ).

Hevner , A. , March , S. T., Park , J. , & S , R. ( 2004 ). Design Science Research in Information Systems. MIS Quarterly: Management Information Systems , 28 ( 1 ), 75 - 105 .

Huotari , K. , & Hamari , J. ( 2017 ). A definition for gamification: Anchoring gamification in the service marketing literature . Electronic Markets , 27 ( 1 ), 21 - 31 .

Iivari , J. ( 2007 ). A paradigmatic analysis of information systems as a design science . Scandinavian Journal of Information Systems , 19 ( 2 ), 39 - 64 .

Iivari , J. ( 2015 ). Distinguishing and contrasting two strategies for design science research . European Journal of Information Systems , 24 ( 1 ), 107 - 115 .

ärvinen , P. ( 2005 ): Action research as an approach in design science , Working paper D-2005-2 , Department of Computer Sciences, University of Tampere, Presented at the EURAM (European Academy of Management) Conference , Munich, May 4-7.

Klapztein , S. , & Cipolla , C. ( 2016 ). From game design to service design: A framework to gamify services . Simulation & Gaming , 47 ( 5 ), 566 - 598 .

Koivisto , J. , & Hamari , J. ( 2017 ). The rise of motivational information systems: A review of gamification literature .

March , S. T. , & Smith , G. F. ( 1995 ). Design and natural science research on information technology.

Decision Support Systems , 15 ( 4 ), 251 - 266 .

Markus , M. L. , Majchrzak , A. , & Gasser , L. ( 2002 ). A design theory for systems that support emergent knowledge processes . MIS Quarterly: Management Information Systems , 26 ( 3 ), 179 - 212 .

Min , S. ( 2007 ). Online vs . Face to Face Deliberation: Effects on Civic Engagement . Journal of Computer Mediated Communications , 12 ( 4 ), 1369 - 1387 .

Morschheuser , B. , Hassan , L. , Werder , K. , & Hamari , J. ( 2017a ). How to design gamification? A method for engineering gamified software . Information and Software Technology . 95 .

Morschheuser , B. , Maedche , A. , & Walter , D. ( 2017b ). Designing Cooperative Gamification: Conceptualization and Prototypical Implementation . In CSCW (pp. 2410 - 2421 ).

Nicholson , S. ( 2012 ). A user-centered theoretical framework for meaningful gamification . In Proceedings of Games+Learning+Society 8.0 (GLS 8.0).

Nicholson , S. ( 2015 ). A recipe for meaningful gamification . In Gamification in Education and Business (pp.

1- 20 ). Springer International Publishing.

Perote-Peña , J. , & Piggins , A. ( 2015 ). A model of deliberative and aggregative democracy . Economics and Philosophy , 31 ( 1 ), 93 - 121 .

Rigby , C. S. ( 2015 ). Gamification and motivation . In S. P. Walz & S. Deterding (Eds.), Gameful World: Approaches, Issues, Applications (pp. 113 - 138 ). Cambridge, MA, USA: MIT Press.

Sanchez-Nielsen , E. , & Lee , D. ( 2013 ). eParticipation in Practice in Europe: The Case of“ Puzzled by Policy: Helping You Be Part of EU.” Proceedings of the 46th Hawaii International Conference on System Sciences (HICSS) (pp. 1870 - 1879 ). IEEE.

Schacht , S. , & Maedche , A. ( 2015 ). Project knowledge management while simply playing! gaming mechanics in project knowledge management systems . In Gamification in Education and Business (pp. 593 - 614 ). Springer International Publishing.

Sein , M. , Henfredsson , O. , Purao , S. , Rossi , M. , & Lindgren , R. ( 2011 ). Action Design Research. MIS Quarterly: Management Information Systems , 35 ( 1 ), 37 - 56 .