Has gamification failed, or failed to evolve? Lessons from the frontline in information systems applications

Abstract

Gamification has grown significantly since its inception in terms of the number of practitioners, technology platforms, university research, and in its application in the information systems domain as a tool to ameliorate system performance though interactive and motivational design. However, questions still remain regarding its long-term sustainability with regard to project outcomes, value creation and return on investment. The findings of this longitudinal research study over a four-year period show that the perceptions and use of gamification of a cohort of early adopter organisations has declined. This paper discusses key areas that are critical for the long-term development and evolution of gamification in information systems research, design and practice.

Keywords

• Gamification
• information systems
• design
• enterprise systems

-

1.1

Introduction

Gamification research in information systems

Gamification is defined as the the transformation of systems, services, organizations and activities to afford similar experiences, motivations and skills as good games in order to support users’ overall value creation [ 1 ]. Gamification in information systems (IS) has become a significant area of research focusing on how organisations can extract greater levels of user satisfaction, system utilisation and system effectiveness. The use of gamification is evident in a wide range of IS domains that include enterprise resource planning [ 2, 3 ], enterprise social networking services [ 4 ], IS modelling languages [ 5 ], user engagement [ 6 ], and customer service applications [ 7 ]. There is also extensive research on gamification in related enterprise application domains such as market research [ 8 ], advertising [ 9, 10 ], peer-to-peer online trading services [ 11, 12 ] and software development [ 13, 14 ] that focus on the motivational affordances of users to improve the effectiveness and efficiency of enterprise systems. The common theme in IS gamification research is a focus on how gamification impacts user behaviour in terms of positive affect [ 15, 11, 12 ], user interaction with a system [ 16, 4 ], user enjoyment in using a system [ 6, 17 ], and generating improved system outputs [ 8, 13 ].

Information systems and the organisations they support are complex, artificial and purposefully designed; they comprise of a complex amalgam of people, structures, technologies and work applications [ 18 ]. Gamification has evolved as an important tool to ameliorate system performance by providing a more cohesive connection between these different and often competing IS elements though interactive and motivational design. Given the disruptive technological and systems changes that are occurring, it is critical to understand the dynamic nature of how organisations utilise gamification to ensure system success, and how this has changed over time.

This longitudinal research focussed on how the use of gamification had shifted for a cohort of early adopter organisations between 2014 and 2018 for the purpose of understanding the nature of change and evolution in the domain. Longitudinal surveys are a useful tool in IS research in investigating contemporary issues such as processes, change, and adaptation in human-technical systems [ 18 ] which provided a suitable approach to explore the research objectives of this study. Longitudinal research involves the repeated observations of the same variables over a period of time from a defined cohort of participants to track changes in behaviour and attitudes over time. This paper presents an outline of the baseline research and methodology, followed by an outline of the key research findings. A discussion is presented on the findings followed by conclusions and opportunities for further research. 1.2

Origins of the baseline research

The first survey was conducted in 2014 as part of a doctoral research project investigating the management experience of organisations that had experimented with gamification, including an exploration of what they saw as the key enablers, barriers and required capabilities for successful gamification implementation [ 20 ] This survey had followed a research project on the development of a gamification taxonomy which was based on an audit of 304 examples of gamification that were self-reported by global organisations [ 21 ]. In setting up the 2014 survey, a selective sample of 40 global organisations was drawn from the taxonomy database and were invited to participate in a confidential online survey based on their experiences with using gamification. The selective sample was based on the criteria that organisations had undertaken a rigorous approach to their gamification project and were perceived leaders in their field. This was determined by examining the contribution of these organisations in publishing their gamification experiences through the business publications and in presentations at global gamification conferences.

A total of 25 responses were received and the combined gamified projects in this sample equated to 11.4 million users that had been affected by these gamified applications. In terms of industry profile, 12% of respondents were from banking, finance and insurance, 20% from professional services, 36% from IT, technology and software services, 8% from travel, accommodation and food services, 12% from government services and 12% miscellaneous. In terms of geographic spread, 28% were from UK/Europe, 28% from Australia/New Zealand, 36% North America, and 4% each from Asia and the Middle East/Africa. A total of 35% of projects covered several continents. The findings of the 2014 baseline study were largely positive in terms of the impact the gamification projects had on motivating target users, improving functional performance and in experimenting with a new and creative approach to their business systems development and user applications. Whilst several areas were mentioned as requiring improvement such as technology performance, data analytics, creative design and project management, respondents expressed high levels of satisfaction and in recommending gamification to colleagues as a useful business tool [ 20 ]. 2 2.1

The follow up survey: Four years later

Methodology and research objectives

The 2018 survey followed up with the same cohort of the 25 respondents from the 2014 survey via email that explained the objective of the longitudinal research and invited them to once again to participate in the study. A total of 23 organisations agreed to participate and a link was sent to the online survey instrument. Attrition of survey respondents is a common challenge in longitudinal research and this research was no exception with an attrition of two respondents (n1=25, n2=23). As the attrition could be classified as MCAR or missing completely at random [ 22 ] and given that this was a selective sample based on a uniform criterion (as detailed in 1.2) any bias and distortion in the data were deemed to be negligible.

The research objectives of the survey were to investigate the following areas:  Identify the proportion of organisations that still used gamification in their projects and to investigate the reasons for any changes;  Understand the current satisfactions levels in relation to gamification design, technology and outcomes, and how this had shifted over time;  Understand the shifts in perceptions that had occurred over time with regard to the usefulness of gamification as a business tool;  Seek recommendations on the future directions for gamification. Three multiple choice questions and four open-ended questions were posed to respondents about changes in their use of gamification and their perceptions on the effectiveness of gamification in meeting business objectives since the initial survey. A mix of quantitative and qualitative tools were used to analyse the results. For the open-ended questions, card-sorting and affinity mapping was used to derive thematic clusters. Grounded theory was applied to derive observations and insight from the data. 2.2

Summary of key findings

A summary of the key findings of the longitudinal survey is as follows: Current use of gamification. The opening multiple-choice question asked, “Compared to four years ago, how much gamification do you use in your projects?” and respondents were given a set of four options of: more, about the same, less, and none. A total of 60% of organisations reported that they no longer use gamification in their work, 22% said they were using it less, 9% stated it was about the same and only 9% said they were using more of it. A summary of the results is presented in Figure 1 below: 60% None 9% More

9% About the same 22%

Less The follow-up question asked respondents the open-ended question of: “What are the reasons for the change in the amount of gamification used in your projects?” Responses were clustered around four key themes of (a) a lack of compelling value proposition, (b) lack of management support, (c) shift in focus and responsibilities, and (d) loss of general interest. An example of the responses for each key theme are as follows: (a) “While we saw some initial improvements in engagement, there was no sustained business value” (b) “The leadership team never really fully bought in to the gamification program” (c) “I switched jobs and it there was no interest in my new department, and my replacement in my old department had no interest in picking it up” (d) “There’s less interest in the organisation now, they’ve moved onto the next big things such as AI, data and agile”.

Levels of satisfaction had declined. In relation to the question: “How would you rate

your overall satisfaction with the results from your gamification projects?” the survey results showed that satisfaction levels had declined over time e.g. in 2014 (n=25) 32% of respondents stated they were “Very Satisfied” however in 2018 (n=23) this dropped to 9% and congregated around the “somewhat satisfied” and “neutral” positions. Those that were dissatisfied rose from 0% to 13% as illustrated in Figure 2 below:

LEVELS OF SATISFACTION WITH THE RESULTS FROM GAMIFICATION PROJECTS 9% 39%

39% 32% 32% 32% 0% 4% 2018 (n=25) 2014 (n=23) 13% 0% Very Satisfied

Somewhat Satisfied

Neutral

Somewhat Dissatisfied

Dissatisfied The follow-up question asked respondents the open-ended question of: “How has your perception of gamification as a business tool changed over the last four years?” Responses were clustered around four key themes of (a) lack of a holistic business tool, (b) design issues and limitations, (c) shifting perceptions to it being a ‘fad’, (d) need for improved research. An example of the responses for each key theme are as follows: (a) “Gamification on its own is not a viable business tool” and “business principles are lacking which makes it risky” (b) “Continuous bad design made it an obvious and dull strategy” and “users became too sensitive and rejected badges and leaderboards” (c) “It’s a fad that’s peaked; it never really developed into a discipline like Design

Thinking or Agile” and “we ended up just folding it into our UX functions” (d) “Great to see more research, helps to be taken seriously” and “growth in information and use-cases has been useful”.

Net Promotor Score. In relation to the Net Promotor Score (NPS), a business measurement tool used as a proxy for customer satisfaction for a product or service [ 23 ] respondents were asked: “How likely are you to recommend gamification as a business tool to a colleague?” and had to rate their answer from zero (not likely to recommend) to ten (highly likely to recommend. The results showed a significant decline compared the same question that was asked in 2014 as illustrated in Figure 3: The results show that in this longitudinal study, project managers shifted from being overwhelmingly promotors of gamification in 2014, to ‘passives’ (disengaged) and ‘detractors’ (actively negative) by 2018. For example, in 2014 84% of respondents answered 9 or 10 out of 10, and in 2018 this dropped to 30.5% Areas for improvement. In looking for respondent recommendations on future directions for gamification, the following open-ended question was posed: “How can the gamification sector improve on its professional services to the business community? The gamification sector includes researchers, consultants, designers, course providers, authors/writers, technology vendors, app developers, game developers.” Responses were processed by using the qualitive method of affinity mapping to sort all the key elements then applied grounded theory to derive six key thematic clusters in the data:  Lack of technological innovation. Several respondents expressed frustration with the lack of technological innovation and despite the large number of vendors that had entered the the market, there was very little product and service differentiation in offerings such as games, platforms and user experiences. Illustrative quotes: “Vendors are sitting on their laurels” and “Haven’t seen much innovation from the big-name platform providers”.  Need for strategic value propositions. Several issues were raised that vendors and consultants need to better target and communicate with leadership teams in order to generate high level buy-in. Often gamification projects were initiated at the middle-management level and failed to reach high level support. Illustrative quote: “Focus less on the tactical aspects and more on the strategic ones such as systemic changes and business transformation”.  Uplift in professionalism. Several respondents stated that while the fun and engaging aspects of a gamification project was attractive, it started to wear thin where it became apparent that there was limited business expertise and substance behind their vendor’s capability. Illustrative quotes: “We’re never inviting game developers here again” and “Gamification is not the holy grail of engagement; we expect that consultants have a deeper knowledge of related business tools and approaches to support the project”.  Use more fact-based assessments. Several mentions were made that gamification should be treated as a design discipline and needs to incorporate several layers of research, stakeholder involvement and strategic design. Illustrative quotes: “Stop throwing around case studies which are not validated or keep pushing standard platforms, start approaching it as a design discipline” and “There’s much more research around now, there needs to be more of it”.  Stop overpromising. Several respondents complained that many consultants and vendors overpromised and underdelivered. They believed that organisational challenges cannot be solved by gamification alone and that vendors need to understand and communicate its limits. Illustrative quotes: “Stop promising the world!” and “Never mention ‘gamification’ to conservative organisations, just incorporate it into the overall solution”.  Full-service product offering. Several comments were made that many gamification consultants were overly focussed on technology i.e. platform vendors or game developers, whereas their organisation was looking full-service gamification offerings that included strategy, design and business transformation. Limited service offerings constrained the ability of an organisation to fully engage with a gamification project due to challenges in project management and implementation. Illustrative quotes: “Our project failed as it was difficult to integrate” and “There needs to be a clear understanding of which areas can be gamified and which shouldn’t”.

Experimentation. The open-ended question on experimentation was: “Have you or your organisation experimented with different types of gamification? If so, what are they? Using affinity mapping the responses were clustered into four key groups: 

No exploration. A total of 30% of respondents answered that they did not experiment with different types of gamification. Responses included: “No, not really”, “No, not yet”, “No, but planning to” and “No interest in exploring it”.

Different Mechanics. A total of 50% of respondents claimed that they did experiment with gamification, and continued to explain that they used different types of mechanics such as badges, feedback, challenges, narrative and points. End Uses. A total of 10% of respondents claimed they experimented with gamification by using it in different areas of the organisation such as training, learning and development, marketing and innovation management.

Strategic Approach. The final 10% of respondents stated that they used different design approaches, different technologies, experimental designs, different design teams, specialist advisory teams, digital and analogue gamified experiences, and short and long forms. These respondents were more likely to be positive about gamification; they were also still using gamification after four years, were more satisfied with project outcomes, and gave it a higher NPS. Very few respondents expressed knowledge of innovative or creative forms of playful or gameful design and affordances beyond the use of basic game mechanics, games and platforms. Nor had they mentioned inspiration or examples from recent innovations in game design such as first-person walker narrative games, or engaging gameful experiences involving wearables or virtual reality that may have a role to play in evolving the enterprise gamification domain beyond the use of basic game mechanics into more holistic forms of experiential design. This supports the findings of Rapp et al. [ 24 ] and Morschheuser et al. [ 25 ] that the full range of game design expertise has not yet been employed in the design of gamification systems. However, the design complexity we face in IS is that business systems are utilitarian with value creation objectives and an implied expectation for users to participate, relative to games which are hedonistic systems for pure entertainment with no expectations to participate. 3

Discussion

The objective of this longitudinal research was to examine how the use of gamification for a cohort of early adopter organisations had shifted over time for the purposes of understanding the nature of change and evolution for the domain. The key finding is that these organisations had significantly reduced their use of gamification due to a perceived decline in sustainable business value creation. While the respondents reported very positive outcomes and experiences with gamification four years ago, the same results could not be replicated or sustained over time.

These results are not dissimilar to a recent extensive review of empirical research into gamification by Koivisto and Hamari [ 26 ] where the researchers found that: “While the results in general lean towards positive findings about the effectiveness of gamification, the amount of mixed results is remarkable” and “the papers report negative or inconclusive results in addition to positive results”. The authors noted that this provides further support to the conclusions of previous gamification reviews “…that gamification is not a silver-bullet type of solution for achieving positive results and success, in either the research sphere, or in practice” [26, page 201].

A key insight can be drawn from the survey question pertaining to experimentation with gamification; if so few respondents engaged with meaningful experimentation with gamification in terms of different affordances, technologies and design methodologies, then perhaps the declining levels of satisfaction has more to do with (a) reaching the limits to extracting value from a narrow range of gamification design elements and technologies, (b) the limited capability, awareness or even courage of the project managers (or their consultants and vendors) in developing more innovative approaches to gamification design, and (c) issues in relation to a more effective integration of gamification within legacy business systems.

This highlights opportunities for further research into organisational competencies in developing effective and sustainable gamified applications. This can be anchored in IS theoretical foundations such as the Information System Success Model [ 27 ] which contends that an enterprise information system comprises of six key interdependent and multidimensional elements that determine system success such as system quality, information quality, service quality, system use, user satisfaction and net benefits. This would provide an important addition to the future research agenda outlined by Koivisto and Hamari [ 26 ] which outlines a comprehensive list of research initiatives to guide future research endeavors in thematic, theoretical and methodological agendas. Gamification research to date has largely focussed on motivational system design [ 12 ], motivational needs [ 28 ], and the relationships between affordances and outcomes [ 29 ], [ 17 ]. The outcomes of this longitudinal research highlight the opportunity to deepen our research perspectives of the unique requirements of gamified information systems to ensure sustainable systems success and how to manage organisational barriers to adoption.

In the broader strategic and operational context of socio-technical systems, this phenomenon can also be observed where organisations experience barriers to adopting innovative business systems in general [ 30, 31, 32, 33 ]. Such barriers are generally formed by organisation culture, legacy systems and systemic constraints [ 31 ], as well as limitations in organisational capabilities [ 34 ]. Thus, the phenomenon seen in this study may not be unique to gamification, but to any strategic innovation project within an organisational context. Therefore, if gamification is to evolve it may need to develop a more rigorous focus in organisational capability development and business transformation particularly if it is to be successfully situated as an IS discipline. Opportunities for further research in this regard is to better understand the dynamic nature of the gamification of information systems over time through longitudinal research. Lessons learnt from this study identifies the opportunity to develop substantial research projects that should include (a) a larger cohort of respondents, (b) a greater number of time points, and (c) additional questions that delve deeper into each of the key gamification elements in information systems such as technology, design, systems implementation and management. 4

Conclusion

The objective of this longitudinal study was to generate insight into how the use and perception of gamification had changed for a cohort of organisations between 2014 and 2018. The results showed that while gamification had initially produced positive results, the incidence of subsequent negative performance had a detrimental impact on organisational confidence and investment in gamification projects. Insights from this study suggest that for gamification to evolve into a preeminent place in information systems research, researchers and practitioners will need focus on enabling capability building, encouraging more innovative forms of gamification design, and taking a strategic approach to business transformation. Opportunities for further research have been identified in understanding the dynamic nature of gamification in information systems though longitudinal research to track how organisations and their business needs evolve over time, and in understanding the interdependent factors that contribute to gamified information systems implementation success.

References

1. Huotari , K. , & Hamari , J.: A Definition for Gamification: Anchoring Gamification in the Service Marketing Literature . Electron Markets 27 : 21 . https://doi.org/10.1007/s12525-015- 0212-z ( 2017 ).

2. Herzig , P. , Strahringer , S. , & Ameling , M. : Gamification of ERP systems-Exploring gamification effects on user acceptance constructs . In Multikonferenz Wirtschaftsinformatik , 793 - 804 . Braunschweig: GITO ( 2012 ).

3. Thiebes , S. , Lins , S. , & Basten , D. : Gamifying information systems-a synthesis of gamification mechanics and dynamics , in Proceedings of the 22nd European Conference on Information Systems , Munster, Germany ( 2014 ).

4. Thom , J. , Millen , D. , & DiMicco , J.: Removing gamification from an enterprise SNS . In proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work , 1067 - 1070 , New York: ACM Press ( 2012 ).

5. Herzig , P. , Jugel , K. , Momm , C. , Ameling , M. , & Schill , A. : GaML-A modeling language for gamification . In Proceedings of the 2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing , 494 - 499 . IEEE Computer Society ( 2013 ).

6. Lee , T. Y. , Dugan , C. , Geyer , W. , Ratchford , T. , Rasmussen , J. , Shami , N. S. & Lupushor , S.: Experiments on Motivational Feedback for Crowdsourced Workers . In Seventh International AAAI Conference on Weblogs and Social Media ( 2013 ).

7. Makanawala , P. , Godara , J. , Goldwasser , E. , & Le , H.: Applying gamification in customer service application to improve agents' efficiency and satisfaction . Lecture notes in Computer Science 8013 LNCS (part 2) , 548 - 557 ( 2013 ).

8. Cechanowicz , J. , Gutwin , C. , Brownell , B. , & Goodfellow , L. : Effects of gamification on participation and data quality in a real-world market research domain . In Proceedings of the First International Conference on Gameful Design, Research, and Applications , 58 - 65 ( 2013 ).

9. Terlutter , R. , & Capella , M. L. : The gamification of advertising: analysis and research directions of in-game advertising, advergames, and advertising in social network games . Journal of Advertising , 42 ( 2-3 ), 95 - 112 ( 2013 ).

10. Bittner , J. V. , & Schipper , J. : Motivational effects and age differences of gamification in product advertising . Journal of Consumer Marketing , 31 ( 5 ), 391 - 400 ( 2014 ).

11. Hamari , J.: Transforming homo economicus into homo ludens: A field experiment on gamification in a utilitarian peer-to-peer trading service , Electronic Commerce Research and Applications , 12 ( 4 ), 236 - 245 ( 2013 ).

12. Hamari , J.: Do badges increase user activity? A field experiment on the effects of gamification. Computers in Human Behavior , http://dx.doi.org/10.1016/j.chb. 2015 . 03 .036 ( 2015 ).

13. Pedreira , O. , García , F. , Brisaboa , N. , & Piattini , M.: Gamification in software engineeringA systematic mapping . Information and Software Technology , 57 , 157 - 168 ( 2015 ).

14. Liu , D. , & Santhanam , R. : Towards meaningful engagement: A framework for design and research of gamified information systems . Available at SSRN 2521283 ( 2016 ).

15. Mollick , E. , & Rothbard , N. : Mandatory fun: consent, gamification and the impact of games at work . The Wharton School Research Paper Series, 30 April , Philadelphia: The Wharton School, University of Pennsylvania, USA ( 2014 ).

16. Nah , F. , Eschenbrenner , B. , Claybaugh , C. , & Koob , P. : “ Gamification of Enterprise Systems,” Systems , 7 ( 1 ), 13 , 1 - 21 ( 2019 ).

17. Mekler , E. D. , Brühlmann , F. , Opwis , K. , & Tuch , A. N. : Disassembling gamification: the effects of points and meaning on user motivation and performance . In CHI'13 extended abstracts on human factors in computing systems , 1137 - 1142 . ACM ( 2013 ).

18. Hevner , A., S. March , J. Park, & S. Ram, S. : Design science in information systems research . MIS Quarterly , 28 ( 1 ), 75 - 105 ( 2004 ).

19. Venkatesh , A. , & Vitalari , N.O. : “Longitudinal Surveys in Information Systems Research: An Examination of Issues , Methods, and Applications,” The Information Systems Challenge: Survey Research Methods , Harvard University Press, 1991 , 115 - 144 ( 1991 ).

20. Raftopoulos , M. : How organisations play: creating stakeholder value with enterprise gamification, Doctor of Philosophy (PhD), Media and Communication , RMIT University ( 2016 ).

21. Raftopoulos , M. , Walz , S. , & Greuter , S. : How enterprises play: Towards a taxonomy for enterprise gamification . In DiGRA conference proceedings, Luneberg, Germany ( 2015 ).

22. Ployhart , R. E. , & Vandenberg , R. J. : Longitudinal research: The theory, design, and analysis of change . Journal of Management , 36 ( 1 ), 94 - 120 ( 2010 ).

23. Reichheld , F. The microeconomics of customer relationships . MIT Sloan Management Review , 47 ( 2 ), 73 ( 2006 ).

24. Rapp , A. , Hopfgartner , F. , Hamari , J. , Linehan , C. , & Cena , F. : Strengthening gamification studies: Current trends and future opportunities of gamification research . International Journal of Human-Computer Studies , 127 , 1 - 6 ( 2019 ).

25. Morschheuser , B. , Hassan , L. , Werder , K. , & Hamari , J. : How to design gamification? A method for engineering gamified software . Information and Software Technology , 95 ( 2018 ).

26. Koivisto , J. , & Hamari , J.: The rise of motivational information systems: A review of gamification research . International Journal of Information Management , 45 , 191 - 210 ( 2019 ).

27. Delone , W. H. , & McLean , E. R. : The DeLone and McLean model of information systems success: a ten-year update . Journal of Management Information Systems , 19 ( 4 ), 9 - 30 ( 2003 ).

28. Deterding , S.: The Lens of Intrinsic Skill Atoms: A Method for Gameful Design . HumanComputer Interaction , 30 ( 3-4 ), 294 - 335 ( 2015 ).

29. Hamari , J. , Koivisto , J. , & Sarsa , H.: Does gamification work? A literature review of empirical studies on gamification . In proceedings of the 47th Hawaii International Conference on System Sciences, Hawaii , USA ( 2014 ).

30. Kotter , J. P. , & Cohen , D. S.: The Heart of Change: Real-Life Stories of How People Change Their Organisations . Harvard Business School Press, USA ( 2012 ).

31. Cohen , W. A. : The Practical Drucker: Applying the Wisdom of the Worlds's Greatest Management Thinker . John Wiley and Sons, USA ( 2013 ).

32. Chesbrough , H. W. : Open Innovation: The New Imperative for Creating and Profiting from Technology: Harvard Business Press , USA ( 2005 ).

33. Christensen , C. M.: The innovator's solution: Creating and sustaining successful growth . Harvard Business Press, USA ( 2003 ).

34. Teece , D.J. , Pisano , G. , & Shuen , A. : Dynamic capabilities and strategic management . Strategic Management Journal , 18 ( 7 ), 509 - 533 ( 1997 ).