Gamification or gameful design attempts to raise participation through the application of game design patterns and principles in nongame environments. It has successfully been applied but in many cases gamification fails due to di↵erent kind of design phase pitfalls. Several game and gamification design taxonomies and guides exists. But it is hard to select the right one for a specific application of gamification. One of the causes is probably the fact that engineers try to implement what experienced game designer should do. We propose to apply data mining on user interaction data of gamified applications to extract insights to support and adapt the application of gamification. Therefore we started the Infoboard experiment - a two phase gamification study of a cutting-edge enterprise information sharing system.
Gamification, the use of game-mechanics in nongaming contexts [DDKN11, HH12], has been widely adapted by di↵erent services during the last years. We see online services like Stackoverflow1 using a reputation leaderboard where users get points for helpful answers. Dropbox2 rewards users helping acquire new users with more space and LinkedIn3 is motivating users to complete their profiles by presenting progress bars.
Coming along with the adaption of gamification in di↵erent domains, new insights about advantages and problems about the right usage are gained. Finding the right means to increase motivation is a non-trivial task since motivation is mainly driven by humancentric factors [Yee07]. Taking a deeper look into e↵ects coming with the increased usage of gamification becomes unavoidable in the enterprise domain. As enterprises starting to adapt gamification to enhance employee engagement and participation, the question arises what motivates people being on top of the leaderboard while others seem to completely ignoring it? The goal of this research is to examine factors motivating people to participate. We argue that gamification design must be user specific to successfully apply gamification. We also argue that especially in enterprises, it is even more important using the right mechanisms. While there are works about gamification design and which elements to apply when, we believe that by using data mining methods to determine the types of users existing in a company and to learn what elements best suits them would be a major leap in successfully introducing gamification. We will not discuss questions regarding implications of di↵erent gamification methods, such as leaderboards, and what a user’s position on the leaderboard says about the work performance, for example.
In this paper, we describe an experiment to collect data needed to learn user types and corresponding mechanisms. We present an enterprise information system we build, the Infoboard application, and the experiment design to collect the data needed for the machine learning approach. The experiment is de3http://linkedin.com/ signed to be conducted in di↵erent phases, and will be explained in detail in section 4. We also present the needed basics and fundamentals to conduct enterprise gamification research. Thus, we summarize the contributions of this work as follows: 1. We give an overview of the current state of the art of enterprise gamification (section 2). 2. We describe user-centric gamification design taking into account user types and cultures (section 3). 3. We describe an experimental set-up to gather user data and outline machine learning approaches to learn and match users and gamification elements (section 4 and 4.2.3).
The goal of the experiment is to collect a dataset that can be used for the previously mentioned machine learning approach to gamification. As to the best of our knowledge no dataset for this topic exists, we believe that this work is an essential step to collect the data and enable further research. We therefore explicitly invite others to give feedback regarding the experimental setup to be able to collect the best data possible. 2
Various studies indicate that gamification has a positive e↵ect on the use of enterprise systems. In [Cas07], Dugan et al. describe the transformation of an enterprise bookmarking system into a guessing game called Dogear. In this game, bookmarks and their tags are displayed on screen and the players have to guess, who created this bookmark. If they guess the correct creator of the bookmark, the players can gain points which is inspired by von Ahn’s ESP game [vAD04]. They report that within the first month of the release of the system, they had 87 active players from 10 different countries. A detailed analysis is missing though.
Farzan et al. [FDM08] examine the impact of game mechanics, more precisely the introduction of a points system, on a social enterprise network system (Beehive, IBM). They evaluate the impact of this points system by performing A/B testing, i.e., one half of all users are made aware of the points system, while the other half (i.e., the control group) cannot see this feature. They observe that overall, the introduction of the points system increased the activity level of users within the system. However, they also report that 72% of the users in the experimental group never visited the page which describes how to earn points. Moreover, they argue that a large portion did not even notice the existence of points. Addressing this issue further, Farzan et al. [FD08] also studied if there is any noticeable e↵ect on the usage when the points system is explicitly explained to the users. Therefore, they provided further details via email and repeated the experiment. They conclude that points systems can successfully be employed to motivate users to contribute more in an enterprise social network system, especially if combined with email notifications. Further, they conclude that the type of contribution can directly be controlled by the type of gamification applied, i.e., increasing the points for certain types of contributions will indeed result in an increase of contributions of this type. In a follow-up experiment, Farzan et al. [FDB09] increase the social interaction and diversity of content even further by introducing a badge based approach on promoting content. Although they observe an increased activity due to the introduction of gamification methods, the authors argue that they cannot make any statement about the quality of the contributions. Further studies are needed to examine this in detail. Overall they show that points and status levels motivates more activity by IBM employees within Beehive and this also inspires further activities by other users. It is also important that the incentive mechanisms incent continually to bring a return, which was the weakness of their static points [FD08] system.
Evaluating the e↵ect of gamification methods from a di↵erent perspective, Thom et al. [TMD12] study whether the removal of gamification features from an enterprise social media system has any measurable effect on user activity. They report a significant decline of user activities after removing gamification features. Interestingly, the authors also noticed some relation between user activity and their geographical location. Further the authors conclude that the organizational culture and the local culture should play a role in gamification design.
Hamari [Ham13] evaluates the use of badges in a peer-to-peer trading service. He observes that the introduction of gamification mechanisms does not automatically result in an increased use of the system by all users, but that those users, who actively inspect their own badges become more active. This supports our assumption that individual behavior plays an important role in the successful application of gamification methods in an oce scenario.
Stanculescu et al. [SBSH16] examined which game design element is more e↵ective for a predefined goal. They applied points, badges and leaderboard to an enterprise learning and social interaction Web application. In total they compared four treatment groups with either enabled a leaderboard or badges or both or none of them. Whereas points were enabled for all groups. The results of the study indicates that there is no di↵erence if only the leaderboard or only badges are visible to the user. Whereas the combination of both, leaderboard and badges visible, will result in an even greater e↵ect.
Summarizing, previous research reports an increase of users’ activity in an enterprise due to diverse game design elements. But only for some users and for a short period of time [FDM08]. Remarkable is that we could not find statements about the usability or user experience of the systems before or without the application of gamification. We think to understand gamification we should aim to measure the pure e↵ect of gamification by minimizing disruptive factors such as bad usability of the system itself. We also notice that the existing work is hardly comparable as studies are usually conducted in closed systems, and no data is publicly available. However, these studies also indicate that individual behavior has a significant influence on the success of gamification. Up to now most studies recommend an examination beyond questionnaires to understand users’ or employees’ actual behavior with gamified application. Therefore, we attempt to better understand employees’ behavior in more detail by gathering users’ interaction data and applying machine learning techniques. The interaction dataset should meet demands regarding reproducibility of results and the collected data should also be ’clean’, e.g. the influence of a bad user experience should be minimized. Thus, the experiment we conduct has the goal to produce ’clean’ data and the data should also be available for research. 3
Before explaining the experiment itself, we will introduce the game design elements and approaches that mainly influenced our Infoboard approach.
In 2011, two definitions of gamification were published. Deterding et al. [DDKN11] define gamification as “the use of game design elements in non-game contexts”. Huotari and Hamari [HH12] define it as “a process of enhancing a service with a↵ordances for gameful experiences in order to support user’s overall value creation”. Hamari et al. summarized in [HKS14] both definitions “as a process of enhancing services with (motivational) a↵ordances in order to invoke gameful experiences and further behavioral outcomes.” We interpret both definitions as implying a goal as the utility of gamification. Both describe elements of the game design world which could change a user’s experience in a di↵erent context (non-game [DDKN11], service [HH12]). Interestingly, for Deterding [DDKN11] “[...] the term ‘gameful design’ – design for gameful experiences – was also introduced as a potential alternative to ’gamification’.” Summarizing, in Deterding’s definition the goal is rather geared towards the (improved) user experience itself, in Huotari and Hamari’s definition it is the outcome driven by the user experience. We agree more with Deterding’s definition aiming on the “improvement of the user experience” achieved by gamification. 3.1
Game designers take advantage of player types [HT14] or play-personas [CD09] to set some boundaries for the game design element selection process towards a user centered game design. Designing gamification is also always a user-oriented process. This is due to the fact that users are all individuals driven by di↵erent input factors like age, gender, education, social skills and cross-cultural influences [HK13, Kha11, YMT+11, Yee07, YDN12]. In the game world this is considered by several player typologies developed on user observations and in-game behavior. The evolution seems to went from Bartles 4 and later 8 player types to Yee’s 3 motivation components or 10 motivation subcomponents [Yee07]. Hamari et al. [HT14] list existing game player typologies and state that player types have their legitimation because of the di↵erent behavior and motivation of players. It is a wide-spread assumption that also for the gamification scenario such types of players or users can be applied. Although many player typologies exist we argue that it is hard to map them to one or more specific game design elements. Beyond that, such types could change over time which seems to be a central criticism on player typologies [HT14]. 3.2
An important aspect of successful gamification is the selection of game design elements. Game design elements determine what type of gameful experiences are generated for the users. In [DDKN11], Deterding et al. provide five levels of game design elements. They distinguish between game interface design patterns, game design patterns and mechanics, game design principles and heuristics, game models and game design methods. Robinson et al. [Rob13] propose a taxonomy built on levels of expected engagement and the required commitment of the user. This taxonomy has been conceived as a decision support for game element selection.
Motivational a↵ordances, interface design patterns with a stimuli to action or “properties of an object that determine whether and how it can support one’s motivational needs” [Zha08], were found by Hamari et al. [HKS14] in 10 di↵erent forms in 24 examined studies on gamification. Jia et al. [JXKV16] examined the relation between the Big Five collections of personality traits and motivational a↵ordances through a survey which, among other things, asked for opinions to example interactions shown with videos. The results of their survey (N=248, mostly AMT4) indicate that considering personality traits helps to make gamification design choices. They plan to analyze the interaction with motivational a↵ordances on a real application.
Previous studies have shown that an improvement towards user activity and user experience is possible [HKS14]. Those studies also showed that the constellation of users (player motivations [Yee07] and player types [Bar96, Bar03, HT14]) and motivational a↵ordances (interface design patterns) and game design elements seems to be important for a successful application of gamification. 3.3
We argue that it is critical to measure challenges and risks that occur due to di↵erent types of users before introducing gamification methods. Applying the right gamification element to the right user will increase the motivation and participation while applying the wrong element can on the other hand have negative e↵ects. More importantly, due to the usually diverse set of employees, and the accompanying set of diverse characters coming from di↵erent cultures, finding the one gamification element satisfying them all is almost impossible.
Finding an optimal user and game design elements relation implies a goal or outcome we want to achieve with that relation. Thus, regarding the predefined goal of a gamification implementation extends the user and game design element relation to a goal, user and game design elements relation. In this relation the right selection of game design elements is crucial to reach the goal.
Under the assumptions that (i) gamification targets various types of users that experience game design elements di↵erently; and (ii) gamification is deployed in order to achieve some goal in the broadest sense:
4
Summarizing the previous sections, we have seen that gamification in enterprises is a hot topic but comes with several aspects that needs more research. We have also seen that current research only analyzes certain gamification aspects and not a set of di↵erent methods for di↵erent users. And, especially from a machine learning perspective, there is no dataset that 4Amazon Mechanical Turk allows us to test and compare results. With this in mind, and the knowledge from the previous section, about the various game design aspects, we have build an application and designed an experiment to start solving those problems. The path taken, starting with this experiment, will allow us to apply di↵erent gamification elements to di↵erent users within the same application, resulting in an overall increase of motivation and participation. 4.1
The Infoboard application is a modern enterprise information system. The main goal of the system is to provide users with relevant information and to enable knowledge exchange across enterprise department borders. It is build upon a distributed search engine which provides information in the form of di↵erent kind of items from indexed data of enterprise sources (enterprise wiki, internal file server) and public sources (news articles, websites, scientific publications, conference calls and funding calls).
Users can define own topics of interest and the Infoboard will continuously search for new information and present results ordered by date allowing users to quickly find the latest information. All items found are arranged as tiles with topic specific background colors. Whenever a new item has been found the tiles on the board will be updated and re-arranged. Fig. 1 shows an exemplary tile. The tile itself contains information about the topic it belongs to (see upper right corner), the source of information and date found (bottom right corner above and under the line). The knowledge sharing is supported by the elements one can find in the upper left corner and the bottom right corner. Users can up or down vote an information item. This vote information a↵ects the ordering of the Infoboard of the user who voted, but may also influence boards of other users with the same item on their board. In the end, users sharing the same interests, will see the most interesting items. The tile also shows who voted the tile last, so that users can see and connect with others sharing the same interest.
Fig. 2a shows an Infoboard of a user with di↵erent topics, marked by the di↵erent colors. As explained, users can vote items and also read the information. Therefore, we included a reader view for usability reasons, see 2b, allowing users to read and then to vote the article directly, without forcing users to switch between di↵erent browser tabs. 4.2
Our goal is not to compare a non-gamified with a gamified version of an enterprise application simultaneously. This has been done before and results have (a) The Infoboard showing information for di↵erent topics. Each topic gets a unique color assigned allowing users to easily distinguish between the topics. (b) The reader view of the Infoboard allowing users to easily read and interact with information items.
We started with a basic version of the Infoboard without any gamification elements. In this warmup phase we want to make sure that the core functionality is usable, understandable and free from major bugs. Because we do not want to apply gamification to distract from deficiencies of a useless or faulty application. This allows us to better analyze the a↵ects on di↵erent users achieved with gamification.
After we started the Infoboard experiment with an 4.2.2
In the second phase of our experiment we will enable gamification which results in visible game design elements for the users. The current Infoboard system already contains a set of di↵erent game design elements. They were carefully selected to ensure coverage of Yee’s three main components of player motivations [Yee07], Achievement, Social and Immersion, in order to address the broad range of user types which allows us to later detect and learn typical interaction patterns of these user types. Currently the following game design elements are implemented and are ready to be activated: Achievement: points; badges; leaderboard; progress bar; levels Social: feedback messages; user activities stream; group achievements (total points) Immersion: customization (color themes); redeemable points to buy a booster (a points multiplier, positive feedback loop) Fig. 3 shows a few examples of these game design elements which are currently not visible for the users. On the left side of Fig. 3a one can see the leaderboards (overall and monthly), in the middle the achieved points of all users and on the right the latest user activities. Fig. 3a shows the points detail view of a specific user with information about the current level status. We already count points and badges in the background on the currently deployed application were gamification is disabled yet. This give us similar insights to A/B testing because we can compare e.g. the achieved points and badges of both phases. The illustrations in Fig. 3 reflect the actual status and numbers after two weeks. If gamification would have been enabled it would exactly look like this to the users (same user behavior assumed) except for the booster bar (red) which is zero because gamification is disabled users can not redeem points for boosters until now.
As we are currently in the first phase and trying to maximize the usability score, the second phase has not yet started. We are currently discussing to further split the second phase into more sub-phases. In each sub-phase, only game design elements for a certain player type will be enabled. We currently review this approach to see if this is necessary or if it introduces negative side e↵ects. 4.2.3
Very similar to the future work stated by Jia et al. [JXKV16], we plan to analyze the collected user interaction data with motivational a↵ordances represented by interface design elements on the Infoboard application. We aim to find similar behaving user groups based on their interactions. In [MJ14] we provide further details on how to select (learn) a model to predict appropriate game design elements. One approach is based on the application of a support vector machine. As we assume sparse user behavior data we treat this as a problem of regression learning for which a plethora of powerful mathematical methods are available. Further we regard the gamification design problem as a special case of a recommendation problem for which matrix factorization constitutes a state-of-the-art solution. 5
Research on enterprise gamification is still in the early stages. Especially given the influence and e↵ects of diverse personnel, regarding character and culture. We think that machine learning approaches can help us determine the best game design element for a user, based on the users’ interaction patterns. Unfortunately, there is currently no dataset available that allows us to apply machine learning methods. In this paper, we introduced the Infoboard and an experiment, which is an essential step to collect the data and enable further research. The experiment is subdivided into di↵erent phases, where users interact with di↵erent functional characteristics. In the first phase, which is currently running, users only interact with a nongamified version to detect and fix influences of errors and usability flaws. In the following phases, we will enable gamification to see which user respond to what element. The game design elements we used in the Infoboard were selected based on research about user and player types and previous gamification studies. We present these works and discuss the current state of the art to provide a comprehensive overview about (a) Gamification statistics over all users of the system (leaderboards, points and recent activities). (b) Points and level information view of a specific user. [DDKN11] Sebastian Deterding, Dan Dixon, R Khaled, and L Nacke. From game design elements to gamefulness: defining gamification. Proceeding of the 15th International Academic MindTrek Conference, pages 9–15, 2011. 6
The authors would like to thank Tom Nick5 for programming and interface design support. The research leading to these results was performed in the CrowdRec project, which has received funding from the European Union Seventh Framework Programme FP7/2007-2013 under grant agreement N 610594. [Bar96]
Proceeding of the 16th International Academic MindTrek Conference, pages 17–22, 2012.
motivations to use gamification: an empirical study of gamifying exercise. Proceedings of the 21st European Conference on Information Systems SOCIAL, (JUNE):1– 12, 2013.
Does Gamification Work? A Literature Review of Empirical Studies on gamification. In In proceedings of the 47th Hawaii International Conference on System Sciences, 2014.
Types: A Metasynthesis. In Transactions of the Digital Games Research Association, 2014. [JXKV16] Yuan Jia, Bin Xu, Yamini Karanam, and Stephen Voida. Personality-targeted Gamification: A Survey Study on Personality Traits and Motivational A↵ordances. In Proceedings of the 34th Annual ACM Conference on Human Factors in Computing
Systems - CHI ’16, 2016. [HKS14] [Kha11] [MJ14] [Rob13] [YDN12]