Designing tailored gamification: A mixed-methods study on expert perspectives and user behavior in a gamified app for sustainability at work

Abstract

The establishment of the Sustainable Development Goals (SDGs) has put the transition to a sustainable society on the global agenda. In this respect, gamification has gained increasing attention as a tool for companies to motivate employees to adopt sustainable behaviors. Specifically, adapting gamification design to the preferences and needs of individual users has been strongly advocated. However, knowledge of personalized gamification design is largely based on conceptual assumptions and self-reported preferences. It remains thus unclear whether actual behavior of different user types matches theoretical conjectures and how user typologies can drive successful gamification design in sustainability contexts. This work addresses this gap by evaluating the design of a gamified app for sustainability at work by comparing expert evaluation (n=10) and analysis of actual user behavior (n=37) of different Hexad player types over a two-month period. In juxtaposing expert opinions and user behavior, our results reveal that actual user behavior greatly differs from expert suggestions and theoretical assumptions. Our results contribute to future research on tailored gamification by questioning the current state of tailored design theory mainly driven by self-report and pointing to the relevance of the context and non-stereotypical approaches for future personalization efforts.

Keywords

• 1 Sustainability
• tailored gamification
• personalization
• player types
• behavior

1. Introduction

Societies’ consumption and production patterns (e.g., resource and energy efficiency) demand fundamental changes towards achieving global sustainable development [1]. In this sense, game-like experiences’ potential to motivate individuals in adopting more sustainable ways of living makes gamification a promising tool to facilitate behavioral changes [2]. Still, previous studies have also pointed to mixed results [ 3,4 ] that may be attributed to a lack of considering individuals’ motivational needs and preferences, as a single gamification design solution cannot be expected to suit every person and situation [5]. While multiple studies investigated the effects of tailored gamification, especially regarding play preferences (e.g., Hexad player types [6]) in educational settings, these outcomes mostly rely on self-report through surveys, whose collected data might be inaccurate or even missing [7].

Therefore, this work draws on two approaches (expert evaluation and user behavior analysis) to investigate the design of a gamified app for sustainability at work. This mobile app was developed based on design science [ 8 ] and evaluated following the Hexad player typology on two levels (i.e., experts and employees). Our research goal is to compare expert opinions and actual user behavior to derive triangulated insights into personalized gamification design for sustainability in workplaces. Accordingly, the research questions that guide this work are: RQ1) How do gamification experts perceive different game elements in a gamified app for sustainability at work to appeal to Hexad player types? and RQ2) How do employees, who have been identified according to Hexad player types, use different game elements in a gamified app for sustainability at work? Our results provide valuable insights into experts’ perceptions and users’ behavior on designing tailored gamification in sustainability contexts. At the same time, it also discusses commonalities and differences between these two levels to contribute to advancing the field by linking existing theoretical knowledge on tailored gamification and its practical observation in the sustainability context.

2. Background

Gamification (using game elements to promote utilitarian goals by hedonic experiences [5]) has gained increasing attention as an approach to encourage sustainable behavior [9]. Previous studies have shown that game elements can have a positive impact on energy conservation [10], public transportation use [11], water conservation [12] and recycling [13]. Moreover, serious games and gamified apps for climate change engagement and sustainable lifestyles are growing [2,14].

However, studies are not unanimous on the outcomes of gamified interventions, pointing to mixed effects on sustainable travel behavior [ 3,4 ] and long-term engagement [15], for instance. In this context, adapting game elements and content to individuals’ specific needs has been advocated as an emerging research direction of personalized or tailored gamification [7]. Among the diverse characteristics analyzed by tailored gamification (e.g., demographics [16], personality traits [17] and goal orientation [18]), player typologies have become the most popular one [7]. Using player typologies to personalize gamification design has led to better outcomes, such as system engagement [19] and task completion [20].

The Hexad typology has received particular attention in tailored gamification literature [7]. Unlike many others, such as Bartle’s typology [21] that was built primarily for gaming contexts, the Hexad typology was developed explicitly for gamification [6]. Since notable efforts have been made to create a valid instrument to measure it [22–25], the Hexad is gaining popularity in practice. It distinguishes six types of players in gamified applications [6]: Achievers, motivated by competence and mastery; Free Spirits, driven by exploration and autonomy; Philanthropists, motivated by altruism and reciprocal support; Players, stimulated by extrinsic rewards; Socializers, driven by social connections; and Disruptors, motivated by change and questioning the system. Despite the clear distinction, these player types overlap [6], and thus, each user is less a definite type and more each type to some degree.

While many studies are investigating the relationship between Hexad types and preferences for specific game elements [23,26,27], previous studies have relied on theoretical assumptions and self-assessments. As a result, literature still needs to understand how different player types actually behave in gamified apps and how their behavior matches existing theoretical knowledge towards successfully tailoring gamification for sustainability, especially in workplaces.

3. Methods and material

This study is part of a design science research project on gamification for sustainability at work [28]. The research project aims to design and evaluate a mobile app for encouraging sustainable employee behavior by employing different game elements. By using the app throughout the workday, employees would be encouraged to change behavior patterns and habits in their daily work to reduce key sustainability measures in companies, such as energy consumption, water consumption and waste production. Following recommendations from the design science paradigm [ 8 ], theoretical insights informed the gamified app design, which was evaluated for further iterative development. The main goal of the current iterative cycle was to understand how the gamified app design appeals to different Hexad types. 3.1.

Participants

Two samples of participants were recruited to answer our research questions. For RQ1, the sample consisted of 7 experts with a particular focus on tailored gamification, who evaluated how the gamified app design might appeal to different Hexad types. Table 1 presents the experts and their backgrounds. They had between 3 and 10 years of experience in gamification research, being 2 (28,6%) women and 5 (71,4%) men. On average, they were around 32 years old (min = 28, max = 41).

For RQ2, the sample involved 37 employees from 5 German companies who pilot-tested the gamified app and served as the basis for evaluating how different Hexad types used the various game elements. The companies varied widely in their operations (from software development to industrial glass manufacturing to banking), but the employees we targeted can all be categorized as "white collar" office workers, as the gamified app was particularly focused on sustainable behavior in office spaces. Of these, 21 (56,7%) were women and 13 (35,1%) were men, 3 (8,1%) did not provide information about their gender. The mean age was 40 (SD = 11.7, min = 20, max = 63). 3.2.

Materials

The proposed gamified app aims to motivate employees to adopt sustainable behavior at the workplace, such as reducing waste and optimizing electricity and water consumption. The applied game elements were based on existing design principles from the literature [29,30], detailed in Table 5 in the Appendix. As a result, we implemented a variety of individual and social game elements, as described below.

Elements related to the individual perception of achievement, based on goal attainment, direct feedback and positive reinforcement principles, were introduced in the app as: a) individual goals (with progress bars), b) a personal sustainability overview with points earned in different categories of sustainable behavior, and personal badges that can be earned through specific milestones in sustainable behavior (Figure 1).

Elements related to individual learning, based on guided paths and multiple choices principles, were included as: a) personalized recommendations for actions that contribute to one's goals, b) the ability to browse all actions, c) detailed information about the relevance and value of each action for sustainable development, and d) tips for sustainability in the form of push notifications outside the app (Figure 2).

Elements related to exploration, based on the continuous excitement over new/hidden content principle, were presented in the form of a) re-rolls for actions and goals (i.e., chance), and b) unlockable actions. In addition, elements related to customization, based on the personalization of the system’s content principle, were available as: c) the possibility of bookmarking actions, and d) customizing the profile picture (Figure 3).

Finally, social elements were based on social comparison and social norming principles. In this case, a) a leaderboard, b) the opportunity to view other users' profiles for indirect competition, c) competitive goals for direct competition as a way to enable social comparison, and d) team goals to enable social collaboration towards sustainability were implemented (Figure 4).

Moreover, we juxtaposed the app prototype (guided by the above principles) with previous research that analyzed the preferences of Hexad types for different game elements [7,23,26,27] during the design process. More specifically, we aggregated the insights from these studies to ensure that the design appeals to all Hexad types from a theoretical perspective, as shown in Table 6 in the Appendix. 3.3.

Procedure

The process for the mixed-methods evaluation was twofold. Regarding RQ1, 10 relevant experts (out of which 7 participated) were invited via their e-mail and ResearchGate to answer an online survey between August and September 2022, in which they rated game elements from the developed gamified app (presenting all nonfunctional interfaces of the final application) according to how they appeal to each of the Hexad types. Although 3 experts did not respond to our request, we considered the sample of 7 experts to be appropriate in light of previous recommendations for sample sizes of 5 to 8 participants in homogeneous samples [31]. Then, Krippendorff's alpha coefficient was calculated to operationalize their agreement on each game design element [32].

Regarding RQ2, participants used the gamified app at work over a two-month period (from September to October 2022), in which they completed a validated short version of the Hexad player type survey [ 33 ] and had their in-app behavior data collected through an interaction log. Representatives of the five companies invited employees via email and intranet messages to participate in the pilot and install the application at the beginning of September 2022. Participation was voluntary and not incentivized, and employees were informed of the data collection by both accepting the privacy policy in the app and giving explicit consent in the survey. From 7,262 event logs, we calculated the frequency of use of the game elements for each of the Hexad types. Afterwards, we performed a correlation analysis in Jamovi (an open-source application for data analysis and statistical tests) using Kendall’s τb (as Hexad typology has partial overlap [23,27]) between the participants’ player type scores and the game elements usage.

4. Results 4.1.

Expert evaluation

Overall, the experts’ evaluation results (displayed in Table 2) show that their perceptions differ regarding how the various game elements of the gamified app for sustainability at work address the Hexad types. For achievementrelated elements, all experts agreed that individual goals and personal badges appealed to Achievers, while the personal sustainability overview was suggested by 6 experts to this type. Also, none of the experts recommended achievement-related elements to Socializers. Still, experts were more undecided about whether they could also be enjoyed by the other player types, which lowered the overall agreement coefficient (α = 0.464).

For learning-related elements, 4 experts indicated that action suggestions and tips for sustainability appeal to Philanthropists, and detailed action information to Achievers. Furthermore, 6 experts suggested that browsing actions appeals to Free Spirits. Yet, overall, there was little agreement on which elements appeal

E2, E4, E6, E7

E3, E7 E2 E2 E1, E2, E4, E5

E2, E3, E5, E7 E1, E2, E4, E5 E3, E7 E1, E4,

E5, E7 E1, E4, E5, E6, E1, E2, E7 E3, E5,

E6 E1, E2, E3, E4,

E5, E6 E5 E1, E3,

E4

E4, E7 E1, E2, E3, E4, E5, E6, E7 E1, E6 E1, E3, E6 E2, E4, E5, E6, E7

E1,E2,E3,E4,E more to which Hexad types (α = 0.219).

For exploration-related elements, 6 experts agreed on using unlockable actions for Achievers, 5 experts suggested chance for Free Spirits. These both elements were also suggested to Players by 4 experts each. However, there was little agreement on whether these game elements appealed to other Hexad types (α = 0.251). Meanwhile, for customization-related elements, all experts suggested profile picture for Free Spirits and 4 of them recommended action bookmarking for this same user type. Still, experts had little to no agreement regarding other user types (α = 0.345).

Finally, for social elements, all experts agreed that team goals appeal to Socializers, and 6 experts suggested competitive goals to Achievers and leaderboards to Players. Moreover, 5 experts indicated that competitive goals are suitable for Philanthropists and Players, while viewing others’ profile might be appropriate for Socializers. Yet, there was little agreement on social elements to other user types (α = 0.317).

While none of the reliability coefficients were higher than 0.8, all experts suggested individual goals and personal badges to Achievers, profile |picture to Free Spirits, and team goals to Socializers. Furthermore, personal sustainability overview, unlockable actions and competitive goals were suggested to Achievers, and browsing actions to Free Spirits by 6 experts. Also, experts agreed that social elements and picture profile were the only game elements that would appeal to Socializers, but little agreement was found to other user types (α = 0.335). 4.2.

User behavior

Descriptive statistics of Hexad types, calculated by summing the scores of respective items [ 33 ] in the sample, show that Philanthropist was the most dominant type (M = 12.4, MD = 13, SD = 1.4), followed by Achiever (M = 12, MD = 12, SD = 1. 69), Free Spirit (M = 11.7, MD = 12, SD = 2.11), Socializer (M = 11.2, MD = 11, SD = 1.99), Player (M = 10.3, MD = 11, SD = 2.72), and Disruptor (M = 7.38, MD = 7, SD = 2.61) as the least represented Hexad type. In total, employees performed 7,262 events in the gamified app, out of which 3,759 (51,7%) event logs were directly related to interaction with the game elements (as opposed to events related to opening or closing the app or completing sustainability actions).

From the descriptive statistics depicted in Table 3, it becomes evident that employees used the elements very differently. There are many logs related to learning-related elements (apart from sustainability tips) and achievement-related elements, while participants seemed to interact less with exploration-related elements. For customization-related elements, the action bookmarking feature was used fairly frequently, but there are only 12 logs related to setting the profile picture. Among social elements, it is interesting that employees predominantly looked at the leaderboard and browsed other profiles, but rarely set team or competitive goals (there was only one person who set a competitive goal).

Due to the small sample in this pilot study, we decided to conduct a one-tailed significance test for positive correlation between Hexad types and game elements (as we wanted to focus on positive relationships and not examine negative or nonexistent relationships [34]). The correlation analysis (shown in Table 4) reveals some notable correlations between Hexad types and interactions with specific game elements, whereby a τb of |0.20.29| represents a moderate association and a τb of ≥ 0.3 represents a strong association [35]. For achievement-related elements, Free Spirits are positively associated with individual goals, and Philanthropists show a positive (though not significant) relationship with the personal sustainability overview. Regarding learningrelated elements that were heavily used by participants, we see positive correlations between action suggestions and Achiever and Free Spirit types, but no significant correlations of action detail information and browsing actions with any player types. Interestingly, there is a positive significant correlation between Disruptors and tips for sustainability, which were the only element that users interacted with in the form of a push notification outside of the gamified application. In addition, we see significant positive correlations between Free Spirits and exploration-related elements (i.e., chance and unlockable actions), as well as bookmarking actions as a customization-related element. There is also a particularly significant correlation between Philanthropists and setting the profile picture. Finally, for the social elements, we see that team goals are positively associated with Philanthropists and the leaderboard has a positive correlation with Free Spirits, while there are no significant correlations for viewing other users' profiles. We refrain from interpreting the results of competitive goals, since these are likely representative only of the player type profile of the one individual who interacted with them.

Overall, we can identify several user patterns that are characteristic of Free Spirit and Philanthropist types, as well as some distinct element interactions that characterize the behavior of Achiever and Disruptor types. In our analysis, however, we cannot find any significant or even salient positive correlation between Player and Socializer types and any game design element.

5. Discussion and implications

This study aimed to extend previous work on tailored gamification and to present a new perspective on tailored design evaluation by comparing expert opinions and actual user behavior to derive triangulated insights into personalized gamification design for sustainability in workplace environments. Following the research questions, we identified how gamification experts perceive different game SD elements in an app for sustainability at work to appeal to Hexad player types (RQ1). Although there was little agreement on the suggested game elements (e.g., action suggestions) and some Hexad player types (e.g., Disruptor), at least 5 out of the 7 gamification experts agreed that: • Individual goals, personal badges, personal sustainability overview, unlockable actions, and competitive goals appeal to Achievers; • Browsing actions, chance, and setting profile picture appeal to Free Spirits; • Competitive goals appeal to Philanthropists; • Competitive goals and leaderboard appeal to

Players; • Team goals and viewing others’ profiles appeal to Socializers.

Yet, when analyzing how the 37 employees, identified according to Hexad types, used different game elements in a gamified app for sustainability at work (RQ2), the interaction logs reported different results, as: • There was no significant difference on the use of the game elements suggested by the gamification experts to Achievers, but rather they interacted more with action suggestions; • While gamification experts did not agree on any game elements to Disruptors, this user type statistically interacted more with tips for sustainability; • Free Spirits interacted more with individual goals, action suggestions, unlockable actions, action bookmarking and leaderboards, which were not noted by gamification experts, and did not interact as much with browsing actions and setting profile picture. However, chance was indeed appealing to these users; • Setting profile picture and team goals had great appeal to Philanthropists, but not competitive goals (as suggested by experts); • There was no significant difference on the use of the game elements for Players and Socializers, which also contrasts with experts’ perception.

Thus, these analyses reveal more differences than commonalities between experts’ perceptions (RQ1) and participants’ usage (RQ2) of the game elements implemented in a gamified app for sustainability at work. On the one hand, this outcome might be influenced by the results found in existing studies on tailored gamification, as the gamification experts provided similar input as theoretical suggestions [7,23,26,27]. Still, as previously explained, literature mainly relies on self-reports, whose data might be inaccurate or even missing [7]. On top of that, existing research of tailored gamification is mostly applied in other contexts than the one from this work, and our setting (as well as the focus on one gamified app design) might also be responsible for some of the discrepancies. On the other hand, we understand that our pilot study can only provide preliminary .227* .171 .066 .285* .081 .095 .088 .310* .225* .223* -.058 .168 -.113 .253* .009 -.124 .206 .113 .088 .012 -.009 .200 -.195 .036 .015 .474*** .314* .211 .053 .067 -.030 -.041 .154 .025 -.096 -.047 .095 .115 -.052 -.293 -.066 -.031 .228 -.137 .004 -.084 .128 .169 .037 .037 .111 .160 -.202 .160 .005 .197 .031 .222 .092 .161 results, given the limited sample. Thus, some results might be a mere coincidence (e.g., the relationship between using leaderboards and Free Spirits), and others might require more interaction to be properly interpreted (e.g., social elements such as team and competitive goals rely on multiple users). Their context might also have affected the interaction with game elements, which could further explain discrepancies of these users in face of experts’ perception given most of them are focused on educational domains (Table 1). Still, our current results raise questions for future research directions on tailored gamification: How reliable are existing theoretical propositions, based on self-reports, in contrast to actual behavior in gamified apps? What is the influence of context in tailored gamification and how generalizable can it be? Should experts tailor gamification based on actual user data rather than theoretical concepts? Finally, should gamification researchers and designers aim to identify which player types (and other self-reported categories) people belong to for defining appropriate game elements, or should we rather focus on less stereotypical forms (e.g., the gameful experience during interaction) for tailored gamification?

6. Conclusion

This work analyzed and compared the preferences of Hexad player types for different game elements in a gamified app for sustainability at work. While this study answers research questions related to experts’ perceptions and participants’ usage of game elements, it also discusses commonalities and differences between these two levels, and potential reasonings for the current results. Yet, this study is the first step in a long journey toward tailored gamification for sustainability at work. The present work ends with more questions than initially started, meaning that there are multiple paths to follow from here. From a theoretical perspective, this study raised questions about the reliability of self-reported preferences for game elements as opposed to actual behavior in gamified apps and about the influence of context on tailored gamification that requires more investigation from future research. As a practical implication, we provide insights that game elements will likely appeal to different types of users in diverse ways. Still, future research should also investigate alternative means to tailor gamification (e.g., context-based, dynamic personalization based on interactions) rather than relying solely on player types and other self-reported categories to create more effective gamification interventions.

7. Acknowledgements

This work was supported by Academy of Finland Flagship Programme (Forest-HumanMachine Interplay (UNITE)) [grant No 337653]; and the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie [grant No 101029543, GamInclusive].

8. References [34] [35]

Validating a Short Version of the Gamification User Types Hexad Scale, in: Proc. 2023 CHI Conf. Hum. Factors Comput. Syst., Hamburg, Germany, 2023: p. forthcoming.

H.C. Cho, S. Abe, Is two-tailed testing for directional research hypotheses tests legitimate?, J. Bus. Res. 66 (2013) 1261– 1266.

R. Botsch, Chapter 12: Significance and measures of association, Scopes Methods Polit. Sci. (2011).

9. Appendix

Individual elements (achievement) Individual goals [27], [7], [27], [7] [23]

Personal

sustainability overview Personal badges

Individual elements (learning) Action [27], [7], suggestions (Path [23] to the goal)

Action detail [27], [7], information [23]

Browse all actions [27], [7],

[23] Tips for [27], [7], sustainability [23]

Individual elements (exploration)

Chance

Unlockable [7], [23] actions

Individual elements (customization)

Actions [7] bookmarking

Set profile picture [7]

Social elements

Team goals

Leaderboard

Other user

profiles Competitive goals [27], [23] [26], [23] Socializer [27], [26], [7], [23] [27], [7], [23] [27], [26], [7], [23] [7], [23]

References

UNEP , ABC of SCP: Clarifying concepts on sustainable consumption and production , ( 2010 ).

Hamari , J. Quist , Gamified apps for sustainable consumption: A systematic review , CEUR Workshop Proc . 3147 ( 2022 ) 135 - 145 .

Policy. 4 ( 2016 ) 269 - 278 .

Rudel , M. Raubal , Beyond Limitations of Current Behaviour Change Apps for Sustainable Mobility: Insights from a User-Centered Design and Evaluation Process , Sustainability. 11 ( 2019 ) 2281 .

Manage. 45 ( 2019 ) 191 - 210 .

A. Marczewski , User Types, in: Even Ninja Monkeys Like to Play, 1st ed., CreateSpace Independent Publishing Platform , 2015 : pp. 65 - 80 .

Stud. 144 ( 2020 ) 102495 .

Inf . Syst. 24 ( 2007 ) 45 - 78 .

D.F. Galeote , M. Rajanen , D. Rajanen , N.- Z. Legaki , D.J. Langley , J. Hamari , Gamification for climate change engagement: review of corpus and future agenda , Environ. Res. Lett . 16 ( 2021 ) [16] [17] [18] [19] [ 20 ] 63004 .

Res. 106 ( 2020 ) 388 - 405 .

M. Ferron , E. Loria , A. Marconi , P. Massa , Play&Go, an Urban Game Promoting Behaviour Change for Sustainable Mobility , Interact. Des. Archit. J. 40 ( 2019 ) 24 - 45 .

K. Koroleva , J. Novak , How to Engage with Sustainability Issues We Rarely Experience? A Gamification Model for Collective Awareness Platforms in WaterRelated Sustainability , Sustainability. 12 ( 2020 ) 712 .

ACM Human-Comp . Int. 5 ( 2021 ) 1 - 21 .

D. Wemyss , F. Cellina , E. Lobsiger-Kägi , V. de Luca, R. Castri, Does it last? Longterm impacts of an app-based behavior change intervention on household electricity savings in Switzerland, Energy Res . Soc. Sci . 47 ( 2019 ) 16 - 27 .

Comput. Interact . Play, ACM, 2017 : pp.

Y. Jia , B. Xu , Y. Karanam , S. Voida , Personality-targeted Gamification, in: Proc. 2016 CHI Conf. Hum. Factors Comput. Syst., ACM , 2016 : pp. 2001 - 2013 .

Technol. 8 ( 2015 ) 261 - 273 .

Learn . Technol. 12 ( 2019 ) 16 - 28 .

M. Passalacqua , S. Sé né cal, M. Fré dette, L.E. Nacke , R. Pellerin , P.-M. Lé ger, [ 27 ] [28] [29] [30] Should Gamification be Personalized? A Self- deterministic Approach , AIS Trans.

Human-Computer Int . 13 ( 2021 ) 265 - 286 .

R. Bartle , Hearts, clubs, diamonds, spades: Players who suit MUDs, J. MUD Res . 1 ( 1996 ) 19 .

Stud. 127 ( 2019 ) 95 - 111 .

Game Mech . HCII 2021. Lect. Notes Comput. Sci . Vol 12789 , Springer Nature , 2021 : pp. 219 - 238 .

N. Taşkın , E.K. Çakmak , Adaptation of Modified Gamification User Types Scale into Turkish, Contemp . Educ. Technol. 12 ( 2020 ) 1 - 17 .

Symp. Comput. Interact . Play, ACM, 2020 : pp. 206 - 218 .

Comput. Interact. , ACM , New York, NY, USA, 2019 : pp. 1 - 8 .

G.F. Tondello , R.R. Wehbe , L. Diamond , M. Busch , A. Marczewski , L.E. Nacke, The gamification user types Hexad scale , CHI Play 2016 - Proc . 2016 Annu. Symp.

Comput. Interact. Play. ( 2016 ) 229 - 243 .

J. Krath , Gamification for Sustainable Employee Behavior , in: Ext. Abstr. 2021 Annu. Symp. Comput. Int. Play , ACM , New York, NY, USA, 2021 : pp. 411 - 414 .

J. Krath , H.F.O. von Korflesch , Designing gamification and persuasive systems: a systematic literature review , in: 5th Int.

[33] GamiFIN Conf ., CEUR Workshop Proceedings , 2021 : pp. 100 - 109 .

A.J. Kuzel , Sampling in Qualitative Inquiry, in: B. Crabtree , W. Miller (Eds.), Doing Qual. Res ., Sage Publications , Newbury Park, CA, 1992 : pp. 31 - 44 .

A.F. Hayes , K. Krippendorff , Answering the Call for a Standard Reliability Measure for Coding Data, Commun . Methods Meas . 1 ( 2007 ) 77 - 89 .

Nacke , Hexad- 12 : Developing and [ 27 ], [7], [ 23 ] [27] [7], [ 23 ] [23] [23] [26], [ 23 ] [23] [23] [27], [7], [ 23 ] [27], [7], [ 23 ] [27], [7], [ 23 ] [27], [7], [ 23 ] [27], [ 23 ] [27], [7], [ 23 ] [27], [7] [27], [7], [ 23 ] [27], [7], [23] [26] [27] [7] [23] [23] [7], [23] [23] [7] [7] [23] [27], [ 26 ], [ 23 ] [27], [7], [ 23 ] [23] [27], [7], [ 23 ] [26], [7], [ 23 ] [27], [ 23 ] [27], [ 26 ], [7], [ 23 ]