Bringing Digital Games to User Research and User Experience

Abstract

In recent years, the gaming industry has grown up and digital games have become more complex products. With this maturity comes an increasing need for formal playtesting methods from user research and scientific methods from academia. Employing user research methods in game development, especially combined qualitative (e.g., questionnaires, interviews) and quantitative (e.g., EEG, EMG, game metrics) methods lead to a better understanding of the relationship and interactions between players and games. This panel gathers game user research industry and academic experts for discussing current methodological advancements and future challenges in playtesting, usability, playability evaluation, and general game user research.

Keywords

• eol>Entertainment
• user experience (UX)
• digital games
• game metrics
• playtesting
• user studies
• empirical methods

-

INTRODUCTION Digital games have grown to be among the favorite leisure activities of billions of people around the world. Today, digital gaming battles for a share of leisure time with other traditional activities such as reading books, watching movies, listening to music, surfing the internet or sports. They attract billions of players on a regular basis online and offline, generating huge revenues. For example, market and sales statistics from the NPD group show an increasing, almost exponential, trend in hardware, software and accessories sales of digital gaming products in the past decade (see Figure 1).

However, digital games are not only a relevant from a commercial perspective. They also impose new research challenges to many scientific disciplines, new and old.

LEAVE BLANK THE LAST 2.5 cm (1”) OF THE LEFT COLUMN ON THE FIRST PAGE FOR THE

COPYRIGHT NOTICE.

With recent advancements in the field of human-computer interaction [ 12, 16 ], new tools, techniques, and methods become available for precisely measuring how people interact with entertainment technology [ 5, 9 ]. With new measurements of player-game interaction, we aim at supporting the traditional game development process and improve game design beyond regular entertainment domains (e.g., games for alternative purposes, such as education, simulation, and professional training). Improving Digital Game Design In the games industry, user testing and user-centered design [ 19, 21 ] together with playability evaluations [ 3, 10 ] have become more common for creating digital games [ 8 ]. Digital games are more about creating experiences than regular software. The full potential of digital gaming unfolds in the interaction of digital entertainment systems with human players. However, the experience of playing games is often unique and individual, consisting of many factors that are hard to assess or even measure.

Most knowledge of game design has been created during years of practice and is often based on personal experience of the game designer, which commonly comes from observing individual reactions to game mechanics. However, since such individual game design knowledge takes years to manufacture, faster insights into the complex player behavior as a reaction to game mechanics are desirable. Recent solutions have used logging of event data [ 9, 16 ] together with subjective and objective player responses to get a more complete image of gameplay experience. In a similar vein, the modeling of player behavior aims at finding optimal spots in the game and level design [ 5 ].

PANEL FOCUS Previous panels and workshops at international conferences have explored user experience (UX) measurement in games [ 2 ] and evaluation of player experience in games [ 15, 17 ]. We aim at taking the discussion of user experience evaluation in games a step further by discussing (1) what the status quo of player experience measurement and game user research is and (2) how novel measurements contribute to designing better games. Of special interest are the differences between traditional user research and emergent, quantitative behavioral tracking systems, such as game metrics or physiological recording (e.g., electroencephalography (EEG), electromyography (EMG) or eye tracking). More specifically, the panel will discuss the role of user research and user experience in games from three specific angles, which cover the current state-of-theart and emergent practices and are aligned with the expertise of the panel participants.

User Research and Digital Games User-oriented research and testing is essential to a game production, because the perceived quality of a game is directly related to UX. Game user research focuses on usability and user experience. Within the games industry a focus has been on adapting methods from usability testing to the specific context of games, while within academia the majority of published research has focused on the properties of UX – what it is, how to define it and how to measure it. Only within the past few years has knowledge from the industry been integrated into academia and vice versa.

Games are entertainment products that – in order to produce a good user experience – require the interaction between player and game to run smoothly and without disturbances of the game interface or environment. Therefore, user testing and user research is of vital importance to game production quality. User testing games can be a major challenge, because of the sheer amount of variables impacting on player-game interaction, but also because of the requirements for resources and expertise involved in many user-oriented methods. Additionally, many of the methods developed for the testing of productivity applications do not apply directly to games, because these need to do more than providing a piece of software with a high degree of usability: Games need to be entertaining. This requires the introduction of approaches such as playability heuristics [ 3, 10 ], which go beyond regular quality assurance and iterative game design. Since user-oriented testing and research is becoming more widespread in the industry [ 8, 19, 21, 26 ], knowledge about how to address the challenges of user testing has gradually been built. Testing methods developed specifically for games such as playability heuristics [ 3, 10 ], RITE [ 14 ] and specialized usability approaches [ 6, 12 ] are making inroads towards establishing a framework of tools for user testing in game production. Player experience research benefits from this development as it is now possible to approach scientific, empirical assessment of digital gameplay. By combining insights gained from numerical recording of parameters (physically from players as well as technically within entertainment software) and approaches toward qualitative assessments of experience (including behavioral observations), it is gradually becoming possible to render a high-resolution image of the complex interactions driving gameplay and player experience. The knowledge integration of player experience research and industry game user research is a focus of the panel discussions. Challenges of Modeling UX in Digital Games Gameplay experience (GX) is different from UX. In a game the focus is usually on recreational and not on functional interaction. While desktop software is primarily created with functionality in mind, digital games are created with an enjoyable experience in mind, which can stimulate cognitive and emotional processing [ 8, 19 ]. While game design is not about usability per se, a digital game benefits from adhering to the tenets of usability and usability appears to be a good foundation for an enjoyable GX [ 20 ]. The focus on the experience in the design and development of digital games results in two specific challenges for the design and user-oriented evaluation of digital games: Complexity and abstraction. Game software and player interface have to be optimized with GX in focus. Within a game, tasks and goals cannot be optimized using classic usability tests, since the difficulty or complexity of the task may provide the necessary challenge to elicit a desirable GX. Digital games are complex software with complex controls and interaction opportunities. Thus, individual GX is hard to predict. For modeling GX, we must abstract to basic forms of interaction taking place between users and the system.

Time. The temporal dimension of GX is linked to psychological experience concepts such as flow [ 18 ], immersion, or presence [ 23 ]. The experience of players will change as time progresses and their understanding of interaction with the game increases. Fundamentally, time influences learning, which is central to gaming. For example, rewards in the game ensure continuous play if they are well-balanced at the right stages of temporal and individual progression in the game.

We will discuss challenges of modeling time and abstraction as part of gameplay experience. In addition, we aim to gather a profound critique of this experience model from the experts in the panel and the audience attending this interactive session. The discussions under this topic will provide fundamental theoretical inputs and critique for modeling game experience.

The Role of Quantitative Analysis in Player Research Testing during and after game production has been performed for decades but developers have commonly been using informal methods. Recently, a variety of methodologies have however been adapted from HCI to assist with this process, for example different forms of usability-testing [ 4, 8, 20 ].

Traditional methods (e.g., audiovisual recording, interviewing players about their experience, and how game design affected it) come with a large set of limitations. Audiovisual recording is time-consuming to analyze, with everything having to be done by hand, and is limited in that not every action of the player in the game world can be tagged and measured. Post-game interviews or surveys suffer from the problem that they are difficult to relate to specific of design features. After a 50-minute play session a player may be asked what they did during the session, but their recall of the events of the game will be to greater or lesser degree imprecise, and their memories already biased. Using smaller in-game surveys in conjunction with limited playtime intervals may alleviate some of these problems; however, no golden rule has emerged yet (as to what time intervals should be used).

In contrast to traditional methods, novel measurements [ 13, 22 ], such as behavioral tracking systems pose methodological and empirical challenges for user researchers. Of special interest to us here are the differences in traditional user research and modern behavioral tracking systems, such as game metrics or physiological recording (e.g., electroencephalography (EEG), electromyography (EMG) or eye tracking). How can we model emotions using physiologically recorded data? Once we understand player emotions, how can we use this knowledge to improve game design? What does knowledge about players’ visual attention in a game contribute to level design? What conclusion can we draw from mental workload assessment during certain player tasks? These are some of the topics that we will address here.

POTENTIAL FOR DISCUSSION & EXPECTED INTEREST Over the past decade, a steadily increasing degree of research interest has been aimed at the emotional and affective aspects of user experience that digital games provide [ 7, 11, 24 ]. How games have been evaluated has historically been an informal process, however this is changing rapidly as the gaming industry has adapted techniques and processes from HCI, notably usability and UX, to evaluate games in production [ 1, 8 ]. In recent years, it has been realized that traditional usability testing does not suffice because its standard metrics (e.g., effectiveness in task completion, error rates, efficiency) are not directly applicable to all aspects of digital games [ 20 ]. Additionally, they do not provide enough information to evaluate UX, which is crucial in game development [ 19 ]. This has prompted the development of new methods for evaluating UX in games, adapted from traditional usability, as well as supplementary development of methods based on physiological measures and instrumentation (user behavior) approaches [ 16, 25 ].

UX is currently a debated topic in HCI-circles, and computer games provide unique challenges for measuring and evaluating UX. These factors alone provide a high expected interest from HCI researchers and practitioners in the panel. Add to this the size of the gaming industry and the amount of research being carried out on games, the panel topics should be of direct interest to many attendees. Key Takeaways We intend this panel to have key benefits for participants from research and industry, while giving hands-on insights into the exciting area of game user research. • • • •

An understanding of the current and emerging methods for user-research and UX evaluation in the digital games application domain.

An overview of the current status quo of know-ledge about UX in games and of the unique challenges of UX modeling in computer games.

Insights into possibilities for merging traditional qualitative with emergent quantitative measures of user behavior and UX in digital games.

How to utilize the results of game user research and game UX evaluation for improving game design. ACKNOWLEDGMENTS We thank Anders Drachen, Regan Mandryk, Heather Desurvire, Hannu Korhonen, Jan Fietz, Gareth White, and Graham McAllister for comments, creative input, and support during the writing process of a draft version of this workshop proposal paper.

References

1. Bernhaupt , R. Evaluating User Experience in Games: Concepts and Methods . Springer, Berlin, 2010 .

2. Bernhaupt , R. , IJsselsteijn, W., Mueller , F. F. , Tscheligi , M. and Wixon , D. Evaluating user experiences in games . In CHI '08 extended abstracts (Florence , Italy). ACM , 2008 , 3905 - 3908 .

3. Desurvire , H. , Caplan , M. and Toth , J. A. Using heuristics to evaluate the playability of games . In CHI '04 extended abstracts (Vienna, Austria). ACM , 2004 , 1509 - 1512 .

4. Desurvire , H. and Wiberg , C. Master of the game: assessing approachability in future game design . In CHI '08 Extended Abstracts (Florence, Italy, April 05 - 10 , 2008 ). ACM, 2008 , 3177 - 3182 .

5. Drachen , A. and Canossa , A. Towards Gameplay Analysis via Gameplay Metrics . In Proc. of MindTrek (Tampere, Finland, October 1-2) . ACM, 2009 .

6. Fabricatore , C. , Nussbaum , M. and Rosas , R. Playability in video games: a qualitative design model . HumanComputer Interaction , 17 , 4 ( 2002 ), 311 - 368 .

7. Hudlicka , E. Affective Computing for Game Design . In Proc. of GAMEON-NA (Montreal , Canada). McGill University , 2008 , 5 - 12 .

8. Isbister , K. and Schaffer , N. Game Usability: Advice from the experts for advancing the player experience . Morgan Kaufmann Publishers, Burlington, MA, 2008 .

9. Kim , J. H. , Gunn , D. V. , Schuh , E. , Phillips , B. , Pagulayan , R. J. and Wixon , D. Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems . In Proc. of CHI 2008 ( Florence, Italy) . ACM , 2008 , 443 - 452 .

10. Korhonen , H. and Koivisto , E. M. I. Playability heuristics for mobile games . In Proc. of Conf . on HCI with mobile devices and services (Espoo, Finland) . ACM , 2006 , 9 - 16 .

11. Lazzaro , N. Why We Play: Affect and the Fun of Games. In The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications . Lawrence Erlbaum, New York, NY, USA, 2003 , 679 - 700 .

12. Mandryk , R. L. , Atkins , M. S. and Inkpen , K. M. A Continuous and Objective Evaluation of Emotional Experience with Interactive Play Environments . In Proc. of CHI 2006 ( Montréal , Québec, Canada, April 2006 ). ACM, 2006 , 1027 - 1036 .

13. Mandryk , R. L. , Inkpen , K. M. and Calvert , T. W. Using Psychophysiological Techniques to Measure User Experience with Entertainment Technologies . Behaviour & Information Technology , 25 , 2 ( 2006 ), 141 - 158 .

14. Medlock , M. C. , Wixon , D. , Terrano , M. , Romero , R. L. and Fulton , B. Using the RITE method to improve products: A definition and a case study . In Proc. of UPA Conference (Orlando, Florida, USA, July 8 -12). UPA, 2002 .

15. Nacke , L. , Ambinder , M. , Canossa , A. , Mandryk , R. and Stach, T. Game Metrics and Biometrics: The Future of Player Experience Research . In Proc. of Future Play @ GDC (Vancouver, BC, Canada, 2009 .

16. Nacke , L. , Lindley , C. and Stellmach , S. Log who's playing: psychophysiological game analysis made easy through event logging . In Proc. of Fun and Games, 2nd Int. Conf. (Eindhoven, The Netherlands, October 20 - 21) . Springer, 2008 , 150 - 157 .

17. Nacke , L. E. , Drachen , A. , Kuikkaniemi , K. , Niesenhaus , J. , Korhonen , H. J. , Hoogen , W. M. v. d., Poels , K. , IJsselsteijn , W. A. and Kort , Y. A. W. d. Playability and Player Experience Research. In Proc. of DiGRA 2009 : Breaking New Ground (London , UK).

18. Nakamura , J. and Csíkszentmihályi , M. The Concept of Flow. In Handbook of positive psychology . Oxford University Press USA, New York, NY, 2002 , 89 - 105 .

19. Pagulayan , R. , Keeker , K. , Wixon , D. , Romero , R. L. and Fuller , T. User-centered design in games . In The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies , and Emerging Applications . L. Erlbaum Associates Inc., New York, NY, 2003 , 883 - 906 .

20. Pagulayan , R. and Steury , K. Beyond usability in games . interactions, 11 , 5 ( 2004 ), 70 - 71 .

21. Pagulayan , R. , Steury , K. R. , Fulton , B. and Romero , R. L. Designing for fun: user-testing case studies . In Funology: From Usability to Enjoyment . Kluwer Academic Publishers, Norwell, MA, USA, 2004 , 137 - 150 .

22. Ravaja , N. Contributions of Psychophysiology to Media Research: Review and Recommendations . Media Psychology , 6 , 2 ( 2004 ), 193 - 235 .

23. Slater , M. Presence and the sixth sense . PRESENCE: Teleoperators and Virtual Environments , 11 , 4 ( 2002 ), 435 - 439 .

24. Sykes , J. and Brown , S. Affective gaming: measuring emotion through the gamepad . In Proc. of CHI 2003. ACM , 2003 , 732 - 733 .

25. Tychsen , A. and Canossa , A. Defining personas in games using metrics . In Proc. of Future Play 08 : Research, Play, Share (Toronto, Ontario, Canada). ACM , 2008 , 73 - 80 .

26. Wixon , D. and Pagulayan , R. Halo 3: the theory and practice of a research-design partnership . interactions, 15 , 1 ( 2008 ), 52 - 55 .