Using A ective Loop as Auxilliary Design Tool for Video Games

Abstract

As modern technologies become more apparent and persistent, human-computer interaction becomes an important research topic. With birth of a ective computing, which aims at developing systems capable of detecting and processing emotionally signi cant data from the environment, new possibilities for applications unfold, and video games can bene t from them as well. Bringing innovative solutions to this area involves new modes of a ective data collection and a ect modelling of various aspects of the game experience. My research, focusing on a ective game design patterns, is located on the intersection of modelling player a ect and a ective game design framework. In this paper, an outline of how a ective computing ideas (especially a ective loop) are introduced to video game design is presented. A new approach to designing video games in the form of a ective game design patterns is proposed, together with research method description and summary of studies conducted so far.

Keywords

• human-computer interaction
• a ective computing
• a ective loop
• video game design

-

In the times of great scienti c and technological progress, almost each and every human activity is accompanied by modern technology. As computers and mobile devices become more persistent and ubiquitous, people's everyday contact with them emerge as a broad topic for discussion, improvement, and { therefore { research. With interfaces being a direct level of interaction between the user and the machine, designers and developers face a challenge to make this interaction as easy and natural as possible.

A movement taking a closer look on these problems, originating in early 1980s, is called Human-Computer Interaction (HCI). However, it was not until late 1990s that Rosalind Picard from MIT Media Lab suggested that HCI should take emotions in the interaction into consideration. She started a new approach in researched, namely { A ective computing, AfC [ 17 ]. It integrates psychology, cognitive science, neurophysiology, and social studies, among others, in order to make a machine a decent participator of interaction with a human.

According to Picard, an a ective computer or system should be able to detect and record emotionally signi cant information in the environment, store the collected data, process it and generate an appropriate response. This data may come from various dimensions of interaction: from the physiological level (heart pulse, electrodermal activity, etc.), through user's behavior (facial expressions, body postures and gestures), towards more general metrics consisting of speci c sequences of operations, mouse cursor paths, and so on. Depending on the context, the gathered data can be forwarded to speci c algorithms and models. A notable feature of such solutions is an a ective loop [ 1 ] { a mechanism, where data detection, collection and processing engage the user and the system in a smooth, continuous cycle of dynamic behaviors and reactions. This brings numerous challenges that were out of concern in case of models processing the data in an o ine and asynchronous manner. Nevertheless, making the system behave like it understood the user's emotions greatly improves the general user experience [ 19 ].

As a domain clearly interdisciplinary in its nature, AfC needs a solid foundation in emotion theories. Currently, three main approaches to modelling emotion can be distinguished. Two of them form a dichotomy of \discreet vs continuous". With Paul Ekman's basic emotion's theory [ 6 ] as an example on one hand, the a ects are grouped into separate categories. On the contrary, according to dimensional a ect theories [ 22, 14 ] an emotion can be though of as a point in a twoor three-dimensional (depending on a speci c model) space. However, most interesting { at least from the AfC's point of view { approach seems to be re ected in so called appraisal-based theories [ 21 ], [ 16 ]. It allows to de ne an emotion as a set of features, ascribe them certain numeric values, and develop mathematical models for their interpretation and simulation. Actually, one of the last of the described group of emotion theories has already been implemented in AfC, speci cally in the eld of video games [ 18 ]. 2

A ective computing in video games

This paper focuses on how a ective approach, especially the a ective loop, can facilitate video game design. In creating AfC systems, two essential steps are considered: rstly, data collection (where various tools and methods for recording the player's a ective state are developed), and secondly { models, where the a ective loop can take its full shape. Video games, besides being a great hypothesis-testing polygon, when enhanced with an a ective loop are capable of raising the player's satisfaction and engagement, thereby increasing the game's recreational, educational, and therapeutic value. 2.1

Data collection in AfC video games

Similarly to other AfC sub elds, as noticed by [ 9 ], in video game research there are several levels of interaction where player's data can be gathered. On the most fundamental level, information about a ective state of the player is derived from her physiological signals, for example heart rate (HR) or galvanic skin response (GSR) [ 1 ]. A layer above comes the behavioral dimension of interaction with the game, which regards data of the player's facial expressions, body postures and gestures while playing, player character movement paths, etc. The level of stress experienced by the player can be also derived from the force used to press the button on the game controller [24]. Player's interpretation and evaluation of in-game events and objects forms yet another level where a ective data can be detected. Here, information of person's a ective state is derived based on models operating on numerically described features of game components and relations between them, such as the event's desirability, or its congruency with the game goal. The features and relations are de ned separately for the player character, and the characters in the game (Non-Player Characters, NPC). For example, the event \the treasure is stolen" for the NPC Guard is calculated as the congruency of this event for the NPC Guard's goal (\protect the treasure", value from 1 to 1) times the utility of this goal (where utility refers to how much the NPC does or does not want the goal to happen, value from 1 to 1). Similar approach may be applied to model the player's a ect. 2.2

A ect modelling in AfC games

Still following Hudlicka's suggestion [ 9 ], three perspectives on modelling the affect in games can be taken. First and foremost, one can focus on the player's emotional state, where her a ects have to be detected and interpreted as reliably and precisely as possible. With regard to di erent levels of data collection, emotion model may be based on physiological signals [ 3 ] and behavior patterns [23]. As has been already noted, the player character may interact with in-game characters, whose emotions may be modelled too. In order to make the NPCs more realistic, their a ect models have to include their goals and motivations. Finally, a ective loop may be tailored into the design of the game itself, on the level of game mechanics and aesthetics. From the very beginning of a ective gaming, the biometric signals of the player were used to in uence the game modes and di culty. Player's GSR and HR can be incorporated into the game engine to dynamically modify player character's speed [ 1 ], as well as interface features and monster spawning rates [ 5, 12 ]. On the other hand, there are whole architectures supporting the a ective game design [ 25, 11 ]. 3

A ective Game Design Patterns

My account as a researcher is situated somewhere in between modelling the player's a ect and a ective design framework. Coming from William James' theory of emotion [ 10 ] with Jesse Prinz's re nement [ 20 ], my team and I take a closer look on game design patterns [ 2 ]. We assume that a set of patterns for evoking emotional responses of the player can be distinguished, and coupled with patterns of physiological reactions of this player. Using non-intrusive sensory devices: Empatica E4 1 and Microsoft Band 2 wristbands2 2, as well as BITalino 3 and e-Health 4 platforms for biometric measurements, together with a simple platform game (designed speci cally for research purposes, with the a ective design patterns in mind, see Figure 1), we are now conducting experiments in order to test our hypothesis. The procedure now consists of three separate phases. Throughout the whole time of the experiment, the subject is wearing one or several of the aforementioned pieces of hardware and is seated in front of the laptop (see Figure 2). The wristbands are paired via Bluetooth with our custom smartphone app for data recording. In the beginning and between each of the phases, there is a 30-second period of inactivity, for acquiring a baseline for signals of each participant. In the rst part, the task is to evaluate subjectively felt arousal in reaction to the brie y presented stimuli from Nencki A ective Picture System [ 13 ]. This steps serves as a calibration phase, to address the problem of individual di erences and acquire data for developing personalized biosignals patterns. Next, the participant plays the platform game for several minutes. The a ective patterns present in the game design are, among others, the Time Limit and randomly spawning Enemies. This is where the timestamps of each game event related to speci c design pattern are recorder, to be included in further analysis in the light of possible correlations 1 https://www.empatica.com/research/e4/ 2 https://www.microsoft.com/en-us/band 3 http://bitalino.com/en/ 4 http://www.my-signals.com/ with physiological responses. In the last phase, the participant is looking at a relaxing picture, when suddenly, after roughly 50 seconds, an unpleasant sound is played { a terri ed scream of a woman. Here, readings of a strong a ective reaction to an unexpected stimuli are acquired. So far, several studies have already taken place. From each of the experiments conducted, readings of participant's HR and GSR responses from each of the three phases were recorded and stored in CSV les. Additionally, timestamps of each time a visual stimuli or a ective event occurred are kept in separate les, for each individual. In April 2017, only the calibration phase was conducted, with participation of 6 subjects, in Eurokreator Lab in Cracow. This rst attempt allowed to verify that pictures with higher arousal values are reacted to with stronger a ective responses, as indicated by increased participants' HR and GSR measurements. Tentative ndings of the rst experiment have been described and published in [ 15 ]. Second experiment in June 2017 included both calibration phase and gaming phase, with 9 participants { AGH UST students.

In order to test the reliability of our experimental setup and explore other possible directions of applicable hardware, experiments conducted in January 2018 and March 2018 were aided with e-Health and BITalino sensory platforms. Moreover, in January we also used Neurobit Optima equipment, a reputable system intended for neurofeedback and biofeedback training, as a reference device. That time, all three phases (including the last one, with the audio stimuli) were conveyed. 107 AGH UST students in total took part in the experiment. In brief, as a result of those two studies we have con rmed that e-Health and BITalino provide best HR and GSR measurements. However, in the future other forms of sensors need to be considered, as ngertip-attached electrodes signi cantly disrupt the player's comfort. This should not pose that much of a challenge, though, as for example BITalino sends the acquired readings using Bluetooth, so the sensors can be compressed into 3D printed wristband.

Close examination of data in search of speci c correlations are still in progress, but nevertheless some observations can be made. Our studies indicate that Empatica E4 wristband and BITalino platform may serve as reliable biometric measuring platforms for AfC purposes. Whether they can be successfully applied to games with a ective loop, so that our requirements for non-invasiveness and undisturbed gaming experience are satis ed, is still to be determined. We also con rmed that out custom application for recording data from the wristbands works impeccably, and is a useful asset for further studies. On the other hand, unfortunately we had to withdraw from further using the Microsoft Band 2 wristband due to its poor raw data quality. 4

Conclusion

Modern technologies are becoming increasingly pervasive in everyday life. Interaction between human and computer has rightfully received much attention within previous decades, but nevertheless there is still much to be done. A ectaware systems, though already studied and developed, are rather crude and are facing a lot of di culties. The need for new methodologies and frameworks for implementing AfC into existing and new applications is evident. One of the areas in need of such concern is video game development.

As for future objectives, we aim at creating a new game, this time with an a ective loop incorporated, and with respect to the conclusions from the analysis of our previous studies. We predict that the outcomes of our research will have a signi cant impact on the how the a ective dimension of games is designed. This will be bene cial both for video games as an entertainment industry, and as a medium for tutoring and therapy. Realistic emotional behavior of in-game characters together with accurate player a ect models will undoubtedly raise credibility of NPCs and make players more engaged in the experience, hence increasing educational value [ 8 ] and e ectiveness of various psychological treatments [ 7, 4 ]. 23. Shang, Z.: Continuous A ect Recognition with Di erent Features and Modeling Approaches in Evaluation-Potency-Activity Space. Master's thesis, University of Waterloo (2017) 24. Sykes, J., Brown, S.: A ective gaming: Measuring emotion through the gamepad.

In: CHI '03 Extended Abstracts on Human Factors in Computing Systems. pp. 732{733. CHI EA '03, ACM, New York, NY, USA (2003), http://doi.acm.org/ 10.1145/765891.765957 25. Szwoch, M.: Design elements of a ect aware video games. In: Proceedings of the Mulitimedia, Interaction, Design and Innnovation. p. 18. ACM (2015)

References

1. Bersak , D. , McDarby , G. , Augenblick , N. , McDarby , P. , McDonnell , D. , McDonald , B. , Karkun , R. : Intelligent biofeedback using an immersive competitive environment ( 2001 ), paper presented at the Designing Ubiquitous Computing Games Workshop at UbiComp 2001

2. Bjork, S. , Holopainen , J.: Patterns in Game Design . Charles River Media ( 2005 )

3. Camilleri , E. , Yannakakis , G.N. , Liapis , A. : Towards general models of player affect . In: A ective Computing and Intelligent Interaction (ACII) , 2017 International Conference on ( 2017 )

4. Chittaro , L. , Sioni , R.: A ective computing vs. a ective placebo: Study of a biofeedback-controlled game for relaxation training . International Journal of Human-Computer Studies 72 ( 8-9 ), 663 { 673 ( 2014 )

5. Dekker , A. , Champion , E.: Please biofeed the zombies: Enhancing the gameplay and display of a horror game using biofeedback . In: DiGRA Conference ( 2007 )

6. Ekman , P. : Facial expression and emotion. American psychologist 48(4) , 384 ( 1993 )

7. El Kaliouby , R. , Picard , R. , BARON-COHEN, S.: A ective computing and autism . Annals of the New York Academy of Sciences 1093 ( 1 ), 228 { 248 ( 2006 )

8. Gilleade , K.M. , Dix , A. : Using frustration in the design of adaptive videogames . In: Proceedings of the 2004 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology . pp. 228 { 232 . ACE '04, ACM , New York, NY, USA ( 2004 ), http://doi.acm. org/10 .1145/1067343.1067372

9. Hudlicka , E.: A ective computing for game design . In: Proceedings of the 4th International North American Conference on Intelligent Games and Simulation (GAMEON-NA) ( 2008 )

10. James , W.: What is an emotion? Mind (34) , 188 { 205 ( 1884 )

11. Kolakowska , A. , Landowska , A. , Szwoch , M. , Szwoch , W. , Wrobel , M.R. : Emotion recognition and its application in software engineering . In: Human System Interaction (HSI) , 2013 The 6th International Conference on. pp. 532 { 539 . IEEE ( 2013 )

12. Lobel , A. , Gotsis , M. , Reynolds , E. , Annetta , M. , Engels , R.C. , Granic , I. : Designing and utilizing biofeedback games for emotion regulation: The case of nevermind . In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems . pp. 1945 { 1951 . ACM ( 2016 )

13. Marchewka , A. , Z _urawski, L., Jednorog , K. , Grabowska , A. : The Nencki A ective Picture System (NAPS): Introduction to a novel, standardized, wide-range, high-quality, realistic picture database . Behavior Research Methods 46 ( 2 ), 596 { 610 ( 2014 )

14. Mehrabian , A. , Russell , J.A. : An approach to environmental psychology . MIT Press. ( 1974 )

15. Nalepa , G.J. , Gizycka , B. , Kutt , K. , Argasinski , J.K. : A ective design patterns in computer games. scrollrunner case study . In: Communication Papers of the 2017 Federated Conference on Computer Science and Information Systems , FedCSIS 2017 . pp. 345 { 352 ( 2017 ), https://doi.org/10.15439/2017F192

16. Orthony , A. , Clore , G. , Collins, A. : The cognitive structure of emotions . Cambridge University Press ( 1988 )

17. Picard , R.W.: A ective Computing . MIT Press ( 1997 )

18. Popescu , A. , Broekens , J., Van Someren, M. : Gamygdala: An emotion engine for games . IEEE Transactions on A ective Computing 5 ( 1 ), 32 { 44 ( 2014 )

19. Prendinger , H. , Dohi , H. , Wang , H. , Mayer , S. , Ishizuka , M. : Empathic embodied interfaces: Addressing users a ective state . In: Tutorial and Research Workshop on A ective Dialogue Systems . pp. 53 { 64 . Springer ( 2004 )

20. Prinz , J.J.: Gut Reactions . A Perceptual Theory of Emotion . Oxford University Press ( 2006 )

21. R., S.K. : Appraisal theory . In: T., D. , M. , P . (eds.) Handbook of cognition and emotion . pp. 637 { 663 . John Wiley and Sons ( 1999 )

22. Russell , J.A. : A circumplex model of a ect . Journal of Personality and Social Psychology 39 ( 6 ), 1161 { 1178 ( 1980 )