=Paper=
{{Paper
|id=Vol-2265/paper8
|storemode=property
|title=Using Affective Loop as Auxilliary Design Tool for Video Games
|pdfUrl=https://ceur-ws.org/Vol-2265/paper8.pdf
|volume=Vol-2265
|authors=Barbara Giżycka
}}
==Using Affective Loop as Auxilliary Design Tool for Video Games==
https://ceur-ws.org/Vol-2265/paper8.pdf
Using Affective Loop as Auxilliary Design Tool
for Video Games
Barbara Giżycka
AGH Univeristy of Science and Technology,
Al. Mickiewicza 30, 30-059 Krakow, Poland
bgizycka@agh.uj.edu.pl
Abstract. As modern technologies become more apparent and persis-
tent, human-computer interaction becomes an important research topic.
With birth of affective computing, which aims at developing systems ca-
pable of detecting and processing emotionally significant data from the
environment, new possibilities for applications unfold, and video games
can benefit from them as well. Bringing innovative solutions to this area
involves new modes of affective data collection and affect modelling of
various aspects of the game experience. My research, focusing on affective
game design patterns, is located on the intersection of modelling player
affect and affective game design framework. In this paper, an outline of
how affective computing ideas (especially affective loop) are introduced
to video game design is presented. A new approach to designing video
games in the form of affective game design patterns is proposed, together
with research method description and summary of studies conducted so
far.
Keywords: human-computer interaction, affective computing, affective
loop, video game design
1 Introduction
In the times of great scientific and technological progress, almost each and every
human activity is accompanied by modern technology. As computers and mobile
devices become more persistent and ubiquitous, people’s everyday contact with
them emerge as a broad topic for discussion, improvement, and – therefore –
research. With interfaces being a direct level of interaction between the user and
the machine, designers and developers face a challenge to make this interaction
as easy and natural as possible.
A movement taking a closer look on these problems, originating in early
1980s, is called Human-Computer Interaction (HCI). However, it was not until
late 1990s that Rosalind Picard from MIT Media Lab suggested that HCI should
take emotions in the interaction into consideration. She started a new approach
in researched, namely – Affective computing, AfC [17]. It integrates psychology,
cognitive science, neurophysiology, and social studies, among others, in order to
make a machine a decent participator of interaction with a human.
According to Picard, an affective computer or system should be able to de-
tect and record emotionally significant information in the environment, store the
collected data, process it and generate an appropriate response. This data may
come from various dimensions of interaction: from the physiological level (heart
pulse, electrodermal activity, etc.), through user’s behavior (facial expressions,
body postures and gestures), towards more general metrics consisting of spe-
cific sequences of operations, mouse cursor paths, and so on. Depending on the
context, the gathered data can be forwarded to specific algorithms and mod-
els. A notable feature of such solutions is an affective loop [1] – a mechanism,
where data detection, collection and processing engage the user and the system
in a smooth, continuous cycle of dynamic behaviors and reactions. This brings
numerous challenges that were out of concern in case of models processing the
data in an offline and asynchronous manner. Nevertheless, making the system
behave like it understood the user’s emotions greatly improves the general user
experience [19].
As a domain clearly interdisciplinary in its nature, AfC needs a solid founda-
tion in emotion theories. Currently, three main approaches to modelling emotion
can be distinguished. Two of them form a dichotomy of “discreet vs continuous”.
With Paul Ekman’s basic emotion’s theory [6] as an example on one hand, the
affects are grouped into separate categories. On the contrary, according to di-
mensional affect theories [22, 14] an emotion can be though of as a point in a two-
or three-dimensional (depending on a specific model) space. However, most in-
teresting – at least from the AfC’s point of view – approach seems to be reflected
in so called appraisal-based theories [21], [16]. It allows to define an emotion as
a set of features, ascribe them certain numeric values, and develop mathemat-
ical models for their interpretation and simulation. Actually, one of the last of
the described group of emotion theories has already been implemented in AfC,
specifically in the field of video games [18].
2 Affective computing in video games
This paper focuses on how affective approach, especially the affective loop, can
facilitate video game design. In creating AfC systems, two essential steps are
considered: firstly, data collection (where various tools and methods for record-
ing the player’s affective state are developed), and secondly – models, where
the affective loop can take its full shape. Video games, besides being a great
hypothesis-testing polygon, when enhanced with an affective loop are capable of
raising the player’s satisfaction and engagement, thereby increasing the game’s
recreational, educational, and therapeutic value.
2.1 Data collection in AfC video games
Similarly to other AfC subfields, as noticed by [9], in video game research there
are several levels of interaction where player’s data can be gathered. On the most
fundamental level, information about affective state of the player is derived from
her physiological signals, for example heart rate (HR) or galvanic skin response
(GSR) [1]. A layer above comes the behavioral dimension of interaction with
the game, which regards data of the player’s facial expressions, body postures
and gestures while playing, player character movement paths, etc. The level of
stress experienced by the player can be also derived from the force used to press
the button on the game controller [24]. Player’s interpretation and evaluation of
in-game events and objects forms yet another level where affective data can be
detected. Here, information of person’s affective state is derived based on models
operating on numerically described features of game components and relations
between them, such as the event’s desirability, or its congruency with the game
goal. The features and relations are defined separately for the player character,
and the characters in the game (Non-Player Characters, NPC). For example, the
event “the treasure is stolen” for the NPC Guard is calculated as the congruency
of this event for the NPC Guard’s goal (“protect the treasure”, value from −1
to 1) times the utility of this goal (where utility refers to how much the NPC
does or does not want the goal to happen, value from −1 to 1). Similar approach
may be applied to model the player’s affect.
2.2 Affect modelling in AfC games
Still following Hudlicka’s suggestion [9], three perspectives on modelling the af-
fect in games can be taken. First and foremost, one can focus on the player’s
emotional state, where her affects have to be detected and interpreted as reliably
and precisely as possible. With regard to different levels of data collection, emo-
tion model may be based on physiological signals [3] and behavior patterns [23].
As has been already noted, the player character may interact with in-game char-
acters, whose emotions may be modelled too. In order to make the NPCs more
realistic, their affect models have to include their goals and motivations. Finally,
affective loop may be tailored into the design of the game itself, on the level of
game mechanics and aesthetics. From the very beginning of affective gaming,
the biometric signals of the player were used to influence the game modes and
difficulty. Player’s GSR and HR can be incorporated into the game engine to
dynamically modify player character’s speed [1], as well as interface features and
monster spawning rates [5, 12]. On the other hand, there are whole architectures
supporting the affective game design [25, 11].
3 Affective Game Design Patterns
My account as a researcher is situated somewhere in between modelling the
player’s affect and affective design framework. Coming from William James’
theory of emotion [10] with Jesse Prinz’s refinement [20], my team and I take
a closer look on game design patterns [2]. We assume that a set of patterns for
evoking emotional responses of the player can be distinguished, and coupled with
patterns of physiological reactions of this player. Using non-intrusive sensory
devices: Empatica E4 1 and Microsoft Band 2 wristbands2 2 , as well as BITalino 3
and e-Health 4 platforms for biometric measurements, together with a simple
platform game (designed specifically for research purposes, with the affective
design patterns in mind, see Figure 1), we are now conducting experiments in
order to test our hypothesis.
Fig. 1. A screenshot from “London Bridge”, the game that was used in the discussed
studies (developed with Game Maker Studio).
3.1 Method
The procedure now consists of three separate phases. Throughout the whole time
of the experiment, the subject is wearing one or several of the aforementioned
pieces of hardware and is seated in front of the laptop (see Figure 2). The
wristbands are paired via Bluetooth with our custom smartphone app for data
recording. In the beginning and between each of the phases, there is a 30-second
period of inactivity, for acquiring a baseline for signals of each participant. In the
first part, the task is to evaluate subjectively felt arousal in reaction to the briefly
presented stimuli from Nencki Affective Picture System [13]. This steps serves as
a calibration phase, to address the problem of individual differences and acquire
data for developing personalized biosignals patterns. Next, the participant plays
the platform game for several minutes. The affective patterns present in the game
design are, among others, the Time Limit and randomly spawning Enemies. This
is where the timestamps of each game event related to specific design pattern are
recorder, to be included in further analysis in the light of possible correlations
1
https://www.empatica.com/research/e4/
2
https://www.microsoft.com/en-us/band
3
http://bitalino.com/en/
4
http://www.my-signals.com/
with physiological responses. In the last phase, the participant is looking at
a relaxing picture, when suddenly, after roughly 50 seconds, an unpleasant sound
is played – a terrified scream of a woman. Here, readings of a strong affective
reaction to an unexpected stimuli are acquired.
Fig. 2. Experimental setup, including of e-Health’s GSR electrodes and pulsioximeter
for HR readings, as well as Empatica E4 wristband. In the latest studies conducted,
either one of the e-Health of Bitalino platforms were used.
3.2 Results and Discussion
So far, several studies have already taken place. From each of the experiments
conducted, readings of participant’s HR and GSR responses from each of the
three phases were recorded and stored in CSV files. Additionally, timestamps of
each time a visual stimuli or affective event occurred are kept in separate files, for
each individual. In April 2017, only the calibration phase was conducted, with
participation of 6 subjects, in Eurokreator Lab in Cracow. This first attempt
allowed to verify that pictures with higher arousal values are reacted to with
stronger affective responses, as indicated by increased participants’ HR and GSR
measurements. Tentative findings of the first experiment have been described
and published in [15]. Second experiment in June 2017 included both calibration
phase and gaming phase, with 9 participants – AGH UST students.
In order to test the reliability of our experimental setup and explore other
possible directions of applicable hardware, experiments conducted in January
2018 and March 2018 were aided with e-Health and BITalino sensory platforms.
Moreover, in January we also used Neurobit Optima equipment, a reputable sys-
tem intended for neurofeedback and biofeedback training, as a reference device.
That time, all three phases (including the last one, with the audio stimuli) were
conveyed. 107 AGH UST students in total took part in the experiment. In brief,
as a result of those two studies we have confirmed that e-Health and BITalino
provide best HR and GSR measurements. However, in the future other forms
of sensors need to be considered, as fingertip-attached electrodes significantly
disrupt the player’s comfort. This should not pose that much of a challenge,
though, as for example BITalino sends the acquired readings using Bluetooth,
so the sensors can be compressed into 3D printed wristband.
Close examination of data in search of specific correlations are still in progress,
but nevertheless some observations can be made. Our studies indicate that Em-
patica E4 wristband and BITalino platform may serve as reliable biometric mea-
suring platforms for AfC purposes. Whether they can be successfully applied to
games with affective loop, so that our requirements for non-invasiveness and
undisturbed gaming experience are satisfied, is still to be determined. We also
confirmed that out custom application for recording data from the wristbands
works impeccably, and is a useful asset for further studies. On the other hand,
unfortunately we had to withdraw from further using the Microsoft Band 2
wristband due to its poor raw data quality.
4 Conclusion
Modern technologies are becoming increasingly pervasive in everyday life. In-
teraction between human and computer has rightfully received much attention
within previous decades, but nevertheless there is still much to be done. Affect-
aware systems, though already studied and developed, are rather crude and are
facing a lot of difficulties. The need for new methodologies and frameworks for
implementing AfC into existing and new applications is evident. One of the areas
in need of such concern is video game development.
As for future objectives, we aim at creating a new game, this time with an
affective loop incorporated, and with respect to the conclusions from the analysis
of our previous studies. We predict that the outcomes of our research will have
a significant impact on the how the affective dimension of games is designed.
This will be beneficial both for video games as an entertainment industry, and
as a medium for tutoring and therapy. Realistic emotional behavior of in-game
characters together with accurate player affect models will undoubtedly raise
credibility of NPCs and make players more engaged in the experience, hence
increasing educational value [8] and effectiveness of various psychological treat-
ments [7, 4].
References
1. Bersak, D., McDarby, G., Augenblick, N., McDarby, P., McDonnell, D., McDon-
ald, B., Karkun, R.: Intelligent biofeedback using an immersive competitive envi-
ronment (2001), paper presented at the Designing Ubiquitous Computing Games
Workshop at UbiComp 2001
2. Björk, S., Holopainen, J.: Patterns in Game Design. Charles River Media (2005)
3. Camilleri, E., Yannakakis, G.N., Liapis, A.: Towards general models of player af-
fect. In: Affective Computing and Intelligent Interaction (ACII), 2017 International
Conference on (2017)
4. Chittaro, L., Sioni, R.: Affective computing vs. affective placebo: Study of
a biofeedback-controlled game for relaxation training. International Journal of
Human-Computer Studies 72(8-9), 663–673 (2014)
5. Dekker, A., Champion, E.: Please biofeed the zombies: Enhancing the gameplay
and display of a horror game using biofeedback. In: DiGRA Conference (2007)
6. Ekman, P.: Facial expression and emotion. American psychologist 48(4), 384 (1993)
7. El Kaliouby, R., Picard, R., BARON-COHEN, S.: Affective computing and autism.
Annals of the New York Academy of Sciences 1093(1), 228–248 (2006)
8. Gilleade, K.M., Dix, A.: Using frustration in the design of adaptive videogames.
In: Proceedings of the 2004 ACM SIGCHI International Conference on Advances
in Computer Entertainment Technology. pp. 228–232. ACE ’04, ACM, New York,
NY, USA (2004), http://doi.acm.org/10.1145/1067343.1067372
9. Hudlicka, E.: Affective computing for game design. In: Proceedings of the 4th
International North American Conference on Intelligent Games and Simulation
(GAMEON-NA) (2008)
10. James, W.: What is an emotion? Mind (34), 188–205 (1884)
11. Kolakowska, A., Landowska, A., Szwoch, M., Szwoch, W., Wróbel, M.R.: Emo-
tion recognition and its application in software engineering. In: Human System
Interaction (HSI), 2013 The 6th International Conference on. pp. 532–539. IEEE
(2013)
12. Lobel, A., Gotsis, M., Reynolds, E., Annetta, M., Engels, R.C., Granic, I.: Design-
ing and utilizing biofeedback games for emotion regulation: The case of nevermind.
In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors
in Computing Systems. pp. 1945–1951. ACM (2016)
13. Marchewka, A., Żurawski, L., Jednoróg, K., Grabowska, A.: The Nencki Affec-
tive Picture System (NAPS): Introduction to a novel, standardized, wide-range,
high-quality, realistic picture database. Behavior Research Methods 46(2), 596–610
(2014)
14. Mehrabian, A., Russell, J.A.: An approach to environmental psychology. MIT
Press. (1974)
15. Nalepa, G.J., Gizycka, B., Kutt, K., Argasinski, J.K.: Affective design patterns in
computer games. scrollrunner case study. In: Communication Papers of the 2017
Federated Conference on Computer Science and Information Systems, FedCSIS
2017. pp. 345–352 (2017), https://doi.org/10.15439/2017F192
16. Orthony, A., Clore, G., Collins, A.: The cognitive structure of emotions. Cambridge
University Press (1988)
17. Picard, R.W.: Affective Computing. MIT Press (1997)
18. Popescu, A., Broekens, J., Van Someren, M.: Gamygdala: An emotion engine for
games. IEEE Transactions on Affective Computing 5(1), 32–44 (2014)
19. Prendinger, H., Dohi, H., Wang, H., Mayer, S., Ishizuka, M.: Empathic embodied
interfaces: Addressing users affective state. In: Tutorial and Research Workshop
on Affective Dialogue Systems. pp. 53–64. Springer (2004)
20. Prinz, J.J.: Gut Reactions. A Perceptual Theory of Emotion. Oxford University
Press (2006)
21. R., S.K.: Appraisal theory. In: T., D., M., P. (eds.) Handbook of cognition and
emotion. pp. 637–663. John Wiley and Sons (1999)
22. Russell, J.A.: A circumplex model of affect. Journal of Personality and Social
Psychology 39(6), 1161–1178 (1980)
23. Shang, Z.: Continuous Affect Recognition with Different Features and Modeling
Approaches in Evaluation-Potency-Activity Space. Master’s thesis, University of
Waterloo (2017)
24. Sykes, J., Brown, S.: Affective gaming: Measuring emotion through the gamepad.
In: CHI ’03 Extended Abstracts on Human Factors in Computing Systems. pp.
732–733. CHI EA ’03, ACM, New York, NY, USA (2003), http://doi.acm.org/
10.1145/765891.765957
25. Szwoch, M.: Design elements of affect aware video games. In: Proceedings of the
Mulitimedia, Interaction, Design and Innnovation. p. 18. ACM (2015)