A BCI-based Assessment of a Player's State of Mind for Game Adaptation

Abstract

Playing videogames is a process driven by both cognitive and emotional factors. Then, developing a mechanism that takes into account players' emotional state for adapting specific game features is a way to increase their engagement and flow during gameplay. In this paper, we present how a passive Brain Computer Interface (BCI) can be used to assess the state of mind of a player that can be used for enhancing his experience through adaptation. In particular, we collected data from EEG signals, in a horror adventure game, to learn a model of flow by monitoring the level of boredom, flow, and stress of the player. To this aim, we set an experiment and collected both subjective data about the perceived emotions and state of flow and data from the BCI that have been used to learn a classifier to recognize and assess the player's affective state. Results are encouraging and the learned model achieves a good accuracy in distinguishing the three player's states.

CCS CONCEPTS

• Human-centered computing → Human computer interaction (HCI); HCI design and evaluation methods; User models; Empirical studies in HCI .

1 INTRODUCTION

Playing videogames is a process driven by both cognitive and emotional factors [ 18 ]. The game should keep the player engaged trying to avoid boredom or a too high level of anxiety or frustration. For this reason the new generation of video games tries to assess the user’s state of mind to adapt, for instance, the dificulty during gameplay [ 10 ].

Emotion-based games are reported to improve a player’s engagement, immersion, excitement, and challenge by dynamically adapting specific game features according to the recognized emotion. At the basis of the adaptation process there is the recognition of the user’s afective state. This process is usually based on two main approaches: i) the analysis of human behavioral signals such as facial expression, gesture, posture, etc., which has the advantage of being easy to acquire and based on solid theories and models; ii) the use of physiological signals (i.e. electroencephalogram (EEG), galvanic skin response (GSR), respiration (RSP), etc.) that are continuous and allows to record changes according to the specific stimulus or situations that people have to face [ 13 ].

In this paper we focus on the description of a study aiming at learning a model for classifying the player’s state into boredom, flow and stress during gameplay from EEG signals trough a BCI. In a general view, a BCI is a direct communication pathway between the brain and an external device [ 16 ]. By means of electroencephalogram, it records human brain activity, through multiple electrodes that are placed on the scalp. In particular, passive BCI are a key approach when dealing with the measurement of emotions. In our study, we focus on a particular type of videogame, an horror adventure, in which the player should not be too relaxed and bored, in this case the level of dificulty should increase, and not too stressed, in this case the level of dificulty should decrease. For eficiently recognize and classify the user’s state from the EEG signals during gameplay we use a commercial BCI, Emotiv EPOC+, and its proprietary API for detecting engagement, stress, interest, focus, excitement, relaxation. To this purpose we performed an experiment to collect data for each of the emotion state we aim to recognize: boredom, flow and stress. In particular, 35 players were involved in the experiment and, in total, we collected 240 recordings of player EEG signals. These physiological data were interpreted and annotated by means of psychological measures collected through https://www.emotiv.com/developer/ a questionnaire to which the players had to answer. Results showed that emotional states extracted by BCI are mainly coherent with both self-evaluated measures since they had the same trend across experimental conditions. In this sense, from results emerged how the more engaging experimental conditions (respectively flow and stress) increased the level of engagement of the participants when it is compared with the ’boredom’ one. Nevertheless being an horror videogame from our results emerged also how higher levels of detected engagement can represent also a first signal of cognitive mastery lost, mainly in women. The collected dataset has, then, been used to learn a model of the player in terms of the above mentioned emotional states. The model’s accuracy is 78% on average on the three classes.

In our future work we plan to perform more experiments in order to gather more data and measure the efect of the game adaptation to the learned model. 2

RELATED WORK

The recognition of the player afective state is usually based on modalities such as video analysis [ 19 ] and physiological measures[ 17 ].

Physiological measures such as Electroencephalogram (EEG) [ 15 ], blood pressure, heart rate, or galvanic skin response have been used widely to measure and monitoring changes in players’ mental state [ 8 ] especially in terms of engagement and quality of the experience [ 3 ]. Examples of the use of physiological information, during the video games fruition, to recognize emotions are [ 2, 12, 20, 21 ]. Granato et al. [ 12 ] propose the use of combination of signals coming from diferent sensors to assess the player’s emotional state in terms of valence and arousal dimensions. Also Tognetti et al. [ 20 ] propose the use of a combination of physiological signals to recognize a user’s enjoyment in a car racing game. The recognition of the player’s afective states and the consequent adaptation of a survival horror games using the analysis of the brainwave and heart rate signals has been investigated by Vachiratamporn et al. [ 21 ]. In particular they investigated on the transition from a pre-fear afect to a post-fear one. Also Chanel et al. [ 2 ], proposed a system where, to keep players engaged in the game, they modulate the dificulty of a game according to felt emotions .

As far as building model of the player’s emotions there are basically two approaches: i) model-based, in which the model of emotion is usually built on a theoretical framework that refers emotion theories , in which signals are mapped directly to specific emotional states. The second approach is model-free (bottom-up), in which the construction of the model comes from the mapping (model) between (player) input and an emotional state representation. In this case, player data and annotated afective states are collected and used to learn the model ([ 22 ] among others).

Models used to enhance the gameplay experience are often related to the notion of flow [ 5 ]. According to the Flow scholars it is a state of elevated concentration, interest, and enjoyment that can be predicted by how much a task is challenging and also under users’ sense of control and selfeficacy. Flow is described as a mental state in which a person is neither anxious nor bored, being in the flow state during game engages the player and keep him in the game [ 4 ]. Flow is related to a persons skill level and the game challenge level. [ 7 ] use a model based on the theory of flow to evaluate a players experience during gameplay.

In our approach we use physiological data coming from the proessing of EEG signals. These are automatically recognized by the BCI and used to monitor the mental state of the player and adapt the gameplay accordingly. For more extensive BCI game reviews, refer to [ 14 ]. 3

THE GAME

The game is a first-person horror adventure very similar to the Slender: The Eight Pages . At the beginning of the game, the player watches at an introduction telling the story of Jack, a very arrogant businessman that cares only for money and power, who, after a fight with his wife, has a violent car-accident (Figure 1a). After recovering from the shock, Jack exits the car and a supernatural voice whispers to him: "You will find yourself in this deserted, lifeless city. Your goal will be to find the seven pages that will rebuild your life. But watch out for the pitfalls that hide in the fog, those are the ghosts of your past." Then, the game begins.

The game set is partially urban and partially constituted by a park (Figure 1b). The player’s objective is to collect all seven pages located in various areas of the game set (Figure 1c) while avoiding the ghosts of the past (Figure 1d), the Non Playing Characters (NPCs). Each page shows a memory of past life and the pain caused to the family. As the player collects pages, the fog in the game set grows thicker, and ghosts appear closer to the player.

A further characterizing game mechanic is given by the mental health of the protagonist, a "resource" represented by a bar that will decrease after significant events such as the encountering of the NPC or the simple passing of time . Collecting pages, however, will make possible to restore a small amount of health, delaying the defeat. Sounds are essential to create a suggestive atmosphere of suspense, they are also used as feedback in correspondence with particular actions, to signal the presence of a page in the surroundings or the approaching of the NPC.

We reproduced a similar game since the final goal of the project is to test afective-based adaptation of the game dificulty and, then, there was the need to modify the game dynamics at runtime The player can move freely in the game set by opening new game areas by collecting the pages and, during the adventure, will be hunted by three NPCs. These will try in every way to prevent the player from collecting the pages. As the latter are collected the range of hunting of the entities will increase and there will be a greater number of antagonist appearances. When these entities appear the mental health of the protagonist will gradually decrease. When the player is reached by the NPC he should move away (running) from the enemy to prevent the falling of mental health to critical levels. There is also a grace period in the beginning of the game, during which the NPC remains inactive until the player collects the first page. However, the dificulty level increases the longer one goes without collecting any pages. A game over occurs when either the player has taken too long to find a page, the player stares at or comes in contact with the hostile entity for too long. 4

THE EXPERIMENT

The Approach

The aim of this study is to assess a player’s state of mind from EEG signals for developing the a model of flow in survival horror games. We collected two diferent kinds of data. Data regarding the understanding of game mechanics, the engagement level, balance between challenge and abilities and intrinsic pleasure and so on that are derived from the questionnaire answers. Data coming from the Emotiv BCI in terms of afective performance metrics. With the prospect of merging BCI and afective research on the gaming field, we collected a dataset of EEG signals under three diferent gameplay conditions: an inherently boring gameplay, an in-flow gameplay and a stressful gameplay.Since Fisher [ 9 ] defines the boredom like an unpleasant afective state with lack of concentration and dificulty during the execution of a task, and Csikszentmihalyi [ 6 ] furthermore denotes it like a state in which player’s skills are greater than required, the boredom game level must be characterized by linearity and repetitiveness with poor challenge, and weak visual assets.

Considering the previously described game, we developed its Boredom Level version, in which the challenge level is low so as to induce in the player a state of boredom. The fog, the scaring sounds and the NPCs have been eliminated. The only possible interaction with the game consists in collecting pages. Moreover, the life does not decrease. Then, each player has the possibility to complete the game. In the In-flow Level of the game, the challenge is balanced to the player abilities; it corresponds to the game as described in Section 3. Then, in the Stress Level the challenge is higher than average players abilities, thus inducing them in a state of stress and frustration. The fog density has been increased and the pages are more dificult to be found. Environment sounds are higher while feedback sounds are lower. The NPCs are five instead of three, and the player life decreases quickly so as to induce more anxiety. Running has been deactivated, so that the player should feel frustrated by the impossibility to escape from NPCs encountering. This level is almost impossible to complete the first time.

A formative test has been performed in order to assess the appropriateness of the preliminary design of the three levels. Six users, 3 female and 3 males, aged between 21 and 35 y.o., all of them experienced players, participated in the test. Semi-structured interviews and explicit questions about the felt emotion helped us in identifying salient features. The three levels resulted appropriately designed to achieved the above mentioned objectives.

The Experimental Setup

Technologies. In this study we used the Emotiv EPOC+ headset, a wireless neuro-signal acquisition device with 14 wet sensors (+2 reference), capable of detecting brainwaves at 128Hz sequential sampling rate. Emotiv provides a set of API that can be used to recognize diferent emotional metrics: • Stress (FRU): is a measure of comfort with the current challenge. • Engagement (ENG): it measures the level of immersion in the task and contrasts with boredom. • Interest (INT): is the degree of attraction or aversion to the current stimuli or activity and is commonly referred to as valence. • Excitement (EXC): is an awareness or feeling of physiological arousal with a positive value. • Focus (FOC): is a measure of fixed attention to a specific task. • Relaxation (MED): is a measure of an ability to switch of and recover from intense concentration.

The Emotiv measurements range from 0 to 1.

Participants. Data were collected from 35 healthy participants (23 males and 12 females), aged between 17 and 42 y.o. (mean: 24.28, std.dev: 6.06). All participants had normal or corrected-to-normal vision and described themselves as daily videogame players with diferent ability levels. Nobody had experience with EEG or BCIs. They participated in the experiment voluntarily.

Procedure. The experiment held in this study consisted of single sessions (see Figure 2). Each experimental session was divided into two phases: a tutorial phase in which the user has to familiarize with the game purpose and commands and the game trial. Each subject played with one of the three game levels described above, without knowing which one.

The experiment held in this study was performed in one of the research lab of the Computer Science Department of the University of Bari. To avoid source of distraction, one user at time experimented the videogame. Before the experiment, each participant has been asked to sign an informed consent and, subsequently, to answer a preliminary questionnaire in order to set an initial profile of the user (pre-test). The participants were then seated in a comfortable chair in front of a computer with a Full HD 24 inch screen, a set of speakers, a mouse and a keyboard. While playing, the lights were turned to a softer brightness to help the immersion into the task and reduce glare. Then, the EEG headset was positioned on the participant’s head. The examiner verified impedance in connections between each electrode and the participant’s scalp. Before starting the experiment, we asked the participant to relax for 10 seconds (baseline). Then they started to play. Data were recorded between events in the gameplay: E1) Page gathering, E2) Encountering of the NPC and E3) Gameover (Death of the player or end of the game). In total we collected 240 examples in total for the 3 classes. At the end of the session, participants were asked to answer to a final questionnaire (post-test) aiming at collecting data about their experience while playing (see Figure 2)). As far as emotions are concerned, a SAM [ 1 ] was used as to measure emotional responses in three dimensions (i.e. valence, arousal and dominance). It was expected that if players became more stressed while playing, this would result in higher arousal and lower valence scores, while a low arousal and valence would denote boredom. The dominance would also be higher if players had the feeling that they were in control of the game. Moreover, we asked the user to explicitly state which was the prevalent felt emotion during the gameplay. In addition questions aiming at assessing the perceived sense of flow are included and in particular we asked to express the perceived relation between the perceived challenge level and skills. These correlated to other psychological cues (understanding of game mechanics, engagement level, balance between challenge and capacity and intrinsic pleasure). Results. Collected data were analysed and used to learn a model of the player afective state. First of all considering answers on self-evaluated emotional state, it is strongly coherent with the experimental conditions since our participants felt themselves bored in boredom condition, stressed in the stress condition and medium level of fear across conditions with a slightly increasing in flow and stress condition [F(2,32)= 4,54; p<0.010] (Figure 3a). With respect to physiological signals a repeated measures ANOVA [F(5,32)= 56,09; p<0.00] showed that engagement with interest are higher across conditions, mainly in the stress condition, followed by the flow one. Also stress is high mainly in the stress condition, by presenting a stronger diference with flow condition, suggesting that a good predictor of the flow state is the level of engagement minus the stress. Lower levels of relax and excitement were showed across conditions (Figure 3b). Focusing on the detected engagement, it is significantly higher in stress and flow condition and [F(2,32)= 4,03; p<0.025] compared with the boredom condition. Also the interest [F(5,32)= 3,95; p<0.032] followed the same trend with a lower levels compared with the engagement, showing how these two states can be related during the game and how interest can be the first step for the full engagement of the user (Figure 4a).

When we consider the gender diferences we can report a significant interaction efect between experimental conditions and the user’s gender [F(2,32)= 3,32; p<0.05] in the sense that women tend to feeling more engagement in the stress condition and less in the boredom one compared to men that basically present a flat engagement across conditions (Figure 4b. Engagement from BCI*Gender). These preliminary results can be related to the fact that women paid more attention to the game condition being, also following recent studies [ 11 ], less used to play with this technological devises compared to men. Women seem more context sensitive to the game settings. The level of the engagement is inversely correlated to cognitive aspects during the game, the higher engagement the less comprehension of game procedure (r=.41; p<0.018), the less comprehension of mode to achieving goals (r=-.30; p<0.05) and the perception of actions’ efects. (r=-.31; p<0.05). Thus, in this sense higher levels of engagement can compromise the game understanding. In line with this results, also the ’dominance’ was significantly correlated with the Interest (r=.48; p<0.005) and not with the Engagement (p=n.s), meaning that the higher levels of Engagement gave to the users a perception of lost mastery’ during the game. In this case we must consider that the engagement was measured during an horror game, thus these results in future studies should be compared with a less arousing game (Figure 5). To implement a process able to use EEG signals as implicit feedback concerning the state of flow we used the data collected during the experiment to learn a classification model. To this aim we used the WEKA platform. For each Performance Metrics (PM) the considered features are:average, std.dev, Pearson correlation coeficient of each PM, minimum value, maximum value,variance Since we did not used raw data but values of the performance metrics we applied the several algorithms and, in particular, Random Forest was the one having the best accuracy on the dataset. The three classes of interest were Boredom, Flow and Stress. Results, calculated using leave-one-out cross-validation, show an average accuracy on three classes of 0.78. Analyzing the classification results in more details by looking at the classification results, we noticed that the majority of instances of the Flow class were misclassified and confused with stress. This result is plausible since in a horror game the player should be in a slightly anxious state. 5

CONCLUSIONS AND FUTURE WORK

In this paper we presented an experiment aiming at collecting data from a passive BCI for learning a model of flow in horror adventure games. To do so we designed and developed three levels of the game: boredom, in-flow, and stress conditions. 35 players were involved in the experiment and, in total, 240 recordings of player EEG signals were collected. At the end of the game the players had to answer to a questionnaire for assessing their perceived emotions, engagement, and how much they felt that their level of skill was appropriate to the game challenge. Results showed that emotional states extracted by BCI are mainly coherent with both self-evaluated measures since they had the same trend across experimental conditions. Moreover, they showed that emotional states extracted by BCI can be considered as good predictors of this kind of videogames (horror) and in particular the level of engagement that was mainly coherent with experimental conditions (boredom, flow and stress). The self-evaluated measures, both cognitive and emotional, helped us to understand in a finer-grained way when engagement was close to a lost of cognitive mastery. This was true mainly for women that were more afected from stress condition, thus future studies could use a less arousing game.

The learned model of flow classifies the three states with an average accuracy of 78%. These results are obtained on the dataset, therefore it is necessary to make new experiments in order to both test the classifier in real-time and collect new examples. Moreover, we plan to investigate on the eficacy of recognition of the player’s afective state with computer vision techniques in the videogames context.

ACKNOWLEDGMENTS

Authors thank Fabio Abbattista for providing the game code and the participants to the study.

References

[1] Teah-Marie Bynion and Matthew Feldner . 2017 . Self-Assessment Manikin. 1-3 . https://doi.org/10.1007/978-3- 319 -28099-8_ 77 - 1

[2] Guillaume Chanel , Cyril Rebetez, Mireille BÃľtrancourt, and Thierry Pun . 2008 . Boredom, Engagement and Anxiety as Indicators for Adaptation to Dificulty in Games . 13 - 17 . https://doi.org/10.1145/1457199. 1457203

[3] Maher Chaouachi , Chalfoun Pierre, Imène Jraidi, and Claude Frasson . 2010 . Afect and mental engagement: Towards adaptability for intelligent . In Twenty-Third International FLAIRS Conference.

[4] Benjamin Cowley , Darryl Charles, Michaela Black, and Ray J. Hickey . 2008 . Toward an understanding of flow in video games . Comput. Entertain . 6 ( 07 2008 ), 1 - 27 . https://doi.org/10.1145/1371216.1371223

[5] Mihaly Csikszentmihalyi . [n. d.]. Flow and the psychology of discovery and invention . ([n. d.]).

[6] Mihaly Csikszentmihalyi and Isabella Csikszentmihalyi . 1975 . Beyond boredom and anxiety . Vol. 721 . Jossey-Bass San Francisco.

[7] S. Z. Diva , R. A. Prorna , I. I. Rahman , A. B. Islam , and M. N. Islam . 2019 . Applying Brain-Computer Interface Technology for Evaluation of User Experience in Playing Games . In 2019 International Conference on Electrical, Computer and Communication Engineering (ECCE). 1-6 . https://doi.org/10.1109/ECACE. 2019 .8679203

[8] Stephen H Fairclough and Louise Venables . 2006 . Prediction of subjective states from psychophysiology: A multivariate approach . Biological psychology 71 , 1 ( 2006 ), 100 - 110 .

[9] Cynthia Fisher . 1993 . Boredom at Work: A Neglected Concept . School of Business Discussion Papers 46 (03 1993 ). https://doi.org/10.1177/ 001872679304600305

[10] Kiel Gilleade , Alan Dix, and Jen Allanson . 2005 . Afective videogames and modes of afective gaming: assist me, challenge me, emote me . DiGRA 2005 : Changing Views-Worlds in Play . ( 2005 ).

[11] Timo Gnambs , Lukasz Stasielowicz, Ilka Wolter, and Markus Appel . 2018 . Do computer games jeopardize educational outcomes? A prospective study on gaming times and academic achievement . (06 2018 ). https://doi.org/10.31234/osf.io/2s8rv

[12] Marco Granato , Davide Gadia, Dario Maggiorini, and Laura Ripamonti . 2017 . Emotions Detection Through the Analysis of Physiological Information During Video Games Fruition . 197 - 207 . https://doi.org/ 10.1007/978-3- 319 -71940-5_ 18

[13] J Matias Kivikangas , Guillaume Chanel, Ben Cowley, Inger Ekman, Mikko Salminen, Simo Järvelä, and Niklas Ravaja . 2011 . A review of the use of psychophysiological methods in game research . journal of gaming & virtual worlds 3 , 3 ( 2011 ), 181 - 199 .

[14] David Marshall , Damien Coyle , Shane Wilson, and Michael Callaghan . 2013 . Games, gameplay, and BCI: the state of the art . IEEE Transactions on Computational Intelligence and AI in Games 5 , 2 ( 2013 ), 82 - 99 .

[15] Anton Nijholt . 2009 . BCI for Games: A 'State of the Art' Survey . In Entertainment Computing - ICEC 2008 (Lecture Notes in Computer Science) , Scott M. Stevens and Shirley J. Saldamarco (Eds.) . Springer, 225 - 228 . https://doi.org/10.1007/978-3- 540 -89222-9_ 29

[16] Jerry J Shih , Dean J Krusienski, and Jonathan R Wolpaw . 2012 . Braincomputer interfaces in medicine . In Mayo Clinic Proceedings , Vol. 87 . Elsevier , 268 - 279 .

[17] Lin Shu , Jinyan Xie , Mingyue Yang, Ziyi Li , Zhenqi Li , Dan Liao , Xiangmin Xu, and Xinyi Yang . 2018 . A Review of Emotion Recognition Using Physiological Signals . Sensors 18 , 7 ( 2018 ). https: //doi.org/10.3390/s18072074

[18] Eduard Sioe-Hao Tan . 2008 . Entertainment is emotion: The functional architecture of the entertainment experience . Media psychology 11 , 1 ( 2008 ), 28 - 51 .

[19] PaweÅĆ Tarnowski , Marcin KoÅĆodziej, Andrzej Majkowski, and Remigiusz Rak . 2017 . Emotion recognition using facial expressions . Procedia Computer Science 108 ( 12 2017 ), 1175 - 1184 . https://doi.org/ 10.1016/j.procs. 2017 . 05 .025

[20] Simone Tognetti , Maurizio Garbarino, Andrea Tommaso Bonanno, Matteo Matteucci, and Andrea Bonarini . 2010 . Enjoyment recognition from physiological data in a car racing game . (01 2010 ). https://doi. org/10.1145/1877826.1877830

[21] Vanus Vachiratamporn , Koichi Moriyama, Ken-ichi Fukui , and Masayuki Numao . 2014 . An implementation of afective adaptation in survival horror games . IEEE Conference on Computatonal Intelligence and Games , CIG, 1 - 8 . https://doi.org/10.1109/CIG. 2014 .6932893

[22] Georgios N Yannakakis , Héctor P Martínez, and Arnav Jhala . 2010 . Towards afective camera control in games . User Modeling and UserAdapted Interaction 20 , 4 ( 2010 ), 313 - 340 .