EmoBrain: Playing with Emotions in the Target

Abstract

Sensing and understanding human emotional behaviour seems to be essential to keep engaging interactions sustained over longer periods. The general purpose of this work is to implement a platform able to recognize and employ human emotions in an interactive game: The EmoBrain Interface (EI). EI interaction is a cycle of stimulus and feedbacks where the user receives a visual input and completes tasks, just controlling his emotional state by activating self-training strategies. EI has been used in the context of a serious game: Quiet Bowman (QB). QB allows the users: (i) to experience emotional behaviours and (ii) to explore the game dynamics related to emotions in order to manage their own emotional state. The outcomes seem to be encouraging. Users appreciated the game: they felt involved and committed to achieve the required goals. This encourages us to make new experiments to improve the accuracy of our classifiers and therefore the impact of the serious game on the autogenous training of the user.

-

INTRODUCTION Emotions are part of our everyday living and influence many human processes such as cognition, perception, and everyday tasks such as learning, communication and GHItaly18: 2nd Workshop on Games-Human Interaction, May 29th, 2018, Castiglione della Pescaia, Grosseto (Italy) Copyright © 2018 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. decision-making [ 1 ]. Being aware of self emotional state and start working on its regulation is important to achieve a better wellness level, since emotion regulation can help to mitigate emotion related biases in our everyday tasks. Recently serious games have been used as a mean to learn how to control and regulate emotions. Many serious games developed in this field use biofeedback information to display player’s emotional state and help them to train in order to improve his own wellness.

Our research work is placed in this context. We carried on previous research [ 2,3,4 ] and developed a serious game, called “Quiet Bowman” (QB). QB allows the users: (i) to experience emotional behaviors; and (ii) to explore the game dynamics related to emotions to manage their own emotional state. This is related to the implementation of a platform able to recognize and use user’s emotions as input to the interactive game: The EmoBrain Interface (EI). The EI implements a cycle of stimulus and feedback where the user receives a visual output and completes tasks, just controlling his emotional state, by activating autogenous training strategies.

Although there are many dimensions associated with emotions, according to Picard [ 5 ] the two most commonly used dimensions of emotion are valence, and arousal. Picard also notes that the valence and arousal dimensions are critical in games applications [ 5 ]. The recognition of user’s emotional state has to be performed in an implicit and transparent way, so as to be non-invasive and more effective. Moreover, traditional methods, such as self-report or interviews, are only partially useful (and concern the preliminary research activities only), because they are based on sampling techniques or simply on the a-posteriori user’s perception of the game environment.

We decided to couple the Emotional Brain Computer Interfaces (EBCI) [ 6,7 ] to traditional methods.

An EBCI is a particular kind of a Brain Computer Interface (BCI). A BCI [ 8 ] is a direct communication pathway between the brain and an external device. By means of an electroencephalogram (EEG), it records human brain activity in the form of electrical potentials (EPs), through multiple electrodes that are placed on the scalp. EPs are processed to obtain features that can be grouped into a feature vector: Depending on the brain activity, distinctive known patterns in the EEG appear. These are automatically recognized by the BCI and associated with a given action on the BCI application. The outcome of this action can be perceived by the user in terms of application feedback. In this case, his brain activity is consequently modulated. The kind of EBCI mostly used in this work is the reactive EBCI [ 9 ]. A reactive EBCI can send stimuli and extract information from user’s brain elaboration. So, if during a session game, the player has been scared or disgusted, the EI will recognize a medium-high value for arousal with a strongly negative valence. Otherwise, will be recognised positive valence and high arousal, for happiness. In the scope of our work, the feedback about the progress of the game is a key tool: it allows the players to monitor their performance, but it also generates new stimuli caused by the achievement of the goal (feedback loop).

The paper is organized as follows: in Section 2 we provide a description of the architecture of the system. In Section 3 we illustrate the experimental study we conducted to train and test our system. In Section 4 the results of the study are discussed. Conclusions and future work directions are reported in Section 5.

OVERVIEW OF THE ARCHITECTURE The EI is a distributed platform to recognize and employ human emotions to drive an interactive system (Fig. 1). The EI includes an input device to record user’s EEG. The brain activity (EEG signal) is then transmitted to the BCI that analyses and processes it for to recognize user’s emotion following a visual stimulation. The EI also includes QB, a serious game for autogenic training, in which the user is a player that shoots arrows in order to hit the target (emotional goal) and receive feedbacks, according to his emotional state.

At the beginning of the interaction, in order to alter his emotional state, an exogenous stimulus (i.e. photos or other multimedia content) is send to the user. The EI detects such emotional alterations and sends feedback messages to the user. The feedback received from the user will, in turn, be used by him to initiate an appropriate endogenous stimulation strategy, to achieve the emotional goal. User Application: Quiet Bowman (QB) In order to let EI platform handle user emotions in the scope of an interactive game, we implemented QB. As stated, QB is a simplified version of a serious game for autogenous training. It is an archery game in which the player plays with the goal of achieving an emotional state represented in the centre of the target (emotional goal). Four emotions could be the emotional goal: Calm, Happiness, Anger or Sadness. We chose these emotions because their components (valence and arousal) better match the emotional chose values: high, medium and low for arousal; positive, neutral and negative for valence. Indeed, calm, for example, has neutral values for valence and slightly low for arousal. On contrary, anger has high arousal and a low valence. In addition, these four emotions are commonly and clearly identified by users.

The only controller in the game is user’s affective state: at each step, the user emotion is recognized by the EI and represented as an arrow on the target. Each step is automatically triggered by a timer. To win the game, the player must be able to drive his emotional state in order to achieve the emotional goal, by activating any endogenous stimulation strategy.

A screenshot of the game can be seen in Fig. 2. Seven shots were fired. The seventh (t7) hit the target. The messages exchange among EI, QB and the user takes the name of feedback loop. So, for example, let the user play whit the emotion of Calm. At the beginning of the interaction, in order to alter his emotional state, a picture is presented to the user. The EI detects such emotional alterations and sends a feedback messages to the user (an arrow in the target: t1, in Fig.2).

The feedbacks received from the user will, in turn, be used by him to initiate an appropriate endogenous stimulation strategy, to reach a state of quiet. In doing so, however, he will send new data to the EI and a new arrow will be fired from the QB. The display of the latter arrow (or the sound, if set, for this event) on the target constitutes a new stimulus (exogenous) for the player, who realizes that he is able to control the bow working of his emotional state. This induces him to elaborate endogenous stimuli which generate the feedback chain.

At the beginning of the interaction, game parameters can be set. Among the other, a target emotional state (emotional goal) is selected. According the Russell’s Circumplex Model of Emotions [ 10 ], in the scope of our work, emotions are represented as a combination of valence and arousal, both ranging from low (-1) to neutral (0) to high (1) values: A 3x3 grid is thus obtained in which only the combinations (0, -1), (1,1), (-1,1) and (-1, -1) represent a emotional goals, respectively calm, happiness, anger and sadness (Fig. 3.a). According to the selected emotional goal, QB draws a target (Fig 3.b). Cartesian axes projected on the target, highlighting the values 1, -1 and 0 used to identify the coordinates in which to shoot the arrows. It should be noticed that such a structured Cartesian pattern places the centre of the target no longer in the coordinates (0,0), but in one of the four aforementioned pairs, depending on the target emotion of the game. As a consequence, the centre of the target represents one of the possible emotions (Calm, happiness, anger or sadness). For this reason, QB performs a last coordinate conversion based on the total pixels, to always get the image centered. The player has 8 chances out of 9 to make mistakes.

Two characteristics of QB make it an application particularly useful for our purposes: a.

Interaction level: The metaphor behind the game gives the user a clear understanding of game rules, as well as a clear interpretation of the effect of every move he makes and of how far he is from achieving his goal (winning the game). This allows to reduce the noise in terms of reduction of emotions due to the interaction

itself, which would certainly affect the progress of the game itself.

User Active participation: As an archer, the user determines the direction of the arrows, by controlling his emotions.

Brain Computer Interface and Classification The BCI is realized by BCI2000 [ 11 ]. BCI2000 is widely used in medical field and research. The main module is the Operator which provides a user graphical interface and allows to define parameters for each experimental session, to start and stop recording sessions. Under the Operator module, the Data Source Module configures the EmotivTMEpoc Headset’s sensors (information channel) [ 12 ] used for brain activity recordings. It converts brain activity into a long byte string to obtain a numeric value form each sensor receiving 4 of 14 channels at 128 Hz with a SampleBlockSize at 32; the SourceChGain, set from manufacturer at 0.003 µV, converts the transmitted values to the amplifier from analogic to digital for each channel. The Data Source Module also removes most of the EEG noise applying a low-pass filter at 30 Hz and a high-pass filter at 0.1 Hz; then the results are stored in a .dat file easily convertible into .csv extension for the offline analysis. Each file is divided into a header with operative information and a payload section that contains the raw signals.

Subsequently data are sent to the Signal Processing Module that uses a filter-chain to analyse and process signals (Spatial and Temporal Filter) converting it in the result output, by using machine learning algorithms. The Signal Processing Module used, includes a Common Average Reference [ 13 ] several used to identify small signal sources in very noisy recordings and a Fast Fourier Transform Filter to extract relevant signal’s features.

Two Support Vector Machine classifiers have been implemented by LibSVM [ 14 ]: one for the valence and one for the arousal. Although the classifiers were distinct, the prediction schemes model and the functions were similar: particularly the classifiers were multiclass and able to recognize between three output classes (one for each of three values for valence and arousal): 1, 0, -1. With a precision of around 0.68 and recall of 0.77, classifiers sent messages to QB that converts these values in arrow's coordinates. To train the classifiers we recorded about 128000 examples (128 features for each one), equally divided for each of the 3 classes both for valence and for arousal. The magnitude of each stimulation frequency has been used for classification. The dataset of signals recorded was scaled and normalized by tools provided by LibSVM. The software also includes algorithms to perform features extraction and selection (through cross validation). The classifiers use a kernel type rbf with gamma 1 for the 3 labels 3, 2, 1 (then converted in 1, 0, -1).

The start-preference panel of QB is not only used to configure the game parameters, but also to perform the classifier training phase. It’s possible, indeed, to turn off the gaming interaction and use only the stimuli selection implementation. After an emotion is chosen, QB loads the opportune initial emotional stimulus for the user. Then, we can record brain activity while QB offers stimuli to the user according to the chosen time and repetition settings. EXPERIMENTAL STUDY Approach Each experimental session opens with a phase of data collection on the user (pre-test) and ends with an interview to the user (post-test). The aim of our analysis is to evaluate the use of the interactive application in inducing chosen emotional states. In particular, we want to evaluate the quality of the interaction in terms of (i) effectiveness of process and (ii) usability of the application, perceived by users. The first evaluation reflects an objective quality of the software in terms of objectives achieved, depending on the heterogeneity of the users; the second, on the other hand, is a subjective measure, linked to each user and influenced by aspects such as satisfaction about the use of the platform, about convenience of the devices and about utility. How quickly the user can change his emotional state with respect to the ongoing feedback cycles will be an evaluation metrics.

Experimental Setup The experiment held in this study consisted of single sessions (see Fig. 4). Each session was divided into two trials, a familiarity trial and the game trial. In the familiarity trial participants could get used to drive their emotional state by selecting endogenous stimulation strategies, that shoot arrows in the target, this to avoid confounding variables such as learning a strategy to play the game. Each game trial consisted of four repetitions of sequences (one for each emotion among Calm, Happiness, Anger, Sadness) starting with a relaxing time (ten seconds) followed by a visual stimulation (five seconds) from The International Affective Picture System (IAPS) [ 15 ], followed by a preparation time (two seconds) and lasted in 45 seconds. Among repetitions, participants were given a break of five minutes.

IAPS is a database of pictures designed to provide a standardized set of pictures for studying emotions that has been widely used in psychological research. It is the essential property of the IAPS that the stimulus set is accompanied by a detailed list of average ratings of the emotions elicited by each picture, in term of valence and arousal. This shall enable we to previously select stimuli eliciting a specific range of emotions: picture were chosen with high arousal and negative valence to elicit in the user anger-disgust, high arousal and positive valence to elicit in the user happiness-joy, low arousal and neutral valence to elicit in the user calm-stillness, low arousal and negative valence to elicit sadness in the user, so that it could be offered to the player, an initial stimulus opposed to the emotion that is the emotional goal of the game. So far, if the user play to the Calm (low arousal and neutral valence), then the initial stimulus will be a picture from IAPS that elicit at least high arousal and neutral/negative valence (i.e. the Anger).

The platform and the game ran on a PC connected to a 15.6-inch screen, placed at a distance of about forty centimetres from the user's gaze. The PC was equipped with a 2.40 Ghz processor, 4 GB of RAM and with Windows 10 64-bit. To avoid source of distraction, one user at time experimented the EI interface. The data acquisition ran on the same PC and exchanged the raw electroencephalography (EEG) data to the platform. Participants and Data Gathering Ten participants (5 female and 5 male), aged between 20 and 60 (µ= 38.4; σ=15.28) participated in the experiment. Before the experiment, each participant has been asked to sign an informed consent and, subsequently, to answer a preliminary questionnaire in order to set an initial profile of the user. All participants had normal or corrected-to-normal vision and described themselves as daily computer users. Nobody had experience with EEG or BCIs.

Prior to the experiment, 14 electrodes were placed according to the international 10- 20 system [ 16 ]. For the EEG data acquisition, the EmotivTM Epoc headset was used. According to the literature [ 9 ], during each session EEG data was used from four frontal electrodes (AF3, AF4, F3, F4). The EEG data was handled as in stated in section Brain Computer Interface.

In order to check the interaction quality in terms of the users’ satisfaction and the perceived-from-user effectiveness in winning the game, at the end of the interaction each user answered a self-assessment questionnaire. It contains 9 questions and is designed to measure different factors linked to immersion (cognitive involvement, emotional involvement, challenge and control).

Furthermore, some game statistics - such as for each emotional goal, the average number of arrows shot in the center of the target, mediated on all users, or the valence and the arousal trend, mediated on all the repetition of all the game sessions - were collected while participants played the game.

The valence and arousal trend would be a good indication of how quickly users can change their emotional state with respect to the ongoing feedback cycles.

After the experiments, scores were obtained for the immersion factors and the game statistics.

RESULTS Based on the post-questionnaire, the autogenous training game was highly appreciated in terms of emotional involvement, control, cognitive involvement and challenge. The scores, expressed on a scale of values from 1 to 5, averaged over participants, are shown in Table1.

Emotional

Control

Cognitive

Challenge

µ 4.7 4.5 4.7 4.9 On average, users claim to have had fun and fully understand how to interact with the platform. These results are opposed to the initial skepticism about the possibility of playing a game only by governing their emotional flow. From the analysis of the pre-test questionnaires, these initial impressions are more evident in users with little digital experience. This could be linked to the classical concept of controller, commonly understood as a tangible hardware device (mouse, keyboard ...). About the EMOTIV device there are differences in terms of comfort: on a scale of values from 1 to 5 where 1 indicates extreme discomfort and with 5 maximum comfort, the device obtains a rating of 4.5. All users have assigned values 4 or 5. The 75% of 4 were from women with long hair. Finally, only one user complained of an excessive pressure on the head during the use of the device, for prolonged periods.

During the experiments, game statistics were collected: The 72% of the arrows were shot in the center of the target or in a close range. Anger is the emotions for which it is easier to activate endogenous strategies (85%); Calm follows (60%). Activating endogenous strategies for happiness or depression seems to be a difficult task. This is probably due to the fact that these two emotions correspond to a mood more than to an-event-related emotion. Furthermore, playing with anger in the target does not seem to strain the user, regardless of which emotions he has played previously, during the same experiment: users needed less time to focus on the target.

Finally, given an emotion, in order to evaluate if and how quickly users changed their emotional state with respect to the ongoing feedback cycles, we calculated Pearson correlation coefficient, with respect to detected values of valence and arousal, over time. Table 2 shows the Pearson coefficients, for each emotional goal, for each user. According to our experimental setup, as the initial stimulus will be opposed to the emotion that is the emotional goal of the game, if users play with anger (resp. calm) in the target, an average negative correlation (resp. positive) of valence over the time and an average positive correlation (resp. negative) of arousal over the time indicate a tendency to approach the emotions of the opposite quadrant (according to Russell's model of representation of emotions), that is a rapprochement with the emotional goals, on average; an average positive correlation of both valence and arousal over the time indicate a rapprochement with the emotional goal of happiness. Sadness is an exception: although the arousal tends to decay on average, the valence tends to remain positive. This may be due to the fact that when playing with sadness, the user is called to activate an endogenous strategy in order to move from an emotion with a very positive valence (happiness), to an emotion with a very negative valence (sadness), this in a playful context. In addition, it is known that the decay of emotions depends on the emotions itself, but also on the context in which emotion appears [ 5 ]: in a playful context, as in our case, happiness could be a slow-decay-emotion and therefore, when sadness appears it is plausible that happiness is not yet completely decayed, causing an overlapping of the two emotions (as in “odi et amo” - microwave metaphor [ 5 ]), as well as noise in the detection of emotions. (V/t)/(A/t) Calm

Happiness

Anger

Sadness User1 User2 User3 User4 User5 User6 User7 User8 User9 User10 Average

A final consideration: some correlation coefficients in Table 2 have value zero. Even these cases can be considered positive since they occurred into two different conditions: (i) when a player has shot an arrow that almost immediately hit the center of the target, (ii) when a large number of arrows shot in the center of the target, when the remaining ones were too far.

CONCLUSIONS In this paper we presented a serious game, called “Quiet Bowman”, in which we employ human emotions for two main purposes: (i) to experience emotional behaviors, and (ii) to explore the game dynamics related to emotions in order to learn how to manage one own emotional state. In particular, the game uses a platform, the EmoBrain Interface, to recognize emotions from EEG signals that are used as input for the interacting with the game. The EI interaction is a cycle of stimulus and feedbacks where the user receives a visual input and completes tasks, just controlling his emotional state, by activating endogenous training strategies. The EI platform is able to recognize emotions according to the Russell’s Circumplex Model of Emotions using two classifiers one for the valence and the other for the arousal dimension.

Results of the study presented in the paper show that the autogenous training game was highly appreciated in terms of emotional involvement, control, cognitive involvement and challenge. Moreover they show that is possible to play a game only by governing one own emotional flow and encourage us to make new experiments in order to improve the accuracy of our classifiers and therefore the impact of the serious game on the autogenous training of the user. ACKNOWLEDGMENTS We thank all the volunteers who participated in the study presented in this paper.

References

1. Norman , D. : Emotion & design: attractive things work better . Interactions 9 ( 4 ), 36 - 42 ( 2002 )

2. Carofiglio , V. , Abbattista , F. Understanding , Modeling and Exploiting User's Emotions for Brain-Driven Interface Design Application to an Adaptive-3DVirtual-Environment . INTERNATIONAL JOURNAL OF PEOPLE-ORIENTED PROGRAMMING .Vol. 3 ,pp. 1 - 21 . ISSN: 2156 - 1796 . 2014 .

3. Carofiglio , V. Abbattista , F. A Rough BCI-based Assessment of User's Emotions for Interface Adaptation: Application to a 3D-Virtual- Environment Exploration Task . Proceedings of the First International Workshop on Intelligent User Interfaces: Artificial Intelligence meets Human Computer Interaction (AI*HCI 2013 ) A workshop of the XIII International Conference of the Italian Association for Artificial Intelligence (AI*IA 2013 ). 2013 .

4. Carofiglio , V. Abbattista , F. BCI-Based User-Centered Design for Emotionally-Driven User Experience . Cases on Usability Engineering: Design and Development of Digital Products . pp. 299 -- 320 . doi: 10 .4018/978-1- 4666 -4046-7; ISBN:9781466640467; Miguel A. GarciaRuiz (Pubs .), University of York, UK, 2013 .

5. Picard , R.W. , Affective computing. 1995

6. Garcia-Molina , G. , Tsoneva , T. , Nijholt , A. , Emotional brain-computer interfaces , International Journal of Autonomous and Adaptive Communications Systems, v.6 n.1 , p. 9 - 25 , December 2013 [doi>10.1504/IJAACS. 2013 .05068]

7. Bos , D. , “ EEG-based emotion recognition - the influence of visual and auditory stimuli” , Emotion , 57 ( 7 ): 1798 - 806 , 2006 .

8. Van Gerven , M. , et Al.. The brain-computer interface cycle . Journal of neural engineering 6 (4) , 041001

9. Zander , T.O. , Kothe , C. , Jatzev , S. , and Gaertner , M. , Enhancing human computer interaction with input from active and passive brain-computer interfaces . In BrainComputer Interfaces - Ap Applying our Minds to Human-Computer Interaction, pp. 181 - 199 , 2010 .

10. J. Posner , J. A. Russell , and B. S. Peterson , “ The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development , and psychopathology”, Development and psychopathology , 17 ( 3 ): 715 - 734 , 2005 .

11. Schalk , G. , Mellinger , J. A practical guide to braincomputer interfacing with BCI2000 . Springer, London. ( 2010 )

12. https://www.emotiv.com/epoc/

13. Cooper , R. , Osselton , J. , Shaw , J.: EEG technology . Butterworths ( 1969 )

14. LIBSVM : A Library for Support Vector Machines ACM Trans . Intell. Syst. Technol., Vol. 2 , No. 3 . (May 2011 ), doi:10.1145/1961189.1961199 by Chih C. Chang , Chih J . Lin

15. Lang , P.J. , Bradley , M.M. , & Cuthbert , B.N. ( 2008 ). International affective picture System (IAPS): Affective ratings of pictures and instruction manual . Technical Report A-8 . University of Florida, Gainesville, FL.

16. Reilly , E.L. EEG Recording and Operation of the Apparatus . In: Electroencephalography: Basic Principles, Clinical Applications and Related Fields , pp. 139 - 160 . Lippincott Williams & Wilkins ( 1999 )