EmoBrain: Playing with Emotions in the Target Vincenzo Liberti Valeria Carofiglio Berardina De Carolis Department of Computer Department of Computer Department of Computer Science Science Science Bari, Italy Bari, Italy Bari, Italy vi.liberti@outlook.com valeria.carofiglio@uniba.it berardina.decarolis@uni ba.it Fabio Abbattista Department of Computer Science Bari, Italy fabio.abbattista@uniba.it ABSTRACT decision-making [1]. Being aware of self emotional state Sensing and understanding human emotional behaviour and start working on its regulation is important to achieve a seems to be essential to keep engaging interactions better wellness level, since emotion regulation can help to sustained over longer periods. The general purpose of this mitigate emotion related biases in our everyday tasks. work is to implement a platform able to recognize and Recently serious games have been used as a mean to learn employ human emotions in an interactive game: The how to control and regulate emotions. Many serious games EmoBrain Interface (EI). EI interaction is a cycle of developed in this field use biofeedback information to stimulus and feedbacks where the user receives a visual display player’s emotional state and help them to train in input and completes tasks, just controlling his emotional order to improve his own wellness. state by activating self-training strategies. EI has been used in the context of a serious game: Quiet Bowman (QB). QB Our research work is placed in this context. We carried on allows the users: (i) to experience emotional behaviours and previous research [2,3,4] and developed a serious game, (ii) to explore the game dynamics related to emotions in called “Quiet Bowman” (QB). QB allows the users: (i) to order to manage their own emotional state. The outcomes experience emotional behaviors; and (ii) to explore the seem to be encouraging. Users appreciated the game: they game dynamics related to emotions to manage their own felt involved and committed to achieve the required goals. emotional state. This is related to the implementation of a This encourages us to make new experiments to improve platform able to recognize and use user’s emotions as input the accuracy of our classifiers and therefore the impact of to the interactive game: The EmoBrain Interface (EI). The the serious game on the autogenous training of the user. EI implements a cycle of stimulus and feedback where the user receives a visual output and completes tasks, just Author Keywords controlling his emotional state, by activating autogenous HCI; Emotions; Game; Brain Computer Interface; training strategies. ACM Classification Keywords H.5.2. User Interfaces. I.2.1 Applications and Expert Systems Although there are many dimensions associated with emotions, according to Picard [5] the two most commonly used dimensions of emotion are valence, and arousal. INTRODUCTION Picard also notes that the valence and arousal dimensions Emotions are part of our everyday living and influence are critical in games applications [5]. The recognition of many human processes such as cognition, perception, and user’s emotional state has to be performed in an implicit everyday tasks such as learning, communication and and transparent way, so as to be non-invasive and more effective. Moreover, traditional methods, such as self-report or interviews, are only partially useful (and concern the preliminary research activities only), because they are based on sampling techniques or simply on the a-posteriori user’s perception of the game environment. GHItaly18: 2nd Workshop on Games-Human Interaction, May 29th, 2018, We decided to couple the Emotional Brain Computer Castiglione della Pescaia, Grosseto (Italy) Interfaces (EBCI) [6,7] to traditional methods. Copyright © 2018 for the individual papers by the papers' authors. Copying An EBCI is a particular kind of a Brain Computer Interface permitted for private and academic purposes. This volume is published and copyrighted by its editors. (BCI). A BCI [8] is a direct communication pathway between the brain and an external device. By means of an At the beginning of the interaction, in order to alter his electroencephalogram (EEG), it records human brain emotional state, an exogenous stimulus (i.e. photos or other activity in the form of electrical potentials (EPs), through multimedia content) is send to the user. The EI detects such multiple electrodes that are placed on the scalp. EPs are emotional alterations and sends feedback messages to the processed to obtain features that can be grouped into a user. The feedback received from the user will, in turn, be feature vector: Depending on the brain activity, distinctive used by him to initiate an appropriate endogenous known patterns in the EEG appear. These are automatically stimulation strategy, to achieve the emotional goal. recognized by the BCI and associated with a given action User Application: Quiet Bowman (QB) on the BCI application. The outcome of this action can be In order to let EI platform handle user emotions in the perceived by the user in terms of application feedback. In scope of an interactive game, we implemented QB. As this case, his brain activity is consequently modulated. The stated, QB is a simplified version of a serious game for kind of EBCI mostly used in this work is the reactive EBCI autogenous training. It is an archery game in which the [9]. A reactive EBCI can send stimuli and extract player plays with the goal of achieving an emotional state information from user’s brain elaboration. So, if during a represented in the centre of the target (emotional goal). session game, the player has been scared or disgusted, the Four emotions could be the emotional goal: Calm, EI will recognize a medium-high value for arousal with a Happiness, Anger or Sadness. We chose these emotions strongly negative valence. Otherwise, will be recognised because their components (valence and arousal) better positive valence and high arousal, for happiness. match the emotional chose values: high, medium and low In the scope of our work, the feedback about the progress of for arousal; positive, neutral and negative for valence. the game is a key tool: it allows the players to monitor their Indeed, calm, for example, has neutral values for valence performance, but it also generates new stimuli caused by and slightly low for arousal. On contrary, anger has high the achievement of the goal (feedback loop). arousal and a low valence. In addition, these four emotions are commonly and clearly identified by users. The paper is organized as follows: in Section 2 we provide a description of the architecture of the system. In Section 3 The only controller in the game is user’s affective state: at we illustrate the experimental study we conducted to train each step, the user emotion is recognized by the EI and and test our system. In Section 4 the results of the study are represented as an arrow on the target. Each step is discussed. Conclusions and future work directions are automatically triggered by a timer. To win the game, the reported in Section 5. player must be able to drive his emotional state in order to achieve the emotional goal, by activating any endogenous OVERVIEW OF THE ARCHITECTURE stimulation strategy. The EI is a distributed platform to recognize and employ human emotions to drive an interactive system (Fig. 1). The A screenshot of the game can be seen in Fig. 2. Seven shots EI includes an input device to record user’s EEG. The brain were fired. The seventh (t7) hit the target. The messages activity (EEG signal) is then transmitted to the BCI that exchange among EI, QB and the user takes the name of analyses and processes it for to recognize user’s emotion feedback loop. So, for example, let the user play whit the following a visual stimulation. The EI also includes QB, a emotion of Calm. At the beginning of the interaction, in serious game for autogenic training, in which the user is a order to alter his emotional state, a picture is presented to player that shoots arrows in order to hit the target the user. The EI detects such emotional alterations and (emotional goal) and receive feedbacks, according to his sends a feedback messages to the user (an arrow in the emotional state. target: t1, in Fig.2). Figure 1. The EmoBrain Interface Architecture. Figure 2. QB Screenshot. The feedbacks received from the user will, in turn, be used itself, which would certainly affect the progress of the by him to initiate an appropriate endogenous stimulation game itself. strategy, to reach a state of quiet. In doing so, however, he b. User Active participation: As an archer, the user will send new data to the EI and a new arrow will be fired determines the direction of the arrows, by controlling from the QB. The display of the latter arrow (or the sound, his emotions. if set, for this event) on the target constitutes a new stimulus Brain Computer Interface and Classification (exogenous) for the player, who realizes that he is able to The BCI is realized by BCI2000 [11]. BCI2000 is widely control the bow working of his emotional state. This used in medical field and research. The main module is the induces him to elaborate endogenous stimuli which Operator which provides a user graphical interface and generate the feedback chain. allows to define parameters for each experimental session, At the beginning of the interaction, game parameters can be to start and stop recording sessions. Under the Operator set. Among the other, a target emotional state (emotional module, the Data Source Module configures the goal) is selected. According the Russell’s Circumplex EmotivTMEpoc Headset’s sensors (information channel) Model of Emotions [10], in the scope of our work, emotions [12] used for brain activity recordings. It converts brain are represented as a combination of valence and arousal, activity into a long byte string to obtain a numeric value both ranging from low (-1) to neutral (0) to high (1) values: form each sensor receiving 4 of 14 channels at 128 Hz with A 3x3 grid is thus obtained in which only the combinations a SampleBlockSize at 32; the SourceChGain, set from (0, -1), (1,1), (-1,1) and (-1, -1) represent a emotional goals, manufacturer at 0.003 µV, converts the transmitted values respectively calm, happiness, anger and sadness (Fig. 3.a). to the amplifier from analogic to digital for each channel. According to the selected emotional goal, QB draws a The Data Source Module also removes most of the EEG target (Fig 3.b). noise applying a low-pass filter at 30 Hz and a high-pass filter at 0.1 Hz; then the results are stored in a .dat file easily convertible into .csv extension for the offline analysis. Each file is divided into a header with operative information and a payload section that contains the raw signals. Subsequently data are sent to the Signal Processing Module that uses a filter-chain to analyse and process signals (Spatial and Temporal Filter) converting it in the result output, by using machine learning algorithms. The Signal Processing Module used, includes a Common Average a. b. Reference [13] several used to identify small signal sources Figure 3. Target implementation in very noisy recordings and a Fast Fourier Transform Filter to extract relevant signal’s features. Cartesian axes projected on the target, highlighting the Two Support Vector Machine classifiers have been values 1, -1 and 0 used to identify the coordinates in which implemented by LibSVM [14]: one for the valence and one to shoot the arrows. It should be noticed that such a for the arousal. Although the classifiers were distinct, the structured Cartesian pattern places the centre of the target prediction schemes model and the functions were similar: no longer in the coordinates (0,0), but in one of the four particularly the classifiers were multiclass and able to aforementioned pairs, depending on the target emotion of recognize between three output classes (one for each of the game. As a consequence, the centre of the target three values for valence and arousal): 1, 0, -1. With a represents one of the possible emotions (Calm, happiness, precision of around 0.68 and recall of 0.77, classifiers sent anger or sadness). For this reason, QB performs a last messages to QB that converts these values in arrow's coordinate conversion based on the total pixels, to always coordinates. To train the classifiers we recorded about get the image centered. The player has 8 chances out of 9 128000 examples (128 features for each one), equally to make mistakes. divided for each of the 3 classes both for valence and for Two characteristics of QB make it an application arousal. The magnitude of each stimulation frequency has particularly useful for our purposes: been used for classification. The dataset of signals recorded was scaled and normalized by tools provided by LibSVM. a. Interaction level: The metaphor behind the game gives The software also includes algorithms to perform features the user a clear understanding of game rules, as well extraction and selection (through cross validation). The as a clear interpretation of the effect of every move he classifiers use a kernel type rbf with gamma 1 for the 3 makes and of how far he is from achieving his goal labels 3, 2, 1 (then converted in 1, 0, -1). (winning the game). This allows to reduce the noise in terms of reduction of emotions due to the interaction The start-preference panel of QB is not only used to configure the game parameters, but also to perform the classifier training phase. It’s possible, indeed, to turn off the user play to the Calm (low arousal and neutral valence), gaming interaction and use only the stimuli selection then the initial stimulus will be a picture from IAPS that implementation. After an emotion is chosen, QB loads the elicit at least high arousal and neutral/negative valence (i.e. opportune initial emotional stimulus for the user. Then, we the Anger). can record brain activity while QB offers stimuli to the user The platform and the game ran on a PC connected to a according to the chosen time and repetition settings. 15.6-inch screen, placed at a distance of about forty EXPERIMENTAL STUDY centimetres from the user's gaze. The PC was equipped with Approach a 2.40 Ghz processor, 4 GB of RAM and with Windows 10 Each experimental session opens with a phase of data 64-bit. To avoid source of distraction, one user at time collection on the user (pre-test) and ends with an interview experimented the EI interface. The data acquisition ran on to the user (post-test). The aim of our analysis is to evaluate the same PC and exchanged the raw the use of the interactive application in inducing chosen electroencephalography (EEG) data to the platform. emotional states. In particular, we want to evaluate the quality of the interaction in terms of (i) effectiveness of process and (ii) usability of the application, perceived by Participants and Data Gathering users. The first evaluation reflects an objective quality of Ten participants (5 female and 5 male), aged between 20 the software in terms of objectives achieved, depending on and 60 (µ= 38.4; σ=15.28) participated in the experiment. the heterogeneity of the users; the second, on the other Before the experiment, each participant has been asked to hand, is a subjective measure, linked to each user and sign an informed consent and, subsequently, to answer a influenced by aspects such as satisfaction about the use of preliminary questionnaire in order to set an initial profile of the platform, about convenience of the devices and about the user. All participants had normal or corrected-to-normal utility. How quickly the user can change his emotional state vision and described themselves as daily computer users. with respect to the ongoing feedback cycles will be an Nobody had experience with EEG or BCIs. evaluation metrics. Prior to the experiment, 14 electrodes were placed Experimental Setup according to the international 10- 20 system [16]. For the The experiment held in this study consisted of single EEG data acquisition, the EmotivTM Epoc headset was sessions (see Fig. 4). Each session was divided into two used. According to the literature [9], during each session trials, a familiarity trial and the game trial. In the familiarity EEG data was used from four frontal electrodes (AF3, AF4, trial participants could get used to drive their emotional F3, F4). The EEG data was handled as in stated in section state by selecting endogenous stimulation strategies, that Brain Computer Interface. shoot arrows in the target, this to avoid confounding In order to check the interaction quality in terms of the variables such as learning a strategy to play the game. Each users’ satisfaction and the perceived-from-user game trial consisted of four repetitions of sequences (one effectiveness in winning the game, at the end of the for each emotion among Calm, Happiness, Anger, Sadness) interaction each user answered a self-assessment starting with a relaxing time (ten seconds) followed by a questionnaire. It contains 9 questions and is designed to visual stimulation (five seconds) from The International measure different factors linked to immersion (cognitive Affective Picture System (IAPS) [15], followed by a involvement, emotional involvement, challenge and preparation time (two seconds) and lasted in 45 seconds. control). Among repetitions, participants were given a break of five minutes. Furthermore, some game statistics - such as for each emotional goal, the average number of arrows shot in the IAPS is a database of pictures designed to provide a center of the target, mediated on all users, or the valence standardized set of pictures for studying emotions that has and the arousal trend, mediated on all the repetition of all been widely used in psychological research. It is the the game sessions - were collected while participants played essential property of the IAPS that the stimulus set is the game. accompanied by a detailed list of average ratings of the emotions elicited by each picture, in term of valence and The valence and arousal trend would be a good indication arousal. This shall enable we to previously select stimuli of how quickly users can change their emotional state with eliciting a specific range of emotions: picture were chosen respect to the ongoing feedback cycles. with high arousal and negative valence to elicit in the user After the experiments, scores were obtained for the anger-disgust, high arousal and positive valence to elicit in immersion factors and the game statistics. the user happiness-joy, low arousal and neutral valence to elicit in the user calm-stillness, low arousal and negative valence to elicit sadness in the user, so that it could be RESULTS offered to the player, an initial stimulus opposed to the Based on the post-questionnaire, the autogenous training emotion that is the emotional goal of the game. So far, if the game was highly appreciated in terms of emotional involvement, control, cognitive involvement and challenge. approach the emotions of the opposite quadrant (according The scores, expressed on a scale of values from 1 to 5, to Russell's model of representation of emotions), that is a averaged over participants, are shown in Table1. rapprochement with the emotional goals, on average; an average positive correlation of both valence and arousal over the time indicate a rapprochement with the emotional µ goal of happiness. Sadness is an exception: although the Emotional 4.7 arousal tends to decay on average, the valence tends to remain positive. This may be due to the fact that when Control 4.5 playing with sadness, the user is called to activate an Cognitive 4.7 endogenous strategy in order to move from an emotion with a very positive valence (happiness), to an emotion with a Challenge 4.9 very negative valence (sadness), this in a playful context. In Table 1. Average results about several factors linked to addition, it is known that the decay of emotions depends on immersion. On a scale of values from 1 to 5, where 1 indicates the emotions itself, but also on the context in which very negative and very positive emotion appears [5]: in a playful context, as in our case, happiness could be a slow-decay-emotion and therefore, On average, users claim to have had fun and fully when sadness appears it is plausible that happiness is not understand how to interact with the platform. These results yet completely decayed, causing an overlapping of the two are opposed to the initial skepticism about the possibility of emotions (as in “odi et amo” - microwave metaphor [5]), as playing a game only by governing their emotional flow. well as noise in the detection of emotions. From the analysis of the pre-test questionnaires, these initial impressions are more evident in users with little digital (V/t)/(A/t) Calm Happiness Anger Sadness experience. This could be linked to the classical concept of User1 -0.144 / 0 0.433 / -0.392 0.204 / 0.405 0.632 / -0.392 controller, commonly understood as a tangible hardware device (mouse, keyboard ...). About the EMOTIV device User2 0 / 0.200 0 / 0.882 -0.790 / 0 0.433 / -0.784 there are differences in terms of comfort: on a scale of User3 0/0 0.474 / -0.685 0.288 / 0 0.866 / -0.002 values from 1 to 5 where 1 indicates extreme discomfort and with 5 maximum comfort, the device obtains a rating of User4 -0.47 / -0.385 -0.316 / 0.204 -0.686 / 0 0/0 4.5. All users have assigned values 4 or 5. The 75% of 4 User5 0.144 / -0.200 -0.408 / 0 0.144 / -0.204 0.433 / -0.203 were from women with long hair. Finally, only one user complained of an excessive pressure on the head during the User6 0.358 / 0.243 0 / -0.500 0.408 / 0 -0.560 / 0.002 use of the device, for prolonged periods. User7 -0.790 / 0.300 0.72 / 0.490 0/0 0 / 0.036 During the experiments, game statistics were collected: The User8 0.351 / -0.153 0.002 / 0,211 0 / 0.35 -0.433 / -0.511 72% of the arrows were shot in the center of the target or in User9 0 / -0.245 -0.351 / 0.057 -0.003 / -0.11 -0.158 / 0.319 a close range. Anger is the emotions for which it is easier to activate endogenous strategies (85%); Calm follows (60%). User10 0.103 / -0.002 -0.206 / 0 -0.31 / -0.888 -0.491 / -0.142 Activating endogenous strategies for happiness or Average 0.045/-0.024 0.034/0.07 -0.074/0.035 0.072/-0.168 depression seems to be a difficult task. This is probably due to the fact that these two emotions correspond to a mood more than to an-event-related emotion. Furthermore, Table 2. Pearson correlation coefficient, with respect to playing with anger in the target does not seem to strain the detected values of valence and arousal, over time. user, regardless of which emotions he has played A final consideration: some correlation coefficients in previously, during the same experiment: users needed less Table 2 have value zero. Even these cases can be time to focus on the target. considered positive since they occurred into two different Finally, given an emotion, in order to evaluate if and how conditions: (i) when a player has shot an arrow that almost quickly users changed their emotional state with respect to immediately hit the center of the target, (ii) when a large the ongoing feedback cycles, we calculated Pearson number of arrows shot in the center of the target, when the correlation coefficient, with respect to detected values of remaining ones were too far. valence and arousal, over time. Table 2 shows the Pearson coefficients, for each emotional goal, for each user. CONCLUSIONS According to our experimental setup, as the initial stimulus In this paper we presented a serious game, called “Quiet will be opposed to the emotion that is the emotional goal of Bowman”, in which we employ human emotions for two the game, if users play with anger (resp. calm) in the target, main purposes: (i) to experience emotional behaviors, and an average negative correlation (resp. positive) of valence (ii) to explore the game dynamics related to emotions in over the time and an average positive correlation (resp. order to learn how to manage one own emotional state. In negative) of arousal over the time indicate a tendency to particular, the game uses a platform, the EmoBrain 7. Bos, D., “EEG-based emotion recognition – the Interface, to recognize emotions from EEG signals that are influence of visual and auditory stimuli”, Emotion, used as input for the interacting with the game. The EI 57(7):1798-806, 2006. interaction is a cycle of stimulus and feedbacks where the 8. Van Gerven, M., et Al.. The brain–computer interface user receives a visual input and completes tasks, just cycle. Journal of neural engineering 6 (4), 041001 controlling his emotional state, by activating endogenous 9. Zander, T.O., Kothe,C., Jatzev, S., and Gaertner, M., training strategies. The EI platform is able to recognize Enhancing human computer interaction with input from emotions according to the Russell’s Circumplex Model of active and passive brain-computer interfaces. In Brain- Emotions using two classifiers one for the valence and the Computer Interfaces - Ap Applying our Minds to other for the arousal dimension. Human-Computer Interaction, pp. 181-199, 2010. Results of the study presented in the paper show that the autogenous training game was highly appreciated in terms 10. J. Posner, J. A. Russell, and B. S. Peterson, “The of emotional involvement, control, cognitive involvement circumplex model of affect: an integrative approach to and challenge. Moreover they show that is possible to play affective neuroscience, cognitive development, and a game only by governing one own emotional flow and psychopathology”, Development and psychopathology, encourage us to make new experiments in order to improve 17(3):715-734, 2005. the accuracy of our classifiers and therefore the impact of 11. Schalk, G., Mellinger, J. A practical guide to brain– the serious game on the autogenous training of the user. computer interfacing with BCI2000. Springer, London. (2010) ACKNOWLEDGMENTS 12. https://www.emotiv.com/epoc/ We thank all the volunteers who participated in the study 13. Cooper, R., Osselton, J., Shaw, J.: EEG technology. presented in this paper. Butterworths (1969) REFERENCES 14. LIBSVM: A Library for Support Vector 1. Norman, D.: Emotion & design: attractive things work Machines ACM Trans. Intell. Syst. Technol., Vol. 2, better. Interactions 9(4), 36–42 (2002) No. 3. (May 2011), doi:10.1145/1961189.1961199 by 2. Carofiglio, V., Abbattista, F. Understanding, Modeling Chih C. Chang, Chih J. Lin and Exploiting User's Emotions for Brain-Driven 15. Lang, P.J., Bradley, M.M., & Cuthbert, B.N. (2008). Interface Design Application to an Adaptive-3D- International affective picture System (IAPS): Affective Virtual-Environment. INTERNATIONAL JOURNAL ratings of pictures and instruction manual. Technical OF PEOPLE-ORIENTED Report A-8. University of Florida, Gainesville, FL. PROGRAMMING.Vol.3,pp.1-21. ISSN:2156-1796. 16. Reilly, E.L. EEG Recording and Operation of the 2014. Apparatus. In: Electroencephalography: Basic 3. Carofiglio, V. Abbattista,F. A Rough BCI-based Principles, Clinical Applications and Related Fields, pp. Assessment of User's Emotions for Interface 139–160. Lippincott Williams & Wilkins (1999) Adaptation: Application to a 3D-Virtual- Environment Exploration Task. Proceedings of the First International Workshop on Intelligent User Interfaces: Artificial Intelligence meets Human Computer Interaction (AI*HCI 2013) A workshop of the XIII International Conference of the Italian Association for Artificial Intelligence (AI*IA 2013). 2013. 4. Carofiglio, V. Abbattista, F. BCI-Based User-Centered Design for Emotionally-Driven User Experience. Cases on Usability Engineering: Design and Development of Digital Products. pp.299--320. doi: 10.4018/978-1- 4666-4046-7; ISBN:9781466640467; Miguel A. Garcia- Ruiz (Pubs.), University of York, UK, 2013. 5. Picard, R.W., Affective computing. 1995 6. Garcia-Molina, G., Tsoneva,T., Nijholt, A., Emotional brain-computer interfaces, International Journal of Autonomous and Adaptive Communications Systems, v.6 n.1, p.9-25, December 2013 [doi>10.1504/IJAACS.2013.05068]