Dynamic models for emotion estimation from physiological signals

Abstract

The ultimate goal of Human-Machine Interaction is to make interaction as natural as possible. To accomplish this, the recognition of the user's emotional state is considered an important factor. The ifeld of emotion recognition and modelling has predominantly employed static machine learning approaches that ignore the dynamic nature of emotions. However, this dynamic character has recently been highlighted by the emergence of appraisal models (e.g., Scherer's Component Process Model, CPM). These recent developments of emotion theory have been combined with Dynamic Field Theory (a wellestablished framework in the field of embodied cognition) to model emotion intensity based on galvanic skin response changes. The present work aims for an extended approach that considers not only the intensity, but also the quality of emotions as well as the dynamic and simultaneous changes of both. To create a dynamic emotion model, we will record and analyse electrophysiological signals. In contrary to most studies in literature where the assessment of the subjective feeling is performed after the exhibition of a stimulus, we will assess the subjective feeling online (during the emotion elicitation and data collection). This will allow us to directly compare the recorded subjective feeling with the dynamic output of our model. The development of such a dynamic model will not only contribute for a better understanding of the emotional processes but will also benefit several real-world applications, such as gaming, mental health monitoring, and driving-assistance technologies.

Keywords

• eol>emotion recognition
• appraisal models
• emotion dynamics
• electrophysiological signals

1. Introduction and Objectives

Emotions play a crucial role in people’s lives, influencing how we think and behave. Besides the introspective character of emotions, they are particularly important in communication and the ability to recognise other people’s emotions is a sign of emotion intelligence. With this in mind, intelligent user interfaces need to exhibit this ability of recognising emotions to replicate a human-like interaction and better adapt the system behaviour [ 1 ].

Over the last two decades, eforts have been made to predict and model users’ emotional states [ 2 ]. Electrophysiological methods represent a powerful tool for this purpose, since they are involuntary, dificult to mask, and able to capture spontaneous and subconscious information continuously. These signals provide complementary information and thus their combination can be used to build a multi-modal approach for emotion recognition [ 3, 4 ].

The dynamic nature of emotions is often recognised in diferent theories [ 5 ] and therefore one could expect that state-of-the-art techniques for emotion recognition would take this into consideration. Nevertheless, emotion dynamics has been vastly disregarded in emotion recognition studies, which represents a lack in the literature that needs to be settled.

In most studies, an afective state is commonly detected within a time window by employing static machine learning approaches [ 2 ], which can either be traditional statistical methods [ 6, 7 ] or deep learning approaches [ 8, 9 ]. On the other hand, the emergence of the so-called appraisal models emphasised the dynamic nature of emotions, since emotions are defined as processes and involve diferent components and their interactions in time [ 10 ], bringing a refreshed architecture to investigate emotion dynamics. Given the early stage of this field, further studies need to be conducted to consolidate the findings and to extend the preliminary models, since the few existent literature is dominated by behavioural studies. Also, the subjectively felt emotion is poorly assessed in most studies, which does not provide a reliable ground truth.

To overcome the presented limitations, Jenke & Peer (2018) [ 4 ] employed galvanic skin response to dynamically model emotion intensity over time. In this study, a specific appraisal model was considered — the Scherer’s Component Process Model (CPM) [ 11, 12 ]. In fact, unlike the machine learning approaches who behave like black-box models to recognise emotions [ 13 ], the CPM provides a mean to combine both theoretical and empirical properties over time (grey-box approach). Therefore, Jenke & Peer (2018) [ 4 ] took advantage of this to develop a dynamic grey-box model for intensity estimation based on the Dynamic Field Theory [ 14 ]. Moreover, the subjectively felt intensity was measured in real-time during the exhibition of IAPS images [ 15 ] and, thus, this information could also be included in the model. This incorporation is essential for a correct interpretation of the model. We consider this study pioneer in multiple aspects, but especially in the introduction of a dynamic model to predict afective states from physiological signals and taking theoretical knowledge from the CPM into account. Despite the relevance of this work, it still presents some limitations that need to be overcome, namely: the intensity model is still requiring extra information for a better prediction; the emotional states were merely divided into positive, negative, and calm; and it just considers changes in intensity over time (other types of dynamics are disregarded).

The abovementioned limitations will be overcome through this project with the addition of extra information provided by other electrophysiologial signals, since the model can embody this information. We will analyse the diferences in intensity for distinct emotion qualities. Moreover, the emotion quality will be introduced in the model to predict trajectories between emotional states, as well as to investigate whether the quality transitions afect emotion intensity.

The subjective feeling, that plays a special role in the CPM, integrates the information from all the emotional components and materialises it in a conscious representation of the afective state. This perception is essential for emotion regulation, in which the individual has the ability to control one’s own emotional state [ 16 ]. In this sense, the mentioned assessment of the subjective feeling may represent an advantage in the incorporation of this type of models in clinical tools capable of helping individuals in this regulation. Moreover, these models may also benefit several “daily-life” applications, such as driving-assistance technologies and gaming.

2. Proposed Approach

2.1. Model The Component Process Model encompasses five components that interact with one another. Among these components, the subjective feeling is characterised by its quality, intensity, and duration. The appraisal component is built upon a set of sequential criteria (the so-called “stimulus evaluation checks”, SECs) that influence other components and, thus, physiological changes are provoked. Diferent SECs can even influence the same mechanism, which stresses their non-linear combination. In this way, the design of an experiment capable of recording these physiological changes and the subjective feeling will help us to build a non-linear model to relate both.

The previous work of Prof. Angelika Peer (the supervisor of this project) [ 4 ] considered the emotion quality a known input of each trial, controlled by the design of the experiment. They adopted the Dynamic Field Theory (DFT) (a mathematical and conceptual framework built to model embodied cognition) [ 14 ] to model this type of dynamics. Since the DFT enables the combination and interaction of diferent DNFs, allowing the model to incorporate information from diferent physiological signals, we will also consider this approach by adding extra layers/physiological signals in the model.

Although this describes one of our first directions, other dynamical approaches as well as Hidden Markov Model and Reinforcement Learning may need to be employed in this project. Nonetheless, the design of our experiment will not be afected, since we will record enough information to include diferent types of models. These models will be trained with both physiological data and the measured subjective feeling (recorded in real-time). 2.2. Experiment and Data Collection Participants: This study will be conducted with healthy individuals, above the age of 18, who do not report any mental disorder1. According to the literature, we expect to have among 20 and 30 participants. This number will be adjusted according to the efect size. Emotion elicitation: Images will be adopted to elicit emotions due to the large consensus in the literature around their usage. More specifically, we will employ images from the International Afective Picture System (IAPS) [ 15 ], one of the most frequently cited tools to induce emotions. Electrophysiological signals: Central and peripheral nervous systems (CNS and PNS, respectively) provide information on emotional states [ 2, 8 ]. We will use information from galvanic skin response (GSR), heart rate (HR), and respiration (RSP) as indicators of the autonomic nervous system (a branch of the PNS), as well as from electroencephalogram (EEG) as an indicator of the CNS.

Experimental design: In the proposed work, we aim for an extended approach that not only considers the intensity, but also the quality of emotions as well as dynamic changes of both. The experiment is divided into 3 diferent parts, as follows:

1Before participating in the study, interested subjects will be screened with a questionnaire to assess the presence of somatization, obsessive-compulsive disorder, interpersonal sensitivity, depression, anxiety, hostility, phobic anxiety, paranoid ideation, psychoticism, and posttraumatic stress disorder. Only participants with a low prevalence of any of these conditions will be included in this study and continue with further steps • Intensity model: To improve the current intensity model [ 4 ] and the analysis of intensity profiles for diferent emotion qualities we are going to choose 3 emotion qualities, representative of the quadrants of the Geneva Emotion Wheel (GEW) [ 17 ]. For each emotion quality (which is fixed over a trial), the intensity of the stimuli changes over the trial. • Quality model: The stimuli range of intensity is assumed to be known and fixed over a trial with 3 intensity ranges are going to be considered to understand whether the intensity afects the quality transitions. • Quality and intensity model: To investigate efects of quality transitions in the felt intensity one part of the experiment is going to involve stimuli who vary in both quality and intensity over the trial.

Experimental procedure: The first step of the experiment is an online pre-screening. The participants who meet our inclusion criteria1 are invited to come to our laboratory and continue the experiment. After an explanation of the study and tasks and the obtainment of written consent, the biosensors to measure GSR, HR, RSP, and EEG is placed. The stimuli presented are part of the IAPS database [ 15 ] and are displayed on a monitor in front of the individuals. The participants provide real-time information about the quality and intensity of the emotion experienced, with access to a polar device with the GEW. The GEW contains 20 diferent labels representing 40 emotion qualities. By adjusting a knob, subjects provide us the felt quality (the angle of the GEW) and intensity (according to the proximity to the border of the GEW).

3. Preliminary Work

A pilot study was conducted with 2 participants, which allowed us to understand the need to slightly revise the experimental design. After fixing these issues, we recorded data from 2 other participants. However, just a part of the trials was recorded due to a software and a hardware problem respectively. In this sense, we were just able to conduct a preliminary analysis of the intensity model with a partial number of trials and in which we considered HR and RSP signals.

We started analysing our data with Linear Regression to serve as a baseline. We computed diferent features, namely: pulse frequency, pulse running rate 2, RSP rate (RSP cycles per minute), RSP running rate, inspiration time, expiration time, inhalation depth, and exhalation depth. We performed a sequential feature selection based on the higher computed bandwidth accuracy3 (Abdw). Preliminary results indicate that for diferent subjects, diferent feature combinations allowed to achieve the highest Abdw, indicating a high subject-dependency of the optimal solution.

Finally, we also used a Nonlinear Autoregressive Network with Exogenous Inputs (NARX neural network), a recurrent dynamic network with feedback connections enclosing several layers of the network, with raw signals as input, testing it with diferent delays. But since deep 2The running rate is computed with respect to a reference interval that is moving along with the evaluaion window as time proceeds

3Rate of samples where the model estimate matches ground truth within an acceptable margin of error, i.e., the bandwidth ( ). We considered of 10%, 15%, and 20% learning approaches require more data we need to first extend our dataset to obtain meaningful results.

References

[1] S. Brave , C. Nass , Emotion in human-computer interaction , in: Human-Computer Interaction Fundamentals, volume 20094635 , CRC Press Boca Raton, FL, USA, 2009 , pp. 53 - 68 .

[2] L. Shu , J. Xie , M. Yang , Z. Li , Z. Li , D. Liao , X. Xu , X. Yang , A review of emotion recognition using physiological signals , Sensors 18 ( 2018 ) 2074 .

[3] J. N. Bailenson , E. D. Pontikakis , I. B. Mauss , J. J. Gross , M. E. Jabon , C. A. Hutcherson , C. Nass , O. John, Real-time classification of evoked emotions using facial feature tracking and physiological responses , Int. J. Hum. Comput. Stud . 66 ( 2008 ) 303 - 317 .

[4] R. Jenke , A. Peer , A cognitive architecture for modeling emotion dynamics: Intensity estimation from physiological signals , Cognitive Systems Research 49 ( 2018 ) 128 - 141 .

[5] P. Kuppens , P. Verduyn , Emotion dynamics, Current Opinion in Psychology 17 ( 2017 ) 22 - 26 .

[6] C.-Y. Chang , C.-W. Chang , Y. -M. Lin , Application of support vector machine for emotion classification , in: 2012 6th International Conference on Genetic and Evolutionary Computing , IEEE, 2012 , pp. 249 - 252 .

[7] M. Li , H. Xu , X. Liu , S. Lu , Emotion recognition from multichannel eeg signals using k-nearest neighbor classification , Technology and Health Care 26 ( 2018 ) 509 - 519 .

[8] W. Lin , C. Li , S. Sun , Deep convolutional neural network for emotion recognition using eeg and peripheral physiological signal , in: International Conference on Image and Graphics , Springer, 2017 , pp. 385 - 394 .

[9] S. Alhagry , A. A. Fahmy , R. A. El-Khoribi , Emotion recognition based on eeg using lstm recurrent neural network , Emotion 8 ( 2017 ) 355 - 358 .

[10] A. Moors , P. C. Ellsworth , K. R. Scherer , N. H. Frijda , Appraisal theories of emotion: State of the art and future development , Emotion Review 5 ( 2013 ) 119 - 124 .

[11] K. R. Scherer , A. Schorr , T. Johnstone, Appraisal Processes in Emotion: Theory, Methods, Research, Oxford University Press, 2001 .

[12] K. R. Scherer , Emotions are emergent processes: they require a dynamic computational architecture , Philos Trans R Soc Lond B Biol Sci 364 ( 2009 ) 3459 - 3474 .

[13] T. Wehrle , K. R. Scherer , Towards computational modeling of appraisal theories , in: Appraisal Processes in Emotion: Theory, Methods, Research, New York: Oxford University Press, 2001 , pp. 350 - 365 .

[14] G. Schöner , Dynamical systems approaches to cognition, Cambridge Handbook of Computational Cognitive Modeling ( 2008 ) 101 - 126 .

[15] P. J. Lang , M. M. Bradley , B. N. Cuthbern , International afective picture system (iaps): Afective ratings of pictures and instruction manual , Technical Report A-8 ( 2008 ).

[16] J. Heylen , P. Verduyn , I. Van Mechelen , E. Ceulemans, Variability in anger intensity profiles: Structure and predictive basis , Cognition and Emotion 29 ( 2015 ) 168 - 177 .

[17] V. Sacharin , K. Schlegel , K. R. Scherer , Geneva emotion wheel rating study ( 2012 ).