Fuzzy gradual rules model for assessing emotions through physiological signals Joseph Onderi ORERO1 1 School of Computing & Engineering Sciences (SCES), Strathmore University, Kenya. Abstract Affective computing agenda is to enhance the quality of human computer interaction by making it more enjoyable by automatically recognizing and adapting to the user’s affective states. More especially, it has a particular interest in the field of health such as providing emotional empathy for people living with autism. Therefore, there is need to develop methodologies for assessing user’s emotional experiences. In this context, among a vast range of possible ways to access a user’s emotional responses, physiological measures have a key advantage as they grant an access to non-conscious and non-reportable processes. However, to map physiological patterns from sensors to user emotional states remains a difficult task. To begin with, physiological signals tend to vary from participant to participant and even within the same participant physiological signals vary from time to time. The current methods tend to rely on some forms of normalization using some baseline yet, the correlation between the baseline and the various emotions also vary from person to person and at different occasions for the same person. In this study, we propose a model based on gradual rules to characterize affective states of the form: the more or less of A, the more or less of B. Specifically, we consider the physiological signals variation with time during a particular affective state, such as: Heart Rate increases with time during Joy more than 60% of the time or Heart Rate increases with time during Disgust less than 40% of the time. Secondly, emotions are conceptual quantities with indeterminate fuzzy boundaries. Besides, the physiological data from sensors is itself imperfect, such that it is difficult to express the results in crisp terms. Therefore, it is more natural to formulate a fuzzy set theory based model to represent these continuous transitions, uncertainties and imperfections. In this study, we consider a fuzzy approach to map physiological patterns to affective states. After we extract the support for each gradual item set, we define fuzzy rules to characterize the various emotions. Keywords Fuzzy sets, gradual rules, affective computing, physiological signals, machine learning 1. Introduction Affective computing has become a major research interest in the Human Computer Interac- tion (HCI) community. Hence, there is a need to develop methodologies for assessing user’s emotional experiences while interacting with these computer applications. In this context, physiology-based emotionally intelligent paradigms provide an opportunity to enhance hu- man computer interactions by continuously evoking and adapting to the user experiences in OLUD 2022: First Workshop on Online Learning from Uncertain Data Streams, July 18, 2022, Padua, Italy. * Corresponding author: Joseph Onderi ORERO. $ jorero@strathmore.edu (J. O. ORERO)  0000-0001-6115-1329 (J. O. ORERO) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) real-time [1]. Research in this area has demonstrated the enormous prospects in developing systems equipped with the ability to assess user emotional states using various aggregation of physiological signal absolute value such as mean, minimum or maximum, power spectrum density . . . and classical machine learning such as K-Nearest Neighbor, Linear Discriminant Analysis , Artificial Neural Networks, Decision Trees. . . [2, 3, 4, 5, 6]. Nevertheless, there is a need to develop more adequate models to represent the mapping of physiological patterns to users’ affective states for real-life emotionally intelligent applications. To begin with, physiological signals tend to vary from participant to participant and even within the same participant physiological signals vary from time to time. The current methods tend to rely on some forms of normalization using some baseline to tackle this variability. However, the correlation between the baseline and the various emotions also vary from person to person and at different occasions for the same person. Thus, these modelling approaches can not lead to a generalized mapping of affective states to physiological signals irrespective of the person and time. In this study, a way of extracting features that are independent of person and time of expression of the emotion, we consider a model based on gradual rules of the form: the more or less of A, the more or less of B [7, 8, 9, 10]. Specifically, we consider the physiological signals variation with time during a particular affective state, such as: Heart Rate increases with time during Joy more than 60% of the time or Heart Rate increases with time during Disgust less than 40% of the time. Secondly,emotions are conceptual quantities with indeterminate fuzzy boundaries [11]. There- fore, it is necessary to express in fuzzy terms the mapping of affective markers from physiological data. In the context of continuously assessing emotions from physiological signals, change from one emotional state to the next is gradual rather than abrupt. Besides, the physiological data from sensors is itself imperfect, such that it is difficult to express the results in crisp terms [12]. Therefore, it is more natural to formulate a fuzzy set theory based model to represent these continuous transitions, uncertainties and imperfections. In a fuzzy set theory based model [13], changes from one rule to another is gradual with fuzzy values [0, 1] instead of crisp values {0, 1} in classical machine learning approaches. In this study, we consider contraction of fuzzy rules based model. After we extract the support for each gradual item set, we define fuzzy rules to characterize the various emotions. 2. Related works 2.1. Emotions and physiology Studies in psychology have proved that certain psychological processes and states are accompa- nied by changes in physiological activity [14, 15, 16]. For example, Winton et al. [15]’s study showed that pleasant and unpleasant emotions could be differentiated through heart rate (HR). Pleasant reaction was found to be followed by heart rate increase while unpleasant slides were characterized by heart rate deceleration. Subsequently, in affective computing, experimental studies have been conducted to propose the use of such inferences as a way to develop machines that can automatically recognize and respond to these emotions [2, 17, 18, 19, 4, 20, 5]. In particular, [2]’s study is well known in this domain. Their experimental study was aimed at discriminating eight emotions (anger, hate, grief, platonic love, joy, love and no emotion) through physiological measures recorded on a trained actor who was asked to express repeatedly these states over several days. Besides the results, one of the most striking revelation of their experiment was the complexity associated to the variability of physiological measures. Despite using the same participant for all the experiments, they observed a significant day-to-day variations. The physiological patterns associated to different emotions on the same day had the tendency to cluster together more tightly than physiological patterns associated to the same emotion on different days. Therefore, part of the aim of this work, is to present a possibility of determining viability of developing generic systems that could be applied independent of the user. 2.2. Methods of characterizing affective states Modeling affective states through physiology has mainly been done through classification machine learning methods such as k-nearest neighbors algorithm, discriminant analysis, support vector machines, bayesian networks and decision trees [2, 18, 3, 4, 5]. These methods use features from the physiological signals during the period the emotion was expressed for each signal such as the average , maximum, minimum, standard deviation, power spectrum density . . . [6]. Absolute values of the signal tend to vary significantly from person to person and therefore they do some form of normalization or use of baseline to make the values comparable for emotion recognition. For example, given values 𝑣1 < 𝑣2 < 𝑣3, the same participant may have a value of 𝑣1 and 𝑣2 for 𝑒𝑚𝑜𝑡𝑖𝑜𝑛1 and 𝑒𝑚𝑜𝑡𝑖𝑜𝑛2 respectively on a particular day but same person may have value of 𝑣2 for 𝑒𝑚𝑜𝑡𝑖𝑜𝑛1 and 𝑣3 for 𝑒𝑚𝑜𝑡𝑖𝑜𝑛2 on a different day. The most widely used normalization is by min max so as to have values between 0 and 1 [0,1] [21]. The disadvantage with this approach is that it relies on only two values, minimum and maximum values. First, these two values may be outliers or suspectable to noise. They may not be a representative/typical of signal values. Secondly, its difficult to use this in real-time system as they have to be done post the emotional experience i.e, the normalization is in comparisons or an emotion vs other emotion values. In this study, we consider a model based on gradual rules of the form: the more or less of A, the more or less of B [7, 8, 9]. The covariation of attributes such as when 𝐴𝑡𝑡𝑟𝑖𝑏𝑢𝑡𝑒1 increases, 𝐴𝑡𝑡𝑟𝑖𝑏𝑢𝑡𝑒2 also increases. It does not matter the absolute value of how much it increased as each person increase level tends to be different. 3. Our approach 3.1. Gradual patterns Definition 3.1 (Dataset). Let the data set 𝒟 consist of 𝑛 transactions: 𝒳1 , · · · , 𝒳𝑘 · · · , 𝒳𝑛 characterised by 𝑚 attributes: 𝒜1 , · · · , 𝒜𝑝 , · · · , 𝒜𝑚 . 𝒜1 𝒜2 ··· 𝒜𝑝 ··· 𝒜𝑞 ··· 𝒜𝑚 𝒳1 𝒜1 (𝑥1 ) 𝒜2 (𝑥1 ) · · · 𝒜𝑝 (𝑥1 ) · · · 𝒜𝑞 (𝑥1 ) · · · 𝒜𝑚 (𝑥1 ) 𝒳2 𝒜1 (𝑥2 ) 𝒜2 (𝑥2 ) · · · 𝒜𝑝 (𝑥2 ) · · · 𝒜𝑞 (𝑥2 ) · · · 𝒜𝑚 (𝑥2 ) .. .. .. .. .. .. .. .. .. . . . . . . . . . 𝒳𝑖 𝒜1 (𝑥𝑖 ) 𝒜2 (𝑥𝑖 ) · · · 𝒜𝑝 (𝑥𝑖 ) · · · 𝒜𝑞 (𝑥𝑖 ) · · · 𝒜𝑚 (𝑥𝑖 ) .. .. .. .. .. .. .. .. .. . . . . . . . . . 𝒳𝑗 𝒜1 (𝑥𝑗 ) 𝒜2 (𝑥𝑗 ) · · · 𝒜𝑝 (𝑥𝑗 ) ··· 𝒜𝑞 (𝑥𝑗 ) ··· 𝒜𝑚 (𝑥𝑗 ) .. .. .. .. .. .. .. .. .. . . . . . . . . . 𝒳𝑛 𝒜1 (𝑥𝑛 ) 𝒜2 (𝑥𝑛 ) · · · 𝒜𝑝 (𝑥𝑛 ) · · · 𝒜𝑞 (𝑥𝑛 ) · · · 𝒜𝑚 (𝑥𝑛 ) Definition 3.2 (Gradual item). A gradual item is a pair made of an attribute and a variation denoted by increase or decrease: 𝒜𝑝 ≥ or 𝒜𝑝 ≤ Definition 3.3 (Gradual itemset). A gradual itemset, 𝒮 is a combination of two or more gradual items i.e a conjunction of two or more gradual items. For example a gradual itemset 𝒮 can be defined by the gradual items the more 𝒜𝑝 , the more 𝒜𝑞 as follows: 𝒮 = 𝒜𝑝 ≥ 𝒜𝑞 ≥ Definition 3.4 (Length of gradual item-set). The length of gradual item-set, is the number of gradual items in a gradual item-set. For example a gradual itemset 𝒮 can be defined below has a length of 3: 𝒮 = 𝒜𝑝 ≥ 𝒜𝑞 ≥ 𝒜𝑟 ≥ Definition 3.5 (Support). The total number of row pairs in the database that comply with a given item set divided by the maximum possible pairs in the database. The maximum number of pairs is given by: 𝑛(𝑛−1)(𝑛−2)! = 𝑛(𝑛−1) (︀𝑛)︀ 𝑛! 𝑛! 𝑟 = 𝑟!(𝑛−𝑟)! = 2(𝑛−2)! = 2(𝑛−2)! 2 If the number of pairs of rows that comply with a given gradual itemset, is 𝑧, then the support is given by: 𝑠 = 𝑛(𝑛−1)𝑧 2 3.2. Gradual patterns in physiological computing In the current study, we consider extraction of attributes that represent the gradual rules. Instead of looking as individual features such as EDA and ECG separately; STEP 1: Define a gradual item, the more the time, 𝒯 ≥. STEP 2: For each attribute, define a gradual item, a pair made of an attribute and a variation denoted by increase or decrease: 𝒜𝑝 ≥. STEP 3: For each attribute, ⋀︀ define gradual itemset, 𝒜𝑡𝑝 as a conjunction of 𝒯 ≥ and 𝒜𝑝 ≥ i.e 𝒜𝑡𝑝 = 𝒯 ≥ 𝒜𝑝 ≥ STEP 4: Compute the support, 𝒜𝑡𝑝 and use it as the input for physiological characterization. STEP 5: Construct fuzzy rules based on 𝒜𝑡𝑝 . 4. Summary In this study, we have proposed a generic model for characterizing affective states through physiology. First, we address the issue of invariability between person to person due to the nature of any bio-signal. In this regard, we proposed gradual rules based model of the form: the more or less of A, the more or less of B. Secondly, we have given direction to the most appropriate machine learning framework to handle the uncertainties and imperfections of online data captured by bio-sensors in real-time. In the characterization task, we considered contraction of fuzzy rules. As this work is a proposal, as a next step, we would like to test the model on real data and improve on its formulation. References [1] D. Novak, M. Mihelj, M. Munih, A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing, Interacting with Computers 24 (2012) 154–172. [2] R. Picard, E. Vyzas, J. Healey, Toward machine emotional intelligence: Analysis of affective physiological state, IEEE Transactions Pattern Analysis and Machine Intelligence 23 (2001) 1175–1191. [3] P. Rani, N. Sarkar, J. Adams, Anxiety-based affective communication for implicit human machine interaction, Advanced Engineering Informatics 21 (2007) 323–334. [4] J. Kim, E. André, Emotion recognition based on physiological changes in music listening, IEEE Transactions on Pattern Analysis And Machine Intelligence 30 (2008) 2067–2083. [5] G. Chanel, J. Kierkels, M. Soleymani, T. Pun, Short-term emotion assessment in a recall paradigm, International Journal of Human-Computer Studies 67 (2009) 607–627. [6] F. Levillain, J. O. Orero, M. Rifqi, B. Bouchon-Meunier, Characterizing player’s experience from physiological signals using fuzzy decision trees, in: Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games, CIG 2010, Copenhagen, Denmark, 18-21 August, 2010, IEEE, 2010. [7] A. Laurent, M.-J. Lesot, M. Rifqi, Graank: Exploiting rank correlations for extracting gradual itemsets, in: Eighth International Conference on Flexible Query Answering Systems (FQAS 2009), Roskilde, Denmark, 2009. [8] L. Di-Jorio, A. Laurent, M. Teisseire, Mining frequent gradual itemsets from large databases, in: Advances in Intelligent Data Analysis VIII, 8th International Symposium on Intelligent Data Analysis, IDA 2009, Lyon, France, August 31 - September 2, 2009., 2009. [9] C. Fiot, F. Masseglia, A. Laurent, M. Teisseire, Evolution patterns and gradual trends, International Journal of Intelligent Systems 24 (2009) 1013–1038. [10] D. O. Owuor, A. Laurent, J. O. Orero, Mining fuzzy temporal gradual emerging patterns, International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems. 29 (2021) 655–676. [11] R. A. Calvo, S. D’Mello, Affect detection: an interdisciplinary review of models, methods, and their applications, IEEE Transactions on Affective Computing 1 (2010) 18 – 37. [12] B. Bouchon-Meunier, Aggregation and Fusion of Imperfect Information, Physica-Verlag, Spring-Verlag Company, 1998. [13] L. Zadeh, Fuzzy sets, Information Control 8 (1965) 338–358. [14] P. Ekman, R. Levenson, W. Friesen, Autonomic nervous system activity distinguishes among emotions, Science 221 (1983) 1208–1210. [15] W. Winton, L. Putnam, R. Krauss, Facial and autonomic manifestations of the dimensional structure of emotion., Journal of Experimental Social Psychology 20 (1984) 195–216. [16] P. Lang, The emotion probe studies of motivation and attention, American Psychologist 50 (1995) 372–385. [17] A. Haag, S. Goronzy, P. Schaich, J. Williams, Emotion recognition using bio-sensors: First stepts towards an automatic system, Lecture Notes in Computer Science, Affective Dialogue Systems 3068 (2004) 36–48. [18] J. Wagner, J. Kim, E. André, From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification, in: IEEE International Conference in Multimedia and Expo, 2005, pp. 940–943. [19] P. Rainville, A. Bechara, N. Naqvi, A. R. Damasio, Basic emotions are associated with distinct patterns of cardiorespiratory activity, International Journal of Psychophysiology 61 (2006) 5–18. [20] J. N. Bailenson, E. D. Pontikakis, I. B. Mauss, J. J. Gross, M. E. Jabon, C. A. Hutcherson, C. Nass, O. John, Real-time classification of evoked emotions using facial feature tracking and physiological responses, International Journal of Human-Computer Interaction 6 (2008) 303–317. [21] R. L. Mandryk, M. Atkins, A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies, International Journal of Human- Computer Studies 65 (2007) 329–347.