Afective computing agenda is to enhance the quality of human computer interaction by making it more enjoyable by automatically recognizing and adapting to the user's afective states. More especially, it has a particular interest in the field of health such as providing emotional empathy for people living with autism. Therefore, there is need to develop methodologies for assessing user's emotional experiences. In this context, among a vast range of possible ways to access a user's emotional responses, physiological measures have a key advantage as they grant an access to non-conscious and non-reportable processes. However, to map physiological patterns from sensors to user emotional states remains a dificult task. To begin with, physiological signals tend to vary from participant to participant and even within the same participant physiological signals vary from time to time. The current methods tend to rely on some forms of normalization using some baseline yet, the correlation between the baseline and the various emotions also vary from person to person and at diferent occasions for the same person. In this study, we propose a model based on gradual rules to characterize afective states of the form: the more or less of A, the more or less of B. Specifically, we consider the physiological signals variation with time during a particular afective state, such as: Heart Rate increases with time during Joy more than 60% of the time or Heart Rate increases with time during Disgust less than 40% of the time. Secondly, emotions are conceptual quantities with indeterminate fuzzy boundaries. Besides, the physiological data from sensors is itself imperfect, such that it is dificult to express the results in crisp terms. Therefore, it is more natural to formulate a fuzzy set theory based model to represent these continuous transitions, uncertainties and imperfections. In this study, we consider a fuzzy approach to map physiological patterns to afective states. After we extract the support for each gradual item set, we define fuzzy rules to characterize the various emotions.
Afective computing has become a major research interest in the Human Computer
Interaction (HCI) community. Hence, there is a need to develop methodologies for assessing user’s
emotional experiences while interacting with these computer applications. In this context,
physiology-based emotionally intelligent paradigms provide an opportunity to enhance
human computer interactions by continuously evoking and adapting to the user experiences in
real-time [
To begin with, physiological signals tend to vary from participant to participant and even
within the same participant physiological signals vary from time to time. The current methods
tend to rely on some forms of normalization using some baseline to tackle this variability.
However, the correlation between the baseline and the various emotions also vary from person
to person and at diferent occasions for the same person. Thus, these modelling approaches can
not lead to a generalized mapping of afective states to physiological signals irrespective of the
person and time. In this study, a way of extracting features that are independent of person and
time of expression of the emotion, we consider a model based on gradual rules of the form: the
more or less of A, the more or less of B [
Secondly,emotions are conceptual quantities with indeterminate fuzzy boundaries [11].
Therefore, it is necessary to express in fuzzy terms the mapping of afective markers from physiological
data. In the context of continuously assessing emotions from physiological signals, change from
one emotional state to the next is gradual rather than abrupt. Besides, the physiological data
from sensors is itself imperfect, such that it is dificult to express the results in crisp terms [ 12].
Therefore, it is more natural to formulate a fuzzy set theory based model to represent these
continuous transitions, uncertainties and imperfections. In a fuzzy set theory based model [13],
changes from one rule to another is gradual with fuzzy values [
Studies in psychology have proved that certain psychological processes and states are accompanied by changes in physiological activity [14, 15, 16]. For example, Winton et al. [15]’s study showed that pleasant and unpleasant emotions could be diferentiated through heart rate (HR). Pleasant reaction was found to be followed by heart rate increase while unpleasant slides were characterized by heart rate deceleration.
Subsequently, in afective computing, experimental studies have been conducted to propose
the use of such inferences as a way to develop machines that can automatically recognize and
respond to these emotions [
Modeling afective states through physiology has mainly been done through classification
machine learning methods such as k-nearest neighbors algorithm, discriminant analysis, support
vector machines, bayesian networks and decision trees [
These methods use features from the physiological signals during the period the emotion
was expressed for each signal such as the average , maximum, minimum, standard deviation,
power spectrum density . . . [
The disadvantage with this approach is that it relies on only two values, minimum and maximum values. First, these two values may be outliers or suspectable to noise. They may not be a representative/typical of signal values. Secondly, its dificult to use this in real-time system as they have to be done post the emotional experience i.e, the normalization is in comparisons or an emotion vs other emotion values.
In this study, we consider a model based on gradual rules of the form: the more or less of A,
the more or less of B [
Definition 3.1 (Dataset). Let the data set consist of transactions: 1, · · · , · · · , characterised by attributes: 1, · · · , , · · · , .
1 2 · · · 1 1(1) 2(1) · · · ...2 1(...2) 2(...2) · · · ... ... 1(...) 2(...) · · · ... ... 1(...) 2(...) · · · ... 1() 2() · · ·
· · · (1) · · · (...2) · · · ... (...) · · · ... (...) · · · ... () · · ·
· · · (1) · · · (...2) · · · ... (...) · · · ... (...) · · · ... () · · ·
(1) (...2) .()
.. .()
..
() Definition 3.2 (Gradual item). A gradual item is a pair made of an attribute and a variation denoted by increase or decrease: ≥ or ≤ Definition 3.3 (Gradual itemset). A gradual itemset, is a combination of two or more gradual items i.e a conjunction of two or more gradual items.
For example a gradual itemset can be defined by the gradual items the more , the more as follows: = ≥ ≥ = ≥ ≥ ≥ Definition 3.4 (Length of gradual item-set). The length of gradual item-set, is the number of gradual items in a gradual item-set.
For example a gradual itemset can be defined below has a length of 3: Definition 3.5 (Support). The total number of row pairs in the database that comply with a given item set divided by the maximum possible pairs in the database.
The maximum number of pairs is given by: ︀( )︀ = !(−! )! = 2(−!2)! = (2−(1)−(2)−! 2)! = (2− 1)
If the number of pairs of rows that comply with a given gradual itemset, is , then the support is given by: = (− 1)
2
In the current study, we consider extraction of attributes that represent the gradual rules. Instead of looking as individual features such as EDA and ECG separately;
STEP 2: For each attribute, define a gradual item, a pair made of an attribute and a variation denoted by increase or decrease: ≥ .
STEP 3: For each attribute, define gradual itemset, as a conjunction of ≥ and ≥ i.e = ≥ ⋀︀ ≥ STEP 4: Compute the support, and use it as the input for physiological characterization.
In this study, we have proposed a generic model for characterizing afective states through physiology. First, we address the issue of invariability between person to person due to the nature of any bio-signal. In this regard, we proposed gradual rules based model of the form: the more or less of A, the more or less of B. Secondly, we have given direction to the most appropriate machine learning framework to handle the uncertainties and imperfections of online data captured by bio-sensors in real-time. In the characterization task, we considered contraction of fuzzy rules. As this work is a proposal, as a next step, we would like to test the model on real data and improve on its formulation. [10] D. O. Owuor, A. Laurent, J. O. Orero, Mining fuzzy temporal gradual emerging patterns, International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems. 29 (2021) 655–676. [11] R. A. Calvo, S. D’Mello, Afect detection: an interdisciplinary review of models, methods, and their applications, IEEE Transactions on Afective Computing 1 (2010) 18 – 37. [12] B. Bouchon-Meunier, Aggregation and Fusion of Imperfect Information, Physica-Verlag,
Spring-Verlag Company, 1998. [13] L. Zadeh, Fuzzy sets, Information Control 8 (1965) 338–358. [14] P. Ekman, R. Levenson, W. Friesen, Autonomic nervous system activity distinguishes among emotions, Science 221 (1983) 1208–1210. [15] W. Winton, L. Putnam, R. Krauss, Facial and autonomic manifestations of the dimensional structure of emotion., Journal of Experimental Social Psychology 20 (1984) 195–216. [16] P. Lang, The emotion probe studies of motivation and attention, American Psychologist 50 (1995) 372–385. [17] A. Haag, S. Goronzy, P. Schaich, J. Williams, Emotion recognition using bio-sensors: First stepts towards an automatic system, Lecture Notes in Computer Science, Afective Dialogue Systems 3068 (2004) 36–48. [18] J. Wagner, J. Kim, E. André, From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification, in: IEEE International Conference in Multimedia and Expo, 2005, pp. 940–943. [19] P. Rainville, A. Bechara, N. Naqvi, A. R. Damasio, Basic emotions are associated with distinct patterns of cardiorespiratory activity, International Journal of Psychophysiology 61 (2006) 5–18. [20] J. N. Bailenson, E. D. Pontikakis, I. B. Mauss, J. J. Gross, M. E. Jabon, C. A. Hutcherson, C. Nass, O. John, Real-time classification of evoked emotions using facial feature tracking and physiological responses, International Journal of Human-Computer Interaction 6 (2008) 303–317. [21] R. L. Mandryk, M. Atkins, A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies, International Journal of HumanComputer Studies 65 (2007) 329–347.