- Facial expressions and speech are elements that provide emotional information about the user through multiple communication channels. In this paper, a novel multimodal emotion recognition system based on visual and auditory information processing is proposed. The proposed approach is used in real affective human robot communication in order to estimate five different emotional states (i.e., happiness, anger, fear, sadness and neutral), and it consists of two subsystems with similar structure. The first subsystem achieves a robust facial feature extraction based on consecutively applied filters to the edge image and the use of a Dynamic Bayessian Classifier. A similar classifier is used in the second subsystem, where the input is associated to a set of speech descriptors, such as speech-rate, energy and pitch. Both subsystems are finally combined in real time. The results of this multimodal approach show the robustness and accuracy of the methodology respect to single emotion recognition systems.
In the last decade, Human Robot Interaction (HRI) has
become one of the most important issues in social robotics.
Within HRI, one of the main objectives is the development
of methodologies focused on non-invasive techniques based
on natural language. This would allow the robot to interact
with users in a similar way to humans, through multimodal
systems that combine information from different channels.
In order to socially interact with humans, a robotic system
should be able not only to understand users behaviour and
intentions, but also to estimate their emotional state.
Knowing and understanding these human emotions helps social
robots adapting the communication in real time, improving
and enriching the interaction [
The design of social robots looks for developing
natural interfaces for an affective interaction. In this context,
most of the current affective HRI techniques use an unique
information channel, called mode, such as speech, facial
expressiveness or body language. However, an emotion is
generally expressed through several modalities. In
multimodal affective interaction, the user can communicate his/her
emotional state to the robot using different several input
channels [
This work presents a novel multimodal emotion
recognition system for affective HRI. The proposed approach
is based on a real-time multimodal system that integrates
speech and facial expression analysis. The main contribution
of this work is a robust feature extraction approach for facial
expression recognition. In this paper, these facial features are
a combination of independent and antagonistic distortions
of the face. Besides, a novel acoustic analysis of speech
is used to extract features relevant to emotion. Both visual
and acoustic features constitute the input of two Dynamic
Bayessian Network (DBN) [
This paper is organized as follows: after discussing known approaches to emotion recognition systems from facial expressions, speech or multimodal systems in Section II, Section III presents an overview of the proposed Multimodal Emotion Recognition System. In Section IV, the experimental results are pointed out, and finally, Section V describes the conclusions and future work of the approach.
In the field of HRI, different automatic emotion
recognition systems have been studied. Most of these approaches
are based on single information channel analysis, such as
video sequences or audio signals. Independent of the nature
of the information source, the raw data is processed and a set
of features are extracted. Then, these features are classified
into different categories, i.e., emotions. On one hand, facial
expressions have been commonly used to detect and
recognize human emotions. An interesting and updated review
was shown in [
III. MULTIMODAL EMOTION RECOGNITION SYSTEM In this section, the proposed multimodal emotion recognition system is presented. The framework consists on two subsystems running in parallel that estimate human emotions using two independent DBN. The facial expression recognition subsystem (see Fig. 1) is based on a fast and robust feature extraction approach where consecutively morphological and convolutional filters are applied to reduce the noise and the dependence against luminosity changes. After that, a Gabor filter is used for efficient edge detection. The output edge image of this filter bank is used to detect and extract scaleinvariant facial features, which will be the input variables of the DBN to estimate the user’s emotional state. In the second system, the user’s speech is analyzed in order to extract a set of independent descriptors. These descriptors are the input of a second DBN. Finally, the proposed system integrates the information associated to both methods in a third DBN, which estimates the final user emotion. The output of both subsystems and a third DBN has as a result an estimate of the emotional state of the user, within four possible emotional states (happiness, sadness, anger, fear) and a non-emotional state (neutral). Each subsystem is described in detail in the following subsections.
A. Emotion recognition from Facial Expressions
The facial expression recognition system proposed in this work uses a video sequence acquired by the robot in real time. An overview of the proposed methodology is shown in Fig. 1. Each video frame is processed and a set of robust and invariant features of the user’s face is detected. The proposed method consists on the following stages:
1) Data acquisition: Data acquisition for the recognition
of facial expressions is based on the processing of a sequence
of RGB images Sfr obtained from the robot camera for
real-time interaction. First, the system recognizes the user’s
face within each frame f r(t) in the instant of time t, using
the well-known Viola and Jones’ method [
2) Pre-processing: In this stage, the ROII is processed to
remove noise, reducing its light dependence and eliminating
unnecessary information. To eliminate errors in the detection
and reduce the processing time of the system, the ROII
is divided in two sub-regions, ROItop and ROIbottom,
respectively. The usage of both sub-regions allow the
elimination of irrelevant information (e.g., nose) and divides
the feature extraction process in two elements (eyes and
mouth). Let ROII being the face image of size W x H,
and let p0 = (n; m) being the central pixel in the image,
which estimates the approximated position of the nose in
the image. Then, ROItop and ROIbottom are defined as
selective copies of ROII as follows: ROItop of size W
x (m NT h) and ROIbottom of size W x (m + NT h)),
where NT h is an user-fixed threshold. In order to detect
and extract robust facial features in different environments,
both ROItop and ROIbottom images are processed to reduce
the effects of the light dependence. The method is based
on the approach described in [
3) Gabor Filter: The Gabor filter is a fast and effective linear filter for the detection of edges with different orientations. In the proposed approach, the Gabor filter is used as a previous stage to the detection and extraction of facial features, which are extracted using the contours of the facial elements (i.e., the eyes, the mouth or the eyebrows). Gabor impulse response in the spatial domain consists of a sinusoidal plane wave of some orientation and frequency, modulated by a two-dimensional Gaussian envelope. Let I(u; v) be the input image, then the output of the Gabor where , and are associated to the sinusoidal plane wave (orientation, wavelength and phase, respectively), and being u and v described as: u
= ucos
v =
usin
+ vsin
+ vcos
(2)
4) Feature extraction: The crucial step in an automatic
facial expressions recognition system is the extraction of
relevant features from the image, F I = f I
f i j i = 1::mg. In
the proposed work, a set of edge-based features is extracted,
which is invariant to scale or distance from the user to the
robot. Each one of the features are directly related to the
Action Units (AUs) described by the Facial Action Coding
System (FACS) [
5) Dynamic Bayesian Network classifier : In order to classify the Facial Expression (F E) produced by the user’s face, a Dynamic Bayesian network is proposed, where the overall classification result achieved is the one foreseen by the belief variable F E, in the scope (F E[neutral], F E[happiness], F E[sadness], F E[f ear], F E[anger]).
(a) (b) Fig. 3. Edge-based facial feature extraction: a) Region of interest in the face image (ROItop and ROIbottom); and b) Features extracted in the image.
This bayesian approach is based on the detection of 11
AUs with antagonistic and exclusive features as leaves of
the DBN, which allows us to reduce the size of the DBN to
7 variables that group these AUs. Thus, these 7 variables are
obtained from combinations of the extracted features of the
previous process: dA, dB and dC. In this case, it is assumed
that these 7 leaf variables are independent given the facial
expression (F E). Although some muscular movements from
one area of the face may slightly affect other areas, this
small influence could not be detected by the robot cameras.
Besides, for correct detection of each emotional state it
is necessary that each AU achieves a minimum threshold
intensity of B (i.e., slight evidence), within the intensity
range described in FACS [
Fig. 4 illustrates the structure of the two level bayesian network, and the time influence that characterizes this DBN. The first level shows the estimation of the user’s emotional state F E, while the second level shows the 7 variables based on the AUs. In addition, one of the main features worth mentioning about the bayesian network is the need to provide it with learning data. The most common method is that each new sample is matched using a threshold. In this work, to avoid the extant gaps, a pre-processing stage is done before the learning stage, fitting a Gaussian distribution to the data. The learning data acquisition was performed by a supervisor, who collected the samples of the 7 random variables manually, correctly classifying them. The leaf random variables of the model, and their respective virtual-scopes are shown in the Table I.
Table II the relationship between the basic emotional states of a user, and the Action U nits associated with different facial distortions. In this case, these AU s depend on the few deformable or mobile elements of the face, such as: the mouth, eyes and eyebrows. This feature of deformation or movement of facial elements is what allows analyze a muscular change associated with a specific emotion, otherwise there would be no facial expressions. However, elements such as the nose does not present specific changes in facial expressions.
In this approach, the data (D) in the classification process is obtained according to the following setup:
D = ((x1; y1):::(xn; yn)); xi 2 Rd; yi 2 R (3) Consider that y1 to y5 are the five possible emotional states (F E[neutral], F E[happiness], F E[sadness], F E[f ear], Emotion Sadness Happiness
F ear Anger Neutral
Action Units AUs AU1 - AU4 - AU15 -AU17
AU6 - AU12 - AU25
AU1 - AU4 - AU20 - AU25 AU4 + AU7 - AU17 - AU23 - AU24
— F E[anger]); and each dimension of x, corresponds to one of the previously described random variables, namely: EB, Ch, LE, LC, CB, M F and M A. Since the learning data may have gaps between its samples, a model is built assuming that (X1; :::; Xn) are independent given F E, and Xi
N (priorT xi; 2) At first, prior U (1=n), however throughout the iterations, the posterior of t 1 becomes the prior on t.
Finally, the posterior equation is obtained using Bayes’ rule:
P (F Ejxm) =
Q1n P (xijF E) P (F E)
P (xm) ; where xm is the most recent visual information acquired. The last dividend can be computed using the Bayesian marginalization rule:
q P (xm) = X Y P (xijF E) P (F E);
F E 1 being q = 7, the number of random variables of the system.
The dynamic properties of the network cause a convergence over time. The resultant histogram from the previous frame is used as prior knowledge for the current frame. Each classification is considered correct, if it converges in a maximum number of 5 frames exceeding a threshold of 80%. Otherwise, if after 5 frames no value is higher than the threshold, the classifier selects the highest probability value (usually referred to as the Maximum a posteriori decision in Bayesian theory) as the result. (4) (5) (6)
Emotion recognition using speech as input poses several
subproblems, such as: detection, extraction and
characterization of a set of significant acoustic features to recognize
emotions. Therefore, the proposed approach has a similar
structure of the subsystem described in Sec. III-A, thus, like
the former, a set of acoustic features is extracted and used
as input to a DBN classifier. The proposed methodology
is based on the work presented in [
1) Detection of the Speech: The audio signal is
preprocessed in order to detect the presence or absence of speech
in real time, within a communication between a human and
a robot. This stage was achieved by using audio library SoX
[
according to eq. 8 for each spectral frame.
C( ) = F 1 log(jF (x(t))j2) (7)
Where F and F 1 represent the Fourier Transform direct and reverse, respectively. The output of this process is a signal that is composed of frames that contain the information of the user’s voice,at a sampling frequency fs of 44,100 KHz. Finally, these frames are input in the function responsible for extracting acoustic features.
2) Acoustic feature extraction: From the selected spoken
sentence, a set of acoustic features, able to characterize the
user’s emotional state, are extracted. The selection of these
characteristics follows a common theme with other methods
of literature, which are related to the use of elements of
prosody, such as: [
Due the previous delivery process the audio frames, it
is possible to extract a set m of features represented by:
F A=ffiA j i = 1::mg, as described in [
In this section, the following features to extract and the respective justification of its relevance for emotion recognition are:
Pitch: also called Fundamental Frequency, is the rate of vibration of the vocal cords to produce sound. The pitch range is a feature that allows to identify not only the gender of users, but also their emotional states. Energy: distribution of the signal amplitude values in time. In an audio signal, the energy in the voice is a determining factor in the generation of emotions. Emotions with higher intensity are associated to higher energy values in the voice. In a similar way, emotions with lower intensity show lower energy values.
Speech-Rate: also called velocity or BPM (beats per minute), is diction speed or number of words in a time period. In a similar way to the Energy, the Speech-Rate is a feature associated directly to the intensity of emotions. Thus, emotions with high intensity present high Speech-Rate values, and emotions with low intensity are associated with low Speech-Rate values.
The methods to extract each of these three acoustic features are briefly described bellow:
Pitch: The pitch range is calculated using the HPS
(Harmonic Product Spectrum) algorithm [
Energy: The quantification of the signal’s energy is defined as:
Y p(!i) = max(Y (!))
E =
Speech-Rate: In order to calculate the Speech-Rate, the beats are evaluated according to the following process: i) the Fast Fourier Transform of the signal is performed; ii) the signal is multiplied with trains of pulses with different speech-rates; and iii) the amount of Energy at each rate is analyzed. Finally, the Speech-Rate is calculated as the signal with the highest Energy value.
These three features are the input of the Bayesian network that estimates the user’s emotional state. The table III shows the relation between the acoustic features and the emotional states of the user. In the table, it is seen as low-intensity emotions, such as sadness and neutral state, have similar features. Similarly, the emotions with a high intensity, such as anger or fear, show also common characteristics, but with slight differences perceptible to the user.
Emotion Sadness Happiness
Fear Anger Neutral
P itch Slightly narrower
Much wider Much wider Much wider much narrower
Energy Lower Higher Normal Higher Normal
T empo Slightly slower Faster or slower
Much faster Slightly faster
Slower
3) Dynamic Bayesian Network classifier: For the classification of the emotional state of the user from the voice, a second DBN is proposed. This second network, represented in Fig. 6, has a two-tier structure, and the scope of the possible emotional states are identical to the facial expressions DBN (mentioned in section III-A.5). The first level is a single node that represents the variable associated with the result of the classified auditory emotion (AE[N eutral; Anger; Happines; Sadness; F ear]). The second level of the network corresponds to three nodes associated with three independent variables between them, such as: P T - Pitch; EN - Energy.and T E- Speech-Rate. These 3 variables quantify the elements of the speech as: speed and the intensity of talk, and are directly related to the intensity of emotions. Moreover, each of the elements of the second level has as its father the node AE. In order to perform the estimation, the network first needs to be filled with learning information about the features in each emotional state. Henceforth, similarly to the facial expression network, Bayes’ rule is once again used for inference given learning, as:
P (AEjxa) =
Q1n P (xijAE) P (AE)
P (xa) Where xa is the most recently acquired auditory information, given by the variables in the second level (PT, EN, TE). Using the Bayesian marginalization rule, we can compute: k P (xa) = X Y P (xijAE) P (AE)
AE 1 being k = 3, the number of variables in the network. C. Multimodal fusion for emotion recognition
In this section, the system analyzes the information obtained by both modalities. Through the use of a multimodal system, it is possible to eliminate errors in the detection or classification, by checking the results with those obtained through another modality. The results of both modalities are used as input to a third bayesian network of three levels, giving rise to the estimate of the emotional state of the user, in a similar way to humans.
The last DBN classifies the results of the two previous DBN, to get the emotional state of the user in the conversation. This third DBN has a structure of three levels that meets all the nodes of the two previous networks, where the node UE is the parent of the nodes of the second level F E and AE. For the nodes in the first and second level: UE , F E and AE, there are only five possible results (Neutral, Happiness, Fear, Anger, Sadness). For the variables of the third level, 7 variables belong to the node F E that estimated the emotions through facial expressions, and 3 variables belong to the node AE that estimate the emotional states through the speech, as shown in Fig. 7.
To estimate the emotional state on the basis of these two results, the joint distribution asociated to the Bayesian Fusion (11) (12) is used, by the following equation:
j P (xc) = X Y P (xijUE ) P (UE )
UE 1 From the joint distribution, using the Bayes’ rule: P (UE jxc) =
Q1n P (xijUE ) P (UE )
P (xc) (13) (14)
Through this third bayesian network, the emotional state of the user is estimated in this multimodal system, giving rise to a robust, real-time result.
Time Control: One of the main problems of the modal
systems is related to the synchronization between the
different blocks that make up the recognizer. The detection of
a facial expression does not have to coincide in time with
a corporal expression, and much less with the emotional
information that can be extracted from the speech analysis.
For this reason, a control block of time that synchronizes the
results of each of the Bayesian networks are established, but
instead of giving the output of each subsystem an estimate
of human emotion in the same instant of time, the proposed
system uses the emotion recognizer based on the analysis of
facial expression as the dominant mode in the entire system
[
In the Fig. 8 shows the behavior in time of the proposed system. Where, the predominant system corresponds to the output of recognizer of facial expressions. Only when there is audio data results are merged into a single output of multimodal system.
In this section, a set of tests evaluates the performance of the proposed system, through the evaluation of each of the two modalities separately and together. In the two first tests, the visual and auditory modalities are tested separatedly. In the last one, the whole set of the two modalities working together is examined.
Test PF E a. Sadness b. Happiness c. Fear d. Anger e. Neutral Besides, in the Table VIII detail the errors in this test in 3 groups: M isclassif ication, Ambiguous (between two emotional states with low recognition rates.) and U nder threshold (not exceed the minimum threshold in a state).
The algorithms presented in this paper were developed
in C ++, and the benchmark tests were performed on a
computer with a 2.8 GHz Intel(R) Core(TM) i7 CPU and 4
Gb RAM running Linux. The software to control the system
is built on the top of the robotics framework RoboComp [
The information for each user group is presented in audio and video files, which are used as the main entrance of the system for each evaluation. In order to differentiate between each group and to avoid confusion, was appointed the first group as ”Database”, and the second group as ”Volunteers”. Finally, the evaluation of the system is divided into the following tests:
1) Recognition from Facial Expression: The first test checks the performance of the system for the recognition of facial expressions. This system detects and extracts features to estimate the emotional states of the user. Therefore, during the test all the visual information is provided by the two database mentioned above, through video files of users with different facial features, to verify the correct detection and classification of this system. The results related to each group of users show important differences, associated to the quantities of errors in the classification, as shown in Table IV and V. These differences between the data for each group is due to the reliability and the conditions for the acquisition of information in each case. On the one hand, the database ”SAVEE” provides best information in a controlled scenario, with high-quality images, trained users and a controlled lighting (without natural light). On the other hand, the information gained from the volunteers is related to non-trained users (mainly students and adults), in a noncontrolled environment with natural light and ambient noise.
TABLE V RECOGNITION FROM FACIAL EXPRESSION (BASED ON THE DATABASE
SAV EE) 2) Recognition from speech: The second test checks the robustness of the emotion recognition system from speech, analyzing the sentences in the audio signal to estimate the emotional states of the user. For this second test all the auditory information from users was provided by the two database mentioned above, to analyze and verify the performance of the system. The results of this second test are seen in the Table VI and VII, where the emotions with less intensity showed better results in the recognition from speech. However, the results of the recognition of emotions based on speech presents a lower performance in comparison with the recognition of emotions based on facial expressions. Since the auditory information can have factors that affect the final results, such as: dialect (cultural), personality (psychological), nervousness (psychological), among others. These factors have consequences that can be analyzed in the Table VI and VII, which despite having a number of differences in the methods of recording and the quality of the data acquired by the volunteers of this experiment and the database ”SAVEE”, exhibit similar results. The details of the errors are illustrated in the Table VIII.
3) Errors in the tests: The main errors described in Table VIII, are associated with the classification and estimation of the emotional states of the user through the auditory information of both groups. These failures in the classification is due to the inconsistency of the data that cause ambiguities between several states and errors of the classifier, called ”misclassification” . In relation to the visual information, the facial expressions show more errors in the classification process as ”misclassification” and ”under the threshold”, due to certain unusual behavior of the user or facial distortions that are not associated with a specific emotion.
4) Emotion recognition from facial expressions and speech: Finally, to demonstrate the benefits of the proposed multimodal system, a third test was performed to verify the multimodal system based on the bayesian network, which analyzes the information from the two methods described above. The last system allows for the correction of detection errors and reduces the uncertainty in the classification. The results of the last test are seen in the Table IX.
Table IX shows the differences between the results of the two databases. Where, the advantage of the database (SAV EE) are based on good performance in the predominant system. However, the benefits of this type of systems are clear. Given that removes a range of problems associated with errors of classification, ambiguity, among others.
In this paper, an emotion recognition system through a multimodal approach is presented. This system recognizes emotional states through two modalities, the visual and the auditory. The visual modality uses observed information to recognize facial expressions. It is based on the use of contours and Gabor filters. This allows the recognition of patterns in the user’s face, even with interference by the light and a wide range of different users. The second modality, auditory, recognizes emotion related items from the audio, using a sentence from a conversation to determine the emotional state of the user, getting emotional information in a similar way as a human would. The results show the improvements of the multimodal approach against those systems based on a single modality. Multimodal solutions show a significant reduction of detection and classification errors.
Future work will focus on using more precise information, through the use of new features based on the spectrum and a microphone array. This would improve the information
Errors Volunteers (PF E ) Volunteers (PAE ) Database (PF E ) Database (PAE )
Misclassification
Ambiguous
Under threshold obtained by the system, allowing the use of other types of information in the multimodal system as: body language, among other.
This work has been partially supported by the MICINN Project TIN2012-38079-C03-01, and by the Institute of Electrical and Electronics of the Universidad Austral de Chile.