In Ambient Assisted Living environments assistance and care are delegated to the intelligence embedded in the environment that, in our opinion, should provide not only a task-oriented support but also an interface able to establish a social empathic relation with the user. To this aim social assistive robots are being employed as a mediator interface and, in order to achieve a relation with the user, they should be endowed with the capability of recognizing the user affective state. Since a natural way to interact with a robot is speech, spoken user's input can be used to give to the robot the capability of recognizing the emotions and attitude of the user, thus providing more detail information about the user state. This paper focuses on this topic and proposes an approach based on the dimensional model of emotions in which the valence and arousal of user's spoken input are recognized. The experimental analysis shows the performance in terms of accuracy of the proposed approach on an Italian dataset. In order to show its application in the context of Ambient Assisted Living, an example is provided.
facilitating users when interacting with environment services in a natural and easy
way. The required assistance may be provided to the user through different devices.
The choice of an assistive robot agent as an interface is supported by several
considerations. First of all, the robot has a physical presence and it may participate in the
user’s daily life. Assistive robots can move around and perform actions, follow and
observe the user in the environment, which is fundamental when designing supportive
technologies for elderly people [
Speech is a natural way for humans to interact with robots [
for the recognition of the emotion. This module is able to extract the prosodic features
of user’s spoken input and, starting from them is able to recognize the two dimension
of emotions: valence and arousal [
background for this work are presented. Section 3 briefly describes the MAS architecture implementing the smart home environment. Section 4 describes how VOCE has been developed and Section 5 shows an example on how it can be applied in the context of AAL. Conclusions and future work directions are illustrated in Section 6. 2
Interaction with services provided by a smart environment may be provided to the user in a seamless way (i.e. by combining smart home technologies based on sensors and effectors embedded in the appliances of the environment), or using an embodied companion as an interface, or combining both approaches. In all cases, research emphasizes the need of natural and user-friendly interfaces for accessing the services provided by the environment. Moreover, research on social and affective computing suggests that such an assistive environment should provide not only a task-oriented support but also an interface able to establish a social empathic relation with the user.
Several studies report successful results on how social assistive robots can be
employed as interface in the assisted living domain. For instance, projects ROBOCARE
[
expressions [
Recognizing emotions in speech through several features has been a key research
issue in robotics, because by recognizing emotional factors the robot can handle social
situations. In emotional classification from speech, a multitude of different features
denoting prosodic cues have been used. Prosodic features, like pitch, loudness,
speaking rate, durations, pause and rhythm were proven to have strong correlations between
them, providing emotional information. In the case of the analysis of an entire
segment of voice, statistical functions like mean, median, minimum, maximum, standard
deviation are applied to the fundamental frequency (F0) base contour [
disadvantages in order to deal with the speech emotion recognition problem. The
more common group includes Hidden Markov Models (HMM) [
The majority of emotion recognition systems from speech have employed a highdimensional speech grouped in a big vector of features. In this paper, the most commonly used features in several researches for capturing emotional speech characteristics in time and frequency were selected. The performance of different well known classifiers was compared in order to select the best result to predict the emotion, based on speech emotional data. 3.
personalized service, accessible through several interfaces available on various devices present in the environment in the optic of pervasive computing.
The objective of the proposed approach is to recognize the users goal starting from percepts (sensors data, user actions, etc.) and provide them with a smart service that integrate elementary services according to the situation. In order to achieve this aim, the environment has to be able to reason on the situation of the user so as to understand which are his/her needs and goals through the composition of the most appropriate smart service. The idea underlying our approach is the metaphor of the butler in grand houses, who can be seen as an household affairs manager with duties of a personal assistant, able to organize the housestaff in order to satisfy the needs of the house inhabitants. To this aim, taking into account the results of a previous project, we have developed a MAS in which the butler agent has to recognize the situation of the user, based on interaction with Sensor Agent, in order to infer possible user’s goals.
of available candidates representing a smart service. Such a selection is made by matching semantically the goals, the current situation features and the effects expected by the execution of the workflow. Once a workflow has been selected, its actions are executed by the effector agents.
One important feature of this architecture is the presence of an agent designed to take care of the interaction with the users. In completely proactive approach, in fact, users may feel a loss of control over the system actions. Therefore we adopt a semiautomatic approach composition of services. The butler proactively propose smart services and leaves, at the same time, the control over proposed service composition to the user to select alternative services, to provide more preference information in order to get a better personalization, to ask for explanation about the proposed services and so on.
The MAS is constituited by the following classes of agents: - Sensor Agents (SA) are used for providing information about context parameters and features (e.g., temperature, light level, humidity, etc.) at a higher abstraction level than sensor data. - Butler Agent (BA) reasons on the user’s goals and devises the workflow to satisfy them (see Figure 1). - Effector Agents (EA) each appliance and device is controlled by an EA that reasons on the opportunity of performing an action instead of another in the current context. - Interactor Agent (IA) is in charge of handling interaction with the user in order to carry on communicative tasks. In this case the IA is embodied in the NAO Robot. - Housekeeper Agent (HA) acts as a facilitator since it knows all the agents that are active in the house and also the goal they are able to fulfill.
Fig. 1. The MAS architecture
Emotions can be classified using two main approaches. Discrete emotion models
focus on a defined set of labels denoting emotions (e.g. anger, fear, disgust, happiness,
surprise and sadness to name the most common ones). The discrete emotion model
has the advantage of clearly distinguishing categories of emotions, however the the
labels and their number differ a lot from one model to another. By contrast,
dimensional models describe the affective space within a limited amount of dimensions. For
instance in the circumplex model of emotions [
fore, the analysis of the prosody user’s spoken utterance is made by two classifiers: one for recognizing the valence dimension and the other for the arousal one.
approach very similar to the one described in [26]. The major steps in speech emotion recognition are audio segmentation, feature extraction and the actual classification of the feature vectors into valence and arousal values.
VOCE can be used in two ways: offline for creating and analysing the emotional speech corpus (Figure 2) and, being a web-service, online for tracking the affect in voice in real-time. While the off-line version allows to build the classifier, the online emotion recognition just outputs the recognised emotions valence and arousal and maps the combination of these values into one of the basic emotions by providing the emotion label.
Let’s see now how the classifier has been trained and how we use it in real-time. 3.1
Albeit our approach is based on the dimensional model, since we could not found any corpus, among the few available for Italian, in which emotions were annotated according to their valence and arousal we used the ∈motion dataset [27]. Among the three availble ones the ∈motion dataset has been used for EVALITA ERT challenge1 and therefore we could compare our results with other research works in this domain.
The emotional speech characteristics were extracted from the Italian subset of
∈motion contains 220 audio files corresponding to sentences for the 6 basic emotions
(joy, anger, surprise, sadness, disgust, fear) and the neutral one recorded by
professional actors. In order to use the dimensional approach on this dataset we mapped
each emotion to the correspondent valence (negative, neutral and positive) and
arousal (low, medium, high) using the approach explained in [
classification algorithms. For this task we used Praat [28]. In particular, besides pitch and energy related features, we extracted features related to the spectrum, harmonicity and the Mel-Frequency Cepstral Coefficients (MFCCs) that is used to describe a spectrum frame, its first and second derivative in time are used to reflect dynamic changes.
Table 1 shows the features extracted from each audio file using Praat functions.
In order to find the best set of features we tested three conditions with several classification algorithms: - Support Vector Machines (SVM), which offers robust classification to a very large number of variables and small samples. - Decision trees, that work with simple classification rules that are easy to understand. The rules represent the information in a tree based on a set of features. - Artificial Neural Network (ANN), and in particular the Multilayer Perceptron algorithm. - k-Nearest Neighbors (kNN) is one of the simplest of classification algorithms available for supervised learning. It classifies unlabeled examples based on their similarity with examples in the training set.
- ALL: all the attributes in Table 1; - MFCC: MFCC features only; - No MFCC: all the features except MFCC.
Feature Pitch PitchMin PitchMed PitchMax PitchMinMaxDiffLog PitchMinLog PitchMedLog PitchMaxLog PitchDevSta PitchSlope Energy EnergyMin EnergyMed EnergyMax EnergyMinMaxDiff EnergyDevSta Spectrum SpectrumCentralMoment SpectrumDevSta SpectrumGravityCentre SpectrumKurtosis SpectrumSkewness Harmonicity HarmonicityMin HarmonicityMed HarmonicityMax HarmonicityDevSta MFCC MFCCnMin MFCCnMed MFCCnMax MFCCnDevSta Description Minimum value Average value Maximum value Logarithmic differentiation Minimum Logarithmic Average logarithmic Maximum logarithmic Standard Deviation Slope Minimum value Average value Maximum value Logarithmic differentiation Standard Deviation Central moment Standard Deviation Central tendency Degree of centralization Degree of asymmetry Minimum value Average value Maximum value Standard Deviation Minimum of nth MFCC Average of nth MFCC Maximum of nth MFCC Standard Deviation of nth MFCC
classification starting from numeric features the most accurate one were MLP (Multi
ning a support vector classifier in Weka2.
cross-validation with k = 10 was used to make validations over the classifiers. This technique allowed the evaluation of the model facing an unknown dataset. Results of the classification of valence, arousal and derived emotion labels are shown in Table 2.
Results show that, for both algorithms, using the complete set of features improves accuracy, however using only MFCC related features we get an accuracy comparable
with the one obtained using the all set of features. The worst setting is when MFCC is not considered. As far as the choice of algorithm is concerned, even if MLP had a slight better accuracy, the time to create the model and classify a vocal input is higher (100:1). Since VOCE has to be employed in real-time classification tasks we selected SMO.
Since the arousal dimension is related to the importance of the goal and the valence dimension is related to the achievement vs. the threatening of the goal, our speech classifier performs well in recognizing negative states, like those related to anger, and allows us to distinguish positive from negative attitudes. However, as expected, some emotions are easier to recognize than others. For example, humans are much better at recognizing anger than happiness; therefore, our results can be considered acceptable under this view.
Features ALL
Valence Arousal
Emotion MFCC
Valence Arousal
Emotion No_MFCC
Valence Arousal Emotion
MLP 70,45 80,90 71,36 69,09 80,00 70,45 64,09 74,09 53,18
SMO 71,36 77,27 68,63 64,54 75,00 68,18 55,45 69,09 53,18
bling emotion recognition during speech-based interaction (see Figure 3).
capability. The system is composed by two fundamental units: the NAO humanoid robot and the workstation connected with NAO robot. Audio files in wav format, recorded from 4 microphones located in the head of the NAO, are collected by the
order to allows the understaning of vocal commands and to recognize the valence and arousal. An Automatic Speech Recognition Module (ASR) performs the first task whereas the second task is accomplished by Voice Classifier for Emotions (VOCE)
Speech-Based Interface
with some sensors (to gather data about the house situation) and some effectors (to control appliances in the environments). Moreover the house is equipped with the
It’s friday evening and Nicola, a 73 y.o. man, is at home alone. He has an appointment with his friends downtown to play cards like he does almost every friday. Nicola is sitting on the bench in the living room that is equipped with sensors and effectors for controlling appliances in the room and with the social robot that acts as a mediator interface between the environment services and the user (see Figure 4). Nicola received a message saying that his doughter cannot accompany him downtown and this makes him a bit angry. Nicola calls NAO to try to find a solution.
In this scenario initially the voice classifier recognizes a negative valence with a high arousal from the prosody of the spoken utterance (Figure 2). This is interpreted as anger and the robot besides expressing empathy (it show to understand the user’s feelings) asks the reasons for it. When the robot understands that the user’s goal is to go downtown it finds a workflow satisfying this goal by matching the constraints of the situation (the daughter is not available). According to the selected workflow the dialog goes on in order to get some information that are necessary (preconditions) to execute some of its step (like the permission to call the taxi given from Nicola’s daughter).
We presented a preliminary work towards implementing a speech-based interface between a Social Robot and a user for handling interaction in a smart environment. In our opinion, besides assisting the elderly user in performing tasks, the robot has to establish a social long-term relationship with the user so as to enforce trust and confidence. The underlying idea of our work, in fact, is that the analysis of user’s spoken utterances can be used for both issuing requests to the robot and understaning his emotional state, which is important when the interaction happens in everyday life environments. To this aim we developed VOCE 2.0 a module to classify emotions from features extracted from the speech signal according to the dimensional model (valence and arousal). The recognition accuracy results are comparable with other research works in which the same dataset was used [29]. We are aware that an improvement is needed and to this aim we plan to collect a new dataset possibly with example of elderly voices, which may have different range of features from those used in the ∈motion dataset. Information on user’s emotions coupled to context and activity recognition may give the robot the capability to infer which is or will be the most probable user’s mood in a given context. 26. W.E. Bosma and E. André, “Exploiting Emotions to Disambiguate Dialogue Acts”, in Proc. 2004 Conference on Intelligent User Interfaces, January 13 2004, N.J. Nunes and C. Rich (eds), Funchal, Portugal, pp. 85-92, 2004. 27. Vincenzo Galata’. 2010. Production and perception of vocal emotions: a crosslinguistic and cross-cultural study. Ph.D. thesis, University of Calabria 28. www.praat.com 29. Antonio Origlia, Vincenzo Galatà e Bogdan Ludusan. Automatic classification of emotions via global and local prosodic features on a multilingual emotional database. In: Speech Prosody. 2010.