=Paper=
{{Paper
|id=Vol-1351/paper2
|storemode=property
|title=Towards an Empathic Social Robot for Ambient Assisted Living
|pdfUrl=https://ceur-ws.org/Vol-1351/paper2.pdf
|volume=Vol-1351
|dblpUrl=https://dblp.org/rec/conf/atal/CarolisFPC15
}}
==Towards an Empathic Social Robot for Ambient Assisted Living==
https://ceur-ws.org/Vol-1351/paper2.pdf
Towards an Empathic Social Robot for Ambient
Assisted Living
B. De Carolis, S. Ferilli, G. Palestra, V. Carofiglio
Dipartimento di Informatica, Universita’ di Bari “Aldo Moro”, Bari, Italy
berardina.decarolis@uniba.it, stefano.ferilli@uniba.it,
giuseppe.palestra@uniba.it
Abstract. In the context of Ambient Assisted Living, assistance and care are
delegated to the intelligence embedded in the environment that, in our opinion,
should provide not only a task-oriented support but also an interface able to es-
tablish a social empathic relation with the user. This can be achieved, for in-
stance, using a social assistive robot as interface towards the environment ser-
vices. In the context of the NICA (Natural Interaction with a Caring Agent) pro-
ject we developed the behavioral architecture of a social robot able to assist the
user in the interaction with a smart home environment. In this paper we de-
scribe how this robot has been endowed with the capability of recognizing the
user affective state from the combination of facial expressions and spoken ut-
terances and to reason on in order to simulate an empathic behavior.
1 Introduction
One of the new trends in the context of Ambient Assisted Living (AAL) concerns the
integration of new technologies with a social environment to support people in their
daily activities increasing their quality of life [1,2]. In this view, an assistive home
environment should provide not only a task-oriented support but also an interface able
to establish a social empathic relation with the user. This is what we call a “caring
home”. Achieving this objective, in our opinion, requires developing:
• Methods and models for defining and developing Ambient Intelligence (AmI) sys-
tems for Assisted Living that are able to define environments that manage devices
and services autonomously and proactively with respect to the needs of the users
populating the environment.
• Methods and models for analysis of the user behavior with particular emphasis on
affective aspects in order to achieve personalization, adaptation and proactivity
that are typical of an AAL system.
• Natural Interaction of the user with the information and services offered by the
system. Such an interface has two fundamental and interconnected objectives: be-
ing a means to interact with the environment and being, for the user, a friendly
caring agent. For this reason it is important to understand not only the meaning of
the communication but also the conveyed emotions and the user’s attitude during
the interaction. This requires the emotional analysis of the user’s verbal and non-
verbal communicative acts (i.e. linguistic and prosodic aspects of the user's vocal
input, facial expressions, postures and gestures).
In the context of the NICA (Natural Interaction with a Caring Agent) project we
developed the behavioral architecture of a social robot able to assist the user in the
interaction with a smart home environment [3]. In this paper we propose the use of a
social empathic robot acting as a virtual caregiver. In particular we discuss how it has
been endowed with the capability of recognizing the user’s affective state from the
combination of facial expressions and spoken utterances and of reasoning on it in
order to simulate an empathic behavior.
The choice of a social assistive robot as an interaction metaphor is driven by the
following considerations. If properly designed, social and conversational agents and
robots may improve the naturalness and effectiveness of the interaction between users
and systems [4]. They have the potential to involve users in human-like conversations
using verbal and non-verbal signals for providing feedback, showing empathy and
emotions in their behavior [5,6]. Indeed, several studies report successful results on
how expressive conversational agents and robots can be employed as an interaction
metaphor in the assisted-living domain and in other ones [7,8] where it is important to
settle long-term relations with the user [9].
Empathy can be defined as “an affective response more appropriate to someone
else’s situation than to one’s own” [10]. Then the expression of empathy aims at
demonstrating that the other’s feelings are understood or shared. Moreover, according
to [11], empathy facilitates the creation of social relations. Empathic agents are per-
ceived as more caring and trustworthy than neutral agents [12] and they can induce
empathy in users [13]. In particular, the simulation of empathy in socially assistive
robotics is supported by the findings of many psychologists showing that empathy
plays a key role for therapeutic improvement and that empathy mediates pro-social
behavior (e.g., [14,10]).
Taking these findings into account, we decided to endow a social assistive robot
with the capability of recognizing the user affective state and attitude, reasoning on it
and, consequently, deciding whether to trigger an empathic behavior toward the user.
Moreover, in order to improve the long-term relation between the user and the robot,
it keeps in its social memory information about which are the antecedents of emotions
for the user, that is what triggers the emotions (events, situations, thoughts, etc.) in
order to improve its empathic capability. These behaviors have being modeled ac-
cording to the analysis of a corpus collected by human caregivers.
The paper is structured as follows: after providing an overview of the related work
in Section 2, in Section 3 we show how the empathic behavior is simulated in the
robot; in Section 4 a brief illustration of a case study is described; finally we conclude
the paper with discussion and directions for future work.
2 Related Work
The main aim of Ambient Assisted Living (AAL) is to improve the life quality of
elderly people who need special care and assistance by providing cognitive and phys-
ical support and access to the environment services [15]. Many of these projects be-
sides developing technological platforms to monitor the health state and comfort of
the user, provide natural and pleasant interfaces for interacting with the smart envi-
ronment services. Several studies report successful results on how expressive conver-
sational agents and robots can be employed as interaction metaphor in the assisted
living domain. For instance, projects ROBOCARE [16], Nursebot [17], Care-o-bot
[18], CompaniAble [19] and KSERA [20] aim at creating assistive intelligent envi-
ronments in which robots offer support to the elderly at home, possibly having also a
companion role. van Ruiten et al. [21] conducted a controlled study using I-Cat [22]
in which they confirmed the results that, as shown in [23], elderly users like to inter-
act with a social robot and to establish a relation with it. The reason of the success of
socially intelligent agents and robots is due to the fact that interaction between human
and machine has a fundamental social component [24]. Thus endowing social agents
with user models that involve the consideration of both cognitive and affective com-
ponents of the user state of mind is a key issue for enabling the adaptation of the
agents behavior to both physical and emotional user’s needs, as in the case of the
simulation of the empathic behavior.
As far as simulating empathic behavior in social agents is concerned, there are sev-
eral studies that aim at evaluating the impact of empathy on the interaction and in
particular on settling a social relation between the agent and the user [29].
Paiva [25] defines empathic agents as “agents that respond emotionally to situa-
tions that are more congruent with the user's or another agent’s situation, or as agents,
that by their design, lead users to emotionally respond to the situation that is more
congruent with the agent’s situation than the user’s one “. In this view, Klein et al.
[26] describe an experimental study aimed at evaluating interfaces that implement
strategies for affectively supporting users experience with negative moods and emo-
tions by showing empathy and by actively supporting them. Results show how the
affect-support was effective in relieving the user negative affective states when inter-
acting with the computer. Along this perspective we find the work by Prendinger et
al. [27] that developed an embodied agent in the scenario of job interviews that is able
to recognize physiological data of users in real-time, to interpret this information as
affective states, and to respond to affect by employing an animated agent. Sabourin et
al. [28] present a study about designing pedagogical empathic virtual agents in a nar-
rative-centered learning environment. They adopt a cognitive model, structured as a
Bayesian network, which includes personal attributes of users (i.e. personality and
goals of students), environment variables (i.e. dynamic attribute capturing a snapshot
of the student’s situation and activity) and physiological data about the user behavior
(i.e. biofeedback parameters such as heart rate or galvanic skin response).
Recently several projects on AAL are endowing assistant robots with social capa-
bilities. In [30] the possible role of empathy in socially assistive robotics is discussed.
Leite et al. [29] propose a multimodal framework for modeling some of the user’s
affective states in order to personalize the learning environment by adapting a robot’s
empathic responses to the particular preferences of the child who is interacting with
the robot.
Looking in more details to human-robot interaction, several EU-projects have ad-
dressed the modeling, definition, and implementation of social and cognitive skills in
Social Assistive Robots (SARs) [44,45,20]. In particular, in order to enhance human
robot interaction, emotional behavior recognition and generation have also been de-
veloped for social robots. In literature, two different approaches can be found to ad-
dress this issue: social robots as agents able to generate emotions in human - robot
interaction and robots able to recognize emotions of the human partners and to conse-
quently adjust their behaviors. We reported here some examples of both approaches
by considering only a mobile humanoid robot, the NAO by Aldebaran [37].
In their work, Cohen et al. [38] proposed two robots, the NAO and the i-Cat, able
to express recognizable emotions and compared the recognition rates of the emotions
in the two cases. For both robots, recognition rates for the expressions were relatively
high but they focused their attention on NAO robot considering its body and colored
eyes to express recognizable emotions. Tielman [39] proposes a model for adaptive
emotion expression for the NAO. The robot communicates these emotions through its
voice, eye colors, posture and gestures. An experiment with 18 children and two NA-
Os was carried on to test the effect of adaptive emotions on robot-child interaction. In
the experiments, the children played a quiz with both an affective robot using the
model for adaptive emotion expression and a non-affective robot. The experiment
results confirmed that children responded more expressively to a robot that adaptively
expressed itself than to a robot that did not.
Others studies present robots able to recognize and generate emotions. In the work
of Zhang et al. [42], Facial Action Coding System has been incorporated in order to
describe physical cues and facial behavior useful for the detection of six basic emo-
tions plus neutral from real-time and posed facial expressions. The system was im-
plemented on NAO humanoid. In Lim et al. [40] a developmental robot able to under-
stand and express emotions in voice, gesture and gait using a model trained with voice
data was presented. The recognized emotions were happiness, sadness, fear and neu-
tral. In experiments, authors assumed an adult-infant simple interaction based on 4
Japanese words for ‘hello’, ‘look’, ‘no’, ‘bye bye’.
Another important field of application are robotic tutors developed with the ability
to perceive emotions experienced by learners, and to incorporate these into pedagogi-
cal strategies. In a recent study, researchers addressed the problem of creating em-
pathic robot tutors to support school students studying geography topics on a multi-
touch table. The NAO robot tutor was equipped with a game-specific AI player that
allowed it to play any of the different roles in the game. The next steps will be to use
the AI to generate appropriate commentary feedback from the robot in a way that it
can seem empathic to the users while still portraying its tutor role [41].
Most of the previous works with empathy in robotics focused on the perception
and impact of empathy on participant attitude towards the robot.
3 Simulating Empathic Behavior
The concept of empathy is related to the understanding of what is happening to the
other person. Therefore, according to [46], a model for simulating empathy in a robot
should be able to i) recognize the affective cues and the affective state of the user and
ii) interpret the motivations that triggered that emotion, iii) answer by expressing its
emotions (as a consequence of the recognized state) by using different modalities
(voice, facial expressions, and body movements and gestures) since the combination
of verbal and non-verbal communication provide social cues that make robots appear
more intuitive and natural. Our first attempts towards simulating empathy with a
socially assistive robot are based on the understanding of the emotions of others (i.e.,
human users). We have developed a simple vision-based facial expression detection
system capable of identifying a basic set of facial expressions including smiling,
frowning, sadness, anger, etc. The list of facial expressions our system is capable of
detecting is a subset of the Ekman’s six basic emotions on human facial expression:
joy, sadness, fear, anger, disgust and surprise [51]. The recognition of facial
expressions is combined with the analysis of speech-based communication. In
particular the speech prosody is analyzed in order to recognize its valence and
arousal.
3.1 Collecting a behavioral information from human caregivers
To define and implement feasible behaviors of the robot, we integrated data collected
from human caregivers with the guidelines that they follow in assistance of elderly
people. In particular two human caregivers recorded their experience during the assis-
tance of two elder women, both affected by a chronic disease, for a period of one
month. These women lived alone and had a son/daughter which could intervene only
in case of need and for solving relevant medical and logistic problems. Data have
been collected using a paper-diary on which the caregiver had to annotate two kinds
of entries: (i) the schedule of the daily tasks and (ii) the relevant events of the day,
using a schema like the one reported in Table 1.
Table 1 - Some entries from the caregivers’ paper-diary
Communicative
Recognized
Time
Event
Signs
Reason
Action
Effect
action
affect
I
remind
Remind
Maria
about
Maria,
I
would
like
to
medical
the
appoint-‐ remind
you
that
today
10.00
…
…
…
…
visit
ment
with
the
you
have
an
appoint-‐
doctor
at
ment
with
the
doctor
11.00.
at
11.00
a.m..
Ask_for
I
ask
and
Today
is
a
wonderful
medical
Maria
is
10.30
…
…
help
Maria
to
day.
You
can
put
on
…
visit
dressed
dress
up.
your
beautiful
dress
that
you
like
so
much!
I
send
a
The
reminder
to
daughter
medical
Maria’s
10.40
…
…
…
…
answered
visit
daughter
that
she
is
about
the
coming.
medical
visit.
Sit
down,
Console
I
go
toward
Moaning
I’m
sorry
to
see
you
so
Maria
is
medical
Maria
and
try
Maria
is
10.45
“Oh
my...
sad!
You
will
see
that
sadness
worried
visit
to
console
less
sad
Oh
my”,
everything
will
be
all
her.
Sad
face
right
.
In particular, each row of the table represents a relevant event with the attributes
for describing it and the action performed by the caregiver when this event occurred.
For example, let’s consider the 4th row: at 10.45 (time) Maria is worried (event). The
caregiver inferred Maria’s state since she was moaning, saying “Oh my, oh my”
(signs) because she had to go to the doctor (reason). The caregiver recognized Ma-
ria’s sadness (recognized affective state). Hence, she went toward Maria (action)
trying to encourage her by saying “Come on, don’t worry! You will not have any
problem for sure.” (communicative action). After this action she noticed that Maria
was less sad (effect).
From the collected data, we extracted the knowledge needed to build the reasoning
strategies of the agent, so as to make its behavior believable. Overall, we collected a
corpus of about 900 entries, which we used for: i) understanding which are the events
and context conditions relevant to goal and action triggering; ii) understanding when
considering affective and social factors is important during the interaction in real-life
scenarios; iii) defining situation-oriented action plans and dialogue strategies; iv)
collecting example dialogues between elderly people and human caregivers useful for
testing the robot behavior.
3.2 An overview of NICA Architecture
As described in [5], the approach that we adopted in designing the architecture of
NICA consists in interfacing the agent’s Body (for example Nao, Aibo, a
conversational agent,…) with a Mind that, using several knowledge bases, reasons on
which goal to pursue. NICA’s Mind has been modeled as a BDI (Belief, Desire,
Intentions) agent, whose behavior is driven by persistent goals [38].
Briefly, the agent has a mission stated in the list of its persistent goals that have to
be pursued during the agent lifecycle. At each stage of its life cycle, the agent evalu-
ates whether there have been changes in the environment or in the user’s state that
may threaten its persistent goals and cause a change in the planned behavior by trig-
gering new goals and/or by modifying the scheduled actions.
At the present stage of development the agent considers a set of persistent goals re-
lated to the user’s wellbeing, the execution of necessary actions of the user’s daily
routine, and so on. These goals correspond to the ones that human caregivers indicat-
ed as the most important ones in their daily assistance.
The agent implements a life cycle based on the following steps:
1. Perception: allows collecting data from sensors present in the environment
and to handle the user input (speech, gestures, facial expressions or actions
in the environment).
2. Interpretation: evaluates changes in the world and user state that are relevant
to the agent’s reasoning and transforms them into a set of agent’s beliefs. In
particular it interprets the user’s input.
3. Goal Activation: goals are triggered based on the current beliefs.
4. Planning and Execution: once a goal has been triggered it is achieved
through the execution of a plan appropriate to the situation.
Although the agent can purse different persistent goals, since this paper focuses on
how it reasons on the user affective state and how to trigger empathic behaviors in our
examples we will consider the following goal as the most relevant one:
(BEL A NOT(Is(U, Negative(affective_state)) - “The Agent A has to belief
that the user U is not in a negative affective state”.
This means that NICA has to believe that the user is not in a negative affective
state. As illustrated in Figure 1, in order to check whether this goal has been threat-
ened the agent has to:
i) interpret the user’s communicative actions expressed through speech and facial
expressions;
ii) in case of expression of an emotion, recognize it and react emotionally to it;
iii) trigger a goal accordingly;
iii) achieve this goal through a communicative plan (“what to say”) that can then
be rendered as a combination of voice and animations of the agent’s body (“how to
say”) [35];
iv) keep in its social memory information about which are the antecedents of emo-
tions for the user, that is what triggers the emotions (events, situations, thoughts, etc.).
Fig. 1. A schema illustrating the triggering of the empathic behavior in the robot.
The social memory is used to remember relations about events and the user’s affec-
tive state. The importance of this piece of knowledge in the agent’s mind is related to
the need of establishing empathy with the elder person by remembering relevant data
and this requirement was outlined several times by the human caregivers during the
data collection phase.
As far as reasoning is concerned, in order to deal with the uncertainty typical of
this domain (e.g. dealing with exceptional situations or with the smooth evolution of
the user’s affective state over time), we employ probabilistic models to reason on the
user and to decide which behavior to adopt, that is the most appropriate set of actions
to perform for satisfying the inferred user’s goal.
At the present stage of the project, we simulate the interaction between the agent
and the user by embodying NICA’s Mind in the Nao robot. We adopted the approach
proposed by Johnson et al. [43] to simulate emotion through Nao eyes by combining
specific LED color patterns (Fig. 2).
Blue - Sadness Red - Anger
Fig. 2. A simulation of two emotion expressions with Nao.
3.3 Recognizing the User Affective State
In the current prototype of the system, we use a simple vision-based facial expression
detection system capable of identifying a basic set of facial expressions in order to
recognize the emotions of the human users. The list of facial expressions our system
is able to detect is a subset of the Ekman’s six basic emotions on human facial
expression. The facial expression recognition system we adopted is fully automatic
and, considering four-class expressions classification, the recognition rates we
achieved were 82%, 76% and 95% using respectively Multi-SVM, k-Nearest
Neighbors and Random Forest.
As far as spoken interaction is concerned, we employ VOCE (VOice Classifier for
Emotions) a module that classifies the valence and arousal in the voice prosody. Our
classifier follows an approach similar to [33,34]. In particular, the valence dimension
is classified from positive to negative along a 4-point scale (from 1=very negative to
4=positive). Arousal is classified in a 3-point scale from high to low.
As far as the valence classification is concerned, the accuracy of the C4.5 algo-
rithm is 83.12%, very close to the one of the K-NN that is 82.45%. As far as the
arousal is concerned, C4.5 has an accuracy of 79.8% while the one of K-NN is 83.63
(validated using a 10 Fold Cross Validation technique).
3.4 Reasoning on the User’s Affective State
In our model of empathy for a virtual caring robot we start from the recognition of the
user’s affective state for monitoring the belief associated to this emotional state. In
this way, during its lifecycle, the agent evaluates whether it is appropriate to trigger
an affective communicative goal aimed at triggering the empathic behavior.
The robot’s beliefs about the user’s affective state are monitored with a dynamic
model based on Belief Network (DBN) [32]. In fact, when modeling affective phe-
nomena we must take into account the fact that affective state smoothly evolve during
the interaction, from one step to the subsequent one and the state at every time of the
interaction depends on the state it assumes in the previous turn. For this reason, the
DBN formalism is particularly suitable for representing situations that gradually
evolve from a dialog step to the next one. Moreover, Belief Networks are a well-
known formalism to simulate probabilistic reasoning and deal with uncertainty in the
relationships among the variables involved in inference process. The DBN model is
shown in Figure 3. The model is employed to infer which is the most probable emo-
tional state the user is experiencing at every step of the interaction by monitoring
speech and facial expressions and it is also used to monitor the overall evolution of
the user’s affective state (i.e. the belief of the agent about the positive or negative
affective state of the user). In the model this is expressed by a temporal link between
the Bel(AffectiveState)Prev and the Bel(AffectiveState) variables. At present we con-
sider only a subset of the affective states that can be relevant for the generating an
empathic response: sadness, happyness and anger.
Fig. 3. The DBN model of the agent’s beliefs about the user affective state.
In particular, every time a new user move is entered, its acoustic features are ana-
lyzed and the resulting evidence are introduced and propagated in the network to
recognize the user’s emotion and the overall polarity of her affective state. The same
happens for the facial expression recognition module. The new probabilities of indi-
vidual emotions are read and contribute to formulate the behavior of the agent; the
probability of the dynamic variable (Bel(AffectiveState)) representing the valence of
user’s affective state is employed by the agent to check the consistency between its
persistent goal of maintaining the user in a positive or neutral affective state and the
actual emotional state the user is in at the time t, thus causing the activation of the
empathic goal.
3.5 Emotion Activation in the Robot’s Mind
In order to activate an affective state in the robot for triggering affective goals we
revised the emotion modeling method that we employed in another project [47]. The
model is based on event-driven emotions according to Ortony, Clore and Collin’s
(OCC) theory [48]. In this theory, positive emotions (happy-for, hope, joy, etc.) are
activated by desirable events while negative emotions (sorry-for, fear, distress, etc.)
arise after undesirable events. In addition we considered also Oatley and Johnson-
Laird’s theory in which positive and negative emotions are activated (respectively) by
the belief that some goal will be achieved or will be threatened [49]. In the context in
which we employ the robot, we consider emotions in the Well-being category (joy,
distress) and those concerning the FortuneOfOthers category (happy-for, sorryfor).
Then, the cognitive model of emotions that is built on these two theories should repre-
sent the system of beliefs and goals behind emotion activation and endows the robot
with the ability to guess the reason why she feels a particular emotion and to justify it.
The model of emotion activation is also represented with a DBN since we need to
reason about the consequences of the observed event on the monitored goals in suc-
cessive time slices. We calculate the intensity of emotions as a function of the uncer-
tainty of the robot’s beliefs that its goals will be achieved (or threatened) and of the
utility assigned to achieving these goals. According to the utility theory, the two vari-
ables are combined to measure the variation in the intensity of an emotion as a prod-
uct of the change in the probability to achieve a given goal, times the utility that
achieving this goal takes to the robot.
Let us consider, for instance, the triggering of Sorry-for in the robot’s model that is
represented in Figure 4. This is a negative affective state and the goal that is involved,
in this case, is preserving others from bad. In this figure R denotes the robot and U
the user. The robot’s belief about the probability that this goal will be threatened (Bel
R (Thr-GoodOf U)) is influenced by his belief that some undesirable event E occurred
to the user (Bel R (Occ E U)). According to Elliott and Siegle [50], the main variables
influencing this probability are the desirability of the event (Bel R not(Desirable E))
and the probability that the robot attaches to the occurrence of this event (Bel R (Occ
E U)). The user moves are interpreted as observable consequences of occurred events,
that activate emotions through a model of the impact of this event on the robot’s be-
liefs and goals. The user may say that a not desirable event occurred to him and may
feel sadness or distress (Feel U(emotion)) that denotes that the event is undesirable.
The probability of this node to be true depends on the emotion node in the network in
Figure 3. This influences R’s belief that U would not desire the event E to occur (Bel
R Goal U ¬(Occ E U)) and (since R is in a empathy relationship with U, R adopts U’s
goals), its own desire that E does not occur (Goal R ¬(Occ E)). This way, they concur
to increase the probability that the robot’s goal of preserving others from bad will be
threatened. Variation in the probability of this goal activates the emotion of sorry-for
in R.
Fig. 4. A portion of the DBN representing the robot’s mental state for the triggering of Sorry-For
The intensity of this emotion is the product of this variation times the weight the
robot gives to the mentioned goal. The strength of the link between the goal-
achievement (or threatening) nodes at two contiguous time instants defines the way
the emotion, associated with that goal, decays, in absence of any event affecting it. By
varying appropriately this strength, we simulate a more or less fast decay of emotion
intensity. Different decays are attached to different emotion categories (positive vs.
negative, FortuneOfOthers vs. Wellbeing and so on) and different temperaments are
simulated, in which the persistence of emotions varies.
3.6 Triggering Empathic Behavior in the Robot
In order to decide how to behave as a consequence of the triggering of an emotion in
the agent state of mind the agent triggers an affective goal. The list of empathic goals
is inspired by the indications that human caregivers gave us during the data gathering
phase at the beginning of the project, by the literature of empathy and pro-social be-
havior [29] and by the results of another study on the influence of empathic behaviors
on people’s perceptions of a social robot [21].
Currently, the empathic goals are the following:
- console by making the user feel loved and cuddled;
- encourage by providing comments or motivations like for example “ don’t be
sad, I know you can make it!”
- congratulate by providing positive feedback on the user’s behavior;
- joke by doing some humor in order to improve the user’s attitude;
- calm down by providing comments and suggestion to make the user feel more
relaxed.
For instance in case the sorry-for emotion is felt by the robot, the console goal
should be triggered. Once a goal has been selected as the most appropriate to the
emotion felt by the agent, the behavior planner module computes the agent behavior
using plans represented as context-adapted recipes. Each plan is described by a set of
preconditions, the conditions that have to be true to select the plan, the effect that the
plan achieves and the body, the conditional actions that constitute the plan. After the
execution of each action in the plan, the correspondent effect is used to update beliefs
in the agent’s mental state.
A sample of a portion of plan used to achieve the Console goal is the following:
………
The tag allows selecting actions on the basis of the current situation.
For instance, the action is
used to express the sorry attitude of the robot R and will be performed only if the user
feel sad. In the same way, the action “Ask” about “Why the user is sad” will be
performed only if the agent does not know why the user is in the current state.
Moreover, if the action is complex, then it can be specified in a subplan describing
elementary agent actions. Each communicative act in the plan is then rendered using
simple template-based surface generation technique [35]. These templates are selected
on the basis of the type of communicative act and its content and are expressed in
metalanguage [36] that is then interpreted and executed by the agent’s body. Plans
and surface generation templates have been created and optimized combining actions
on the basis of pragmatic rules that were derived from the corpus dataset.
4 A Case Study
In this section we show an example of an empathic behavior of the agent in a typical
interaction scenario that we envisaged as a suitable one for testing our agent frame-
work.
It’s morning and Nicola, a 73 y.o. man, is at home alone. He doesn’t feel very well
since he has a cold and fever. Nicola is sitting on the bench in his living room that is
equipped with sensors and effectors. According to the situation the smart environment
selects a workflow and starts to execute scheduled tasks accordingly. The caring robot
has to check Nicola’s health state and recommend him to take some medicine. After a
while Nicola starts whispering and says with a sad facial expression: “Oh My …oh
poor me…”. This is perceived by the robot that interprets it and activates the most
appropriate behavior.
The voice classifier recognizes a negative valence with a low arousal from the
prosody of the spoken utterance and the facial expression classifier recognizes the
sadness emotion. These evidences are propagated in the DBN and the belief about the
affective state of the user is in a negative affective state with the higher probability
(65.56), as shown in Figure 3. Then, since the goal of keeping the user in a state of
well-being is threatened, the DBNs modeling the robot’s affective mind are executed
to trigger the robot’s affective goal (sorry-for in this case). As described in the previ-
ous section, the goal to pursue in this situation is the “console” one. Then, the corre-
spondent plan is selected (see previous section) and the execution of its actions be-
gins. The plan includes the following actions since the agent does not know why the
user is sad and it will ask the user about it:
MoveTo(NAO,NICOLA)
Express(NAO,Sorry-
for(NAO,NICOLA))
Ask(NAO,NICOLA,Why(Feel(NICOLA,Sa
dness)))
Express(NAO,Console(NAO,NICOLA))
Fig. 5 A simulation of the scenario with an elderly person.
When a new belief about the event that occurred to the user related to a particular
affective state is acquired by the robot during the interaction, it is stored in the agent
Social Memory. In this way the robot will remember which event causes a particular
affective state in the user, for instance the event “has_disease” is associated to the
affective state “sadness”. This information can be used by the agent in the dialogue
with the user for preventing this state or for improve the relation between the user and
the robot.
5 Discussion and Future Work
This paper presented issues concerning the importance of taking into account affec-
tive factors when modeling the user in social interaction with a caring agent. In our
opinion, besides assisting the elderly user in performing tasks, the agent has to estab-
lish a social long-term relationship with the user so as to enforce trust and confidence.
The underlying idea of our work, in fact, is that endowing the robot with a social
empathic behavior is fundamental when the devices of a smart home are integrated
pervasively in everyday life environments. In this paper we illustrated how this capa-
bility has been designed and implemented in a caring assistant for elderly people.
Evaluating the efficacy of the empathic behavior of the social robot in a real-
context at the moment is not feasible due to the lack of enough smart homes equipped
with social robots. Therefore, we performed a quantitative evaluation of the decisions
and plans executed by the agent compared to the behaviors of the human caregivers
that we annotated in a previous phase of the project. To this aim we randomly split
our corpus into 70/30 training/test partitions. For each item of the test set, we formal-
ized the corresponding scenario in order to set the evidences in the simulation test.
Then, we observed the robot’s behavior in terms of selected communicative acts: the
behavior of the robot was classified as ‘correct’ if it matched the choice of the human
caregiver, as ‘incorrect’ vice versa. Results of the evaluation are encouraging and
indicate that the system performance is quite good since the choices of the agent
match the human actions in the dataset in the 79% of cases. We are aware that it is
important to conduct an evaluation study with real elderly users. This kind of experi-
ment should aim at assessing the impact of the use of a social robot vs. seamless in-
teraction with the environment services in smart environments. Another important
issue to be addressed in our future work concerns the interpretation of gestures and
postures of the user.
Acknowledgements
This work fulfils the research objectives of the PON02_00563_3489339 project
"PUGLIA@SERVICE - funded by the Italian Ministry of University and Research (MIUR).
References
1. Bierhoff, I. and van Berlo, A. More Intelligent Smart Houses for Better Care and Health,
Global Telemedicine and eHealth Updates: Knowledge Resources",vol. 1, 322-325, 2008.
2. Steg, H. et al. Ambient Assisted Living – European overview report, September, 2005
3. De Carolis, B., Ferilli S., Greco D. Towards a Caring Home for Assisted Living.Workshop
"The Challenge of Ageing Society: Technological Roles and Opportunities for Artificial
Intelligence"co-located with the AI*IA 2013.
4. Brahnam, S. & De Angeli, A. (2008). Editorial - Special issue on the abuse and mis-
use of social agents. Interacting with computers, 20 (3).
5. R. Niewiadomski, M. Ochs and C. Pelachaud, Expressions of Empathy in ECAs, in Pro-
ceedings of the 8t Int. Conf. on IVA, LNAI, vol. 5208. Springer-Verlag, Berlin, Heidelberg,
pp. 37-44, 2008.
6. W.S.N. Reilly. Believable Social and Emotional Agents. PhD thesis (1996)
7. Nijholt, A., de Ruyter, B., Heylen, D. and Privender, S. Social Interfaces for Ambient
Intelligence Environments. Chapter 14 in: True Visions: The Emergence of Ambient Intel-
ligence. Aarts, E. and Encarnaçao, J., Eds. Springer, New York, 2006, 275—289.
8. Ortiz, M. Del Puy Carretero, D. Oyarzun, J.J.Yanguas, C. Buiza, M. F. Gonzalez, and I.
Etxeberria. Elderly users in ambient intelligence: does an avatar improve the interaction?.
In Proceedings of the 9th conference on User interfaces for all (ERCIM'06), Constantine
Stephanidis and Michael Pieper (Eds.). Springer-Verlag, Berlin, Heidelberg, 99-114, 2006.
9. T. Bickmore and R. W. Picard, Establishing and maintaining long-term human-computer
relationships, ACM Transactions on Computer Human Interaction, 12, 2, 293-327, 2005.
10. Hoffman, M. L. (1981). The development of empathy. In J. Rushton & R. Sorrentino
(Eds.), Altruism and helping behavior: Social personality and developmental perspectives
(pp. 41–63). Hillsdale, NJ: Erlbaum.
11. C. Anderson and D. Keltner (2002). The role of empathy in the formation and maintenance
of social bonds. Behavioral and Brain Sciences, 25, pp 21-22.
12. S Brave, C Nass, and K Hutchinson. Computers that care: investigating the effects of orien-
tation of emotion exhibited by an embodied computer agent. Int. J. of Human Computer
Studies 62(2):161–178, 2005.
13. A. Paiva, et. al. “Caring for Agents and Agents that Care: Building empathic relations with
synthetic agents”, AAMAS 2004, ACM Press, 2004.
14. Eisenberg, N., & Miller, P. A. (1987). The relation of empathy to prosocial and related
behaviors. Psychological Bulletin, 101, 91–119.
15. http://www.aal-europe.eu/wp-content/uploads/2012/04/AALCatalogue_onlineV4.pdf
16. A. Cesta, G. Cortellessa, F. Pecora and R. Rasconi, Supporting Interaction in the RoboCare
Intelligent Assistive Environment, AAAI 2007 Spring Symposium, 2007.
17. J. Pineau, M. Montemerlo, M. Pollack, N. Roy and S. Thrun, Towards Robotic Assistants
in Nursing Homes: Challenges and Results, Robotics and Autonomous Systems 42(3–4),
pp. 271–281, 2003.
18. Graf B, Hans M, Schraft RD (2004) Care-O-bot II – development of a next generation
robotics home assistant. Auton. Robots 16, 193–205.
19. CompanionAble project (2011) http://www.companionable.net/
20. Cuijpers, R.H. (2012). http://ksera.ieis.tue.nl/
21. A.M. van Ruiten, A. M., Haitas, D., Bingley, P., Hoonhout, H.C.M., Meerbeek, B.W. and
Terken, J.M.B. Attitude of elderly towards a robotic game-and-train- buddy: evaluation of
empathy and objective control. In R. Cowie and F. de Rosis (Eds.) Proceedings of the Doc-
toral consortium, in the scope of ACII2007 Conference, 2007.
22. J. N. van Breemen, (2004). iCat: a generic platform for studying personal robot applica-
tions. Paper presented at the IEEE SMC, Den Haag.
23. J. Broekens, M. Heerink, H. Rosendal. Assistive social robots in elderly care: a review.
Gerontechnology 2009; 8(2):94-103; doi: 10.4017/gt.2009.08.02.002.00
24. Reeves and C. Nass, The media equation: how people treat computers, television, and new
media like real people and places, Cambridge University Press, New York, NY, 1996.
25. A. Paiva et al. Empathy in Social Agents. International Journal of Virtual Reality, Vol. 10,
No. 1, pg. 65-68, 2011.
26. Klein J, Moon Y, Picard R (2002) This computer responds to user frustration: Theory,
design, and results. Interacting with Computers, 14:119–140.
27. Prendinger H, Mori J, Ishizuka M (2005) Recognizing, modeling, and responding to users
affective states. In Proceedings of User Modeling 2005, Lecture Notes in Computer Sci-
ence, Volume 3538/2005, 149, DOI: 10.1007/11527886_9 2005
28. Sabourin J, Mott B, Lester J (2011) Computational Models of Affect and Empathy for.
Pedagogical Virtual Agents. Standards in Emotion Modeling, Lorentz Center International
Center for workshops in the Sciences.
29. I. Leite and C. Martinho and A, Paiva Social Robots for Long-Term Interaction: A Survey.
International Journal of Social Robotics, pg. 1--18, January, 2013.
30. Maja Mataric', Adriana Tapus, and David Feil-Seifer (2007) “Personalized Socially Assis-
tive Robotics”, Workshop on Intelligent Systems for Assisted Cognition, Rochester, New
York, USA, October, 2007.
31. C. Castelfranchi and F. Paglieri, The role of beliefs in goal dynamics: Prolegomena to a
constructive theory of intentions, Synthese 155, pp. 237-263, 2007.
32. F.V. Jensen, Bayesian Networks and Decision Graphs, Statistics for Engineering and In-
formation Science, Springer, 2001.
33. Vogt T, Andre’ E, Bee N (2008) EmoVoice - A Framework for Online Recognition of
Emotions from Voice. In Proceedings of the 4th IEEE tutorial and research workshop on
Perception and Interactive Technologies for Speech-Based Systems: Perception in Multi-
modal Dialogue Systems (PIT '08), Springer-Verlag, Berlin, 188-199.
34. Sundberg, J., Patel, S., Björkner, E., & Scherer, K.R. (2011). Interdependencies among
voice source parameters in emotional speech. IEEE Transactions on Affective Computing,
2(3).
35. Reiter E, Dale R (2000) Building Natural Language Generation Systems. Studies in natural
language processing. Cambridge University Press, Cambridge, United Kingdom. ISBN 0-
521- 62036-8.
36. De Carolis B, Pelachaud C, Poggi I, Steedman M (2004) APML, a Mark-up Language for
Believable Behavior Generation, in H. Prendinger Ed, Life-like Characters, Tools, Affec-
tive Functions and Applications, Springer.
37. http://www.aldebaran-robotics.com/ (last visited Oct 3, 2012).
38. Cohen, I., Looijeand, R. & Neerincx, M. A. (2011). Child’s recognition of emotions in
robot’s face and body. In Proceedings of the 6th international conference on human-robot
interaction (pp. 123–124).
39. Tielman M. Adaptive emotional expression in robot-child interaction. Proceedings of the
2014 ACM/IEEE international conference on Human-robot interaction - HRI '14. New
York, New York, USA: 2014. p. 407-414.
40. Lim, A.; Okuno, H.G., "The MEI Robot: Towards Using Motherese to Develop Multimodal
Emotional Intelligence," Autonomous Mental Development, IEEE Transactions on , vol.6,
no.2, pp.126,138, June 2014
41. Ribeiro, T., Pereira, A., Deshmukh, A., Aylett, R., & Paiva, A. (2014, May). I'm the mayor:
a robot tutor in enercities-2. In Proceedings of the 2014 international conference on Auton-
omous agents and multi-agent systems (pp. 1675-1676). International Foundation for Auto-
nomous Agents and Multiagent Systems.
42. Zhang, L., Jiang, M., Farid, D., & Hossain, M. A. (2013). Intelligent facial emotion recog-
nition and semantic-based topic detection for a humanoid robot. Expert Systems with Ap-
plications, 40(13), 5160-5168.
43. Johnson, D. O., Cuijpers, R. H., & van der Pol, D. (2013). Imitating human emotions with
artificial facial expressions. International Journal of Social Robotics, 5(4), 503-513.
44. DOMEO (Domestic Robot for Elderly Assistance) (2012). www.aal-domeo.eu
45. MOVEMENT website (2012). http://www.is.tuwien.ac.at/fortec/reha.e/ projects/movement/
46. Feshbach, N. D. 1987. Parental empathy and child adjustment/maladjustment. In Eisenberg,
N., and Strayer, J., eds., Empathy and its development. Cambridge University Press.
47. de Rosis, De Carolis, Carofiglio, Pizzutilo 2003. Shallow and inner forms of emotional
intelligence in advisory dialog simulation. In H. Prendinger and M. Ishizuka (Eds.): “Life-
Like Characters. Tools, Affective Functions and Applications”. 271-294. Springer 2003.
48. Ortony, A., Clore, G.L. and Collins, A., 1988. The cognitive structure of emotions. Cam-
bridge University Press.
49. Oatley, K. and Johnson-Laird, P.N., 1987. Towards a Cognitive Theory of Emotions. Co-
gnition and Emotion, 29-50.
50. Elliott, C. and Siegle, G., 1993. Variables influencing the intensity of simulated affective
states. In Proceedings of the AAAI Spring Symposium on Mental States.’93, 58-67.
51. Ekman, P. & Oster, H. (1979). Facial Expressions of Emotion. Annual Review of Psycho-
logy, 30, 527-554.