=Paper=
{{Paper
|id=Vol-1122/paper2
|storemode=property
|title=Towards a Caring Home for Assisted Living
|pdfUrl=https://ceur-ws.org/Vol-1122/paper2.pdf
|volume=Vol-1122
|dblpUrl=https://dblp.org/rec/conf/aiia/CarolisFG13
}}
==Towards a Caring Home for Assisted Living==
https://ceur-ws.org/Vol-1122/paper2.pdf
Towards a Caring Home for Assisted Living
Berardina De Carolis, Stefano Ferilli, Domenico Grieco
Dipartimento di Informatica, Università di Bari
70126 Bari, Italy
@uniba.it
Abstract. Ambient Assisted Living aims at developing systems that assist
people in their daily tasks. In this context, assistance and care are delegated to
the intelligence embedded in the environment that, in our opinion, should
provide not only a task-oriented support but also an interface able to establish a
social empathic relation with the user. This is what we call a “caring home”. In
this paper we report our experience in designing a caring home environment
that combines C@sa, a multi-agent system for managing the smart home
behavior, and NICA, a conversational interface embodied in a social robot able
to recognize the user attitude and to provide the appropriate social response.
1 Introduction
One of the research goals in the context of Ambient Assisted Living (AAL) concerns
the integration of new technologies with the social environment to support people in
their daily activities and increase their quality of life [1,2,3]. The use of intelligent
technologies to support people at home has been addressed in several AAL research
projects [4]. For instance, the amiCA system aims at increasing the quality of life by
means of un-obtrusive sensors. The SOPRANO project aims at developing highly in-
novative smart services with natural and comfortable interfaces for ageing people in
order to support independent living. Finally, the PERSONA project also aims to help
the elderly at home to cope with the loss of skills due to the normal ageing process, by
providing a set of services supporting social inclusion, daily life activities, health mo-
nitoring and risk prevention. The main focus of these projects is to develop technolo-
gical platforms that allow a natural and pleasant interaction with a smart environment,
by implementing an easy access to its services. As far as interaction is concerned, as-
sistance may be provided to the user in a seamless way (i.e. by combining smart home
technologies based on sensors and effectors embedded in the appliances of the envi-
ronment), using an embodied companion as an interface, or combining both approa-
ches. In all cases, research emphasizes the need of natural and user-friendly interfaces
for accessing services provided by the environment. In our opinion, an assistive home
environment should provide not only a task-oriented support but also an interface able
to establish a social empathic relation with the user. This is what we call a “caring
home”. Achieving this objective requires developing:
- Methods and models for defining and developing Ambient Intelligence (AmI) sy-
stems for Assisted Living that are able to define environments that, exploiting
multi-agent techniques, manage devices and services autonomously and proactive-
ly with respect to the needs of the users populating the environment. In particular,
the environment must be able to learn user behaviors and control physical devices
placed in an environment (home,office, etc.) so as to improve their comfort.
- Methods and models for analysis of the user behavior with particular emphasis on
affective aspects in order to reach personalization, adaptation, proactivity that are
typical of an AmI system. This model should be integrated with the multi-agent
system in such a way that, by sensing the most significant parameters and by tran-
smitting their values to the system, suitable inferential strategies are applied on the
available models to assess the distance between the current situation and the situa-
tion presumably desired by the user or needed to guarantee her aims and to satisfy
her needs, recognizing her emotional state, and, based on all these elements, plan a
set of actions, to be performed by suitable effectors, that are useful to improve the
situation. The model should be refined incrementally by analyzing the user's feed-
back and reactions to improve performance in similar situations. The whole reaso-
ning activity of the system must be traced, and exhibited upon request to explain
its behavior. This will improve the degree of trust of the user in the system, resul-
ting in a better and more natural interaction that will simplify subsequent activity
of the system.
- Natural Interaction of the user with the information and services offered by the
system. Such an interface has two fundamental and interconnected objectives:
being a means to interact with the environment and being, for the user, a friendly
caring agent. For this reason it is important to understand not only the meaning of
the communication but also the conveyed emotions and the user’s attitude during
the interaction. This requires the emotional analysis of the linguistic and prosodic
aspects of the user's vocal input, of her facial expressions and gestures.
In this paper we propose the integration of a social empathic agent, acting as a vir-
tual caregiver, in C@sa, an agent based architecture for handling the smart behavior
of the home environment [5,6]. The choice of an embodied agent as an interaction
metaphor is driven by the following considerations. If properly designed, social and
conversational agents may improve the naturalness and effectiveness of the interac-
tion between users and smart environment services [7,8]. They have the potential to
involve users in a human-like conversation using verbal and non-verbal signals for
providing feedback, showing empathy and emotions in their behavior [9,10]. Indeed,
several studies report successful results on how expressive conversational agents and
robots can be employed as an interaction metaphor in the assisted-living domain and
in other ones [11,12,13] where it is important to settle long-term relations with the
user [14]. For instance, projects ROBOCARE [15], Nursebot [16], Care-o-bot [17],
CompaniAble [18] and KSERA [39] aim at creating assistive intelligent environments
in which robots offer support to the elderly at home, possibly having also a compa-
nion role. Indeed, the results of several studies, conducted to investigate the hu-
man-robot interaction, show how robots can be successfully employed as a good inte-
raction metaphor when they act in the role of assistants, companions, therapeutic and
socially assistive robots [19]. For example, van Ruiten et al. [20] conducted a control-
led study using I-Cat, a robot developed in order to study personal robotic applica-
tions and human-robot interactions [21]. They confirmed the results, shown in [22],
about the fact that elderly users like to interact with a social robot and to establish a
relation with it. The reason of the success of socially intelligent agents is probably due
to the fact that interaction between human and machine has a fundamental social com-
ponent [23].
The paper is structured as follows: in Section 2 we briefly describe the C@sa mul-
tiagent architecture, and in Section 3 we illustrate the main features of NICA. Then, in
Section 4 we provide an example on how NICA can serve as a caring agent of C@sa;
finally we conclude the paper with discussion and directions for future work.
2 An overview of C@sa
A smart environment should observe lifestyle and desires of its inhabitants to learn
how to anticipate and accommodate their needs by using Machine Learning
techniques [24]. The environment, then, must be able to reason on the situation of the
user so as to understand his/her needs and goals and satisfy them through the
composition of the most appropriate services. To enforce this view we have developed
C@sa [5,6], a Multi Agent System (MAS) based on the metaphor of the butler in
grand houses, whose architecture we briefly recall in the following.
Fig. 1. The MAS architecture
As its main tasks, the butler must know the habits of the house inhabitants,
perceive the situation of the house, and coordinate the housestaff. To this aim we have
designed the following classes of agents:
- Sensor Agents (SA): are used for providing information about context parameters
and features (e.g., temperature, light level, humidity, etc.) at a higher abstraction
level than raw sensor data.
- Butler Agent (BA): reasons on the user’s goals and devises the workflow to sati-
sfy them [12] (see Figure 1).
- Effector Agents (EA): each appliance and device is controlled by an EA that rea-
sons on the opportunity of performing an action instead of another in the current
context.
- Interactor Agent (IA): is in charge of handling interaction with the user in order
to carry on communicative tasks.
- Housekeeper Agent (HA): acts as a facilitator since it knows all the agents that
are active in the house and also the goal they are able to pursue.
All agents are endowed with two
main behaviors, reasoning and
learning (see Figure 2). Although all
agents share the same architecture,
they differ in level of complexity,
techniques that can be exploited by
the reasoning functionality, tools that
implement the techniques, and
theories used for reasoning. Of
course, different agents work on
different portions of knowledge on the
domain and may require different Fig. 2. The architecture underlying the C@sa agents
effort and pose different problems.
Reasoning uses the agent’s knowledge to perform inferences that determine how the
agent achieves its objectives. Learning exploits possible feedback on the agent’s
decisions to improve that knowledge, making the agent adaptive to the specific user
needs and to their evolution in time. The learning behavior returns new knowledge
gained from experience, that extends or refines the model on which the reasoning
behavior is based. The reasoning behavior mixes mathematical and statistical
processing techniques with more powerful kinds of reasoning and knowledge
management based on First-Order Logic, in order to handle the complexity of
real-world environments where relationships among several entities and possible
situations play a significant role. The input to an agent is processed by its reasoning
layer, for:
- deciding which signals are to be ignored and which ones are to be sent to other en-
tities that can understand and exploit them (e.g. agents or user or devices, depen-
ding on the kind of agent) and/or to its learning functionality.
- processing and combining input data to detect significant patterns and produce
more complex information, using different kinds of inference techniques.
- deciding which part of this information is to be ignored and which part is to be
forwarded to other entities (see above) and/or to the learning functionality.
The learning behavior, on the other hand, is used by an agent to refine and improve
its future performance. For the specific needs of adaptation posed by the present
application, an incremental approach to learning new information is mandatory,
because the continuous availability of new data and the evolving environment require
continuous adaptation and refinement of the available knowledge. An incremental ILP
system that is able to exploit different kinds of inference strategies (induction,
deduction, abduction, abstraction), and hence fits the above requirements, is
InTheLEx [25]; also abstraction and abduction theories can be learned automatically
[26]. The main inference strategy that characterizes the learning layer of our agents is
induction, although a cooperation with other strategies, such as those exploited in the
reasoning behavior, is strongly advised, for a better integration of the new knowledge
with the reasoning engine.
3 NICA: a Socially Intelligent Caring Agent
In order to show how the considerations outlined in the Introduction can be
successfully employed in designing and implementing a social and empathic virtual
caregiver [27, 28], we have developed a caring agent named NICA (Natural
Interaction with a Caring Agent). In order to provide assistance and, at the same time,
to settle a social long-term relation with the user, NICA, starting from the multimodal
analysis of the user behavior (see [29] for more details on the recognition framework)
provides proactively and reactively the needed assistance by acting as a social
conversational interface between the user and the home services. To this aim, NICA
has to: i) start from the interpretation of the user multimodal input; ii) reason on the
information the user intends to convey (emotion, social attitude, performative,
content, etc.) and then trigger communicative goals according to the current belief
representation of the state of the world; iii) achieve these goals through a set of
communicative plans (“what to say”) that can then be rendered as a combination of
voice and animations of the agent’s body (“how to say”); iv) keep in its social
memory information about which are the antecedents of emotions for the user, that is
what triggers the emotions (events, situations, thoughts, etc.).
In order to handle dialogs in a dynamic environment, NICA has been modeled as a
BDI (Belief, Desire, Intention) agent and interleaves reactive and deliberative
behaviors [30].
Fig. 3. NICA’s architecture proposal
For taking care of the user, NICA implements a life-cycle based on the following
steps (see Figure 3):
- Perception: allows collecting data from sensors present in the environment and
to handle the user input (speech, gestures, facial expressions or actions in the
environment).
- Interpretation: evaluates changes in the world and user’s state that are relevant to
the agent’s reasoning and transforms them into a set of agent’s beliefs. In particular it
interprets the user’s input.
- Goal Activation: conversational goals are triggered based on the current beliefs.
- Planning and Execution: once a goal has been triggered it is achieved through
the execution of a communicative plan appropriate to the situation.
In order to adapt the robot’s behavior to the user’s needs and preferences, NICA’s
mental state reasons on and stores different types of knowledge:
- the World Model that represents a set of relevant beliefs about the current
environment context.
- the User Model that contains the representation of beliefs of various type. In
particular, we model long-term factors concerning stable user data (i.e. sex, age,
chronic diseases, allergies, main personality traits, interests, etc.) and short-term
factors concerning belief about the current user situation, affective state, etc.
- the Conversational Resources to be used to handle the dialog.
- the Agent Social Memory stores structured information about feelings associated
with events. It is used to remember relations about events and the user affective state.
The importance of this piece of knowledge in the agent’s mind is related to the need
of establishing empathy with the elderly person and this was outlined several times by
the human caregivers during the data collection phase.
As the agent reasons and updates its beliefs, infers goals, plans conversational
behaviors and executes them, it keeps an image of this process in its mental state. In
order to deal with the uncertainty typical of this domain (e.g. dealing with exceptional
situations or with the smooth evolution of the user’s affective state over time), we
employ probabilistic models to reason on the user and to decide which behavior to
adopt, that is the most appropriate set of actions to perform for satisfying the inferred
user’s goal. At the moment, as shown in Figure 4, we simulate the interaction between
NICA, the user and the environment in a toy house equipped with a robot (Lego
MindStorm with sensors for detecting its position in the house), light, temperature and
presence sensors, and a microphone for capturing the user’s voice.
Fig. 4. A simulation of NICA’s dialog capability in a toy house
Moreover, we developed an interface for setting some parameters concerning the
world state and some other user data that at the moment we are not capturing in real
time (i.e. physical parameters such as fever, blood pressure, etc.). We do not consider
this as a strong limitation of our approach since many wearable and wireless devices
are coming out on the market and therefore in a real setting we will be able to receive
these data. However, we are aware that in real settings the appearance of Lego
Mindstorm could provoke a negative effect in the user, but before testing the system
in real settings we wanted to be sure that the agent’s mind was reasoning in a reliable
and consistent way and therefore we employed the toy house scenario. Our research
group used the AIBO robot or the conversational agent Valentina [31] in other
projects and they might be used also in this case in future experiment with elderly
users. Moreover, of particular interest to our research is the approach adopted in the
Florence project (http://www.florence-project.eu/). In Florence the Social Robot was
built using low-cost parts that have been assembled in order to allow movements in
the house, communication with the SHE architecture and a friendly conversational
interface. From the implementation point of view, changing the embodiment of our
agent is not a problem since we adopt the mind-body architecture developed in a
previous project [31]. Therefore the plan computed by the “mind” module contains
the meaning to express and the “body” has to convey this meaning according to its
communicative capabilities. For space reasons, in this paper we will focus on the
description of the affective component of the user model. For more details about
NICA, dialog and affect modeling see [11, 29, 31].
3.1 NICA’s User Model
The user’s move is a rich information source that allows extracting knowledge about
her intention, social attitude, emotional state, and so on. In our approach, the user’s
model maintained by the agent allows to reason on the user’s beliefs (i.e. the user’s
move “I love fruit!” will be transformed into the corresponding belief that can be used
to adapt the dialog strategy) and on the user’s attitude during the dialog. Beliefs on
knowledge, preferences and interests of the user are inferred according to an approach
previously employed in another system [31] and to the one about the user’s affective
state that is recognized and monitored with a dynamic model based on Belief Network
(DBN) [32]. In fact, when modeling affective phenomena we have to take into
account the fact that affective states smoothly evolve during the interaction, from one
step to the subsequent one. As a consequence, the affective state should be monitored
and modeled as a temporal phenomenon, whose value at every time of the interaction
depends on the value it assumes in the previous dialogue turn. For this reason, the
DBN formalism is particularly suitable for representing situations that gradually
evolve from a dialog step to the next one.
Figure 5 shows an example of DBN in which low-level beliefs, deriving from per-
ceptions, can be used to infer beliefs about the affective state of the user. In particular,
this network can be used to infer the probability that the user is in a negative, neutral
or positive affective state or the probability that the user feels a particular emotion. In
this model we consider only sadness, anxiety, anger and happiness that are relevant
for the purpose of the system since they were reported in the diary of the human care -
givers.
Fig. 5 Inferring low-level beliefs from perceptions’
This dynamic model allows us to take into account the influence of the user’s sta-
te at the previous step. For instance, in the DBN in Figure 5 this is expressed by a
temporal link between the Bel(AffectiveState)Prev variable and the Bel(AffectiveSta-
te). Analogously, the evidences and the probability values of the root nodes of the BN
may be extracted from other modules. This allows to manage the complexity of the
network and to integrate in the model evidence deriving from different modules per-
forming different and independent analysis.
For example, the Voice node in Figure 5 is evaluated according to the results of
the acoustic analysis of the user’s utterances. Research in emotional speech has shown
that acoustic and prosodic features can be extracted from the speech signal and used
to develop models for recognizing emotions and attitudes [33] In fact, the effects of
emotion in speech tend to alter the pitch, timing, voice quality, and articulation of the
speech signal and reliable acoustic features can be extracted from speech that vary
with the speaker's affective state. We used Praat functions in order to perform a ma-
cro-prosodic or global analysis and to extract from the audio file of each move featu-
res related to the variation of the fundamental frequency (f0), energy (RM), harmoni-
city, central spectral moment and speech rate.
The sensors responsible for capturing the speech classify the user’s affective state
and attitude towards the system, adopting an approach analogous to the one described
in [34]. At present, our classifier exploits the Nearest Neighbor with generalization
(NNge) algorithm and recognizes the valence with an accuracy of 85%, evaluated on
a dataset of 4 speakers and 748 user moves overall. The accuracy of the classifer has
been validated with a 10-fold Cross Validation technique.
The aim of the model described in Figure 5 is twofold: on one hand the model is
employed to guess which specific emotional state the user is experiencing at every
step of the interaction; on the other hand it is used to monitor the overall evolution of
the user’s affective state (i.e. the agent’s belief about the positive or negative affective
state of the user). In particular, every time a new user move is entered, its linguistic
and acoustic features are analyzed with respect to the context variable and the resul-
ting evidence is introduced and propagated in the network to recognize the user’s
emotion and the overall polarity of her affective state. The new probabilities of indivi-
dual emotions are read and contribute to formulate the next move of the agent; the
probability of the dynamic variable (Bel(AffectiveState)), representing the valence of
user’s affective state, is used by the agent to check the consistency between its persi-
stent goal of maintaining the user in a positive affective state and the actual emotional
state the user is in at time t, thus causing the activation or the revision of high-level
planning of the agent’s behavior. Then, the values of the node Vocal Input derive from
a module that integrates the linguistic content of the user move with the recognition of
its acoustic features to recognize the actual user’s communicative goal [34].
At present we concentrated on spoken interaction and we are not working at the re-
cognition of facial expressions, gestures, postures, and so on. For this reason the va-
lues of the Physical Situation and Emotional Action nodes of the network are simula-
ted using an interface for setting parameters for running a scenario simulation like the
one that will be described later on.
Starting from what has been inferred by the user model component, the dialog
management module computes the agent’s move using a strategy based on the
information state approach [35]. It represents and stores beliefs about the current state
of the world, the user, the dialog, the dialog history, the current dialog move and the
move scheduled for execution.
The dialog manager, and in particular the deliberative module, decides which goals
to trigger and to pursue during the dialog, starting from the interpretation of the user’s
move and the recognized affective state. In order to handle interruptions, variation of
the user’s situation (for instance the recognized emotion), the agent has a Reactive
Layer. The idea is that the agent has an initial list of goals, each with its priority, some
of which are inactive: every goal is linked, by an application condition, to a plan that
the agent can perform to achieve it. The communicative actions corresponding to acti-
ve plans are put in the agenda maintained by the information state. The agent starts
the dialog by executing these actions but, as we said in the Introduction, the agent ap-
plies some form reasoning on the user’s move. The recognized social attitude and the
emotion triggered in the agent’s mind are used to implement social and emotion-based
dynamic revision of goals and consequently of the dialog. To achieve the selected
communicative goal we use plans represented as context-adapted recipes.
With these rules, we formalize a situation of empathic reaction in which the agent
temporarily substitutes the presumed goals of the user for its own, when these goals
are due to an emotional state of the user. If an undesirable event occurs to the users,
what they are presumed to need is to be convinced that the agent understands the
situation and does its best to solve the problem. If something desirable occurs to them,
they need to know that the agent shares their positive experience. If, on the contrary,
the undesirable event does not concern the users, they probably want to be sure that
this will not interfere negatively with the dialog.
4 NICA as a Social Interface of C@sa
Let us now provide an example of how the proposed systems can be integrated in or-
der to support a “caring home” for assisted living. Consider the following scenario,
depicted from the analysis of data from the diaries that we collected with human care -
givers and elderly people [11]. More details on the formalization of this scenario can
be found in [5].
It’s morning and Maria, a 73 y.o. woman, is at home alone. She has a cold and fe-
ver. She is a bit sad since she cannot go to the market downtown and talk with her
friends, like she does every morning. Maria is sitting on the bench in her living room
in front of the TV. The living room is equipped with sensors, which can catch
sound/noise in the air, time, temperature, status of the window (open/close) and of the
radio and TV (on/off), and the current activity of the user, and with effectors, acting
and controlling windows, radio and TV and also the execution of digital services that
may be visualized on communication devices. According to the situation, the BA infers
possible user’s goals and triggers the appropriate workflow whose tasks are executed
by effectors agents. When a communicative or an interaction task is required, NICA,
that is close to the user, acts as an IA. While the selected workflow is executed, NICA
has to check Maria’s health state and recommend some medicine to her. After a while
Maria starts whispering and moaning and says: “Oh My…oh my…”. This utterance
is perceived by NICA that interprets it and activates the most appropriate emphatic
behavior.
Let’s see how this scenario is simulated in our system.
The physical sensors send in real-time the values they gather to the reasoning be-
havior of the associated SA, which uses abstraction to strip off details that are useless
for the specific current tasks and objectives. For instance, the SA providing informa-
tion about temperature will abstract the centigrade value into a higher level represen -
tation such as “warm”, “cold”, and so on. This abstraction process may be done accor-
ding to the observed specific user’s needs and preferences (e.g. the same temperature
might be cold for a user but acceptable for another). For instance, let us denote the
fact that the user Y is cold in a given situation X with cold(X,Y). This fact can be deri-
ved from the specific temperature using a rule of the form:
cold(X,Y) :- temperature(X,T), T<18, user(Y), present(X,Y), maria(Y).
(it is cold for user Maria if she is present in a situation in which the temperature is lo -
wer than 18 degrees). In turn, the above rule can be directly provided by an expert (or
by the user herself), or can be learned (and possibly later refined) directly from obser-
vation of user interaction [36]. For instance, assume that the following events have
been recorded in the past:
temperature 28 16 8 20 32 18 37 26 22 19 29 23 12 25 4
action C H H - C H C C - - C - H - H
where the first row reports a set of temperatures sensed in situations where Maria was
present, and the second row reports her action in those situations (C = cooling, H =
heating, - = no action). Then, the SA controlling the temperature may automatically
learn that the user turns on heating (i.e., she is cold) whenever the temperature is be-
low 19 degrees, and turns on the cooling system (presumably because she is warm)
whenever the temperature is above 25 degrees:
cold(X,Y) :- temperature(X,T), T<19, user(Y), present(X,Y), maria(Y).
warm(X,Y) :- temperature(X,T), T>25, user(Y), present(X,Y), maria(Y).
Starting from percepts received by SAs, the BA infers user goals and selects a work-
flow that integrates elementary services suitable for the particular situation. Situations
can be formally described as conjunctive logic formulas under the Closed World As-
sumption (what is not explicitly stated is assumed to be false). A model consists of a set
of Horn clauses whose heads describe the target concepts and whose bodies describe
the pre-conditions for those targets to be detected. A very simple model might be:
improveHealth(X) :- present(X,Y), user(Y), has_fever(Y).
improveHealth(X):- present(X,Y), user(Y), has_headache(Y), cold(X,Y).
improveHealth(X) :- present(X,Y), user(Y), has_flu(Y).
improveMind(X) :- present(X,Y), user(Y), sad(Y).
improveMind(X) :- present(X,Y), user(Y), bored(Y).
meaning “A user Y that is present in situation X and has a fever, or has a headache and
has a cold, or has a flu, might want to improve his health” and “A user Y that is present
in situation X and is sad or bored might want to improve his mind”, respectively. A
sample observation might be:
morning(t0),closedWindow(t0),present(t0,m),maria(m),user(m),
temperature(t0,14), has_flu(m), sad(m).
(i.e., “in situation at time t0 it is morning, the window is closed and the temperature is
14°; user Maria is present and she has flu”). Reasoning infers that Maria is cold:
cold(t0,m). Being all the preconditions of the third and fourth rules in the model
satisfied by this situation for X = t0 and Y = m, the user goals improveHealth and
improveMind are recognized for Maria at time t0, which may cause activation of
suitable workflows aimed at attaining those results. Conversely, the other rules in the
model are not satisfied – e.g., considering the last rule, user Maria is present, but she is
assumed not to be bored. Although predicates such as fever(X), headache(X) and flu(X)
are already abstractions of the specific value provided by SAs, further levels of
generalization can be automatically performed by the reasoning layer, e.g. using a
predicate has_disease(Y), defined as
has_disease(X) :- has_fever(X).
has_disease(X) :- has_flu(X).
such that the first and third rule in the model can be reduced to:
improveHealth(X) :- present(X,Y), user(Y), has_disease(Y).
making it applicable to other kinds of diseases, in addition to just fever and flu. Re-
ferring back to the previous observation, the reasoning behavior would infer that Maria
has a disease – has_disease(m) – from the fact that she has a flu.
The BA reasons not only on goals but also on workflows. Indeed, once a goal is
triggered, it selects the appropriate workflow by performing a semantic matchmaking
between the semantic IOPE (Input Output Preconditions Effects) description of the
user's high-level goal and the semantic profiles of all the workflows available in the
knowledge base of the system [37]. As a result, this process will return a (possibly
empty) set of workflows that are semantically consistent with the goal (possibly ranked
by a function of semantic similarity with the goal). For instance, in Figure 6 the
semantic matchmaking process leads to two different workflows associated,
respectively, to the two high-level goals improveHealth and improveMind previously
recognized. The semantic matchmaking process starts from these goals and leads to the
desired workflow.
Fig 6. An example of a Smart Service Workflow composed by the Butler Agent
Complex workflows may involve both simple actions and other sub-goals corre-
sponding to subflows, which are again processed according to the matchmaking phase
described above. In our case, the main workflow includes two goals that need to be
executed by selecting two different subflows corresponding, respectively, to improve-
Health and improveMind. These subflows include both simple actions, that can be di-
rectly executed, and subflows that need to be satisfied, such as setTemperature. In turn,
the subgoal setTemperature is satisfied by applying once more the matching process to
find a suitable workflow. Using this workflow, the reasoning behavior of the BA will
process the information collected by the temperature sensors in order to understand
whether to raise or reduce the environment temperature:
doReduceTemperature(X) :- present(X,Y), user(Y), warm(X,Y).
doRaiseTemperature(X) :- present(X,Y), user(Y), cold(X,Y).
This hierarchical matchmaking process stops when the resulting workflow is com-
posed of goals that can be directly satisfied by invoking a net-centric service or throu-
gh simple actions performed on the effectors. In both cases, the BA asks to the HA
which EAs can satisfy each planned action and sends the specific request to the EA in
charge for handling actions regarding changes of a particular parameter (i.e. tempera-
ture, light, etc.). In particular, when the goal regards a communicative action, its exe-
cution is delegated to the IAs, and the HA returns to the BA the list of agents that are
responsible for implementing the interaction with the user through different modali-
ties (e.g. on a touch screen, on the smartphone or by using the social robot).
In order to find the best solution to satisfy the user needs, EAs reason about differ-
ent possible solutions to attain the same goal in the current context. For instance, if
the goal is reducing the temperature, the EA in charge of temperature control may de-
cide whether turning on air conditioning or opening the window; additionally, it de-
cides how to control those devices (in the former case, which fan speed to select; in
the latter case, how widely the window must be opened).
When the environment has to perform a communicative task it delegates it to the
IA, that in our case is instantiated in NICA. There are several communicative goals
that this agent may carry out. In Information Seeking the IA exploits interaction with
the user to get hints on how to attain a simple goal and, based on this, possibly learns
new preferences of the user with respect to the given context and situation, in order to
continuously and dynamically improve adaptation. Information Providing is used to
answer and provide explanations about the SHE appliances behavior or about the de-
cision to include some specific subflows in the main workflow built by the BA. For
example, referring to the previous scenario, the user may ask the robot to provide
more information to justify the choice for a given medicine. Through Remind NICA
may remind to the user to perfom an action or about an appointment, and so on. So-
cial and Affective Support is used for communications.
In the following we refer to the previously described scenario and we propose an
example of interaction. In particular the subflow called ‘Medicine Recommender’ in-
cluded in the workflow in Figure 6. It may be satisfied differently according to the
specific situation of the user. Let’s suppose, for example, that the information about
the user’s disease is not complete, that is the BA knows that the user needs to improve
her health because she has got a flu but it is necessary to decide which drug is the
most suitable for her to take. In such a situation, it is necessary to further investigate
the user’s physical state in order to select the most appropriate drug to suggest to him.
In the following we provide an example of interaction with the social intelligent
robot present in the house.
NICA: ‘Hi Maria, how are you doing?’
Maria: ‘I have got flu, I think I need some medicine’
NICA: ‘Let’s check the fever’.
NICA: ‘You have 37°C. I would suggest you to take a pill of paracetamol’
Maria: ‘Are you sure? I don’t feel like having fever, I rather have some headache and sore
throat’
NICA: ‘Well, paracetamol is ok for headache too. But, if you prefer, you can take a pill of
anti-inflammatory, which could solve both problems’
… after a while
Maria: Oh My …oh my…
NICA: Oh I’m sorry to hear that you are anxious. Why do you feel so?
Maria: I’m worried for my health.
NICA: Don’t worry, you have just a flu, you will soon feel good. If you won’t feel better
by tomorrow I will call the doctor.
The last utterance by Maria is recognized as having the meaning expressing an af-
fective state and its prosody identifies a negative state and, in particular, the recogni-
zed emotion is sadness. NICA does not know what caused sadness in Maria. The ini-
tial probability values of the nodes Voice and Vocal Input are derived automatically by
the speech recognition module of NICA, while the setting of the other values of the
root nodes in the DBN is done through the framework interface. This evidence is pro-
pagated in the DBN and the belief about the affective state of the user has a negative
valence with a high probability (66,89) (see Figure 5). This belief triggers “encoura-
ge” as the most convenient goal to pursue in this situation, since it is the one with the
highest probability. Then, the most appropriate plan is selected according to its pre-
condition and the execution of its communicative actions begins. The plan includes
the following actions, since NICA does not know the reason for that affective state
and it will ask the user about it:
MoveTo(N,MARIA)
Express(N,Sorry-for(N,MARIA))
Ask(N,MARIA,Why(MARIA,Anxious))
Express(N,MARIA,Encurage(N,MARIA))
5 Conclusions and Future Work
We presented a preliminary work towards the integration of a MAS aimed at handling
the situation-aware adaptation of a Smart Environment behavior and NICA, a Social
Robot for handling interaction in the SHE. In the MAS different types of agents co-
operate to the adaptation process: Sensor Agents, a Butler Agent, a Housekeeper, Ef-
fector and Interactor Agents. The main peculiarity of the proposed architecture lies in
the fact that all kinds of agents in the MAS are a specialization of an abstract class en-
dowed with both reasoning and learning behavior. In particular, interaction with the
user is implemented using NICA, a conversational social interface to the SHE ser-
vices. In our opinion, besides assisting the elderly user in performing tasks, the agent
has to establish a social long-term relationship with the user in order to enforce trust
and confidence. The underlying idea of our work, in fact, is that endowing the agent
with a socially intelligent behavior is fundamental when the devices of a smart home
are integrated pervasively in everyday life environments. As an example, we illustra-
ted how NICA, a social agent acting as a caring assistant for elderly people living in a
smart home, has been developed. At present we evaluated the agent’s behavior using a
toy house, and hence we could not evaluate the effectiveness of the communication
with real elderly users, which requires a smart home environment and a suitable social
robot like for instance the Nao [40]. This kind of experiment should aim at assessing
the impact of the use of a social robot on seamless interaction with the environment
services. Another important issue to be addressed in our future work concerns the
interpretation of multimodal human communicative actions in order to recognize the
user’s attitude. In [29] we started working on the recognition of the user’s social
attitude from a combination of spoken and gestural communication using a
probabilistic approach that allows accommodating for the uncertainty typical of this
domain. Moreover we are developing a model for giving NICA the capability to infer
the user’s activity in the house [38].
Acknowledgements
This work fulfils the research objectives of the PON02_00563_3489339 project "PUGLIA@-
SERVICE - Internet-based Service Engineering enabling Smart Territory structural develop-
ment" funded by the Italian Ministry of University and Research (MIUR).
References
1. Bierhoff, I. and van Berlo, A.: More Intelligent Smart Houses for Better Care and Health,
Global Telemedicine and eHealth Updates: Knowledge Resources,vol. 1, 322-325, 2008.
2. McFadden T. and Indulska J.: Context-aware Environments for Independent Living, in
Proceedings of 3rd National Conference of Emerging Researchers in Ageing 2004.
3. Steg, H. et al.: Ambient Assisted Living – European overview report, September, 2005 14
4. http://www.aal-europe.eu/wp-content/uploads/2012/04/AALCatalogue_onlineV4.pdf
5. Ferilli S., Cavone D., De Carolis B. and Novielli N.: A Layered Architecture for Situation
Aware Home Environments, to appear in Proceedings of 6th Workshop on "Artificial
Intelligence Techniques for Ambient Intelligence" (AITAmI'11), 2011.
6. De Carolis B., Cozzolongo G. and Pizzutilo S.: A Butler Agent for Personalized House
Control. ISMIS 2006: 157-166, 2006.
7. M. Dehn and Mulken S.V.: The Impact of Animated Interface Agents: A Review of
Empirical Research. International Journal of Human-Computer Studies, 52, pp. 1-22, 2000.
8. Brahnam S. and De Angeli, A.: Editorial - Special issue on the abuse and misuse of social
agents. Interacting with computers, 20 (3), 2008.
9. Niewiadomski R., Ochs M. and Pelachaud C.: Expressions of Empathy in ECAs, in
Proceedings of the 8t Int. Conf. on IVA, LNAI, vol. 5208. Springer-Verlag, Berlin,
Heidelberg, pp. 37-44, 2008.
10. Reilly W.S.N.: Believable Social and Emotional Agents. PhD thesis, 1996.
11. De Carolis B., Mazzotta I., Novielli N., Pizzutilo S.: User Modeling in Social Interaction
with a Caring Agent. In: UMADR - User Modeling and Adaptation for Daily Routines,
Springer HCI Series. Springer, 2013.
12. Nijholt A., de Ruyter B., Heylen D. and Privender S.: Social Interfaces for Ambient
Intelligence Environments. Chapter 14 in: True Visions: The Emergence of Ambient
Intelligence. E. Aarts and J. Encarnaçao (Eds.), 275-289, Springer, 2006.
13. Ortiz A., Del Puy Carretero M., Oyarzun D., Yanguas J.J., Buiza C., Gonzalez M.F. and
Etxeberria I.: Elderly users in ambient intelligence: does an avatar improve the interaction?.
In Proceedings of the 9th conference on User interfaces for all (ERCIM'06), C. Stephanidis
and M. Pieper (Eds.), 99-114, Springer-Verlag, 2006.
14. Bickmore T. and Picard R.W.: Establishing and maintaining long-term human-computer
relationships, ACM Transactions on Computer Human Interaction, 12, 2, 293-327, 2005.
15. Cesta A., Cortellessa G., Pecora F. and Rasconi R.: Supporting Interaction in the RoboCare
Intelligent Assistive Environment, AAAI 2007 Spring Symposium, 2007.
16. Pineau J. , Montemerlo M., Pollack M., Roy N. and Thrun S.: Towards Robotic Assistants
in Nursing Homes: Challenges and Results, Robotics and Autonomous Systems
42(3-4):271-281, 2003.
17. Graf B., Hans M., Schraft R.D.: Care-O-bot II – development of a next generation robotics
home assistant. Auton. Robots 16, 193-205, 2004.
18. CompanionAble project (2011) http://www.companionable.net/
19. Fong T., Nourbakhsh I. and Dautenhahn K.: A survey of socially interactive robots,
Robotics and Autonomous Systems 42, 143-166, 2003.
20. van Ruiten A.M., Haitas D., Bingley P., Hoonhout H.C.M., Meerbeek B.W. and Terken
J.M.B.: Attitude of elderly towards a robotic game-and-train- buddy: evaluation of empathy
and objective control. In R. Cowie and F. de Rosis (Eds.) Proceedings of the Doctoral
consortium, in the scope of ACII2007 Conference, 2007.
21. van Breemen J.N.: iCat: a generic platform for studying personal robot applications. Paper
presented at the IEEE SMC, Den Haag, 2004.
22. Broekens J., Heerink M., Rosendal H.: Assistive social robots in elderly care: a review.
Gerontechnology, 8(2):94-103, 2009.
23. Reeves B. and Nass C.: The media equation: how people treat computers, television, and
new media like real people and places, Cambridge University Press, 1996.
24. Muggleton S.H.: Inductive Logic Programming. New Generation Computing,
8(4):295-318, 1991.
25. Esposito F., Fanizzi N., Ferilli S., Basile T.M.A. and Di Mauro N.: Multistrategy Operators
for Relational Learning and Their Cooperation. Fundamenta Informaticae Journal,
69(4):389-409, IOS Press, 2006.
26. Ferilli S., Esposito F., Basile T.M.A. and Di Mauro N.: Automatic Induction of Abduction
and Abstraction Theories from Observations. In: S. Kramer, B. Pfahringer. Inductive Logic
Programming, 103-120, Springer, 2005.
27. Cearreta I., López J.M., López De Ipiña K., Hernandez M., Garay N., Graña M. and
Álvarez A.: Affective Computing as a Component of Ambient Intelligence.
INFORMATION SCIENCES 2007, Proceedings of the 10th Joint Conference, Salt Lake
City, Utah, USA, 18 - 24 July 2007, 1580-1586, 2007.
28. Leite I., Martinho C. and Paiva A.: Social Robots for Long-Term Interaction: A Survey.
International Journal of Social Robotics, 1-18, January, 2013
29. De Carolis B., Ferilli S. and Novielli N.: Recognizing the User Social Attitude in
Multimodal Interaction in Smart Environments. AmI 2012: 240-255, 2012.
30. Castelfranchi C. and Paglieri F., The role of beliefs in goal dynamics: Prolegomena to a
constructive theory of intentions, Synthese 155, 237-263, 2007.
31. De Carolis B., Mazzotta I. and Novielli N.: Enhancing Conversational Access to
Information through a Socially Intelligent Agent. In G. Armano, M. de Gemmis, G.
Semeraro and E. Vargiu (Eds): Intelligent Information Access, Studies in Computational
Intelligence 301, 1-20, Springer, 2010.
32. Jensen F.V.: Bayesian Networks and Decision Graphs, Statistics for Engineering and
Information Science, Springer, 2001.
33. Sundberg J., Patel S., Björkner E. and Scherer K.R.: Interdependencies among voice source
parameters in emotional speech. IEEE Transactions on Affective Computing, 2(3), 2011.
34. De Carolis B. and Cozzolongo G.: Interpretation of user’s feedback in human-robot
interaction, Journal of Physical Agents, 3(2):47-58, 2009.
35. Larsson S. and Traum D.R.: Information state and dialogue management in the TRINDI
dialogue move engine toolkit. Nat. Lang. Eng. 6, 3-4, 323-340, 2000.
36. Ferilli S., Basile T.M.A., Di Mauro N., Esposito F.: On the LearnAbility of Abstraction
Theories from Observations for Relational Learning. In: J. Gama, R. Camacho, G. Gerig, P.
Brazdil, A. Jorge, L. Torgo. Machine Learning: ECML 2005, 120-132, Springer, 2005.
37. Meyer H.: On the Semantics of Service Compositions. Lecture Notes in Computer Science
4524, 31-42, 2007.
38. Ferilli S., De Carolis B. and Redavid D.: Logic-based Incremental Process Mining in Smart
Environments. In M. Ali, T. Bosse, K.V. Hindriks, M. Hoogendoorn, C.M. Jonker, J. Treur
(Eds.), Recent Trends in Artificial Intelligence (IEA-AIE 2013), Lecture Notes in Artificial
Intelligence 7906, 392-401, Springer, 2013.
39. ksera.ieis.tue.nl/
40. http://www.aldebaran-robotics.com/ (last visited Oct 3, 2012).