=Paper=
{{Paper
|id=Vol-2352/paper8
|storemode=property
|title=A Knowledge-based Planning and Acting System for Assistive Robots
|pdfUrl=https://ceur-ws.org/Vol-2352/short8.pdf
|volume=Vol-2352
|authors=Amedeo Cesta,Gabriella Cortellessa,Andrea Orlandini,Alessandro Umbrico
|dblpUrl=https://dblp.org/rec/conf/aiia/CestaCOU18a
}}
==A Knowledge-based Planning and Acting System for Assistive Robots==
https://ceur-ws.org/Vol-2352/short8.pdf
A Knowledge-based Planning and Acting System for
Assistive Robots
Amedeo Cesta, Gabriella Cortellessa, Andrea Orlandini, and Alessandro Umbrico
Institute of Cognitive Sciences and Technologies,
CNR – National Research Council of Italy, Rome
Email: {name.surname}@istc.cnr.it
Abstract. In this work, we present a Knowledge-based Planning and Acting ar-
chitecture to control assistive robots pursuing the integrating of knowledge rep-
resentation and reasoning techniques with automated planning and acting solu-
tions. The architecture aims to endow assistive robots with some proactivity and
self-configuration capabilities needed to deploy them in domestic environment to
providing an effective and useful support to elderly people for independent living.
1 Introduction
The recent advancements in robotic technologies are fostering the development of new
Artificial Intelligence (AI) research solutions to deploy robots in assistive scenarios and
allow them to dynamically interact with humans living in their own domestic environ-
ment. In general, autonomous robots should be able to represent and reason over a wide
set of information constituting the knowledge about the application scenario, the needs
of the human user, etc. while acting in the environment. The knowledge structure of a
robot depends on the particular application context and objectives as well as the partic-
ular behavior a robot must achieve. Our long-term research objective aims at designing
“companion robots" capable of taking autonomous decisions to support older persons
in their living environments through effective and safe interactions. We focus on sce-
narios where a senior user, with mild cognitive and/or physical impairments, lives in
her home with the need of continuous assistance from a personal robotic assistant. In
this regard, a robot acting in this scenario should (i) acquire information about the user
and the environment via a sensor network, (ii) analyze and reason over such informa-
tion and (iii) proactively take decisions to support with continuity a user in her daily
home-living activities. We are developing such reasoning capabilities for assistive tasks
on top of a mobile/telepresence robot. In particular we are exploring the integration of
knowledge representation and reasoning features with automated planning and execu-
tion techniques. As a case study, we consider GiraffPlus-like scenarios [1], a research
project whose aim was to create a sensor-integrated environment to support seniors in
their living environments. The objective of the project was to support prevention and
long-term monitoring as well as to foster social interaction and communication through
a telepresence robot by proposing a solution built around the primary users (i.e., the
seniors) [2]. This paper presents the main features of a novel Planning and Acting ar-
chitecture, called KOaLa (for Knowledge-based cOntinuous Loop) [3, 4], for the syn-
thesis of a continuous “sense-reason-act" control cycle. The pursued control approach
relies on the integration of (i) an abstraction process over the data collected via the sen-
sor network, (ii) a semantic representation and reasoning module to provide the control
system with a Knowledge Base (KB) representing the user, her/her needs and the envi-
ronment conditions and, (iii) a planning and execution module to implement decisional
autonomy. When deployed in a GiraffPlus-like sensorized environment, such modules
should endow an assistive robot with a enhanced level of proactivity and autonomy.
2 KOaLa as a Cognitive Architecture
The KOaLa architecture [4, 3] consists of a knowledge processing module, called the
KOaLa Semantic Module, and a planning and execution module1 , called the KOaLa
Acting Module, that constitute an integrated high-level control loop. Fig. 1 shows a con-
ceptual representation of the envisaged control loop together with the different steps of
the control flow. It starts with data gathered through sensors and it ends with the execu-
tion of actions in the environment to realize the decided supporting tasks. The KOaLa
KOaLa Semantic Module KOaLa Acting Module
KOaLa Ontology
Problem
Formulation Timeline
Goal -based
KB Recognition New Goals Plan
Data Processing Planning & Execution
Sensed Action
Data Execution
Environment / Robot / Sensor Network
Fig. 1: The Semantic and Acting modules of the KOaLa sense-reason-act cycle
Semantic Module is responsible for the interpretation of data gathered from sensors and
the management of the knowledge of the robot. This module relies on the KOaLa Ontol-
ogy [5] (see also Sec. 3) to provide sensor data with semantics and incrementally build
an abstract representation of the application context i.e., the Knowledge Base (KB). It
implements a data processing mechanism which leverages the Web Ontology Language
(OWL) [6] and semantic technologies 2 to continuously refine the KB and infer addi-
tional information (e.g., recognized user activities). Then, a goal recognition process
analyzes the KB to detect specific situations that require the proactive execution of sup-
porting tasks and therefore it dynamically trigger planning goals. The KOaLa Acting
Module is responsible of planning and executing (see Sec. 4) assistive tasks according
to the events or activities inferred by the semantic module. These tasks are encoded as
1
Planning and Acting is also used as interchangeable with Planning and Execution.
2
See the Apache Jena Library - http://jena.apache.org
planning goals by the problem formulation process and, given to a planner in order to
generate a timeline-based plan which describes the sequence of actions that must be
executed. A planning and execution process leverages the timeline-based approach [7]
to continuously refine and update the plan according to the triggered planning goals and
the status of the execution. The executive iteratively dispatches actions to the environ-
ment according to the temporal behaviors of the robot and the sensors encoded by the
timelines of the plan.
3 Ontology and Data Processing
The Semantic Module in Fig. 1 provides the control approach with the cognitive ca-
pabilities needed to interpret data collected from sensors and dynamically build, re-
fine and analyze an internal representation of the user and home environment status
(i.e., the KB). It relies on a dedicated ontology which formally characterize the gen-
eral concepts, entities and properties the system must deal with, and a data processing
mechanism which iteratively refines the KB by interpreting sensor data according to the
ontology. The KOaLa ontology extends the SSN ontology [8] and the DUL ontology 3
and, it is structured according to a context-based approach. Each context characterizes
the knowledge of an agent with respect to a particular perspective and a specifc level
of abstraction. They are organized in a hierarchical way in order to incrementally build
knowledge starting from raw data interpretation. Following a bottom-up approach, (i)
the sensor context, (ii) the environment context and (iii) the observation context have
been defined. The sensor context characterizes the knowledge about the sensing de-
vices that compose a particular environment, their deployment and the properties they
may observe. This context is strictly related to SSN. The aim of the sensor context is
to provide a more detailed representation of the different types of sensor that can com-
pose an environment as well as the different types of property that can be observed.
Such knowledge allows the system to dynamically recognize the actual monitoring ca-
pabilities as well as the set of supporting tasks that can be performed according to the
available sensors and their deployment i.e., the configuration. The environment context
characterizes the knowledge about the structure and physical elements that can compose
a home environment together with sensor deployment. It defines the properties of the
different types of element that can be observed to characterize the state of the environ-
ment. Finally, the observation context characterizes the knowledge about the features
that can actually produce information as well as the events and the activities that can be
observed through them.
4 Goal, Plan, Act
The last step of the data processing mechanism is represented by the Goal Recognition
module (GR) which dynamically generates goals for the Acting Module of the architec-
ture. GR is not responsible for the refinement of the KB, rather it leverages the inferred
KB to connect knowledge representation with planning. It can be seen as a background
3
http://www.loa-cnr.it/ontologies/DUL.owl
process that monitors the updated KB in order to generate operations (i.e., goals) the
system must perform. GR is the key feature of KOaLa to achieve proactivity.
The set of goals the Acting Module can deal with depends on the particular configu-
ration of the environment and the available capabilities. A planning model encapsulates
the knowledge about the capabilities of the controllable elements of the environment
and how such capabilities must be coordinated to realize complex supporting tasks. It
describes the primitive capabilities of assistive robots like e.g., make a call, send a mes-
sage or move to a particular location, as well as the primitive capabilities of sensors
like e.g., turn on, turn off a sensor or set a particular configuration on a sensor. Plan-
ning goals represent supporting tasks that can be performed by properly controlling and
coordinating these primitive capabilities. Such a planning model can be dynamically
configured by analyzing the “static" knowledge about the configuration of the environ-
ment. This means that no observation data is needed and therefore the planning model
can be generated during an initial setup phase. Figure 2 shows the configuration pro-
cess pipeline which generates such a model by leveraging mechanisms similar to those
described in [9] for a reconfigurable manufacturing system.
Primitive Assistive
Configuration Constraint
Capability Functionality
Detection Modeling
Extraction Extraction
Fig. 2: The KOaLa configuration pipeline for planning model inference
The Configuration Detection step extracts the configuration of the environment
from the KB in order to identify the set and types of sensor, their deployment as well
as information about the assistive robot used. Then, the Primitive Capability Extrac-
tion further analyzes these elements in order to extract the primitives representing the
low-level functionalities available to control the environment. Two types of primitives
can be identified: (i) environment primitives; (ii) robot primitives. The environment
primitives represent the capabilities of the elements of the environment that can be con-
trolled. Namely, they characterize the controllable elements of the environment and the
operations that can be performed on and with them. They do not model directly the
sensors of the environment, but rather they model the elements that can be controlled
through the deployed sensors. For example a sensor deployed on the socket where a
TV is plugged in can be used to turn off and on the TV. In such a case, the TV be-
comes controllable and the related turn on and turn off capabilities are part of the en-
vironment primitives. Similarly, the robot primitives represent the capabilities of the
assistive robot available within the environment. They model the the functional layer of
the robot which provides the basic functionalities that must be used to perform assistive
tasks. For example a robot like the GiraffPlus robot provides navigation capabilities that
can be used to move the robot within the environment, messaging capabilities that can
be used to send/receive messages to/from patient’s relatives, videocall capabilities that
can be used to make calls or receive calls with or from doctors and patient’s relatives.
All these functionalities compose the robot primitives. The Assistive Functionality Ex-
traction step extracts the high-level supporting tasks the system can perform. As shown
in the previous section, the data processing mechanism is capable of dynamically infer-
ring the set of events and activities that can be actually monitored/detected according to
the specific configuration of the environment and the properties of the deployed sensors.
Given such knowledge, this step analyzes the KB in order to extract the set of assistive
tasks an assistive robot is actually capable of performing in the considered scenario.
Finally, the Constraint Modeling step finalizes the control model by linking complex
supporting tasks to the environment and robot primitives. Specifically, it leverages the
results of the two previous steps in order to correlate supporting tasks to the primitives
needed to perform them. The result of the described pipeline is a control model that
completely characterizes the high-level supporting tasks an assistive robot can perform
(i.e., the goals generated by the GR) and the constraints that must be satisfied to realize
them and properly coordinate the available primitives.
5 Planning and Execution with PLATINUm
The planning and execution capabilities of the Acting Module rely on PLATINUm
[10, 11], a novel framework which has been successfully applied in real-world man-
ufacturing scenarios [12] and relies on the formal characterization of timeline-based
approach proposed in [7]. The timeline-based approach is a particular temporal plan-
ning paradigm which has been introduced in early 90s [13] by taking inspiration from
the classical Control Theory, and successfully applied in many real-world scenarios
[14, 15, 16]. This planning paradigm aims at controlling a complex system by synthe-
sizing temporal behaviors for a set of identified domain features that must be controlled
over time. According to the formalization proposed in [7], a timeline-based model is
composed by a set of state variables describing the possible temporal behaviors of the
domain features that are relevant from the control perspective. Each state variable spec-
ifies a set of values that represent the states or actions the related feature may assume
or perform over time. Each value is associated with a flexible duration and a control-
lability tag which specifies whether the value is controllable or not. A state transition
function specifies the valid temporal behaviors of a state variable by modeling the al-
lowed sequences of values (i.e., the transitions between the values of a state variable).
State variables model “local" constraints a planner must satisfy to generate valid tempo-
ral behaviors of single features of the domain i.e., valid timelines. It could be necessary
to further constrain the behaviors of state variables in order to coordinate differente
domain features and realize complex functionalities or achieve complex goals (e.g.,
perform assistive functionalities). A dedicated set of rules called synchronization rules
model “global" constraints that a planner must satisfy to build a valid plan. Such rules
can be used also to specify planning goals.
6 Feasibility Evaluation of KOaLa
As a case study, we considered a typical GiraffPlus scenario where a senior person lives
alone in a single floor apartment composed by a kitchen, a bedroom, a bathroom and a
living room. A telepresence robot shares the apartment with the senior user. The robot
is capable of navigating the apartment, interacting with the senior user through gestures
and voice commands as well as making videocalls, sending and reproducing text/audio
messages. A set of sensors capable of gathering information about the temperature,
luminosity and presence are installed inside the apartment. Each room of the apartment
is endowed with one of these sensors in the experimental scenario. There are additional
sensors capable of detecting energy consumptions and they have been installed in order
to detect the usage of particular devices like e.g., a TV or a microwave oven.
The data processing pipeline shown analyzes the configuration of the house to de-
tect the observable features of the environment and the associated observable proper-
ties. According to this information, all the rooms of the environments are classified as
ObservableFeature and the associated properties Temperature, Luminosity
and Presence can be observed. Sensor data is processed by applying inference rules
to recognize events and activities. The semantic module successfully recognizes events
HighLuminosity and HighTemperature when data received by sensors is higher
than a known threshold. Then, tests show that it is capable to further process this
information and infer more complex information. As an example, the semantic mod-
ule successfully detects the activity Cooking when the events HighTemperature,
HighLuminosity and Presence are detected inside the Kitchen. The configu-
ration pipeline shown in Figure 2 leverages the knowledge generated by the data pro-
cessing pipeline to generate a timeline-based planning model for the acting module. It
leverages knowledge about the configuration to generate the state variables needed to
model the robot and environment primitive capabilities. Specifically, the configuration
pipeline generates the AssistiveRobotNavigationSV and the AssistiveRobotCommunica-
tionSV to model respectively the robot primitives that can be used to move the telep-
resence robot within the house, interact and allow the senior user to communicate with
the external world. Additional state variables like e.g., the KitchenTemperatureSV, Liv-
ingroomLuminositySV are generated to model environment primitives and the status of
the house. Finally, the HumanSV is generated to model the behavior and the status of
the monitored senior user over time.
The synchronization rules composing the timeline-based planning model as well as
the planning goals (i.e., supporting tasks) the GR can trigger depend on the set of events
and activities the data processing pipeline is capable of recognizing. As an example, the
GR generates the planning goal SupportMealTime when the data processing pipeline
recognizes the activity Cooking. Given such a goal, the acting module synthesizes a
set of flexible timelines each of which represents an envelope of valid temporal behav-
iors that allow an assistive robot to carry out the supporting task. Considering the assis-
tive task SupportMealTime, the synthesized and executed timelines associated with the
robot primitives determine the actions performed to support the Cooking activity of
the user. The timeline of the AssistiveRobotNavigationSV specifies the tokens that allow
the assistive robot to navigate the home environment and reach the kitchen. Similarly,
the timeline of the AssistiveRobotCommunicationSV specifies the tokens that allow the
assistive robot to remind to the user the dietary restrictions he/she must follow as soon
as it reach the kitchen. Then, after a known (flexible) interval of time the user ends eat-
ing the meal and the timeline of the AssistiveRobotCommunicationSV plans to remind
the user to take his/her pills for the therapy, after this interval of interval of time.
7 Related Works
Different works have been presented in the literature taking into account different per-
spectives and different levels of abstraction. Some works focus on the problem of man-
aging sensor data to extract knowledge that can be leveraged to realize complex ser-
vices. The works [17, 18, 19] propose an ontology-based approach for activity recogni-
tion for a home-care service, and a constraint-based approach for proactive human sup-
port. Some works address more specific problems like e.g. RoboSherlock [20] which
proposes a knowledge-based approach for representing realistic scenes and reasoning
on manipulation tasks that can be performed. Other works deal with the problem of
endowing autonomous agents with cognitive capabilities to represent knowledge about
contexts and leverage such knowledge to improve the flexibility of control processes.
The works [21, 22] propose the integration of knowledge processing mechanisms with
planning to improve the efficiency and performance of deliberation processes. Sim-
ilarly, the works [23, 24] propose the integration of knowledge representation with
machine learning to improve the flexibility and efficiency of robots while interacting
with humans. The novelty of our approach consists in the design and development of a
cognitive architecture which relies on a holistic approach to knowledge representation
based on a well-defined ontology. The key idea is the development of a control archi-
tecture integrating knowledge processing mechanisms that leverage standard semantic
technologies to dynamically generate a model of the application context. Such standard
knowledge can be leveraged by different services to be integrated into a control archi-
tecture. Thus, several “independent" services can be built on top of such knowledge and
leverage the related information for different purposes like e.g., human behavior learn-
ing services, human monitoring services or decisional autonomy for assistive robots.
8 Conclusions
This paper presented a cognitive architecture which integrates sensing, knowledge rep-
resentation and planning to constitute a control loop enhancing the proactivity of an
assistive robot that supports an older person living at home in her daily routine. A
semantic module leverages a dedicated ontology to build a KB by properly process-
ing data collected from a sensor network. An acting module takes advantage of the
timeline-based planning approach to control robot behaviors. A goal recognition pro-
cess connects these two modules and provides the key enabling feature to endow the
robot with a suitable proactivity level. At this stage, some tests have been performed to
show the feasibility of the approach. Further work is ongoing to perform more exten-
sive integrated laboratory tests and better assess the performance and capabilities of the
overall system. Future work will also investigate the opportunity to integrate machine
learning techniques to better adapt the behavior of the assistive robot to specific daily
behaviors of different targeted people.
References
1. Coradeschi, S., Cesta, A., Cortellessa, G., Coraci, L., Gonzalez, J., Karlsson, L., Furfari, F.,
Loutfi, A., Orlandini, A., Palumbo, F., Pecora, F., von Rump, S., Štimec, A., Ullberg, J., Öt-
slund, B.: GiraffPlus: Combining social interaction and long term monitoring for promoting
independent living. In: The 6th International Conference on Human System Interactions
(HSI). (2013) 578–585
2. Cesta A., Cortellessa G., Fracasso F., Orlandini A., and Turno M.: User Needs and Pref-
erences on AAL Systems that Support Older Adults and Their Carers. Journal of Ambient
Intelligence and Smart Environments 10(1) (2018) 49–70
3. Cesta, A., Cortellessa, G., Orlandini, A., Umbrico, A.: A cognitive loop for assistive robots -
connecting reasoning on sensed data to acting. In: RO-MAN 2018 - 27th IEEE International
Symposium on Robot and Human Interactive Communication, Institute of Electrical and
Electronics Engineers Inc. (2018) 826–831
4. Umbrico, A., Cesta, A., Cortellessa, G., Orlandini, A.: A goal triggering mechanism for con-
tinuous human-robot interaction. Lecture Notes in Computer Science (including subseries
Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11298 LNAI
(2018) 460–473
5. Cesta, A., Cortellessa, G., Orlandini, A., Sorrentino, A., Umbrico, A.: A semantic representa-
tion of sensor data to promote proactivity in home assistive robotics. Advances in Intelligent
Systems and Computing 868 (2019) 750–769
6. Bechhofer, S. In: OWL: Web Ontology Language. Springer US, Boston, MA (2009) 2008–
2009
7. Cialdea Mayer, M., Orlandini, A., Umbrico, A.: Planning and execution with flexible time-
lines: a formal account. Acta Informatica 53(6-8) (2016) 649–680
8. Compton, M., Barnaghi, P., Bermudez, L., García-Castro, R., Corcho, O., Cox, S., Graybeal,
J., Hauswirth, M., Henson, C., Herzog, A., Huang, V., Janowicz, K., Kelsey, W.D., Phuoc,
D.L., Lefort, L., Leggieri, M., Neuhaus, H., Nikolov, A., Page, K., Passant, A., Sheth, A.,
Taylor, K.: The ssn ontology of the w3c semantic sensor network incubator group. Web
Semantics: Science, Services and Agents on the World Wide Web 17(Supplement C) (2012)
25 – 32
9. Borgo, S., Cesta, A., Orlandini, A., Umbrico, A.: A planning-based architecture for a re-
configurable manufacturing system. In: The 26th International Conference on Automated
Planning and Scheduling (ICAPS). (2016)
10. Umbrico, A., Cesta, A., Cialdea Mayer, M., Orlandini, A.: Integrating resource management
and timeline-based planning. In: The 28th International Conference on Automated Planning
and Scheduling (ICAPS). (2018)
11. Umbrico, A., Cesta, A., Cialdea Mayer, M., Orlandini, A.: PLATINU M: A new Frame-
work for Planning and Acting. In: AI*IA 2016 Advances in Artificial Intelligence: XVth
International Conference of the Italian Association for Artificial Intelligence, Genova, Italy,
November 29 – December 1, 2016, Proceedings, Springer International Publishing (2017)
508–522
12. Pellegrinelli, S., Orlandini, A., Pedrocchi, N., Umbrico, A., Tolio, T.: Motion planning and
scheduling for human and industrial-robot collaboration. CIRP Annals - Manufacturing
Technology 66 (2017) 1–4
13. Muscettola, N.: HSTS: Integrating Planning and Scheduling. In Zweben, M. and Fox, M.S.,
ed.: Intelligent Scheduling. Morgan Kauffmann (1994)
14. Muscettola, N., Smith, S., Cesta, A., D’Aloisi, D.: Coordinating Space Telescope Operations
in an Integrated Planning and Scheduling Architecture. IEEE Control Systems 12(1) (1992)
28–37
15. Frank, J., Jonsson, A.: Constraint Based Attribute and Interval Planning. Journal of Con-
straints 8(4) (2003) 339–364
16. Cesta, A., Cortellessa, G., Fratini, S., Oddi, A.: M R SPOCK: Steps in Developing an End-
to-End Space Application. Computational Intelligence 27(1) (2011)
17. Alirezaie, M., Renoux, J., Kockemann, U., Loutfi, A.: An ontology-based context-aware
system for smart homes: E-care@home. Sensors 17(7) (2017)
18. Federico Pecora and Marcello Cirillo and Francesca Dell’Osa and Jonas Ullberg and
Alessandro Saffiotti: A constraint-based approach for proactive, context-aware human sup-
port. JAISE 4(4) (2012) 347–367
19. Suh, I.H., Lim, G.H., Hwang, W., Suh, H., Choi, J.H., Park, Y.T.: Ontology-based multi-
layered robot knowledge framework (OMRKF) for robot intelligence. In: Intelligent Robots
and Systems, 2007. IROS 2007. IEEE/RSJ International Conference on. (Oct 2007) 429–436
20. Beetz, M., Bálint-Benczédi, F., Blodow, N., Nyga, D., Wiedemeyer, T., Márton, Z.C.: Ro-
bosherlock: Unstructured information processing for robot perception. In: 2015 IEEE Inter-
national Conference on Robotics and Automation (ICRA). (2015) 1549–1556
21. Behnke, G., Ponomaryov, D., Schiller, M., Bercher, P., Nothdurft, F., Glimm, B., Biundo, S.:
Coherence across components in cognitive systems–one ontology to rule them all. In: Proc.
of the 25th Int. Joint Conf. on Artificial Intelligence (IJCAI 2015). AAAI Press. (2015)
22. Hartanto, R., Hertzberg, J.: Fusing DL Reasoning with HTN Planning. In Dengel, A., Berns,
K., Breuel, T., Bomarius, F., Roth-Berghofer, T., eds.: KI 2008: Advances in Artificial Intel-
ligence. Volume 5243 of Lecture Notes in Computer Science. Springer Berlin Heidelberg
(2008) 62–69
23. Tenorth, M., Beetz, M.: Representations for robot knowledge in the KnowRob framework.
Artificial Intelligence (2015)
24. Lemaignan, S., Ros, R., Mosenlechner, L., Alami, R., Beetz, M.: ORO, a knowledge man-
agement platform for cognitive architectures in robotics. In: Intelligent Robots and Systems
(IROS), 2010 IEEE/RSJ International Conference on. (Oct 2010) 3548–3553