A Knowledge-based Planning and Acting System for Assistive Robots

Abstract

In this work, we present a Knowledge-based Planning and Acting architecture to control assistive robots pursuing the integrating of knowledge representation and reasoning techniques with automated planning and acting solutions. The architecture aims to endow assistive robots with some proactivity and self-configuration capabilities needed to deploy them in domestic environment to providing an effective and useful support to elderly people for independent living.

Introduction

The recent advancements in robotic technologies are fostering the development of new Artificial Intelligence (AI) research solutions to deploy robots in assistive scenarios and allow them to dynamically interact with humans living in their own domestic environment. In general, autonomous robots should be able to represent and reason over a wide set of information constituting the knowledge about the application scenario, the needs of the human user, etc. while acting in the environment. The knowledge structure of a robot depends on the particular application context and objectives as well as the particular behavior a robot must achieve. Our long-term research objective aims at designing “companion robots" capable of taking autonomous decisions to support older persons in their living environments through effective and safe interactions. We focus on scenarios where a senior user, with mild cognitive and/or physical impairments, lives in her home with the need of continuous assistance from a personal robotic assistant. In this regard, a robot acting in this scenario should (i) acquire information about the user and the environment via a sensor network, (ii) analyze and reason over such information and (iii) proactively take decisions to support with continuity a user in her daily home-living activities. We are developing such reasoning capabilities for assistive tasks on top of a mobile/telepresence robot. In particular we are exploring the integration of knowledge representation and reasoning features with automated planning and execution techniques. As a case study, we consider GiraffPlus-like scenarios [ 1 ], a research project whose aim was to create a sensor-integrated environment to support seniors in their living environments. The objective of the project was to support prevention and long-term monitoring as well as to foster social interaction and communication through a telepresence robot by proposing a solution built around the primary users (i.e., the seniors) [ 2 ]. This paper presents the main features of a novel Planning and Acting architecture, called KOaLa (for Knowledge-based cOntinuous Loop) [ 3, 4 ], for the synthesis of a continuous “sense-reason-act" control cycle. The pursued control approach relies on the integration of (i) an abstraction process over the data collected via the sensor network, (ii) a semantic representation and reasoning module to provide the control system with a Knowledge Base (KB) representing the user, her/her needs and the environment conditions and, (iii) a planning and execution module to implement decisional autonomy. When deployed in a GiraffPlus-like sensorized environment, such modules should endow an assistive robot with a enhanced level of proactivity and autonomy. 2

KOaLa as a Cognitive Architecture

The KOaLa architecture [ 4, 3 ] consists of a knowledge processing module, called the KOaLa Semantic Module, and a planning and execution module1, called the KOaLa Acting Module, that constitute an integrated high-level control loop. Fig. 1 shows a conceptual representation of the envisaged control loop together with the different steps of the control flow. It starts with data gathered through sensors and it ends with the execution of actions in the environment to realize the decided supporting tasks. The KOaLa KOaLa Semantic Module

KOaLa Ontology

KOaLa Acting Module

KB Sensed Data

Goal Recognition

Problem

Formulation

New Goals Data Processing

Planning & Execution Environment / Robot / Sensor Network

Timeline -based

Plan Action Execution Semantic Module is responsible for the interpretation of data gathered from sensors and the management of the knowledge of the robot. This module relies on the KOaLa Ontology [ 5 ] (see also Sec. 3) to provide sensor data with semantics and incrementally build an abstract representation of the application context i.e., the Knowledge Base (KB). It implements a data processing mechanism which leverages the Web Ontology Language (OWL) [ 6 ] and semantic technologies 2 to continuously refine the KB and infer additional information (e.g., recognized user activities). Then, a goal recognition process analyzes the KB to detect specific situations that require the proactive execution of supporting tasks and therefore it dynamically trigger planning goals. The KOaLa Acting Module is responsible of planning and executing (see Sec. 4) assistive tasks according to the events or activities inferred by the semantic module. These tasks are encoded as 1 Planning and Acting is also used as interchangeable with Planning and Execution. 2 See the Apache Jena Library - http://jena.apache.org planning goals by the problem formulation process and, given to a planner in order to generate a timeline-based plan which describes the sequence of actions that must be executed. A planning and execution process leverages the timeline-based approach [ 7 ] to continuously refine and update the plan according to the triggered planning goals and the status of the execution. The executive iteratively dispatches actions to the environment according to the temporal behaviors of the robot and the sensors encoded by the timelines of the plan. 3

Ontology and Data Processing

The Semantic Module in Fig. 1 provides the control approach with the cognitive capabilities needed to interpret data collected from sensors and dynamically build, refine and analyze an internal representation of the user and home environment status (i.e., the KB). It relies on a dedicated ontology which formally characterize the general concepts, entities and properties the system must deal with, and a data processing mechanism which iteratively refines the KB by interpreting sensor data according to the ontology. The KOaLa ontology extends the SSN ontology [ 8 ] and the DUL ontology 3 and, it is structured according to a context-based approach. Each context characterizes the knowledge of an agent with respect to a particular perspective and a specifc level of abstraction. They are organized in a hierarchical way in order to incrementally build knowledge starting from raw data interpretation. Following a bottom-up approach, (i) the sensor context, (ii) the environment context and (iii) the observation context have been defined. The sensor context characterizes the knowledge about the sensing devices that compose a particular environment, their deployment and the properties they may observe. This context is strictly related to SSN. The aim of the sensor context is to provide a more detailed representation of the different types of sensor that can compose an environment as well as the different types of property that can be observed. Such knowledge allows the system to dynamically recognize the actual monitoring capabilities as well as the set of supporting tasks that can be performed according to the available sensors and their deployment i.e., the configuration. The environment context characterizes the knowledge about the structure and physical elements that can compose a home environment together with sensor deployment. It defines the properties of the different types of element that can be observed to characterize the state of the environment. Finally, the observation context characterizes the knowledge about the features that can actually produce information as well as the events and the activities that can be observed through them. 4

Goal, Plan, Act

The last step of the data processing mechanism is represented by the Goal Recognition module (GR) which dynamically generates goals for the Acting Module of the architecture. GR is not responsible for the refinement of the KB, rather it leverages the inferred KB to connect knowledge representation with planning. It can be seen as a background 3 http://www.loa-cnr.it/ontologies/DUL.owl process that monitors the updated KB in order to generate operations (i.e., goals) the system must perform. GR is the key feature of KOaLa to achieve proactivity.

The set of goals the Acting Module can deal with depends on the particular configuration of the environment and the available capabilities. A planning model encapsulates the knowledge about the capabilities of the controllable elements of the environment and how such capabilities must be coordinated to realize complex supporting tasks. It describes the primitive capabilities of assistive robots like e.g., make a call, send a message or move to a particular location, as well as the primitive capabilities of sensors like e.g., turn on, turn off a sensor or set a particular configuration on a sensor. Planning goals represent supporting tasks that can be performed by properly controlling and coordinating these primitive capabilities. Such a planning model can be dynamically configured by analyzing the “static" knowledge about the configuration of the environment. This means that no observation data is needed and therefore the planning model can be generated during an initial setup phase. Figure 2 shows the configuration process pipeline which generates such a model by leveraging mechanisms similar to those described in [ 9 ] for a reconfigurable manufacturing system.

Configuration

Detection

Primitive Capability Extraction

Assistive Functionality

Extraction

Constraint Modeling The Configuration Detection step extracts the configuration of the environment from the KB in order to identify the set and types of sensor, their deployment as well as information about the assistive robot used. Then, the Primitive Capability Extraction further analyzes these elements in order to extract the primitives representing the low-level functionalities available to control the environment. Two types of primitives can be identified: (i) environment primitives; (ii) robot primitives. The environment primitives represent the capabilities of the elements of the environment that can be controlled. Namely, they characterize the controllable elements of the environment and the operations that can be performed on and with them. They do not model directly the sensors of the environment, but rather they model the elements that can be controlled through the deployed sensors. For example a sensor deployed on the socket where a TV is plugged in can be used to turn off and on the TV. In such a case, the TV becomes controllable and the related turn on and turn off capabilities are part of the environment primitives. Similarly, the robot primitives represent the capabilities of the assistive robot available within the environment. They model the the functional layer of the robot which provides the basic functionalities that must be used to perform assistive tasks. For example a robot like the GiraffPlus robot provides navigation capabilities that can be used to move the robot within the environment, messaging capabilities that can be used to send/receive messages to/from patient’s relatives, videocall capabilities that can be used to make calls or receive calls with or from doctors and patient’s relatives. All these functionalities compose the robot primitives. The Assistive Functionality Extraction step extracts the high-level supporting tasks the system can perform. As shown in the previous section, the data processing mechanism is capable of dynamically inferring the set of events and activities that can be actually monitored/detected according to the specific configuration of the environment and the properties of the deployed sensors. Given such knowledge, this step analyzes the KB in order to extract the set of assistive tasks an assistive robot is actually capable of performing in the considered scenario. Finally, the Constraint Modeling step finalizes the control model by linking complex supporting tasks to the environment and robot primitives. Specifically, it leverages the results of the two previous steps in order to correlate supporting tasks to the primitives needed to perform them. The result of the described pipeline is a control model that completely characterizes the high-level supporting tasks an assistive robot can perform (i.e., the goals generated by the GR) and the constraints that must be satisfied to realize them and properly coordinate the available primitives. 5

Planning and Execution with PLATINUm

The planning and execution capabilities of the Acting Module rely on PLATINUm [ 10, 11 ], a novel framework which has been successfully applied in real-world manufacturing scenarios [ 12 ] and relies on the formal characterization of timeline-based approach proposed in [ 7 ]. The timeline-based approach is a particular temporal planning paradigm which has been introduced in early 90s [ 13 ] by taking inspiration from the classical Control Theory, and successfully applied in many real-world scenarios [ 14, 15, 16 ]. This planning paradigm aims at controlling a complex system by synthesizing temporal behaviors for a set of identified domain features that must be controlled over time. According to the formalization proposed in [ 7 ], a timeline-based model is composed by a set of state variables describing the possible temporal behaviors of the domain features that are relevant from the control perspective. Each state variable specifies a set of values that represent the states or actions the related feature may assume or perform over time. Each value is associated with a flexible duration and a controllability tag which specifies whether the value is controllable or not. A state transition function specifies the valid temporal behaviors of a state variable by modeling the allowed sequences of values (i.e., the transitions between the values of a state variable). State variables model “local" constraints a planner must satisfy to generate valid temporal behaviors of single features of the domain i.e., valid timelines. It could be necessary to further constrain the behaviors of state variables in order to coordinate differente domain features and realize complex functionalities or achieve complex goals (e.g., perform assistive functionalities). A dedicated set of rules called synchronization rules model “global" constraints that a planner must satisfy to build a valid plan. Such rules can be used also to specify planning goals. 6

Feasibility Evaluation of KOaLa

As a case study, we considered a typical GiraffPlus scenario where a senior person lives alone in a single floor apartment composed by a kitchen, a bedroom, a bathroom and a living room. A telepresence robot shares the apartment with the senior user. The robot is capable of navigating the apartment, interacting with the senior user through gestures and voice commands as well as making videocalls, sending and reproducing text/audio messages. A set of sensors capable of gathering information about the temperature, luminosity and presence are installed inside the apartment. Each room of the apartment is endowed with one of these sensors in the experimental scenario. There are additional sensors capable of detecting energy consumptions and they have been installed in order to detect the usage of particular devices like e.g., a TV or a microwave oven.

The data processing pipeline shown analyzes the configuration of the house to detect the observable features of the environment and the associated observable properties. According to this information, all the rooms of the environments are classified as ObservableFeature and the associated properties Temperature, Luminosity and Presence can be observed. Sensor data is processed by applying inference rules to recognize events and activities. The semantic module successfully recognizes events HighLuminosity and HighTemperature when data received by sensors is higher than a known threshold. Then, tests show that it is capable to further process this information and infer more complex information. As an example, the semantic module successfully detects the activity Cooking when the events HighTemperature, HighLuminosity and Presence are detected inside the Kitchen. The configuration pipeline shown in Figure 2 leverages the knowledge generated by the data processing pipeline to generate a timeline-based planning model for the acting module. It leverages knowledge about the configuration to generate the state variables needed to model the robot and environment primitive capabilities. Specifically, the configuration pipeline generates the AssistiveRobotNavigationSV and the AssistiveRobotCommunicationSV to model respectively the robot primitives that can be used to move the telepresence robot within the house, interact and allow the senior user to communicate with the external world. Additional state variables like e.g., the KitchenTemperatureSV, LivingroomLuminositySV are generated to model environment primitives and the status of the house. Finally, the HumanSV is generated to model the behavior and the status of the monitored senior user over time.

The synchronization rules composing the timeline-based planning model as well as the planning goals (i.e., supporting tasks) the GR can trigger depend on the set of events and activities the data processing pipeline is capable of recognizing. As an example, the GR generates the planning goal SupportMealTime when the data processing pipeline recognizes the activity Cooking. Given such a goal, the acting module synthesizes a set of flexible timelines each of which represents an envelope of valid temporal behaviors that allow an assistive robot to carry out the supporting task. Considering the assistive task SupportMealTime, the synthesized and executed timelines associated with the robot primitives determine the actions performed to support the Cooking activity of the user. The timeline of the AssistiveRobotNavigationSV specifies the tokens that allow the assistive robot to navigate the home environment and reach the kitchen. Similarly, the timeline of the AssistiveRobotCommunicationSV specifies the tokens that allow the assistive robot to remind to the user the dietary restrictions he/she must follow as soon as it reach the kitchen. Then, after a known (flexible) interval of time the user ends eating the meal and the timeline of the AssistiveRobotCommunicationSV plans to remind the user to take his/her pills for the therapy, after this interval of interval of time. 7

Related Works

Different works have been presented in the literature taking into account different perspectives and different levels of abstraction. Some works focus on the problem of managing sensor data to extract knowledge that can be leveraged to realize complex services. The works [ 17, 18, 19 ] propose an ontology-based approach for activity recognition for a home-care service, and a constraint-based approach for proactive human support. Some works address more specific problems like e.g. RoboSherlock [ 20 ] which proposes a knowledge-based approach for representing realistic scenes and reasoning on manipulation tasks that can be performed. Other works deal with the problem of endowing autonomous agents with cognitive capabilities to represent knowledge about contexts and leverage such knowledge to improve the flexibility of control processes. The works [ 21, 22 ] propose the integration of knowledge processing mechanisms with planning to improve the efficiency and performance of deliberation processes. Similarly, the works [ 23, 24 ] propose the integration of knowledge representation with machine learning to improve the flexibility and efficiency of robots while interacting with humans. The novelty of our approach consists in the design and development of a cognitive architecture which relies on a holistic approach to knowledge representation based on a well-defined ontology. The key idea is the development of a control architecture integrating knowledge processing mechanisms that leverage standard semantic technologies to dynamically generate a model of the application context. Such standard knowledge can be leveraged by different services to be integrated into a control architecture. Thus, several “independent" services can be built on top of such knowledge and leverage the related information for different purposes like e.g., human behavior learning services, human monitoring services or decisional autonomy for assistive robots. 8

Conclusions

This paper presented a cognitive architecture which integrates sensing, knowledge representation and planning to constitute a control loop enhancing the proactivity of an assistive robot that supports an older person living at home in her daily routine. A semantic module leverages a dedicated ontology to build a KB by properly processing data collected from a sensor network. An acting module takes advantage of the timeline-based planning approach to control robot behaviors. A goal recognition process connects these two modules and provides the key enabling feature to endow the robot with a suitable proactivity level. At this stage, some tests have been performed to show the feasibility of the approach. Further work is ongoing to perform more extensive integrated laboratory tests and better assess the performance and capabilities of the overall system. Future work will also investigate the opportunity to integrate machine learning techniques to better adapt the behavior of the assistive robot to specific daily behaviors of different targeted people.

References

1. Coradeschi , S. , Cesta , A. , Cortellessa , G. , Coraci , L. , Gonzalez , J. , Karlsson , L. , Furfari , F. , Loutfi , A. , Orlandini , A. , Palumbo , F. , Pecora , F. , von Rump , S. , Štimec , A. , Ullberg , J. , Ötslund , B. : GiraffPlus: Combining social interaction and long term monitoring for promoting independent living . In: The 6th International Conference on Human System Interactions (HSI) . ( 2013 ) 578 - 585

2. Cesta A. , Cortellessa G. , Fracasso F. , Orlandini A. , and Turno M.: User Needs and Preferences on AAL Systems that Support Older Adults and Their Carers . Journal of Ambient Intelligence and Smart Environments 10 ( 1 ) ( 2018 ) 49 - 70

3. Cesta , A. , Cortellessa , G. , Orlandini , A. , Umbrico , A. : A cognitive loop for assistive robots - connecting reasoning on sensed data to acting . In: RO-MAN 2018 - 27th IEEE International Symposium on Robot and Human Interactive Communication , Institute of Electrical and Electronics Engineers Inc . ( 2018 ) 826 - 831

4. Umbrico , A. , Cesta , A. , Cortellessa , G. , Orlandini , A. : A goal triggering mechanism for continuous human-robot interaction . Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11298 LNAI ( 2018 ) 460 - 473

5. Cesta , A. , Cortellessa , G. , Orlandini , A. , Sorrentino , A. , Umbrico , A. : A semantic representation of sensor data to promote proactivity in home assistive robotics . Advances in Intelligent Systems and Computing 868 ( 2019 ) 750 - 769

6. Bechhofer , S. In: OWL: Web Ontology Language. Springer US, Boston, MA ( 2009 ) 2008 - 2009

7. Cialdea Mayer , M. , Orlandini , A. , Umbrico , A. : Planning and execution with flexible timelines: a formal account . Acta Informatica 53 ( 6-8 ) ( 2016 ) 649 - 680

8. Compton , M. , Barnaghi , P. , Bermudez , L. , García-Castro , R. , Corcho , O. , Cox , S. , Graybeal , J. , Hauswirth , M. , Henson , C. , Herzog , A. , Huang , V. , Janowicz , K. , Kelsey , W.D. , Phuoc , D.L. , Lefort , L. , Leggieri , M. , Neuhaus , H. , Nikolov , A. , Page , K. , Passant , A. , Sheth , A. , Taylor , K.: The ssn ontology of the w3c semantic sensor network incubator group . Web Semantics: Science, Services and Agents on the World Wide Web 17 ( Supplement C) ( 2012 ) 25 - 32

9. Borgo , S. , Cesta , A. , Orlandini , A. , Umbrico , A. : A planning-based architecture for a reconfigurable manufacturing system . In: The 26th International Conference on Automated Planning and Scheduling (ICAPS) . ( 2016 )

10. Umbrico , A. , Cesta , A. , Cialdea Mayer , M. , Orlandini , A. : Integrating resource management and timeline-based planning . In: The 28th International Conference on Automated Planning and Scheduling (ICAPS) . ( 2018 )

11. Umbrico , A. , Cesta , A. , Cialdea Mayer , M. , Orlandini , A. : PLATINUM: A new Framework for Planning and Acting . In: AI*IA 2016 Advances in Artificial Intelligence: XVth International Conference of the Italian Association for Artificial Intelligence , Genova, Italy, November 29 - December 1 , 2016 , Proceedings, Springer International Publishing ( 2017 ) 508 - 522

12. Pellegrinelli , S. , Orlandini , A. , Pedrocchi , N. , Umbrico , A. , Tolio , T. : Motion planning and scheduling for human and industrial-robot collaboration . CIRP Annals - Manufacturing Technology 66 ( 2017 ) 1 - 4

13. Muscettola , N.: HSTS: Integrating Planning and Scheduling . In Zweben, M. and Fox , M.S., ed.: Intelligent Scheduling. Morgan Kauffmann ( 1994 )

14. Muscettola , N. , Smith , S. , Cesta , A. , D'Aloisi , D. : Coordinating Space Telescope Operations in an Integrated Planning and Scheduling Architecture . IEEE Control Systems 12 ( 1 ) ( 1992 ) 28 - 37

15. Frank , J. , Jonsson , A. : Constraint Based Attribute and Interval Planning . Journal of Constraints 8 ( 4 ) ( 2003 ) 339 - 364

16. Cesta , A. , Cortellessa , G. , Fratini , S. , Oddi , A. : MRSPOCK: Steps in Developing an Endto-End Space Application . Computational Intelligence 27 ( 1 ) ( 2011 )

17. Alirezaie , M. , Renoux , J. , Kockemann , U. , Loutfi , A. : An ontology-based context-aware system for smart homes: E-care@home . Sensors 17 ( 7 ) ( 2017 )

18. Federico Pecora and Marcello Cirillo and Francesca Dell'Osa and Jonas Ullberg and Alessandro Saffiotti: A constraint-based approach for proactive, context-aware human support . JAISE 4 ( 4 ) ( 2012 ) 347 - 367

19. Suh , I.H. , Lim , G.H. , Hwang , W. , Suh , H. , Choi , J.H. , Park , Y.T.: Ontology-based multilayered robot knowledge framework (OMRKF) for robot intelligence . In: Intelligent Robots and Systems , 2007 . IROS 2007 . IEEE/RSJ International Conference on. (Oct 2007 ) 429 - 436

20. Beetz , M. , Bálint-Benczédi , F. , Blodow , N. , Nyga , D. , Wiedemeyer , T. , Márton , Z.C. : Robosherlock: Unstructured information processing for robot perception . In: 2015 IEEE International Conference on Robotics and Automation (ICRA) . ( 2015 ) 1549 - 1556

21. Behnke , G. , Ponomaryov , D. , Schiller , M. , Bercher , P. , Nothdurft , F. , Glimm , B. , Biundo , S. : Coherence across components in cognitive systems-one ontology to rule them all . In: Proc. of the 25th Int. Joint Conf. on Artificial Intelligence (IJCAI 2015 ). AAAI Press. ( 2015 )

22. Hartanto , R. , Hertzberg , J.: Fusing DL Reasoning with HTN Planning . In Dengel , A. , Berns , K. , Breuel , T. , Bomarius , F. , Roth-Berghofer , T., eds. : KI 2008: Advances in Artificial Intelligence. Volume 5243 of Lecture Notes in Computer Science . Springer Berlin Heidelberg ( 2008 ) 62 - 69

23. Tenorth , M. , Beetz , M. : Representations for robot knowledge in the KnowRob framework . Artificial Intelligence ( 2015 )

24. Lemaignan , S. , Ros , R. , Mosenlechner , L. , Alami , R. , Beetz , M. : ORO, a knowledge management platform for cognitive architectures in robotics . In: Intelligent Robots and Systems (IROS) , 2010 IEEE/RSJ International Conference on. (Oct 2010 ) 3548 - 3553