New generations of robotic systems capable of taking care of human-level tasks are becoming more and more desirable especially if considering the lack of human support in health-care assistance for the elderly. Assistive Robotics is a growing research eld that is also applied to support both older adults and caregivers in a variety of situations and contexts. It leverages and integrates results from di erent research areas like e.g., Arti cial Intelligence (AI), Cognitive Systems, Psychology and of course Robotics. Concerning AI, many of the technological skills that assistive robots could bene t of to achieve their objectives represent important challenges for that eld. Some of the most relevant are the capability of monitoring and understanding information coming from the environment, the capability of interacting with humans in a exible and human-compliant way, the capability of proactively performing supporting tasks inside the environment and also the capability of personalizing both interactions and services according to the speci c needs of the assisted person. Thus, there are many techniques of AI that must be \integrated in a loop" to realize a needed set of advanced capabilities. This paper presents a research initiative which aims at realizing an enhanced (cognitive) control architecture to endow autonomous and socially interacting robots with a number of such advanced functionalities. Speci cally, the paper presents an initial version of the envisaged control architecture, called KOaLa (Knowledge-based cOntinuous Loop) which integrates sensor data representation, knowledge reasoning and decision making functionalities.
Nowadays, there are many widely di used commercial robotic solutions like e.g., robot vacuums or industrial lightweight robots, while a new generation of Intelligent Robots are entering our working and living environments, taking care of human-level tasks. Such robotic systems are becoming more and more important also in elderly healthcare assistance. Indeed, recent advancements in Arti cial Intelligence (AI) and Robotics are fostering the di usion of robotic agents with the capabilities needed to support both older adults and their caregivers in a variety of situations (e.g., in their homes, in hospitals, etc.). Such robotic agents must be capable of monitoring and understanding information coming from the environment, interacting with humans in a exible and human-compliant way, autonomously performing tasks inside the environment and also personalizing interactions and services according to the speci c needs of the assisted person.
The ability of representing and reasoning diverse kind of knowledge constitutes a key feature for allowing intelligent robotic assistants to understand the actual (and possibly time changing) needs of older persons as well as the status of the environment in which they are acting and inferring new knowledge to adapt their behaviors and better assist humans. The need of supporting long-term monitoring and deploying personalized services for di erent users opens to the exploitation of sensor networks to gather information about the status of the assisted persons and their living environments in order to gure out which is the actual situation and how e ective assistance can be provided. New social robots are entering the market (e.g., Pepper by SoftBank Robotics) but they still lack advanced reasoning capabilities to provide well suited and e ective impact in healthcare assistance, contributing in prolonging elderly independence as well as increasing their quality of life.
AI techniques constitute a key enabling technology for realizing adaptive assistive services to implement continuous monitoring and support daily-home living of seniors. This paper tries to identify the main requirements that an intelligent assistive robot must satisfy to realize effective services aimed at taking care of older adults inside their home living environment. According to this requirements we make an hypothesis concerning the main AI techniques that can contribute to achieve the desired objectives. We propose an advanced cognitive architecture integrating these AI techniques into a uni ed control loop and we discuss the related responsibilities and contributions with respect to the desired intelligent assistive behaviors. Then, we show a (partial) implementation of the envisaged cognitive architecture called KOaLa (Knowledge-based cOntinuous Loop) which has been designed by leveraging the results and the experience earned with Gira Plus [8] (a research project funded by the European Commission representing a successful example o the use of AI in domestic care contexts). 2
The development of reliable AI and robotic technologies aimed at supporting the daily-home living of persons directly at home is a really challenging research objective. There are many heterogeneous situations such systems must properly deal with to e ectively support a person and, therefore many features and capabilities must be taken into account. Taking inspiration from the experience in Gra Plus [8], it is possible to identify a set of key requirements characterizing the capabilities of intelligent assistive robotic systems. These requirements can be characterized according to four correlated perspectives: (i) environment perspective; (ii) autonomy perspective; (iii) interaction perspective; (iv) adaptation perspective.
{ Environment perspective. Pursuing the idea of Gira Plus
different types of sensor can be used to gather information about the
environment and the health status of the assisted person. The
number and the type of the sensors deployed into the environment
depend on the speci c purposes and objectives that must be achieved.
Broadly speaking, there are two categories of sensing devices that
are relevant in domestic assistance scenarios. Environmental sensors
produce data about the state of a particular area of the house like
e.g., the kitchen, the living-room and so on. Physiological sensors
produce data about physiological parameters of a person like e.g.,
blood pressure, hearth rate and so on. Thus, IoT and sensing
devices represent a precious source of information characterizing
different features of a working context. The envisaged assistive robotic
system must be capable of dealing with a continuous ow of
heterogeneous data coming from sensors to monitor the state of the
environment and autonomously recognize particular situations that
require support. Namely, the system must be capable of recognizing
activities the assisted person is performing inside the house as well
as recognizing events related to the state of the house or the health
of the assisted person.
{ Autonomy perspective. Analyzing the knowledge gathered from
the environment the envisaged system can recognize particular
situations that may require to proactively execute supporting tasks. This
means that a \causal knowledge" is needed to characterize the basic
rules that enable a safe and correct interaction of the system with
the environment. According to this knowledge, the system knows its
internal capabilities and how they interact with the environment and
therefore can autonomously decide the sequence of activities needed
to achieve a desired objective like e.g., a supporting task. A decision
making process is needed to achieve the level of autonomy needed
to automatically synthesize and carry out supportive actions.
{ Interaction perspective. The assistive robot must be capable of
interacting with humans at di erent levels and with di erent
modalities like e.g., gestures and/or voice. In general, the interactions
between humans and robots must be as safe and \natural" as possible.
Older adults should interact with an assistive robot in a \natural
way" using a \natural language" and they should not feel the robot
as an obstacle into the house and/or as a danger for their safety. To
achieve this an assistive robot must correctly understand commands
and instructions coming from humans and must show behaviors that
are safe but also socially acceptable by humans. Namely, the
behaviors of an assistive robot must comply with so-called social norms
that are necessary to e ectively take part to \social life". The work
[
The envisaged assistive robotic system must be capable of integrating a heterogeneous and complex set of (intelligent) capabilities. There are several techniques in AI that address some of the challenges raised by the requirements described above. Machine learning, knowledge representation and reasoning, automated planning and execution represent three well-established eld of AI that can play a key role in this context. A proper integration of these techniques can endow an assistive robot with the capabilities needed to achieve the desired objectives. Thus, the long-term research objective we are pursuing aims at realizing an enhanced cognitive architecture for assistive robots integrating three core AI techniques.
Research in cognitive architecture aims at endowing an arti cial agent
with a hybrid set of cognitive capabilities that range from learning and
perception to problem solving and acting. As stated in [13], research in
cognitive architecture is important because it enables the creation and
understanding of (synthetic) agents that support the same capabilities as
humans by integrating results in cognitive sciences and AI. A key point in
the design of cognitive architectures is the management of di erent source
of knowledge and the basic capabilities needed to access and process such
knowledge. For example, knowledge from environment comes through
perception, knowledge about opportunities of a particular state of the
environment comes through planning, reasoning and prediction. Many
works in the literature have analyzed cognitive architecture [13, 15] and
have introduced and applied systems based on cognitive architectures
with interesting results. ACT-R [
and sed g lgyo -txab issen tnoO teCno rcPo
Knowledge y m o n o x a T F C I O H W
Task Execution and Interaction icenh irgnn reo aM Lae C n o i tta re n o e C rsep ignn e o R s ged aeR leow and n K and reo g C in g nn itn laP cA AI techniques mentioned above. The Knowledge Representation and Reasoning Core is the part of the architecture responsible for processing information coming from the environment. There can be two types of information the system must deal with. Sensor data about the environment the system must properly acquire, interpret and contextualize. Human instructions and commands the system can receive by interacting with persons through di erent modalities (voice and gestures). The elements Knowledge Acquisition and Data Interpretation and Multimodal Human-Robot Interaction are responsible for dealing with these two different sources of information respectively. The information received from these two di erent \channels" must be processed in a uniform way, according to a well-de ned semantics in order to extract useful knowledge. Speci cally, the Ontology and Context-based Processing relies on an ontological approach to de ne a semantics guiding the interpretation of sensor data and the related processing mechanisms. Indeed, it de nes a set of context-based rules used to process external data and extract knowledge about the events, activities and tasks that characterize the state of the environment and the assisted person. it enables a contextbased knowledge processing mechanism which allows an assistive robot to continuously re ne its internal knowledge, i.e., the element Assistive Robot Knowledge shown in Figure 1.
The knowledge generated by means of this processing mechanism is central to the (enhanced) control loop and synchronizes the three AI-cores composing the architecture. The Machine Learning Core is in charge of further enriching this knowledge by taking track of interactions and events to learn the particular needs of a speci c person/patient and build a model of his/her possible behaviors. Broadly speaking it is possible to distinguish two basic elements. The Behavior Learning element is in charge of recognizing patterns or repetitive behaviors by analyzing stored information about interactions between the assistive robots and a patient as well as the activities of a patient inside the house. The Behavior Prediction element is in charge of analayzing learned behaviors to build a suitable model of a patient and \predict" his/her possible activities and interactions accordingly. Such a model can be used to enrich the knowledge of the assistive robot and build a pro le of a patient. The pro le of a patient can also include a description of his/her health-related needs by leveraging a proper representation of the ICF Taxonomy made by WHO1. Leveraging all this information an assistive robot can build a quite rich model of a patient characterizing his/her health status as well as his/her behaviors inside the house.
The Planning and Acting Core is the part of the architecture responsible for actually interacting with the patient and the environment. It leverages the built knowledge of the assistive robot to characterize the operations that can be performed into the environment as well as the set of events and/or activities that can require support. Speci cally, the Online task reasoning element is responsible for proactively identify tasks that must be executed according to the events and activities detected by knowledge processing mechanisms of the architecture as well as commands/instructions received by a patient. These tasks are integrated into the Daily Task Planning element which maintains the (temporal) plan of the supportive tasks planned within the day. The daily plan is synthesized by taking into account a (temporal) model of the assisted person characterizing his/her speci c needs and behaviors. The tight integration of these two elements allow an assistive robot to dynamically adapt its behaviors and the executed supportive tasks according to the speci c pro le of the assisted person. Then, these tasks are executed by actually acting into the environment through a closed-loop control cycle which executes actions and receives feedbacks about their execution from the environment. 4
The AI3 architecture elicited in Figure 1 represents a sort of roadmap
for the challenging research objective we are pursuing within the KOaLa
research initiative recently started. This initiative has been inspired by
the successful results obtained with the Gira Plus research project [8].
This project developed an integrated system composed by a sensor
network and a telepresence robot aimed at supporting and monitoring the
daily-home living of a senior person directly in his/her house. Several
pilot studies were made during which a telepresence robot (the Gira ) was
actually deployed in the house of people for several months [
Gira Plus envisages an application context consisting of a mobile telepresence robot capable of autonomously interact with an older person through audio/text messages and gestures [9] as well as navigate her living environment. The robot is also endowed with videoconferencing 1 http://www.who.int/classi cations/icf/en/ 2 https://youtu.be/9pTPrA9nH6E functionalities that allow the user to communicate with a caregiver in the \external world" (e.g., a relative or a doctor). The robot is supposed to move inside a sensorized environment that can produce data about the status of the house as well as the status and activities of the user. Thus, the control architecture of a Gira Plus-like assistive robot must continuously process sensor data to understand the status of a person (according to his/her speci c health-related needs) and the environment (the operative context) and, then dynamically synthesize the actions needed to better support the user.
Following the AI3 architecture, the improvement introduced by KOaLa consists of the integration of a knowledge processing module, called the KOaLa Semantic Module, and a planning and execution module, called the KOaLa Acting Module. The integration of these two modules realizes an cognitive high-level control loop enhancing the capabilities of Gira Plus. Fig. 2 shows a conceptual representation of the envisaged cognitive architecture and highlights the di erent phases of the control ow which starts with the gathering of data from the sensor network and ends with the execution of actions in the environment involving, e.g., the robot or sensor con gurations.
Sensed Data
New Goals
-based
Action
Execution
The KOaLa Semantic Module is responsible for the interpretation of
sensor network data and the management of the resulting knowledge of the
robot. This module relies on the KOaLa Ontology to provide sensor data
with semantics and incrementally build an abstract representation of
the application context i.e., the Knowledge Base (KB). A data
processing mechanism uses standard semantic technologies based on the Web
Ontology Language (OWL) [
The KOaLa Ontology The KOaLa ontology has been de ned by leveraging SSN [7] and DUL3, two stable and publicly available ontologies. The KOaLa ontology has been structured according to a contextbased approach which characterizes the knowledge by taking into account di erent level of abstractions and perspectives. Speci cally, three levels (i.e. contexts) have been identi ed: (i) the sensor context; (ii) the environment context; (iii) the observation context. The sensor context characterizes the knowledge about the sensing devices that compose a particular environment, their deployment and the properties they may observe. This context strictly relies on SSN by providing a more detailed representation of the di erent types of sensor that can compose an environment as well as the di erent types of property that can be observed. Leveraging this general knowledge, it is possible to dynamically recognize the actual monitoring capabilities as well as the set of operations that can be performed according to the types of sensor available and their deployment. The environment context characterizes the knowledge about the structure and physical elements that compose a home environment, and the deployment of sensors. This context models the di erent physical objects that may compose a home environment, their properties and the particular deployment of the sensors. Thus, this context provides a complete characterization of a domestic environment and the relate con guration of the sensor network. Finally, the observation context characterizes the knowledge about the features that can actually produce information in a give con guration as well as the events and the activities that can be observed through them. This context identi es the observable features of a domestic environment as the physical elements that are actually capable of producing information through the deployed sensors. Similarly, this context identi es the observable properties as the properties of the observable features that can be actually observed through the deployed sensors. In this way, the KB is capable of representing observations and processing/interpreting received data by taking into account the associated environmental information like e.g., the are of the house data comes from or the type of object data refers to.
Knowledge Processing Given the semantics de ned by the KOaLa ontology, a knowledge processing mechanism elaborates sensor data to incrementally build a KB. The pipeline depicted in Fig. 3 shows the main steps of this knowledge processing mechanism. The pipeline is composed by a sequence of reasoning modules each of which elaborates data and the KB at a di erent level of abstraction (i.e., ontological context) by means of a dedicated set of inference rules. Such rules de ne a semantics to link di erent ontological contexts in order to incrementally abstract data and infer additional knowledge which is integrated into the KB. 3 http://www.loa-cnr.it/ontologies/DUL.owl
Data Filtering
and Normalization The KB is initialized on a con guration speci cation which describes the structure of the domestic environment, the set of sensors available and their deployment. Then, the Con guration Detection and Data Interpretation module generates an initial KB by analyzing the con guration speci cation. Such initial KB is then re ned by interpreting ( ltered and normalized) sensor data coming from the environment. The Feature Extraction module identi es the observable features of the environment and the related properties. It processes sensor data in order to infer observations and re ne the KB accordingly. Finally, the Event and Activity Detection module analyzes inferred observations by taking into account the knowledge about the environment. Di erent inference rules detect di erent types of events and activities according to the particular set of features and properties involved within the observations4. 4.2
The Acting Module The KOaLa Acting Module is responsible of planning and executing operations according to the events or activities inferred by the semantic module. These events are inferred by the Goal Recognition module (GR) of the knowledge processing pipeline show in Fig. 3. GR is a key element of the cognitive architecture because it provides the link between the semantic and the acting modules. Speci cally, it leverages the inferred KB to connect knowledge representation with planning. It can be seen as a background process that monitors the updated KB in order to generate operations the Gira Plus robot must perform. GR is the key feature of KOaLa to achieve proactivity. Operations that GR generates are modeled as planning goals the problem formulation process encodes into a planning problem speci cation. Such a problem speci cation is then given to a timeline-based planner which synthesizes a plan describing the sequences of operations needed to support the user. Thus, a planning and execution process leverages the timeline-based approach [6] and the PLATINUm framework [16] to continuously execute and re ne the plan according to the input goals and the status of the execution.
Planning and Execution with PLATINUm The planning and execution capabilities of the acting module rely on a novel timelinebased framework, called PLATINUm [16]. PLATINUm complies with the formal characterization of the timeline-based approach proposed in 4 The knowledge processing mechanism has been developed by means of the Apache Jena software library (https://jena.apache.org/) [6] which takes into account temporal uncertainty, and has been successfully applied in real-world manufacturing scenarios [14] recently. Broadly speaking, a timeline-based model is composed by a set of state variables describing the possible temporal behaviors of the domain features that are relevant from the control perspective. Each state variable speci es a set of values that represent the states or actions the related feature may assume or perform over time. Each value is associated with a exible duration and a controllability tag which speci es whether the value is controllable or not. A state transition function speci es the valid temporal behaviors of a state variable by modeling the allowed sequences of values (i.e., the transitions between the values of a state variable). State variables model \local" constraints a planner must satisfy to generate valid temporal behaviors of single features of the domain i.e., valid timelines. it could be necessary to further constrain the behaviors of state variables in order to coordinate di erente domain features and realize complex functionalities or achieve complex goals (e.g., perform assistive functionalities). A dedicated set of rules called synchronization rules model \global" constraints that a planner must satisfy to build a valid plan. Such rules can be used also to specify planning goals.
Given such a model, a PLATINUm planner synthesizes a set of timelines each of which represents an envelope of valid temporal behaviors of a particular state variable. These timelines allow the Gira Plus robot to perform the desired assistive tasks. Then, an PLATINUm executive carries out the timelines by temporally instantiating the associated sequences of values, called tokens. Namely, an executive decides the exact start time of the tokens composing the timelines of the plan. In general, the actual execution of these tokens cannot be controlled by the executive which must dynamically adapt the plan according to the feedbacks received during execution. For example, the actual time the Gira Plus robot needs to navigate the environment and reach a particular location cannot be decided by the planner or the executive. Indeed, the navigation can be slowed-down by obstacles and therefore the end and therefore the actual duration of a navigation operation is known only when the executive receives the associated execution feedback. 5
Let us consider a typical assistive scenario consisting of older adult living alone in his/her single oor apartment composed by a living room, a kitchen, a bathroom, a bedroom and a central corridor connecting all the rooms with the entrance. There are many sensors that can be installed to track activities and events inside the house. Each window and the entrance door have been endowed with a sensor to check whether they are open or close. There is (at least) one sensor for each room to track temperature and luminosity and detect motions. Finally, there are additional sensors to track the usage of electronic devices like e.g., the TV, the oven or the microwave. In addition to these environmental sensors, there are other sensing devices that track physiological parameters of the assisted person like blood pressure, heart rate, glucose level, etc. All these sensing devices provide a rich and heterogeneous set of data the assistive robot can continuously analize through KOaLa to recognize activities the person is performing, or events/situations a ecting the status of the house or the health of the person. Below there are some examples concerning typical situations we are focusing to inside the house during the daily-home living of a person in need of assistance. These examples show the objective of the enhanced assistive services: { The sensor network detects some activities in the kitchen of the house. The information gathered from sensors is saying that someone is moving inside the kitchen, the TV is on, the luminosity is high and also that the temperature close to the ame is a bit higher than usual. Given this information and given the time, the assistive robot understands that \Nonna Lea" is cooking and therefore it plans to move towards the kitchen to remind the dietary restrictions she must follows. Then, the assistive robot plans to remind the patient to take his/her pills for the therapy in forty- ve minutes which is the time she usually takes to complete the meal. In addition, the robot plans to send a message to her sons to ask them to call their mother in one hour and a half in order to check whether she has actually taken the pills or not. { The sensor network detects some activities in the living room of the house. The information gathered from sensors is saying that someone is sitting on the sofa, the TV is on and that the luminosity of the room is high. According to these information the robot understands that the person is watching the TV and therefore it plans to move inside the living room and inform the person about the programming of the day. In addition, the robot notices that the person has neither made any calls to nor received calls from her sons today and therefore it plans to suggest to the person to call her sons before going to sleep. { The sensor network detects some activities in the bedroom of the house. The information gathered from sensors is saying that someone is moving inside the bedroom and that the light inside the room is on. The robot checks the time and recognizes that it is the time at which the person usually goes to bed. However, it detects that the window inside the kitchen is open and that the light of the bathroom is on. Thus, the robot plans to move towards the bedroom in order to alert the person about the fact that the window must be closed and that the light in the bathroom must be turned o before going to bed. After a while, the information gathered from sensors in the bedroom says that someone is laying on the bed and that the luminosity of the room is low. The robot understands that the person is sleeping and decides to notify her sons about this. In addition, the robot notices that the temperature inside the bedroom is a bit higher than the ideal temperature for a good sleep. Thus, it decides to cool down a bit the temperature of the room by controlling either the air-conditioner or the heater according to season. The day after, the robot detects that the person is still sleeping at the typical time she wakes up. Thus, the robot plans to send a message to her sons about this unusual behavior within thirty minutes if she does not wake up before. Such scenarios show ordinary life situations of a senior person and some roles that an assistive robot, with the help of KOaLa, can play to support her living at home. In particular, these scenarios show that KOaLa, through the combination of simple inference rules like e.g., someone is moving inside the kitchen and the temperature close to the ame is higher then usual, can endow a telepresence robot with the capability of autonomously reasoning on the state of the environment, inferring complex situations and dynamically triggering goals accordingly. A rst set of inference rules has been developed to realize a \strati ed" reasoning mechanisms capable of abstracting sensor data and inferring events situations concerning the status of the environment. Currently an extended set of rules is under development to realize the goal triggering mechanism needed to proactively link knowledge reasoning to planning and acting. 6
This paper presented an AI-based cognitive architecture which integrates sensing, knowledge representation and automated planning techniques to constitute a high-level control loop to enhance proactivity features of an assistive robot designed to support an older persons living at home in her daily routine. A semantic module leverages a dedicated ontology to build a KB by properly processing data collected by means of a sensor network installed in the environment. An acting module takes advantage of the timeline-based planning approach to control robot behaviors. A goal triggering process acts as a bridge between the two modules and provides the key enabling feature to endow the robot with suitable proactivity levels. At this stage, some tests have been performed to show the feasibility of the approach. Further work is ongoing to enable more extensive integrated laboratory tests to better assess performance and capabilities of the overall system. Future work will also investigate the opportunity to integrate machine learning techniques to better adapt the behavior of the assistive robot to speci c daily behaviors of di erent targeted people.