Social Robotics is a research eld aiming at designing robots able to interact with people in a natural manner. Within the domain of Socially Assistive Robotics the capability of adapting and personalizing behaviors and assistive services of robots, according to the speci c assistive context and needs of a person is crucial to improve the e cacy in users support and hence acceptance. The authors rely on some recent results concerning the realization of a cognitive control approach for assistive robots supporting the synthesis of personalized and exible assistive behaviors. This paper takes into account a general rehabilitation scenario and presents some initial steps toward the integration of perception, knowledge representation and planning capabilities to pursue exibility, adaptation and personalization of assistive robot behaviors.
The research eld of Socially Assistive Robotics (SAR) aims at designing robots
capable to assist fragile users supporting their daily living activities and also rely
on Social Interaction features [
End-users have speci c health-related needs determining the type of
assistance they need. Secondary users see in a SAR system a means to improve their
quality of work and facilitate communication with patients. Previous experiences
in domestic assistance scenarios like e.g. the [
In this context, a SAR control system should integrate a large variety of
(con icting) requirements. To satisfy such requirements in a exible and
adaptable way, SAR systems should encapsulate a number of cognitive capabilities to
autonomously i) reason about these requirements, ii) nd out the most suitable
set of assistive services needed in a speci c context and, iii) con gure and adapt
these services in order to achieve the desired assistive objectives. To endow an
assistive robot with such cognitive capabilities we have recently started a research
initiative called KOaLa (Knowledge-based cOntinuous Loop) aimed at developing
a novel cognitive control architecture for assistive robots [
A recently started research project, called SI-Robotics (SocIal ROBOTics for active and healthy ageing) presents a number of interesting assistive challenges that represent a good opportunity to enhance and evaluate the capabilities of KOaLa in realistic assistive scenarios. Taking into account the challenges raised by a generic rehabilitation scenario where an assistive robot should support both therapists and patients, the contributions of the paper consist of: (i) a general presentation of SI-Robotics and the considered applicative scenarios; (ii) a discussion of Human-Robot Interaction issues related to adaptive SAR systems; (iii) a high-level cognitive control architecture for SAR extending KOaLa with the integration of brain-inspired perception and emotion recognition capabilities. 2
SI-Robotics is an Italian research project whose aim is to design and develop novel solutions of collaborative assistive robotics capable of supporting humans in healthcare scenarios and interacting with them in a socially acceptable way. The scienti c objective of the project is to investigate and develop advanced software and robotic solutions for assisting seniors in a variety of situations that range from daily-home living support to continuous monitoring of health-related
RESIDENTIAL SCENARIOS • Teleservice • Health Monitoring • Coaching • Cognitive Stimulation
HOSPITAL SCENARIOS • Welcoming and orientation • Therapy and Rehabilitation
Support • Patient Monitoring
• Knowledge Representation and
Reasoning • Ontology • Common Sense Reasoning • Natural Language Processing • Sensor Data Fusion • Decision Making • Planning and Execution • Learning and Adaptation conditions to possibly facilitate an early detection of cognitive decline like e.g., early dementia or mild cognitive impairment.
SI-Robotics identi es the integration of core technologies like Robotics, Internet of Things (IoT) and Arti cial Intelligence (AI) as the key enabling feature to realize an innovative SAR system capable of synthesizing exible assistive behaviors tailored on the speci c needs of primary users.
Figure 1 shows a conceptual view of SI-Robotics. Central to the project is the development of novel sensorized robotic platforms. On the left side of the gure, there are a number of heterogeneous assistive services and scenarios considered within the project. Such services are supported by means of the integration on a novel modular robotic platform of a number of advanced AI technologies Speci cally, two types of scenarios are considered in SI-Robotics: (i) residential scenarios and; (ii) hospital scenarios.
Residential scenarios concern assistive services where usually the assistance is carried out in restricted environments like the social houses or the house of the patient. The assistive services are mainly targeted in supporting the daily home living of a person over a long temporal horizon. The envisaged services in this context are: (i) Teleservice or teleassistance where the robot acts as a communication channel allowing external persons like e.g., doctors or relatives to contact and talk to the target senior; (ii) Health monitoring where the robot through a number of physiological and environmental sensing devices continuously monitor the activities and health parameters of the target senior and proactively triggers alerts and/or noti cations when some \not regular" event happens; (iii) Coaching where the robot is meant to support the target seniors in continuing his/her rehabilitation therapy when he/she is back from from hospital. (iv) Cognitive stimulation where the robot continuously interact with the target person and constantly stimulate and evaluate his/her cognitive capabilities through a number of dedicated games properly integrated into the system (gami cation).
Hospital scenarios concern assistive services where the assistance is carried out in public environment (i.e., a hospital) and where the robot could interact simultaneously with several target seniors. However, unlike the other scenarios, the assistance and the interactions needed to realize services that span over a reduced temporal horizon and concern: (i) Welcoming and orientation where the robot is placed at the entrance of the hospital and is in charge of realizing social functionalities to interact with di erent people providing useful information; (ii) Rehabilitation Support where the robot is meant to support a therapist in performing the rehabilitation tasks with one ore more seniors. (iii) Patient Monitoring where the robot is in charge of constantly monitoring the health conditions of bedridden patients, autonomously identify critical conditions and promptly notify/alarm the medical sta . 3
The scenarios and services of SI-Robotics are well suited to evaluate the holistic
approach pursued within KOaLa [
Environment Perspective IoT Devices
Smart
Environments Autonomy Perspective Skills, Navigation, & Communication Interaction Perspective User Categories Personalization Perspective FUesaetruNreeseds &
The left-side shows the technological features of a SAR system. Here, we have two perspectives that characterize the capabilities of the system. The environment perspective concerns perception capabilities, taking into account the IoT devices like e.g., environmental or physiological sensors that allow the system to gather information about the environment and the state of the assisted person. The autonomy perspective concerns the skills, the operations and the autonomy levels of the assistive robot, determining the possible interactions with the environment.
The right-side of Figure 2 instead characterizes the behavioral features of a SAR system. The related two perspectives determine the set of assistive services needed for the considered users as well as the \shape" of the robot behaviors that carry out such services. The interaction perspective concerns the de nition of the di erent types of user that interact with the system. The personalization perspective concerns the identi cation of the health-related and cognitive features that a ect the modalities of interaction between the robot and the person who receives the assistance. It determines the parameters/constraints to consider in order to realize e ective assistive behaviors.
Given this multi-perspective approach and considering again the assistive services of SI-Robotics, a particularly interesting one is the Rehabilitation Support for hospital scenarios. Such a service requires a robot to support both a patient and a therapist during the execution of a general rehabilitation. This service presents a number of interesting aspects with respect to behavior adaptation and human-robot interactions. First of all, the robot should be capable of interacting with two di erent types of user (i.e., the patient and the therapist), providing them with di erent information and functionalities. Then, the robot should be capable of facing a variety of situations in terms of types of rehabilitation procedure and health conditions of the assisted person to monitor. Namely, the robot should be capable of supporting several rehabilitation therapies and monitoring di erent technical and physiological parameters to assess the correct execution of planned exercises.
To achieve the desired quality level of assistance we conceive a three-step procedure consisting of: (i) a con guration and training step allowing the robot to interact with the therapist learn the exercises composing the rehabilitation procedure and the technical parameters/features as well as the quality metrics to monitor for exercise assessment; (ii) a pro ling step allowing the robot to interact with the patient and learn his/her health-related needs and cognitive capabilities representing additional information to consider during the execution of the exercises; (iii) a monitoring and control step alllowing the robot to show to the patients the exercises he/she must perform, to monitor the parameters/features con gured for the assessment and then to intervene when necessary. 3.1
The role of the robot is to support the therapist in the administration of rehabilitation exercises. The therapist trains the robot in pro ling the user (and his/her rehabilitation) and in providing data about correct executions of the rehabilitation therapy.
During the rst step of the con guration, the therapist inserts generic data of the user (i.e. name, age, gender, nationality) as well as clinical data related to health condition. As second step, the list of exercises and the parameters of interest are con gured by the therapist himself. These data are important to determine the kind of interaction the user is more prone to and to assess the rehabilitation procedure. In the training phase, the robot learns the features to monitor during the execution in order to evaluate observed performances. Therefore, the robot acts passively to the rehabilitation procedure. It stands close to the therapist and collects data.
We can suppose that the robot is endowed with an internal representation of rehabilitation exercises and the associated physiological and physical parameters that are considered for performance assessment. The therapist con gures the robot by selecting the exercises the robot is going to \learn" and the parameters to observe for the assessment. During the training phase, the robot monitors the exercises through the selected parameters. Then, the therapist enters his/her own evaluation at the end of each exercise. In this way, the robot internally build a \training set" enabling the autonomous evaluation of the (known) exercises. 3.2
To assess the pro le of the patient, the assistive robot collects information about the health-related needs of the patient who performs the therapy. It is important to know physical as well as cognitive capabilities and features of a patient in order to properly interpret observed behaviors. The importance of collecting these kinds of information is twofold. From one side, it allows the robot to update the information of the patient and make it accessible to the therapist(re ning phase). Furthermore, it is important to automatically customize the rehabilitation exercises the patient needs to perform. For example, patients a ected by short-term memory loss can be constantly reminded by the robot about the exercises they are going to perform, explain the steps that compose the rehabilitation procedure and how they should be executed.
On the other side, this information is crucial to allow the robot to interact with the patient in an e ective way and to explain decisions and possible changes in the execution or in the structure of the proposed therapy. For example, the robot can prefer a text-based interaction modality if the patient is a ected by hearing impairments or a voice-based interaction modality if the patient is affected by eyesight impairments.
More in general, knowing health-related needs of a patient allows a robot to justify exercises with respect to the health conditions of the patient but also to recognize hazardous or critical situations and ask the intervention of a therapist. 3.3
The tasks performed by the robot in this phase require the ability of monitoring the performance and the engagement of the patient who is executing the exercise. It needs to make the user feel safe and comfortable in exercising. According to the outcomes of monitoring activities the robot may decide to interact with the patient in di erent ways during the execution of the exercise. For example the robot can encourage the patient if an exercise has been performed well or viceversa can interrupt and explain again the exercise if the patient performs too bad. The following subsections detail the monitoring procedures. Performance Monitoring From a technical point of view, the robot monitors the patient's performance by collecting data from visual sensors (i.e. cameras) Heartrate80.0 Respiratoryrate 12
Heartrate80.0 Respiratoryrate 12
Heartrate80.0 Respiratoryrate 12 (a) (b) (c) and wearable sensors (i.e. inertial sensors), if requested by the therapist. The data are then displayed on the screen to provide a feedback to the patient. On the same time, the online analysis of physiological data helps the robot to assess the quality of the performance. Due to this analysis, the robot can identify 3 scenarios, as shown in Figure 3: 1. Good performance: the patient is correctly performing the rehabilitation exercise and the value of the physiological parameters are uncritical (Figure 3(a)); 2. Bad performance: the patient is not correctly performing the rehabilitation exercise (Figure 3(b)); 3. Alert performance: the patient is correctly exercising but the physiological parameters got critical values (Figure 3(c)).
Engagement Monitoring The engagement monitoring allows the robot to modify its interaction plan based on the feelings expressed by the patient during the exercise. Based on the scenario described in the previous subsection, if the robot notices that the patient is correctly exercising, it will keep motivating him, like a personal coach. Otherwise, if the robot recognizes that the patient is incorrectly performing the rehabilitation task, it will slow down the exercise and it will suggest the correct execution. In case the user is not motivated in performing the rehabilitation, the robot will try to get his attention in order to increase the engagement. When Alert performance scenario is detected, the robot will stop the rehabilitation session and it will comfort him with some routine's questions. 4
To realize the desired assistive services it is necessary to design and develop an advanced intelligent control system capable of implementing and integrating the numerous cognitive capabilities needed to successfully achieve the desired objectives. Starting from the cognitive architecture de ned within KOaLa, advanced perception capabilities are needed to gather and process data from di erent sensing devices (e.g., video-cameras, environmental sensors and physiological sensors). Representation and abstraction capabilities are needed to integrate and contextualize sensory information in order to recognize di erent \assistive contexts" and situations, allowing the robot to incrementally build a kind of consciousness. According to this knowledge decision making and acting capabilities are needed to decide which high-level action to perform and synthesize an interaction plan to proactively support an end-user.
Taking inspiration from research in cognitive architecture [
Figure 4 proposes a schematic representation of a cognitive architecture for adaptive rehabilitation. The cognitive control approach relies on three main modules: (i) A perception module is in charge of realizing the raw-data processing mechanisms needed to extract useful information from sensory inputs; (ii) A knowledge module is in charge of encoding information about the therapy and rehabilitation exercises, metrics for performance evaluation and patients' state in terms of health-related needs and emotions; (iii) A planning and acting module is in charge of making decisions about how to support the rehabilitation therapy, selecting the exercises to perform and the interactions needed to correct or support the rehabilitation.
Perception Engagement Monitoring Performance Monitoring
Therapy Measures
Knowledge
KB Exercise Monitoring
Physiological
State Emotional
State Exercise Support and Adaptation
Planning & Acting
Goal Reasoning Therapy Execution
Therapy
Synthesis
Perception module allows to assess the emotional and engagement state of the
user, while he/she is performing the exercise. In details, the perceptual system
aims at converting raw data coming form the sensory equipment into behavioural
patterns. The perceptual system presented in this work resembles the abstraction
process occurring in the human brain. The capability of the brain to process
stimuli from the environment is mimicked by the interconnection of three modules,
denoted as: thalamus, sensory cortex and associative cortex. The functionality
of each module shows analogies with the abilities of the corresponding
humanbeings' neural structure. Namely, the thalamus module is the one responsible of
gathering sensory data. Sensory cortex is composed by multiple modules, one
for each sensory modality, which extract the features of interest used by the
associative cortex to assess the multi-modal pattern describing the human's
behaviour. The ow of information characterizing the overall system recalls the
human inherent ability of automatically assessing the mental state, feeling and
other personal traits, as described by the Theory of Mind [
To support the desired level of interaction and adaptability, the robot should exchange information with users as well as understand information or instructions from them and correctly interpreting \signals" from the environment. It is therefore necessary to endow an assistive robot with some sort of knowledge in order to deal with information about rehabilitation exercises and parameters considered for the assessment. To characterize such knowledge, we follow an ontological-based approach to de ne a clear semantics of the general concepts and properties the robot deals with in the considered rehabilitation scenarios.
To this aim, we will extend the KOaLa ontology, previously designed for
domestic assistance [
The KOaLa ontology is also extended with the representation of the emotional states of a person that can be recognized by the integrated perception capabilities. This knowledge allows the robot to maintain an internal representation of the mental state of the assisted person and contextualize the observations accordingly. 4.3
The ontology de nes a clear semantics of the heterogeneous concepts and
properties the robot must deal with in di erent assistive scenarios. This semantics
guides the knowledge reasoning processes that link perception module to
planning and acting module of Figure 4. KOaLa reasoning processes interpret and
contextualize perception information according to the semantics de ned by the
ontology [
This paper presents some initial design e orts aimed at realizing a novel cognitive control system for SAR systems within the Italian research project SI-Robotics. The paper focuses on a speci c assistive scenario for rehabilitation where an assistive robot is supposed to support therapists as well as patients during the execution of some exercises. The contribution of the paper consists in proposing an initial integrated view of perception, knowledge representation and acting capabilities. Perception capabilities should extract information useful for the assessment of an exercises (correctness and engagement). Then, information should be modeled by a dedicated ontology and then integrated into a knowledge-based control architecture (KOaLa) to dynamically adapt the behavior of an assistive robot to the speci c health-related needs and state of a person. Next steps, will push on the concrete integration and development of a prototype that can be evaluated in real rehabilitation scenarios.
Research supported by \SocIal ROBOTics for active and healthy ageing" (SIROBOTICS) project founded by the Italian \Ministero dell'Istruzione, dell' Universita e della Ricerca" under the framework \PON 676 - Ricerca e Innovazione 2014-2020", Grant Agreement ARS01 01120.