The research field of Socially Assistive Robotics (SAR) aims at designing robots capable to assist fragile users supporting their daily living activities and also rely on Social Interaction features [11]. This kind of robots is to provide people with continuous support and assistance, possibly facing a significant number of heterogeneous tasks [16] such as reminding the dietary restrictions and medical appointments or monitoring the heart rate or the sleep quality of a person. To this purpose adaptivity constitutes a key capability. Personalization and adaptation features in robotics architecture are strongly required to effectively address the specific needs of a person as well as to achieve a good level of acceptance [17,19]. To this aim, a key point is to see Socially Assistive Robots (SAR) as complex systems (SAR systems) merging requirements that come from end-users (or patients), secondary users (e.g., caregivers or health-care professionals) and the social robot itself.
End-users have specific health-related needs determining the type of assistance they need. Secondary users see in a SAR system a means to improve their quality of work and facilitate communication with patients. Previous experiences in domestic assistance scenarios like e.g. the [6,10,7] have clearly pointed out the role of SAR systems as technological means capable of performing assistive functionalities that can support the daily living of an assisted person as well as support the proactive intervention of external/third persons like e.g. caregivers or health-care professionals. Finally, a social robot itself has specific capabilities like e.g., autonomous navigation, object manipulation, multi-modal interaction and so on, that may determine the set of assistive services the resulting SAR system can actually support.
In this context, a SAR control system should integrate a large variety of (conflicting) requirements. To satisfy such requirements in a flexible and adaptable way, SAR systems should encapsulate a number of cognitive capabilities to autonomously i) reason about these requirements, ii) find out the most suitable set of assistive services needed in a specific context and, iii) configure and adapt these services in order to achieve the desired assistive objectives. To endow an assistive robot with such cognitive capabilities we have recently started a research initiative called KOaLa (Knowledge-based cOntinuous Loop) aimed at developing a novel cognitive control architecture for assistive robots [22,2,3]. The developed prototype has been evaluated in simulated domestic assistance scenarios showing the desired level of proactivity, personalization and adaptation, concerning robot behaviors [21].
A recently started research project, called SI-Robotics (SocIal ROBOTics for active and healthy ageing) presents a number of interesting assistive challenges that represent a good opportunity to enhance and evaluate the capabilities of KOaLa in realistic assistive scenarios. Taking into account the challenges raised by a generic rehabilitation scenario where an assistive robot should support both therapists and patients, the contributions of the paper consist of: (i) a general presentation of SI-Robotics and the considered applicative scenarios; (ii) a discussion of Human-Robot Interaction issues related to adaptive SAR systems; (iii) a high-level cognitive control architecture for SAR extending KOaLa with the integration of brain-inspired perception and emotion recognition capabilities.
SI-Robotics is an Italian research project whose aim is to design and develop novel solutions of collaborative assistive robotics capable of supporting humans in healthcare scenarios and interacting with them in a socially acceptable way. The scientific objective of the project is to investigate and develop advanced software and robotic solutions for assisting seniors in a variety of situations that range from daily-home living support to continuous monitoring of health-related conditions to possibly facilitate an early detection of cognitive decline like e.g., early dementia or mild cognitive impairment. SI-Robotics identifies the integration of core technologies like Robotics, Internet of Things (IoT) and Artificial Intelligence (AI) as the key enabling feature to realize an innovative SAR system capable of synthesizing flexible assistive behaviors tailored on the specific needs of primary users. Residential scenarios concern assistive services where usually the assistance is carried out in restricted environments like the social houses or the house of the patient. The assistive services are mainly targeted in supporting the daily home living of a person over a long temporal horizon. The envisaged services in this context are: (i) Teleservice or teleassistance where the robot acts as a communication channel allowing external persons like e.g., doctors or relatives to contact and talk to the target senior; (ii) Health monitoring where the robot through a number of physiological and environmental sensing devices continuously monitor the activities and health parameters of the target senior and proactively triggers alerts and/or notifications when some "not regular" event happens; (iii) Coaching where the robot is meant to support the target seniors in continuing his/her rehabilitation therapy when he/she is back from from hospital. (iv) Cognitive stimulation where the robot continuously interact with the target person and constantly stimulate and evaluate his/her cognitive capabilities through a number of dedicated games properly integrated into the system (gamification).
Hospital scenarios concern assistive services where the assistance is carried out in public environment (i.e., a hospital) and where the robot could interact simultaneously with several target seniors. However, unlike the other scenarios, the assistance and the interactions needed to realize services that span over a reduced temporal horizon and concern: (i) Welcoming and orientation where the robot is placed at the entrance of the hospital and is in charge of realizing social functionalities to interact with different people providing useful information; (ii) Rehabilitation Support where the robot is meant to support a therapist in performing the rehabilitation tasks with one ore more seniors. (iii) Patient Monitoring where the robot is in charge of constantly monitoring the health conditions of bedridden patients, autonomously identify critical conditions and promptly notify/alarm the medical staff.
The scenarios and services of SI-Robotics are well suited to evaluate the holistic approach pursued within KOaLa [22] and to enhance the developed cognitive capabilities [21]. Figure 2 gives a general view of the proposed holistic approach. The left-side shows the technological features of a SAR system. Here, we have two perspectives that characterize the capabilities of the system. The environment perspective concerns perception capabilities, taking into account the IoT devices like e.g., environmental or physiological sensors that allow the system to gather information about the environment and the state of the assisted person. The autonomy perspective concerns the skills, the operations and the autonomy levels of the assistive robot, determining the possible interactions with the environment.
The right-side of Figure 2 instead characterizes the behavioral features of a SAR system. The related two perspectives determine the set of assistive services needed for the considered users as well as the "shape" of the robot behaviors that carry out such services. The interaction perspective concerns the definition of the different types of user that interact with the system. The personalization perspective concerns the identification of the health-related and cognitive features that affect the modalities of interaction between the robot and the person who receives the assistance. It determines the parameters/constraints to consider in order to realize effective assistive behaviors.
Given this multi-perspective approach and considering again the assistive services of SI-Robotics, a particularly interesting one is the Rehabilitation Support for hospital scenarios. Such a service requires a robot to support both a patient and a therapist during the execution of a general rehabilitation. This service presents a number of interesting aspects with respect to behavior adaptation and human-robot interactions. First of all, the robot should be capable of interacting with two different types of user (i.e., the patient and the therapist), providing them with different information and functionalities. Then, the robot should be capable of facing a variety of situations in terms of types of rehabilitation procedure and health conditions of the assisted person to monitor. Namely, the robot should be capable of supporting several rehabilitation therapies and monitoring different technical and physiological parameters to assess the correct execution of planned exercises.
To achieve the desired quality level of assistance we conceive a three-step procedure consisting of: (i) a configuration and training step allowing the robot to interact with the therapist learn the exercises composing the rehabilitation procedure and the technical parameters/features as well as the quality metrics to monitor for exercise assessment; (ii) a profiling step allowing the robot to interact with the patient and learn his/her health-related needs and cognitive capabilities representing additional information to consider during the execution of the exercises; (iii) a monitoring and control step alllowing the robot to show to the patients the exercises he/she must perform, to monitor the parameters/features configured for the assessment and then to intervene when necessary.
The role of the robot is to support the therapist in the administration of rehabilitation exercises. The therapist trains the robot in profiling the user (and his/her rehabilitation) and in providing data about correct executions of the rehabilitation therapy.
During the first step of the configuration, the therapist inserts generic data of the user (i.e. name, age, gender, nationality) as well as clinical data related to health condition. As second step, the list of exercises and the parameters of interest are configured by the therapist himself. These data are important to determine the kind of interaction the user is more prone to and to assess the rehabilitation procedure. In the training phase, the robot learns the features to monitor during the execution in order to evaluate observed performances. Therefore, the robot acts passively to the rehabilitation procedure. It stands close to the therapist and collects data.
We can suppose that the robot is endowed with an internal representation of rehabilitation exercises and the associated physiological and physical parameters that are considered for performance assessment. The therapist configures the robot by selecting the exercises the robot is going to "learn" and the parameters to observe for the assessment. During the training phase, the robot monitors the exercises through the selected parameters. Then, the therapist enters his/her own evaluation at the end of each exercise. In this way, the robot internally build a "training set" enabling the autonomous evaluation of the (known) exercises.
To assess the profile of the patient, the assistive robot collects information about the health-related needs of the patient who performs the therapy. It is important to know physical as well as cognitive capabilities and features of a patient in order to properly interpret observed behaviors. The importance of collecting these kinds of information is twofold. From one side, it allows the robot to update the information of the patient and make it accessible to the therapist(refining phase). Furthermore, it is important to automatically customize the rehabilitation exercises the patient needs to perform. For example, patients affected by short-term memory loss can be constantly reminded by the robot about the exercises they are going to perform, explain the steps that compose the rehabilitation procedure and how they should be executed.
On the other side, this information is crucial to allow the robot to interact with the patient in an effective way and to explain decisions and possible changes in the execution or in the structure of the proposed therapy. For example, the robot can prefer a text-based interaction modality if the patient is affected by hearing impairments or a voice-based interaction modality if the patient is affected by eyesight impairments.
More in general, knowing health-related needs of a patient allows a robot to justify exercises with respect to the health conditions of the patient but also to recognize hazardous or critical situations and ask the intervention of a therapist.
The tasks performed by the robot in this phase require the ability of monitoring the performance and the engagement of the patient who is executing the exercise. It needs to make the user feel safe and comfortable in exercising. According to the outcomes of monitoring activities the robot may decide to interact with the patient in different ways during the execution of the exercise. For example the robot can encourage the patient if an exercise has been performed well or viceversa can interrupt and explain again the exercise if the patient performs too bad. The following subsections detail the monitoring procedures.
Performance Monitoring From a technical point of view, the robot monitors the patient's performance by collecting data from visual sensors (i.e. cameras)
The engagement monitoring allows the robot to modify its interaction plan based on the feelings expressed by the patient during the exercise. Based on the scenario described in the previous subsection, if the robot notices that the patient is correctly exercising, it will keep motivating him, like a personal coach. Otherwise, if the robot recognizes that the patient is incorrectly performing the rehabilitation task, it will slow down the exercise and it will suggest the correct execution. In case the user is not motivated in performing the rehabilitation, the robot will try to get his attention in order to increase the engagement. When Alert performance scenario is detected, the robot will stop the rehabilitation session and it will comfort him with some routine's questions.
To realize the desired assistive services it is necessary to design and develop an advanced intelligent control system capable of implementing and integrating the numerous cognitive capabilities needed to successfully achieve the desired objectives. Starting from the cognitive architecture defined within KOaLa, advanced perception capabilities are needed to gather and process data from different sensing devices (e.g., video-cameras, environmental sensors and physiological sensors). Representation and abstraction capabilities are needed to integrate and contextualize sensory information in order to recognize different "assistive contexts" and situations, allowing the robot to incrementally build a kind of consciousness. According to this knowledge decision making and acting capabilities are needed to decide which high-level action to perform and synthesize an interaction plan to proactively support an end-user.
Taking inspiration from research in cognitive architecture [15,14,1], we here propose the integration of three AI-based processes implementing the cited capabilities. Specifically, we here propose an extension to a knowledge-based framework for cognitive control called KOaLa [4,3] (Knowledge-based cOntinuous Loop) in order to integrate additional sensory information and enhance the reasoning and acting capabilities of the approach.
Figure 4 proposes a schematic representation of a cognitive architecture for adaptive rehabilitation. The cognitive control approach relies on three main modules: (i) A perception module is in charge of realizing the raw-data processing mechanisms needed to extract useful information from sensory inputs; (ii) A knowledge module is in charge of encoding information about the therapy and rehabilitation exercises, metrics for performance evaluation and patients' state in terms of health-related needs and emotions; (iii) A planning and acting module is in charge of making decisions about how to support the rehabilitation therapy, selecting the exercises to perform and the interactions needed to correct or support the rehabilitation. Perception module allows to assess the emotional and engagement state of the user, while he/she is performing the exercise. In details, the perceptual system aims converting raw data coming form the sensory equipment into behavioural patterns. The perceptual system presented in this work resembles the abstraction process occurring in the human brain. The capability of the brain to process stimuli from the environment is mimicked by the interconnection of three modules, denoted as: thalamus, sensory cortex and associative cortex. The functionality of each module shows analogies with the abilities of the corresponding humanbeings' neural structure. Namely, the thalamus module is the one responsible of gathering sensory data. Sensory cortex is composed by multiple modules, one for each sensory modality, which extract the features of interest used by the associative cortex to assess the multi-modal pattern describing the human's behaviour. The flow of information characterizing the overall system recalls the human inherent ability of automatically assessing the mental state, feeling and other personal traits, as described by the Theory of Mind [12]. The final output of this system is to assess specific emotional and engagement states. Among the emotional states described by [9], joy, anger, fear, sadness and surprise are of interest. The engagement state of the user is adopted to assess an additional feelings, like boredom (low level of engagement) and excitement (high level of engagement).
To support the desired level of interaction and adaptability, the robot should exchange information with users as well as understand information or instructions from them and correctly interpreting "signals" from the environment. It is therefore necessary to endow an assistive robot with some sort of knowledge in order to deal with information about rehabilitation exercises and parameters considered for the assessment. To characterize such knowledge, we follow an ontological-based approach to define a clear semantics of the general concepts and properties the robot deals with in the considered rehabilitation scenarios.
To this aim, we will extend the KOaLa ontology, previously designed for domestic assistance [3], by introducing concepts and properties needed to manage the needed information. The KOaLa ontology relies on the DOLCE foundational ontology [13] and the SSN ontology [5]. Concepts and properties related to rehabilitation exercises are modeled as DOLCE:Process and characterized in terms of the effects on some physical/physiological parameters of a person. In this regard, the KOaLa ontology integrates a representation of the ICF [18] classification proposed by WHO (World Health Organization) to generally describe health-related conditions of persons. This knowledge is crucial in this context to build profiles of end-users but also to link rehabilitation exercises to healthrelated parameters to monitor for the assessment. Also, this knowledge is crucial for supporting explainability [8]. Using the ICF taxonomy indeed the robot can explain and motivate the exercises as well as positive or negative evaluations to end-users by taking into account his/her health-related needs.
The KOaLa ontology is also extended with the representation of the emotional states of a person that can be recognized by the integrated perception capabilities. This knowledge allows the robot to maintain an internal representation of the mental state of the assisted person and contextualize the observations accordingly.
The ontology defines a clear semantics of the heterogeneous concepts and properties the robot must deal with in different assistive scenarios. This semantics guides the knowledge reasoning processes that link perception module to planning and acting module of Figure 4. KOaLa reasoning processes interpret and contextualize perception information according to the semantics defined by the ontology [2,20]. Such processes continuously refine an internal Knowledge Based (KB) which characterizes the assistive scenario with respect to different abstraction levels and perspectives.
This paper presents some initial design efforts aimed at realizing a novel cognitive control system for SAR systems within the Italian research project SI-Robotics. The paper focuses on a specific assistive scenario for rehabilitation where an assistive robot is supposed to support therapists as well as patients during the execution of some exercises. The contribution of the paper consists in proposing an initial integrated view of perception, knowledge representation and acting capabilities. Perception capabilities should extract information useful for the assessment of an exercises (correctness and engagement). Then, information should be modeled by a dedicated ontology and then integrated into a knowledge-based control architecture (KOaLa) to dynamically adapt the behavior of an assistive robot to the specific health-related needs and state of a person. Next steps, will push on the concrete integration and development of a prototype that can be evaluated in real rehabilitation scenarios.
Research supported by "SocIal ROBOTics for active and healthy ageing" (SI-ROBOTICS) project founded by the Italian "Ministero dell'Istruzione, dell' Università e della Ricerca" under the framework "PON 676 -Ricerca e Innovazione 2014-2020", Grant Agreement ARS01 01120.