- Assistive technology is playing a crucial role in supporting caregivers and patients with neurological disorders to carry out home care services. Nevertheless, the technological advances allowing for the automation of such services often lead to systems composed of several devices that make patients not always comfortable when interacting with them, so limiting their effective use. In order to improve the acceptance level of an assistive system, a robotic system is proposed to act as a smart sensor whose functionalities to control its behavior as well as to provide meaningful observations from the input data are modeled as services. The system is supported by a middleware able to automatically schedule a set of home care services that are personalized for each patient considering the personal daily routine, the cognitive status, and the personality profile. In addition, the proposed system is able to react to dynamic changes in the patient's state by modifying the robot behavior and adapting on the fly the set of proposed home care services.
People with mild cognitive impairment, as well as many old people, prefer to live in their own home, when possible. However, they are not always in the condition to autonomously take care of themselves, since, for example, they can eat in a not proper way, they can make mistakes while following the medical prescriptions, or even not to perform activities that could improve their physical state. The possibility to monitor the proper handling of their personal needs, referred to both the so-called Instrumental Activities of Daily Living (IADL) and the Activities of Daily Living (ADL), and to detect deviations from previous routine patterns, is a primary challenge in supporting this type of people and their caregivers. Even though assistive technology is mature enough to theoretically provide an accurate monitoring of a patient, its use in a concrete scenario is still far from being widely adopted because of the associated costs, and the low acceptance level from people not used to it.
Usually, the human activities recognition and monitor
problem was tackled using environmental sensors (e.g.,
cameras and RFID), and “wearable” sensors (e.g., mobile
phones, wrist watches). In the first case, the devices are
positioned at certain points of interest distributed
throughout the space, so requiring structural environmental setup,
while the use of wearable sensors represents a more viable
approach from an economic point of view that does not
require a structural intervention in the environment. Both
1are with the Department of Electrical Engineering and Information
Technologies, Universita` degli Studi di Napoli Federico II, Napoli, Italy
fs2ialrevia.rossi,giovanni.ercolanog@unina.it
with Istituto di Calcolo e Reti ad Alte Prestazioni
- C.N.R., Napoli, Italy claudia.dinapoli@cnr.it,
marco.valentino@icar.cnr.it
approaches have disadvantages. In particular, elderly patients
are reluctant to continually wear multiple sensors on the body
[
In addition, assistive technology products do not take
into account the cognitive and personality characteristics of
their end-users, such as their specific deficit, emotional and
behavioral problems, the attitude towards technology, and
their physical and social environment, which could affect
their acceptance, use, and effectiveness. A valid assistive
system with a high degree of user acceptance must be
based on the knowledge of the potential users, as well as
on contextual information, that provides essential parts of
effective planning of the assistance process [
In this paper, we introduce the overall architecture of an assistive technology system, composed of a low-cost mobile robot, wearable sensors, and a middleware software infrastructure responsible for automatically providing a monitoring plan that is generated by considering patient’s needs and preferences. The plan can be adapted according to detected changes both in the patient’s conditions, and in the environment. The proposed system is being developed within the UPA4SAR project with the goal of providing an affordable and well-accepted monitoring system. The mobile robot is used as an active sensor for monitoring the user’s activities, together with a wearable device, in order to minimize structural interventions and therefore to increase the spread and the acceptance of such applications. The robot, given its physical presence, represents an added value for a monitoring system since it can proactively provide additional services to the user, such as to communicate with relatives and caregivers in video-conferencing, and to provide cognitive support with reminders and notifications. These services play an important role to improve the system acceptance and usefulness perceived by the user. The possibility to schedule a set of personalized and adaptive monitoring tasks relies on the possibility to profile the patient’s individual skills related to IADL, the cognitive status, and the personality. This collected information together with dynamic information gathered from ambient sensors is used to customize the monitoring tasks with respect to the current detected situation. We present an example where state of the art machine learning techniques are adopted for implementing different monitoring activities that could be dynamically selected considering the available contextual information and whose execution can be adapted to the specific user.
The proposed framework for the generation and the execution of personalized assistive plans for home patients affected by neurological disorders is composed of different modules organized according to a layered architecture depicted in Figure 1. The aim of the design is to decouple low-level functions for managing devices, for data elaboration, and for basic robotics behaviors, from high-level functions adopted for reasoning on the assistive plans.
ator (DAWG) [
The middle layer is composed of DAAs. The goal of such
actions is to effectively provide either different algorithms for
analyzing input data to monitor the user state and behaviors
(i.e., using different input data and modalities to obtain
such information), and so updating the observations and the
user profile, but also to implement different navigation and
interaction strategies to be used by the robot. This approach
is in the direction of integrating robot functionalities (DAAs)
as services that can be requested for the seamless integration
of robots, as well as other IoT devices, into a web or cloud
computing environment [
To obtain a better adaptation to the user, the project proposes to equip the user cognitive profile with a psychological personality profile, and to adapt the robotic behavior not only with respect to the choice of the single activity to be undertaken (selected by the DAW), but also with respect to the way in which the same activity is performed. Indeed, in order to be effectively deployed, also the robot should be able to regulate its social interaction parameters (e.g., the interaction distances, proxemics, the speed of movements, and the same modality of interaction) based on personality factors as well as of the cognitive state of the user. Hence, the user profiling plays a fundamental role both to generate a DAW tailored for each patient, but also to modulate the execution of Daily Assistive Actions. In fact, according to the personality of a patient, some actions can be performed with a different interaction modality, such as direct interaction with the robot if the user is in a state of inactivity and calm, or remote interaction with the robot staying at a certain distance, if the user is in a state of agitation.
The upper layer is represented by the Smart Environment composed of sensors and actuators that play the twofold role of gathering information on the patient’s state, and of performing assistive actions. Low-level functionalities make direct use of sensors and actuators installed into the Smart Environment, which are respectively managed by the DAA. Figure 1 shows how low-level nodes are combined to compose high-level functionalities.
In this section, we present some modules of the proposed general frameworks that have been already developed in the first phase of the UPA4SAR project. The Robot Operating System (ROS) is employed to ensure modularity and scalability of the architecture.
Problems related to functional limitations, behavioral and cognitive limitations, reduced capabilities, etc., that are specific to each individual, could strongly influence the user acceptance as well as the effectiveness of assistive technologies. Such knowledge is typically in the hands of clinicians that are provided with a variety of cognitive and behavioral evaluation instruments that aggregate a large amount of information that, potentially, could be extremely important
...
Request
Sensor Reminder Actuator Camera
Microphones ... Sensor
Speakers Motors Actuator Laser Sensor User Location Activity Recognition 1 Remind Activity 1 ...
Activity Recognition 2 ...
Dialogue System Remind ... Activity 2 Navigation System
Response
O b s e r v a it o n s U s e r ' s P r o fi l e
in order to provide personalized ICT technologies for a large class of the population with psychiatric and cognitive disabilities. Since the assessment of cognitive impairments is typically carried out using of a wide range of clinical tests, starting from the analysis of such tests, the project’s goal is to develop such computational models of the user’s profiles to provide the possibility of obtaining a personalized interaction with the robot in the assistive context.
In details, in this project, we considered characteristics
that are related to the user personality since they affect the
way public spaces are shared and the perception of socially
acceptable movements [
The smart environment relies on a series of sensors and actuators. More specifically, we considered two types of robots. A Pepper robot, developed by Softbank, that is able to navigate using sonars, lasers and bumpers. It has two frontal cameras and one RGB-D camera for object recognition, face recognition and people recognition. It can interact with the user using the speech recognition and speech synthesis by 4 directional microphones and two loudspeakers but also using a tablet. It can also locate the user position and identify the user emotions using microphones. The second considered robot is Turtlebot 2, that is a low-cost mobile base configured with a tablet on the top of it and a Microsoft Kinect 2 (RGB-D camera). The base of the robot is a Kobuki, endowed with bumpers and infrared sensors to navigate and automatic docking for charging the battery pack. iBeacons are used for the indoor positioning system. Such devices are capable of transmitting a signal at low cost and energy, using Bluetooth Low Energy (BLE) technology. We used the strength of the signal, the RSSI (Receive Signal Strength Indicator), to define proximity relations in order to pinpoint the position of the door of the rooms. Therefore, iBeacons are displaced near room doors, and the signals are captured through an Android smartphone. Finally, we used a Polar M-600 smartwatch that mounts accelerometer, gyroscope, optical heart rate measurement with 6 LEDs. The optical heart rate can alert us in dangerous situations, giving in real time the heart condition and the state of health of the patients.
Specifically, each Sensor Node publishes collected information over the time on its corresponding Sensor Topic, whilst a generic Actuator Node subscribes to a specific Actuator Topic to handle its parametric execution. Time = 9:00
The Daily Assistive Workflow Generator (DAWG) is the component responsible for the generation of monitoring assistive plans. In order to perform this task, the DAWG takes into account the user’s profile described in Sec. III-A along with the daily routine, the current observations and the entire set of Abstract Assistive Actions, and Daily Assistive Actions. The daily routine is represented by a set of activities that the user has to perform throughout the day, each of them labeled with a time constraint. The daily routine is then encoded into a set of goals, one for each activity, that the system has to achieve with respect to the time constraints. The workflow generation process consists of two main steps. The first step is responsible for the generation of the abstract daily assistive workflow, representing the set of actions to be scheduled to monitor the daily routine of the patient, organized as a set of goals to be fulfilled. Figure 2 shows an example of an abstract workflow for the Wake-Up activity, representing the set of abstract actions necessary to monitor the Wake-Up activity. Each Abstract Assistive Action has to be instantiated by a concrete Daily Assistive Action in order to be executed. Therefore, an abstract workflow represents a high-level template for the sequence of actions required to achieve a certain goal.
The second step is responsible for the instantiation of a specific Abstract Assistive Workflow. This process starts as soon as a certain time constraint triggers a new goal. For instance, with respect to the workflow shown in Figure 2, the instantiation process will be activated at 9:00 am by triggering the corresponding goal Wake Up at that time. When a goal is triggered, the system retrieves the abstract workflow associated with the current activity to be monitored and turns it into a concrete workflow. The instantiation process follows the structure of the corresponding abstract workflow. Its general structure is represented as a graph G(V; E) in which each vertex v 2 V represents an abstract action to be instantiated and each edge e 2 E represents a transition labeled with a condition. Starting from the first vertex, each abstract action is instantiated by selecting the most suitable one among the available concrete Daily Assistive Actions. For instance, in Figure 2, the abstract assistive action Check Wake Up may be implemented by different concrete actions offering the same functionality with different modalities. If we consider the environmental setting depicted in Figure 1, Check Wake Up can be actually realized by the Activity Recognition module provided by the Smartwatch, as well as by the Robot via Camera. Moreover, even the Robot’s Dialogue System can be suitable for this task. The main characteristics we consider to differentiate a concrete action from each other are its reliability and the interaction modality. These non-functional parameters are then matched against the user profile to determine a ranking over possible concrete implementations to select the one that represents the best trade-off between user needs and reliability. Once an abstract action is instantiated, the selected concrete implementation can be executed by the corresponding device, e.g., the robot. In addition, the concrete action can be executed in different modes (e.g., interaction modes), i.e. with different values of some nonfunctional parameters. Here, the execution of a certain action produces as output new observations deriving from sensors installed into the environment. These observations are used to determine whether a certain state is reached. Hence, the system is able to determine the transition to the next vertex in the graph after the execution of each concrete assistive action. When a final condition is reached, the workflow execution is completed and the system waits until a new goal is triggered.
The set of Daily Assistive Actions considered in the
project includes, for example, the IADL recognition [
1) Dialogue System: For the robot to approach the user and to perform a user-tailored interaction, it has to find him/her, and subsequently position itself properly with respect to the activity requirement. When the robot is instructed to approach or start monitoring the subject, the wander module - implemented with a ROS node - executes a walking routine in order to find the person in the room: it will move forward for a few seconds, then it will steer the robot first to its right and then to its left, for a number of seconds each turn, that has to differ to avoid getting stuck in a dead end. The obstacle avoidance relies respectively on the Softbank NAOqi-API for the Pepper robot and on bumpers placed at the base of the Turtlebot2, and if an impediment is found, the robot will turn to direct elsewhere. The robot will navigate in the room until the user is found.
Once the user has been detected, it is necessary to recognize his/her pose so that the robot can move to the correct approach/monitoring position. Pepper and the Turtlebot2 recognize the human skeleton with an RGB-D camera. The pose/skeleton recognition algorithm considers only the single depth image of each frame to predict the human skeleton joints. The predicted joints are translated on the average point between the points of the torso, left and right shoulder, left and right hand. Then they are normalized with the standard score, computing the mean and the standard deviation on the entire sequence of the instance. The normalization reduces the dependencies with the length of the limb and the height of the person.
Thanks to these skills, the robot can search a human by wandering in the rooms, considering the user position given by the iBeacons to narrow down the search area. Once the user position and his/her pose are recognized, the robot approaches the person with a direction and a speed that depends on the user profile. The robot will then ask directly the user a confirmation for the hypothesized activity by using the tablet, speech interaction or both (see Figure 3 (right)).
This action is considered to have a strong reliability since it directly asks for a confirmation to the user, but, conversely, it is considered invasive since it implies a direct interaction with the user and consequently a distraction from his/her current activity.
previous action, the robot could also try to recognize the
user activity by observing directly the user behavior. In
this case, the robot will stop at a longer distance from the
user (that again depends on the user preferences). Once the
robot approaches the user a ROS node, that implements the
Activity Recognition algorithm proposed in [
The Activity Recognition via camera action is less reliable than the previous one, but also less invasive, since it would not distract the user from his/her current activity. However, also the use of the camera is sometimes considered invasive.
possibility is to monitor the user behavior by directly considering data coming from a wearable device. The Activity Recognition via wearable service is a ROS node which is in charge of gathering data from an accelerometer to predict low-level activities. The recognized activities are standing up, getting up, walking, running, going up, jumping, going down, lying down, sitting down. Each instance contains the accelerations along each of the three Cartesian axes with a frequency of 50 Hz. The accelerometer information is given in input to a deep learning model formed by two LSTM layers.
With respect to the direct observation, Activity Recognition via wearable action is considered less invasive, since it does not use a camera or interrupt the user. However, the recognition ability is related only to human posture, so in order to recognize different high-level behaviors, such for example, watching tv, this information has to be integrated with other information, such as the location of the user. Hence, it is not as reliable as the direct observation.
Adaptation and user modeling play a key role when it
comes to design smart assistive systems to help elderly
people live a longer and more independent life at home. A
recent survey on self-adaptation for cyber-physical systems
[
Several systems have been designed for home-care
assistance providing a different degree of personalization, but
no adaptation on the fly that has been a major research
issues in ambient intelligence scenarios [
An effective assistive robotic system for home care assistance should be affordable and well accepted by end users. In order to meet these requirements, our aim is to further explore user’s adaptation by considering a dynamic environmental setting where each functionality can be realized in several ways and concrete implementations are selected according to the user’s profile.
In this paper, we present a framework, under development
within the project UPA4SAR, that allows generating
personalized and adaptive assistive plans for home patients affected
by neurological disorders, that are executed by the smart
environment devices that are part of the framework. Hence,
the framework integrates business processes with real-world
objects, humans and digital services [
The proposed framework is designed according to a
layered architecture that allows decoupling the actual execution
of an assistive plan, from its planning and adaptation. The
planning and adaptation depend on the cognitive and
personality profile of the patient, in addition to dynamic information
on his/her state as detected by the smart environment. This
separation allows to guarantee the system functioning
according to both the available information about the patient’s
profile and the specific assistive actions available in a specific
setting of the Smart Environment. The rationale of this choice
is to provide a modular and extensible framework where
assistive actions to be performed can be added, removed and
customized according to the specific setting of the Smart
Environment, and the high-level reasoning that allows for
the personalization and the adaptation of the daily assistive
plan that is performed according to the collected profiling
information [
This study has been supported by MIUR (Italian Ministry of Education, Universities, and Research) within the PRIN 2015 research project “UPA4SAR - User-centered Profiling and Adaptation for Socially Assistive Robotics” (grant n. 2015KB-L78T), http://www.upa4sar.unina.it.