Workshop "From Objects to Agents" (WOA 2019) Assessing Usability of a Robotic-Based AAL System: A Pilot Study with Dementia Patients Claudia Di Napoli1 , Emanuela Del Grosso1 , Giovanni Ercolano2 , Federica Garramone3 , Elena Salvatore4 , Gabriella Santangelo3 , and Silvia Rossi2 1 Istituto di Calcolo e Reti ad Alte Prestazioni, C.N.R., Naples, Italy Email: claudia.dinapoli@cnr.it, emanuela.delgrosso@icar.cnr.it 2 DIETI, Universit degli Studi di Napoli Federico II, Naples, Italy Email: {silvia.rossi, giovanni.ercolano}@unina.it 3 Department of Psychology, Universit degli Studi della Campania Luigi Vanvitelli, Caserta, Italy Email: {gabriella.santangelo, federica.garramone}@unicampania.it 4 Department of Neuroscience, Universit degli Studi di Napoli Federico II, Naples, Italy Email: elena.salvatore@unina.it Abstract—Ambient Assisted Living is playing a crucial role in In this work, we report the approach adopted in the supporting dementia patients to live in their preferred environ- User-Centred Profiling and Adaptation for Socially Assistive ment, so limiting the involvement of careers and/or relatives. In Robotics (UPA4SAR) project, whose objective is to develop order for such systems to become a reality, patients need to feel comfortable when interacting with them, and so an agent-based an affordable, easy to deploy, and well accepted AAL system modular approach is adopted to make it possible to personalize based on a social robot to deliver assistive services for home the provision of digital services for each specific patient’s needs. patients affected by Alzheimer’s disease. The aim of the Here, we experiment with a robotic-based ambient assisted project is to improve the level of acceptability of social environment to analyze the perceived usability when real patients robotics through the possibility of adapting robots’ behavior interact with it in a controlled research laboratory where the system is deployed, by taking into account both their personality to the patient. The innovative character of the project concerns traits and cognitive status. The perceived usability is evaluated the realization of new models of assistance and provision of through a survey with a set of patients filling a questionnaire services in the health sector, which aim at a “patient-centric” specifically designed for the experimentation that is based on the vision. To this end, the project proposes the use of a robotic Unified Theory of Acceptance and Use Technology (UTAUT). The system that allows the automatic adaptation of the robot’s preliminary obtained results show that the perceived usability of the system is related to some traits of patients’ personality, while behavior to the personality profile, preferences, and cognitive their cognitive status impacts the provided assistive services. status of the user, in order to provide an adaptive interaction. Index Terms—Assistive robotics, workflow of services, person- The main task of the robot is the monitoring of the patient’s alization, QoS adaptation quality of life (through the recognition of the activities he/she performs) and cognitive support. In this work, we present the I. I NTRODUCTION results of a first experimentation performed with the complete Ambient Assisted Living systems equipped with social prototype of the system. robots are reaching a great potential due to the advances in the Information and Communication Technologies, so making II. A ROBOTIC - BASED AAL S YSTEM it possible their adoption for supporting home care of patients In order to support home-care assistance for patients af- with mild neurological disorders, or at initial stages of the fected by neurological disorders, a robotic-based ambient Alzheimer’s disease. Such systems may help to provide cog- assisted environment has been developed within the UPA4SAR nitive and physical stimulation to patients, crucial to limiting project whose goal is to provide an affordable and well- their cognitive reserve, to remind tasks that have to be carried accepted robotic assistive system, able to generate and execute out during the day, and to monitor their activities, so allevi- assistive plans and actions that are personalized for each ating the already heavy burden on careers and/or relatives. specific patient, and that can be adapted during execution to Nevertheless, the interaction of a very vulnerable category changing conditions in both the home environment, and the of users, such as Alzheimer’s patients, with these systems patient’s conditions [2]. may strongly impact their effective use [1]. In fact, interactive At this purpose, a service-oriented approach (SOA) is devices, as robotic systems, whose behavior is not compliant adopted to decouple a required functionality from its concrete with the needs and the characteristics of each patient, may implementation that is characterized both by the device that cause discomfort so preventing her/him from using it. provides it, and by Quality of Service (QoS) parameters 59 Workshop "From Objects to Agents" (WOA 2019) referring to the way it is delivered [3]. This service-based III. AGENT- BASED A BSTRACT S ERVICES FOR H OME approach allows both the execution of services on computing A SSISTANCE resources outside the robot, and the integration of services Starting from the patient’s daily routine (as reported in that can be provided by the robot with services provided by Table I, the Assistive Workflow Management subsystem is other devices in the house, or even by services provided by a responsible for the generation of high level plans listing the caregiver, thanks to the possibility of representing an assistive necessary daily assistive tasks required for assisting patients. action as a service according to an interface and standard It is an agent-based middleware that, once assistive tasks communication protocols. are declared in an XML format, it generates plans to per- The system architecture is composed of different layers: form them, composed of the required functionalities, and their execution order requirements. According to the adopted service-oriented approach, such assistive task is represented • the Data Computational Model, i.e. the knowledge base as a workflow of abstract services, i.e. abstract functionality containing static and dynamic information both on the necessary to perform the task. An example of daily routine patient, and on the home environment; dynamic informa- and the corresponding high level plan, known as an Abstract tion includes the current user activity, his/her physical Assistive Plan, represented in XML, are here reported. and emotional state, his/her current location, that are collected through the available sensors (including the TABLE I robot) and updated from time to time; static information DAILY ROUTINE contains: the patient Daily Routine, i.e. the set of daily Time Range Daily Activity activities that he/she has to carry out throughout the 07:00 - 07:30 Wake-up day at given times, the Personality Profile, reporting 07:30 - 08:00 Breakfast measures of five personality traits, i.e. Neuroticism, Ex- 09:30 - 11:00 Cognitive Entertainment traversion, Openness, Agreeableness, and Conscientious- 11:30 Take medicine 12:30 - 13:30 Lunch ness, assessed through the NEO Personality Inventory 14:00 - 16:00 Resting test [4]; the Cognitive Profile, characterizing cognitive 16:00 - 18:00 Go out and functional performance of Alzheimer’s patients and 18:30 - 19:30 Physical Entertainment assessed through the ACE-R (Addenbrooke’s Cognitive 20:00 - 21:00 Dinner Examination) test [5]; • the Assistive Workflow Management, i.e., the middleware responsible for the execution of personalized assistive plans represented as a workflow of services [6]; WakeUp • the Daily Assistive Actions, i.e. the set of concrete service WakeUpMonitoring 2019−04−24 implementations provided by the technological devices, 7 : 0 0 : that is periodically updated to take into account their 7 : 3 0 : dynamic availability; • the Smart Environment, i.e. the set of technological B r e a k f a s t devices deployed in the home environment that are: a B r e a k f a s t M o n i t o r i n g Sanbot Elf, that is a low-cost mobile humanoid robot 2019−04−24 7 : 3 0 : endowed with a tablet and an RGB-D camera (see Figure 8 : 0 0 : 3), iBeacons used for the indoor positioning system, able to transmit a signal using Bluetooth Low Energy (BLE) C o g n E n t e r t a i n m e n t technology, a Polar M-600 smartwatch, and a Samsung S u g g e s t C o g n S t i m u l Smartphone. 2019−04−24 09 : 3 0 11 : 0 0 All the considered devices are android-based, hence an- droid applications are developed to communicate with their T a k e M e d i c i n e sensors and actuators. The workflow management subsys- MedicineRemind tem is running on a PC where also user data are stored. 2019−04−24 The communication among all components of the system, 11 : 3 0 11 : 3 0 the services and the IoT devices, is based on Web Socket protocol using Socket.IO, a JavaScript library for real-time ..... web applications that enables real-time, bi-directional and ..... event-based communication between web clients and servers, and JSon messages. A server, running on the PC, manages Each assistive task is referred to as an AbsWorkflow, the communication between the workflow manager and the a graph of abstract services, i.e. high level functionalities concrete services. each one managed by an AbstractServiceAgent responsible 60 Workshop "From Objects to Agents" (WOA 2019) Fig. 1. The abstract workflow for suggesting a stimulation activity Fig. 2. The concrete workflow for suggesting a stimulation activity for the lookup of the concrete services available to provide A. Selection of Concrete Services the required functionality, and to contact the corresponding ConcreteServicesAgents, within the android applications, providers represented as a ConcreteServiceAgent. are android services (i.e., threads working in background that In Figure 1 the SuggestCognitiveStimulation ab- can be invoked via socket messages) and activities (for GUI). stract workflow is reported, with the abstract services required A web interface used to activate concrete service individually to perform it. For each abstract service, the corresponding Ab- is shown in Figure 4. stractServiceAgent issues a request to the available Concrete- Currently, the types of available services are: 1) Monitoring ServiceAgents, and it selects the first ConcreteServiceAgent Services that include activity recognition via a wearable de- that replies or more complex mechanisms for ordering the vice or via camera using pose/skeleton recognition, emotion replies can be implemented [7]. Once selected, the Con- and disengagement recognition, 2) Navigation Services for creteServiceAgent is responsible for instantiating the QoS searching and approaching the user, 3) Interaction Services parameters of the concrete service it provides, according to for speech recognition and speech synthesis using multimodal the cognitive status and the personality profile of the specific interaction with the user. patient. Currently the only QoS parameter considered is the 1) Monitoring Services: Monitoring services are developed repetition frequency at which each service is executed. to monitor and recognize the current state of the user and the performed Activities of Daily Living (ADLs) and instrumental This selection process is repeated for all abstract services in Activities of Daily Living (IADL). the workflow, until all of them are instantiated with a concrete In the current version of the system the state of the user is implementation resulting in a concrete workflow, as the one evaluated through HR Detection service and Emotion Recog- reported in Figure 2, ready to be executed by the selected nition service. The first aims at getting the current heart-rate providers. The type of entertainment activities are randomly of the patient from the smartwatch and the average value of selected among the ones preferred by the patient and reported 15 second of lectures is provided as a result. In the second, in the user profile. the robot takes a video of the person that is analyze by the During the execution of the concrete workflow, as shown in Affectiva SDK; this service returns the emotion (joy, surprise, Figure 3, concrete services may fail due to either a timeout contempt, disgust, sadness, anger) of the person with the for getting the service output, or for an unavailability of the highest mean on the whole video. device delivering the service. In such cases, the corresponding ADL and IADL can be monitored by using either Pose AbstractServiceAgent receives the error message and it selects Detection, Activity Classification, or Dialogue Check. The the second ConcreteServiceAgent that replied to the initial Pose Detection service aims at evaluating the current pose call, and so on. If all available devices fail, then the workflow of the user from wearable data. The robot communicates with execution ends with a failure and a notification to a human the smartwatch via Bluetooth to take 512 data samples from career is signaled. the smartwatch accelerometer and use a deep neural network 61 Workshop "From Objects to Agents" (WOA 2019) Fig. 3. Selection of different devices for the same abstract service for the classification process [8]; the recognized activities are: standing up, getting up, walking, running, going up, jumping, going down, lying down, sitting down. The Activity Classifica- tion service implements an Activity Recognition algorithm by analyzing a video sampled from the robot camera to detect Fig. 4. Server interface to control individual services the skeleton pose for each video frame [9]. The activity recognition works again with a deep neural network on a window of 140 frames of human skeleton data with 30 fps. The the Dialogue Suggest service is used to suggest the activities, activity recognized are watching tv, relaxing on couch, ironing, while the services Video Entertainment, Audio Entertainment, making coffee, working at PC, and talking on the phone. and Game Entertainment are invoked to show respectively, Finally, the Dialogue Check service allows to use the robot video, audio or games to the user. GUI to directly ask a question to the user regarding his/her IV. T HE P ILOT S TUDY activity. Two buttons (for positive and negative answers) are A prototype implementation of the robotic-based AAL sys- showed on the robot tablet (e.g. a question can ask if the user tem has been developed and tested in a controlled environment has taken the medicines). with real patients with different degrees of the Alzheimer’s Two additional services, In Room Detection and In Room disease, recruited by the team of neurologists of the project1 . Detection with Robot, are developed to infer additional infor- The purpose of this preliminary experimentation was to mation on the user state and activity. In details,with the In evaluate a set of ratings to assess the general acceptance Room Detection service the robot contacts the smartwatch via degree of the system according to the different cognitive and Bluetooth to know the position of the person in the house; the personality characteristics of the selected patients, by varying smartwatch communicates with the beacons placed in each the repetition frequency of the interaction services suggesting room of the house to calculate the distances between the various entertainment activities, in a user-transparent manner. user (that wears the smartwatch) and the beacon; this service To determine the strength of predictors for elderly participants’ returns the label of the room with the beacon at minor distance intention to accept and use the system, we adopted a modified from the smartwatch, so locating the person into the house. version of the Unified Theory of Acceptance and Use of The same behavior can be exploited by the robot itself, when Technology (UTAUT) questionnaire [11] specifically designed it is in the presence of the user. 2) Navigation Services: These services are developed in or- by the team of psychologists of the project. This represents an der to make the robot able to navigate within the environment instrument to measure the variety of perceptions of information and to locate the user [10]. Look user is a simple service that technology innovations. moves only the camera, that is located in the robot head, to The rationale of the experimentation is to collect informa- scan if there is a user nearby the robot. Images from the camera tion on possible relationships between the modalities in which are processed by the use of the PoseNet service to identify a services are executed, and the cognitive and personality traits possible user. On the contrary, the service Find User is used of patients. Such information will be used for a customization by the robot to randomly navigate within the environment, of the assistive tasks delivered by the system, so to improve avoiding obstacles, and searching for a user. Once the user has its acceptance level, by selecting behaviors that result less been detected, the Approach service can be requested to make disturbing for each patient because more compliant with the robot move to the correct interacting/monitoring position. his/her profile. The patients were left alone in an area of the controlled Finally, the Stop Robot service is used to start the search for environment resembling a house room, where they could the charging station in order for the robot to charge its battery. 3) Interaction Services: Interaction modalities of the robot perform different entertainment activities (read a book or a (voice interaction and GUI) can be used to suggest and show 1 This experimental study has been approved by the ethical committee of personalized entertainment activities to the user. In details, the University Federico II with protocol number 167/18. 62 Workshop "From Objects to Agents" (WOA 2019) TABLE II UTAUT RESULTS Code Construct Max Min Avg Std ANX Anxiety 5 3 3.9 0.8 ATT Attitude 4 2 3.2 0.8 FC Facilitating conditions 4 2 2.8 0.9 ITU Intention to use 4 3 3.3 0.5 PAD Perceived adaptability 4.3 3 3.7 0.6 PENJ Perceived enjoyment 4.6 2.8 3.4 0.8 PEOU Perceived ease of use 3.6 2.4 3.1 0.5 PS Perceived sociability 5 2.5 3.8 1.0 PU Perceived usefulness 3.7 3 3.3 0.3 Fig. 5. One of the selected patients interacting with the system (left). The SI Social influence 4 1 2.5 1.3 robot (right) SP Social presence 4.4 1.6 3.1 1.1 TR Trust 4 2 3.1 0.9 magazine, watching tv, listen to music, playing at a PC, take a refreshment). From time to time, a monitoring workflow was executed to check if the user was doing some activity. In The questionnaire was translated from English to Italian case he/she was not performing any activity, an entertainment by two psychologists and an engineer, that were proficient workflow was executed by varying the frequency when the in English and Italian and familiar with HRI. The translation entertainment was suggested. Each user interacted with the was examined at a consensus meeting, back-translated, and system for approximately three hours. approved at a second consensus meeting. A comprehension test was carried out in a subgroup of 15 individuals aged A. Patients’ Traits Classification 18 years. This consisted of a face-to-face interview during The present study included four subjects (two men and two which the interviewer inquired whether the subject had any women) between 71 and 75 years old. All subjects performed difficulty in understanding the questions and the pre-coded a cognitive screening test (ACE-R) [5], and a personality answers. A comprehension rate was obtained as the percentage questionnaire (NEO PI -3) [4]. The first subject (man; age of questions and pre-coded answers of all items correctly = 74; years of education = 8) showed a cognitive decline with understood by subjects. In the test, more than 90% of subjects significant difficulties of memory and fluency. The personality found the questions easy to understand and had no difficulty profile showed low levels of neuroticism and normal levels in interpreting the answer modes. The final Italian version of of openness, without depressive symptoms. The second sub- the questionnaire is available from the authors upon request. ject (woman; age = 75; years of education = 5) showed a Results of the UTAUT evaluation is shown in Table II. cognitive decline with significant difficulties of memory, at- We consider a positive perception of a participant when the tention, fluency and language. The personality profile showed construct score is greater than 3, while a negative perception higher levels of neuroticism and low levels of openness, with is when average score is lower than 3 (in a scale from 1 depressive symptoms. The third subject (woman; age = 71; to 5). Facilitating Conditions and Social Influence are the years of education = 8) did not show a cognitive decline, only two constructs obtaining a score lower than 3. However, but light memory difficulties. The personality profile showed these results are compatible with our experimental setting normal levels of neuroticism and high levels of openness, that was tested by the patients left alone, without involving without depressive symptoms. The last subject (man; age = caregivers, in a simulated home environment in the research 75; years of education = 18) did not show a cognitive decline, laboratory, and so not a real domestic environment. Indeed, but only slight difficulties in memory tasks. The personality Facilitating Conditions refers to factors in the environment profile showed lower levels of neuroticism and normal levels that facilitate use of the system, while Social Influence refers of openness, without depressive symptoms. to the perception that people who are important for the subject B. A UTAUT-based Usability Test think he/she should or should not use the system. We adopted the version of the UTAUT questionnaire pro- In addition, it was noticed that the patient with a low posed by [12] because it was already adapted and validated education level, depressive symptoms, and a high neuroticism in the similar context of assistive robotics applied to elderly was less incline to be engaged in the interaction with the robot, users. This UTAUT questionnaire consists of 41 items and and so to accept its suggestions. On the contrary, patients explores 12 constructs: Anxiety (ANX), Attitude (ATT), Fa- with similar cognitive decline, but with a higher education cilitating conditions (FC), Intention to use (ITU), Perceived level and lower neuroticism showed a better acceptance and adaptability (PAD), Perceived enjoyment (PENJ), Perceived a stronger engagement with the robotic system. These results ease of use (PEOU), Perceived sociability (PS), Perceived are encouraging in showing that personality traits, more than usefulness (PU), Social influence (SI), Social Presence (SP) cognitive decline, play a crucial role in the acceptance of the and Trust. Subjects are required to reply to each item on a robotic system and so it has to be taken into consideration for Likert type scale (range: 1-5). personalization of patient-system interaction. 63 Workshop "From Objects to Agents" (WOA 2019) V. C ONCLUSIONS R EFERENCES In this work, we presented a general overview of the proposed approach and the software services developed in [1] E. Broadbent, R. Stafford, and B. MacDonald, “Acceptance of healthcare the context of the UPA4SAR project. Moreover, we presented robots for the older population: Review and future directions,” Interna- the results of a pilot study that we conducted in a research tional Journal of Social Robotics, vol. 1, no. 4, p. 319, Oct 2009. [2] S. Rossi, G. Ercolano, L. Raggioli, M. Valentino, and C. Di Napoli, laboratory environment with 4 real end-users recruited for the “A framework for personalized and adaptive socially assistive robotics,” project. in Proceedings of the 19th Workshop ”From Objects to Agents” - Vol. The experimentation highlighted a general good acceptance 2215. CEUR, 2018, pp. 90–95. rate of the system, even if, in the case of the patient with an [3] C. Di Napoli, “Software agents to enable service composition through negotiation,” Studies in Computational Intelligence, vol. 170, pp. 275– high neuroticism, the interaction with the robot was limited. 296, 2009. Moreover, from the users’ free comments, we observed a [4] R. R. McCrae, J. Paul T. Costa, and T. A. Martin, “The neo-pi-3: A immediate attachment to the robot. These results are encour- more readable revised neo personality inventory,” Journal of Personality Assessment, vol. 84, no. 3, pp. 261–270, 2005. aging for the next experimental stage that will be conducted [5] E. Mioshi, K. Dawson, J. Mitchell, R. Arnold, and J. R. Hodges, “The directly in the patients’ homes. In this stage, 40 patients will Addenbrooke’s cognitive examination revised (ace-r): a brief cognitive be recruited (20 with a mild level of Alzheimer, 10 with a test battery for dementia screening,” International Journal of Geriatric Psychiatry, vol. 21, no. 11, pp. 1078–1085, 2006. moderate level and 10 with a subjective memory disorder), and [6] C. Di Napoli, M. Valentino, L. Sabatucci, and M. Cossentino, “Adaptive the robot will stay in their home for 14 days each. This longer workflows of home-care services,” in Proceedings of the 27th IEEE period, will allow us to collect more reliable data regarding International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (in press). IEEE, 2018. the acceptance rate of the robotic-based AAL system to better [7] C. Di Napoli, P. Pisa, and S. Rossi, “Towards a dynamic negotiation tune the adaptation of the robot’s behavior with respect to mechanism for qos-aware service markets,” in Trends in Practical Ap- each single patient. Indeed, the robot is also able to adapt its plications of Agents and Multiagent Systems, ser. Advances in Intelligent Systems and Computing. Springer International Publishing, 2013, vol. own social interaction parameters, such as the proxemics, the 221, pp. 9–16. speed of movements, and the interaction modalities, so future [8] S. Rossi, L. Bove, S. Di Martino, and G. Ercolano, “A two-step work are planned to consider also these QoS parameters to be framework for novelty detection in activities of daily living,” in Social tuned to validate the reliability of the proposed design choices. Robotics. Cham: Springer International Publishing, 2018, pp. 329–339. [9] G. Ercolano, D. Riccio, and S. Rossi, “Two deep approaches for adl It is expected that the adaptation of these parameters to the recognition: A multi-scale lstm and a cnn-lstm with a 3d matrix skeleton specific user’s profile will allow to overcome the novelty effect representation,” in 26th IEEE International Symposium on Robot and caused by the robot presence and to evaluate its long-term Human Interactive Communication (RO-MAN), Aug 2017, pp. 877–882. [10] G. Ercolano, L. Raggioli, E. Leone, M. Ruocco, E. Savino, and S. Rossi, effectiveness. “Seeking and approaching users in domestic environments: Testing a reactive approach on two commercial robots,” in 27th IEEE International ACKNOWLEDGMENT Symposium on Robot and Human Interactive Communication (RO- This work has been supported by MIUR (Italian Ministry of MAN), Aug 2018, pp. 808–813. [11] V. Venkatesh, M. G. Morris, G. B. Davis, and F. D. Davis, “User Education, Universities, and Research) within the PRIN 2015 acceptance of information technology: Toward a unified view.” MIS research project “UPA4SAR - User-centered Profiling and Quarterly, vol. 27, no. 3, pp. 425–478, 2003. Adaptation for Socially Assistive Robotics” (grant n. 2015KB- [12] M. Heerink, B. Kröse, V. Evers, and B. Wielinga, “Assessing acceptance L78T). The authors thank OMITECH s.r.l. for providing the of assistive social agent technology by older adults: the Almere model,” International Journal of Social Robotics, vol. 2, no. 4, pp. 361–375, Dec Sanbot Elf robots for the project experimentation phase. 2010. 64