Personalized robot interactions to intercept behavioral disturbances of people with dementia

Abstract

People with Dementia (PwD) exhibit Behavioral Disturbances (BD) that can be alleviated by personalized interactions, revisiting memories and promoting comfort and quality of life. However, caregivers are unable to spend a lot of time on these interactions. This work-in-progress poster details the design and deployment of a semantic Internet of Robotic Things (IoRT) platform that enables personalized interactions of a robot with a PwD to reduce and intercept BDs.

Introduction

2 IoRT Platform Overview

The overall architecture of the designed IoRT platform is visualized in Fig. 1.

Sensors are integrated into the care setting that monitor the context. The PwD are equipped with wearables to track their location and walking patterns. To hide the different protocols and interfaces of the devices, the DYAMAND middleware is used [ 4 ]. It allows application developers to discover, request info and control devices in a transparent manner. DYAMAND maps the output of the sensors on a uniform model and wraps into a JSON syntax. We have extended DYAMAND with a plugin that maps its internal model on the Semantic Sensor Network (SSN) Ontology2, which allows to link the measured data to each other and known background info, e.g., the sensor location and type of data it monitors. The plugin converts the data from the sensors into JSON-LD, which is then pushed to a Kafka Messaging Bus running in the cloud. There it can be dynamically picked up by various services interested in this type of sensor data.

To design services that detect BDs and steer the robot, MASSIF [ 5 ] is used. It enables the seamless collaboration of modular reasoning services to facilitate complex decision making. As common knowledge model, the Ambient-Aware Continuous Care Ontologies (ACCIO)3 are used. The Context Adapter picks up the JSON-LD messages from Kafka and converts them to OWL individuals that adhere to the ACCIO ontology. As this ontology extends the SSN ontology, this is a straight-forward conversion. These individuals are pushed on the Semantic Communication Bus (SCB), which enables a publish-subscribe mechanism based on high-level semantic concepts. The services express the data they interested in by defining semantic filter rules, i.e., OWL Class expressions. Reasoning is used by the SCB to match incoming data on the filter rules and

2 https://www.w3.org/TR/vocab-ssn/ 3 http://ibcnservices.github.io/Accio

decide to which services it should be forwarded. The services push their conclusions on the SCB. As such, services can be chained together in a flexible and data-driven manner.

Per type of BD, a BD Detection Service is made, which registers a filter rule with the SCB indicating the types of data it needs, e.g., noise or location data. Two services have already been defined, i.e., Wandering and Yelling Detection Services. They incorporate an extension of the ACCIO ontology that models domain-specific knowledge about dementia, e.g., the profiles of the PwD (e.g, historical information about exhibited BDs, severity of dementia or relationships with other PwD) and the layout and activity schedule of the care center. This info is extracted from databases and mapped on the ontology using Ontop4. These services incorporate probabilistic pattern recognition algorithms to infer with a particular probability whether a BD was detected [ 3, 7 ], i.e., a PwD is yelling or wandering. To further assess the certainty of this detection and derive the risk level of the detected BD, the probabilistic output is combined with semantic reasoning algorithms on the background knowledge. The risk level indicates whether the behavior is disturbing other people or is harming the PwD him/herself, e.g., he/she has been wandering for too long and is becoming tired. The detected BD, the confidence and the risk level are pushed back on the SCB. The use of MASSIF allows that new services can easily be incorporated to detect other BDs or use alternative detection algorithms.

The Task Assignment Service subscribes itself to detected BDs and current context to determine the appropriate IoRT actuation strategy. Possible actions are alerting a caregiver, sending a robot or a combination of both. For example, when a BD is detected with low probability and risk level, the robot is sent to the scene to gather additional info through conversation and its camera. However, if yelling is detected in area where people are present that have a history of violent behavior, the staff is immediately alerted. When the robot is sent, the Robot Task Selection Service decides which task this robot should perform. This decision algorithm adapts itself to the current context and the personality of the PwD. For example, the robot should not make a lot of noise at night or a PwD might respond better to a song than to conversation.

The Task Executor is responsible for executing the assigned tasks. The Staff Communication sends the alerts to the caregivers. The Robot Controller splits up the robot tasks into individual robotic actions and the Robot Manager decides to which robot these commands should be sent. The latter keeps track of the charging schemes, activities and locations of the robots. The Path Planning Service finds an optimal path to the PwD exhibiting the BD. The Robot Manager and the Path Planning Service use an Environment Model that encodes the floor plan of the care institution. The status, e.g., executing or finished, of the robot actions are communicated to MASSIF to detect whether the BD has been alleviated or whether additional measures should be taken. 3

IoRT Platform Deployment

The platform is deployed in two care institutions in Flanders. The rooms of 26 elderly are equipped with a GrovePi5 to measure environmental parameters, i.e., luminance, 4 http://ontop.inf.unibz.it/ 5 http://www.dexterindustries.com/grovepi/ temperature, sound, loudness, movement and humidity. The Xetal Kinsei6 is used to track the number of people in the room and perform localization. Doors are equipped with enocean7 sensors to detect whether they are opened or closed. The common rooms are also equipped with GrovePi systems. Finally, the PwD wear a Xiami Mi Band8 to track their movement. Their walking aids, e.g., wheelchair, are outfitted with a BLE tag. These wearables and tags are picked up by seven Raspberry Pi 3 gateways, enabled with BLE bluetooth, spread out across the institution. Each of the GrovePi and Raspberry Pi gateways runs a DYAMAND instance to uniformly expose the data. The ZORA9, which combines the NAO with optimized software for healthcare settings, interacts with the PwD.

The caregivers use a smartphone application to indicate when a PwD is exhibiting a BD. This labeled dataset will allow us to train and optimize the algorithms of the BD Detection Services. Decision tree workshops [ 8 ] have been organized together with the staff members to derive which parameters, profile and context info are important to take into account to assess the reliability of a BD detection, to decide who should be alerted and which action should be performed by ZORA. These workshops shaped the algorithms in the Task Assignment and Robot Task Selection Services. User tests are being performed to see how the PwD react to certain robot actions in particular situations to further optimize these services.

The next research step consists of using the collected dataset to optimize the BD Detection Services. Afterwards, the IoRT platform will be used in both institutions to steer the robots and we will study the effect this has on the behavior of the PwD. Acknowledgment: This research was realized in the WONDER project, co-funded by VLAIO, iMinds, ZoraBots, Xetal, WZC De Vijvers and Weverbos.

6 http://xetal.net/

7 https://www.enocean.com/en/

8 http://www.mi.com/sg/miband/

9 http://zorarobotics.be/

References

1. A. Sabelli , et al.: A conversational robot in an elderly care center: an ethnographic study . In: HRI Conference . pp. 37 - 44 ( 2011 )

2. Dautenhahn , K. : Robots we like to live with? a developmental perspective on a personalized, life-long robot companion . In: ROMAN Workshop . pp. 17 - 22 ( 2004 )

3. F. Palumbo , et al.: Multisensor data fusion for activity recognition based on reservoir computing . EvAAL pp. 24 - 35 ( 2013 )

4. J. Nelis , et al.: Dyamand: dynamic, adaptive management of networks and devices . In: IEEE LCN Conference . pp. 192 - 195 ( 2012 )

5. P. Bonte , et al.: The massif platform: a modular & semantic platform for the development of flexible iot services . KAIS ( 2016 )

6. P. Simoens , et al.: Internet of robotic things: Context-aware and personalized interventions of assistive social robots . In: IEEE CloudNet conference ( 2016 )

7. Q. Lin , et al.: Managing elders' wandering behavior using sensors-based solutions: a survey . IJGE 8 ( 2 ), 49 - 55 ( 2014 )

8. S. Elprama , et al.: Opportunities to use robots to alleviate behavioral disturbances of nursing home residents . In: ISG Conference ( 2016 )