The increasing number of People with Dementia (PwD) increases the strain and burden on caregivers in nursing homes. When PwD exhibit behavioral disturbances, they need personalized interventions of the nursing sta , however, the sta only has limited time to spend on these interventions. Within this paper, an Internet of Robotic Things platform and an accompanying robotic intervention strategy are presented that enable the nursing sta to rely on robots to assist them during these personalized interventions.
Along with the ageing population, the number of people with dementia (PWD)
is steadily increasing [
A robot solution could help to provide such person-centric care by interacting
with the PwD in order to prevent and alleviate BDs [
In the WONDER project1, we are developing an Internet of Robotic Things
(IoRT) platform to enable personalized robot interactions with PwD to reduce
and intercept BDs. In an IoRT platform the robot is integrated in a smart
environment out tted with a variability of sensors and wearables that capture the
current context. A semantic cloud back-end then analyzes this captured
information and combines it with other context information sources (e.g. pro le of
the PwD or day schedule of the institution) to extract valuable knowledge about
the context and activities of the persons active in it. This derived knowledge is
then used to steer the actions of the robot. The IoRT platform autonomously
detects whether a BD occurred and determines which actions should be
performed by the robot to respond to it in a personalized manner. Although several
researchers have looked towards semantics to support the care for people with
dementia, none of them focused on the combination of IoT and semantics in
order to detect behavioral disturbances and derive the accompanying personalized
intervention strategy [
This paper brings two contributions. First, the technical architecture of the IoRT platform is presented, together with the knowledge-base that was built up to capture the gathered sensor data, model the environment & context and capture the PwD pro les. Second, the co-design methodology, where researchers and nursing sta join forces to determine the robotic intervention strategy in case of a detected BD, is discussed, together with the resulting personalized algorithms. 2
The overall architecture to realize the robotic task assignment platform will be
based on the Internet of Robotic Things Architecture (IoRT) [
Within the platform, data from various heterogeneous sources is gathered. First, sensors are placed within the continuous care setting to monitor the context, for
Process
Management ProcessingService MCI Path Planning Service Service RoSbeorvtiTcaesk …
TaskExecutor
Static
Data Messaging
Bus
JSON-LD
Nurse - Caregiver Alerts
Status updates Robot …
Controllers
DYAMAND Smartphone
Data
Sensor input Humidity
Temperature Movement
Status updates Commands
Robot example, humidity, movement and temperature sensors can be deployed in every room of a nursing home to get an overview of the personal setting of a patient. Moreover, also sensors can be deployed in the common spaces, for example, to track patients and/or caregivers in the hallways by using a wearable. Data produced by these sensors can then also be used to analyze walking patterns or other speci c behaviors. Second, the caregivers can make use of smart devices, such as a smart phone or a smart watch, to give status update to the platform or to report irregularities. Third, input can also be expected from the robots that are deployed within the continuous care setting. These robots are equipped with a variety of sensors, which can have valuable input for the system. Moreover, the robot is also capable of sending status updates to the platform, for example,\the patient is currently looking at me and I am calming this patient down". 2.2
Controllers Data enters the platform through the Controllers. These controllers are responsible for mapping the data from the sensors, robot and caregivers, which is often described using JSON, on a uniform model. These controllers are also responsible for communicating towards the outside world, for example, sending an alert to a nurse or assigning a task to a speci c robot.
Currently, three di erent controllers are de ned in this architecture:
{ DYAMAND [
MAND maps its internal model onto the Semantic Sensor Network (SSN)
ontology [
Similar to these three Controllers, other Controllers can easily be created within this component to support newly de ned input sources. All Controllers will publish the received data onto the Messaging Bus, using JSON-LD. Messaging Bus As data is coming from a wide variety of di erent sources, the platform needs to cope with a huge amount of data. To make sure the platform is responsive enough and other components do not get overloaded, a Messaging Bus, using a publish/subscribe pattern, was chosen as a central component of the architecture. This Messaging Bus will be deployed in the cloud and implemented, using Apache Kafka 2. Data published by the Controllers onto the Bus will be distributed to all speci c components that have indicated a speci c interest in that type of data fragment. This will be done by analyzing the JSON-LD message.
Static Data Not only real-time data will be used within the platform. But also static data, such as calendar information or pro le information, has to be accessed. This data is made available through the Static Data component. Updates or new data will be pushed directly to the Messaging Bus, while static data will be directly queried by the speci c Processing Services.
Processing Services The Processing Services are at the heart of the platform.
These services contain the business logic components of the platform. Each
service is responsible for speci c functionality and will registered its lter rules to
the Messaging Bus to indicated its interest in speci c data. There are several
examples of di erent Processing Services:
{ MCI Services [
from the Messaging Bus will be reasoned upon and possible ndings of these services will be pushed to the Messaging Bus. This enables other Processing Services to use and incorporate the knowledge extracted in another service. Examples of MCI Services for this speci c use case are the Behavioral Disturbance Detection MCI Service and the Task Assignment MCI Service. { Robot Task Service: Within this service, tasks are selected that should be executed by a robot. The dynamic algorithm takes the current context and the personality of the person into account { Path Planning Service: When the task is created, a robot should be directed towards the person. This is done within this service, using an Environment Model the oor plan of the care institution.
Task Executor Services within the Processing Service component will send out tasks, actions and gained knowledge to the Messaging Bus. The Task Executor will publish a lter rule to indicate its interest in all task information. Once the Task Executor receives a task, this component will decide which Controller is responsible for the execution of the task and delegates this to the Controllers component.
Process Management The Process Management component will be noti ed of all actions and tasks taken by the Processing Services component. Based on the data that this component receives, decisions can be made to overrule a speci c decision, as not all services are aware of the decisions made by other services. For example, when the platform assigns a task to a robot to assist a patient, the Process Management component can decide to cancel this task because of a higher priority task. 2.3
Communication to the outside world will always go through the Controllers. These Controllers are responsible for sending out commands and alerts. If a task needs to be sent to the robot, the Robot Controller will be used, while the DYAMAND Controller can be used for sending actions to actuators or smart devices. 3
To ideally tune the personalized intervention strategy and interaction with the robot to the PwD, it is important to derive the relevant personal information about the PwD and link it to the current context. However, the context and pro le information is provided by a plethora of sensors, devices, databases and software components. A semantic model can ideally be used to consolidate all this information and abstract it to high-level concepts that can be used by the intervention strategy to base its decisions on. Moreover, semantic reasoning can be applied by the various services to derive actionable robot insights out of the consolidation of context and pro le data.
Data provided by the various sensors & devices in the environment needs to
be captured and consolidated to available background knowledge about these
devices, their capabilities and set-up within the environment. To realize this, an
extension of SSN 3) [
First, all the sensors and devices were modeled which are deployed within the nursing homes in order to track the behavior of the PwDs, e.g., motion sensors, sound sensors, light sensors, wearables, etc.
Next, the SSN was extended with an observation pattern [
Fig. 3. Impressions of the co-design workshops
certain amount of time, a LongLoudNoiseSymptom is detected. Queries or axioms
can then be de ned that detect undesirable combinations of Symptoms and
classify them as Faults, e.g., when the presence of a person, who is known to exhibit
yelling as a behavioral disturbance, is detected within a room with a loud noise
and no other rooms are exhibiting LoudNoiseSymptoms, a yellingPersonFault
is detected. These detected Faults can then be coupled to Solutions that
resolve them, e.g., soothPersonSolution. Finally, this Solution can be mapped
on one or more Actions that need to be performed to reach this solution, e.g.,
RobotPlaysSoothingMusicAction or sendNurseWarningAction. As such, the
various incidents and behavioral disturbances that are detected within the
nursing home, can easily be linked to (robot) actions that need to be performed.
To model the available robots, their capabilities and actions, is based on the
ontologies provided by KnowRob4 [
To derive which action should be performed in which situation, co-design sessions with the caregivers of the participating nursing homes were organized. These workshops focused on eliciting from the sta which information they take into account intuitively to decide how one should intervene or interact with the PWD. Some impressions of the workshops are visualized in Figure 3.
At the start of the workshop, the participants described a complex situation involving an intervention strategy for a PwD exhibiting a behavioral disturbance, e.g., wandering or yelling. Next, participants were asked to suppose they were an intelligent system that had a complete overview of the current situation. This system takes detected BDs by PwDs as input and is tasked with assessing the priority of the situations and deciding which actions should be taken to resolve them, i.e., ensure that the PwD stops exhibiting the BD. The real life situations described by the participants were used to start the discussions by visualizing them, e.g. the location of the PWD, the BD, the available robots, on a blue print of the work environment of the participants. To gather more context and make an informed decision, the participants asked questions, e.g., who is the PwD? Which BD occurred? Which type of music does the PwD like? What is the time of day? Instead of answering the question, discussions were rst held about the importance of the requested info and possible answers the participants envisioned. This way, the researchers could tap into the reasoning made by the participants. The reasoning process of the participants was visualized by the researchers as decision trees in which the formulated questions formed the various nodes and the possible answers indicated the possible branches. The order of the information in the tree re ects its importance, while the di erent nodes represent the parameters that should be taken into account to reach the robotic intervention strategy. More information about the derived algorithms is detailed in the next section.
Femke
Ongenae, Femke
Decision tree behavioral disturbances
No Which type of robot interaction does this PwD respond to?
Yes
What is the Priority Level of the BD? PwD has shown Aggressive Behavior towards robot at this Time & Location?
Was already introduced to Robot?
Unknown
No Highest priority
...
High priority
...
Normal priority
What is the BD? Yes
Send message to nurse containing: BD, Location, Priority, PwD, Agressive Behavior
Wandering or lingering
Determing PwD, Timestamp & Location of BD Etc. ...
Storytel ing
Music
Conversation ...
...
Degree of dementia of PwD?
Severe Length conversation = 2 short sentences Has the Night Shift started?
No Yes ...
Pick topic from PwD profile (e.g. Leisure activity, Family history, Occupation, etc.) Determine Native
Language PwD Average ...
Mild ...
Is Pwd hard hearing?
No Volume conversation =
normal Send robot to patient to perform Conversation Action with parameters topic, language, length,
volume Robot executes Conversation Action Action successful?
Care Ontology (ACCIO) . ACCIO is a modular ontology, consisting of 7 core high-level ontologies modeling information prevalent to o ering contextaware and personalized
healthcare services, namely the Upper, Sensor, Context, Pro le, Role & Competence, Medical and
Task continuous care core ontologies. It
https://github.com/IBCNServices/Accio-Ontology/
also consists of several low-level care ontologies, modeling knowledge exchanged
within nursing homes. These low-level ontologies extend the core ontologies with
concepts and relationships speci c to these care settings, e.g., speci c roles,
competences, tasks and pathologies. More information about these ontologies
can be found in Ongenae, et al. [
To illustrate how this knowledge base, containing the gathered pro le and context data, is then used to steer the interventions, an example of a co-designed algorithm is shown in Figure 4. The bold text indicates concepts from the knowledge base. 5
In this paper, we presented the rst steps towards a semantic IoRT Platform and how it could be used to optimize the care for people with dementia through personalized robot interventions in case of behavioral disturbances. In future work, we will focus on the design of algorithms for the accurate detection of behavioral disturbances based on the pro le and context information captured in the knowledge base of the IoRT platform.
The imec WONDER is a project cofunded by imec, a research institute founded by the Flemish Government. Companies and organizations involved in the project are ZoraBots, XETAL, WZC De Vijvers, and WZC Weverbos, with project support of VLAIO.