The DemaWare Service-Oriented AAL Platform for People with Dementia?

Abstract

This work presents DemaWare, an Ambient Intelligence platform that targets Ambient Assisted Living for people with Dementia. DemaWare seamlessly integrates diverse hardware (wearable and ambient sensors), as well as software components (semantic interpretation, reasoning), involved in such context. It also enables both online and offline processes, including sensor analysis and storage of context semantics in a Knowledge Base. Consequently, it orchestrates semantic interpretation which incorporated defeasible logics for uncertainty handling. Overall, the underlying functionality aids clinicians and carers to timely assess and diagnose patients in the context of lab trials, homes or nursing homes.

Introduction

DemaWare Architecture

Overall, system requirements include an abstraction from data and functions, modularity for multiple components, orchestration of data transfer and reasoning, support for various roles and sites. The proposed architecture follows a layered approach (Figure 1) to address them. The hardware layer entails multi-modal sensors for collecting context information, each of which stores or handles data of specific format (video, audio, text or binary). Due to hardware constraints, some data have to be manually transferred offline by the clinicians. The analysis layer primarily addresses format heterogeneity, extracting higher-level information called observations to be stored in the Knowledge Base (KB). In detail: – The SleepClock (SC)2 logs residential patients’ sleep state patterns (deep, shallow or no sleep) and summary (total deep/shallow sleep/awake time). Data is manually retrieved and parsed by the system’s SC Library. – The DTI2 Wristwatch (WW)3 monitors physical activity (accelerometer), skin conductivity and temperature, ambient temperature and light. Binary WW data are collected offline and parsed by the system’s WW Library. – An ambient, Depth Camera is used for Complex Activity Recognition (CAR) [ 1 ] events related to patient location within zones of interest (e.g. “Kitchen”, “Out of bed” etc.) or posture (e.g. “Standing”, “Walking”, “Sitting”). Multiple CAR nodes reside on linux-based mini-PCs with attached Depth Cameras in each residence. – A second ambient, IP camera is used for Human Activity Recognition (HAR) [ 2 ] such as “Eat” or “Drink”. – Wearable Camera videos, processed by the Wearable Camera Processing Unit (WCPU) detect rooms and objects that the patient’s encounter [ 3 ]. – Wearable wireless microphones capture audio for Offline Speech Analysis (OSA) [ 4 ], measuring various indicators for the progress of patient dementia. – The KB Manager stores Observations to the KB in RDF triples (triple store). – The Semantic Interpretation (SI) [ 5 ] performs analysis on KB-stored data and enriches it with new observations. SI also combines different sensor data (Fusion), detects various complex events in time (Complex Event Processing - CEP) and handles uncertainty.

The Service layer lifts platform heterogeneity based on the WSDL W3C standard4 for remote access. It also allows using the XML/XSD-based Dema@Care Exchange Model to type-define observations, facilitating their mapping to KB constructs. Implementation-wise, the services wrap analysis components using Java JAX-WS5. Services (WSDL/SOAP) are pull-based and invoked as soon as Observations are available since most components perform offline processing (due to manual data transfer or time-demanding analysis). Meanwhile, CAR (non-WSDL service) processes streaming video and pushes observations. 2 Gear4 Renew SleepClock: http://www.stage.gear4.com/ 3 Phillips Healthcare: http://www.healthcare.philips.com/ 4 http://www.w3.org/TR/wsdl 5 JAX-WS: https://jax-ws.java.net/

The application layer consists of various Graphical User Interfaces (GUIs) as well as application logic (Controller). The Controller backend, resolves requirements related to information flow, as it orchestrates the retrieval of observations from components and stores them into the KB. It also performs certain hardware operations (e.g. start and end recordings) and gathers metadata for component invocation. GUIs are used to invoke analysis (Technician role) and view assessment results (Clinician, Carer roles). 3

Semantic Interpretation Layer

While individual sensing modalities monitor different perspectives, the Semantic Interpretation (SI) layer provides inferencing capabilities for the derivation of complex activities over the combination of those modalities. To do so, it encapsulates the ontology vocabularies for modeling the Dem@Care application context, such as activities, measurements, summaries, patients, locations and objects. The ontologies reuse the conceptual model provided by the SSN ontology [ 6 ] to model observations, measurements and sensors, as well as, relevant dementia-specific vocabularies. In detail they model: – atomic activities and measurements detected by means of monitoring and analysis components (e.g. body temperature, luminance level, having meal, sleeping, etc.). – problems and situations that the clinicians need to be informed about (e.g. missed meals, excessive napping, insufficient communication attempts, nocturia, etc.) . – clinically relevant attributes and summaries (e.g. sleep efficiency and duration, number of daily telephone interactions, etc.).

SI’s reasoning framework supports a hybrid combination of the OWL 2 reasoning paradigm and the execution of SPARQL rules in terms of a CONSTRUCT and a WHERE clause: the former defines the graph patterns, i.e. the set of triple patterns that should be added to the underlying RDF graph upon the successful pattern matching of the graphs in the WHERE clause. For example, the recognition of the PrepareTea activity is performed by fusing tea-related objects (detected from wearable camera) and the PrepareDrink intermediate activity (detected from static camera).

Since the proposed framework is required to handle data that is vastly heterogeneous, inherently uncertain and noisy, this work proposes Defeasible Logics [ 7 ] as an extremely suitable tool for handling this type of data. This can be illustrated by the following example that involves two complex activities makeT ea and eatLunch, which are respectively defined via the following sets of primitive observations as follows: makeT ea = {kitchenZone, cup, kettle, teabag}, and eatLunch = {kitchenZone, cup, f ork, dish}.

In multi-sensor environments with multi-modal and often incomplete information, where the absence of primitive observations is frequent, Defeasible Logics can offer a flexible and human-intuitive formalism for efficiently handling such situations. For instance, the following defeasible theory (written in Defeasible Logics) can handle some cases involving the above two activities: r1 : kitchenZone ∧ cup ⇒ makeT ea r2 : kitchenZone ∧ cup ∧ f ork ⇒ eatLunch r3 : kitchenZone ∧ cup ∧ kettle ⇒ makeT ea r2 > r1, r3 > r2 and C = {makeT ea, eatLunch}

Defeasible rule r1 reads as “ if the user is in the kitchen and uses the cup then he is probably making tea” and similar interpretations accompany defeasible rules r2 and r3). Moreover, rules r2 and r3 are superior to rules r1 and r2, respectively, meaning that they will prevail in potential conflicts - a conflict between two rules is initiated by complementary rule heads or heads with conflicting literals (i.e. pairs of mutually exclusive literals that cannot both be derived at the same time, see e.g. set C in the sample rule base above). 4

STATE OF THE ART

The work in [ 8 ], introduces openAAL, a general-purpose open source middleware for AAL, which provides context management, service matching, composition and workflow execution. However, while some components are similar to Dem@Care (e.g. KB Manager), openAAL does not yet handle hardware. FamiWare [ 9 ] implements a Publish/Subscribe approach, discovery, fusion etc., but targets limited hardware, e.g. Android smartphones and TinyOS sensors. Previous work in aWESoME-S [ 10 ] [ 11 ], AIM [ 12 ] and Hydra [ 13 ] have focused on energy and environmental sensors, excluding higher-level analysis. Work in [ 14 ] also provides context-sensing and user profiling. In contrast to those works, DemaWare unifies ambient and wearable devices, but also offers higher-level analysis e.g. speech, image recognition and interpretation.

CONCLUSIONS AND FUTURE WORK

The proposed system integrates both low and high level processes in the context of AAL for people with dementia i.e. sensor data retrieval, analysis and semantic interpretation under uncertainty. The framework is applicable to various pilot scenarios for patient monitoring and assessment. Current limitations include hardware constraints, e.g. manual data transfer, and the lack of even richer context information, for which we plan to investigate alternative sensors.

References

1. R. Romdhane , C.F. Crispim , F. Bremond , and M. Thonnat , ' Activity recognition and uncertain knowledge in video scenes' , in Advanced Video and Signal Based Surveillance (AVSS) , 2013 10th IEEE International Conference on, pp. 377 - 382 , ( Aug 2013 ).

2. K. Avgerinakis , A. Briassouli , and I. Kompatsiaris , ' Recognition of activities of daily living for smart home environments' , in 9th International Conference on, pp. 173 - 180 , ( July 2013 ).

3. H. Boujut , J. Benois-Pineau , and R. Megret , ' Fusion of multiple visual cues for visual saliency extraction from wearable camera settings with strong motion' , in 12th International Conference on Computer Vision - Volume Part III , ECCV'12 , pp. 436 - 445 , ( 2012 ).

4. A. Satt , A. Sorin , O. Toledo-Ronen , O. Barkan , I. Kompatsiaris , A. Kokonozi , and M. Tsolaki , ' Evaluation of speech-based protocol for detection of early-stage dementia .', in INTERSPEECH , pp. 1692 - 1696 . ISCA, ( 2013 ).

5. G. Meditskos , S. Dasiopoulou , V. Efstathiou , and I. Kompatsiaris , ' Sp-act: A hybrid framework for complex activity recognition combining owl and sparql rules' , in PerCom Workshops , pp. 25 - 30 , ( 2013 ).

6. Michael Compton , Payam Barnaghi, and Luis Bermudez et al., ' The SSN ontology of the W3C semantic sensor network incubator group' , Web Semantics: Science, Services and Agents on the World Wide Web , 17 ( 0 ), 25 - 32 , ( 2012 ).

7. D. Nute , ' Handbook of logic in ai and logic programming (vol. 3)' , chapter Defeasible Logic, 353 - 395 , Oxford University Press, Inc., New York, NY, USA, ( 1994 ).

8. P. Wolf , A. Schmidt , J.P. Otte , M. Klein , S. Rollwage , B. Konig-Ries , T. Dettborn , and A. Gabdulkhakova , ' openaal - the open source middleware for ambient-assisted living (aal)' , in AALIANCE conference, Malaga, Spain, March 11 -12, 2010 , ( 2010 ).

9. N. Ga´mez and L. Fuentes, ' Famiware: a family of event-based middleware for ambient intelligence' , Personal and Ubiquitous Computing , 15 ( 4 ), 329 - 339 , ( 2011 ).

10. T. G. Stavropoulos, K. Gottis , D. Vrakas , and I. Vlahavas , ' awesome: A web service middleware for ambient intelligence' , Expert Syst. Appl. , 40 ( 11 ), 4380 - 4392 , ( 2013 ).

11. T. G. Stavropoulos, E. Kontopoulos , N. Bassiliades , J. Argyriou , A. Bikakis , D. Vrakas , and I. Vlahavas , ' Rule-based approaches for energy savings in an ambient intelligence environment' , Journal of Pervasive and Mobile Computing , to appear.

12. A. Capone , M. Barros , H. Hrasnica , and S. Tompros , ' A new architecture for reduction of energy consumption of home appliances', in TOWARDS eENVIRONMENT, European conference of the Czech Presidency of the Council of the EU , pp. 1 - 8 , ( 2009 ).

13. M. Eisenhauer , P. Rosengren , and P. Antolin , ' Hydra: A development platform for integrating wireless devices and sensors into ambient intelligence systems' , in The Internet of Things , 367 - 373 , Springer, ( 2010 ).

14. A. De Paola , S. Gaglio , G. Lo Re, and M. Ortolani , ' Sensor9k: A testbed for designing and experimenting with wsn-based ambient intelligence applications' , Pervasive and Mobile Computing , 8 ( 3 ), 448 - 466 , ( 2012 ).