Activity Recognition and Personalized Feedback Solution for Active and Healthy Ageing Thanos Stavropoulos Georgios Meditskos Stefanos Vrochidis Ioannis Kompatsiaris Centre for Research & Technology Hellas, Information Technologies Institute 6th Km Charilaou – Thermi, 57001, Thessaloniki, Greece {athstavr, gmeditsk, stefanos, ikom}iti.gr ABSTRACT However, assistive technologies promise to alleviate those As the average lifespan increases, the care of diseases barriers by providing low cost self-management or at least related to lifestyle and age, such as chronic and remote and efficient clinical care. In detail, several such neurodegenerative ones, becomes costlier and less technologies, employing Internet of Things (IoT) on the accessible, highlighting the need for self-management using rise, are used to subjectively and intelligently enhance technology. This paper proposes a pervasive system for clinical diagnosis and decision making e.g. by efficiently autonomous healthy ageing, which integrates two layers of sensing and estimating cognitive status and disease intelligence via semantic interpretation: multi-modal sensor progression faster than standard neuropsychological tests fusion from smart devices, wearables and multimedia, and [2]. Furthermore, assistive technology, also met as Ambient personalized spoken feedback based on context-sensing and Assisted Living (AAL), is expected to play a critical role in user input. Aiming for a practical, acceptable system, the improving quality of life, both on cognitive and physical proposed architecture considers aspects of integration, level, by providing tailored interventions, advice and security, privacy and cost. The currently implemented support, without the need of costly in-person care [3]. components include activity recognition and problem detection, complemented by end-user applications and Still, current systems present many drawbacks, namely personalized spoken feedback. The proof-of-concept many of them target a single purpose (e.g. pharmacological implementation is evaluated both in a lab setting, for the treatment) or aspect (e.g. sleep quality or exercise). Other more complex personalized feedback component, and in systems are still based on end-user interviews, leading to four real home environments, presenting efficient activity generic interventions. Even though remote monitoring of recognition, and improvement in several patients is a promising “patient-centered” management neuropsychological areas, such as mood, physical approach that provides specific and reliable data, enabling functional and cognitive condition of elders. the clinicians to monitor daily function and provide adaptive and personalized interventions, these systems must Author Keywords provide a multi-modal view of several aspects combined Ambient Assisted Living; Active and Healthy Ageing; and be complemented with self-management functionality Sensors; Knowledge Management; Reasoning; Internet of to reduce the effort of clinicians. Things; Personalized Feedback; Activity Recognition; Towards this direction, we propose a holistic approach for INTRODUCTION context-aware monitoring and personalized home care, The increase of the average lifespan across the world has together with intuitive end-user interfaces to autonomously been accompanied by an unprecedented upsurge in the prolong independent living. To begin with, the system occurrence of dementia, with high socio-economic costs, integrates a wide range of sensor modalities and high-level reaching 818 billion US dollars worldwide, in 20151. analytics to support accurate monitoring of all daily life Nevertheless, its prevalence is increasing as the number of aspects including physical activity, sleep and activities of people aged 65 and older with Alzheimer's disease may daily living (ADLs). After all gathered knowledge is nearly triple by 2050, from 46.8 million to 131 million represented in a universal format, semantic interpretation, people around the world, the majority of which, living in an via a hybrid reasoning scheme, is used for complex activity institution [1]. Dementia, as well as several other ailments recognition from atomic events, emotional and well-being such as depression, cardiovascular diseases, obesity and bad status and highlighting clinical problems. habits (like smoking), require consistent lifestyle changes, usually through interventions driven by experts. However, The high-level meaningful information is presented in as the number of people in need of care together with the applications tailored to clinicians, but most importantly high costs as well as the inability in several regions for such end-users themselves. They are also to be exploited further high quality services prohibit in-person treatment. so as to automatically provide support and suggest interventions, combined with spoken user input as context. The proposed system is intended for real-life and wide- 1 Dementia Statistics by Alzheimer’s Disease Internationl - spread usage, hence, security and privacy aspects, cost, https://www.alz.co.uk/research/statistics equipment, acceptance, integration and interoperability 45 aspects are also discussed. A proof-of-concept has been Ontologies have been extensively used in natural language deployed and evaluated either in home settings focused in interfaces and information extraction [11][12], offering dementia, showing effective performance for activity vocabularies and reasoning services to fuse contextual recognition and long-term improved in several domains. information [13] and solve disambiguation problems [14]. However, the ability to provide personalized responses The system builds upon lessons learned from previous requires not only language understanding, but also coupling work: smart home, wearable, image and audio sensing profile and clinical knowledge. In our work, this coupling integrated in an existing service-oriented middleware, for personalized responses is realized through a together with semantic models for activity recognition [4]. combination of ontology reasoning and SPARQL. The novelty of this paper is their further integration of sensed qualities as context, with an additional layer of THE PROPOSED SYSTEM intelligent spoken feedback, enabling autonomy and The scope of the system is to assemble a secure, compact previously impossible self-management, in a platform that solution, deployable for a wide audience and enabling self- accounts for security, usability and cost aspects. management for healthy ageing, utterly reducing effort and cost of clinical dependence and early hospitalization. To do The following sections present: related work, overall so, the system must not only employ reliable sensing, but aspects and requirements, the proposed architecture, also adaptive personalized and human-intuitive interaction, sensing and analysis, the personalized spoken feedback while clinical oversight and visits are becoming rarer. The method, the end-user interface, proof-of-concept evaluation, proposed system addresses these requirements by conclusions and future work. integrating a set of best practices, the latest technological RELATED WORK standards as well as valuable lessons learned from past Pervasive technology solutions have already been employed research. The overall concept is shown on Figure 1. In in several ambient environments, either homes or clinics, general, the system incorporates two layers of intelligence: but most of them focus on a single domain to monitor, holistic multi-modal sensing interpretation and personalized using only a single or a few devices. Such applications spoken feedback. At the sensing layer, sensor and include wandering behavior prevention with geolocation multimedia analysis are semantically combined to provide devices, monitoring physical activity, sleep, medication and higher-level meaningful qualities. These constitute the user performance in daily chores [3] [2]. status after interpretation, namely physical, emotional, cognitive, medical and social state (e.g. via monitoring In order to assess cognitive state, activity modelling and heart rate, stress, word utterance and medication). The recognition appears to be a critical task, common amongst second layer of intelligence capitalizes this information as existing assistive technology. OWL has been widely used context, which combined with spoken user input, can lead for modelling human activity semantics, reducing complex to further personalized feedback. On the other hand, activity definitions to the intersection of their constituent clinicians are constrained to only providing the parts. In most cases, activity recognition involves the recommended set of interventions to the agent, perform segmentation of data into snapshots of atomic events, fed to seldom clinical visits, elicit end-user and clinical the ontology reasoner for classification. Time windows [5] requirements and evaluating the system. and slices provide background knowledge about the order or duration [6] of activities. In this paradigm, ontologies are Moving from concept to implementation, the proposed used to model domain information, whereas rules, widely system follows a multidisciplinary approach to integrate embraced to compensate for OWL’s expressive limitations, and bring into effect clinical expert knowledge in an AAL aggregate activities, describing the conditions that drive the system. The system employs a synergy of the latest derivation of complex activities e.g. temporal relations. advances in sensor technologies, fusion and mining, knowledge representation and personalized feedback. The Focusing on clinical care through sensing, the work in [7] detailed architecture is shown on Figure 2. A sensing has deployed infrared motion sensors in clinics to monitor submodule integrates several heterogeneous devices and sleep disturbances, limited, though, to a single sensor. protocols in order to satisfy the variety of modalities Similarly, the work in [8] presents a sensor network mandated by the requirements. The modalities are retrieved deployment in nursing homes to continuously monitor vital by various lifestyle sensors and wearables. However, the signs of patients. Other systems employ environmental sensing submodule also integrates more complex data sensors to observe and assess activities [9] or security format retrieval in the form of image and audio together monitoring with actuators to control doors [10]. with the corresponding specialized processing techniques. Nevertheless, it so far lacks the ability to fuse more sensor modalities such as sleep and ambient sensing, with limited After an early fusion and preprocessing takes place, interoperability. On the other hand, the proposed system information is unanimously stored in the Knowledge Base offers a unified view of many life aspects, including sleep (KB) where it is interpreted and fused into higher-level and activities, to automatically assess disturbances and their meaningful information such as physiological, medical, causes, to support end-users and clinicians. cognitive, emotional and social aspects. Together with 46 provide Zero Knowledge, but it can also be implemented easily in owned cloud infrastructure. In the proposed architecture, the clinical process is continuously supported in two modes: the validation and the operation mode. During the validation mode, clinicians can transcode their expert knowledge for interventions into the system while they also perform clinical visits to the end- users for complimentary assessment and interventions. The system continuously supports them. End-user and clinician requirements periodically reform the system according to validation results. On the other hand, in the operational mode, interventions are more or less pre-decided, clinical visits are rare and the system is not reformed until critical updates, to allow smooth deployment and operation. Figure 1. Overall system concept, sensing, interpretation, Since this proposed architecture is multi-layered and intelligent coaching and end-user interaction. diverse to be thoroughly presented here, this paper focuses on the key components for personalized feedback. The expert knowledge apart from showing the progress to end- following sections present an overview of sensing users and clinicians, the system employs personalized modalities and analysis, fusion and personalized feedback, feedback to provide advice and support for end-user self- which are then evaluated in a proof-of-concept management. implementation. To maintain compatibility with global IoT solutions, the SENSOR DATA RETRIEVAL AND ANALYSIS proposed system is built as a service-oriented middleware The sensing submodule includes two streams for data which can provide adapters to integrate with open IoT retrieval and processing, sensors and multimedia, as well as platforms. This process helps provide service discovery, the modules for early fusion, i.e. preprocessing and matching and composition both internally to the system but transformation. After all information is unanimously stored also externally. The system can benefit from existing in the knowledge base, it can be further interpreted and solutions for the lifecycle that open platforms provide and displayed to end-user or clinician interfaces. Further details disseminate its own services to their wider ecosystems of for each of these modules is given below. solution providers. Some of the most popular IoT platforms to provide adapters for, are FIWARE, universAAL and the IoT Lifestyle and Wearable Sensing emerging and most relevant, ACTIVAGE2. The system currently integrates a wide selection of proprietary, low-cost, ambient or wearable devices, Regarding the privacy, the proposed techniques for an AAL originally intended for lifestyle monitoring, repurposed to a system extend beyond simple approaches, such as the medical context. This variety satisfies both the required removal or masking of the direct identifiers (i.e. names, modalities and the user needs according to context, always identifications IDs, etc.), to mature technologies such as k- finding a balance between comfortability and functionality. anonymity and methods such as differential privacy, syntactic anonymity, homomorphic encryption, secure In detail, Ambient depth cameras3 are collecting both image search encryption and secure multiparty computation. and depth data. The Plug sensors4 are attached to electronic devices, e.g. to cooking appliances, to collect power Regarding security, the proposed system employs standard consumption data. Tags5 are attached to objects of interest, enterprise protocols for secure authentication (OAuth) and e.g. a drug-box or a watering can, capturing motion events transmission/retrieval (HTTP/SSL) from end-user sites to and Presence sensors are modified Tags that detect people’s the cloud. Beyond most existing commercial cloud services, presence in a room using IR motion. A selection of AAL systems should not only encrypt transmissions (to wearable Wristwatches6 according to needs may measure repel man-in-the-middle attacks) but also encrypt stored physical activity levels in terms of steps, heart rate and data and the passwords to decrypt them in the client side. This approach, namely Zero Knowledge cloud storage, is 3 considered to be the most current and trustworthy method Xtion Pro as the beholder (service provider) himself is unable to view (http://www.asus.com/Multimedia/Xtion_PRO/) (and further exploit) sensitive information. Depending on 4 Plugwise sensors (https://www.plugwise.nl/) the installation, commercial cloud infrastructures can 5 Wireless Sensor Tag System (http://wirelesstag.net/) 2 6 https://www.fiware.org, http://universaal.sintef9013.com, Jawbone UP24, FitBit Charge HR, Microsoft Band, == http://www.activageproject.eu and Empatica E4 47 𝑃𝑟𝑒𝑝𝑎𝑟𝑒𝑇𝑒𝑎 ≡ 𝐶𝑜𝑛𝑡𝑒𝑥𝑡 ⊓ ∃𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠. 𝐾𝑒𝑡𝑡𝑙𝑒𝑂𝑛 ⊓ ∃𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠. 𝐶𝑢𝑝𝑀𝑜𝑣𝑒𝑑 ⊓ ∃𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠. 𝐾𝑒𝑡𝑡𝑙𝑒𝑀𝑜𝑣𝑒𝑑 ⊓ ∃𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠. 𝑇𝑒𝑎𝐵𝑎𝑔𝑀𝑜𝑣𝑒𝑑 ⊓ ∃𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠. 𝐾𝑒𝑡𝑡𝑙𝑒𝑂𝑓𝑓 According to clinical experts involved in the development so far, highlighting problematic situations next to the entire set of monitored activities and metrics would further facilitate and accelerate clinical assessment. This is addressed by a set of predefined rules (expressed in SPARQL) with numerical thresholds that clinicians can adjust and personalize to each of the individuals in their care. Furthermore, each analysis is invoked for a period of time allowing different thresholds for different intervals e.g. before and after a clinical intervention. Problematic Figure 2. System architecture, end-user and clinician situations supported so far regard night sleep (short evaluation process. duration, many interruptions, too long to fall asleep), physical activity (low daily activity totals), missed activities EDA, while a pressure-based Sleep sensors7 are placed (e.g. skipping daily lunch) and reoccurring problems underneath the mattress to record sleep duration, phases and (problems for consecutive days). The following example interruptions. illustrates a rule for a short sleep duration problem: Each device is integrated by using dedicated modules that CONSTRUCT { wrap their respective API, retrieve data and process them ?new a :SleepDurationProblem; accordingly to generate atomic events from sensor :duration ?D; :date ?date. observations e.g. through aggregation. In the case of image } data, computer vision techniques are employed to extract WHERE { information about humans performing activities, such as ?activity a :Sleep; :startTime ?st; :endTime ?et. opening the fridge, holding a cup or drinking [15]. Standard BIND(:duration(?st, ?et) as ?D) microphones are used to retrieve audio. { Fusion, Activity Recognition and Problem Detection SELECT ?_d ?ActivityType To obtain a more comprehensive image of an individual’s WHERE { condition, semantic fusion is used to transform atomic ?p a :SleepDurationPattern; sensor events to complex ones, such as daily activities, and :hasDescription [ :definesActivityType[ identify problematic situations. For this purpose the system :classifiesActivity :Sleep; employs a hybrid combination of OWL 2 reasoning and :hasDurationDescription [ SPARQL. time:seconds ?_d]]]. Regarding activity recognition, a simple pattern models the } context of complex activities. Each activity context is } FILTER(?D > _d) described through class equivalence axioms that link them BIND (extract_date(?startTime) as ?date) with lower-level observations of existing domain models } (which can be found in [4]). The instantiation of this pattern is used by the underlying reasoner to classify context PERSONALISED SPOKEN FEEDBACK instances, generated during the execution of the protocol, as Empowering and motivating people in need of guidance complex activities. The instantiation involves linking ADLs and care due to age-related conditions is essential in order with class equivalence axioms. For example, given that the to preserve elderly’s ability to remain active and activity PrepareTea involves the observations KettleOn, independent, with the highest quality of life. This second CupMoved, KettleMoved, TeaBagMoved and KettleOff, its layer of intelligence in the system shows how monitored semantics are defined as: qualities can be exploited for personalized assistance, suggestions and recommendations, using again ontologies and rules. This closed loop between the elderly and the system is realized through a natural language interface. Users are able to ask questions about their daily activities and habits, getting feedback and suggestions about health- related problems and situations in return. The underlying 7 Withings Aura and Beddit processes entailed are: 48 Figure 3 Conceptual dependencies among user profile, topics and clinical guidelines 1. Automated Speech recognition: In order to support the topics into concrete ones, defining necessary and sufficient transformation of spoken language into text, we use a OWL 2 restrictions for class membership. Examples of state-of-the-art ASR system8 that employs statistical such restrictions that capture domain knowledge and speech models for both acoustic and language modeling, formulate the verbal vocabulary the system can understand specifically trained for basic and healthcare domains. in the form of a Description Logic theory model [17] are shown below: 2. Language analysis: The language analysis consists in itself of four stages: a) surface-syntactic parsing, b) deep- − 𝑃𝑎𝑖𝑛𝑇𝑜𝑝𝑖𝑐 ≡ 𝐶𝑜𝑛𝑡𝑒𝑥𝑡 ⊓ ∃𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠. (𝑃𝑎𝑖𝑛 ⊔ 𝐻𝑢𝑟𝑡) syntactic parsing, c) frame-semantics parsing, and d) − 𝐻𝑒𝑎𝑑𝑎𝑐ℎ𝑒𝑇𝑜𝑝𝑖𝑐 ≡ 𝑃𝑎𝑖𝑛𝑇𝑜𝑝𝑖𝑐 ⊓ ∃𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠. 𝐻𝑒𝑎𝑑 projection to ontological representations [16]. The output − 𝐵𝑎𝑐𝑘𝑃𝑎𝑖𝑛𝑇𝑜𝑝𝑖𝑐 ≡ 𝑃𝑎𝑖𝑛𝑇𝑜𝑝𝑖𝑐 ⊓ ∃𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠. 𝐵𝑎𝑐𝑘 is a set of FrameNet-based structures projected to a − 𝐴𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝑇𝑜𝑝𝑖𝑐 ≡ 𝐶𝑜𝑛𝑡𝑒𝑥𝑡 ⊓ ∃𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠. 𝐴𝑐𝑡𝑖𝑣𝑖𝑡𝑦 DOLCE+DnS UltraLite9 compliant representation. − 𝑆𝑙𝑒𝑒𝑝𝑇𝑜𝑝𝑖𝑐 ≡ 𝐴𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝑇𝑜𝑝𝑖𝑐 ⊓ ∃𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠. 𝑆𝑙𝑒𝑒𝑝 3. Question topic understanding: This tasks is responsible − 𝑁𝑖𝑔ℎ𝑡𝑆𝑙𝑒𝑒𝑝𝑇𝑜𝑝𝑖𝑐 ≡ 𝑆𝑙𝑒𝑒𝑝𝑇𝑜𝑝𝑖𝑐 ⊓ ∃𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠. 𝑁𝑖𝑔ℎ𝑡 for bringing conversational awareness into the system, − 𝑆𝑡𝑟𝑒𝑠𝑠𝑇𝑜𝑝𝑖𝑐 ≡ 𝐶𝑜𝑛𝑡𝑒𝑥𝑡 ⊓ ∃𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠. 𝑆𝑡𝑟𝑒𝑠𝑠 recognizing the topic of the question based on the − 𝐷𝑢𝑟𝑎𝑡𝑖𝑜𝑛𝑇𝑜𝑝𝑖𝑐 ≡ 𝐶𝑜𝑛𝑡𝑒𝑥𝑡 ⊓ ∃𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠. 𝐷𝑢𝑟𝑎𝑡𝑖𝑜𝑛 language analysis results and on a topic OWL 2 ontology. − 𝐴𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝐷𝑢𝑟𝑎𝑡𝑖𝑜𝑛𝑇𝑜𝑝𝑖𝑐 ≡ 𝐴𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝑇𝑜𝑝𝑖𝑐 ⊓ 𝐷𝑢𝑟𝑎𝑡𝑖𝑜𝑛𝑇𝑜𝑝𝑖𝑐 4. Question interpretation and reasoning: Each concrete topic of the topic ontology is associated with a rule The root of the hierarchy is the 𝐶𝑜𝑛𝑡𝑒𝑥𝑡 class that allows template that couples profile and clinical knowledge to 𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠 property assertions to be defined for associating derive the context of the response to a given question. language analysis results. More specifically, the context of each question is represented as an instance of the 𝐶𝑜𝑛𝑡𝑒𝑥𝑡 5. Language generation: The verbal communication concept that is associated though multiple 𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠 capitalizes on the ontological representations returned assertions with language analysis frame entities (verbal from question interpretation, following the inverse domain model). For example, a context instance (i.e. user cascade of processing stages described in language utterance) containing the domain elements 𝑁𝑖𝑔ℎ𝑡, 𝑆𝑙𝑒𝑒𝑝 analysis. and 𝐷𝑢𝑟𝑎𝑡𝑖𝑜𝑛 is automatically classified by the ontology In this paper, the focus is given on topic understanding, reasoned in the topic 𝐴𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝐷𝑢𝑟𝑎𝑡𝑖𝑜𝑛𝑇𝑜𝑝𝑖𝑐. As interpretation and reasoning (points 3 and 4). The rest of described in the next section, such multiple classifications this section describes the ontologies used to capture topics, act as semantic annotations of utterances and are used to clinical knowledge and how knowledge derived from extract relevant information from the KB through rules. sensor monitoring can be combined with rules to provide Reasoning and Feedback personalized feedback and suggestions. Like in domain models and spoken vocabulary, clinical Topic Understanding experts were involved to define the logic behind coupling An important aspect is the ability to recognize the context templates with user information. A context-aware of the question, so as to trigger the appropriate rule knowledge extraction module using rule templated was template and meaningfully extract and combine knowledge developed to retrieve information. Intuitively, a rule to respond to the user’s inquiry. The topic ontology was template acts as a conceptual link between question topics, designed collaboratively with the clinical experts that user profiling information and clinical logic (when needed). suggested both the needed topics and the underlying spoken Topic detection triggers the execution of a rule base that language semantics that characterize each of them. The defines the logic to extract, process and return information modelling follows a hierarchical decomposition of abstract relevant to user’s inquiry. The association of topics with templates is done in an abstract manner, exploiting the subsumption ontological hierarchy. Figure 3 graphically 8 illustrates the concept behind rule templates. http://www.vocapia.com/speech-to-text.html 9 One simple example of template-based reasoning is the http://www.ontologydesignpatterns.org/ont/dul/DUL.owl ability to answer questions about the duration of certain 49 activities. Such functionality is useful both for care- ?activityType rdf:subClassOf :Activity . recipients and care-givers that have the ability to get FILTER (?activityType != :Activity) activity logging information with a natural way. In this ?ad :contains [rdf:type ?period] . context, the classification of utterances in the ?period rdf:subClassOf :Period . FILTER (?period != :Period) 𝐴𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝐷𝑢𝑟𝑎𝑡𝑖𝑜𝑛𝑇𝑜𝑝𝑖𝑐 class is used to trigger the } corresponding rule base, without needing to couple clinical logic. This simple case can be handled by two template By inserting the two 𝑉𝑎𝑟𝑖𝑎𝑏𝑙𝑒 instances, the graph pattern SPARQL rules (SPIN rule10). More specifically, Rule 1 is in Rule 1 is now matched and a 𝐷𝑢𝑟𝑎𝑡𝑖𝑜𝑛 triple is returned executed on top of the RDF graph with the activity logs to and sent to the language generation. It is worth mentioning return the duration of an activity. that the query for extracting the variables for Rule 1 #Rule 1 requires the presence of a 𝑃𝑒𝑟𝑖𝑜𝑑 concept, denoting the CONSTRUCT { date of the activity (in this case : 𝑌𝑒𝑠𝑡𝑒𝑟𝑑𝑎𝑦). So, a [] a :Duration; :value ?d . question of the form: “What was the duration of the prepare } breakfast” does not return any result, since the system WHERE { misses information about the date when the activity should [] a :Variable; :name “ACTIVITY_TYPE”; have taken place. :value ?$ACTIVITY_TYPE . [] a :Variable; :name “DATE”; :value ?$DATE. Example of Spoken Feedback Based on Monitoring ?activity a ?$ACTIVITY_TYPE; :start ?start ; One of the use cases, inspired by the KRISTINA project11, :end ?end . involves the interaction of users with the system in order to FILTER (:match(xsd:date(?start) == ?$DATE))) acquire feedback about problems that may have. For BIND (:duration(?start, ?end) as ?d) example, the use may ask the system: “Why does my head } hurt?”. This is a question that requires the coupling of clinical knowledge, in order to give as feedback potential The variables starting with ‘$’ denote template variables causes of the headache, based on the user profile. For that are instantiated at runtime, based on the context example, clinicians suggested that the sleep quality of the captured in the topic. The custom SPARQL function previous night should be checked (based on the results : 𝑚𝑎𝑡𝑐ℎ tests the equality of a date value (? 𝑠𝑡𝑎𝑟𝑡) against provided by the sleep sensor of the framework), together another symbolic date value (e.g. Yesterday). The with the number of coffees the user had the last 24 hours. : 𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛 function computes the duration given two date Both are implemented in terms of SPARQL rules that time values. search the user activity monitoring graph to detect relevant Assuming that the user asked the question: “What was the patterns. duration of the prepare breakfast activity yesterday?”, the It is important to highlight that the detection of a question following context instance is generated (in Turtle format): topic (e.g. 𝐻𝑒𝑎𝑑𝑎𝑐ℎ𝑒𝑇𝑜𝑝𝑖𝑐 based on language analysis :ctx1 a :Context ; results 𝐻𝑢𝑟𝑡 and 𝐻𝑒𝑎𝑑) triggers a rule base that usually :contains [a :Duration]; contains more than one rule. Intuitively, each topic is :contains [a :PrepareBreakfast]; associated with a small rule-based application that tries to :contains [a :Yesterday] . match graph patterns in the activity log of the user. As an example, we present the rule for checking the sleep quality where 𝐷𝑢𝑟𝑎𝑡𝑖𝑜𝑛, 𝑃𝑟𝑒𝑝𝑎𝑟𝑒𝐵𝑟𝑒𝑎𝑘𝑓𝑎𝑠𝑡 and 𝑌𝑒𝑠𝑡𝑒𝑟𝑑𝑎𝑦 are of the previous night. If the value computed by the sleep domain concepts detected through language analysis. sensors is lower than a threshold, then this fact is marked as According to the domain model described earlier, 𝑐𝑡𝑥1 is a potential cause and returned as feedback to the user. automatically classified in the 𝐴𝑐𝑡𝑖𝑣𝑖𝑡𝑦𝐷𝑢𝑟𝑎𝑡𝑖𝑜𝑛𝑇𝑜𝑝𝑖𝑐 class, which triggers the second rule of the rule base (Rule CONSTRUCT { 2) to assert triples for the two variables used in Rule 1. [] a :Feedback; :value :NightSleep. } #Rule 2 WHERE { CONSTRUCT { ?activity a :NightSleep; :start ?s ; [] a :Variable; :name “ACTIVITY_TYPE”; :quality ?q . :value ?activityType . FILTER (?q < 0.4) [] a :Variable; :name “DATE”; :value ?period . FILTER (:match(xsd:date(?start)==:Yesterday))) } } WHERE { ?ad a :ActivityDurationTopic ; If sleep quality is less than 0.4, the rule asserts a 𝐹𝑒𝑒𝑑𝑏𝑎𝑐𝑘 :contains [a ?activityType] . instance in the RDF graph that is collected by another rule 10 11 http://spinrdf.org/ http://kristina-project.eu 50 accurate information detection. In this cycle, the knowledge from pilots was used to test the component by experts instead of pilot users to avoid confusing them with an early prototype building the prototype. Finally, clinical aspects are also presented for the pilot installations. Pilot Installations The system was evaluated in four home installations, in the residences of individuals living alone (for clinical aspects please refer to clinical evaluation below), and maintained for four months. Two additional installations are still sustained for a one-year total duration study. Sensors and relevant home areas or devices of the installation were selected after a visit from the clinician to the participants and follow the placement guidelines of Table 1. The Figure 4. The end-user tablet application interface. majority of deployed sensors covered the areas of kitchen, bathroom and bedroom, since these rooms are strongly of the rule base to compile an RDF response graph, which linked with most activities. is finally forwarded to language generation. Figure 5 shows a real-world installation, an image END-USER INTERFACE processing instance, collection of sensor events and Besides the personalized spoken feedback the users also aggregated information on an end-user tablet application have access to a tablet application with a Graphical User (although the installation is real, a professional actor is Interface (GUI), tailored to provide comprehensive, depicted here to preserve the privacy of actual participants). intuitive monitoring of their daily life aspects and feedback. These pilots were used so far to evaluate sensor data In detail it provides a restricted, simplified view of the most processing and interpretation performance such as activity important measurements so as to avoid overwhelming the recognition, stress detection, sleep problem detection and users or even stressing them out. The displayed interval long-term clinical evaluation, some of which are presented spans across three days of information regarding Physical below. Activity (daily steps and burned calories), Sleep, Usage of Activity Recognition Performance Evaluation Appliances and Medication. Besides user status and trends, High-level activity recognition via ontology-based fusion the application also provides feedback with regards to has been evaluated from the four real-world pilot problems detected such as sleep problems. It also provides installations. Ground truth was obtained via annotation, educational material, such as recipes or step-by-step based on images from ambient cameras. The metrics here instructions to perform routine tasks, and the ability to are Recall (or True Positive Rate, TPR) and Precision exchange messages between end-users and clinicians. (Positive Predicted Value. PPV), corresponding to activities Figure 5 shows an example view with a digested view of recognized with respect those actually performed. Clinical three-day trends of sleep aspects and a warning for many experts suggested five activities, which are shown on. Table sleep interruptions. Overall, the application is designed to 2 shows the activities together with pertinent context help patients feel confident and secure with the system they dependency models. are using, but also their relatives and carers as well as to encourage social interaction between them. End-user and The evaluation dataset spans over 31 days, in July 2015. As carer feedback for the application through questionnaires observed on Table 3, the more atomic and continuous an was so far positive. activity is, the more accurate the detection. In practice, PROOF-OF-CONCEPT EVALUATION BathroomVisit, the activity most accurately detected, is Due to its very high complexity and the limited scope of never interleaved to do something else. On the contrary, this paper, evaluation results are presented only for the most cooking is a long-lasting activity interrupted by instances of representative of components. Real-world pilot installations other events (e.g. watching TV) and influenced by were used to test sensor data retrieval, analysis and uncertainty and the openness of the environment. WatchTV interpretation, from which we present here the most and PrepareTea are fairly short in duration, causing less sophisticated result, namely activity recognition. The other uncertainty and interleaved events in-between, yielding key aspect of this paper, spoken feedback, is a rather decent precision and recall rates. complicated component to evaluate as it also requires prior Personalized Spoken Feedback Evaluation 51 After evaluating the accuracy of sensor recordings and dependencies. For example, it is assumed that for an high-level activity recognition, the personalized spoken utterance with 𝐻𝑢𝑟𝑡 and 𝐻𝑒𝑎𝑑 the system should always feedback component was separately examined, as it is not return possible causes of headache. However, the same yet part of the pilots. Instead, internal IT and clinical staff concepts are detected when the user just says: “My head was invited to test the current implementation of the hurts”, for which recommendation on how to stop the pain component. The assessment was performed in accordance would be more relevant than possible causes. Currently, the with Good Clinical Practice (GCP) following the system is not able to distinguish such cases, since the topic procedure: (i) Informed consent was given for recording hierarchy defines very abstract dependences between topics voice and filling a questionnaire, after a briefing of how and concepts, while language analysis does not provide the data will be used internally. (ii) The participants conducted type of utterance (e.g. statement, question, etc.). a guided conversation with the system. (iii) They filled in Improvement of Clinical Condition the questionnaire with assistance of personnel. The process Six individuals living in six separate homes participated: took about one hour per participant and the questionnaires five female (four Amnestic MCI, one mild dementia – AD) revealed the following critical aspects: and one male (mild dementia – AD). Besides regular Speech recognition and language analysis: Topic approximately weekly clinical visits, the system supported detection depends solely on the speech and language clinical objective insights as well as the participants and analysis output, which is used by the reasoner to classify their family. Significant improvement was found in post- utterance contexts in the topic hierarchy. The current pilot clinical assessment in multiple domains, utterly implementation is not able to handle missing information bringing about positive change in mood and cognitive state, and uncertainty. Therefore, the absence of a domain measured objectively via neuropsychological tests. In descriptor from the input, e.g. a missing 𝐻𝑢𝑟𝑡, preventing detail, the first participant has overcome insomnia the detection of the correct topic (e.g. 𝐻𝑒𝑎𝑑𝑎𝑐ℎ𝑒) to trigger (p=0.001) and neglecting daily chores (p=0.000), the the corresponding rule base. second has shown improvement in sleep (p=0.000) and lingering in the bath (p=0.03), while the other two have Level of detail of the topic hierarchy: The rules for been benefited with respect to sleep interruptions (p<0.02), collecting and returning feedback are based on abstract lack of sleep (p<0.08) and medication. Physical exercise increased for participant three (p=0.0) while TV watching Sensor Placement area or object reduced (p=0.03) and personal hygiene improved for Camera Kitchen, Living room, Hall participant four (p-0.001). While this information was Plugs TV, Iron, Vacuum, Cooking device, derived from statistical processing of system knowledge, Boiler, Kettle, Bathroom lightsDrug neuropsychological assessment post pilot showed Tags TV remote, Iron, Fridge door, statistically significant improvement (pair sample t-test) in Presence cabinet, Kitchen,Drug box, Tea Bathroom, bag,room Living Cup scales: Rivermead Behavioral Memory Test (p=0.03), Wristwatch Worn on the wrist MMSE (p=004), Hamilton depression scale (p=0.01), Sleep Under the mattress MoCA (p=0.004) and Rey Auditory Verbal Learning Test sensor (p=0.04). Two participants converted from aMCI to SCI, Table 1. Sensors in home installation. with no pathological depression or anxiety symptoms, and one with moderate dementia switched to mild. For the last Activity Context dependency set Concept participant, state was unchanged but symptoms related to PrepareDrugBox DrugBoxMoved, DrugCabinetMoved, Parkinson’s were highlighted, showing the multi-domain Cooking KitchenPresence TurnCookerOn, KitchenPresence coverage of the system. Detailed clinical aspects are PrepareTea TurnKettleOn, TeaBagMoved, available in [18]. WatchTV CupMoved, KitchenPresence, TurnTvOn, RemoteControlMoved, CONCLUSION AND FUTURE WORK TurnKettleOff LivingRoomPresence BathroomVisit BathroomPresence, TurnBathrLightsOn This work has showcased a proposed system for self- managing healthy ageing, based on multi-modal sensor data Table 2. Context dependency models for the evaluation. analysis and personalized spoken feedback. The proposed architecture has considered not only the required Activity TPV PPV Activity functionality, but also interoperability, acceptability, cost, security and privacy aspects. Evaluation was carried out PrepareDrugBox 0.86 0.89 PrepareDrugBox through four real-world pilots, assessing activity Cooking 0.61 0.68 Cooking recognition effectiveness and clinical condition PrepareTea 0.81 0.86 PrepareTea improvement, while the more complex spoken feedback WatchTV 0.87 0.80 WatchTV was evaluated by experts. BathroomVisit 0.91 0.94 BathroomVisit To reach its full potential the system has yet to breach many Figure 5. Real-world installation, data collection and end- barriers, especially regarding feedback. Further language Table 3. Precision and recallapplication. user feedback for activity recognition. 52 semantics and handling uncertainty can provide for more a nursing home,” Int. J. Distrib. Sens. Networks, use cases of feedback, as shown in evaluation. Additional vol. 2012, 2012. pervasive and human intuitive modalities can be added such [9] S. Helal, W. Mann, J. King, Y. Kaddoura, and E. as an expressive compassionate avatar in hand with emotion Jansen, “The gator tech smart house: A sensing. Finally, decision making and acting on the programmable pervasive space,” Computer (Long. environment, e.g. setting home lighting, teleconferencing Beach. Calif)., vol. 38, no. 3, pp. 50–60, 2005. with friends and step-by-step guidance, can have much greater positive impact in people’s health. [10] S. G. Bonner, “Assisted interactive dwelling house,” in Proc. 3rd TIDE Congress: Technology ACKNOWLEDGMENTS for Inclusive Design and Equality Improving the This work has been funded by the projects H2020-ICT- Quality of Life for the European Citizen, 1998, vol. 645012 KRISTINA and H2020-IOT-732679 ACTIVAGE. 23, p. 25. REFERENCES [11] J. Sleeman and T. Finin, “Type prediction for [1] M. Prince, A. Wimo, M. Guerchet, G. Ali, Y. T. efficient coreference resolution in heterogeneous Wu, and M. Prina, “World Alzheimer Report 2015. semantic graphs,” Semant. Comput. (ICSC), 2013 The global impact of dementia. An analysis of IEEE, 2013. prevalence, incidence, cost and trends,” Alzheimer’s [12] R. Prokofyev, A. Tonon, M. Luggen, and L. Dis. Int. London, 2015. Vouilloz, “SANAPHOR: ontology-based [2] P. N. Dawadi, D. J. Cook, M. Schmitter- coreference resolution,” Int. Semant., 2015. Edgecombe, and C. Parsey, “Automated assessment [13] S. Dourlens, A. Ramdane-Cherif, and E. Monacelli, of cognitive health using smart home “Multi levels semantic architecture for multimodal technologies.,” Technol. Health Care, vol. 21, no. interaction,” Appl. Intell., vol. 38, no. 4, pp. 586– 4, pp. 323–43, 2013. 599, Jun. 2013. [3] C. Kerssens et al., “Personalized Technology to [14] D. Damljanović, M. Agatonović, and H. Support Older Adults With and Without Cognitive Cunningham, “Improving habitability of natural Impairment Living at Home,” Am. J. Alzheimers. language interfaces for querying ontologies with Dis. Other Demen., vol. 30, no. 1, pp. 85–97, 2016. feedback and clarification dialogues,” Web Semant. [4] T. G. Stavropoulos, G. Meditskos, and I. Sci., 2013. Kompatsiaris, “DemaWare2: Integrating sensors, [15] K. Avgerinakis, A. Briassouli, and I. Kompatsiaris, multimedia and semantic analysis for the ambient “Recognition of Activities of Daily Living for care of dementia,” Pervasive Mob. Comput., 2016. Smart Home Environments,” in 2013 9th [5] G. Okeyo, L. Chen, H. Wang, and R. Sterritt, International Conference on Intelligent “Dynamic sensor data segmentation for real-time Environments, 2013, pp. 173–180. knowledge-driven activity recognition,” Pervasive [16] L. Wanner et al., “Kristina: A knowledge-based Mob. Comput., vol. 10, pp. 155–172, 2014. virtual conversation agent,” in Lecture Notes in [6] T. Patkos, I. Chrysakis, A. Bikakis, D. Plexousakis, Computer Science (including subseries Lecture and G. Antoniou, “A reasoning framework for Notes in Artificial Intelligence and Lecture Notes in ambient intelligence,” in Artificial Intelligence: Bioinformatics), 2017, vol. 10349 LNCS, pp. 284– Theories, Models and Applications, Springer, 2010, 295. pp. 213–222. [17] M. Ballesteros, B. Bohnet, S. Mille, and L. Wanner, [7] R. Suzuki, S. Otake, T. Izutsu, M. Yoshida, and T. “Data-driven sentence generation with non- Iwaya, “Monitoring Daily Living Activities of isomorphic trees.” ACL (Association for Elderly People in a Nursing Home Using an Computational Linguistics), 2015. Infrared Motion-Detection System,” Telemed. e- [18] I. Lazarou et al., “A Novel and Intelligent Home Health, vol. 12, no. 2, pp. 146–155, Apr. 2006. Monitoring System for Care Support of Elders with [8] Y.-J. Chang, C.-H. Chen, L.-F. Lin, R.-P. Han, W.- Cognitive Impairment,” J. Alzheimer’s Dis., vol. T. Huang, and G.-C. Lee, “Wireless sensor 54, no. 4, pp. 1561–1591, Oct. 2016. networks for vital signs monitoring: Application in 53