The KnowSense framework, presented in this work, supports monitoring behavioral aspects of individuals in goal-oriented scenarios, within controlled, pervasive environments. Semantic Web technologies, such as OWL 2, are extensively employed in KnowSense to represent sensor observations and application domain specifics as well as to implement hybrid activity recognition and problem detection solutions. Although the framework can be beneficial in a variety of domains that require multi-sensing and goal-oriented data analytics such as smart homes, it is currently applied in the eminent field of healthcare. In this proof-of-concept health application, it provides the semantic models and intelligent detection of Instrumental Activities of Daily Living (IADLs) to assist in the clinical assessment of autonomy at different stages of dementia.
A key clinical feature of the Alzheimer’s disease (AD) is impairment in daily
function, reflected on the difficulty to perform complex tasks, such as the Instrumental
Activities of Daily Living (IADL) [
Treatment of AD begins with its diagnosis, based on behavioral and cognitive assessment that highlight quantitative and qualitative changes in cognitive functions, behaviors and ADLs. Currently, such methods involve questionnaires and clinical rating scales, which unfortunately, cannot often provide objective and fine-grained information. In contrast, pervasive technologies promise to overcome such limitations using sensor networks and intelligent analysis to capture the disturbances associated with autonomy and goal-oriented cognitive functions. This way, they could extract objective and meaningful information about individuals’ condition for timely diagnosis.
In this direction, the paper presents KnowSense, a semantically-enriched framework for monitoring IADL activities in goal-oriented scenarios. KnowSense aims to provide the means to formally capture and integrate sensory observations, describe domainspecific use case scenarios of IADL, and support intelligent data analytics, interpretation and assessment services pertinent to each deployment. To this end, KnowSense follows an ontology-driven approach to data modelling and analysis, using OWL 2 ontologies to capture deployment-specific properties and sensory observations, while interpretation and assessment are performed using DL reasoning and rules.
KnowSense was derived from the Dem@Care suite [
The rest of the paper is structured as follows: Section 2 presents relevant work. Section 3 gives an overview of the framework, while Section 4 describes the ontologies used to represent goal-oriented scenarios and sensory observations. Sections 5 and 6 elaborate on data analytics, presenting the activity recognition and problem detection capabilities of KnowSense. Section 7 describes the GUIs supported by the framework to provide feedback to the clinical experts and Section 8 concludes the paper. 2
Pervasive technologies have already been employed in several ambient sensing
environments [
OWL has been widely used for modelling human activity semantics, reducing
complex activity definitions to the intersection of their constituent parts [
Focusing on medical care and ambient sensing, the work in [
KnowSense supports a rich selection of ambient and wearable sensors, listed on Table 1, which introduce multiple data modalities, such as image and video for specialized analysis, and more self-contained measurements, such as physical activity2, object motion and presence. A core objective of KnowSense is to recognize activity events which may be relevant to direct sensor outputs, e.g. activation of motion sensors, or even require intermediate data analysis e.g. posture recognition on video data. Its conceptual architecture, as depicted in Fig. 1, consists of three core layers: Semantic Knowledge Graphs: OWL vocabularies are used to build semantic knowledge graphs capturing (i) domain protocols, (ii) sensor and analysis observation types and (iii) IADL contextual models. The GraphDB3 triple store is used for persisting ontologies and data.
Semantic Knowledge Graphs
Activity Recognition OWL Ontologies RDF Triple Store
Activity Models OWL Reasoning Observations & Events
Problem Detection Feedback
SPARQL Queries
2 A wrist-worn 3D-accelarometer device provides physical activity metrics as described in [
KnowSense allows end users to model domain knowledge about (i) goal-oriented protocols, (ii) domain observation entities and events and (iii) IADL contextual models i.e. semantics of complex activities involved in each scenario.
A protocol (or scenario) is represented as instance of the Protocol class and is used to store information about its date, the participating individual and the involved steps (Fig. 2). The Participant instances allow profile-related assertions about participants to be defined, such as demographic, clinical and experimental records. A protocol step involves some tasks and has a start and an end timestamp. Our deployment implements three protocol steps, relevant to directed activities, semi-directed activities and discussion with the clinicians. Fig. 2 depicts the conceptualization of the semi-directed task step, along with some examples of IADL tasks involved. 4.2
Sensor observations, intermediate analysis results (e.g. posture) and recognized
activities are captured by extending the leo:Event class of LODE [
KnowSense provides a simple pattern (Fig. 4) for modelling the context of complex activities (IADL) i.e. semantics for activity recognition. Each activity context is described through class equivalence axioms that link them with lower-level observations.
The instantiation of this pattern is used by the underlying reasoner to classify context instances, generated during the execution of the protocol, as complex activities. The instantiation involves linking IADLs with context containment relations through class equivalence axioms. For example, given that the activity PrepareTea involves the observations KettleOn, CupMoved, KettleMoved, TeaBagMoved, KettleOff, TeaZone, its semantics are defined as: ≡ ⊓ ∃ .
⊓ ∃ ⊓ ∃ ⊓ ∃
⊓ ∃ ⊓ ∃ . . . Read article Water plant Prepare pillbox
Calculator
Answer phone Estabblaislhanaccecount (a)
Turn radio on Prepare tea
(b) . .
KnowSense implements a location-driven context generation and classification approach. The deployment room is divided into zones, according to the location each activity takes place (Fig. 5 (a)). When a participant enters a zone, KnowSense generates a Context instance and starts associating it with collected observations using contain property assertions, until he leaves it. The resulting context instances generated in each session are fed into the ontology reasoner to classify them in the activity hierarchy.
Fig. 5 (b) depicts two example context instances associated with a set of observations relevant to tea preparation. Based on the semantics of PrepareTea described in Section 4.3, c1 will be classified in this class, since all existential restrictions are satisfied. However, c2 will not be classified as tea preparation, since the context is not associated with any observation of type KettleOn, but rather translated into an incomplete activity, as described in Section 6.
Table 2 summarizes the performance on KnowSense on a dataset of 50 participants. TP is the number of IADLs correctly recognized, FP is the number of IADLs incorrectly recognized and FN is the number of IADLs that have not been recognized. Our approach achieves the best accuracy for “Prepare tea”, “Answer phone”, “Watch TV”, “Water the plant”, and “Write check”, whose activity models encapsulate richer contextual information, compared to “Prepare pill box” and “Read article”. On the other hand, the recall of these activities is relatively low, as they entail richer contextual dependencies and are, therefore, more susceptible to false negatives.
Notably, the activity contexts do not involve temporal restrictions. E.g. the semantics of PrepareTea in Section 4.3 do not involve temporal relations6. As activities do not usually manifestate in a predefined order, KnowSense uses loosely coupled activity models, based on containment relations, instead of highly structured ones. 6
The clinical experts highlighted the fact that, apart from recognizing protocol activities, the derivation of problematic situations would further support them for the diagnosis/assessment. Towards supporting this requirement, KnowSense has been enriched with a set of SPARQL queries to detect and highlight situations of possibly problematic behavior and of critical value to the clinical experts. Currently, abnormal situations detected include highly repeated, excessively long, incomplete and missed (absent) activities. The closed-world reasoning (e.g. instance counting or negation as failure) required to detect them, was implemented with SPARQL queries.
1: select ?x ?s ?e 2: where { 3: { 4: 5: 6: 7: } 8: 9: 10:} } FILTER (?n > 1) select (count(?o) as ?n) ?x ?s ?e { ?x a :Context; :contains ?o; :starts ?s; :ends ?e.
FILTER NOT EXISTS {?x a :Activity.}
6 The native OWL semantics do not support temporal reasoning. However, it can be simulated
using custom property assertions, as described in [
Activity repetitions correspond to the number of context instances classified into each activity type, highlighting a problem if there is more than one of them. Activity duration, computed from start and end activity timestamps, is compared to a reference duration per IADL suggested by the clinical experts. Missed activities correspond to IADLs not performed i.e. absent in the knowledge base while incomplete activities correspond to orphan context instances, i.e. those with more than one contains property assertion, but with no pertinent Activity classification.
The query in Fig. 6 defines a nested graph pattern (lines 3 to 8) to retrieve context instances ?x not classified as activities (line 6), while counting their contains property assertions (line 4). In order for the query to be successfully pattern matched, there should be more than one associated observations (line 9) apart from the location-related observation associated with all context instances. This helps eliminate cases where participants just enter zones without performing any action. In case of a match, the query returns the context instance ?x along with its start and end timestamps, used to provide pertinent feedback to the end users. 7
At the application level, KnowSense provides a multitude of user interfaces to assist clinical staff, summarizing an individual’s performance and highlighting abnormal situations. Fig. 7 depicts the Assessment screen, prior to the initialization of a protocol, where users can check the status and activate/deactivate sensors, according to the current protocol step, as described in Section 4.2. An example of the Results page for 4 IADLs is shown in Fig. 8 where both complete and incomplete activities are visualized (highlighted in green and red respectively). Various additional details for each activity are provided, such as their relative order, total duration and number of repetitions.
Fig. 7. KnowSense assessment application with real-time data collection
Meanwhile, the bottom of the screen shows a line-chart of the person’s moving
intensity, indicative of the time he has been walking (beginning and end of the session), as
measured by the DTI-2 sensor. The KnowSense framework deployment in Greece has
already been successfully carried out for more than a hundred participants, achieving a
mean accuracy of clinical assessment close to 83% among healthy and MCI participants
[
KnowSense enables complex task monitoring of individuals in controlled pervasive environment. The framework is currently applied in the field of healthcare, providing the semantic models and detection of IADLs to assist in the clinical assessment of autonomy and cognitive decline.
The activity recognition capabilities of KnowSense present certain limitations, significant to consider as future research directions. First, it cannot handle missing information, since activity semantics are modelled using fixed TBox axioms that should be all satisfied. Second, it does not handle uncertainty and conflicts, as it assumes that all observations have the same confidence. Although these limitations do not significantly impact the current lab deployment (given the predefined activity zones that simplifies activity recognition and compensates for sensor errors), deployment in more realistic environments, e.g. in homes, imposes additional challenges to be met.