Continuous person monitoring systems usually assume singleperson households. Presence of visitors, such as family members and friends, and multi-person households are therefore not taken into consideration, although they can still in uence results. However, in order to reason about a speci c monitored person's evolution and trends, it is needed to accurately recognise the di erent activities that these persons perform throughout a day. This paper considers multi-person households and focuses on the problem of assigning activities detected by a continuous monitoring system to the person that has performed these activities. The proposed solution consists of modelling the domain terminology into an ontology, as well as that person's typical habits by means of a Finite State Machine { represented in its own ontology { using the concepts from the domain ontology. Sequences of recognized activities are then compared to the Finite State Machines associated to the di erent persons in a household, and assigned to the person with the most similar modelled sequence.
Our aging society is facing huge challenges on a variety of domains, not least in the home-care domain. A lot of research is performed into how people can be continuously monitored remotely, so as to keep track of how they are doing, whether they stick to their treatment protocol, etc. In order to be able to reason about the evolution and trends, one rst needs to understand a person's habits, and, to this end, activity recognition is needed. One of the driving factors for the research presented in this paper, is the fact that most of the current StateOf-The-Art (SOTA) is focused on individuals. Oftentimes, the fact that human beings live and grow old together is neglected. One might argue that technology is not or less necessary when people are still living together, as they can monitor one another, but this is certainly not always the case. Some people already are mentally or physically less able to care for themselves, let alone for a partner or someone else living in.
A wearable cannot uniquely identify a person, as, per se, it is unaware of who is wearing it. The system should therefore allow to link a wearable with a person (e.g. by its unique name or an ID). Environmental sensors { sensors placed in the living environment of a resident, e.g. Passive Infrared Sensors (PIR) sensors, ambient temperature sensors, etc. { that complement the wearable can pick up signals from other (non-)monitored persons. Such sensors can also identify a person, in case the person wears a tag, when its (unique) signal is captured by a tag reader, e.g. in a speci c room of the household. Current monitoring solutions do not deal with this multi-person situation. Even in single-person households, visitors can be present. In order to correctly analyse a person's daily living habits, these potential problems need to be properly catered for.
This paper investigates multi-person scenarios and more speci cally focuses on the problem of assigning a recognised activity to one of the persons present in that environment. Conceptually, the following approach has been taken to tackle the aforementioned research challenge. Residents in a household have been asked to keep a diary of their daily activities. This diary is then used to generate a model that captures the typical activities that follow one another, with appropriate guards (e.g. the resident only takes a shower when he/she has gotten out of bed). In this work, a Finite State Machine (FSM) has been chosen as supporting technology. In principle, this FSM can be generated based on other data, where such patterns of daily living have been identi ed from historical data. The FSM-based approach does not assign single activities to a single resident directly, but considers whether an activity that is observed ts a (set of) FSMs that are linked to (a) user(s). Since each user has its own FSM, one strives to end up with a single FSM linked to that activity, which basically then assigns that activity to the resident corresponding to that FSM.
Using the xed scenario below, i.e. the morning scenario at an elderly couple's place, the developed algorithms and services can be validated and demonstrated and is used throughout this paper as a running example. The scenario requires a number of sensors to be installed at the household to detect the correct activity (later on referred to as Activity Profile) performed by the persons in that household. The goal of the activity attribution is to automatically distinguish which person is doing which activity in a multi-person household setting. 1. One person wakes up at 7:00, 2. and starts preparing breakfast at 7:02. 3. The second person wakes up ve minutes later, 4. and takes a shower at 7:07. 5. They have breakfast together at 7:20. 6. One of them starts reading the newspaper at 8:00, and 7. the other one takes a shower at 8:00.
The research presented in this paper was conducted in close collaboration with partners in the SMARTpro project.3.
In this section the context of the research challenge has been sketched. The adopted technologies, the actual ontology decomposition and the software architecture of the presented solution are detailed in Section 3 to 5. The next section, Section 2 introduces relevant related work. This paper is concluded in Section 6. 2
Research presented at the Pervasive Health Conference in Oldenburg, 2014 [
Another interesting approach can be found in this thesis proposal [
The algorithm and framework presented and evaluated in [
The evaluation of the algorithms presented in [
Although [
The authors in [
The heart of our proposed system is based on semantic technologies. An ontology is used to x the vocabulary used throughout this system. This vocabulary is modelled into a taxonomy, in which relationships (i.e. properties) and constraints (i.e. axioms) link the terms of this taxonomy together. The generic ontology structure is graphically illustrated in Figure 1. To keep the ontology properly maintainable, adaptable and con gurable, it has been split up in a number of re-usable ontology modules. All those ontology modules model a speci c part of the domain. Should the scenario evolve in such a way that certain aspects are not needed anymore, then the import approach can be modi ed in such a way that the redundant modules are omitted.
The following subsections detail the purpose and the contents of the actual domain ontology modules. 3.1
All common concepts representing the minimal ontological commitment are included in the SMARTpro Base ontology module, which is illustrated in Figure 2. 3.2
Each taxonomy module in this subsection extends the base ontology module, and thus also imports the http://users.atlantis.ugent.be/svrstich/SMARTpro/ SMARTpro_Base.owl ontology. One module for every room in the household or for every Activity Pro le to be processed, has been created.
Finally, these are all brought together, at least those modules needed for the current demonstration scenario, in the overall SMARTpro ontology. A representative sample can be seen in Figure 2.
import SMARTpro_ LyingInBed
import SMARTpro_
Base
SWRL Rules import
import SMARTpro_ Taxonomy SMARTpro_
General Bathroom Bedroom Kitchen Taxonomy together by the import approach.
According to the Free Online Dictionary of Computing4, a FSM or also known as a Finite State Automaton (FSA) is an abstract machine consisting of a set of states (including the initial, intermediate and nal states), a set of input events, a set of output events, and a state transition function. 4.1
We expect residents to behave in a more or less repetitive and structured manner. Note that this may not always be the case, and our approach will not be performing as desired, when the residents are rather unstructured in their daily active living. The states of the FSM are aligned with the SegregatedActivity concept from the domain ontology model, according to the scenario de ned in Section 1.
As a base ontology model to represent the FSM, the results from the IST
ELENA project [
Waking Up
Preparing Breakfast Taking Shower
Compulsory PriorStateVisit
Having Breakfast SimpleState FinalState
Reading Newspaper
Guard
Reading
Newspaper Transition
Condition
Guard
In Figure 3, the actual FSM is presented. It has to be read from left to right. According to the de ned scenario, for two residents: 1. The resident wakes up (Waking Up). 2. The resident takes either a shower (Taking Shower) or prepares breakfast (Preparing Breakfast). 3. After that, any of the two residents have breakfast (Having Breakfast) together. 4. One of the residents starts reading the newspaper (Reading Newspaper). 4a. However, before being allowed to do 4 http://foldoc.org/ that, there is a conditional guard (Reading Newspaper Guard) on the transition between having breakfast and reading the newspaper. 4b. In this situation this guard models that the resident should rst have taken a shower (Compulsory Prior State Visit). 5. The other resident, which does not satisfy the condition de ned in the guard, takes a shower (Taking Shower). 6. Now the second resident can also start reading the newspaper (Reading Newspaper).
In an M:N setting, i.e. more than one FSM belonging to more than one resident, multiple instances of such FSMs, each modelling slightly di erent situations, might be present. The goal of the detailed system is that, by analysing the incoming Activity Pro les from the low-level activity pro le recognition algorithms { the actual detection of the activities is beyond the scope of this research, i.e. it is assumed that certain activities can be detected by an external system, but whom actually performed those is to be determined { it can nd either one or more correct FSMs, still satisfying the sequence of already detected Activity Pro les. Moreover, as every FSM is linked in the system belonging to / representing the behaviour of a resident, we can conclude which resident has been responsible for the detection of those Activity Pro les. The system has been engineered in such a way that a true M:N situation can be supported. That is, every resident can have multiple instances of nite state machines. On the other hand, any given FSM can be linked to one or more residents. 4.2
The incoming sequence of detected Activity Pro les might not always correspond 100% with the pre-determined knowledge model. Perhaps, though, a similarity of 90% can be su cient to make a well educated guess as to whom was actually executing those activities. To be able to calculate this similarity a certain similarity measure is needed.
The problem of measuring \similarity" of objects arises in many
applications, and many domain-speci c measures have been developed, e.g., matching
text across documents or computing overlap among item-sets. Jeh and Widom [
In the previous subsections, the approach with using FSMs for the modelling and attribution of sequences of Activity Pro les to members of the household, has been detailed. Of course, before this methodology and corresponding algorithms can be used in a real-life setting, the actual FSMs corresponding to the typical behaviour of the residents need to be instantiated. The approach adopted for this research has been to ask the residents to keep track of their typical behaviour during a number of days. The resulting log les were then fed into the system to generate the corresponding FSMs from. A small sample of such a log le is given in the listing below. 29/06 7 : 0 0 : Kate wakes up 29/06 7 : 0 2 : Kate p r e p a r e s b r e a k f a s t 29/06 7 : 0 5 : William wakes up 29/06 7 : 0 7 : William t a k e s a shower 29/06 7 : 2 0 : Kate and William have b r e a k f a s t t o g e t h e r 29/06 8 : 0 0 : William r e a d s t h e newspaper 29/06 8 : 0 0 : Kate t a k e s a shower 5
Denoted with the title \DYAMAND", reference is made towards the integration
with the DYAMAND platform [
The raw low-level data produced by the sensors can potentially generate highfrequency data samples.The aim of the services in \External Sources" module is to gather this data, execute proprietary algorithms on it in order to generate what has been referred to as \Activity Pro les" earlier on. 5.3
Detected Activity Pro les are communicated to the MASSIF platform by means of the interface provided by the \Gateway" component.
Multiple sub-ontology modules have been created to support transparent personalisation and easy adaptation. The second component in Module III can be used to convert non-semantic data into semantic information, as well as to verify the correct formation of the incoming JSON fragment. 5.4
Module IV represents a data brokerage system. Its purpose is to streamline incoming data distribution. After all, the services responsible for the actual processing of the incoming data. The native message brokering system, as provided by the platform in Module IV, lacks support for intelligent data processing, e.g. by means of descriptionlogics reasoning. The service subscribes for speci c information, but cannot do this using rst order logic. Therefore, the Semantic Communication Bus (SCB) enhances the data brokerage system by subscribing to all passing messages.
In what has been presented so far, data engineering and communication mechanisms have been detailed.
The following paragraphs present in more detail how the algorithms are enabled in the MASSIF platform.
MCI Service In MASSIF a service that converts data into information, and processes this information to create knowledge is named a Meta-Context Information (MCI) Service.
Activity Pro le Attribution Service This is the service responsible for attributing the detected Activity Pro les to a resident in the monitored household, and this based on what is speci ed in the knowledge model, i.e. in the FSMs. On a high level, the process of performing this attribution uses a combination of code-based programming and SPARQL queries (between brackets where applicable) and can be summarised as follows: 1. The FSM instance that contains a state representing the activity pro le is queried. (=FIND FSM FOR ACTIVITY PROFILE) 2. For the given state from the FSM at hand, several speci c cases need to be analysed and treated di erently:
2.a. Check whether the given state represents an initial state in the FSM. (=ASK STATE INITIAL)
2.a.1. Execute the logic for an initial state in the FSM, i.e. check all clusters currently being found for all actors, until an empty cluster is found. After all, an initial state has to start from an empty cluster.
2.a.2. If it is an initial state, add it to an empty cluster or create a new cluster.
2.b. Check whether the given state represents a simple state in the FSM. (=ASK STATE SIMPLE)
2.b.1. Execute the logic for a simple state in the FSM, i.e. check all clusters currently being found for all actors, until an appropriate cluster is found, i.e. a cluster where the last activity pro le matches the previous FSM state of the current activity pro le being analysed. (=QUERY PREVIOUS STATE) 2.b.2. Iterate through all potential previous states according to the FSM ontology, until in the inner loop a cluster has been found where it can be added to.
2.b.3. Iterate through all clusters already present for one or more actors, and see whether the current activity pro le being analysed can be added to that cluster.
2.b.4. Check whether a guard is attached to the transition between the two states, and if so, if the guard has been satis ed. (=QUERY TRANSITION GUARD STATE)
2.c. Check whether the given state represents a nal state in the FSM. (=ASK STATE FINAL)
2.c.1. Execute the logic for a Final State in the FSM, i.e. check all clusters currently being found for all actors, until an appropriate cluster is found, i.e. a cluster where the last activity pro le matches the previous FSM state of the activity pro le being analysed.
2.c.2. Finalise the current FSM.
Performance evaluation The performance in terms of Round Trip Time (RTT) has been evaluated. On a laptop, running Windows 10, with an Intel(R) Core(TM) i7-6600U CPU, running at 2.81 GHz and having 15.7 GB of RAM available, the following averaged results were achieved: (i) Platform startup: 29 024ms, (ii) Activity Pro le Attribution service startup: 9ms, (iii) Gateway startup + adapter registration: 60 507ms, (iv) Activity Pro le Attribution algorithm cycle: 426ms, (v) Shutdown: 64ms. Apart from (iv), these timings represent a one-time execution. The algorithmic cycle can be subdivided into JSON fragment processing and the actual FSM matching aspect. On average 90% of the time is spent on the actual algorithm itself, with the remaining on the JSON processing. 6
Attributing detected activities in a multi-person household is a di cult task. In this paper, a semantic approach to follow-up on daily activities in a multihousehold setting has been presented. Both the software platform as well as the ontology model itself have been introduced. The algorithmic heart of the software is realised by means of ontology-based Finite State Machines. This system can help in allowing people to stay home in their habitual residences as long as possible.
The core concepts and technologies that have been used for this research are plenty, but appropriate for the tasks at hand. Ontologies have been used as central knowledge component. The SMARTpro project has allowed us to install sensors in residences, allowing us to collect data, together with a journal kept by the residents themselves. This data has been ingested through DYAMAND and processed by algorithms in the semantic MASSIF platform.
One aspect that needs more e ort is the generation of the log les. It has been observed that keeping track of one's daily behaviour, in order to be able to generate a corresponding FSM, can be a very labour intensive task. More intelligent / automated approaches are de nitely needed. This research has been supported by the Flemish Innovation and Entrepreneurship Agency (VLAIO) in the context of the SMARTpro VIS-trajectory.