Decision making by real time rule-based reasoners is prone to errors. Modern systems must implement decisions based on partial, statistical, or approximate information. If a part of the information is missing or corrupted, the reasoner might be misled into reaching wrong conclusions. Systems using monotonic static logic are less affected, as they must be more careful in their decisions by design. On the other hand, human reasoning would prompt for some conclusions even if the conclusions may be wrong, and later rectify them if a new fact was discovered. To mimic similar reasoning in order to achieve users' satisfaction, the use of non-monotonic reasoning is preferred as it allows us to easily model a situation whereby a missing piece of information completely changes the outcome of the reasoner. However, detecting reasoning incoherence is very challenging, and thus there is a need to formulate strategies to deal with reasoning defaulting. In this paper, we propose the use of a replay mode on a part of the dataset, including ground truth data. This would also address the problem of late data arrival. As a case study, this paper discusses the implementation of a decision rectification in our ontologybased tracking system that is currently deployed in real conditions.
Ontology-based reasoners are a convenient way to create an executable
description of a speci c universe using declarative programming [
The powerful combination of IoT and Linked Data deviates pervasive
computing away from prede ned bindings and from static communication protocols.
Semantic technologies are used to perform a context-aware service provision in
smart environments [
Besides, systems using this architecture rely on the information coming from outside, possibly sensor events and it is necessary to handle the uncertainty of the environment. Many of these systems are used to make a decision in real time and never get an opportunity to question this decision. The most common strategy in case of any delayed or missing information is simply to discard it. Obviously, this strategy disables a possibility of taking decisions back. As a result, it may be preferable to use repaired decision to make future suggestions or to enhance the uncertainty handling. We propose a methodology, design and implementation that enables the system to\re-think" and backtrack its inferences. In terms of validation, our approach has been implemented and tested in an IoT Ambient Assisted Living (AAL) platform currently deployed in 23 homes.
In this paper we present an implementation of a reasoning replay in an AAL platform in order to open a discussion on its potential as a new paradigm. In the following section, we compare our work to other projects and concepts. In the next section, we explain our implementation, followed by another section describing achieved results. Then, we discuss the relative signi cance of this approach, in addition to useful cases of its application. Finally, we conclude with the overall outcome. 2
Related Work “Reasoning replay” is a term that emerged from our platform implementation: it is the mechanism of resubmitting the knowledge to the reasoner in order to include facts omitted because they were unavailable at the moment when the decision making process was performed. Some systems accomplish a similar task without using this term { either because of its triviality (in trace based systems) where the reasoning replay is the key component of the design and the main functionality; or because of little or no added value in real-time systems with critical decision process and dilemmas (e.g. in driver-less cars). Our type of application of activity tracking ts in-between: decision repair is technically possible, and can greatly bene t the user monitoring the activity in real time. In the domain of activity tracking and ontology-based systems, we can nd several technological solutions neighbouring our domain.
Trace-driven systems provide an interesting insight into data processing. One
of the engines allowing to build such systems is kTBS, “a kernel for Trace-Based
Systems” [
Amongst applications built with kTBS, we can mention semantic wiki
interaction tracking via a browser plugin described in [
Similar concepts to ours, and their recent evolution are presented in [
Using a di erent perspective, eld of stream reasoning provides a solid
formalisation of reasoning over a possibly massive knowledge stream (e.g. sensor
events). The last decade displayed intensive e orts summarised in [
In order to assess the suitability of our solution, in this section, we describe our system's context, characteristics, and relevant con guration. 3.1
The purpose of the platform is to collect quality of life indicators, using them
to detect risky situations [
To achieve its goals, the platform UbiSmart makes use of standard commercialised sensors producing events, a gateway UbiGate that relays the structured event data to the main component { our server written in Node.js. Eventually, noti cations are sent to other devices as part of service provisioning. The reasoning is situated in the server: Incoming events are stored in a database, translated in triples using Notation3 format, and queued to be processed within the reasoning cycle.
Reasoning cycle Whenever the server is ready to process a queued event, reasoner (EYE4) is executed with knowledge originating from three ontological layers (as illustrated in Figure 1 as content of knowledge base, and in Listing 1.1, Figure 2): 4 Euler Yet another proof Engine, https://github.com/josd/eye
– Abstract model : Static ontology describing the world that does not change (sensor type associated to its abilities and characteristics, type and description of detectable activities); – Instance model : Instantiated ontology comprising of persisted information that was instantiated and has current information (e. g. sensor-room associations, durations of previously detected activity, . . . ); – Action: Injected knowledge produced by the sensors or a user interaction. Using the described knowledge base, we apply rules that bring conclusions about what the user's current activity is, compute new durations, update the persisted information about this instance, and decide what noti cations are to be sent. The conclusions are passed to software components performing decided actions. The reasoning cycle ends and waits till next event triggering a new one. Reasoning context In this paper, we operate with the term \reasoning context". It is simply a set of knowledge elements that are made available from previous reasoning cycles and carry an important message about the events. Reasoning context is volatile because it is not persisted and theoretically can be re-created. Keeping track of the past is useful when we want to reason over past usage of objects. However, restarting the platform have for consequence a loss of this data, as the data is stored only in the RAM.
Origin of semantic components Ontology in our system has been created
manually for UbiSmart [
qol:residentIn hom:house. hom:bed qol:locatedIn hom:house. hom:nap a qol:Activity. hom:livingroom a qol:Livingroom. hom:SleepMatSensor rdfs:subClassOf qol:Sensor. hom:SleepMatSensor qol:hasPossibleState _:b1. hom:SleepMatSensor qol:hasPossibleState _:b2. _:b2 qol:hasValue "bed_empty". _:b1 qol:hasValue "bed_occupied". _:b1 qol:indicateUse "true"^^xsd:boolean. _:b2 qol:indicateUse "false"^^xsd:boolean. # OUTPUT/INPUT originating from Persisted knowledge: hom:sensor_sleepmac_b8 rdf:type hom:SleepMatSensor. hom:sensor_sleepmac_b8 qol:attachedTo hom:bed. # OUTPUT/INPUT originating from Real-time injected knowledge: hom:clock qol:hasValue "2017-07-20T17:41:02+08:00"^^xsd:dateTime. hom:sensor_sleepmac_b8 qol:hasCurrentState _:b1. # OUTPUT Computed knowledge hom:bed qol:lastUsed "2017-07-20T17:31:02+08:00"^^xsd:dateTime. hom:sensor_sleepmac_b8 qol:lastUpdate "2017-07-20T17:41:02+08:00"^^xsd:dateTime. hom:clock qol:lastUpdate "2017-07-20T17:41:02+08:00"^^xsd:dateTime. hom:johndoe qol:detectedIn hom:livingroom. hom:johndoe qol:inRoomFor "6213.0"^^xsd:decimal. hom:livingRoom qol:motionMeasured "0"^^xsd:integer. # OUTPUT Computed knowledge Conclusions important for the end user: activity tracking hom:johndoe qol:believedToDo hom:nap. hom:johndoe qol:doesActivitySince "2017-07-20T17:31:02+08:00"^^xsd:dateTime. qol:believedToDo hom:johndoe
qol:doesActivitySince qol:locatedIn hom:bed hom:nap rdf:type qol:residentIn qol:lastUsed rdfs:subClassOf qol:hasPossibleState rdf:type qol:detectedIn hom:house 2017-07-20T17:31:02+08:00
qol:Sensor rdf:type qol:hasValue qol:indicateUse qol:hasValue qol:indicateUse qol:Activity qol:Resident hom:livingroom qol:House bed_occupied true
bed_empty false qol:attachedTo hom:sensor_sleepmac_b8
hom:clock qol:hasCurrentState rdf:type
qol:lastUpdate qol:lastUpdate
qol:hasValue qol:hasPossibleStatehom:SleepMatSensor 2017-07-20T17:41:02+08:00
Fig. 2. Visualisation of the knowledge base in Listing 1.1.
Listing 1.2. Example of activity detection rule: “nap” activity gets 4 more points if the user is detected in a livingroom for 600 or more seconds and motion was measured less than twice (within last 300 seconds). Generally, the rule-based scoring system (not detailed in this paper), selects the activity having the highest score to become the object of a Computed knowledge Conclusions triple hom:johndoe qol:believedToDo [] If there are too many activities with a similar score, then no activity is inferred. @prefix qol: <http://www.ubismart.org/n3/qol-model#>. @prefix hom: <http://www.ubismart.org/n3/home#>. @prefix math: <http://www.w3.org/2000/10/swap/math#>. {?u qol:detectedIn ?r. ?r a qol:Livingroom. ?u qol:inRoomFor ?d. ?d math:notLessThan 600. ?r qol:motionMeasured ?m. ?m math:lessThan 2} => {hom:nap :getScore 4}. 3.2
Some assumptions are necessary in order to apply enhancements. – Decision is an inference conclusion. The most obvious requirement is that the decision is a subset of inferred knowledge, e.g. all objects of a triple :Reasoner :concludes ?conclusion. – Independent cycles of iterations. The system works in cycles, a cycle is composed of one call or of a sequence of calls to a reasoner. The number of iterations within a cycle is supposed to guarantee the stability of conclusions. It depends on the complexity and the order of rules. Usually, the output of previous iteration is used as the input of the following one. – Time independence. The system must not depend on the processor's clock. All time-related computation must take inputs from a virtual clock that emulates the processing in the past. Time-related context must be entirely provided from the input parameters. – Full context available. Not only all of the events but also the reasoning context must be fully available for each cycle, otherwise, only full replay is possible. Usually, a database may contain signi cant snapshots is needed, otherwise, replay would be needed from the beginning of the last restart. 3.3
In this section, we detail some aspects of our system that is necessary to understand our implementation.
No repair strategy Our system performs a decision making process to infer the activities of a person. Inputs are events from various sources (remote sensor events, user interaction, and open data). Figure 3 illustrates the inputs, events, and main outputs, activities. In our real deployment, recurrent problems impacted the output: events were blocked on the gateway and could not reach our reasoner due to a connectivity problem, gateway malfunction, or temporary unavailability of the server intercepting the data. The system was not able to deal with delayed data. The information that eventually reached the server or could be extracted from backed up local copies of the data was not included in reasoning. The deployed strategy was to discard (ignore) delayed events. (a) (b) Time-awareness and “clock event” Our reasoning needs to take into account the time ow. To achieve it, we use the fact that events trigger a new reasoning cycle and carry the information about the time. To take care of the cases when the delay between the events is too long, we set a timer. If no event arrives, we issue a special volatile “clock event”. As any other event, it resets the timer and launches a new reasoning cycle. This way, we guarantee a minimum time resolution for the time dependent activity detection (if the person is in a room for more than 30 minutes and last sensor event happened 20 minutes ago, we can infer that the person might have fallen). In addition, within the cycle, we are able to compute durations and trigger time-related actions (e.g. noti cations). Setting the timer delay to shorter period gives higher time resolution but also may create a processing overhead. Unlike \real" events, “clock events” are not persisted. A representation of a “clock event” in the knowledge base is shown in Listing 1.3.
Listing 1.3. A “clock event” representation in knowledge base hom:clock qol:hasValue "2017-07-22T18:36:06+08:00"^^xsd:dateTime.
In terms of kTBS, recorded events are stored trace and the union of recorded events and “clock events” is computed trace of rst order. The activities inferred are computed trace of second order. 3.4
Fundamentally, requirements (Section 3.2) were respected, only time-independence had to be implemented. Initially, system was time-dependent and every event got evaluated at its arrival time using the server machine's system time to provide current time. This time measure was used to compute the duration of activities, quantity of movement of the person and other indicators.
Replacing the system time by the time included in every “clock event” with an optional delay of arbitrary one second (in order to get non-zero duration activities) ful lled the requirement of time-independence.
Automatic clock event has to be disabled during the replay. As every regular event starts a timer for an automatic clock event, we added a replay tag on all events prepared for replay. This property is used to discriminate and omit the replayed events, so that they are not duplicated in the event database and they do not trigger the automatic clock. If a new event (not a part of replay) enters the queue, the timer for clock events will naturally resume. 4
Our implementation yielded following results and observations that prove the feasibility of the approach, and underline some issues and open problems. Parameters For the actual replay test, we used a dataset containing 22 034 events covering 84 days of event data. After completing the dataset with 105 138 generated clock events indicating the points in time when the evaluation should take place, we obtained the total of 127 172 events to be replayed. The “clock events” were generated in the same way as the system would generate them in real-time mode: if the gap between two consequent events is more than 30 seconds, a “clock event” is inserted every 30 seconds in-between. Scheduling of clock events is illustrated in Listing 1.4.
Listing 1.4. Example of scheduled events (in JSON format) with replay tag, showing 30-second delay of the inserted clock event. { "date": "2017-07-20T21:31:23.000Z", "house": 101, "sensor": "sleepMac_b8", "value": "Bed_Occupied", "id": 708409, "createdAt": "2017-07-20T21:31:23.000Z", "updatedAt": "2017-07-20T21:31:23.000Z", "replay": true }, { "event": "clock", "date": "2017-07-20T21:31:53.000Z", "replay": true }, ...
This “clock event” timer parameter was chosen by tests with di erent values: with 10 seconds, the number of events was too high and with 10 minutes (600 seconds), the resolution of resulting reasoning was too low to activate rules that are capturing activities shorter than 10 minutes. As the event data came from the real deployment, we already had a dataset to compare with { the real-time inference. The parameters di erences are summed up in Table 1. Execution The process was executed on x86 64 architecture virtual host with 2 400 MHz CPU. It took 78 hours and 58 minutes to complete (using 63 hours and 12 minutes of processor time) in our UbiSmart platform built with Node.js. Observations Resulting activities have been compared to a running instance in real time. The di erences were observed and are illustrated in Figure 4. The expected di erences that the replay displays are as follows: 1. higher activity granularity (darker colour indicates more short activities), 2. new activities previously undetected (nap between 00:00 and 03:00), 3. over owing activities (18:00 to 20:00 nap, where it was not supposed to be detected). 5
The results suggest several questions around the proposed approach and its possible future. 5.1
As it can be clearly seen in the execution details in previous section, the replay is very resource intensive. Therefore, strategies for partial replay have to be considered for more e ective resource use. For example, store checkpoints of current state regularly and resume at a checkpoint to make it more e cient. However, depending on the ontology, the decision taken in the past can have impact deeply in the future. The question of handling such cases remains open.
We can distinguish two main motivations for such a replay: enhance the resulting dataset or enhance the decision process. For instance, in the rst case, the priority is to provide the best results even if they come later; the second case might consider delaying certain decisions or make them more fuzzy if a certain type of data tends to arrive late. These two points of view are mutually compatible, and can lead to a richer event handling and greater user satisfaction. kitchen activity bed activity watch TV
nap kitchen activity bed_activity watch TV
nap Wed 14 Jun 18:00
Wed 14 Jun 21:00
Thu 15 Jun 00:00
Thu 15 Jun 03:00
Thu 15 Jun 06:00
Thu 15 Jun 09:00
Thu 15 Jun 12:00
Thu 15 Jun 15:00
Thu 15 Wed 14 Jun 18:00
Wed 14 Jun 21:00
Thu 15 Jun 00:00
Thu 15 Jun 03:00
Thu 15 Jun 06:00
Thu 15 Jun 09:00
Thu 15 Jun 12:00
Thu 15 Jun 15:00
Thu 15 Replay implementation also opens a question about what to do with the old decisions. Shall we just replace them? If we keep them, how many layers of replay?
For our system, we decided to keep the older decisions in order to: keep track of the state of the system in the past (we have to know what an observer might have seen when looking at the system). This can also be used for a di erential analysis. We label previous decisions as \disabled by [a reference to a new decision]". As for our use-case, the optimal solution is to keep the oldest decisions and the newest one. It allows us to explain why the noti cation was sent without confusing the end user (caregiver) with too many versions of what the activity might have been. 5.3
In our case, the replay implementation helped to point out the di erence between an uninterrupted operation of the platform during the replay and the case when the platform is interrupted and restarted. When the platform is restarted, we may lose some context information. Replay allows us to test these di erences.
Another observation exposes some issues with previous implementation and brings a solution. It depended on the time when the event was e ectively evaluated within the reasoner. In the case of processor overload, e.g. unrelated heavy processing on the server, the events queued and were processed much later. We were computing the duration of an activity as the time di erence between the current machine time, and the rst in an uninterrupted series of the same event. As a consequence, the activities appeared to have a longer duration than they actually had. Another reason of discrepancies was the misalignment of the clocks of event producer and consumer where we could get negative duration of an inferred activity.
Particularity of the execution was its non-uniformity in time. In the beginning, the execution pace was quick and at around two cycles per second. In the middle, the pace slowed down to as low as 2 minutes per cycle. At the end, the execution resumed a quick pace at one cycle per second. 5.4
For future development, we consider these topics: \dynamic replay", when an event arrives late, roll back the events and replay; \real resume", context conservation during of platform restart; \progress monitoring" to show how the replay performs as part of other user interface replay management; \augmented visualisation" when the platform is confronted with a replayed information, serve it to the end user (caregiver). 6
In this paper, we explained our motivations for a semantic replay. We implemented and tested it in an existing Ambient Assisted Living platform. We demonstrated that activity tracking is a domain where this approach has some potential, in particular, because of the availability of unused data. We believe that the replay paradigm can bring useful insights for other systems as well, in the case of real time user-oriented systems, such as care-giving. Additional motivation is driven by users' needs to understand why a false alarm was triggered while being able to see a correction.
Even if it is clear that discarding events is the simplest strategy, the bene t of replay approach is highly case-dependent. Therefore, we believe that a discussion about this topic can be very enriching.
As any other feature, it makes sense to implement it only in certain conditions. In our case, the recalculation cost is high but its bene ts are higher, and we believe the process can be enhanced and be made more e cient in case of partial replay.