With the always increasing availability of sensor devices, there is constant unseen monitoring of our environment. A physical activity has an impact on more sensor modalities than we could imagine. It is so vivid that distinctive patterns in the data look almost interpretable. Such knowledge, which is innate to humans, ought to be encoded and reason upon declaratively. We demonstrate the power of Markov Logic Networks for encoding uncertain knowledge to discover interesting situations from the observed evidence. We formally relate distinguishable patterns from the sensor data with knowledge about the environment and generate a rule basis for verifying and explaining occurred phenomena. We demonstrate an implementation on a real dataset and present our results.
With the always-changing physical environments, uncertainty and incompleteness are innate in them. Context-aware pervasive systems have been the centre of research regarding approaches to modelling uncertain contextual information and reasoning upon it [Bettini et al., 2010]; moving from lowlevel contextual data (i. e., sensors) to higher-level contextual information, where it is most commonly referred to as “situation” [Dey, 2001; Gellersen et al., 2002]. Setting up systems to observe an environment includes deploying probes (e. g., sensors) tailored to specific situations. Today, such efforts fell under the terms “internet of things” and “smart homes”. Many situations are worth identifying using sensors in a single room, ranging from “is someone present” to “water boiling”. Considering an entire home, we may end up with hundreds of such situations. An office building could have thousands, increasing dedicated sensors to cover all the above situations, driving higher economic and maintenance costs.
A compelling method in such deployments is to use indirect sensing, which is employed when the property in need (e. g., a situation) is not attainable to direct sense, either due to sensor malfunctions, connectivity issues or energy loss. ∗Contact Author In the literature, indirect sensing is interwoven with remote sensing or sensing from afar [Zhang et al., 2019]. In our study, we translate indirect sensing to a cooperative model of sensor fusion [Durrant-Whyte, 1990], where surrounding heterogeneous sensors capture different aspects of the same phenomenon (i. e., activity1). Activity is often described by a specific temporal organisation of low-level sensor data, or as we call it, a “dimensional footprint”( DF). The low-level sensor data in a DF are the primary source of information used as evidence to understand and recognise the observed situation. Such techniques following a bottom-up approach to recognising situations are well-established in the area of contextaware pervasive computing [Schmidt, 2003]. Dealing with a concept as the DF requires handling both uncertainty and the relational organisation. Existing approaches for an indirect sensing task typically fail to capture such aspects at the same time.
For the mechanics of an indirect sensing task, recent research targets data analysis techniques employing machine learning to train complex models labelling the property they want to infer from the data. For example, in [Laput et al., 2017] the authors train Support Vector Machine (SVM) models, in an automatic learning mode à la “programming by demonstration” [Dey et al., 2004; Hartmann et al., 2007], with raw sensor data while performing the activity of interest. The major limitation of such systems is that they use representations that are not relatable to humans. In addition, they do not support explicit encoding of knowledge about the environment. Background knowledge (e. g., contextual, domain or commonsense) may describe situations absent in training data or challenging to grasp and annotate. In addition, apart from the definition of knowledge, the occurred observables (i. e., events) in sensor data may be uncertain, as much as the manifestations of knowledge are (i. e., rules) in an analytical reasoning process.
We address these limitations by choosing a probabilistic logic-based approach using an amalgam of Event Calculus (EC) [Kowalski and Sergot, 1989] and Markov Logic Network (MLN) [Richardson and Domingos, 2006] to model uncertain knowledge about the relational manifestations of different and heterogeneous sensors reasoning to infer interesting situations. EC drives the modelling task by a set of meta1A situation, in that case, is the state of activity. rules that encode the interaction between the sensor events and their effects over discrete time. One of the exciting properties of EC is that a situation of interest persists over time unless it gets interrupted by the occurrence of other events. On the other hand, MLN combines first-order logic and concepts from probability theory to tackle uncertainty, which has received considerable attention in recent years with applications in video activity analysis [Cheng et al., 2014], maritime surveillance [Snidaro et al., 2015], music analysis [Papadopoulos and Tzanetakis, 2016] and others. Our goal is to design a reasoning mode for indirect sensing that handles uncertainty and uses interpretable representations from data. To this end, we make the following contributions: (1) We model existing sensor data into interpretable symbolic representations as elements in a narrative on a running scenario (cf. Section 2.2), (2) design a knowledge base (KB) within MLN for supporting indirect sensing while emulating commonsense reasoning, (3) evaluate the realisation of the approach using an open-source implementation of MLN, (4) demonstrate how the probability of an occurred situation changes over time while using different combinations of sensors.
Section 2 provides the terminology used in this document, including the running example and background information on Event Calculus and Markov Logic Networks. This leads to Section 3 where we introduce the concept of DF and how to model it. In Section ,4 we elaborate on MLN definitions, while in Section 5, we present the results and experiments. Section 6 provides a brief related work around the topic of event modelling and recognition. In Section 7, we summarise the main contributions and discuss details, including future work. 2 2.1
The Oxford English Dictionary gives a general definition for an event as “ a thing that happens or takes place, especially one of importance”. In our context, a “ thing” is represented by a (sensor) data pattern. “ Importance” matches the (subjective) interest in finding an explanation for this pattern. Many researchers try to use the term event in their way, depending on the context and the investigated environment, even though the definition of the word event remains the same.
We assume that an (interesting) event occurred on identifying a visible change in the sensor data. The identification involves a pre-processing step using some pattern extraction techniques [Patel et al., 2002; Lin et al., 2003; Yeh et al., 2016]. Therefore, the timestamps for the respective pattern represents the event’s temporality. This work clarifies a time point and a series of time points (exhibiting the concept of duration) bounded by a predefined window value. For example, the increase in the temperature readings is an interesting event and reflects the development of sensing data (temperature) over time. Therefore, a representation should semantically annotate an event’s time point.
Interpreting symbols as representations of objects is a proxy to describe something instead of the actual thing. For example, if something is an ambient “high” temperature 2, that temperature does not reside in our heads when we think of it. The “it” of the temperature is a representation of the actual natural environmental property. This representation of something is an entity that transmits to us the idea of the real something. Perhaps we think of our discomfort or imaging ourselves reacting to this phenomenon (e. g., sweating) to represent the high ambient temperature. Alternatively, we use the colour red accompanied by the temperature degree.
An event representation in our work is a lexical word embedded in a “sentence” among other additional contextual words, which we understand. Therefore, the development of sensing data over time (i. e., a time series) is wrapped in a word that best describes its nature (e. g., data pattern). The event representation has two lexical parts. The one part is the trend of the pattern, and the other one is the type of the pattern. The trend of a pattern is represented by the words upward or downward. The patterns we may derive in the sensor readings could resemble a shape currently named shapeoid. For the sake of presentation, the lexical shapeoids are the following: ANGLE A gradual, continuous line with an increasing (upward) or a decreasing (downward) trend in the sensor readings.
HOP A stage shift in the sensor readings, where the data have an apparent difference between two consecutive recognition time points (e. g., binary sensor values). HORN This pattern is a transient increase or decrease in the sensor readings curve.
FLAT A horizontal line in the data, with either unchangeable values in the pattern duration or minimal changes.
We extract the shapeoids using the Symbolic Aggregate Approximation (SAX) technique. Many time series representation alternatives exist, but most of them result in a downsampled real-valued representation. In contrast, SAX boils down to a symbolic discretised form of the time series, which is abstract enough to extract the shapeoids generally. The paper’s focus is not to describe how to obtain the proposed patterns from the sensor data but to put forward a concept of using temporal organisations of such representations to reason in a robust and declarative way. 2.2
In Figure 1, we illustrate the activity of opening and closing a window and its impact (i. e., their DF) on five surrounding sensor types that happen to be in the same room. Later in the paper (cf. Section 3.3), we will showcase the extracted shapeoids from the raw data, which put forward a sufficient abstraction, serving as an input for a reasoning task.
The data are from a real-world public dataset [Birnbach et al., 2019], where the authors collected sensor data while performing different activities. The data timeline spawns over two minutes, sufficient for demonstrating the essence of our approach.
2We use a threshold-based term to describe the comfort level for a human to endure. Window opened
Window closed Representing and reasoning about actions and temporallyscoped relations has been a critical research topic in the area of Knowledge Representation and Reasoning (KRR) since the 60s [Shoham and McDermott, 1988]. Since then, various approaches have been proposed to overcome the Frame Problem in classical Artificial Intelligence (AI) [McCarthy and Hayes, 1981; Shanahan, 2006]; the challenge of representing the effects of actions. Among them, EC, which Kowalski and Sergot have initially proposed in 1986 [Kowalski and Sergot, 1989], is a system for reasoning about events (or actions) and their effects in the scope of Logic Programming. It comprises excellent expressiveness with intuitive and readable representations, making it feasible to extend reasoning. It is an adequate tool to fit domain knowledge representing how an entity progresses in time using events. It has found applications ranging from the scope of robotics [Russel et al., 2013], game design [Nelson and Mateas, 2008] and commonsense reasoning [Shanahan, 2004; Mueller, 2014] to name a few.
From a technical point, the core ontology of the EC involves events, fluents and time points. The continuum of time is linear, and integers or real numbers represent the time points. A fluent can be whatever whose value is subject to change over time. At the occurrence of an event, it may change the value of a fluent. This could be a quantity, such as “the temperature in the room”, whose value varies in numbers, or a proposition, such as “the window is open”, whose truth state changes from time to time. In EC, the core axioms are domain-independent and define whether a fluent holds or not at a particular time point. In addition, these axioms can capture what is known as the common sense law of inertia; formal logic is a way of declaring that an event is assumed not to change a given property of a fluent unless there is evidence to the contrary [Shanahan and others, 1997].
We use a simplified version of EC (named MLN-EC), based on a discrete-time reworking of EC [Mueller, 2008], which was proven to work in a probabilistic setting [Skar(1) Predicate Happens(e, t) HoldsAt(f, t) InitiatedAt(f, t) TerminatedAt(f, t) HoldsAt (f, t + 1) ⇐
InitiatedAt (f, t) ¬ HoldsAt (f, t + 1) ⇐
TerminatedAt (f, t)
Meaning Event e happens at time t Fluent f holds at time t Fluent f is initiated at time t
Fluent f is terminated at time t Axioms
HoldsAt (f, t + 1) ⇐
¬ TerminatedAt (f, t) ¬ HoldsAt (f, t + 1) ⇐ ¬ HoldsAt (f, t) ∧ ¬ InitiatedAt (f, t) latidis et al., 2015]. Other dialects may have additional restrictions (e. g., complex time quantification) that hinder the realisation of the approach. For more information, we point the reader to this paper [Mueller, 2004]. The basic predicates and the domain-independent axioms are presented in Table 1. One can read the upper line of two axioms from left to right: (1) a fluent f holds at time t if it was initiated at a previous time point, and (2) that the fluent f continues to hold, providing it was not previously terminated. The domain-dependent predicates initiatedAt/2 and terminatedAt/2 are expressed in an application-specific manner guiding the logic behind the occurrence of events and some contextual constraints. One example of a common rule for initiatedAt/2 is:
InitiatedAt (f, t) ⇐
Happens (e, t) ∧
Constraints[t]
The above definition states that a fluent f is initiated at time t if an event e happens, and some optional constraints depend on the domain. EC supports default reasoning via circumscription, representing that the fluent continues to persist unless other events happen. Therefore, in our definition of the event narrative, we assume these are the only events that occurred.
A Markov Logic Network (MLN) amalgam of a Markov Network (aka. Markov Random Field) and a first-order logic KB [Richardson and Domingos, 2006]. Specifically, it softens the constraints posed by the formulas with weights that support (positive weights) or penalise (negative weights) worlds in which they are satisfied. As opposed to classical logic, all the statements are hard constraints (i. e., preserving truthfulness).
The formulas, being first-order logic objects [Genesereth and Nilsson, 1987], are constructed using four symbols: constants, variables, functions and predicates. Predicates and constants start with an upper-case letter, whereas the functions and variables have lower-case letters. The variables are quantifiable over the given domain (e. g., type={Temperature, Humidity} ). The constants are objects in the respective domain (e. g., sensor types: Temperature, Air Quality, Microphone etc.). Variables are ranges over the objects of the domain. The functions (e. g., downwardAngleTemp) represent actual mappings from a single object to a value or another object. Finally, the predicate symbols represent relations among objects associated with truth values(e. g., Happens(DownwardAngle_Temp,4)).
A KB in MLN consists of both hard- and soft-constrained formulas. Hard constraints (clauses with infinite weight) are interwoven with unequivocal knowledge. Therefore, an acceptable world fulfils all of the hard constraints. By contrast, the soft constraints are related to the imperfect knowledge of the domain, which can be falsified in the world’s existence in discourse. This means that when a world violates a formula, it is less probable but not impossible.
Formally, a MLN is a set of pairs (Fi, wi), where Fi is a first-order logic formula and wi is a real numbered weight. The KB L, with the weighted formulas together with a finite set of constants C = c1, c2, . . . , c|C| , defines a ground Markov Network ML,C as follows [Richardson and Domingos, 2006]: • ML,C has one binary node for each possible grounding of each predicate in L. The value of the node is 1 if the grounded atom is true and 0 otherwise. • ML,C contains one feature for each possible grounding of each formula Fi in L. The value of this feature is 1 if the formula is true and 0 otherwise. The weight of the feature is the wi associated with Fi in L.
An MLN is a template for constructing Markov networks: it will produce different networks given different constants. The grounding process is the replacement of variables with a constant in their domain. The nodes of ML,C correspond to all ground atoms that can be generated by grounding a formula Fi in L, with constants of C. Thus there is an edge between two nodes of ML,C iff the corresponding ground predicates are conditionally dependant on a grounding of a formula Fi in L. A possible world from the MLN must satisfy all of the hard-constrained formulas and be proportional to the exponential sum of the weights of the soft-constrained formulas satisfied in this world (cf. Equation 2). Hence, a MLN defines a log-linear probability distribution over Herbrand interpretations(i. e., possible worlds).
In an indirect sensing task context, we know a priori that we will have two kinds of predicates; the evidence variable X, containing the narrative of real-time input events, translated with the Happens predicates of EC, and the set of query HoldsAt predicates Y , as well as other groundings of “hidden” predicates (i. e., neither query nor evidence); in EC these are the InitiatedAt and TerminatedAt predicates. Finally, the conditional likelihood of Y given X is defined as follows [Singla and Domingos, 2005]:
P (y | x) = 1 Zx exp
X wini(x, y) i∈FY ! (2) x ∈ X and y ∈ Y represent the possible assignment of the evidence set X and the query set Y , respectively. FY is the set of all MLN clauses produced from the KB L and the finite set of constants C. The ni(x, y) is the number of true groundings of the i-th clause involving the query atoms y given the evidence atoms x. Finally, Zx is a partition function that normalises for all the possible assignments of x.
Equation 2 shows the probability distribution of the set of query variables conditioned over the set of observations. By modelling the conditional probability directly, the model remains agnostic about potential dependencies between the variables in X, and any factors that depend on X are eliminated. Instead, the model makes conditional independence assumptions among the Y and assumptions on its inherent structure with dependencies of Y on X. Therefore, in such a way, the number of the possible words is constrained [Singla and Domingos, 2005; Sutton and McCallum, 2006] and the inference is much more efficient. However, calculating exactly the formula might become intractable even for a small domain. Consequently, other approximate inference methods are preferred.
Originally, the authors in [Richardson and Domingos, 2006] propose to use Gibbs sampling to perform inference, but they found out that the sampling breaks down when the KB has deterministic dependencies3 [Poon and Domingos, 2006; Domingos and Lowd, 2009]. The authors proposed another Markov Chain Monte Carlo method called MCSAT [Poon and Domingos, 2006] based on satisfiability with slice-sampling. Another type of inference is the Maximum A Posteriori (MAP) which described the problem of finding the most probable state of the world given some evidence, which reduces to find the truth assignment that maximises the sum of weights of satisfied clauses (i. e.,argmaxp(y | x)). y
The problem is generally NP-hard, but both exact and approximate satisfiability solvers exist [Domingos and Lowd, 2009]. In our experiments, we run approximate inference using the MC-SAT algorithm. 3
An activity affects various fundamental environmental properties, such as speed, pressure, temperature, luminosity, etc. Surrounding sensors may capture the various changes (forming the activity’s DF), which depends on different contextual information, such as their proximity from the occurred phenomenon and their type (cf. Section 3.1). In addition, a sensor may observe ambient values (e. g., temperature) or require manual intervention to observe a change (e. g., separating the two magnetic elements of a contact sensor) (cf. Section 3.2).
This “change” (i. e., the forming pattern) is the “interesting event” we want to focus on. This observed change mostly stays unobserved. Thus, the emitted DF indicates its occurrence. In addition, its state is a continuous value in time, which is tracked under the definition of the “fluent” . With no sensor modality to identify the occurrence of an activity, due to its unavailability at the given time, or by simply stating that there does not exist any direct one, we account its DF as a space with equivalent options that “indirectly” account for the same activity.
Our work uses commonsense knowledge (CK) to characterise how activity affects its environment. From the running 3They are formed from hard-constrained formulas in the KB. example in Section 2.2, some distinct data patterns exist, almost as recognisable to the human eye where one may exercise a hypothesis against the data. We consider that a dataprocessing step is viable to extract such patterns, but it is out of the scope of the current paper. The abstracted representations (cf. Section 2.1) from low-level sensor data reflect their organisations in shapes and trends (e. g., an increasing angle in the sensor data). Therefore, one with a naive knowledge of physics can make hypotheses about the occurrence of an activity using the abstractions from sensor data as evidence (cf. Section 3.3). 3.1
Sensors are interfaces that serve as occurrence indicators for various monitored situations. The sensor numbers could increase accordingly as their numbers increase, making the instrumentation, deployment and maintenance cumbersome tasks. A sensor primarily measures an environmental change as accurate as possible, varying between the different manufacturers. Selecting a sensor to monitor a situation ought to obey some criteria, which formulate the sensing fidelity of its output. In this paper, we propose the following criteria: Type There exist different vendors for various sensors.
Nonetheless, the type of sensor is of key importance. There is no doubt that different manufacturers may offer a better sensor device, affecting accuracy. Semantically, the sensor type determines if the sensor participates in the verification process, not its model.
Location The location is another important aspect of determining the credibility of the sensor output. Either the physical location or the position of the sensor in the space should affect the decision of selecting any sensor of a given type in a location (e. g., a room).
As discussed later in the paper, the above criteria are minimal constraints for a sensor to participate in reasoning. However, the sensors have a fundamental high-level classification, making the shapeoid extraction from their data clearer and focused. 3.2
A sensor is an interface between the physical and the digital world. The raw sensor data rarely matches human semantics, but the representations of patterns in them are. The kind of sensor classification the paper foresees, bases on the nature of the resulting sensor data, is as follows: Binary sensors restrict their result to two possible values.
Usually, the values resemble the category itself (i. e., being binary); thus, one and zero. Furthermore, depending on the context4 the result may take values from it. For example, the output of a physical switch is “on” or “off”, the result of a motion sensor may be “present” or “not present”, and so on. The suitable data patterns for the binary sensors are the HOP and FLAT representations. 4Context is any information that one can use to characterise the situation of an entity. An entity is a person, place, or object considered relevant to the interaction between a user and an application, including the user and application themselves [Dey, 2000] Numerical sensors are almost every sensor with an arithmetic output in the set of real numbers R. Some examples of quantifiable sensors are an accelerometer, humidity, temperature, pressure sensor etc.. Accordingly, the data patterns, which we found in the raw sensor data, are those of ANGLE, HORN and FLAT.
One could say that a binary sensor is a subset of numerical sensors. However, we make the distinction explicit, as the binary sensors are semantically a practical standalone class. In the running example, we use numerical sensors. The sensor data’s available observations (i.e., shapeoids) are the simple events with their respective time point in the focused bounded time window. We represent them with the Happens predicate, where finally a collection of such predicates form the so-called “narrative” in EC. 3.3
An event “just” happens, with an accompanied discrete time point to keep a reference in the timeline. The chosen representation of it, according to the dialect of EC, is the predicate Happens(e, t). Time t can be quantified over the spectrum of integers, exhibiting coherence among the occurred events. The events in the sensing timeline are the formed shapeoids, and by using lexical words for the symbolic representation, the intuition behind them is human-readable (e. g., downward ANGLE). For example, in Figure 1, the two activities of opening and closing the window produce an impact in the five surrounding sensors. We observe that around the time of opening the window, distinct patterns are forming. Figure 2 contains in separate graphs a more clear view of the data in Figure 1, after performing a dimensionality reduction step (e. g., Piecewise Aggregate Approximation (PAA) [Ding et al., 2008]). The patterns were extracted empirically, resembling the proposed lexical shapeoids (cf. Section 2.1): . . .
. . .
. . .
. . . (3) 2.0 1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.00.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 (a) Temperature (b) Air Pressure .
In the following, we elaborate on constructing the KB containing the representations of the sensor events, using contextual words in “sentences” that comply with the formalism of EC and are expressed in first-order logic. 4.1
For our purposes, the KB, or the so-called “theory”, contains a few function definitions, predicate definitions, as well as the inertia laws axioms of EC5 as seen in (2). We consider the observed patterns as a continuous narrative of Happens predicates (cf. (3)). InitiatedAt and TerminatedAt determine under which factors a fluent is initiated or terminated at a given time point, using the form in (1). Finally, the query predicate HoldsAt incorporate a possible quantification over the verification of a monitored situation (i. e., a fluent).
Table 2 shows a fragment of the KB and the associated weights. The formulas are converted to a clausal form during the grounding phase, also known as conjunctive normal form (CNF), a disjunction of literals. The next step is the replacement of the variables with the constants, which formulate grounded predicates. As such, the construction of the Markov Network consists of one binary node V for each possible grounding of each predicate. A world is an assignment of a truth value to each of these nodes.
The definition of the indirect sensing rules follow CK represented as a theory in MLN enacting it as part of commonsense reasoning (CR)6; the sort of reasoning people perform in daily life [Mueller, 2014], which is vague and uncertain. For example, the Table 2 contains two separate rules, which reflect an atomic instruction of the DF, using a temperature sensor and a microphone. For our purposes, we consider that the events in the narrative are the only one occurred.
A rise in the temperature readings, or a sudden spike in the sound pressure levels, could be anything in an open world, including the opening/closing of a door in a room. However, with the help of context, we may exercise the hypothesis that a temperature sensor close to the window could indicate its 5They should remain hard-constrained; otherwise, the recognition of the situation will converge to be uncertain up to the horizon of probability.
6CR is implemented as a valid (or approximately valid) inference [Davis, 2017] in MLN as part of the EC law of inertia. state. The hypothesis is asked in the form of a query, representing the probability for the situation of an opened window to be true for the given observations (i. e., ground truth). For example, if we require to encode an “opened door”, we may include the same rule with a lower weight encoding our confidence for the result. Then, using the background knowledge that the sensors are closer to the window, we encode this with a higher weight value to the opened window rule. MLN has many learning algorithms [Richardson and Domingos, 2006] to determine the weight assignment; however, as we do not intend to select the absolute probabilities of a specific occurred situation, we opt for the most likely situation given the evidence. 4.2
The evidence contains ground predicates (facts) (e. g., the narrative of events in Section 3.3) and optionally ground function mappings. A function mapping is a process of mapping a function to a unique identifier. For example, the first formula in Table 2 contains the function downwardAngleTemp(r). During the grounding phase, constants from the domain of the variable r substitute it7. Thus, a function mapping could be the following: DownwardAngle_Temp_LocA = downwardAngleTemp(LocationA). All the events of the grounded Happens predicates in Section 3.3 follow the same procedure for their function mappings. 5
In this section, we evaluate our approach in the domain of smart homes. As presented in Section 2.2, we use a publicly available dataset. The data timeline spawns over twelve consecutive full days. The dataset was in a zip format, which contains multiple comma-separated value (CSV) files with a total size of approximately 50 Gigabytes (GB)8. We selected one device close to the interest situation (i.e., close to the window). We extracted the relevant data points using the five sensors capturing the DF of opening/closing the room’s window. We do not process the raw data points, but instead, we use the shapeoids from the data; their extraction was possible via our tool Scotty9. The total number of shapeoid events are 4393, where the ground truth events from the window contact sensor are 87.
7We assume a single room and its context is not reflected in the naming scheme of the function.
8The actual size of the raw data exceeds the 250 GB. 9This work is meant to be published in a forthcoming conference.
InitiatedAt (openedWindow (r) , t) ⇒Happens (downwardAngleTemp (r) , t) ∧
Happens (flatTemp (r) , t − 1) InitiatedAt (openedWindow (r) , t) ⇒Happens (upwardHornMic (r) , t) ∧ Happens (flatMic (r) , t − 1)
We put forward two scenarios (cf. Table 3), which contain rules for declaring the alternatives in recognising the situation of an opened/closed window. The purpose of the scenarios is to run the computation against the existing narrative with the discovered events but using different sensor compositions. Each recognition rule also contains a weight value, which was empirically assigned, as we consider them confidence values of the rule.
We implemented the KB and the narrative evidence file to demonstrate the approach’s feasibility using an opensource implementation of Markov Logic Networks, named LoMRF [Skarlatidis and Michelioudakis, 2014]. Together with the domain-dependent rules for each scenario, the full KB and the evidence file are publicly available online10, enabling the reproducible results. The KB, given the evidence, is transformed into a Markov Network of 26353 ground clauses and 13177 ground predicates. We run marginal inference from the developed MLN on vanilla runs without any interference from other processes. All the results are averaged over five runs with a corresponding standard deviation. The experiments are executed on a virtual machine(VM) running in a self-hosted data centre at the University of Ulm running on OpenStack under the series “Victoria”. The VM runs with 8 cores (16 threads) and 16 GB of RAM. Scenario S#1 S#2
TP 288 1016
In the experimental analysis, we present the results for the marginal inference in terms of F1 score for a range of thresholds between 0.0 and 1.0. We consider the situation recognition task successful with a probability above the specified threshold. In Table 4, we present a snapshot of the performance using the threshold value 0.6 in terms of True Positives (TP), True Negatives (TN), False Positives (FP), False Negatives (FN), Precision, Recall and F1 score.
The scenarios have a certain flavour. The basic intuition from the experiments is to showcase that we may use sensors that have an obscure interpretation (e. g., a spike in the microphone can be anything, even being next to the window) and sensors that act as a more direct verification step (e. g., air quality, temperature). We assume that the shapeoid events are the only ones that happen in the environment in focus. More alternative sentences may be encoded accordingly, using shapeoids of the humidity or the air pressure sensor. Based on the inertia laws of EC, the fluent start to hold at the time point t+1, and therefore the assignment to the next time point from the used pattern event in the narrative (3). In Figure ,3 the F1 score is higher for the marginal inference in S#2 due to the additional strong sensor. The S#1, similar to S#2, contains a shapeoid in the microphone data (increasing horn), which matches both the fluent’s initiation and termination rules. Hence, during the inference process, the probability always strives towards 0.5, which is regulated by another sensor in the rules (air quality sensor) with a higher weight value.
We note here that in a real setting, the verification of situation (i. e., the fluent) depends on whether the required observation is made (e. g., the shapeoid event from the temperature sensor), which may be a delayed effect of the activity itself - in other words: it takes some time until the open window affects the temperature sufficiently. In the experimental analysis, we calculate the performance measures strictly based on the time range of an opened window. Therefore the ground truth is the single point of reference for calculating the performance. The delay between the activity and its observable DF should be accounted for a more accurate timing prediction. We observe a considerable amount of FP, which indicates a plausible calculation of an opened window but with a certain recognition delay. Thus, we consider the F1 scores in the scenarios to be slightly higher.
Research in context modelling, context reasoning, and their unified view via various middleware systems is tremendous; for a recent survey, we point the reader to [Perera et al., 2013]. In the paper, we focus more on a bottom-up approach to the recognition of occurred situations. We employ a probabilistic rule-based approach, using occurred sensor events as evidence for the reasoning task.
In [Liu et al., 2017], the authors create a bottom-up hierarchical model using the raw sensor data as evidence while creating inference rules encoded in an MLN to recognise complex events. In order to create abstractions from the raw data, they use various thresholds per sensor type. In our approach, we use generic template abstractions which base on the data shapes and trends. The core contribution of their paper is the dynamic assignment of weights learned from a training dataset; we do not assume that the user has a training dataset to learn the weights from because we use them as confidence values for the inference rules. Finally, in our paper, we foresee scalability issues that may arise from the free variables in the MLN rules, which may drive the computation times to higher levels.
Considering our choice for a rule-based reasoning technique has a broad spectrum of applications to many domains, making it a commonly used technique [Perera et al., 2013]. Another interesting technique, which bases on previously acquired knowledge, is case-based reasoning (CBR) [Aamodt and Plaza, 1994; Biswas et al., 2014]. It offers solving mechanisms by adopting solutions that have been suggested to similar issues in the past. The authors in [Kofod-Petersen and Aamodt, 2003] use CBR to understand an occurred situation based on available contextual information. A case-based solution is not favourable in our case because collecting and maintaining previous cases is a cumbersome task. Our work does not require any previous known input from sensor observations and domain-dependent knowledge during the rule specification.
In the paper, we focus on finding alternatives for recognising a situation. Similarly, Loke [Loke, 2006] advocates that the situation in_meeting_now has different recognition ways based on contextual cues. The author follows an abductive treatment of the subject as we also do. In the forthcoming years, the author developed a formalism to represent compositions of sensors that can act on an understanding of their situations [Loke, 2016].
Finally, although sensing data contain implicit information, explicit domain knowledge is required for situation recognition. Many research works employ logic-based models for situation recognition in smart homes, such as the Event Calculus (EC) [Chen et al., 2008]. Other works have also employed EC in activity recognition from video streams [Artikis et al., 2014] and health monitoring [Falcionelli et al., 2019]. However, it is unclear how they move from the raw data to the tagged symbolic representations in these systems. 7
In the paper, we employed Markov Logic Networks for the modelling and reasoning over uncertain alternatives for the method of indirect sensing. We use the temporal formalism of EC as a “linchpin” for driving the reasoning about the sensing objects and creating observations for the occurrence of certain situations (e. g., “is the window open”). The concept of the DF allows using different sensor setups to monitor the same situation(s). In other words, it is parallel to interpreting the given evidence (e. g., sensor data) for finding the most likely explanation, which created the DF. As such, we declare these logical “inference” sentences in a human-readable form of reasoning that incorporates commonsense logic.
Due to the nature of environmental situations, the interpretation (i. e., evaluation) of such sentences depends on the full context. For example, the same sentence in Table 2 might not apply if the weather outside is warmer than the sensor’s environment. In this case, the temperature may not decrease but stay the same or even increase. Therefore, one will never evaluate the according to sentence to true. Instead, a fallback to another sensor is needed. Nevertheless, the approach defends the redundancy, or alternatives, in detecting the desired situation, considering that we usually use direct means for sensing (e. g., use a contact sensor to detect if the door is open).
The lack of sensors to capture the whole DF of activity leads to an incomplete “view of the world”. The question thereby is, which physical effects are of specific relevance for interpreting an event and omitted. These conditions may vary enormously between different events, e. g., a person speaking or the sun rising both have other effects on the environment and thus (to a degree) require various sensors for interpretation, but also both could be observed using additional information: sound, visual, temperature, time etc.
Concerning the employed method of MLNs, there is an issue using predicates with free variables in the body of a rule; during the grounding phase, it creates a disjunction of the cartesian grounded conjunction of the formulas, translating these variables to existentially quantified leading to a possible combinatorial explosion. We consider any additional constraint in a domain-dependent rule should contain as variables only the time t and the location r. Any knowledge engineer should follow this and remove any existentially quantified variables, using the technique of skolemisation [Broeck et al., 2013], overcoming this limitation for the solution’s scalability.
Finally, the observed data patterns may also result from multiple overlapping activities challenging to separate, such as speaking in traffic, leading to uncertainty about the interpretation. As future work, we want to overcome the limitations of MLN concerning the free variables in the rules and concentrate on a dynamic ecosystem that realises the proposed work.
This work was partially funded by the Federal Ministry of Education and Research (BMBF) of Germany under Grant No. 01IS18072.