The establishment on the market, witnessed in the very last years, of home assistance devices (e.g., Google Home, Amazon Alexa), provided new incentives to the development of sensors and actuators based on wireless protocols (e.g., Zigbee) to be installed in private houses. These devices have the advantage, with respect to previous "domotic" technologies, of not requiring complex wiring and installation work, thus attracting the curiosity of end users who, with a limited expense, can create their own smart houses.
If, on the one hand, data coming from these devices are currently mainly employed by defining manual automation rules, on the other hand, there is now the concrete possibility of applying automatic techniques from the intelligent environment research community outside the controlled world of research laboratories.
Research work focused in particular on the following automated tasks: activity recognition, anomaly detection and activity prediction. As discussed in [1], prediction can be applied at the sensor level or at the activity level. The two types of prediction have different purposes. Activity level prediction is used to predict the next activity/ies the user is going to perform. Activity level prediction is usually intended for prompting expected activities to users (e.g., elderly) if they did not started at the right time [2,3]. A survey of sensor and activity prediction is provided in [4]. Activity prediction is usually based on probabilistic models [2,3], especially based on Markov chains, or on rules learned through classical data mining method such as rule mining [5] and clustering [6].
In this paper, we tackle the problem of Activity Daily Living (ADL) prediction in smart houses given past ADLs and sensor measurements, as discussed in our papers [7,8]. We outline a prediction model for daily activities based on (Marked) Temporal Point Processes -MTPPs, that is a probabilistic method modeling random processes. In recent years, there has been an increasing number of applications to MTPPs [9,10] in various areas including health-care analysis, finance, modeling earthquakes and aftershocks, etc. In particular, we applied MTPPs to a freely available dataset in the smart house community provided by the CASAS project. Achieved performance are evaluated against the state-of-the-art probabilistic algorithm defined in [2].
This discussion paper is organized as follows. Section 2 introduces the theoretical background behind MTPPs and their neural network implementations. Section 3 describes our approach to apply MTPPs to activities of daily living. Section 4 reports an initial evaluation of our approach based on MTPPs against the state-of-the-art technique proposed in [2]. Finally, Section 5 concludes the paper with final considerations and future works.
(Marked) Temporal Point Process (shortly MTPP) is a powerful mathematical tool for modeling the latent mechanisms that govern the occurrence of random events observed over time. In particular, these models can be used as predictive models, that are capable of specifying the timing of future events, based on the history of the past.
Formally, a marked temporal point process is a stochastic process modelling a list of discrete events 𝑒 𝑗 = (𝑡 𝑗 , 𝑎 𝑗 ), with 𝑡 𝑗 ∈ ℛ + , 𝑗 ∈ 𝒵 + and 𝑎 𝑗 ∈ 𝒜 where the domain of 𝒜 is application dependent [10]. Note that, with respect to the basic formulation of a TPP, the concept of event point is extended with a marked 𝑎 𝑗 , i.e. additional information associated with an event that can be of separate interest (e.g. in the prediction of an earthquake we are interested in its position and magnitude) or may simply be included to make a more realistic model of the event times (as in our case). Let the history 𝐻 𝑡 be the list of events (time and marker pairs) up to the time 𝑡, we can explicitly specify the conditional density function that the next event will occur during the interval [𝑡, 𝑡 + d𝑡] with mark 𝑎 by 𝑓 * (𝑡, 𝑎) = 𝑓 (𝑡, 𝑎|𝐻 𝑡 ) where the notation * from [11] means that this density is conditional on the past (right up to but not including the present) rather than writing explicitly that the function depends on the history. From the density functions 𝑓 ((𝑡 1 , 𝑎 1 ), . . . , (𝑡 𝑗 , 𝑎 𝑗 )) specifying the distributions of all events time and marker pairs, one by one, starting in the past, thus the distribution of all events is given by the joint density:
A common way to model temporal point process with marker is by the conditional intensity function that could be specified by the conditional density 𝑓 (𝑡, 𝑎|𝐻 𝑡𝑛 ) and its corresponding cumulative distribution function 𝐹 (𝑡, 𝑎|𝐻 𝑡𝑛 ) for any 𝑡 > 𝑡 𝑛 . Formally, the conditional intensity function is defined by:
The conditional intensity function can be interpreted heuristically by considering an infinitesimal interval around 𝑡, say [𝑡, 𝑡 + d𝑡] and the number of points falling in it 𝑁 : 𝜆 * (𝑡, 𝑎)d𝑡d𝑎 = 𝐸[𝑁 (d𝑡 × d𝑎)|𝐻 𝑡 ] that is, the mean number of points in a small time interval 𝑑𝑡 with the mark in a small interval d𝑡. Based on the application domains, different forms of the conditional intensity function has been proposed aiming at capturing the phenomena of interests [9]. For example, the specific parametric could be as in Poisson process [12] when keep 𝜆 fixed over time, and to be independent of the history 𝐻 𝑡 ; as non homogeneous Poisson process when keep 𝜆(𝑡) as a function of only time and not the history of events you get; as Hawkes process [13], where exponential formulation of the conditional intensity simulate the fact that a point increases the chance of getting other points immediately after, and Self-correcting process [14] where as opposed to Hawkes process the chance of new points decreases immediately after a point has appeared. Beside the specific form of the conditional intensity, as those introduced above, all these different parametrizations imply specific assumptions about the functional forms of the generative processes, which may or may not reflect the reality and definitely would be unknown in the most common cases.
Differently to the approaches where the conditional intensity has a specified parametric form, recent research on MTPP emphasizes more on the neural-network-based point process models as a promising approach that can better capture the dynamics of a complex system. In particular, two recent approaches, namely RMTPP [15] and ERPP [16], have demonstrated that Recurrent Neural Network (RNN) can be used to model and automatically learn the conditional intensity function (without any particular prior assumptions) by achieving better prediction results than simple TPP models (e.g. Point, Hawkes or Self-correcting processes).
RMTPP [15] is the seminal work that initially proposed to use RNN in the MTPP framework. Specifically, the event history is embedded into a vector ℎ 𝑗 , and then used to define the conditional intensity as 𝜆 * (𝑡) = exp(𝑤 𝑡 (𝑡 − 𝑡 𝑗 ) + 𝑣 𝑇 ℎ 𝑗 + 𝑏 𝑡 ), where 𝑣 𝑇 is a column vector, 𝑤 𝑡 , 𝑏 𝑡 are scalars learned during the train of the network and the exponential function is used as a non-linear transformation and guarantees that the intensity is positive. Formulation of 𝜆 summarizes three contributions: (1) the first term emphasizes the influence of the current event j; (2) the second term 𝑣 𝑇 ℎ 𝑗 represents the accumulative influence from the marker and the timing information of the past events; (3) the last term gives a base intensity level for the occurrence of the next event.
By following the path traced by [15], ERPP proposed a model that automatically learns the conditional intensity function of a point process by synergically using two RNNs, one RNN with asynchronous events as input and another RNN with time series as input. The underlying rationale is that time series are more suitable to carry the synchronously and regularly updated, while the event sequence can compactly catch event driven, more abrupt information, which can affect the conditional intensity function over longer period of time. Adopting a twin RNN structure permits to deal with timeseries and event via separate RNN, and can be suitable to model point process in which the dynamics of these two source of data can be rather varying [16].
We use (marked) Temporal Point Process (MTPP) as a mathematical framework for the modeling and learning of human's behaviors, associated with their side habit information. In fact, human activities are captured by sensors and recorded as many discrete events in continuous time.
Definition 3.1. Let the input be a set of sequence of activities performed in the time with its timestamp, i.e. 𝑆 = {𝑒 1 = (𝑡 1 , 𝑎 1 ), . . . , 𝑒 𝑚 = (𝑡 𝑚 , 𝑎 𝑚 )} is a sequence of pairs marked by a kind of activity and its timestamp. Given two events 𝑒 𝑖 and 𝑒 𝑗 with 𝑖 > 𝑗, we have that 𝑡 𝑖 ≥ 𝑡 𝑗 . Considering the j-th observation, our goal is to predict the next observation (j+1), i.e. when the next activity will happen and what type of activity it will be, given the past sequence of observations.
To build the model prediction based on the RMTPP's architecture, we first embed the sequence of daily activities (see Definition 3.1) into a latent space. Then, the embedded vector and the temporal features are fed into the recurrent layer. The recurrent layer learns a representation that summaries the nonlinear dependency over the previous events. Based on the learned representation ℎ 𝑗 , it outputs the prediction for the next marker 𝑎 𝑗+1 and timing 𝑡 𝑗+1 to calculate the respective loss functions. As evidenced in [16], based on the hidden unit of RNN, we are able to learn a unified representation of the dependency over the history. In consequence, the direct formulation 2 of the conditional intensity function 𝜆 * (𝑡 𝑗 + 1) captures both of the information from past event timings and event markers. On the other hand, since the prediction for the marker also depends nonlinearly on the past timing information, this may improve the performance of the classification task as well when both of these two information are correlated with each other.
By adopting the ERPP's architecture, the model differs substantially, since time series and event sequence are predicted separately. Specifically, we fed the different timestamps into a LSTM and the sequence of activities is embedded and fed into a separate LSTM.
The experimental analysis has been conducted on the ARUBA dataset provided by the CASAS project 1 . The dataset has been acquired by monitoring the activities performed by an elderly during two years of daily life in the house depicted in Figure 1.
Furnitures and walls are depicted using black lines. The environment contains different category of sensors including passive infrared sensors -PIRs (denoted with ellipses and code MXXX), door-attached switch sensors (denoted with empty rectangles and code DXXX), and temperature sensors (denoted with code DXXX). In particular, the house is equipped with 31 passive infrared sensors (PIR) installed on the ceiling in correspondence of sensitive places for human activities (e.g., M002 and M003 mounted in correspondence of the bed, M009 and M010 mounted in correspondence of the sofa) and pointing at the floor. As soon as the elderly walks in the area covered by a sensor, he/she makes it trigger to the ON value. Two kinds of PIR sensors are available with two different detection area sizes: small cone sensors (denoted with small ellipses) and large cone sensors (denoted with large ellipses). The former have the goal of detecting the elderly interacting with the specific furnitures in correspondence of the sensors, whereas the latter have the goal of detecting him/her generically entering a room or an area (e.g., M019 detects the person entering the kitchen).
The dataset is labeled with the activity the elderly is performing when each sensor measurement onsets. The dataset reports 12 different activities. The dataset has been pre-processed in order to remove sensor measurements other than those coming from the PIR sensors and to remove OFF measurements for PIR sensors, which do not contain any information (PIR sensors trigger to OFF after a predetermined amount of time). The goal of the evaluation is to measure the ability of the proposed technique to predict the next activity and when this activity will be performed. Achieved performance have been compared with the state-of-the-art technique proposed in [2]. The Aruba dataset has been split in a training set (70% of the measurements) and a validation set (30% of the measurements). We evaluated the time prediction using the Mean Absolute Error (MAE), Precision and Recall and F1-score.
The performance of our solution have been compared with those obtained with the technique proposed by the CASAS research group in [2]. The source code of the algorithm is freely available on the project website, even though some re-coding was needed in order to compute performance envisioned in our work. In particular, performance in [2] are not computed considering all the different activities together, but with respect to a specific activity. Evaluation in the original paper [2] was conducted with a different, less recent and smaller dataset from the very same research group.
The technique proposed in [2] is based on two components. The current activity -RA, is recognized by applying Support Vector Machines -SVM, to sensor measurements. At this point the likelihood of a predicted activity -PA, is obtained by taking into account (i) the likelihood that activity PA occurs after activity RA, (ii) the confidence in the distribution of the occurrence times of PA relative to RA, and (iii) the mean delay between RA and PA. In addition pattern mining over sensor measurements is employed.
Performance has been computed by asking to the algorithm at each step, which is the next activity and when it is going to happen. On a total of 238 748 decisions, the next activity was correctly predicted in the 56% of the cases, which is fairly good performance considered that the number of different activities is 12. The precision obtained though was 0.19, the recall 0.18 and F1-score 0.18. The MAE with respect to the onset of a specific activity is instead of 182.1 minutes.
In order to evaluate the effectiveness of our approach, we performed the same experiments described in the section 4.1 using both RMTPP and ERPP. As introduced in section 2.2, RMTPP is a way to connect recurrent neural networks and point processes, giving us the opportunity to predict not only the marker but also the timing of the future events. It is important to notice that no prior knowledge about the hidden functional forms of the latent temporal dynamics is needed. The reference paper for the implementation of RMTPP is [15]. Using the same settings described in 4.1, we obtained the following results using RMTPP: the MAE is 4.672 with a precision of 0.648, recall of 0.907 and F1-score equals to 0.756. In order to evaluate ERPP, we used as reference the paper [16], using a single layer LSTM of size 32 with Sigmoid gate activations and tanh activation for hidden representation. The results are better than RMTPP in terms of MAE. We obtained the following values: MAE: 0.008, precision: 0.660, recall: 0.905 and F1-score: 0.763. Notably, MAE is an indication of the average deviation of the predicted values w.r.t the corresponding observed values, suggesting us that ERPP outperform RMTPP in long term model prediction.
Table 1 recaps the results obtained during our performance evaluation. From these preliminary tests, the proposed approaches clearly outperform the state-of-the-art algorithm proposed in [2]. The algorithm presented in [2] is essentially based on classical pattern mining and machine learning. In particular, starting from the activity currently recognized as ongoing, based on an SVM classifier, patterns are extracted on a statistical basis taking into account time relationships between activities and the value of specific sensor measurements as indicators of future activities. RMTPP [15] and ERPP [16] are instead based on recurrent neural networks, which proved themselves very effective for many different learning tasks.
In this paper, we have proposed the application of Marked Temporal Point Process for human action analysis, which has not been explored before. We have adopted ERPP and RMTPP as methods describing the generative mechanism of event sequences. Empirical results on challenging daily life datasets demonstrate the efficacy of the proposed methods and consistent performance gain for event prediction and causality analysis w.r.t. state-of-the-art approach. In future work, we plan to enrich our test set with more references and to extend this work with a novel architecture incorporating external features in order to improve its temporal dependence modeling and to better embed the dynamics that govern the generation process of the events. We will extend validation also to other techniques.