We have been working with dynamic data from an oil production facility in the North sea, where unstable situations should be identified as soon as possible. Monitoring in such a complex domain is a challenging task. Not only is such a domain typically volatile and following non-linear dynamics, but sensor input to the monitoring system can also often be high dimensional, making it difficult to model and classify the domain's states. Dynamic latent classification models are dynamic Bayesian networks capable of effective and efficient modeling and classification. An approximate inference algorithm utilizing Gaussian collapse has been tailormade for this family of models, but the approximation's properties have not been fully explored. In this paper we compare alternatives approximate inference methods for the dynamic latent classification model, in particular focusing on traditional sampling techniques. We show that the approximate scheme based on Gaussian collapse is computationally more efficient than sampling, while offering comparable accuracy results.
In the oil drilling, monitor the complex process and
identify the current system state is actually very
difficult. Monitoring the complex process often involves
keeping an eye on hundreds or thousands of sensors to
determine whether or not the process is stable. We
report results on an oil production facility in the North
sea, where unstable situations should be identified as
soon as possible [
Motivated by this problem setting, a generative
model called dynamic latent classification models (dLCM)
[
Dynamic Latent classification models [
Ct−1
C t
This model is described by a prior distribution over
the class variable P (c0), a conditional observation
distribution P (yt|ct), and transition probabilities for the
class variable P (ct|ct−1); we assume that the
model is stationary, i.e., P (yt|ct) = P (yt−1|ct−1) and
P (ct|ct−1) = P (ct+1|ct), for all t. For the continuous
observation vector, the conditional distribution may
be specified by a class-conditional multivariate
Gaussian distribution with mean μct and covariance matrix
Σct , i.e., Y |{Ct = ct} ∼ N (μct , Σct )
In a standard HMM, it assumes that the class
variable and attributes at different time points are
independent given the class variables at the current time,
which is violated in many real world setting. In our
oil drilling data, there is also a strong correlation
between attributes given the class [
Following [
Ct−1 The latent variable Zt is assigned a multivariate Gaussian distribution conditional on the class variable and the attributes Y t are assumed to be linear multivariate Gaussian distributions conditional on the latent variables:
Zt|{Ct = ct} ∼ N (μct , Σct ),
Y t|{Zt = zt} ∼ N (Lzt + Φ, Θ), where Σct and Θ are diagonal covariance matrix and L is the transition matrix, Φ is the offset from the latent space to attribute space; note that the stationarity assumption encoded in the model.
In this model, the latent variables capture the dependence between the attributes. They are conditionally independent given the class but marginally dependent. Furthermore, the same mapping, L, from the latent space to the attribute space is used for all classes, and hence, the relation between the class and the attributes is conveyed by the latent variables only.
At this step, the temporal dynamics of the model is
assumed to be only captured at the class level. When the
state specification of the class variable is coarse, then
this assumption will rarely hold. This assumption does
not hold in our oil drilling data, as the conditional
correlation of the attribute in successive time slices is
evident [
Y t|{Zt = zt, M t = mt} ∼ N (Lmt zt + Φmt , Θmt ), where 1 ≤ mt ≤ |sp (M )| (|sp (M )| denotes the dimension of variable M space), P (M t = mt|Ct = ct) ≥ 0 and P|mspt(=M1 )| P (M t = mt|Ct = ct) = 1 for all 1 ≤ ct ≤ |sp (C)|, Φmt is the offset from the latent space to attribute space.
The final model is then called dynamic latent
classification model which is shown in Figure 4. The
dynamic latent classification model is shown to be
effective and efficient through the experiment with our oil
drilling data, and the significant improvement is
also demonstrated when comparing dLCM with static
models (such as NB or decision tree) and HMM [
Ct−1 Zt−1 1
Zt−1 2
C t Z t 1
Z t
2
A discrete mixture variable M is further introduced to
the model at each time slice for the purpose of
reducing the computational cost while maintaining the
representational power [
The exact inference for dLCM is intractable. To make dLCM applicable and effective in practice, approximate inference is then proposed. 3.1
Seen from the dLCM in Figure 4, an equivalent
probabilistic model is
p(y1:T , z1:T , m1:T , c1:T ) =
p(y1|z1, m1)p(z1|c1)p(m1|c1)p(c1) ·
T
Y p(yt|zt, mt)p(zt|zt−1, ct)p(mt|ct)p(ct|ct−1).
t=2
In dLCM, exact filtered and smoothed inference is
shown to be intractable (scaling exponentially with
T [
The structure of the proposed dLCM is similar to
the linear dynamical system (LDS) [
During the forward recursion of dLCM, the filtered posterior p(zt, mt, ct|y1:t) is computed with a recursive form. By a simple decomposition, p(zt, mt, ct|y1:t) = p(zt, mt, ct, yt|y1:t−1)/p(yt|y1:t−1) ∝ p(zt, mt, ct, yt|y1:t−1).
Dropping the normalization constant p(yt|y1:t−1),
p(zt, mt, ct|y1:t) is proportional to the new joint
probability p(zt, mt, ct, yt|y1:t−1), where
p(zt, mt, ct, yt|y1:t−1) = p(yt, zt|mt, ct, y1:t−1)·
p(mt|ct, y1:t−1)p(ct|y1:t−1).
(1)
To build the forward recursion, a recursive form for
each of the factors in Equation 1 is required.
Given the filtered results of the previous time-step, the
recursive form for each of the factors are shown to
be feasible [
Similar to the forward pass, the backward pass also relies on a recursion computation of the smoothed posterior p(zt, mt, ct|y1:T ). In detail, p(zt, mt, ct|y1:T ) is computed from its smoothed result of the previous step p(zt+1, mt+1, ct+1|y1:T ), together with some other quantities obtained from forward pass. The first smoothed posterior is p(zT , mT , cT |y1:T ), which can be directly obtained as it is also the last filtered posterior from the forward pass. To compute p(zt, mt, ct|y1:T ), factorize it as p(zt, mt, ct|y1:T ) = = mt+1,ct+1
X
X mt+1,ct+1 p(zt, mt, ct, mt+1, ct+1|y1:T ) p(zt|mt, ct, mt+1, ct+1, y1:T ) · p(mt, ct|mt+1, ct+1, y1:T )p(mt+1, ct+1|y1:T ). Due to the fact zt⊥⊥{yt+1:T , mt+1, ct+1}|{zt+1, mt, ct}, the p(zt|mt, ct, mt+1, ct+1, y1:T ) can be found from that term p(zt|mt, ct, mt+1, ct+1, y1:T ) = Z
p(zt|zt+1, mt, ct, y1:t) · zt+1 p(zt+1|mt, ct, mt+1, ct+1, y1:T )dzt+1.
To complete the backward recursive form, two
essential assumptions are further made in the
backward pass that makes the approximate inference
applicable and effective. The first assumption
is to approximate p(zt+1|mt, ct, mt+1, ct+1, y1:T ) by
p(zt+1|mt+1, ct+1, y1:T ) [
The exact inference is not applicable in practise as
its computation cost is increasing exponentially over
time. The approximate inference is then essential to
dLCM. Gaussian collapse is adopted during building
the recursive form for both forward and backward pass.
At the same time, p(zt+1|mt, ct, mt+1, ct+1, y1:T ) is
also approximated by p(zt+1|mt+1, ct+1) in dLCM. As
the approximations are made within the inference, the
quality of the overall learning and inference for
dLCM is rather sensitive to these approximations. Our
experimental results [
Even though the proposed approximate inference in
dLCM is satisfactory, is there any improvement space
with alternative approximation methods? With this
question in mind, we decide to investigate the
approximation’s properties by incorporating new
approximation method. The traditional sampling techniques
(e.g., [
In general, we will replace Gaussian collapse by sampling in the approximate inference of dLCM, and further investigate the effectiveness and efficiency of this proposal through a comparison experiment between original Gaussian collapse based dLCM and sampling techniques based dLCM. 4 4.1
The sampling is to select a subset of samples from within a population to estimate the characteristics of the original population. There is a commonly known tradeoff in sampling. When less samples are selected from within a population, which means the sampling process takes shorter time, the estimation of the characteristics to the original population is relatively worse. On the other hand, if more samples are selected from within the same population, which of course is much more time consuming, the characteristics of the original population is better estimated. The efficiency (time consuming) and effectiveness (characteristics estimation) are the essential concerns in the sampling techniques. In general, more samples should be selected within the tolerable time, and the better estimation of characteristics of the population can be expected. This feature of traditional sampling techniques makes it attractive to the approximate inference of dLCM, a balance between efficiency and effectiveness is expected to be achieved according to application requirement. Meanwhile sampling can approximate any distribution as long as the sample number is sufficient. Sampling is expected to replace the Gaussian collapse for the approximation in the both forward and backward pass. We introduce particle filtering next, which will further motivate our discussion on the utilizations detail of the sampling in the inference of dLCM. 4.2
Particle filtering (PF) [
The sampling process that we required for the
inference of dCLM is similar to the PF. In the forward pass,
we know that the mixture components of p(zt−1|yt−1)
is increasing exponentially over time in the
exact inference. Instead of a recursive approximation
on p(zt−1|mt−1, ct−1, y1:t−1) in the Gaussian collapse
scheme, an recursive approximation on p(zt−1|y1:t−1)
by sampling is adopted. With the obtained
approximated distribution p(zt−1|y1:t−1) at time slice t−1, N
weighted samples {(stn−1, πnt−1)}nN=1 are selected from
this approximated distribution. These selected
samples are propagated to the next time slice t with a
linear transition dynamics Act . As the discrete class
variable Ct has the size of |sp (C)|, then each of the
selected samples will become |sp (C)| new samples. These
|sp (C)|·N propagated samples are further updated by
the observation yt. The updating rule is the same as
the Kalmar filter updating [
Algorithm 1 Sampling in the forward pass
1: for t = 2 : T do
2: Select N samples from the previous
appxoximated distribution p(zt−1|y1:t−1) to form a
weighted sample set {(stn−1, πnt−1)}nN=1
3: Propagate these selected samples to the next
time slice t by a transition dynamics Act
4: Update the propagated samples by the
observation Y t.
5: Then the updated samples form a new
weighteadn asapmprpolxei mseatti{o(nfnot,fγpt()z}t|n|sy=p1(1C:t))|·|sp(M)|·N , which is
n
6: end for
In the backward pass, with a similar sampling
process as the forward pass, samples are firstly
selected from the approximated distribution p(zt+1|y1:T ).
p(zt+1|y1:T ) is approximated by a weighted sample
set denoted as {(btn+1, ρtn+1)}n=1. Next, the
selected samples are then back-propagated to the previous
time slice t (which is the next step of the backward
pass) with the reverse transition dynamics of Atc. The
back-propagated samples is later updated by the
observation Y t−1 [
In the discussion of this section, the approximate inference of dLCM based on sampling is established by mimicking the particle filtering process both in the forward and backward pass. To investigate the effectiveness and efficiency of sampling based dLCM, a comparison experiments test will be conducted in the next section. 5
In this section, the comparison experiments on simulation data and oil drilling data are conducted and their results are discussed. 5.1
A set of simulation data is firstly generated from dLCM, and we investigate the classification accuracy and time-consumption between Gaussian collapse scheme and sampling scheme. 5.1.1
The simulation data-set are generated from dLCM with parameters that is chosen by a “semi random” process. The model parameters of dLCM have two parts: model structure and model parameters with fixed model structure. For each time slice, the model structure is decided by four factors: the size of class variable C (activity state), the dimension of latent variable space Z, the dimension of attribute space Y and the size of mixture components M . These values are fixed as described in Table 1. After choosing the model structure, its associated model parameters were randomly generated. The above process of choosing model structure and model parameters together is the ”semi random” process. We then generate a data set with this model.
For convenience we call the model structure and its parameters as model parameters in the remaining of the paper. The generated data set and the model parameters are used as true model for the classification test purpose next. A comparison experiment between data |sp (C)| |sp (M )| |sp (Z)| |sp (Y )| set1 2 1 12 6 set2 2 2 18 9
Gaussian collapse and sampling is then conducted. 5.1.2
The comparison experiment is conducted with both Gaussian collapse based and sampling based dLCM. Among sampling based scheme, there are three chosen sample sizes 40, 200, 1000 respectively. The classification results on simulation set1 and set2 are summarized in Table 2. The classification accuracy results scheme are recorded with the average results of ten runs of each scheme, and it shows that Gaussian collapse based dLCM performs better than three sampling based dLM in both set1 and set2. Among sampling based scheme, scheme with larger sample size achieves better classification accuracy in a general sense.
scheme (samples) set1/accuracy set2/accuracy Gaussian collapse 99.60% 99.90% Sampling (40) 96.75% 95.55% Sampling (200) 97.60% 97.95% Sampling (1000) 97.75% 98.40% After investigating the effectiveness of each scheme, we continue to discuss the efficiency. The efficiency of each scheme is evaluated by the average time-cost of ten run that is required to accomplish the classification task, and the time-cost detail is shown in Table 3. In set1, the classification task is accomplished 0.47 second with Gaussian collapse, whereas the sampling scheme with 40 samples cost 2.01 second. The larger sample size in sampling scheme, the more time it costs to accomplish the classification task. Meanwhile it is clear that Gaussian collapse requires much less time to accomplish the classification task.
Compared to sampling based scheme, Gaussian collapse based scheme achieves comparable (slightly better) classification results with much less time on simulation data test. 5.2
Next the same comparison experiment is conducted with the oil drilling data from North sea.
As we mentioned in the introduction section, we will tie the development to the task of activity recognition in this paper. In total, there are 5 drilling acclivities in the dataset used for classification task. These activities are “drilling”, “connection”, “tripping in”, “tripping out” and “other”. The original oil drilling data contains more than 60 variables. Advised by oil drilling domain expert, 9 variables for the classification task here. There are two chosen data set, which contains 80000 and 50000 time slices with all 5 activities presented respectively.
For classification purpose in this paper, we combine
these 5 activities into 2 activities and conduct the
classification test on the combined data set. Three
activities including “drilling”, “connection” and
“other” activities are combined as one activity, and we do
the similar combination for “tripping in” and
“tripping out” activities. The reason behind is that these
combined actives are physically close and may have
quite similar dynamics. This combination also
simplify our experiments with the oil drilling data, while
maintaining the comparison experiment purpose.
Before we can compare the inference of each scheme
on the oil drilling data set, we learn a dLCM with the
learning method proposed in [
With the fixed dLCM, the average (by ten runs) classification accuracy and average time-cost for each scheme are obtained. There are 4 scheme are presented, Gaussian collapsed based scheme and sampling techniques based scheme with 40, 200, 1000 samples respectively. The experiments results are summarized in Table 4.
Among the sampling techniques based scheme, more samples achieves higher classification accuracy. However, with more samples in sampling techniques based scheme, the computation cot for the classification task is much more expensive. It is clearly shown in the table that sampling with 1000 samples requires more than one hour to accomplish the classification task which is around 40 times than that of Gaussian collapse, and they achieve a similar classification accuracy. In general Gaussian collapse still achieves comparable results (slightly better than Sampling), while keeping the computation cost in a rather low standard compared to sampling based scheme. 6
In the approximate inference of the dLCM, the Gaussian collapse is originally adopted as the core of the approximation method. In this paper, alternatively sampling technique is proposed to do the approximation. A process similar to particle filtering, utilizing sampling as the basis, is then incorporated into the approximate inference of the dLCM. We then conduct the comparison experiment results on both simulated data and real oil drilling data. The experimental results from both sets show that the approximate scheme based on Gaussian collapse is computationally more efficient than sampling, while offering comparable accuracy results.
I would like to thank Helge Langseth who helped a lot during the whole process of organizing and writing this paper.