Fault isolation is the art of localizing faults in a process, given observations from it. To do this, a model describing the relation between faults and observations is needed. In this paper we focus on learning such models both from training data and from prior knowledge. There are several challenges in learning fault isolators. The number of data, as well as the available computing resources, are often limited and there may be previously unobserved fault patterns. To meet these challenges we take on a Bayesian approach. We compare five different methods for learning in fault isolation, and evaluate their performance on a real fault isolation problem; the diagnosis of an automotive engine.
We consider the problem of fault isolation, i.e. the problem of localizing faults that are present in a process given observations from this process. To do this, a model of the relations between observations and faults is needed. In the current work we investigate and compare different methods for learning from training data and prior knowledge.
We are motivated by the problem of fault isolation in
an automotive engine, and the learning methods are
evaluated using experimental training data and
evaluation data from real driving situations. In engine fault
isolation there may be several hundreds of faults and
observations. There will be fault patterns, i.e.
cooccuring faults, from which there are no training data.
Furthermore, training data is typically experimental
and obtained by implementing faults, running the
process, and collecting observations. On the other hand,
there is often engineering knowledge available about
the process. The engineering knowledge can for
example be used to determine the structure of dependencies
between faults and observations. This kind of
knowledge is often the only basis in previous algorithms for
fault isolation [
Due to the fact that there are previously unobserved fault patterns in training data, frequentist and purely data-based methods are bound to fail. To meet these challenges we use a Bayesian approach to learning in fault isolation. We consider five different methods of learning a model from training data, which are all previously present in the literature in different forms. We taylor these methods to incorporate the available background information. The methods we consider are Direct Inference (DI), Logistic Regression (LogR), Linear Regression (LinR), Naive Bayes (NB) and general Bayesian Networks (BN).
The main contributions of the current work are the
investigation of Bayesian learning methods and
regression models for fault isolation by comparing the five
methods mentioned above, the application and
evaluation of the methods on real-world data, and the
combination of data-driven learning and prior knowledge
within these methods. In order to do this
investigation, we first discuss the characteristics of the fault
isolation problem in terms of probability theory, and
performance measures that are meaningful for fault
isolation. Consecutively we show how the five
methods can be adopted to the isolation problem. We apply
them to the task of fault isolation in an automotive
diesel engine. Finally, we compare the five methods,
and discuss their advantages and drawbacks.
Bayesian methods for fault isolation are previously
studied in literature. In these previous works it is
generally assumed that the model is given [
The paper is structured as follows. We introduce notation, and formulate the diagnosis problem in Section 2. Therein we also define relevant performance measures. In Section 3 we briefly describe the five methods used, and in particular how they are applied to the diagnosis problem, before we perform the evaluating experiments and compare the results obtained in Section 4. Finally, in Section 5 we conclude the paper by summarizing our results and discussing future work directions. 2
PROBLEM FORMULATION Before going into the details of each of the learning methods we introduce some notation, and discuss the characteristics of the fault isolation problem. Then we carefully state the problem at hand and define performance measures. 2.1
The fault isolation problem can be formulated as a prediction problem, where the task is to determine the fault(s) present in a system, given a set of observations from the system. Let the faults be represented by the binary variables Y = (Y1, . . . , YK ), and let the observations from the system be represented by the variables X = (X1, . . . , XL), where each Xl is discrete or continuous. Generally, we use upper case letters to denote variables, and lower case letters to denote their values. Boldface letters denote vectors. We write p(X = x) (or simply p(x)) to denote either probabilities or probability distributions both in the continuous and in the discrete case. The meaning will be clear from the context.
We are given a set of training data D, consisting of samples (yn, xn), n = 1, . . . , ND, pairs of fault and observation variables. The training data is collected by implementing faults and then collecting observations, meaning that training data is experimental. To evaluate the system we use a set E consisting of NE samples. The evaluation data is collected by running the system, meaning that it is observational. Furthermore, we assume that the fault isolation algorithm is
X1 X2 X3 X4 X5 X6 X7 X8 triggered by a fault detector telling us there must be at least one fault present in the process.
The structure of dependencies between the faults and observations has three basic properties, illustrated in the example Bayesian network of Figure 1.
The first property is that faults assumed to be a priori independent, i.e. that
K M
p(y) = Y p(yk|y1, . . . , yk−1) ≈ Y p(yk),
k=1 k=1
(1)
meaning that faults cannot cause other faults to occur.
Although not necessary for the methods in the current
work, this is a standard assumption in many fault
isolation algorithms [
Second, faults may causally affect one or several of
the observation variables introducing dependencies
between faults and variables. A dependency between
fault variable Yk and observation variable Xl means
that the fault may be visible in the observation.
The third property is that an observation variable Xl
may be dependent on other observation variables.
Dependencies between observation variables may arise
due to several reasons. For example they can be caused
by unobserved factors, such as humidity, driver
behavior, and operation point of the process. These
unobserved factors could be modeled using hidden nodes,
but since they are numerous and unknown they are
here simply modeled with dependencies between
observation variables. This is more carefully discussed in
[
We take a Bayesian view point on fault isolation. The objective is to find the probability for each fault to be present given the current observation, the training data, and the prior knowledge I, i.e. to compute the probabilities p(yk|x, D, I), k = 1, . . . , K. The probability for each fault can be found by marginalizing over y−k = (y1, . . . , yk−1, yk+1, . . . , yK ), p(yk|x, D, I) = X p(y−k, yk|x, D, I).
y−k (2) Note that (y−k, yk) = y, and (2) means that we seek the conditional distribution p(y|x, D, I). To simplify the notation we will omit the background information I in the equations.
Computing the conditional distribution p(y|x, D) is generally difficult. To approximate it we need a model M and a method for determining the parameters of the model. To evaluate the different models to be used in Bayesian fault isolation, we use two performance measures: logscore and percentage of correct classification. All the methods considered in this paper – with the exception of DI – build separate models for each fault and thus assume independence among these. A priori this corresponds to approximation (1). However, when we build separate models for each fault, we also make a stronger assumption, namely that the faults remain independent given the observations,
K K p(y|x) = Y p(yk|x, y1, . . . , yk−1) ≈ Y p(yk|x) (5) k=1 k=1 The log-loss is a commonly used measure [1], and given by This approximation is (after applying Bayes’ rule and canceling terms) equivalent to μ(E , M) = 1 NE
X log p(yj |xj , M),
NE j=1
(3)
The scoring function μ measures two important
properties of the fault isolation system; both the ability to
assign large probability mass to faults that are present,
and also the ability to assign small probability mass
to faults that are not present. Furthermore, the
logscore is a proper score. A proper score has the
characteristic that it is maximized when the learned
probability distribution corresponds to the empirically
observed probabilities. In the fault isolation problem the
conditional probabilities for faults is often combined
with decision theoretic methods for troubleshooting
[
MODELLING APPROACHES In this section we briefly present the inference methods used to tackle the fault isolation problem. We carefully state all assumptions made, and describe the adjustments of each method to apply it to the diagnosis problem. However, we begin by describing two assumptions that need to be made for all methods except DI.
K K
Y p(x|yk) ≈ Y p(x|y1, . . . , yk),
k=1 k=1
meaning that the observation x is dependent on each
fault yk, but this dependency is assumed to be
independent of all other faults yk′ , k′ 6= k. In other words,
we assume no “explaining away” [
Assumption (5) is primarily made for technical reasons, in order to be able to build separate models for each fault. But often it is also the case (as in the application of Section 4) that there is training data only from single faults. This means we do not have any training data telling us about the joint effect of multiple faults.
Remember that it is known that there is at least one fault present when the fault isolator is employed, see Section 2.1. Therefore, instead of computing p(y|x), we search p(y|x, X yk > 0) = p(y|x)(1 − p(y ≡ 0|x)).
k Unfortunately p(y|x, X yk > 0) 6= Y p(yk|x, X yk > 0),
k k k a fact which recouples the single-fault models introduced in (5). This fact is ignored during the learning phase and the single-fault models are trained individually. We then apply (7) in the evaluation phase. 3.2
Several previous fault isolation algorithms rely on prior
knowledge about which observations may be affected
(6)
(7)
(8)
by each fault [
The direct inference method is developed for sparse sets of training data, particularly when there is only training data from a subset of the fault patterns to isolate. 3.3
When using Bayesian networks for prediction, we search the joint distribution p(y, x|θ), where θ are parameters describing the conditional probability distributions in the network. From the joint distribution, the conditional distribution for y can be computed. We consider two types of Bayesian networks: Naive Bayes and general Bayesian Networks. 3.3.1
The Naive Bayes classifier assumes that the
observations are independent given the fault. Naive Bayes is
is one of the standard methods for Bayesian prediction
and often performs surprisingly well [
S(V ) = 1 ND
X log P (ykn|xn, V, D \ {(yn, xn)}, α), ND n=1 (10) where V ⊂ X is the variable set under consideration and α is the Dirichlet hyper-parameter for the NBmodel. 3.3.2
Since it is known that the faults causally precede the
observations, and since the observations are known to
be dependent given the faults, a natural step forward
from the Naive Bayes structure is a Bayesian network.
In the network we constrain the fault to be a root
node, but otherwise leave the structure unconstrained.
One such network was learned for each fault using a
BDe score (with an equivalent sample size parameter
of 1.0). For small systems (< 30 variables) learning can
be performed using the exact algorithm in [
Fault isolation is a discriminative task, where we are to predict the fault vector y given the observations x, i.e. estimate the conditional likelihood p(y|x, θ) =
p(y, x|θ) Py p(y, x|θ) .
(11)
It is well known [
The most straight-forward regression method is linear regression, where each fault variable is assumed to be a linear combination of the observations plus a gaussian noise term, yk = wkT x + wk0 + ǫk, ǫ ∼ N (0, σ).
Here wk, wk0, and σ are parameters to be determined. This gives the probability distribution p(yk|x) =
exp(− c′(α, β) − 2 log(exp(α) + exp(−α))
ND w∗ = arg min − X log p(ykn|xn, w) w n=1
ND = arg min − X(wkT xn + wk0 − ykn)2.
w n=1 When the parameters w∗ are known, the parameter σ can also be computed. The normalization constant in (12) is given by Z = exp(−((w∗)kT x+ wk∗0 − 1)2/2σ2)+ exp(−((w∗)kT x + wk∗0 − 0)2/2σ2). 3.4.2
Learning parameters to maximize (11) for a Bayes Net
B is known to be equivalent to logistic regression under
the condition that no child of the class can be a
“bastard”, a common child of two variables that are not
interconnected directly. More formal definition and
proofs can be found in [
To start with, for each fault we learn a logistic regression model corresponding to a discriminative Naive Bayes classifier 1.
We name the parameters of the logistic regression model α and β such that the conditional likelihood is defined as
p(yk = 1|x, α, β) := where exp s(x, α, β) exp s(x, α, β) + exp −s(x, α, β) (13)
L s(x, α, β) := α + X xlβl.
l=1 We also include a smoothing term c(α, β) in our objective function which takes the place of a prior in the corresponding NB classifier. To unify its role for different observations, we first normalize our data by shifting and scaling such that for l = 1, . . . , L X xln = 0 and n max |xln| = 1 n Starting out from the uniform prior, we pretend to have seen one vector of each class at node Yk and two vectors of each class with extreme values ±1 at each node Xl, with all other values zero (∼unobserved).
1possible other choices include tree-augmented Naive
Bayes (TAN) [
L − 4 X log(exp(βl) + exp(−βl)). (16)
l=1 However, we found this smoothing term problematic, since it is flat near zero. Therefore, we never get any parameters exactly zero. But in logistic regression many small parameters can make a difference, while they may be weakly supported. We choose to replace log(exp(x) + exp(−x)) by |x|. This is a good approximation away from zero, but forces unsupported parameters to zero, implicitly performing attribute selection. For fault Yk we search parameters as to maximize log p(yk|x, α, β) + c(α, β)
ND L = X log p(ykn|xn, α, β) − 2|α| − 4 X |βl|. (17) n=1 l=1 We do this by simple line search, one parameter at a time2.
Finally, we try a variant of this algorithm which weights the training vectors. We have prior knowledge about the probabilities p(yk) with which to expect some fault yk in the real-world setting or, in this case, the evaluation set. These probabilities differ from the relative frequencies observed in the training set. The idea is to weight the training vectors in the objective as to focus the optimization on areas of the data space more likely to be seen later on. The corresponding objective for fault Yk becomes
ND X log wkp(yin|xn, α, β) + c(α, β) n=1 (18) where the weight wk is the prior p(yk) divided by the observed relative frequency #{n : ykn = yk}/ND. 4
To evaluate the different methods learning fault isolation models, we apply them to the diagnosis of the gas flow in a 6-cylinder diesel engine in a Scania truck. In automotive engines, sensor faults are one of the most common faults, and here we consider five faults that may appear in different sensors. The faults are listed together with their prior probabilities in Table 2.
2There are much faster optimization techniques, some
of which are compared in [
For the gas flow of the diesel engine there is physical
model from which a set of 29 diagnostic tests are
automatically generated using structural analysis [
For training and evaluation data we use measurements from real operation of the truck, with faults implemented. The training data consists of 100 samples each from the five single faults. Evaluation data consists of data from the five single faults, but also of data from two multiple faults y1&y2, and y1&y4. Evaluation data is observational, and consists of 1000 samples, distributed roughly according to the prior probabilities in Table 2.
The data we consider is originally continuous, but all
except the regression algorithms take in discrete data.
The data is discretized in two different ways: binary,
with thresholds set such that all fault free data is
known to be contained in the same bin; and discretized
using k-means clustering [
In Table 3 the log-score (μ) and percentage of correct classification (ν) are presented for the different methods. In addition we report the number of parameters used by each predictor. This is relevant, since for on-board fault isolation the computing and storage capacity is often limited. For comparison we also report the default which is obtained by simply using the prior probabilities given in Table 2. We observe, that among the four best methods in Table 3 three are discriminative and learn the conditional distribution instead of the joint distribution. Furthermore, LogR with training sample weighting performs best on this data in log-score sense, while using a small number of parameters. Surprisingly the weighting trick has made quite a difference and LogR without weights it is outperformed by NB-disc. NB performs better when it is fed with discretized observations instead of binary, while for BN the effect is reversed. Clearly the discretized data contain more information, but it seems that in more complex Bayes Nets the conditional probability tables easily grow too large. In DI good results are obtained by exploiting prior knowledge in terms of that some faults never cause an observation to pass certain thresholds.
Measured by the ν-score the relative differences between the methods become smaller. We observe that this score favors the regression models and the Bayesian methods using binary data. The reason for the good performance of the methods using binary data is the particular way of thresholding the data such that all fault free samples are contained in the same bin.
Table 4 compares the log-scores of the predictions given for the single faults by DI and LogR+weights. Note that because of inequality (8) the columns do not sum to the corresponding entries in Table 3. Not surprisingly, both methods (as all others) have most trouble with faults y1, y2 and y4, the ones appearing simultaneously in evaluation data, but not in training data. This gives evidence for explaining away being important in this problem. Figure 2, in which the probabilities for each fault using LogR + weights are plotted, shows this in more detail. In the Figure we have ordered the evaluation data such that the rightmost samples have multiple faults, visualizing that the double faults are most difficult to predict.
nx)1 | 10.5 y ( p00 nx)1 2|0.5 y ( p00 nx)1 3|0.5 y ( p00 nx)1 4|0.5 y ( p00 nx)1 5|0.5 y ( p00
We have considered the problem of fault isolation in an automotive diesel engine. We have discussed the special characteristics of this problem. There is experimental training data available which is distributed differently from what we expect to see in the real-world setting. In particular, evaluation data consists partly of previously unseen fault patterns. In addition there is prior knowledge available about which faults may affect each observation, and also the knowledge that at least one fault is present.
We have studied different Bayesian and regression approaches to combine this by nature heterogeneous information into probability distributions for the faults conditioned on given observations. We have compared the performance of the methods using real-world data, and have found that the discriminative logistic regression method to perform best. Among the best methods we have also found the naive Bayes classifier and the direct inference method.
One of the clearest implications of this work is that all methods have difficulties with handling unobserved fault patterns. Unfortunately, unobserved patterns are common in fault isolation, so this problem should be tackled in future work. All the methods used, except direct inference, ignore explaining away. However, this explaining away effect can possibly be helpful when diagnosing unseen patterns. Furthermore, it is crucial to include background information in the learning phase whenever it is available.
In our work to come we will investigate models capable of both explaining away and taking prior knowledge into account, while providing an efficient inference procedure, as on-board computers offer very limited resources. We expect further improvement of performance is possible.
[1] Christopher M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1995. [2] Johan de Kleer and Brian C. Williams. Diagnosis with Behavioral Modes. In Readings in Modelbased Diagnosis, pages 124–130, San Francisco, CA, USA, 1992. Morgan Kaufmann Publishers Inc. [3] Luc Devroye, Laszlo Gy¨orfi, and Gabor Lugosi.
A Probabilistic Theory of Pattern Recognition.
Springer, New York, 1996. [4] Henrik Einarsson and Gustav Arrhenius. Automatic design of diagnosis systems using consistency based residuals. Master’s thesis, Uppsala University, 2004. [5] Russel Greiner and Wei Zhou. Structural Extension to Logistic Regression: Discriminative Parameter Learning of Belief Net Classifiers. In 13th international conference on uncertainty in artificial intelligence, 2002.