Machine learning such as data based classification is a diagnosis solution useful to monitor complex systems when designing a model is a long and expensive process. When used for process monitoring the processed data are available thanks to sensors. But in many situations it is hard to get an exact measure from these sensors. Indeed measure is done with a lot of noise that can be caused by the environment, a bad use of the sensor or even the conversion from analogic to numerical measure. In this paper we propose a framework based on a fuzzy logic classifier to model the uncertainty on the data by the use of crisp (non fuzzy) or fuzzy intervals. Our objective is to increase the number of good classification results in the presence of noisy data. The classifier is named LAMDA (Learning Algorithm for Multivariate Data Analysis) and can perform machine learning and clustering on different kind of data like numerical values, symbols or interval values.
Data classification is the process of dividing pattern space
using hard, fuzzy or probabilistic partitions into a number
of regions [
Most of the time, data are issued from sensor
measurements and are corrupted by noise. This noise can have
different origins, for example environment disturbances, bad
use of the sensor, hysteresis effect or numerical conversion
and representation of the data. Many domains of
application have to deal with noise problems like medical
diagnosis [
Fuzzy logic is a multi-valued logic framework
introduced by Zadeh [
In the first part of the paper the LAMDA algorithm is shortly presented then in a second time a method to use the algorithm to classify noisy data is introduced. This method is in two parts: the first presents a general solution to model uncertainty on data with crisp intervals based on confidence intervals and the second shows an improvement to model Gaussian noise with fuzzy intervals. In both cases examples of application are introduced to show the improvement of the method compared to the use of the data without transformation. 2
This section presents the principle of the LAMDA algorithm. 2.1
LAMDA is a classification algorithm based on fuzzy logic
created on an original idea of Aguilar [
The algorithm takes as input a sample x made up of N features. The first step is to compute for each feature of x, an adequacy degree to each class Cj , j = 1..J where J is the total number of class. This is obtained by the use of a fuzzy adequacy function. So J vectors of N adequacy degrees are computed, these vectors are called Marginal Adequacy Degree vectors (MAD). At this point, all the features are in a common space. Then the second step is to take all the MADs and aggregate them into one global adequacy degree (GAD) by means of a fuzzy aggregation function. Thus the J MAD vectors (composed of N MADs) become J scalar GADs, the higher the GAD, the better the adequacy to the class. The simplest way to assign the sample x to a class is to keep as result the class with the biggest GAD.
All the process is summarized in Fig. 1.
During the learning step, the algorithm creates prototype data for each class and for each feature. These data are called classe descriptors or prototypes; they can be for example means or variances. We define as Cj,n the class prototype of the n-th feature for the class j.
As previously mentioned the first step of the algorithm
is a comparison between the sample vector x and all the
Cj,n. This operation is performed with membership
functions and gives as result a membership adequacy degree.
Thus M ADj,n is the MAD for the j-th class and the
nth feature. As the framework is based on fuzzy logic, all
memberships are numbers in the [
M ADj,n = f (Cj,n, xn) (1)
The class prototype Cj,n depends on two things: the type of data and the function used. Some functions may require only one data into Cj,n whereas others need a list of parameters.
In the following section, some examples of membership functions are presented.
• Quantitative data:
Many functions are available for this kind of data. For example the Gaussian: or the binomial function: f (xn) = e − (xn − ρj,n)2
2σj2,n f (xn) = ρjx,nn.(1 − ρj,n)1−xn (2) (3)
Where xn is the n-th feature of the sample x, ρj,n is the mean of the n-th feature for the class j and σj,n is the standard deviation of the n-th feature for the class j. • Qualitative data:
Qualitative can take values in a set of modalities. The membership function of qualitative data returns the frequency of modality taken by the feature into the class during the learning phase. We introduce a qualitative variable with K modality {Q1, ..., QK } and the frequency Φjk of the modality Qk for the class j. The membership is described by: f (xn) = (Φj1,n)q1 ∗ ... ∗ (ΦjK,n)qK (4) with qk = 0 if xn 6= Qk qk = 1 if xn = Qk • Intervals:
The membership function for interval data is a function which tests the similarity between two fuzzy intervals. In this case similarity is defined by two components: the distance between the intervals and the surface that these intervals have in common. Indeed the class prototype for crisp interval data is a mean interval. The similarity function is:
S(A, B) = 1 ( RV μA∩B(ξ)dξ 2 RV μA∪B(ξ)dξ + 1 − ∂[A, B] $[V ] ) (5) where μX (x) is the value of x in the fuzzy set X, ∂[A, B] is the distance between intervals A = [a−, a+] and B = [b−, b+]and $[X] is the size of a fuzzy set into a V universe. This is described by:
Z $[X] =
μX (ξ)dξ
V In the case of crisp intervals and in a universe between 0 and 1: (6) S(A, B) = 1 $[A ∩ B]
( 2 $[A ∪ B] + 1 − ∂[A, B]) (7) where $[X] in this case can be replaced by the length of the interval: $[X] = upperbound(X)-lowerbound(X) (8) and distance ∂[A, B] is defined as: ∂[A, B] = max[0, max(a−, b−)−min(a+, b+)] (9) In the case where an interval feature is used the prototype for a class j is given by [ρjn−, ρjn+] where ρjn−, respectively ρn+ represents the mean value of lower j bounds (respectively upper bounds) of all the elements belonging to class j for this feature.
Once the MAD are computed whatever the feature type, it is possible to perform any type of processing as described on Fig. 2 Once all the features are grouped into the membership space the next step of the algorithm is to transform the MAD vectors into a set of single value which depicts the global membership of the sample to a class. These values were introduced in section 2.1 and are called GAD. To perform this transformation a fuzzy aggregation function Ψ is used.
The aggregation function is the following: Ψ(M AD) = α.γ(M AD) + (1 − α).β(M AD) (10) where γ is a fuzzy T-norm and β is a fuzzy T-conorm. α parameter is called exigency indicator. It enables to give more or less significance to the union operation and the intersection operation. Two fuzzy T-norm and T-conorm are currently implemented in the algorithm, the min-max and the probabilistic. For example if min-max is used, (10) becomes: Ψ(M AD) = α.min(M AD)+(1−α).max(M AD) (11) When all GAD are computed they give the membership of the data x to each class. The final result depends on the application but the simplest way to give a result is to class the sample in the class which has the highest GAD. A limit membership can also be fixed: if no GAD is higher than the limit, the sample is defined as unclassifiable. 3 3.1
Every data measurement is performed with noise. In some cases noise has enough bad effect to increase the error of classification. Thus the point is to model the imprecision of the data to decrease the number of bad classifications.
A technique used in several fields of application is the
use of intervals to symbolize data uncertainty [
In a situation where the probability law followed by the noise on a variable is unknown, it may be possible to obtain a confidence interval. It is an interval in which the real value of the measure is present with a certain amount of confidence (for example a confidence interval of 95% is an interval in which the exact value of the measure can be found with a probability of 95%). Introducing xˆ the measured value and l the length of a centered on zero confidence interval based on the measurement error, the interval l ; xˆ + 2l ]. used by the algorithm is calculated: X = [xˆ − 2
The main aim of the transformation is to improve the classification on the transition zones where data is really sensitive to noise and a small change can modify the output of the classifier. The use of intervals to model uncertainty is effective only if the “clean” data is relevant for the classification problem. If it is not the case a better solution is to remove the irrelevant feature. It will in most cases provide better output results. This expresses the fact that if the “clean” data is difficult to classify it is not improved by using confidence intervals. 3.2
A set of data has been created for an application test which can be interpreted as sensors time evolution of a continuous process. This set of data is composed by three quantitative (numerical) features of 101 samples that are shown on the Fig. 3. Three classes are specified and used as targets for the classifier. These classes are chosen arbitrarily to represent different behaviors of a system that could be healthy or failure modes. Nevertheless the classes are built to make all the data relevant for the system monitoring which means the three features do not have a global negative impact on the classification results.
The three features x, y and z are defined by the following time functions:
−t • x = e 2
t • y = 12 · e 4 − 1 • z = tanh(t − 5)
This example is used to measure the improvement in the classification results in the case of all data are noisy. Artificial noise is added by the following: x is the ideal variable without noise and xˆ the noisy variable, xˆ = x + Y with Y a random variable following a uniform distribution on an interval I.
The experiment has been performed with these conditions: α parameter of (10) is set at 0.8 with the [min,max] functions to compute the fuzzy aggregation and the membership function used for quantitative data is the binomial.[min, max] aggregation is chosen because experiments on the algorithm showed that this kind of aggregation provides better results on noisy data that the probabilistic one. A first classification without any noise gives a result of 91% of good classification. Then the experiment is repeated a great many times to avoid statistical mistakes. In this case, the experiment has been run fifty thousand times, xˆ is recomputed at each new run. Results are given on table 1.
Interval for random data Mean success percentage with binomial function Mean success percentage with interval function As it can be seen, this method provides an improvement on the results in the two first cases where noise deteriorates the classification with the quantitative method but when the data is still globally consistent. In these cases, the intervals method gives better results than binomial method 82% of the time. But when noise amplitude is much higher than the data like in the [−2; +2] error interval, the interval method does worse in general than the binomial function.
Most of the time, noise on physical measure follows a Gaussian distribution centered on the real value. Thus it is interesting to model this specific kind of uncertainty. Nevertheless, it is difficult to handle fuzzy intervals with an exact Gaussian shape. That is why we suggest approximating the Gaussian with a triangular fuzzy interval. This interval is described with a lower boundary x− and an upper boundary x+: X = [x−; x+] which leads to a similar description as crisp intervals. So: μX (x−) = 0 and μX (x+) = 0 and μX ( x++2 x− ) = 1 with μX (x) the fuzzy value of x into the fuzzy set X. As a Gaussian of ρ mean is centered on the true measure value the maximum fuzzy value of the triangle x++x− is equal to 2 ρ. To compute x− and x+ we propose to use the full width at half maximum (FWHM) that can be calculated this way:
F W HM = 2p2ln(2) · σ (12) with σ that is the standard deviation of the measure. Thus for a Gaussian function that has a mean value ρ and a standard deviation σ the approximated interval X is defined by X = [ρ − 2p2ln(2) · σ; ρ + 2p2ln(2) · σ]. An example of this approximation is given on Fig. 5.
Until now all the implementations of the LAMDA
algorithm were using only crisp intervals despite the fact that
the general method was introduced. The class prototype is
now a triangle interval computed with the means of upper
and lower boundaries of the data used to train the algorithm.
Thus the membership function is still a similarity measure
between two fuzzy intervals like in (5) but it is necessary to
redefine the distance function between the intervals. A
solution has been proposed to measure a distance with the center
of gravity of triangular fuzzy intervals [
The intersection A ∩ B needed in (5) is calculated with an analytical solution based on geometry and trigonometry. It avoids numerical integration that could be less precise and longer to compute. 4.2 As we did previously with the crisp method, a test is performed with a Gaussian noise on the same data set (Fig. 3). The test is done in the same conditions as in the previous section. The difference is on the construction of the noisy data xˆ = x + Y . Y is now a random variable that follows a normal distribution of standard deviation σ and centered on 0. Results of the simulation are given on the table 2. σ Mean success percentage with binomial function Mean success percentage with crisp interval function Mean success percentage with fuzzy interval function 0.2 0.5 0.7 1 83.2% 79.8% 79.8%
79.6% 86.8% 82.5% 77.2%
71.3% 93.1% 84.5% 79.3% 74.8% Similarly to the previous test, the interval method increases the rate of good classifications until the standard deviation σ becomes too high and the binomial function provides better results. This point is reached here for σ = 0.7 which corresponds to a signal to noise ratio (SNR) of 6 dB for the signal with the smallest amplitude. Also it is important to notify that in all cases the fuzzy interval provides better results than the crisp interval method. 4.3
As a second example we use the classical iris dataset[
We assume that the data follow a normal distribution centered on a mean μj,n and with a standard-deviation σj,n. This hypothesis can be verified by using a statistical test. The Kolmogorov-Smirnov test has been used for each class with a 5% significance level, it shows that the hypothesis is true for the iris Setosa and the iris Versicolour but not for the iris Virginica. Nevertheless all the data are processed as if they follow a normal distribution.
The classifications are performed using the crossvalidation method. The percentages of well classified data for the two methods are: • using binomial function (scalar): 81.3% • using fuzzy triangular intervals: 94.0%
Once again the classification rate is increased by the use of the fuzzy interval method instead of the binomial one. 5
We presented in this article two methods to model uncertainty for classification applications. An example showed that these methods can improve classification results even when the signal to noise ratio is high. The second method based on fuzzy intervals demonstrated that try to model more precisely the probability law of the noise can provide better results than use confidence intervals modelled by crisp intervals. However this process to model uncertainty reveals limits when the SNR reaches a low level. A future important work is to limit the classification error of the interval method at the level of the numerical method.
These methods will now be tested on data out coming from a real industrial process.
Another way to manage uncertainty on classifiers like
LAMDA could be to use type-2 fuzzy functions [
To provide a better solution to manage uncertainty in the LAMDA classifier it can be useful to extend the problem to the qualitative features. It is often difficult to determine if a qualitative element is close to another, for example the color "orange" is closer to "red" than "blue". But on small training dataset consider this kind of information can improve final classification results. This could be done by using similarity matrix which are already used in some artificial intelligence problems.
LAMDA algorithm can work with a feature selection
algorithm named MEMBAS (Membership Margin Based
Feature Selection) [