An important problem that arises when processing large amounts of observations is data compression to highlight the most essential information and identify certain latent factors that implicitly determine the nature of the phenomenon being studied. One of the most effective approaches to solving this problem is the apparatus of factor analysis, which has found wide application in problems of processing empirical data in various fields. Fuzzy clustering is a popular approach for soft data partitioning, its use always encounters difficulties in solving the problems of processing high-dimensional real data with complex hidden distributions. This paper proposes a disclosure of a kind of stack fuzzy clustering method where the data is represented in a new feature space created by a staking neural network. This approach aims to overcome the challenges associated with processing complex data and can bring significant improvements in the quality of clustering.
Clustering is a technique in machine learning and data analysis that involves grouping a set of data points into subsets, or clusters, based on the similarity between them. Fuzzy clustering is a variation of traditional clustering methods that allows for more flexible and nuanced assignments of data points to clusters [1-5]. In contrast, fuzzy clustering allows data points to belong to multiple clusters simultaneously with varying degrees of membership. This reflects the inherent uncertainty or ambiguity present in real-world data.
The Fuzzy C-Means (FCM) algorithm, proposed by James Bezdek in 1973, is a prominent method in fuzzy clustering [6]. FCM assigns membership degrees to data points, indicating the likelihood of each point belonging to different clusters. This flexibility makes fuzzy clustering particularly useful in scenarios where data points may exhibit overlapping characteristics or uncertainty in their categorization.
Over the years, fuzzy clustering has found applications in diverse fields, including pattern recognition, image processing, and Data Mining. Researchers have developed various extensions and enhancements to the original FCM algorithm, addressing specific challenges and improving its adaptability to different data patterns.
The validity of fuzzy clustering solutions became a key focus, leading to the introduction of indices to assess the quality of clustering results. These indices help researchers and practitioners evaluate the effectiveness of fuzzy clustering algorithms in capturing meaningful patterns within datasets.
The evolution of fuzzy clustering has seen ongoing advancements, with researchers exploring sophisticated membership functions and integrating fuzzy clustering with other machine learning techniques. This integration has expanded the capabilities of fuzzy clustering, making it applicable to complex problems in large-scale data analysis.
The era of big data has significantly influenced the field of clustering, including traditional clustering methods and the development of fuzzy clustering techniques. Deep learning in big data represents a powerful combination that has transformed various fields by enabling more sophisticated analysis, pattern recognition, and decision-making capabilities.
Recently, there has been significant research into leveraging deep learning to uncover meaningful data representations through neural networks. A notable area of exploration involves the integration of unsupervised clustering algorithms with stack neural networks. This synergy has become a vibrant field of research, aiming to jointly optimize the performance of deep learning models and clustering algorithms.
The goal of the work is propose the stack neuro-fuzzy system for Data Stream Mining using credibilistic approach and designed to work both in batch and its recurrent online version.
An important problem that arises when processing large amounts of observations is data compression to highlight the most essential information and identify certain latent factors that implicitly determine the nature of the phenomenon being studied. One of the most effective approaches to solving this problem is the apparatus of factor analysis [7], which has found wide application in problems of processing empirical data in various fields: psychology, sociology, technology, economics, medicine, criminology, etc.
The basic idea of factor analysis, which allows for the presence of a priori unknown hidden factors, leads to the following informal task: by observing a large number of measured parameters (indicators), identify a small number of parameter-factors that mainly determine the behavior of the measured parameters, or in other words: knowing the values of a large number of measured functions parameters, set the appropriate values of the factor arguments common to all functions and restore the form of these functions.
The initial information for factor analysis is the ( N × n) observation matrix:
x1(
1 N x (k) = ∑ x(k), x(k) =x(k) − x (k)
N k=1 vectors of measured indicators centered relative to the average of the data array.
One of the most common and effective methods for finding factors is the principal component method or component analysis, which is widely used in problems of data compression, pattern recognition, coding, image processing, spectral analysis, etc. and known in pattern recognition theory as the Karhunen-Loeve transform.
The task of component analysis is to project data vectors from the original n-dimensional space into a m-dimensional one (m < n) space of principal components and reduces to searching for a system w1, w2 ,..., wm orthonormal eigenvectors of the matrix R(N ) such that w = (w11, w12 ,..., w1n )T corresponds to the largest eigenvalue λ1 matrix R(N ) , w2 - second largest 1 eigenvalue λ2 , etc. In other words, the problem comes down to finding solutions to the matrix T that formed by an array of N n-th dimensional vectors x(k) = ( x1, x2 ,..., xN ) and autocorrelation matrix (n × n) where
R(N ) = 1 N
∑( x(k) − x (N ))( x(k) − x (N ))T = N k =1 1 N
∑ x(k)xT (k), (
The dimension of the space of principal components m is determined, as a rule, from empirical considerations and the required degree of compression of the data array.
Thus, in algebraic terms, solving a factor problem is closely related to the problem of eigenvalues and finding the rank of the correlation matrix; in a geometric sense, this is the problem of moving to a lower-dimensional space with minimal loss of information; in a statistical sense, this is the problem of finding a set of orthonormal vectors in the input space that “accept” the maximum possible variation of the data, and finally, in an algorithmic sense, this is the problem of sequentially determining a set of eigenvectors w1, w2 ,..., wm by optimizing a set of local criteria that form a global objective function with constraints w1(k + 1) =w1(k ) +η (k ) y1(k ) ( x(k ) − w1(k ) y1(k )), y1(k ) =)x(k w1T (k ), w1(0) ≠ 0 the first principal component can be isolated.
Next, following the procedure of standard principal component analysis, from each vector x(k ), k = 1, 2,..., N its projection onto the first principal component is subtracted and the first principal component of the differences is calculated, which is the second principal component of the original data and the orthonormal first. The third principal component is calculated by projecting each original vector x(k ) into the first two components, subtracting this projection from x(k ) and finding the first principal component of the differences, which is the third principal component of the original data array. The remaining principal components are calculated recursively according to the described strategy.
It is this idea of recursive calculation of principal components that forms the basis of the algorithm proposed by T. Sanger [8] and in a modified form having the form [9]
The first principal component w1 can be found by maximizing the criterion: by solving a nonlinear programming problem using uncertain Lagrange multipliers.
However, if data processing must be carried out in real time, neural network technologies come to the fore, among which the self-learning rule and E. Oya’s neuron should be noted.
It is with the help of Oya's rule in the form:
Ek =1∑m E kj k j=1
1 m k =∑ ∑ ( wTj x( p)) k j=1 p=1
2 wT w j l =0, j ≠ l, wTj wj
=1.
E1k = 1 k
∑ ( w1T x( p))
k p=1
2
(
It is easy to see that the first principal component is calculated using the Oya algorithm, then the projection of the input vectors onto w1(k) are subtracted from the inputs and the differences are processed by the next neuron, etc.
In Fig. 1 shows a diagram of a modified artificial T. Sanger’s neural network, composed of
E. Oya’s neurons and implementing the algorithm (
The first layer of the network is formed by encoder neurons that pre-process signals by centering and normalizing them. Further signals x1(k), x2 (k),..., xn (k) are processed in the second hidden layer formed by E. Oya's neurons, after which they are sent to the output layer formed by elements with activation rectifier functions with a dead zone
u, if u ≥θ , ψ (u) =
0, otherwise which allows you to highlight informative signals y j (k) and filter out the noise.
The Sanger neural network is an effective means of compressing information with minimal loss of accuracy, but its capabilities are limited by the fact that, implementing essentially the standard technique of factor analysis, it solves a linear problem, while the main advantage of neural network technologies is the ability to work in purely nonlinear situations.
The problem of nonlinear factor analysis can be effectively solved using credibilistic theory and clustering analysis.
Alternatively, to probabilistic and possibilistic procedures [10] it was introduced credibilistic fuzzy clustering approach using as its basis the credibility theory [11] and is largely devoid of the drawbacks of known methods.
The most common approach within the framework of probabilistic fuzzy clustering is associated with minimizing the goal function [12-14]. with constraints
N m E (uq (k),cq ) = ∑ ∑uqβ (k)d 2 ( x(k),cq )
k=1 q=1 m ∑uq (k) = 1, q=1
N 0 < ∑uq (k) < N.
k=1
Solution of nonlinear programming problem using the method of Lagrange indefinite
multipliers leads to the well-known result [9, 11, 15]:
(
If the data are fed to processing sequentially, the solution of the nonlinear programming
problem (
The goal function of credibilistic fuzzy clustering has the form [6, 11] close to (
= for any q and k, for which Credq (k ) ≥ 0,5.
uq (k ) =ϕ q (d ( x(k ), cq )) uq (k ) =
1 1 + d 2 ( x(k ), cq )
.
uq (k ) = 1 +
−1 d 2 ( x(k ), cq ) ,
σ q2
σ q2 = ∑m d 2 ( x(k ), cl )
l=1
l≠ε
−1
(
It should be noted that the goal functions (
In the procedures of credibilistic clustering, there is also the concept of fuzzy membership, which is calculated using the neighborhood function of the form: monotonically decreasing on the interval [0, ∞] so that ϕ q (0) =1,ϕ q (∞) → 0.
Such a function is essentially an empirical similarity measure of [13, 15, 16] related to distance by the relation:
Note also that earlier it was shown in [14] that the first relation (
In batch form the algorithm of credibilistic fuzzy clustering in the accepted notation can be written as −1 uq (k ) = (1 + d 2 ( x(k ), cq )) , uq∗ (k ) = uq (k ) (sup ul (k ))−1 , Credq (k ) ∑N (Credq (k ))β x(k ) cq = k=1 ∑N (Credq (k ))β k=1 =12 uq∗ (k ) + 1 − sup ul∗ (k ) ,
l≠q
and in the online mode, considering (
−1 d 2 ( x(k + 1), cq (k ))
, σ q2 (k + 1) uq∗ (k + 1) =uq (k + 1) ,
sup ul (k + 1) Credq (k + 1) =12 uq∗ (k + 1) + 1 − sup ul∗ (k) ,
l≠q cq (k + 1) =cq (k ) +η (k + 1)Credqβ (k + 1) ( x(k + 1) − cq (k )). (18) (19)
From the point of view of computational implementation, algorithm (19) is not more
complicated than procedure (
Conducting experimental studies and comparative analysis of the quality of data clustering using various metrics allows you to objectively assess the effectiveness of the developed method in accordance with analogues. To estimate the quality of the method we used quality criteria partitioning into clusters such as [3, 6]: − Partition Coefficient (PC); − Classification Entropy (CE); − Partition Index (SC); − Separation Index (S); − Xie and Beni's Index (XB); − Dunn's Index (DI).
Partition Coefficient (PC): measures the amount of "overlapping" between clusters.
Classification Entropy (CE): it measures the fuzzyness of the cluster partition only, which is similar to the Partition Coefficient.
Partition Index (SC): is the ratio of the sum of compactness and separation of the clusters. It is a sum of individual cluster validity measures normalized through division by the fuzzy cardinality of each cluster. SC is useful when comparing different partitions having equal number of clusters. A lower value of SC indicates a better partition.
Separation Index (S): on the contrary of partition index (SC), the separation index uses a minimum-distance separation for partition validity.
Xie and Beni's Index (XB): it aims to quantify the ratio of the total variation within clusters and the separation of clusters. The optimal number of clusters should minimize the value of the index.
Dunn's Index (DI): this index is originally proposed to use at the identification of "compact and well separated clusters".
So the result of the clustering has to be recalculated as it was a hard partition method. The specific information of the data sets is shown in Table 1. Online Stack Fuzzy Credibilistic Clustering for Data Stream Mining SOM based on possibilistic fuzzy clustering
SOM based on probabilistic fuzzy clustering Online Stack Fuzzy Credibilistic Clustering for Data Stream Mining SOM based on possibilistic fuzzy clustering SOM based on probabilistic fuzzy clustering 7 0 е 9 4 2 1 . 9 5. Discussions Upon analyzing the results acquired, it can be inferred that irrespective of the volume of the initial data provided, the processing through the proposed method exhibits comparable speed and clustering quality when contrasted with established clustering algorithms and methodologies.
The obtained results confirm that the performance stack neuro-fuzzy system is better than other network structures, and it can be a viable structure for Data Stream Mining.
The results of accuracy that demonstrated in Table 4 confirm that proposed method online stack fuzzy credibilistic clustering for Data Stream Mining time more superiority regardless of the number observations that are fed on clustering process.
Based on the experimental findings, it is advisable to endorse the proposed system for practical application in addressing the challenges associated with automatic clustering of large datasets.
The problem of fuzzy clustering of data streams by stack neuro-fuzzy system is considered. In the paper was proposed the stack neuro-fuzzy system for Data Stream Mining using credibilistic approach and designed to work both in batch and its recurrent online version.
The network shows that stack structures based on fuzzy models can be applicable in data clustering. The proposed stack neuro-fuzzy system is quite simple in numerical implementation and can use the well-known online fuzzy clustering methods intended for solving Data Stream Mining problems.
Future research endeavors could explore the potential of employing stack neuro-fuzzy systems for fuzzy clustering of data streams, aiming to address the complexities inherent in automatic clustering of big data.
The work is supported by the state budget scientific research project of Kharkiv National University of Radio Electronics "Deep hybrid systems of computational intelligence for data stream mining and their fast learning" (state registration number 0119U001403).