Clustering is an unsupervised learning form which aims at revealing intrinsic patterns in data based on the distances or similarities of instances among each other. In real world datasets, clustering procedures may be significantly affected by noise, outliers and low-confidence data points. This paper presents a fuzzy clustering approach based on modified objective functions employing adaptive reliability measures, striving to enhance the robustness of the clustering results. The modification on the objective function integrates the notion of a point's influence into the clustering procedure. Therefore, the influence of each point will be controlled by a reliability score which will be evaluated adaptively. The root mean square propagation (RMSprop) algorithm is applied to dynamically assess the reliability score of the data points. Finally, this framework will be experimentally tested on several benchmark datasets to assess the quality of generated clusters and compare it to classical fuzzy clustering algorithms.
In machine learning, clustering is a core unsupervised learning task which intends to group similar data instances together, forming clusters (subsets) where elements share strong similarities with one another and clear dissimilarities from those in other clusters. The clustering process is guided merely by the inter-point similarity or distance, without any available external information about the latent structure of the dataset. Clustering has proven to be a valuable and flexible technique due to its wideranging applications, including recommendation systems in e-commerce, customer profiling in marketing, topic discovery in text mining, behavioral pattern analysis in psychology, and species categorization in biological sciences [1]. For any sizes of datasets, clustering constitutes a powerful tool for exploring and summarizing data, but especially for larger volumes of data it remains essential for uncovering patterns, revealing hidden structures, and often guiding downstream machine learning tasks [2].
There are multiple approaches to the clustering problem, among which can be distinguished hard clustering and fuzzy clustering. In the hard clustering approach, each data instance is assigned to exclusively one cluster, consequently the cluster boundaries are crisp and without overlapping. In contrast, fuzzy clustering allows instances to belong to multiple clusters simultaneously with varying degrees of membership (values between 0 and 1), making it a more flexible and realistic approach in circumstances where data points are not clearly separable. The traditional fuzzy clustering algorithm operates by implying an equal contribution of the data points to the clustering procedure, nonetheless this makes the algorithm susceptible to presence of noise and outliers [3, 4].
This paper presents a modification to the classical fuzzy clustering algorithm, incorporating adaptive reliability scores in the definition of the objective function of the algorithm. The central idea of this modification is to regulate the influence of the data points into the clustering process, based on a dynamically assessed reliability score. More specifically, the proposed method computes the reliability score of each point using entropy-based confidence metrics and the entire model is trained using the RMSprop optimizer, in order to adaptively scale learning rates. This approach allows the algorithm to down-weight the influence of uncertain or noisy data points during the clustering procedure, avoiding distortions that they may induce to the generated clusters. As a result, the model strives to be more resilient towards the presence of outliers and operates more effectively upon realworld, imperfect datasets. Together, the use of reliability scoring and adaptive optimization are expected to contribute to more stable, accurate, and interpretable clustering results. This approach will be tested extensively on several benchmark datasets and slightly distorted variants of benchmark datasets to assess quantitively the quality of the generated clusters.
In the recent decades, significant progress has been made in the field of unsupervised learning, designing various forms of clustering approaches and enhancements tailored to specific scenarios. Nevertheless, clustering remains a challenging problem, there is no algorithm capable of learning the patterns of every dataset. The idea of modifying the objective function and the idea of reliability measures usage have been presented in different flavors in various research works, often separately, but also occasionally intertwined together. In this section, the main approaches in these directions will be summarized and the differences in our approach will be accentuated.
F. Hoppner et al. have explored how the traditional fuzzy c-means objective function can be modified to support different levels of fuzziness in cluster memberships. They have analyzed how these modifications of the objective function allow for more flexible, partial membership values, enabling the clustering algorithm to better capture different data distributions and overlapping between clusters. These modifications were demonstrated to enhance the interpretability and robustness of fuzzy clustering in practical applications [5].
M. Menard et al. have proposed approaches to fuzzy clustering devising objective functions based on the principle of Extreme Physical Information. In their work, authors have exhibited how this method can systematically incorporate into the objective functions effectively minimal constraint terms. Their work is very well semantically explained for the physical perspective compared to the traditional algorithms [6].
H. Timm and R. Kruse have addressed a drawback in classical possibilistic fuzzy clustering, where the objective function is really minimized only if all cluster centers are equivalent and have proposed a modification to the objective function to trigger mutual repulsion among clusters, thus enhancing the clustering process [7].
J. Kang et al. have proposed a modified fuzzy c-means (FCM) algorithm that augments spatial neighborhood information into the conventional objective function. This approach was demonstrated to enhance the robustness of fuzzy clustering, especially when applied for image segmentation purposes [8].
H. Wang et al. have presented an automated multiscale fuzzy c-means (MSFCM) method for magnetic resonance images (MRI). The objective function of the conventional FCM method is modified to allow multiscale classification processing, thus improving the robustness especially when operating on low-contrast MR images [9].
X. Xiong et al. have proposed a modified generalized objective function for prototype-based fuzzy clustering incorporating a p-norm distance measure. Their approach induces cluster merging and the key innovation is the integration of principal component analysis (PCA) into the objective function. This methodology successfully captures the directional structure of the clusters utilizing the principal components [10].
K. Zhao et al. have introduced a generalized fuzzy c-means (FCM) clustering strategy that modifies the objective function involving a mechanism to control the degree of fuzziness in clustering results. Via this this mechanism, the algorithm can tune between hard and fuzzy clustering, making it more adaptable to various datasets [11].
A. Bagherinia et al. have presented a reliability-driven cluster indicator to assess the reliability of fuzzy clusters within an ensemble framework. This methodology assigns weights to multiple clustering outcomes based on their reliability, which achieved an overall higher clustering quality and robustness [12].
The approach presented in this paper revives partially ideas by F. Hoppner et al. and K. Zhao et al., but the execution strategy is vastly different as instead of Lagrange multipliers, the numerical root mean squared propagation (RMSprop) method is employed.
The classical Fuzzy C-Means algorithm (FCM) is the most significant algorithm in the field of fuzzy
cluster analysis. It generalizes the well-known K-Means algorithm, involving the concept of partial
degree of membership, thus allowing the instances of dataset to belong to several clusters
simultaneously, with different degrees of membership. Therefore, the outcome of this algorithm,
instead of being a set of clusters with their exclusive member points (like in the case of K-Means),
will be the fuzzy membership matrix (U) containing the respective membership values of the
instances into the clusters. The FCM algorithm operates as in iterative procedure aiming to
approximate the nonlinear optimization of the objective function formulated classically as:
n C
J ( X , U , V ; m , C ) = ∑ ∑ μimj d2 ( xi , c j)
i=1 j =1
(
The algorithm runs in iterations where it updates the membership degree of each point into each cluster, then adjusts the cluster centers based on the memberships. Eventually the convergence is achieved, settling into a configuration that best fits the dataset. The hyperparameters collectively influence the clustering results, so good choices of them typically assisted by tuning procedures are helpful. The following pseudocode describes the FCM algorithm [14]: 1. Randomly initialize the centers of the clusters. 2. Initialize the fuzzy membership matrix with zero values. 3. Let k = 1 (iteration counter) 4. Evaluate the distances of data points from cluster centers (dij values). 5. Update the fuzzy membership matrix, according to: μij= c ∑ di-kφ2-1 k =1 n
φ ∑ μij X j 6. Calculate the new centers of the clusters, according to: ci = j =1n ∑ μiφj j =1 7. k = k+1 (increment the iteration counter) 8. If ‖U k -1 - U k -2‖> Tol repeat at step 4. 9. END. Despite the multiple successful applications of the FCM algorithm across a wide range of domains, it struggles when operating in datasets characterized by complex structures such as overlapping clusters, varying cluster sizes and shapes, presence of noise and or outliers. In such datasets, the FCM algorithm typically underperform resulting in poorly constructed clusters [15]. In this work, is proposed a modification (detailed in the next section) which modifies the objective function of FCM incorporating the notion of influence of a data point in the cluster. Moreover, the optimization approach in this work will be based on a numerical optimization method: the root mean squared method.
The key idea of the modification proposed in this work is to control the influence that each point will have in the clustering process based on a reliability score that will be dynamically adapted. The definition of the objective function is:
n C J ( X , U , V ; m , C ) = ∑ ∑ ri μimj d2 ( xi , c j) + λR ( r )
i=1 j =1 In the above definition, X, U, V, m, C, μij, c j are the same as in the classical FCM, while ri is the reliability score of each instance which will be dynamically updated, R(r) is a regularization term upon the vector of reliabilities and λ is a coefficient controlling the weight of the regularization term. The usage of the regularization term intends to avoid degenerate or extreme reliability assignments, inducing stability in the clustering process. There are several possibilities how the regularization term can be designated; in our case is used the L2 regularization, which discourages extreme values in the reliability scores vector: On the other hand, in order to evaluate the reliability scores of the data points, an entropy-based approach is employed. For each data point will be evaluated the entropy value which quantifies how well distributed are the point memberships into the clusters:
C R ( r ) = ∑ ri2
j =1
C
H i = - ∑ μij log μij
j =1
(
In order to approximate a solution to the non-linear optimization problem defined by the given
objective function, the root mean square propagation (RMSprop) method will be used. This method
is an improvement over the classical gradient descent optimization technique. The fundamental
principle of gradient descent is tracking the direction of the steepest descent, using a constant
learning rate, while RMSprop adapts the learning rate adjusting it to the current landscape of the
objective function. So generally, for a parameter θ, the update is handled as [16]:
Parameter η represents the learning rate, g(t) denotes the gradient at the k-th iteration, E[g2](k) denotes
the root mean square of the recent gradients and ɛ is a small constant to avoid division by zero.
Furthermore, the update of the root mean square will be handled based on a decay parameter β as [17,
18]:
The general iterative scheme of the modified fuzzy clustering algorithm will remain the same as in
the classical FCM, with the primary distinction that the updates for the cluster centres and the fuzzy
membership values will be carried out numerically, according to the equations (
E [ g2μij ](k +1) = β ∙ E [ g2μij ](k ) +( 1 - β )∙( g(μkij))2 θ(k +1) = θ(k )
η √ E [ g2 ](k ) + ϵ
∙ g(k ) E [ g2μ ](k +1) = β ∙ E [ g2μ ](k ) +( 1 - β )∙( g(μt ))2
ij ij ij
g(ckj ) = ∂∂ cJj = 2( ∑i=N1 ri ∙ μimj ∙ ( xi - c j))+ λ ∙ ∂∂ Rcij
g(μkij) = ∂∂ μJij
= ri ∙ m ∙ μimj -1 d2 ( xi , c j) + λ ∙
∂ R
∂ μij
(
(k +1) = 7. Normalize the fuzzy membership matrix: μij η 8. Update the gradients of the centers, as: g(ckj ) = 2( ∑i=N1 ri ∙ μimj ∙ ( xi - c j))+ λ ∙ ∂∂ Rcij (k +1) = ckj 9. Update the centers, as c j
η √ E [ g2c ](t ) + ϵ ij ∙ g(ck ) j 10. Update the reliability values: ri = 1 - H i
log C 11. k = k+1 (increment the iteration counter) 12. If ‖U k - U k -1‖> Tol jump to step 4.
13. END.
In order to assess the robustness and the quality of the generated clusters generated by the proposed modified version of FCM algorithm, a series of experimental tests are conducted on several benchmark datasets. In order to evaluate the stability of the algorithm, in addition to the original versions of the benchmark datasets, two distorted versions are created for each benchmark dataset by adding artificial noise at different levels.The employed benchmark datasets were: Breast Cancer, Ionosphere, Vertebral Column, Dermatology, E. coli and Shuttle [19]. For each of the aforementioned datasets, two distorted versions are also created, with an additional quantity of respectively 2% and 5% noise points being added. The noise points are randomly placed at a distance from the cluster centres that is 8-10% larger than the average distance of the top-5 farthest genuine points from the respective cluster centre. The details of the original datasets are displayed in Table 1 below: Dataset Although the class labels for the employed classes are known, this information is not provided to the clustering procedures, instead it is utilized as the ground truth for the assessment of the clustering results. The quality assessment of the generated clusters is done via the V-measure, which is evaluated as the harmonic mean of homogeneity and completeness scores, so: 1 + 1 H C (10) The homogeneity score measures the degree to which each cluster contains data points of only one particular label, while the completeness score measures how well all data points with the same label are grouped into the same cluster. In order to be compatible with the fuzzy scenario, firstly the conditional fuzzy entropy between the generated clusters and the ground truth is evaluated, and afterwards these results are utilized to calculate the fuzzy homogeneity and fuzzy completeness scores.
The results of the experimental procedures are summarized in Table 2 in the following page: As noticed from the table, the modified version of the FCM performs typically with a higher Vmeasure, pointing out the effectiveness of this approach. Moreover, it can be noticed that the difference in the V-measure values increases as the distortion level increases, which indicates a better robustness of this methodology. However, a drawback of this method is the increased computational complexity compared to the classical FCM.
This paper presented a modified reliability-based fuzzy clustering algorithm devised by modifications on the objective function of the classical FCM algorithm. The primary goal of this approach was to construct a more robust clustering framework, less sensitive to the presence of noise, outliers and uncertain instances. The proposed approach leverages entropy-based metrics to dynamically evaluate a confidence value for each data point, in order to control their influence during the clustering process. The optimization of the modified objective function is carried out by the RMSprop optimization techniques, in order to achieve a more flexible and adaptive clustering process. The experimental procedures applied on several benchmark datasets and their slightly distorted variants demonstrated that the proposed algorithm generally performs with better fuzzy V-measure scores compared to the classical FCM algorithm. These findings indicate the method’s improved resilience to noise and uncertain data and its capability to distinguish inherent clusters in challenging scenarios. Despite the natural computational overhead introduced by the adaptive modification, the overall gains in clustering quality and stability suggest that this reliability-based framework offers promising directions for robust fuzzy clustering in real-world applications.
The author(s) have not employed any Generative AI tools. [9] Wang, Hesheng, and Baowei Fei. "A modified fuzzy C-means classification method using a multiscale diffusion filtering scheme." Medical image analysis 13, no. 2 (2009): 193-202. [10] Xiong, Xuejian, Kap Chan, and Kian Lee Tan. "Similarity-driven cluster merging method for unsupervised fuzzy clustering." arXiv preprint arXiv:1207.4155 (2012). [11] Zhao, Kaixin, Yaping Dai, Zhiyang Jia, and Ye Ji. "General fuzzy c-means clustering strategy: Using objective function to control fuzziness of clustering results." IEEE Transactions on Fuzzy Systems 30, no. 9 (2021): 3601-3616. [12] Bagherinia, Ali, Behrooz Minaei-Bidgoli, Mehdi Hosseinzadeh, and Hamid Parvin. "Reliabilitybased fuzzy clustering ensemble." Fuzzy Sets and Systems 413 (2021): 1-28. [13] Bedalli, Erind, and Ilia Ninka. "Exploring an educational system’s data through fuzzy cluster analysis." In 11th Annual International Conference on Information Technology & Computer Science. 2014. [14] Bedalli, Erind, Enea Mançellari, and Ozcan Asilkan. "A heterogeneous cluster ensemble model for improving the stability of fuzzy cluster analysis." Procedia Computer Science 102 (2016): 129136. [15] Liu, Zhe, and Sukumar Letchmunan. "Enhanced fuzzy clustering for incomplete instance with evidence combination." ACM Transactions on Knowledge Discovery from Data 18, no. 3 (2024): 120. [16] Zou, Fangyu, Li Shen, Zequn Jie, Weizhong Zhang, and Wei Liu. "A sufficient condition for convergences of adam and rmsprop." In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 11127-11135. 2019. [17] Liu, Jinlan, Dongpo Xu, Huisheng Zhang, and Danilo Mandic. "On hyper-parameter selection for guaranteed convergence of RMSProp." Cognitive Neurodynamics 18, no. 6 (2024): 3227-3237. [18] Babu, D. Vijendra, C. Karthikeyan, and Abhishek Kumar. "Performance analysis of cost and accuracy for whale swarm and RMSprop optimizer." In IOP Conference Series: Materials Science and Engineering, vol. 993, no. 1, p. 012080. IOP Publishing, 2020. [19] Amarnath, B., S. Balamurugan, and Appavu Alias. "Review on feature selection techniques and its impact for effective data classification using UCI machine learning repository dataset." Journal of Engineering Science and Technology 11, no. 11 (2016): 1639-1646.