When it comes to the task of dimensionality reduction, the Principal Component Analysis (PCA) is among the most well known methods. Despite its popularity, PCA is prone to outliers which can be traced back to the fact that this method relies on a covariance matrix. Even with the variety of sophisticated methods to enhance the robustness of the PCA, we provide here in this work-in-progress an approach which is intriguingly simple: the covariance matrix is replaced by a socalled comode matrix. Through this minor modi cation the experiments show that the reconstruction loss is signi cantly reduced. In this work we introduce the comode and its relation to the MeanShift algorithm, including its bandwidth parameter, compare it in an experiment against the classic covariance matrix and evaluate the impact of the bandwidth hyperparameter on the reconstruction error.
In cases where we have to deal with high-dimensional data, a common strategy is
to perform a Principal Component Analysis (PCA)[
Copyright c 2019 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
Many e orts have been made to make the PCA more robust towards noise. As
such in the work of [
Given a data matrix D where each of its rows represents a data record and its columns represent the features (A1; :::; Ad). When performing a PCA, rst a covariance matrix is computed. The covariance is a generalization of the variance. For computing the covariance from every feature, the expected value E (mean) is subtracted. Since the mean is sensitive towards outliers, the computation of the covariance matrix is also sensitive against outliers. A more robust measure is the mode. Generalizing the mode like variance is generalized to the covariance leads to the comode: . . .
com(A1; Ad) 1
... CA mode(Ad)
But how do we actually compute com(Ai; Aj) which represents the mode of
a variable Ai in dependence of another variable Aj? One solution we propose
here relies on the so called MeanShift[
PiN=(1x) K(x PiN=(1x) K(x xi)xi xi) (1) where K(X) = k de nes a kernel and N(x) are all objects in the neighborhood of x within a bandwidth . The bandwidth follows the intuition of an -range. It de nes the local neighborhood of an data object x. The mean shif t of x is the di erence m(x) x. At this point it may still be unclear how the MeanShift may lead to the detection of the comodes? A comode com(Ai; Aj) is determined by computing the MeanShift shift(DAi;Aj ) on a given dataset D considering only its features (= random variables) Ai and Aj.
The comode matrix can thus be expressed in terms of MeanShift computations:
What remains unclear so far is: how do we approach the fact that there can be not only one (unimodal) but several (multimodal) modes? In this work-inprogress we have focused on taking the mode with the highest frequency, meaning we take the top mode which has the highest number of objects belonging to it. In the case where we have several equally high top-modes, we randomly select one of them. Due to the brevity of this paper, we pursue it in future work to elaborate and evaluate the impact of choosing di erent top-modes. 4
In our experiments we use the MeanShift with a at kernel from sklearn1. For reproducibility purposes, the code is made available on github2. We conduct our experiments on the iris dataset3 which consists of 150 instances and 4 features. We compute the PCA using the comode instead of the covariance matrix. We discard all principal components from the eigenvector matrix U except the rst one, and project the data D down to its lower (one)dimensional representation Y through: Y = D Uk=i. We reconstruct the data by projecting it back to its original full-dimensional representation through: Z = Y Uk=i. By projecting the data down and back again, enables us to compute the Mean Squared Error (MSE) which is de ned as M SE(D; Z) = Pin=1(ndi zi)2 where di 2 D, zi 2 Z and n denoted the number of objects for which holds n = jDj = jZj. We apply this procedure for taking the second, third,...,d-th principal component. Since we want to investigate the e ect of the choice of the bandwidth we further perform the comode computation choosing the bandwidths = (0:1; 4:0; 5:5). The results can be seen in Figure 4 where the horizontal axis represents the principal components in descending order of their corresponding eigenvalues. The vertical axis represents the MSE. It can be seen from Figure 4 that PCA with comode and a bandwidth of = 0:1 yields an MSE ( 1) which is signi cantly below that of the classical PCA with a covariance matrix ( 7) taking only the rst 1 https://scikit-learn.org/stable/modules/generated/sklearn.cluster.MeanShift.html 2 https://github.com/hamilton-function/comode 3 https://archive.ics.uci.edu/ml/datasets/iris principal component. By taking additionally the second principal component the MSE of the covariance-based PCA drops even slightly below the MSE of the comode variant with = 0:1 which holds for taking three principal components as well. A larger bandwidth of = 4:0 yields to MSE values which exhibit a course similar to that of the covariance-based PCA but with overall higher MSE values. At this point it is interesting for further research to investigate if there is an 'even'-point regarding the bandwidth where the MSE for di erent number of principal components equals the MSE of the covariance-based PCA. A bandwidth of = 5:5 yields MSE results which are by far worse compared to the other settings.
However, we have to take the MSE results with a grain of salt for the
following observation which has been made in [
In this work-in-progress we have presented the Comode in context of PCA. A rst experiment showed promising results, outperforming a covariance-based PCA while being intriguingly simple. There are however interesting aspects which demand further research: rst and foremost, it remains an open question on how to deal with multimodal cases. What are the e ects on the PCA if we choose the second or third strongest mode(s)? How are good bandwidths determined? For these aspects we may seek previous works which aimed at estimating good bandwidths for MeanShift. Further it is of interest to investigate if featurespeci c bandwidths have any impact on the robustness of the Comode-based PCA. As for now, we have one bandwidth which is valid for all features, neglecting feature-speci c bandwidths. We hope to stimulate further research on the Comode, revealing its limitations as well as its potentials.
This work has been funded by the German Federal Ministry of Education and Research (BMBF) under Grant No. 01IS18036A. The authors of this work take full responsibilities for its content.