Dimensionality reduction is defined as the search for a low-dimensional space that captures the “essence” of the original high-dimensional data. Principal Component Analysis (PCA) is one of the most used dimensionality reduction technique in psychology and behavioral sciences for data analysis and measure development. However, PCA can capture linear correlations between variables, but fails when this assumption is violated. In recent years, a variety of nonlinear dimensionality reduction techniques have been proposed in other research fields to overcome this limitation. In this paper, we focus on non-linear autoencoder, a multi-layer perceptron, with as many inputs as outputs and a smaller number of hidden nodes. We investigate the relation between the intrinsic dimensionality of data and the autoencoder's internal nodes in a simulation study, comparing autoencoders and PCA performances in term of reconstruction error. The evidence from this study suggests that autoencoder's ability in dimensionality reduction is very similar to PCA, and that there is a relation between internal nodes and data dimensionality.
The transformation of data from high-dimensional space into a meaningful low-dimensional space,
which ideally corresponds to the intrinsic dimensionality of the original data, is referred as
“dimensionality reduction”. This transformation is important in several domains because it mitigates
undesired properties of high-dimensional spaces such as the curse of dimensionality [
However, determining the number of dimensions of a data set requires researchers to take several
important decisions: in particular, the choice of extraction method and the decision about how many
components to retain are considered among the most critical in psychological scale development [
Intrinsic dimensionality, as an important intrinsic characteristic of high-dimensional data, can be
defined as the minimum number of coordinates which are necessary to describe data points without
significant information loss: because the process of dimensionality reduction inevitably leads to
information loss, it is very important to preserve the main and important characteristics of the original
data as much as possible. So, dimensionality reduction is not only related to data compression, but also
to feature extraction [
Traditionally, dimensionality reduction is performed using linear techniques such as Principal
Components Analysis (PCA), which is one of the most used statistical techniques in behavioral sciences
and is a standard part of measure development [
PCA was first introduced by Pearson in 1901 and developed by Hotelling in 1933 [
In contrast to the traditional linear techniques for dimensionality reduction, machine learning techniques can deal with complex nonlinear data, and they represent a valuable alternative to classical methods. In this context, a considerable amount of work has been done on non-linear extensions of PCA and a variety of approaches has been proposed. Among others, Autoencoders seem a valuable alternative to PCA for dimensionality reduction.
Autoencoder, also called auto-associative neural network or bottleneck network, is a multi-layer
perceptron with as many inputs as outputs and a smaller number of hidden feature units. During training,
the targets for the output units are set to be equal to the inputs. The weights in the network are then
trained to minimize the square error of the reconstruction [
In line with these considerations, the aim of this paper is to investigate in a systematic way a possible relation between the number of hidden layer nodes and the intrinsic dimensionality of data, comparing PCA and autoencoders reconstruction error on artificial datasets.
Datasets are generated from factor-based population, a choice due to their diffusion in psychometric
research [
The rest of article is organized as follows: first, methods are briefly described, and the study is presented in a more detailed way; then, experimental procedures, data analysis and results are showed; finally, section 4 concludes the paper and discusses several future research directions.
In many areas among the social and life sciences the amount of high-dimensional data has rapidly increased within the past year: to handle such real-world data adequately, dimensionality needs to be reduced into meaningfully expression in low-dimensional space.
In this section two approaches for dimensionality reduction will be described: subsection 2.1 discuss Principal Component Analysis, the most famous linear dimensionality reduction technique; then, we describe Autoencoders, an alternative non-linear approach for dimensionality reduction more recently proposed. 2.1.
PCA can be defined as the orthogonal projection of data onto a lower dimensional linear space, such
that the variance of the projected data is maximized [
To derive the form of PC’s, suppose is a vector of variables with a covariance matrix . The first step is to search for a linear function 1′ of the elements of having maximum variance: 1′ = 11 1 + 12 2 + … + 1 = where 1′ is a vector of constants 11 1, 12 2 , … , 1 and ‘ denotes transpose. The variance of the projected data is given by: multipliers. It follows that: and is maximized under the normalization constraints 1′ 1 = 1 using the techinque of Lagrange
( 1 ) = 1′ 1 = 1 the largest eigenvalue. where λ is the Lagrange multipliers. By setting differentiation with respect 1 equal to zero, the solution of this problem can be obtained as a unit eigenvector of the covariance matrix S corresponding to the largest eigenvalue. Thus, 1 is the eigenvector corresponding to the largest eigenvalue of , and
In general, the kth PC of is and its variance is , where is the largest eigenvalue of S and is the corresponding eigenvector or, also, the vector of loadings for the kth component. We can define additional principal components in an incremental fashion by choosing each new direction to be that which maximizes the projected variance amongst all possible directions orthogonal to those already considered. To summarize, principal component analysis involves evaluating the covariance matrix of the dataset and then finding the k eigenvectors of S corresponding to the k largest eigenvalues.
PCA can be also viewed as a linear projection of data points into a lower dimensional space such that the squared reconstruction loss is minimized. In general, a dimension reduction technique provides an approximation ̂( ) to ( ) which is the composition of two functions f and g:
[ 1′ ] = 1′ 1 1′ 1 = ( 1′ 1 − 1) ∑ 1 =1 (1) (2) (3) (4) (5) (6) ( ) = ̂( ) + ( ) = ( ( ( ))) + ( )
The projection function ∶ → projects the original P-dimensional data ( ) onto a
Zdimensional subspace, while the expansion function ∶
→ defines a mapping from the
Zdimensional space back into the original P-dimensional space with ( ) as the residue. The feature
extraction problem may involve the determination of functions f and g. The mean square error (MSE)
in reconstructing the original data is:
= [‖ − ( ( ))‖ ]
2
It can be shown that PCA is the algorithm which obtains the smallest MSE among all techniques with
linear projection and expansion functions f and g [
Autoencoder, also called auto-associative neural network, is a multi-layer perceptron having the
same number of outputs as inputs, designed to learn an approximation to the identity function, so as the
output is as similar to the input as possible [
We can view this network as two successive functional mappings and as indicated in Figure 2. The first mapping projects the original -dimensional data into a Z-dimensional subspace S defined by the activations of the units in the second hidden layer. Because of the presence of the first hidden layer of nonlinear units, this mapping is very general, and is not restricted to being linear. =1 where ( ) is the weight between the i-th neuron of layer + 1 and the j-th neuron of layer m, and
Autoencoder has the advantage of not being limited to linear transformations and can learn more
complicated relations between visible and hidden units, although it contains standard principal
component analysis as a special case. However, unlike PCA, the coordinates of the output of the
bottleneck are correlated and are not sorted in descending order of variance. Moreover, computationally
intensive nonlinear optimization techniques must be used, and there is the risk of finding a suboptimal
local minimum of the error function [
Dimensionality reduction implies capturing the "essence" of the data, that is, extracting the most important information. In PCA, this is achieved by selecting the principal components that explain most of the relationships among the variables and, so, reflect the intrinsic dimensionality of data, but little research has been done on autoencoder ability in dimensionality extraction.
In 2016, Wang et al. [20] investigated a possible relation between the number of hidden layer nodes, the performance of autoencoder and the intrinsic dimensionality of data. This study was conducted on MNIST and Olivetti face datasets by recording the change of performance of the classifier when the dimensionality of the projected representation varies. Results of this study showed a possible relation between the hidden nodes, the intrinsic dimensionality of MNIST dataset and the autoencoders accuracy.
Similarly, in this paper we want to investigate this relation in a more systematic way. The aim is to compare PCA and autoencoder ability in dimensionality extraction on different factor-based simulated datasets. Our hypothesis is that autoencoder’s representation of data lying in bottleneck layer captures the most important data characteristics and is in relation with data intrinsic dimensionality. So, the performance of autoencoder should be optimal when the number of internal nodes is equal to data dimensionality. More details and results are showed in the next section.
In order to investigate the relation between data dimensionality and autoencoder’s internal nodes, a simulation approach is chosen, because of the possibility to analyze different scenarios by varying only the selected design-factors. In this section, the simulation study is described in detail, and results are showed. 3.1.
Analyses were conducted on artificial data generated from different factor-based population, using the R package Lavaan [21]. Relationships in the model were set assuming the theoretical path model represented in Figure 4 and then data were simulated considering the given values of the parameters.
The simulation study considered different scenarios, varying the following design-factors: sample size, number of components and number of observed variables. The considered levels for each designfactor are presented in Figure 4. The total number of scenarios obtained from the combination of these levels of the design-factors was equal to 48 (4 sample sizes × 3 number of components × 4 number of observed variables). For each considered scenario, we generated one dataset.
Model used for generating data:
n = 200 Generated sample n = 500 size: n = 700 n = 1000
Number of components: Loadings:
All factor loadings are equal to 0.7
Number of observed variables
for each component:
Non-linear autoencoders used for dimensionality reduction were implemented in Python using Keras module [22]; the non-linearity of choice was the hyperbolic tangent activation function (tanh) except for the bottleneck and the output layers which used a linear activation function.
All layers but for the bottleneck had the same number of neurons, equal to the number of observed variables. The bottleneck layer’s nodes varied from one to the number of observed variables. That is, for a dataset with n observed variables, n autoencoders were trained on the whole dataset, with neurons in the bottleneck layer varying from 1 to n. MSE was computed for each autoencoder.
Weights were initialized based on the uniform distribution suggested by Glorot and Bengio [23] and Adam optimizer was used with 0.0001 learning rate as it offers both fast training and good generalization performance [24].
Finally, PCA was performed using Scikit-learn module in Python [25] and MSE was computed for each possible number of components (from 1 to the number of observed variables).
Figures 5, 6, and 7 show results for three scenarios: a) 3 components and 9 observed variables. b) 5 components and 25 observed variables. c) 7 components and 63 observed variables.
These scenarios are, respectively, the smallest, the medium and the largest among those obtained from all the possible combinations of the chosen design-factors.
Results show that MSE for both Autoencoders and PCA are very similar and are about the same when sample size is sufficiently large. Plots always display a downward curve, starting high on the left, falling rather quickly, and then flattening out at some point: this "elbow" coincides with the intrinsic data dimensionality. This pattern is always repeated, except in models with many observed variables and low sample size. In these cases, autoencoders results don’t follow the same trend as PCA and don’t provide information about the data dimensionality.
For the maximum number of components, MSE score for PCA is equal to zero, because considering all components data are perfectly reconstructed, and all the variation is retained. MSE scores for autoencoder, when the number of nodes is near or equal to the number of observed variables, are low, but doesn’t display the same pattern as PCA and are not equal to zero.
In this work, we have compared PCA and non-linear autoencoder and we have hypothesized a relationship between the number of autoencoder’s internal nodes and the intrinsic dimensionality of data. Results shows that autoencoder can perform dimensionality reduction as well as PCA, with an adequate sample size. Furthermore, results show that neurons in internal layer have a relation with the dimensionality of data. In fact, after the “elbow” of the graph, in which the number of neurons coincides with the dimensionality of data, the decrease in the MSE is slower and thus not sufficient to compensate for the increase in complexity. Because the choice of number of hidden neurons is a priori choice, it is useful to know that nodes of the bottleneck layer have a relation with data dimensionality.
The most important difference between PCA and autoencoders is that autoencoders can utilize nonlinear activation functions at the different layers of the neural network whereas, in PCA, dimensionality reduction is done in a linear transformation. The use of non-linear activation functions is what makes autoencoders a more flexible method for learning patterns in data.
However, in this work, PCA and Autoencoder seems to have the same behavior in dimensionality reduction: even if both methods offer information on the intrinsic dimensionality of data, it is important to consider that relations between simulated variables are linear; therefore, it is likely that a linear method performs better. However, relations met in real world are not always linear: for this reason, a future work will evaluate autoencoder performances on simulated data with non-linear relations between variables. Moreover, future research will also focus on autoencoder performances on real datasets. This future work will allow deeper understanding of the similarities and differences between these methods.
PCA and autoencoders share architectural similarities, but despite this fact, an autoencoder by itself does not have PCA properties. Incorporating some PCA constraints, autoencoder’s solution would have the following benefits: a) uniqueness; b) components would be uncorrelated and sorted in descending order of variance; and c) when reducing the data from dimension to dimension , the first vectors ( < ) would be the same as the solution for reduction from dimension to [26].
Despite machine learning methods are increasingly prevalent in several areas of psychology [27, 28], autoencoders are absent in psychometric research. Nevertheless, we believe that autoencoder can be used where some traditional methods show their limits. For example, in future research, autoencoders will be applied to the development of short form of psychological test. In fact, despite the potential benefits of using shorter measures, development of short forms shows several limitations: first, development of an abbreviated measure can be a relatively laborious process and second, most short forms of existing measures are not guaranteed to achieve optimality because their developers typically consider only a small fraction of possible alternate forms. In this context, neural networks can help to automatize and optimize short-form development process [29]. In particular, an autoencoder trained on a long-form of a measure, could be useful in selecting the short form that better reconstructs the long form, among the many possible and alternative short-forms. In this case, keeping the number of hidden neurons equal to the number of dimensions in the original test could help to choose short-forms that have the same dimensionality of the original measure.
In conclusion, despite additional investigations are required, we believe that autoencoders, which are already widely used in other scientific fields, are an interesting alternative to standard PCA also in psychometric models. 5. References [19] Hinton, Geoffrey E., and Ruslan R. Salakhutdinov. "Reducing the dimensionality of data with neural networks." science 313.5786 (2006): 504-507. [20] Wang, Yasi, Hongxun Yao, and Sicheng Zhao. "Auto-encoder based dimensionality reduction." Neurocomputing 184 (2016): 232-242. [21] Rosseel, Yves. "Lavaan: An R package for structural equation modeling and more. Version 0.5– 12 (BETA)." Journal of statistical software 48.2 (2012): 1-36. [22] Chollet, François. "keras." (2015). [23] Glorot, Xavier, and Yoshua Bengio. "Understanding the difficulty of training deep feedforward neural networks." Proceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010. [24] Kingma, Diederik P., and Jimmy Ba. "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980 (2014). [25] Pedregosa, Fabian, et al. "Scikit-learn: Machine learning in Python." the Journal of machine
Learning research 12 (2011): 2825-2830. [26] Plaut, Elad. "From principal subspaces to principal components with linear autoencoders." arXiv preprint arXiv:1804.10253 (2018). [27] Dwyer, Dominic B., Peter Falkai, and Nikolaos Koutsouleris. "Machine learning approaches for clinical psychology and psychiatry." Annual review of clinical psychology 14 (2018): 91-118. [28] Dolce, Pasquale, Davide Marocco, Mauro N. Maldonato, and Raffaele Sperandeo. "Toward a Machine Learning Predictive-Oriented Approach to Complement Explanatory Modeling. An Application for Evaluating Psychopathological Traits Based on Affective Neurosciences and Phenomenology." Frontiers in psychology 11 (2020): 446. [29] Gonzalez, Oscar. "Psychometric and machine learning approaches to reduce the length of scales." Multivariate Behavioral Research (2020): 1-17.