-Various data mining methods are used for examining large financial data sets to uncover hidden and useful information. Ability to access big data sources raises new challenges related with capabilities to handle such enormous amounts of data. This research focuses on big financial data visualization that is based on dimensionality reduction methods. We use data set that contains financial ratios of stocks traded on NASDAQ stock exchange. A brief overview of the most popular dimensionality reduction and visualization methods is presented in this paper. We also show how to adjust the algorithms of these methods for parallel computing. The MPI technology is applied in computer cluster to perform dimensionality reduction. The results show that Random projection and Multidimensional scaling methods can effectively classify data and find the most promising stocks.
This template, We use big data analytics for examining large financial data sets to uncover hidden and useful information. Ability to access new data sources provides many opportunities, for example create new investors sentiment indexes from online social media streams. But it also raises new challenges related with capabilities to handle such enormous amounts of data. We need effective methods and powerful environments to complete large and complex tasks.
This research focuses on big financial data visualization that is based on dimensionality reduction methods. We use data set that contains financial ratios of stocks traded on NASDAQ stock exchange. Our goal is to find the most effective ways to analyze and visualize such type data. In the second section, we present a brief overview of the most popular dimensionality reduction and visualization methods. In the third section we present how to adjust the algorithms of these methods for parallel computing. We use MPI technology in computer cluster to perform dimensionality reduction. In the last section we present the classification and visualization results which where get while using Random projection and Multidimensional scaling methods. The current data set is relatively small, but the results give suggestions what techniques to choose for future works with big online data streams.
The detailed reviews of dimensionality reduction methods were done by I. K. Fodor (2002), M. Mizuta (2004), C.O.S. Sorzano, J. Vargas et. al. (2014). In this section we present the brief summary of most popular methods in order to choose best suitable in our case.
Dimensionality reduction refers to the process of taking a
data set with a usually large number of dimensions, and then
creating a new data set with a fewer number of dimensions,
which are meant to capture only those dimensions that are in
some sense “important”. The idea here is that we want to
preserve as much “structure” of the data as possible, while
reducing the number of dimensions it possesses [
The demand for such methods rises because various sources generate enormous amount of data, e. g laboratory instruments can report thousands measurements for a single experiment, and the statistical methods face challenging tasks when dealing with such high‐ dimensional data. However, according to C.O.S. Sorzano, J. Vargas et. al. (2014), much of the data is highly redundant and can be efficiently brought down to a much smaller number of variables without a significant loss of information.
Fig. 1. Dimensionality reduction methods
As long as data have a near-linear structure, the
singularities of the data can be pointed out using Principal
Component Analysis (PCA) [
PCA find components that make projections uncorrelated
by selecting the highest eigenvalues of the covariance matrix
and maximizes retained variance [
Random Projection finds components that make projections
uncorrelated by multiplying by a random matrix and minimizes
computations for a particular dimension size [
This method involves taking a high-dimensional data set
and then mapping it into a lower-dimensional space, while
providing some guarantees on the approximate preservation of
distance. If the input data is an n × d matrix, then to do a
projection, we choose a “suitable” d × k matrix R, and then
define the projection of A to be E = AR, which stores
kdimensional approximations for our n points (matrix E is n × k)
[
Steve Vincent (2004) made comparison of principal
component analysis and random projection in text mining. He
found that in general Random projection is faster by many
orders of magnitude over PCA, but in most cases produced
lower accuracy. This lead to suggestion to use Random
projection method if speed of processing is most important. [
Like PCA, factor analysis is also a linear method, based on
the second-order data summaries. Factor analysis assumes that
the measured variables depend on some unknown, and often
immeasurable, common factors. The goal of this method is to
uncover such relations, and thus can be used to reduce the
dimension of data sets following the factor model. [
Factors are assumed to follow a multivariate normal
distribution, and to be uncorrelated to noise [
ICA is a higher-order method that seeks linear projections,
not necessarily orthogonal to each other, that are as nearly
statistically independent as possible. Statistical independence is
a much stronger condition than uncorrelatedness. ICA can be
considered as a generalization of the PCA and the Projection
pursuit concepts. While PCA seeks uncorrelated variables, ICA
seeks independent variables. In contrast with PCA, the goal of
ICA is not necessarily dimension reduction. [
This method specifies the likelihood of the noise-free ICA
model, and uses the maximum likelihood principle to estimate
the parameters. The advantages of this method include the
asymptotic efficiency of maximum likelihood estimates under
regularity conditions. However, it is computationally intensive,
which make it undesirable in many practical situations. [
Probably the simplest way of reducing dimensionality is by
assigning a class (among a total of K classes) to each one of the
observations xn. This can be seen as an extreme case of
dimensionality reduction in which we go from M dimensions to
1 (the discrete class label Ƙ). Each class, Ƙ, has a
representative xƘ which is the average of all the observations
assigned to that class. If a vector xn has been assigned to the
Ƙn‐ th class, then its approximation after the dimensionality
reduction is simply xn = xƘn. Widely used K‐ means method also
belongs to this group of dimensionality reduction methods. [
PCA is the perfect tool to reduce data that in their original
M‐ dimensional space lie in some linear manifold. However,
there are situations at which the data follow some curved
structure. In this case, approximating the curve by a straight
line will not perform a good approximation of the original data.
For such type data the solution is to use principal curves,
surfaces and manifolds. [
Curve fitting to data is an important method for data
analysis. When we obtain a fitting curve for data, the
dimension of the data is nonlinearly reduced to one dimension.
[
PCA is ineffective in analyzing nonlinear structures, i.e.
curves, surfaces or clusters. In such cases Projection pursuit
method might be a solution. The goal of projection pursuit is to
find a projection that reveals interesting structures in the data.
“Interesting” is defined as being “far from the normal
distribution”, i.e. the normal distribution is assumed to be the
most uninteresting. The degree of “far from the normal
distribution” is defined as being a projection index. There are
various standards for interestingness. [
Given n items in a d-dimensional space and n x n matrix of
proximity measures among the items, multidimensional scaling
(MDS) produces a k-dimensional, k ≤ d, representation of the
items such that the distances among the points in the new space
reflect the proximities in the data. [
The proximity measures the (dis)similarities among the
items, and in general, it is a distance measure: the more similar
two items are, the smaller their distance is. Popular distance
measures are Euclidian distance, the Manhattan distance and
the maximum norm. [
Self‐ Organizing Maps (SOM) are generalizations of the
vector quantization approaches presented above. SOM work by
assigning to each input vector a label Ƙn corresponding to the
class closest to its representative vector. Kohonen’s SOMs start
by creating a set of labels on a given manifold (usually a
plane). Labels are distributed in a regular grid and the
topological neighborhood is defined as the neighbors in the
plane of each point of the grid. [
Support vector machines (SVMs) have been recognized as
one of the most successful classification methods [
Important comparison of different methods was made by R. S. Rosaria, I. Adae et. al. (2014). They concentrated on a few state-of-the-art methods to reduce input dimensionality and examined how they might affect the final classification accuracy. In particular, they tried removing data columns with too many missing values and used low variance filter that calculates each column variance and removes those columns with a variance value below a given threshold. They also reduced highly correlated columns and implemented dimensionality reduction via tree ensembles (Random forests). But the best results were got by using backward feature elimination and forward feature construction together with removing data columns with too many missing.
The Backward Feature Elimination loop performs
dimensionality reduction against a particular machine learning
algorithm. Similarly to the Backward Feature Elimination
approach, a Forward Feature Construction loop builds a
number of pre-selected classifiers using an incremental number
of input features. The Forward Feature Construction loop starts
from 1 feature and adds one more feature at a time in the
subsequent iterations. [
III. ALGORITHM ADOPTION FOR PARALLEL COMPUTING In this section we briefly describe the MPI technology that is used for parallel computing in computer cluster. We also present the algorithm of selected dimensionality reduction method which is adapted for MPI.
A. MPI
MPI is message-passing library interface specification. MPI addresses primarily the message-passing parallel programming model, in which data is moved from the address space of one process to that of another process through cooperative operations on each process. [7] CPU CPU CPU CPU CPU CPU
Network
Memory Memory CPU CPU CPU CPU CPU CPU
According to R. Rabenseifner, G. Hager, etc. (2009), most systems in high-performance computing feature a hierarchical hardware design: shared memory nodes with several multi-core CPUs are connected via a network infrastructure.
The comparison between MPI and hybrid models was made by F. Cappello, D. Etiemble (2000). Their test results showed that a unified MPI approach is better for most of the benchmarks.
Rifat Chowdhury (2010) designed an algorithm using the libraries of OpenMP to compute Matrix Multiplication efficiently. The speedup was close to the ideal 4 times using four cores of a processor to compute the final matrix instead of one. This is an example how parallel computing can increase the performance.
Our goal is to combine speed and accuracy. For initial data dimensionality reduction we selected Random projection method because it is very fast. Moreover we adapted this method for parallel computing. This method can be applied for many purposes. For example, G. E. Dahl, J. W. Stokes et. al. successfully used Random projections for large-scale malware classification to further reduce the dimensionality of the original input space. E. Bingham and H. Mannila used Random projection as dimensionality reduction method tool in cases of processing both noisy and noiseless images, and information retrieval in text documents. In our case we use this method for financial data mining.
To use Random projection method, we firstly declare 3 matrixes: n × d matrix1 (contains initial financial data), d × k matrix2 (operating matrix) and n × k matrix3, which is our results matrix. During the next step we initialize random integer values to all elements of matrix1. Then we assign values of 1 or -1 to matrix2 with probability 0,5 each. Then matrix1 and matrix2 are multiplied and results are assigned to the elements of matrix3.
The code is constructed accordingly to MPI requirements (Fig. 3). Program includes MPI library. All matrices and variables are declared in serial code region, but MPI environment is initialized by special functions before parallel region begins. The program uses determined amount of threads (“ranks”). “Zero“ thread is used to divide multiplication task into multiple processes and handle incoming results from the remaining threads. All computing operations are simultaneously done by multiple threads.
In this section we present the results of applying Random projection and Multidimensional scaling methods for our financial market data set.
The data set contains information about 1400 companies that are traded on NASDAQ stock exchange. Every company is described by 54 different parameters. All these parameters are grouped into 6 categories: overview, valuation, financial, performance, technical and ownership. All data is from finviz.com website.
In the Overview group there are such parameters as sector, industry, market capitalization, price, volume. In Valuation group we have P/E, PEG, P/B. EPS and sales parameters. Financial parameters group contains ROA, ROE, ROI, earnings, debt and margins parameters. Performance indicators present price changes during different periods of time, volatility and recommendation values.
Technical indicators are ATR, Beta, Simple moving averages of different periods of time, RSI, the lowest and highest prices at which a stock has traded in the previous 52 weeks. And the last group contains parameters of stock ownership.
Usually our goal is to visualize data and uncover hidden information. This means that for example in this case we have to reduce the dimensions from 54 to 2 or 3. It’s complicated to quickly perform such tasks with single machine. So in the first step we executed Random projection algorithm in the cluster having 120 nodes to reduce the dimensions from initial 54 to smaller amount: 25, 15, 10 and 5 (got several new data sets). When the most suitable number of threads were chosen it took less than 0,04 seconds to complete each task (Figure 4). It should be noted that using more threads not always leads to better performance. As this data set was relative small, 10 to 30 threads was optimal choice to execute the algorithm in the fastest way. In comparison with serial code, MPI code worked from 10 to 20 times faster.
In the second step we applied Multidimensional scaling method (with Sammon) to visualize the reduced data set containing 5 variables. We investigated two stock classification cases: based on analysts’ recommendation and based on multicriteria indicator. A. Stock classification based on analysts’ recommendation
One of our initial 54 factors was the recommendation of financial analysts. This variable ranges from 1 (strong recommendation to buy stock) to 5 (strong recommendation to sell stock). We raised hypothesis that “good” companies (having recommendation ratio from 1 to 2) should be separated from the rest companies after dimensionality reduction and visualization processes.
However, figure 6 shows that the combination of Random projection and Multidimensional scaling methods couldn’t separate these two classes. They just overlapped each other. We tried to use 3D plot, but it was uninformative too (Fig. 7).
The first thought could be that the methods are not efficient enough. But it also could be that in reality the potentiality of “good” and “bad” companies (as determined by analysts) don’t differ so much. This would explain why using just the opinion of financial advisors often leads to unstable returns or even losses. It also suggests that opinions might be more based on intuition than various stock ratios.
In the second case we constructed multicriteria indicator for separating stocks that are worth investing (“good”), neutral stocks and “bad” stocks.
In our assumption “good” stocks meet all following criteria: P/E < 20, EPS this year > 5, ROE > 5, 50-period SMA > 0, RSI > 50. Table 1 shows the ratios by which “good” and “bad” classes of stocks are constructed. “Neutral” stocks are those which don’t fall within any of previously defined two classes.
We used the same Multidimensional scaling method to visualize the reduced data set, so the arrangement of data points is the same as presented in figure 6. But in this case we found that the three different classes of stocks were separated quite well.
Figure 8 shows that “good” companies were presented in the left side of the plot. The “bad” and “neutral” companies overlapped, but “bad” companies separated from “good” companies.
This leads to suggestion, that dimensionality reduction and visualization methods can effectively classify data and find the most promising stocks. However, in order to explain the differences between classes we need to use several different ratios.
We used the combination of Multidimensional scaling and Random projection methods because the latter one is very fast. However it might be not so accurate. So we tried to apply MDS method for full initial data set and it actually led to better results. 3D plot in figure 9 shows that “bad” stocks were separated from “neutral” stocks. Fig. 9. Visualizing initial data set using MDS
Various data mining methods are used for examining large financial data sets to uncover hidden and useful information. This research focused on financial data visualization that is based on dimensionality reduction methods. We used data set that contained financial ratios of stocks traded on NASDAQ stock exchange. A brief overview of the most popular data dimensionality reduction and visualization methods was presented in this paper. We also showed how to adjust the algorithm of Random projection method for parallel computing. The MPI technology was applied in computer cluster to perform dimensionality reduction. The performance results revealed the advantages of parallel computing. Our goal was to visualize data and uncover hidden information. In order to do this we had to reduce the dimensions to 2 or 3. In the first step we executed Random projection algorithm in the cluster to reduce the dimensions from initial 54 to smaller amount. In the second step we applied Multidimensional scaling method to visualize the reduced data set.
One of the data set ratios was the recommendation of financial analysts. We raised hypothesis that companies having recommendation to buy them should be separated from the rest of companies. But the results showed that the combination of Random projection and Multidimensional scaling methods couldn’t do this. This might have happened because in reality the potentiality of “good” and “bad” companies (as they are determined by analysts’ recommendations) doesn’t differ so much. However, in the second case of stock classification based on multicriteria indicator “good” and “bad” stocks were separated quite well. This leads to suggestion, that dimensionality reduction and visualization methods can effectively classify data and find the most promising stocks. But in order to explain the differences between classes we need to use several different ratios. It should be also noted, that MDS method alone was more accurate than combination of two methods.
MPI technology. 3.1/mpi31-report.pdf
Access: