Traditional stock market analysis is based on the assumption of a stationary market behavior. The recent financial crisis was an example of the inappropriateness of such assumption, namely by detecting the presence of much higher variations than what would normally be expected by traditional models. Data stream methods present an alternative for modeling the vast amounts of data arriving each day to a financial analyst. This paper discusses the use of a framework based on an artificial neural network that continuously monitors itself and allows the implementation on a multivariate financial non-stationary model of market behavior. An initial study is performed over ten years of the Dow Jones Industrial Average index (DJI), and shows empirical evidence of concept drift in the multivariate financial statistics used to describe the index data stream.
Data streams are generated naturally within several domains.
Network monitoring, web mining, telecommunications data
management, stock-market analysis and sensor data processing are
applications that have vast amounts of data arriving continuously. In such
applications, the process may not be strictly stationary, i.e., the target
concept may change over time. Concept drift means that the concept
about which data is being collected may shift from time to time, each
time after some minimum permanence [
In this paper we address the detection and analysis of concept drift
in financial markets by employing a methodology based on Artificial
Neural Networks (ANN). ANN are a set of biologically inspired
algorithms and well-established data mining methods popular for
technical market analysis and price predictions. We are currently
undergoing a wider research on using ANN in Ubiquitous Data Mining.
This work, in essence, is a real-world application of a mechanism to
detect concept drift while processing data streams. The motivation
for this approach in the financial field can be easily explained.
Mathematical finance has made wide use of normal distributions in stock
market analysis to maximize return rates, i.e., they assume
stationary distributions, which are easier to understand and work well most
of the times. However, this traditional approach neglects big
heavytails, i.e.,huge asset losses, in the distributions and their potential risk
evaluation [
These contributions adhere to the impositions of data stream
models in [
At the end of the paper, Section 4, discussion of the results are made together with final conclusions. 2
The presented methodology for drift detection comprises two modules. The first module uses an ART network that receives the incoming stream and produces aggregations, or data synopsis, compressing the data and retaining the intrinsic relationships within the distribution (Section 2.1). This module feeds a second module that takes a fixed set of these aggregations and through simple computations produces an output that can be used to detect concept drift. 2.1
One should point out that algorithms performing on data streams are
expected to produce “only” approximated models [
Hence, we create q “weighted” prototypes of data stored in tuples Q = {M1, ..., Mj , ..., Mq}, each containing: a prototype of data Pj ; the number of inputs patterns Nj assigned to that prototype and a timestamp Tj that contains the point in time that prototype was last accessed, hence Mj = {Pj , Nj , Tj }. The prototype together with the number of inputs assigned to it (weighting) is important to preserve the input space density if one is interested in creating offline models of the distribution. The timestamp allows the creation of models from specific intervals in time.
ART [
More formally, a data stream is a sequence of data items
(observations) x1, ..., xi, ..., xn such that the items are read once in
increasing order of the indexes i. If each observation contains a
set of d-dimensional features, then a data stream is a sequence of
X1d, ..., Xid, ..., Xnd vectors. We employ an ART2-A [
The degree of similarity is limited to the range [
The constant learning rate η ∈ [
η = NJ (3)
NJ + 1
This does not guarantee the convergence to local minimum,
however, according to the adaptive vector quantization (AVQ)
convergence theorem [
Another needed modification arises from the fact that ART networks, by design, form as much prototypes as needed based on the vigilance value. At the extremes, ρ = 1 causes each unique input to be encoded by a separate prototype, whereas ρ = 0 causes all inputs to be represented by a single prototype. Therefore, for decreasing values of ρ coarser prototypes are formed. However, to achieve exactly q prototypes solely on a manually tuned value of ρ is a very hard task, mainly due to the input space density, that can change over time, and is also different from application to application.
To overcome this, we make a modification to the ART2-A algorithm to impose a restriction on creating a maximum of q prototypes and dynamically adjusting the vigilance parameter. We start with ρ = 1 so that a new micro-cluster is assigned to each arriving input vector. After learning an input vector, a verification is made to check if q = j + 1, where j is the current number of stored microclusters. If this condition is met, then to keep only q we need to merge the nearest pair of micro-clusters. Let Tr,s = min{kPr − Psk2 : r, s = 1, ..., q, r 6= s} be the minimum Euclidean distance between prototypes stored in micro-clusters Mr and Ms. We merge the two micro-clusters into one:
Mmerge = {Pmerge, Nr + Ns, max{Tr, Ts}} with the new prototype being a “weighted” average between the previous two:.
Pmerge =
Nr Nr + Ns
Pr +
Ns Nr + Ns
Ps
With d-dimensional input vectors, Equation 1 defines a hypersphere around any stored prototype with radius r = (1 − ρ) · √d. By solving this equation in respect to ρ, we update the vigilance parameter dynamically with Equation 6, hence ρ(new) < ρ(old) and the radius, consequently, increases.
ρ(new) = 1 − T√r,s (6) d
Our experimental results show that this approach is effective in providing a summarization of the underlying distribution within the data streams. The inclusion of these results is out of the scope of this paper.
We must point out that the aggregation module produces more information that it is actually necessary for the concept drift detection, (4) (5) namely the weighting of the prototypes and the timestamps. This module is an integrating part of a larger framework that also generates offline models of the incoming stream for specific points in time. 2.2
Our method assumes that if the underlying distribution is stationary
that the error-rate of the learning algorithm will decrease as the
number of samples increases [
We use a queue B of b aggregation results, such that B = {Ql, Ql−1, ..., Ql−b+1}, where Ql is the last aggregation obtained. For each Ql that arrives, we compute the average Euclidean distance between each prototype Pi in Ql and the closest one in Bl−1 = {Ql−1, ..., Ql−b+1}. Equation 7 formalizes this Average Quantization Error (AQE) computation for the lth aggregation, where k · k2 is the Euclidean distance and q is the number of prototypes in Ql by definition. This computes the error of the last aggregation in “quantifying” previous aggregations in a particular point in time.
q AQE(l) = 1 X q i=1 min( k Pi − Pj k2, ∀Pj ∈ Bl−1 ) (7)
By repeating this procedure over time, we obtain a series of errors that stabilizes and/or decreases when the underlying distribution is stationary and presents increases on this curve when the underlying distribution is changing, i.e., concept drift is occurring. This series of errors is the drift curve.
Larger values of b are used to detect abrupt changes in the underlying distribution, whereas to detect gradual concept drift a lower value should be adopted. We exemplify the automatic concept drift detection in this drift curve using a moving average in Section 3.2. 3
We present an application of the previous methodology to the stock
market, namely to the Dow Jones Industrial index (DJI). Instead of
using daily prices of several stocks that compose the DJI, our
approach to this problem uses the DJI daily index values themselves
and other computed statistical and technical indicators, which are
explained in Section 3.1. We make extensive use of moving averages,
as they reduce the short term volatility of time series and retain
information from previous market events; another statistical indicator
is the Hurst index [
This application makes use of data gathered from the period comprised between the 1st of January of 2001 to the 31st of December of 2011, in a total of 2767 observations. 3.1
The data gathered was composed by a set of technical variables including different index values for one trading day like Open, Close, High and Low values. From these we chose the lowest daily price (PX LOW) because it provides better insight to the risk of a fall. Other available technical indicator was the trading Volume.
In terms of statistical indicators, we initially considered a large number of them, like moving averages (MA) from 20 to 180 trading days, relative numbers, i.e., the DJI index value divided by moving averages (AVG), price fluctuation and Hurst index. However, it was important to reduce the number of variables because redundant variables can reduce the model efficiency. For this purpose we performed an analysis with the VARCLUS procedure (SAS/STAT).The VARCLUS procedure can be used as a variable-reduction method. The VARCLUS procedure divides a set of numeric variables into disjoint or hierarchical clusters through principal component analysis. All variables were treated as equally important. VARCLUS created an output was used by the TREE procedure to draw a tree diagram of hierarchical clusters (SAS/STAT R 9.1 User’s Guide p. 4797). The tree diagram is depicted in Figure 1. We can observe in the hierarchical clustering that the price variables and moving averages are correlated, so it was only chosen PX LOW of Cluster 1. In Cluster 2 all variables were selected because, although they are correlated, they measure different characteristics. In the case of relative numbers different averages were selected because it is interesting to see the differences between the analysis of short, medium and long term. Finally in Cluster 3 and Cluster 4 just Hurst index and price fluctuation appeared, because they are not correlated with any other variable, so these variables were included in the final data set.
Hence, the complete set of features in the data stream is the following:
PX VOLUME: Volume of daily business; IX HURST: Hurst index computed for 30 days; IX CAP FLUTUATION: PX LOW(t)/ PX LOW (t 1). This variable represents price fluctuation for one day interval; AVG 20: PX LOW / 20-day moving average. This variable represents the relative number of current price divided by the 20-day Moving Average. This shows whether the current price is cheap, average value, expensive or really expensive. The same applies to the next indicators but within other time frames;
AVG 60: PX LOW / 60-day moving average; AVG 100: PX LOW / 100-day moving average; AVG 120: PX LOW / 120-day moving average; AVG 180: PX LOW / 180-day moving average;
The dataset is depicted in Figure 2, where the behavior of all
variables can be seem. This data is our data stream. The stream comprises
10 features, e.g., a 10-dimensional stream.
The methodology presented in Section 2 was applied to the above
data. It is converted into a data stream by taking the data input
order as the order of the streaming. All features were previously
normalized to the range [
However, since we are dealing with financial time-series, it is important to retain the time dependency of the sequence of observations. Therefore, in this application, we use a sliding-window of 100 trading days, i.e., approximately a trimester of trading as input to each aggregation phase. Note that a year of trading has approximately 260 days. This means that the stream is processed in blocks of 100 observations that are kept in a queue. For each new observation that arrives the oldest in the queue is discarded and the new one added. The parameterization used was the following: Block size: S = 100; Number of micro-clusters: q = 10; Concept drift buffer size: b = 15
The result of the procedure of Section 2.2 applied to the data stream is presented in Figure 3. Each point of the series corresponds to the error of the model for a particular trading day, thus providing possible indications of drifting. It can be seen an overall shape of a curve that indicates the drift over time. Since this drift is being computed for every trading day, the “noise” around the curve is considered normal since it is affected by the daily volatility of the index values.
To obtain a “clean” curve we apply a convolution filter along this drift series of the same size as b, i.e., 15 days. An alarm scheme is created through the generation of an empirical moving average of 60 days performed over the drift series. The cleaned drift curve and its moving average are depicted in Figure 4a).
We then compare the differences between the drift series and its moving average obtaining a line that oscillates around zero. We call this line the drift trend, shown in Figure 4b). Whenever the drift series has values lower than its moving average we are in a descending trend. This is reflected in the drift trend with values lower than zero. Whenever the moving average is crossed by the drift series it signals a shift in the trend and the drift trend crosses zero. This reasoning to detect trends is also very popular in financial technical analysis. In this context, the 60 trading days moving average reflects the intuitive notion of long-term “decreasing” or “increasing” trend of the drift.
All plots in Figure 4 are aligned in time for easy comparison. Figure 4c) shows the time series of PX LOW, i.e., the DJI index, that we compare to the detection of drift performed. 4
Based on experiments we found that a tenth of prototypes relative to the number of observations are sufficient in most applications to represent them adequately, hence, q = 10. Usage of higher values of q did not improve the results with the additional problem of increased computational time. Additionally, since we are both interested in abrupt and gradual drift detection we used a moderate sized buffer of aggregations (b = 15) to compute the series of quantification errors. During our experiments we found that this value was appropriate for the established goals.
By inspecting Figure 4 and comparing the drift trend with the behavior of the DJI index we can make two important observations: (i) the drift trend crossed zero before the market crash of 2008 (around day 1500). It appears that the concept that was being learned changed sometime before the crash occurred. (ii) it may be reasonable to assume that in periods of normality the long-term tendency of these indexes is upwards. One of such periods is after the recovery of the 2002 market crash, i.e., the dot-com bubble, until the other crash of 2008 (approximately between days 300 and 1300). During such period it is interesting to see that the drift trend was always below zero.
In the present work we have shown a methodology to detect concept drift in financial markets. We intend to apply this same methodology to intra-day trading as soon as it is possible, thus reinforcing the need to efficient processing of large volumes of data. The proposed methodology applied over a data stream comprised of carefully chosen technical and statistical indicators seems promising in detecting changes in markets events ahead of time that can reduce the exposure to risk.
The characterization of the drifts, i.e., trying to understand what is
really changing in the markets through inspection of hidden changes
in the indicators is reserved for future work. Work is under way in
this subject and we are using Self-Organizing Maps [