Devising Equity markets forecasting is relevant not only for the parties directly involved in price formation (e.g., companies and investors), but also for policymakers and regulators. Central banks have shown a growing interest in modelling equities to decide macro prudential policy, assess investors' attitude towards risk, and in some cases deploy capital in the market directly. For example, according to the 30th of June 2022 13F SEC filings 1 , the Swiss National Bank owns about $11.11 billions of Apple and $7.49 billions of Microsoft.
This study is driven by the need to review machine learning techniques applied to equity markets time series forecasting problems, with the objective to provide an overview for practitioners, highlight areas under explored and warranting further research.
This study explores research from finance and computer science, from industry and academia. As such, the selection of papers for this literature review is not purely bibliometric. We aim to cover a selection of high impact publications and original contributions from the field for the critical steps of the machine learning deployment process, from pre-processing to algorithm selection. We cite papers that are now part of history of financial theory for the benefit of non-experts. Research from other domains is included where it is reasonable to assume that the related techniques could be ported to financial time series forecasting.
The contributions of this study are as follows:
The non-stationarity and non-linearity of financial variables are the primary attributes to estimate a forecasting model for equity markets.
Under the Efficient Markets Hypothesis (EMH) developed by Fama [1], the sequence of price changes must be unpredictable if they fully incorporate information and expectations of market participants. Yet, in the real world, analysing past returns and processing publicly available information are instrumental to build a forecasting model. Jagadeesh and Titman [2], showed that it is possible to generate statistically significant abnormal returns, by buying the stocks that exhibited top decile returns in the past (i.e., winners) and selling stocks in the bottom decile, losers. Malkiel [3] documented several market anomalies that increase predictability; however, these inefficiencies do not persist.
If market participants firmly believed in EMH, it would be irrational to trade, hence there would not be a financial market. Grossman and Stiglitz [4] offer the following solution: the information derived by skilled investors is not entirely reflected in the market. As a result, investment research is compensated. Bauer et al. [5] showed skill persistence amongst retail option traders: top decile performers over one year outperform individuals in the bottom decile the following year.
Traditional financial theories assume that market participants are rational and with the same utility function. When these assumptions are violated, behavioural biases such as dissimilar risk appetite and inconsistent probability beliefs play a significant role to account for abnormal profit opportunities. Grinblatt et al. [6] showed that individuals' personal traits and sentiment tend to affect trading behavior. Bernstein [7] points out how financial markets undergo regime shifts through an evolutionary process, which explains the market dynamics.
The Adaptive Market Hypothesis (AMH) [8] offers an alternative framework: different agents interact and adapt in response to ever changing market environment, competing to capture returns opportunities.
Machine learning, being adaptive in nature, has the set of tools to conduct research in the new paradigm: models are non-parametric in nature and have the capability to approximate complex non-linear, non-continuous functions.
The emergence of big data gives rise to a paradigm shift in the field of financial engineering where the state-ofthe-art techniques and data processing capacity are of utmost importance. Not only newsflow, economic and stock market data are widely available in copious size, however, unstructured data (e.g., geo-location), can be accessed at any time, processed in different ways. As a result, investors equipped with the latest technological advancement can make use of an extensive set of features. Machine learning algorithms do not require data transformations to make a time series stationary, being able to learn complex patterns in high dimensions.
Quian [9] showed the outperformance of several machine learning algorithms (logistic regression, Neural Network (NN), Support Vector Machines (SVM), denoising Autoencoders) compared to ARIMA. Gu et al. [10] compared ordinary least squared regression to tree-based methods and NNs on the task of predicting equity excess returns for US stocks. The authors ascribe the superior performance of machine learning techniques to the ability to accommodate a large number of predictors and learn non-linear relationships between covariates.
In this section, different machine learning algorithms and their applications to stock market time series are presented. Figure 1 depicts the ML algorithms and their related references. Here the literature is reviewed focusing on the main steps required for an end-to-end machine learning application, rather than looking, almost exclusively, at the different algorithms deployed.
Framing time series forecasting problems as regression might present an issue in the financial domain: the prediction might be very close to the actual value and yet induce the wrong action. For example, given a value of the S&P 500 of 4200, assume the 1-day forward forecast generated by two different models to be 4210 and 4150 respectively, with the actual future level being 4190: while the first model is more accurate, it will suggest entering a long position causing a loss. The second model, despite being further away from the observed value, would trigger the correct (profitable) action, establishing a short position.
In several time series classification studies, the labeling is conditional on a gating value g: an instance is labelled +1 if the response variable π¦ > π, -1 otherwise. For example, Ballings et al. [11] compared the performance of ensemble methods against single classifiers on the task to group European shares based on whether they will be up at least 25% in a year. Dixon et al. [12] framed securities time series problems as multiclass attributing +1 to a positive movement, -1 to a downward direction and 0 as flat market, setting a value to determine 'no action' in order to balance the classes. Labelling examples, using an arbitrary fixed threshold does not take into account that returns are realised in a specific market regime. Defining for example a threshold of Β±5% to determine the bounds of the 0 class, might be appropriate for a quiet market environment, however, it could be too small in a high volatility phase. Research should be conducted to scale raw data in ways that would result in predictors that carry information content related to the respective market regime.
Accuracy in time series classification problems might not reflect profitability if a model fails to capture the correct direction for large moves. For example, consider the sequence of returns +0.8%, +1%, +0.5%, -3%, and two binary classifiers (+1, -1): the first predicts four upwards movements, it is therefore 75% accurate. The second classifier outputs -1, -1, +1, -1: it is 50% accurate. If the outputs were to be fed into a trading system, the first classifier would generate a 0.7% loss while the second would yield a profit of 1.7%. An approach to mitigate this issue, suggested by industry practitioners [13] is to train a model to predict returns and then value it based on direction accuracy (also called hit ratio).
Despite the potential of machine learning algorithms to extract knowledge in many dimensions, several studies do not go beyond using lags of the target series as predictors. For example, Fischer and Krauss [14] used only percentage changes of adjusted prices of the S&P 500 index constituents.
An interesting feature engineering technique that tries to capture both past observations and market regime can be seen in [15]. The authors implement a relative change transformation of the predictors in each data window: each element in a sample is subtracted and divided by the first observation of the sequence. As a result, the network will be learning from a price change that is put in the context of the short-term market environment in which it was generated. Possible avenues of research would be to devise ad hoc methods to rescale data dynamically.
The overwhelming majority of the papers surveyed in this study employs technical indicators [16] as additional features. Patel et al. [17] ran experiments on Indian equities using different representations of technical indicators: in their original form and in discrete form. The latter, called "trend-deterministic," is obtained by discretising continuous values of oscillators (e.g the Relative Strength Index, RSI) in two categories: +1, -1. Their study shows that the trend deterministic representation improves results irrespective of the algorithm.
Sentiment analysis is performed to take the pulse of the market participants' emotional state. Extracting sentiment from social media, particularly for short time frames applications has gained paramount importance given the rise of retail traders, whose decisions combined can have a major impact on price formation. Sentiment indicators can be the direct result of investors polls like for example, the American Association of Individual Investors (AAII) survey. The Investopedia portal2 computes the Investopedia Anxiety Index (IAI) based on their readers' interests for topics such as macroeconomics, negative market and credit disruptions.
Schumaker and Chen [18] proposed a model to predict the intraday share price of the companies after a piece of news was released. They used as predictors the price forecast using a linear regression model and features derived from text analysis of news articles. Kazemian et al. [19] deployed an SVM to classify the sentiment extracted from financial news on US stocks. Bollen et al. [20] showed that the emotional state, particularly the 'Calm' dimension, of the broad population has predictive power on the daily changes in the DJIA. Othan et al. [21] proposed a model to predict the direction of Turkish stocks based exclusively on sentiment extracted from Twitter posts.
It is noteworthy that these sentiment centred techniques aim to output short term predictions. It would be interesting to extend this research over longer time horizons, using texts from commentators with domain knowledge, for example analysing blog posts from the likes of "Seeking Alpha".
Cho et al. [22] simulated an investment strategy based on the text characteristics of equity research reports published by brokerage houses. The authors deployed Part of Speech tagging to engineer features such as: numbers of nouns, adjectives, number of sentences. Classifiers, using the text extracted features, were trained on the task of recognising successful buy recommendations on Korean equities. Fundamental analysis does much more than assessing the intrinsic value of an asset or company given that it includes both macro-econometrics and micro-econometrics approaches. As such it focuses on financial statements and macroeconomic data. Specific attention is warranted to avoid look-ahead bias: new data releases and revisions should enter the time series only when the updated information is disclosed. For example, quarterly GDP data, after the initial release, are later revised twice to get to the final figure . Only a limited number of studies in the field adopt fundamental data; this is probably due, on one hand to the aforementioned issues, and on the other to the low frequency nature. Olson and Mossman [23] studied predictive modelling of Canadian stock excess returns and direction using as inputs 61 accounting ratios. Tsai et al. [24] studied the application of ensemble learning methods to Taiwanese stock market quarterly direction forecast. Predictors included company financial ratios and macroeconomic indicators.
Fundamental reported data are backward-looking. Alberg et al. [25] and Chauhan et al. [26] while studying neural network applications to factor investing [27] in US stocks, built a predictive model of companies' fundamental metrics. Stocks were then ranked and picked to form portfolios based on the forecasted fundamental values. These studies show that investing based on predicted company data outperforms stock selection based on reported information. An alternative solution could be using, as predictors consensus forecast figures from the financial analyst community. This approach would also allow sampling data with a frequency higher than the quarterly earnings cycle.
As summarised in Table 1, technical features can be easily retrieved from data vendors or calculated from price series. They are mostly used by practitioners to have short term timing indications, albeit with mixed results. Studying sentiment allows to read the emotional state of market participants. Sentiment data is mostly unstructured therefore, requires specific Natural Language Processing (NLP) tasks. Fundamental data are low frequency and potentially subject to the dangers of lookahead bias. However, this kind of data reveals the state of the economy. A predictive model aiming to generate a medium-or long-term forecast should take fundamental variables in consideration.
The
Ensemble methods prescribe combing multiple algorithms to achieve better performance than inducing only the best member. Ensemble architectures are determined by how a population of diverse models is deployed and by the choice of aggregation mechanism to derive the final prediction. The Random Forest (RF) algorithm [28] represents a prototypical example.
In Krauss et al. [29], a statistical arbitrage strategy is simulated after an ensemble architecture with majority voting (composed by a deep neural netwowk, gradient boosted tree and RF) has classified S&P 500 constituents according to the probability to outperform the index. Yuan et al. [30], deployed NN, SVM and RF, to classify Chinese stocks based on belonging to the top 30% of companies in generating excess return over one month.
The authors highlight an issue arising adopting standard k-folds cross validation for time series problems: future data might be used to predict past observations, overstating the performance. They evaluate algorithms using a sliding window train and test set instead. Evaluating a model on periods prior to the training set might be desirable in certain cases: for example, for stress testing purposes.
The Support Vector Machine (SVM) [31] is a very effective algorithm able to solve non-linearly separable problems by enlarging the feature space thanks to the "kernel trick". Using the kernel functions allow to solve the problem in the original dimension without having to transform the data in a more complex space.
SVMs were compared in [32] to linear and quadratic discrimination analysis, and Elman network to forecast the weekly direction of the Nikkei 225 index. Zbikowski [33] proposed to consider volume data by plugging this information directly in the SVM formulation, multiplying the hyper-parameter representing the cost of margin violation, C, by a coefficient v based on the transaction volume in a given security over the input window. Patel et al. [34] deployed a 2 stages model with the task of predicting forward closing prices. Their architecture comprises a Support Vector Regressor (SVR) to predict the features' values, and then feed this output into a different model to produce the estimated index close price.
SVM incurs in high computational costs with a large dataset. The adoption of deep learning has grown in popularity thanks to the capability to overcome SVM scalability problems without sacrificing performance. Chong et al. [35] run experiments on stocks listed on the Korean market to make intraday returns prediction (5 minutes). The authors considered a network to be deep with just three hidden layers. Over the subsequent few years, structures significantly more complex will be developed. The impact of the network depth on a financial time series classification task performance was studied in [36].
A Recurrent Neural Network (RNN) processes at any given stage as inputs information from that time step, and a hidden state derived from the previous one. Dixon et al. [37] show that a RNN, with linear activation, can be assimilated to a Auto Regressive time series model. The output of a given time step can be written as: ππ‘ = π (ππ₯π₯π‘ + ππ¦π¦π‘β1 + π).
The Long Short Term Memory (LSTM) cell [38] overcomes the RNNs limitations (exploding or vanishing gradient). A LSTM neuron adds to the recurrent unit hidden state (short term memory), a long-term cell state controlled by different gates. LSTM network can therefore handle long sequences.
A comparison of LSTM to RF, NNs can be seen in [39]: the task in question was the prediction of the direction of 5 components of the BOVESPA index in a high frequency setting. Fischer and Krauss [14] used as feature only the return series of the past 240 observations to classify 1 day forward returns, showing the ability of LSTM cells to learn long sequences. Alonso et al. [13] tackled predicting the 1 day ahead return and the related direction of 50 constituents of the S&P 500. This study showed that increasing the number of time steps to train the model from 1 to 10 improves the performance of the LSTM.
Convolutional Neural Networks (CNNs) have been particularly effective in computer vision. Convolution layers exhibit equivariance to translation; in time series data this property allows the algorithm, to detect patterns within a sample and recognise them, notwithstanding at which point in time they appear. Convolution filters over 1-dimension (1D) can be deployed to slide across time, deriving hidden patterns within samples. 1-dimensional convolution layers can be deployed as feature extractors before LSTMs, forming a CNN-LSTM architecture proposed in [15].
The dilation rate d of a convolution layer indicates the frequency of the number of elements in the input to which apply a convolution. Researchers from DeepMind conceived an architecture called WaveNet [40], able to achieve state-of-the-art performance in several text to speech tasks without the use of RNNs. It is obtained by stacking several 1D convolution layers, doubling the dilation rate at every layer. Thus, the receptive field is larger, albeit without additional parameters. The initial layers learn short term patterns, while as data progresses towards the output, sequences of longer term are extracted.
Borovykh et al. [41] adapted the Wavenet architecture, using ReLU as activation function, for multivariate financial time series forecasting: 1D dilated causal convolutions run independently for each input time series to be combined before making the prediction. Experiments were run on several financial instruments, on 1 day forward return prediction tasks. Wavenet and LSTM performed equally in terms of forecasting the direction, however, Wavenet outputs more accurate point estimation. Borjesson et al. [42] proposed a WaveNet inspired model, using as activation function the Scaled exponential Linear Unit (SeLU) for the convolution layers. They ran experiments with the goal of predicting the next day price and trend of the S&P 500.
Convolutions over two dimensions (2D) could be used to extract the most salient features and the most significant sub sequence in a sample. This approach, recently emerging in the literature, offers the advantage to learn jointly patterns within each predictor time series, and the relationship between features. This could be particularly advantageous with economic time series, where the correlation between variables changes over time, albeit in a recurrent fashion.
Gudelek et al. [43] used 2D convolutions simulating trading strategies on several Exchange Traded Fund (ETFs). Prices were differenced once to mitigate the issues of non-stationarity and transformed to be in a range between +1 and -1. The authors interpret the prediction in this range as confidence values. It would be interesting to conduct further research applying to financial time series tasks, architectures centred on 2D convolutions that have proven to be successful in other domains.
A recent stream of research leverages the success achieved in computer vision directly: time series data is transformed in order to assume the characteristics of pixel values and intensity. Cohen et al. [44] converts a time series classification problem (spotting technical patterns) as an image recognition task; Zeng et al. [45] ported a multivariate time series forecasting problem in US equities to the video prediction domain.
Autoencoders (AE) are models conceived to replicate their inputs without supervision. The design usually prescribes an input layer, an encoding layer with a smaller number of units and a decoding layer of the same size of the input, providing a reconstructed representation. Autoencoders can be applied to financial time series prediction problems to reduce dimensions or noise. Troiano et al. [46] focused on the impact of the feature reduction stage (starting from 40 technical indicators). Denoising Autoencoders were used in [9] in order to create a latent feature representation, adding a noise component that randomly alters the raw data, forcing the algorithm to reconstruct robust inputs. Bao et al. [47] stacked 5 autoencoders connecting the final output to a LSTM network to predict the one step ahead level of several stock market indices.
Generative Adversarial Networks (GANs) [48] are composed by 2 competing models: the generator aims to approximate the data distribution and outputs synthetic samples; the discriminator takes either actual or synthetic data and estimates the probability that the sample in question is genuine. The generator aims to maximise the probability that the discriminator will make a mistake. Zhou et al. [49] applied GANs to make one step ahead predictions in 42 Chinese stocks within a high frequency framework. The generator is an LSTM network, while a 1D-CNN plays the role of the discriminator.
Back testing is a common approach to measure the profitability of a model against past trends in the precise order in which they occurred. GANs could be employed to provide synthetic test sets, as if engineers were forward testing algorithms devised by researchers. A recent architecture fit for this purpose is Time-series Generative Adversarial Network (TimeGAN) [50]. This framework was conceived to capture the time dependent conditional distribution of data. In addition to the unsupervised adversarial loss, the model prescribes the use of a supervised loss based on the original data.
Attention mechanisms, pioneered by Bahdanau et al. [51], allow a decoder to focus at each step of a sequence on the most relevant (encoded) input. The decoder computes a weighted sum of the output of the encoder; the weights are learned by an attention layer, using as inputs the encoder output concatenated with the decoder previous hidden state. These techniques, developed for neural machine translation problems, have been applied to time series forecasting: attention mechanism weights differently each time step of a sequence, this is then fed to a forecasting model to derive a prediction. Zhang et al. [52]proposed the AT-LSTM architecture: an attention mechanism using LSTM as encoder, assigns different weights to input features (time steps). Attention weighted sequences are than used as inputs into an LSTM network to output the prediction. A more complex version of this architecture can be seen in [53]: here the input series is first encoded with an LSTM and an attention layer assigns weights to the features at each time step, obtaining an attention weighted features matrix. In the next stage another LSTM based attention mechanism weights the different hidden states across time steps. The final stage prescribes an LSTM bloc to make a prediction using as inputs the output of the previous stage and the target time series. This Dual-Stage Attention RNN (DA-RNN) aims to capture the most important features while learning time dependencies.
So far we have discussed researchers tackling the problems of non-stationarity of financial data, different market regimes and low signal to noise ratio, deploying more and more complex algorithms. Further study should be conducted focusing on feature engineering: rescaling and labelling examples considering the related market environment, therefore avoiding a fixed arbitrary threshold to define classes, would put the data in relation to the context in which they were generated.
Framing the prediction problem in terms of classification is vulnerable to not identifying large movements. On the other hand, developing a model to predict future securities prices could have the pitfall that a fairly accurate point estimate, albeit in the wrong direction, could result in a loss-making course of action.
Research could be pursued in developing alternative approaches in terms of training, for example, conceiving models learning jointly regression and classification tasks.
In contrast with many of the studies reviewed that use as input the target series and technical indicators, further research could be conducted combining features from different domains. A diverse data pool could point towards unexplored avenues in economics research.
The literature reviewed in this study adopts input time series exclusively in tabular form, however, financial professional place considerable value in extracting knowledge from the relationship and interaction amongst different variables. Graph Neural Networks (GNNs) [54] are able to extract knowledge from the interplay between different nodes in a graph, therefore could represent a novel approach, worthy of further research in our field. An exploratory study related to Japanese equities can be seen in [55]. The authors motivated the use of GNNs to leverage inter-market and inter-company relationships.
The main paradigm adopted by both industry and academia when devising a machine learning solution has been to consider a dataset as fixed, focusing on algorithm development. This survey shows that there is potential for progress, keeping the algorithm fixed, placing the data at the centre of the research process instead.
Ensemble learning approaches combining neural networks have been the winning architecture in image recognition competitions. Similar ideas could be explored, applying diversified network ensembles to financial data. Different models could be trained on different market regimes. One way to deal with a changing environment, is to constantly discard old data and retrain the model with more recent examples. Nevertheless, often, market dynamics observed in the past occur in a similar way at a later stage. Conceiving a way to use past, however, relevant information is an interesting avenue to pursue further study.
While the value of machine learning methods to equity time series forecasting has been shown in the short term, it would be interesting to test further these techniques sampling data with lower frequency. Extending findings to monthly data would graduate machine learning methods to applications beyond the realm of trading.
This survey shows how the playground for machine learning experiment in equities is the S&P 500. Few studies try to corroborate findings extending the research to different countries. European Equities in particular, emerge as an area which remains rather under-explored.
Moreover, given that different algorithms are simulated on different data, it is difficult to assess what could be considered the state of the art. It would be very fruitful for the field if, perhaps as a cooperative project between industry and academia, different financial multivariate datasets ( e.g., one for US equities and one for European shares) would be engineered as standard, in order to provide a more objective common ground to conduct research.
Researchers tackled the problem of equity market forecasting, initially deploying statistical time-series forecasting techniques and then experimenting with complex deep learning architectures.
In particular, LSTM has been proven to be useful in solving the vanishing/exploding gradient problem while offering the advantage of modelling non-linear time series data. Dilated convolutions, on their own, or as feature extractors, constitute an effective technique when dealing with long sequences. C2D centred architectures are an emerging method capable of extracting local knowledge about the interaction of different features.
With this review, we advocate the pursuit of research on every component of the machine learning value chain, rather than focusing exclusively on the algorithmic core. Most studies show results only on a specific market or related to a specific period, thus, the general robustness of findings could be improved. Applying machine learning to financial time series is a challenging, however, rewarding endeavour. Given the importance of the decisions based on equity market forecasts, even a small improvement in model performance can have a major impact.