Day Ahead Forecasting of FAANG Stocks Using ARIMA, LSTM Networks and Wavelets.

Abstract

Facebook Inc., Apple Inc., Amazon.com Inc., Net ix Inc. and Alphabet Inc., known collectively as FAANG, are a group of the best performing tech stocks in recent years. In this study, we present linear and non-linear methods for predicting the closing price of each stock on the following day. We decompose each time series into component series using wavelet methods and develop an novel ensemble approach to improve forecast accuracy.

Keywords

• Machine Learning
• Time Series Analysis
• Neural Networks
• Signal Processing

Introduction

Linear modelling methods such as the Autoregressive model (AR) and Moving Average (MA) models have been prominent in time series forecasting for decades, particularly due to their intuitive nature and robustness in producing short-term forecasts over a wide range of subject matters.

In their seminal textbook [ 9 ], Box and Jenkins, described their method for applying an ARIMA model and nding a best t for a time series. Since then ARIMA models have been employed in modelling diverse series such as in ation in the Irish economy [ 17 ] and failures in repairable systems [23].

LSTM Networks date from 1997 [ 13 ] as an improvement the previous RNN algorithms, which took a prohibitive amount of time to learn long term time lags. They are widely used in natural language processing (NLP) [ 22 ] and speech recognition tasks [ 12 ].

Wavelet decomposition, the theory of using small wave like functions or wavelets to deconstruct a time series into its approximation and detail components, is described in [ 18 ]. Wavelet methods are prominent in signal processing [ 19 ] and time series analysis [ 6 ] and have been used e ectively in multiresolution image processing [ 14 ].

Ensemble approaches combining wavelet decomposition with forecasting methods have been extensively researched with two notable examples:

Youse et al [24] use wavelets to forecast crude oil prices over di erent time horizons using monthly WTI1 spot prices. These forecasts were compared with NYMEX2 oil futures prices and are shown to better predict the future spot prices, challenging the perception the futures market is e ciently priced. The authors decompose the series into wavelet component series and extend them using a sinusoidal model, before reconstructing. They suggest expanding the research to other time series methods for extending the component series such as ARIMA or GARCH.

Liu et al [ 16 ] combine wavelet decomposition and principal components analysis (PCA) to create input features for multiple models. The performance of these models is compared when training and testing on two classical datasets: the arti cial Mackey-Glass equation and the real-world, mean daily ow of the Oldman River near Brocket.

The ve implemented models were as follows: Cascade correlation and back propagation neural networks (CCNN and BPNN) on the underlying the di erenced, stationary time-series. CCNN's on the wavelet decomposed time series, in one instance with all the decomposed series fed into one network, the another with each sub-series as an input to it's own individual network (WCCNN and WCCNN multi-models). The nal network introduces PCA as a preprocessing step on the decomposed series and the rst k principal components that account for 85% of the information are selected as inputs (PCA-WCCNN).

As a result of this work, the researchers suggest PCA-WCCNN as more accurate than WCCNN and quicker to train than WCCNN multi-models. 1 West Texas Intermediate 2 New York Metal Exchange

Methodology

In this section we describe the techniques utilised in our proposed models by describing the mathematical concepts behind ARIMA models and LSTM networks, used in forecasting future values of our selected time series. Finally, we discuss wavelet methods for transforming our series into component sub-series, which we can consider individually or use as a multivariate input to our models.

We de ne a univariate time series X as a time ordered nite set of t observations X = fx1; x2; x3; : : : ; xtg commonly recorded at evenly spaced intervals, with each xi 2 X representing the value of the series at that time. 3.1

Autoregressive Integrated Moving Average (ARIMA) Model

An ARIMA process is a linear combination of p Autoregressive (AR) terms and q Moving Average (MA) terms, modelling a d order di erencing of a time series X [ 9 ]: 1

p X i=1

! iLi (1

L)dxt =

q 1 + X iLi

! i=1 t + c where c is a constant and L is a lag operator, such Lxt = xt 1, L2xt = xt 2 etc.

The selection of appropriate values for the parameters p, d and q is crucial to tting the most accurate model to the time series. The Auto-Correlation Function (ACF) and Partial Auto-Correlation Function (PACF) can be used to provide a reasonable estimate for p [ 9 ].

The Akaike Information Criterion (AIC) is useful for determining the relative performance of a model to another with di erent parameters [ 20 ]. 3.2

Long Short-Term Memory (LSTM) Networks

LSTM Networks are a subclass of Recurrent Neural Network (RNN) that are capable of learning long term dependencies or \long term memory" in time series data [ 10 ] as opposed to RNNs which place more emphasis on recent inputs or \short term memory". We provide a brief description of each but start with a general de nition of an arti cial neural network.

At its core, an arti cial neural network (ANN) is a collection of nodes or neurons, connected by weighted edges or synapses, mimicking the biological neurons in the human brain.Each neuron can receive multiple inputs, update it's current state and emit an output based on an internal activation function, for example a sigmoid or tanh function [ 10 ]. The weights of each synapse determine the in uence the output of neuron, with a low weighted synapse meaning that the output of the preceding node is e ectively ignored.

One limitation of ANN's is that due to their acyclic nature, all inputs and outputs are independent of each other, that is, the predicted value, x^t does not depend on x^t 1 or x^t 2 etc. This is not ideal for time series modelling with a high level of autocorrelation between previous terms.

RNNs address this issue by allowing feedback loops within the hidden layers, where a neuron can store information from current step to make predictions in the following steps. In practice, however, RNN's are only suitable for storing a few previous steps and cannot remember long sequences of data which may better inform the prediction. [ 21 ]

Long term memory is improved in LSTM's with the introduction of gate functions within the neurons [ 13 ] for transforming the state of current step, to be used in the following steps. Three types of gate exist within each node: { Filter Gate: A sigmoid function which determines how much of the previous state should persist in the current state, a value of 1 implies everything should be remembered, while 0 completely disregards the previous state. { Memory Gate: Determines how much of the new data should be added to the current state of the neuron. { Output Gate: Combines the the new data and the current state of the node to produce an output.

As the LSTM is trained it should learn which values in the series to remember or disregard over a longer term than the RNN alone. 3.3

Maximal Overlap Discrete Wavelet Transform (MODWT)

A wavelet transform breaks a time series into components of di erent frequencies [ 7 ] and is better than other decomposition methods as it provides information on the di erent frequencies present in a series while preserving the positions of these frequencies in the time domain. In this section we discuss the fundamentals of wavelet transformation theory and the application of Discrete Wavelet Transform (DWT) and Maximal Overlap Discrete Wavelet Transform (MODWT) to a time series.

There are two main functions involved in wavelet analysis, the wavelet function (mother wavelet ) and the scaling function (father wavelet ). In this paper we consider the Haar and Daubechies of order 4 (db4 ) [ 7 ] wavelet functions, with the Haar father and mother wavelets as follows: (t) = 1; if 0 t < 1 0; otherwise and (t) = 8 > > < > > : 1; 0 t <

1 1;

2 0; otherwise 1 2 t < 1 In general, the mother and father wavelets have the property

Z (t)dt = 1 and

(t)dt = 0 Z In a discrete wavelet transform (DWT) the time series is decomposed using the father and mother wavelet functions :

j j;k(t) = 2 2 (2j t k) and

j j;k(t) = 2 2 (2j t k) Where j 2 [1; J ] in a J -level decomposition is the scale and k is the shift parameter [ 7 ]. The set of these functions provide an orthonormal basis [ 7 ] and as such we can represent our original time series x(t) as x(t) =

X AJ;k J;k(t) + X DJ;k J;k(t) + : : : + X D1;k 1;k(t) k k k The set of coe cients of the wavelet functions in the decomposition, fAJ ; DJ ; DJ 1; : : : ; D1g, is known is a decomposition crystal, with each element Dj = fdj;1; dj;2; : : : ; dj;kg a series in its own right.

The AJ series re ect the approximation component or trend of the series, the Di components represent the detail or deviation from this trend.

The Maximal Overlap Discrete Wavelet Transform (MODWT) is a non orthogonal redundant transform which repeats information in neighbouring coefcients by overlapping time series values [ 7 ]. This allows the decomposed coe cient sets to have the same cardinality as the original series. Similarly, the Inverse Maximal Overlap Discrete Wavelet Transform (IMODWT) takes the decomposition crystal from the MODWT and reconstructs the original series. 4

Proposed Models

In this section we discuss the implementation of the models used in our experiments and the data acquired for training them. The implementation logic for ARIMA and LSTM have been adapted from [ 4 ] and [ 5 ] respectively. 4.1

Data Model

log The time series we focus on the daily closing prices of the shares in FB, AAPL, AMZN, NFLX and GOOG over the period 1st January 2010 to 1st January 2017. The number of observations vary between each stock due a number of corporate actions that occurred during the interval: Facebook Inc. was not listed on the NASDAQ for the entire window, meaning there are only closing prices subsequent to their IPO in May 2012. Alphabet Inc. (aka Google) undertook a corporate restructure in March 2014 and issued a new class of share. It is this new class that is the subject of our analysis. All of these time series were obtained from Alpha Vantage Inc. [ 1 ] and are made available for replication.

These series have been adjusted to account for stock-splits and buybacks [ 2 ] so the transformation required is for the target feature. We are predicting log returns based on the assumption the prices are log normally distributed [ 8 ]. For each stock we have a time series of the form X = fx1; : : : ; xtg where xi = pi pi 1 The models described in the following sections will be used to produce 100 oneday forecasts for the nal 100 terms in each series.

and pi, pi 1 are the current and previous closing prices respectfully. This is our base line model for measuring the relative prediction improvements in subsequence models. Here, the predicted value for xt is the previous day's closing price, denoted x^t = xt 1.

The iterative forecasting process continues until we have a complete set of forecast values fx^t 99; : : : ; x^tg. Two di erent LSTM network topologies were implemented, a 1 node and a 4 node LSTM hidden layer with both having an input layer of 3 nodes. Both are shown in Figure 1.

In order to t the network, the time series X, needs to be transformed into feature and target spaces X~ and Y respectively. In the 3 node input layer, the previous 3 observations are used to predict the current:

X~ = f(x1; x2; x3); (x2; x3; x4); : : : ; (xt 3; xt 2; xt 1)g

Forecasts were produced with the following steps. { Step 1: The training feature and target sets are de ned as X = fx~1; : : : ; x~t 100g and Y = fy1; : : : ; yt 100g. { Step 2: The elements of X and Y are standardised such that all values lie in the interval [ 1; 1]. This is to improve convergence performance and avoid local minima [ 15 ]. { Step 3: The network is trained over 1,000 epochs using the standardised training set X0. The internal memory of the network is reset after each pass to ensure that future observations do not inform the parameter tting. { Step 4: The set X0 is input to the tted network so that the internal memory re ects the time t 100. This memory will be available for use in forecasts { Step 5: The standardisation function for X is applied to the test tuples fx~t 99; : : : ; x~tg which are sequently input into the model to output the forecasts fy^t 99; : : : ; y^tg. { Step 6: The inverse standardisation function for Y is applied to fy^t 99; : : : ; y^tg to produce a complete set of forecast values fx^t 99; : : : ; x^tg. 4.5

WAV-ARIMA

An extension to the ARIMA model, this method applies the MODWT to the series before applying the ARIMA model as above. The steps are as follows: { Step 1: The training set X, for the model is extracted from the series X. { Step 2: The J -level MODWT is applied to X to produce the component series AJ ; DJ ; DJ 1; : : : ; D1 { Step 3: The ARIMA process above is applied to each component series

Dj = fdj;1; : : : ; dj;t 100g to produce the forecast d^j;t 99 { Step 3: The forecasts are appended to each component series, the IMODWT is applied and the nal term in the constructed series is the forecast x^t 99. { Step 4: The observed value xt 99 is added to the training set X and Steps 2-3 are repeated to produce a forecast for xt 98.

The iterative forecasting process continues until we have a complete set of forecast values fx^t 99; : : : ; x^tg. 4.6

WAV-LSTM

The network topologies implemented were the same as those in LSTM, as are most of the steps for producing forecasts. The main di erence lies in the transformation of the series into the feature and target spaces, where the real values xi 2 R, are replaced with vectors haJ;i; dJ;i; : : : ; d1;ii 2 RJ+1. Out target and feature sets are now: X~ = > > > : 8 (haJ;1; : : : ; d1;1i; haJ;2; : : : ; d1;2i; haJ;3; : : : ; d1;3i); 9 > > >>< (haJ;2; : : : ; d1;2i; haJ;3; : : : ; d1;3i; haJ;4; : : : ; d1;4i); >>= . .

. (haJ;t 3; : : : ; d1;t 3i; : : : ; haJ;t 1; : : : ; d1;t 1i) > > > ;

The Steps 1-6 are repeated from the LSTM process to produce a complete set of forecast values fx^t 99; : : : ; x^tg. The model implementations from the previous section were used to perform several pieces of analysis. By way of determining a baseline accuracy of forecasting, we applied PREVCLOSE to the ve series FB, AAPL, AMZN, NFLX and GOOG. These baseline forecasts were used to calculate a comparative uplift in accuracy between the models for each series.

Primarily, we were interested in assessing the impact of the MODWT on forecast accuracy by comparing performance the wavelet and non-wavelet models. We rst applied the ARIMA and the two LSTM network topologies in Figure 1 to each series. For WAV-ARIMA we applied 4-level and 7-level Haar MODWTs and for the WAV-LSTM, both Haar and db4 4-level and 7-level MODWT's. In total this was 13 models for each of the 5 series. 6

Evaluating Performance

With each of the 100-step log return forecasts fx^t 99; : : : ; x^tg, we were able to produce a set of predicted closing prices fp^t 99; : : : ; p^tg with p^i = p^i 1ex^i and p^t 100 = pt 100. Figure 2 displays the day ahead forecast of AAPL using WAVLSTM 4haar 4nodes. With these forecasts we calculated the root mean squared error (RMSE) as a measure of a model's accuracy. The RMSE metric has the bene t of having the same units as the closing prices we are predicting. 7

Results

In this section we display the model forecast accuracies for the each stock, a results summary can be seen in Table 1. For each stock, we present PREVCLOSE with AAPL the only instance where there is a notable improvement (> 1% increase in uplift) when using the LSTM.

We suspect this is because of the di erence in forecasting methods between the two approaches. With ARIMA, a new model is trained with each step of the 100 day forecast to incorporate each new current day observation. With LSTM, the model is t on the initial training data only and while the node states will update as the test data is played through the network, the synapse weights are xed. We suggest there may an increase in LSTM accuracy if the network is retrained after each forecast to include the observed value for the current day.

The 1 node LSTM hidden layer continually outperformed the 4 node layer in both RMSE and MODWT RMSE, with AAPL the only exception, demonstrating that an increase in model complexity does not always translate to an improvement in model performance.

All of the ARIMA models outperformed their WAV-ARIMA counterparts, suggesting that the component series are less suitable to this type of modelling than the original series. The majority of the di erences in uplifts between the LSTM and WAV-LSTM models were negligible (< 1% di erence in uplifts), with the exception of the APPL 4 node WAV-LSTM with a 4-level Haar decomposition which had a 2:86% increase in accuracy uplift and the AMZN 4 node WAV-LSTM 7-level Haar with a 4:21% decrease.

In summary, the results show no signi cant gains in forecast accuracy resulting from the MODWT in either the ARIMA or LSTM models. This contrary to the research in [24] and [ 16 ] and would suggest there are further re nements to be made to our approach.

Our selection of the 4-level and 7-level decomposition were arbitrary and further work in our method could make this a more informed decision by incorporating the PCA as seen in [ 16 ]. Similarly our choice of the number of input nodes in LSTM, in this instance a 3-lag input, could better inferred by making using of the Partial Autocorrelation Function (PACF) in the ARIMA model.

A major assumption of the ARIMA model is constant variance or volatility. This seems unlikely in a real world scenario given the trading volumes that surround company speci c or geo-political events like earnings reports or the USA Presidential election. An ARIMA with addition explanatory variables (ARIMAX) or the General Autoregressive Conditional Heteroskedasticity (GARCH) model as suggested in [24] could better describe this behaviour. 9

Conclusion

In this paper we deployed the ARIMA and LSTM models for forecasting future values in a univariate time series. We demonstrated the MOWDT, a technique for decomposing a time series into approximation and detail components, while preserving the temporal information of the original series. We combined these techniques and developed an ensemble approach in order to produce 100 1-day ahead forecasts of the daily closing price of FAANG stocks on the NASDAQ exchange. We presented our results and o ered possible suggestions for the outcomes. Finally we recommended further avenues of investigation in order to expanded and improve upon our work. 23. Walls, L., Bendell, A.: Time series methods in reliability. Reliability Engineering 18(4), 239 { 265 (1987) 24. Youse , S., Weinreich, I., Reinarz, D.: Wavelet-based prediction of oil prices. Chaos, Solitons & Fractals 25(2), 265{275 (2005)

References

1. Alpha Vantage . https://www.alphavantage.co ( 2018 ), [Online; accessed 10-Aug2018]

2. Investopedia - What are Corporate Actions? https://www.investopedia.com/ articles/03/081303.asp ( 2018 ), [Online; accessed 10-Aug-2018]

3. Yahoo! Finance. https://finance.yahoo.com ( 2018 ), [Online; accessed 10-Aug2018]

4. Auquan , T. : Time Series Analysis for Financial Data V - ARIMA Models . https://medium.com/auquan/ time-series -analysis-for-finance-arima-models- acb5e39999df ( 2017 ), [Online; accessed 10-Aug-2018]

5. Brownlee , J.: Time Series Forecasting with the Long Short-Term Memory Network in Python . https://machinelearningmastery.com/ time-series -forecasting-long-short-term-memory-network-python/ ( 2017 ), [Online; accessed 10-Aug-2018]

6. Conlon , T. , Crane , M. , Ruskin , H.J.: Wavelet multiscale analysis for hedge funds: Scaling and strategies . Physica A: Statistical Mechanics and its Applications 387 ( 21 ), 5197 { 5204 ( 2008 )

7. Crowley , P.M.: A guide to wavelets for economists . Journal of Economic Surveys 21 ( 2 ), 207 { 267 ( 2007 )

8. Egan , W.J.: The distribution of s&p 500 index returns ( 2007 )

9. George E. P. Box , G.M.J. : Time Series Analysis: Forecasting and Control. Holden Day ( 1970 )

10. Graves , A. : Generating sequences with recurrent neural networks . arXiv preprint arXiv:1308.0850 ( 2013 )

11. Hannan , E. , McDougall , A. : Regression procedures for arma estimation . Journal of the American Statistical Association 83 ( 402 ), 490 { 498 ( 1988 )

12. Hasim Sak , A.W.S. , Beaufays , F. : Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition . CoRR abs/1402 .1128 ( 2014 )

13. Hochreiter , S. , Schmidhuber , J.: Long short-term memory . Neural computation 9(8) , 1735 { 1780 ( 1997 )

14. Kingsbury , N.: Image processing with complex wavelets . Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 357 ( 1760 ), 2543 { 2560 ( 1999 )

15. Laurent , C. , Pereyra , G. , Brakel , P. , Zhang, Y. , Bengio , Y. : Batch normalized recurrent neural networks . arXiv preprint arXiv:1510.01378 ( 2015 )

16. Liu , T. , Wei , H. , Zhang, C. , Zhang , K. : Time series forecasting based on wavelet decomposition and feature extraction . Neural Computing and Applications 28 ( 1 ), 183 { 195 ( 2017 )

17. Meyler , A. , Kenny , G. , Quinn , T. : Forecasting irish in ation using arima models . Central Bank and Financial Services Authority of Ireland Technical Paper Series 1998 (3/RT/98), 1 { 48 ( 1998 )

18. Percival , D.B. , Walden , A.T. : Wavelet Methods for Time Series Analysis . Cambridge University Press ( 2000 )

19. Rioul , O. , Vetterli , M. : Wavelets and signal processing . IEEE Signal Processing Magazine 8 ( 4 ), 14 { 38 ( 1991 )

20. Ripley , B. : Statistical theories of model tting . INATO ASI SERIES F COMPUTER AND SYSTEMS SCIENCES 168 , 2 { 26 ( 1998 )

21. Siami-Namini , S. , Namin , A.S.: Forecasting economics and nancial time series: Arima vs . lstm. arXiv preprint arXiv: 1803 . 06386 ( 2018 )

22. Sundermeyer , M. , Schluter, R., Ney , H.: Lstm neural networks for language modeling . In: Thirteenth annual conference of the international speech communication association ( 2012 )