This paper is discusses the problems of the short-term forecasting of financial time series using supervised machine learning (ML) approach. For this goal, we applied several the most powerful methods including Support Vector Machine (SVM), Multilayer Perceptron (MLP), Random Forests (RF) and Stochastic Gradient Boosting Machine (SGBM). As dataset were selected the daily close prices of two stock index: SP 500 and NASDAQ, two the most capitalized cryptocurrencies: Bitcoin (BTC), Ethereum (ETH), and exchange rate of EUR-USD. As features we used only the past price information. To check the efficiency of these models we made out-of-sample forecast for selected time series by using one step ahead technique. The accuracy rates of the forecasted prices by using ML models were calculated. The results verify the applicability of the ML approach for the forecasting of financial time series. The best out of sample accuracy of short-term prediction daily close prices for selected time series obtained by SGBM and MLP in terms of Mean Absolute Percentage Error (MAPE) was within 0.46-3.71 %. Our results are comparable with accuracy obtained by Deep learning approaches.
Forecasting financial tine series have been in focus of researchers for a long time. This topic continues to be relevant from both theoretical and applied points of view. Brokers, financial analysts and traders make daily decisions about buying and selling various financial assets, including currency, stocks, bonds and others. To reduce the risk of such transactions and to obtain the expected return on their investments, each of them must ___________________ Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). analyze a number of factors that affect market conditions and generate upward or downward trends.
In this regard, the problem of developing adequate forecasting approaches is relevant to the scientific community as well as to financial analysts, investors and traders.
There are two main approaches to solving the problem of forecasting financial assets. The first one is to construct a casual model that describes the relationship between the asset’s value and other macroeconomic factors. This approach was implemented within the framework of fundamental analysis and based on different mathematical tools, such as econometric modeling and systems of differential equations [13; 17; 28].
Another approach is based on the analysis of past observations selected asset and used variety of technical indicators and oscillators that help predict market trends. This approach has been realized in technical analysis which is actively used now in addition to time series analyses [7; 13; 28]. Within time series framework has been developed manifold class of linear and nonlinear approaches, such as ARIMA-GARCH models [6; 27].
Recent time the methods and algorithms of Machine Learning (ML) which have developed within Data Science paradigm [14; 36] ML have been also applied to forecasting financial and economic time series [2; 12], and various automated trading systems (bots) built on these algorithms began to be used for trading. Results of numerous empirical studies have shown that ML approaches outperform time series models in forecasting different financial assets [10; 18; 22; 26; 31; 38].
The main advantage of ML is that the algorithms themselves interpret the data, so we don’t need to perform their initial decomposition. Depending on the purpose of the analysis, these algorithms themselves build the logic of modeling on the basis of available data.
This avoids the complex and lengthy pre-model stage of statistical testing of various hypotheses about studied process. The main hypothesis, in particular, in terms of the purpose of our study, is only the thesis of the ability of ML methods to effectively analyze the financial time series, to identify hidden patterns and time correlations, which are the basis for making qualitative short-term forecasts.
The main goal of our paper is to compare the predictive properties of the most efficient ML algorithms: Artificial Neural Network (ANN), Support Vector Machine (SVM), Random Forest (RF) and Gradient Boosting Machine (GBM) for short-term forecast financial time series (stock indices, currencies and cryptocurrencies). At the same time, as predictors (features) we used only the past values of the studied time series. Our main assumption is that ML methods be able to extract latent patterns from the data, which allows us to make more efficient predictions.
This paper is organized as follows: in Section 2 is represented brief literature review devoted using ML approaches in the field of financial time series forecasting. Section 3 is described the main concept of applied methods. Data description and empirical results are given in Section 4. Concluding remarks and future perspectives are given in Section 5.
It should be noted that financial time series forecasting have been studied for a long time. Since, ML approaches proved their efficiency in many areas and became popular; they have been widely used for research financial time series. Numerous articles in scientific journals, reviews, conferences and internet resources are devoted to this topic.
Last five years researchers basically have been focused attention on novel network
ML approaches Deep Learning (DL), which includes asset of powerful methods, such
as Recurrent Neural Network (RNN), Long-Short Time Memory (LSTM), and
Convolutional Neural Network (CNN) and so on [22; 24; 32; 34; 38]. Recently was
published detailed overview devoted to using DL approaches in the field of financial
forecasting [
The key advantage of DL models is very powerful in feature learning and selection of input data using a general-purpose learning procedure. But DL models have such disadvantage that it takes much more time to train them, besides, it is a nontrivial problem is tuning hyperparameters. At the same time, traditional ML models often show comparable accuracy in time series forecasting.
As for using ML algorithms in financial forecasting, the most common are Neural Networks (ANNs) of various architecture [1; 8; 10; 20; 21; 33; 38], Support Vector Machines (SVM) [23; 25; 29; 30; 35], and Fuzzy Logic (FL) [25; 39].
The application of these approaches for forecasting task has shown their efficiency for both traditional financial assets [1; 8; 18; 20; 21; 23; 24; 29; 30; 35] and cryptocurrencies [8; 26; 31; 38].
Several studies [1; 8; 20; 21] presented the results that ANNs have better predictive
properties then other ML approaches for forecasting financial time series. At the same
time, there are a number of research papers (see, for example, Okasha, [
It should be noted, that much less attention has been paid to another powerful class of ML approaches of designing ensembles Classification and Regression Trees (C&RT): Random Forest (RF) [4; 5] and Gradient Boosting Machine (GBM) [15; 16], which used bagging (RF) and boosting (GBM) technique. Both RF and GBM are powerful methods that can efficient capture complex nonlinear patterns in data.
Thus, Varghade and Patel [
Kumar and Thenmozh [23] explored set of classification models for predicting direction of index S&P CNX NIFTY. Their empirical results suggest that both the SVM and RF outperforms the other classification methods (NN, Linear Discriminant Analysis, Logit), in terms of predicting the direction of the stock market movement, but at the same time SVM it turned out to be more accurate.
Recently there have been appeared several papers devoted to applying ensembles approaches for forecasting cryptocurrency prices [3; 9; 11]. Borges and Neves [3] tested four ML algorithms for prediction price trend: LR, RF, SVM and GBM. All learning algorithms outperform the Buy and Hold investment strategy in cryptomarket. The best result was obtained by ensembles voting (accuracy 59.3%).
Chen et al. [9] applied a set of learning models including RF, XGBoost, Quadratic Discriminant Analysis, SVM and LSTM for Bitcoin 5-minute interval and daily prices. Authors used wide dataset including as features technological, market and trading, socio-media and fundamental factors. Somewhat unexpected was that for daily prices better results were obtained by using statistical methods (average accuracy 65%) unlike ML methods (average accuracy 55.3%). Among the best ML the SVM was the best, with an accuracy of 65.3%. (1) 3
In this paper we have been applied supervised ML technique for forecasting financial time series. Consider a sample of pairs of features = , , . . . , , . . . , and the labels y: ( , ) , ,..., length n. In our case labels (or target) are values of selected financial assets, and features are only lagged daily values these assets , , . . . , , > .
Our main goal is to predict future value of target variable on the next time period (next day since we used daily quotes) by using several ML approaches (SVM, ANN, RF and GBM) and compare their forecasting performance.
Thus, our task is to construct some functional (regression) or rule-based (decision tree based) dependence of the form = , , where = ( ) , = 1,2, . . . , are vectors of features; – weights of the features, n – total number of samples in dataset; k – number of features. 3.1
The support vector machine (SVM) is an extension of the support vector classifier that results from enlarging the feature space in a specific way by using kernels function. The main idea of the SVM method is to map the original vectors into a space of a higher dimension and search for a separating hyperplane with a maximum margin in this space. Two parallel hyperplanes are constructed on both sides of the hyperplane separating the classes. The separating hyperplane will be the hyperplane that maximizes the distance to two parallel hyperplanes. The algorithm works under the assumption that the lager difference or distance between these parallel hyperplanes (margine) provides the smaller average error of the classifier.
Support Vector Regression (SVR) is the regression process performed by SVM which tries to identify the hyperplane that maximizes the margin between two classes and minimize the total error. In order for an efficient SVM to be constructed, a penalty of complexity is also introduced, balancing forecasting accuracy and computational performance.
Unlike classic regression problem SVR seeks coefficients that minimize a different type of loss, where only residuals larger in absolute value than some positive constant contribute to the loss function. This is an extension of the margin used in support vector classifiers to the regression setting.
The mathematical formalization of SVR is reduced to the following. Let’s regression equation is written in the form ( ) = ⟨ , ⟩ − , where ⟨⋅,⋅⟩ – is operator of inner product; is a constant.
Then the problem is reduced to minimizing functional: ⟨ , ⟩ + ∑ (|⟨ , ⟩ − − | − ) →
, = 1,2, . . . , , , where C is the regularization parameter or penalty coefficient for incorrectly estimating the output associated with input vectors, which also controls the relationship between a smooth boundary; l is the number of samples in training set (l < n, as a rule ≈ 0.7 ÷ 0.8 ); is the margin value.
After changing variables and some algebraic transformations loss function for SVR can be presented in such form: 3.2
ANNs are the most popular methods of ML. Numerous empirical studies show the efficiency of ANNs in the different fields both for classification and regression problem: pattern recognition, image and voice analysis, machine translation and so on. Several last decides they are widely used for analysis and forecasting financial time ⟨ , ⟩ + ∑ ( + ) → , ,, ,, , = 1,2, . . . , , where = (− ( ) + − ), = ( ( ) − − ) are slack variables, that allow individual observations to be on the wrong side of the margin or the hyperplane; , is the kernel function. The most commonly used kernel functions are Linear, Polynomial, Gaussian, Radial Based Function (RBF) and so on.
Loss function (4) is minimized under condition − , − .
(2) (3) (4) (5) series. In [12; 22; 31] it was shown that ANNs have better predictive properties than time series models and other ML algorithms for financial time series forecasting problem.
In this paper we have been used network model of the most common architecture: Multi-Layer Perceptron (MLP) with three layers: input layer, one hidden layer and output layer with one neuron that represent target variable (predicted value). It should be noted that despite simple structure MLP be able to take into account complex patterns in data due using different nonlinear activation functions.
The network output depends on its configuration, weights and activation functions of neurons on the hidden and output layers: = ∑ ∑ + + , (7) where (⋅), (⋅) – activation functions of neurons of the hidden and input layer, respectively; – the weight of the connections between the i-th neuron of the hidden layer and the output of the network; – the weight of the connections between the jth neuron of the input and the i-th neuron of the hidden layers; , – bias neurons of the output and hidden layers.
Network learning consists in finding and setting the neurons weights (synaptic weights) which minimized difference between the target variable and the network output. The search of minimum of the loss function was performed by the gradient descent method, embodied in the back-propagation algorithm. 3.3
Gradient Boosting Machine (GBM) Boosting is a procedure for sequentially building a composition of machine learning algorithms, when each of them seeks to compensate for the shortcomings of the composition of all previous algorithms. In contrast to bagging, boosting does not use simple voting but a weighted one. The major attractions of boosting are that it is easy to design computationally efficient weak classifiers (as a rule used shallow decision trees). Boosting over decision trees is considered one of the most efficient methods in terms of classification quality.
Gradient Boosting Machine method (GBM) was proposed Friedman [15; 16]. Commonly the basic steps of GBM are the next.
The final classifier ( ) is constructed as a weighted sum of N basic algorithms ℎ ( , ) (Decision Trees): ( ) = ∑ ℎ ( , ), (8) where is vector of adjusted parameters, is the weight coefficient.
Let’s we choose the initial classifier ℎ ( , ), for example, it may be the median or mean of the time series (target variable).
If we on the N–1 step have already built new classifier ( , ), then we select the next basic algorithm ℎ ( ) that reduced the error given by previous classifier as much, as possible: , ( , ) + ℎ ( , ) → min , , where (⋅) – is loss function.
We can select ℎ ( . ) that minimized sum of squares deviations for all samples in training set ℎ ( , ) = argmin ∑ ( , ) (ℎ( , ) − ) ,
(9) (10) where is deviations that equal to the anti-gradient of the loss function (⋅).
In this way we perform predictions for samples in the training set by using gradient descent in the l-dimensional space.
If a new basic algorithm has been found, it is possible to select its coefficient by analogy with the gradient descent: = argmin ∑ , ( , ) + ℎ ( , ) , (11) It should be note, that boosting usually does not result the overfitting problem because shallow decision trees are used. These trees have a large bias, but are not inclined to overfitting.
The effective ways to solve this problem is to reduce the step: instead of moving to the optimal direction of the anti-gradient, a shortened step can be taken by ( , ) = ( , ) + ℎ ( ) , (12) where
∈ [0,1] is the learning rate. 3.4
The main concept of the RF is that a composition of weak classifiers can give good results for both classification and regression problems. Proposed by Breiman [4; 5] in 1996 the RF is based on bagging technique (bootstrap aggregation) over decision trees. Bagging reduces the variance of the base algorithms if they are weakly correlated. In RF the correlation between trees is reduced by randomization in two directions.
Firstly, each tree is trained on a bootstrapped subset. Secondly, the feature by which splitting is performed in each node is not selected from all possible features, but only from their random subset of size m. The main distinction between bagging and RF is the choice of these features subset. RF works well when all of the features are at least marginally relevant, since the number of features selected for any given tree is small. Using a small value of m will typically be helpful when we have a large number of correlated predictors.
The RF algorithm generates each of the N trees independently, which makes it very easy to parallelize. For each tree, it constructs a full binary tree of maximum depth. The main concept is that classifiers (trees) do not correct each other’s mistakes, but compensate for them when voting. Basic classifiers should be independent and they can be based on different groups of methods or trained on independent datasets. Bagging allows us to reduce prediction error in the case when the variance of the error base method is high.
Thereby efficiency of RF performance is achieved even though some trees will query on useless features and make random predictions. But some of the trees will happen to query on good features and will make good predictions (because the leaves are estimated based on the training data).
If we have enough trees, the random ones will wash out as noise, and only the “good” trees will have an effect on the final result (classification or prediction). 4 4.1
To reduce our analysis to the most popular financial assets, we were used daily close
prices two stock index: Nasdaq and SP&500, two most capitalized cryptocurrencies:
Bitcoin (BTC), Ethereum (ETH), and exchange rate EUR USD. Our initial dataset
covers the period from 01/01/2015 to 30/06/2020 for all series (for ETH from
06/08/2020) according to the Yahoo Finance [
So, our dataset includes 1384 observations for Nasdaq, 1383 for SP&500, 1434 for exchange rate EUR USD, 2008 for BTC and 1278 for ETH.
It should be noted that selected time series during this period had different type of dynamics due to we can better estimate forecasting performance for ML approaches (see fig. 1). Plot of selected variables (series)
Include cases: 653:1383
On purpose of training models, fitting and tuning their parameters dataset was divided into the training and test subsets in the ratio of 80% and 20%. Moreover, the last 100 observations (from 22/03/2020 to 30/06/2020) were reserved for validation which was performed by out-of-sample one-step ahead forecast.
Since we focus on ML approach of forecasting financial time series data, the main purpose of our paper is to get the most accurate one-step ahead forecast of daily prices, based on only their past value.
According to some empirical studies devoted forecasting financial time series, there is a seasonal lag which is a multiple of 5 if we use daily observations and a multiple of 7 for cryptocurrencies because the fact that cryptocurrencies are traded 24/7.
For stabilization variance all features were taken in to natural logarithm. This is special case of Box-Cox transform. 4.2
It should be noted that hyper-parameters tuning is an important and sophisticated step of the model design. First of all, it is necessary to choose the functional form of the loss function. In the point of main purposes of our study the quadratic loss, which generally used for solving the regression problem, was selected.
According to our hypothesis regarding lag length as MLP models, we tested the following architectures: ─ 7 inputs and from 5 to 14 hidden layer neurons for cryptocurrencies; ─ 5 inputs and from 5 to 10 hidden layer neurons.
The most common functions such as logistic, hyperbolic tan, exponential and ReLu were tested as activation functions. Training MLP for each time series and different lag values (number of input neurons) was conducted over 100 epochs, of which the best 5 architectures were selected for each case (in terms of minimum PE error on the test sample and matching the model residuals to normal distribution).
The final prediction for each asset was obtained as the prediction of the ensemble of networks, that is, average of the best 5 corresponding MLP models.
For SVM models we have chosen RBF as a kernel which is the best for regression problem. Regularization parameter was estimated by the greed search in the range from 1 to 15 and it was selected C=10.
Both of tree-based methods (RF, and GBM) based on partitions the data into training and testing sets by randomly selecting cases. We applied in this study stochastic modification of GBM (SGBM) which based on such partition. The training sample is used to fitting models by adding simple trees to ensembles. Testing set is used to validate their performance. For regression tasks validation is usually measured as the average error. We select 30% of the dataset as test cases for both approaches.
Since the RF is not inclined to overfitting, one can choose a large number of trees for the ensemble. We designed RF model with 500 trees. At the same time, in order for the model to be able to describe complex nonlinear patterns in data, it is necessary to use complex trees. So, we have been chosen 15 the maximum number of levels.
Other important parameter for RF is the number of features to consider at each split. As noted in the Section 3.2 it is recommended to choose this value as (where M ≈ is the total number of features) for regression task. We tested different RF models with value m within 8 to 12.
As stop condition for number of trees in SGBM (boosting steps) we took the number of trees at which the error on the test stops decreasing. This is necessary in order to avoid the overfitting. For boosting, unlike the RF, the simple trees are usually used. That’s why we fitted maximum number of levels in trees and number of terminal nodes by the criteria of lowest average squared error on both training and test samples.
For GBM an important parameter is a learning rate (shrinkage). Regularization by shrinkage consists in modifying the update rule (12) by tuning . We selected this value on the grid search according to minimum prediction error on the test set. The final values of hyper-parameters setting are reported in table 1. 4.3
The short-term forecasts for selected time series were made for absolute values of prices (log prices). The target variable is the prediction the value of close prices for each series in the next time period (day) although we used daily observation. All models were trained with the same set of features.
On figures 2-3 were shown quality models fitting for BTC, and NASDAQ obtained by MLP and SVM. figures 4-5 presented results for RF, and SGBM respectively.
Samples: Train
Include cases: 16:1284 10500 10000 9500 9000 8500 t)u 8000 tup 7500 O (Q 7000 AD 6500 S AN 6000 5500 5000 4500 4000 35030500 4000 4500 5000 5500 6000 6500 7000 7500 8000 8500 9000 9500 1000010500
NASDAQ (Target) (a)
NASDAQ (Observed) vs. NASDAQ (Predictions) (NASDAQ.sta)
Include cases: 16:1284 10500 10000 9500 9000 ) 8500 itsno 8000 icd 7500 e (rP 7000 AQ 6500 D AS 6000 N 5500 5000 4500 4000 35030500 4000 4500 5000 5500 6000 6500 7000 7500 8000 8500 9000 9500 10000
NASDAQ (Observed) (b) Fig. 2. Fitting accuracy on the training and test subsets for NASDAQ: (a) MLP, (b) SVM.
Samples: Train Include cases: 1:1908
BTC (Observed) vs. BTC (Predictions) (Spreadsheet41)
Include cases: 16:1908 0 2000
6000 4000 8000 10000
14000 12000
18000 16000 20000 22000 0 2000 4000 6000
10000 8000
14000 12000 16000 18000
22000 20000 TC 8000 B 6000 4000 2000
0 26000 24000 22000 r rro 20000 E d re 18000 a u q eS16000 g a r ve 14000 A 12000 10000 8000 2,1E5 2E5 1,9E5 r 1,8E5 o r rE1,7E5 d rae 1,6E5 u q S1,5E5 e g rae 1,4E5 v A1,3E5 1,2E5 1,1E5 1E5
Number of Trees
(a) Summary of Random Forest
Response: BTC Number of trees: 200; Maximum tree size: 100
Include cases: 15:1908
Number of Trees (c) These graphs characterize the dependence of the predicted values (vertical axis) on the actual data (horizontal axis) on the test set and allow us to visually determine the quality of the fitting.
Observed value 12000 (d) 2,5E5 r rroE 2E5 d e r a qu1,5E5 S e g rvea 1E5 A 50000
0 3,5E6 3E6 ro2,5E6 rr E reda 2E6 u q eS1,5E6 g a r veA 1E6 5E5 0 50 100 150 200 250 300 350 400
Numberof Trees 50 100 150 200 250 300 350 400
Numberof Trees (c)
Thus, we can conclude MLP, SVM and SGBM methods have the same order of accuracy for the out-of-sample dataset prediction, although boosting also was somewhat more accurate. The best prediction performance is produced by SGBM for EUR-USD – 0.46 % (MAPE), and the best result for NASDAQ also provided SGBM – 2.38%. For BTC better performance shown SVM but MLP outperformed other models for SP&500. 1,15 1,14 1,13 1,12 1,11 1,10 1,09 1,08 1,07
EUR SVM MLP
10500 10000 9500 9000 8500 8000 7500 7000 6500 BRTFC 6000 SGBM 5500 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 (b)
300 280 260 240 220 200 180 160
ETH 140 SRGFBM 120 100 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101
(b) SP500 SVM
MLP
MLP 3600 3400 3200 3000 2800 2600 2400 2200 10500 10000 9500 9000 8500 8000 7500 7000 3600 3400 3200 3000 2800 2600 2400 2200 10500 10000 9500 9000 8500 8000 7500 7000 SP500 RF SGBM
SGBM 2000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 2000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 (a) (b)
Fig. 9. Out of sample prediction S&P 500. 6500 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 (a) Fig. 10. Out of sample prediction NASDAQ.
(b)
6500 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97101
EUR/USD
BTC/USD
ETH/USD
S&P 500
NASDAQ
It should be noted that our results are comparable with accuracy obtained by Deep
learning approaches [
Our research has shown efficiency of using ML approaches to predicting financial time series. According to our results, the out of sample accuracy of short-term forecasting daily quotes obtained by SGBM has the same rate as MLP and SVM. In terms of MAPE for selected time series it was within 0.46-5.9 %. Moreover, for NASDAQ and ETH SGBM outperformed other approaches. The worse results were obtained by RF which was used as a baseline.
At the same time all models showed the worst results for ETH, accuracy rate (MAPE) turned out in the range from 5.9 (SGBM) to 8.38 % (RF).
By designing models, we explored different sets of features: from 5 to 15 lags of target variable (from 7 to 14 for cryptocoins). Our final dataset contained only past values of target variable with 14 and 15 lag depth. In this case larger dataset provided better training for all models and given more efficient results.
It should be noted that we used a minimal dataset - only lag values of the studied series (closing prices). In our opinion, forecasting accuracy can be improved by including additional features, for example, open, max, min and average prices, fundamental variables, different indicators and oscillators, such as, Price rate-ofchange, Relative strength index, and so on.
Future research should extend by investigating of the prediction power of described ML approaches by using additional features. In the conclusion, we note that the proposed methodology by the development of combined ensemble of C&RT with other powerful ML models, such as NN and SVM is a promising approach to forecasting financial time series. Moreover, it seems to us promising to use DL approaches for features selection and making prediction. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22.