These days, attention-based Recurrent Neural Networks (RNNs) have been widely used for learning the hidden temporal structure of raw time series data. More recently, attention-based RNNs have been further enhanced to represent multivariate temporal or spatio-temporal structure underlying multivariate time series. This latest study achieved more efective prediction by employing attention structure that simultaneously capture the spatial relationships among multiple diferent time series and the temporal structure of those time series. That method assumes single time-series samples of multi- or uni-variate explanatory variables, and thus, no prediction method was designed for multiple time-series samples of multivariate explanatory variables. Moreover, such previous studies have not explored on financial time series incorporating macroeconomic time series, such as Gross Domestic Product (GDP) and stock market indexes, to our knowledge. Also, no neural network structure has been designed for focusing a specific industry. We aim in this paper to achieve efective forecasting of corporate financial time series from multiple time-series samples of multivariate explanatory variables. We propose a new industry specific model that appropriately captures corporate financial time series, incorporating the industry trends and macroeconomic time series as side information. We demonstrate the performance of our model through experiments with Japanese corporate financial time series in the task of predicting the return on assets (ROA) for each company.
In recent years, a huge amount of information is accumulated day by day with the developing information technology. One such information is corporate financial time series data in the economic and financial fields. Many economic experts have interest in gaining new insights from these data. Corporate financial time series are particularly complex since they are often affected by various information, such as business conditions of each company, trends in the industry, and business sentiment in society. Traditional time series analysis and modern deep learning technology have addressed the problem of time-series prediction (or forecasting); however, there is plenty of room for new research on complex multivariate time series, such as corporate ifnancial time series. Among the widely recognized time series analysis methods, the Autoregressive Integrated Moving Average (ARIMA) model and the kernel methods can capture one aspect of spatio-temporal patterns; however, it is not easy to achieve accurate forecasting of multivariate time series[1, 10]. Moreover, Recurrent Neural Networks (RNN) and Long ShortTerm Memory (LSTM) can take into account time dependencies and have been well accepted lately even in the fields of financial time series analysis, such as stock price forecasting [8]. However, it may not be easy to achieve accurate long-term prediction for multivariate time series, since a part of multivariate explanatory variables may not contribute to the prediction and even do harm the prediction accuracy. It can be considered that, when some explanatory variables have relatively small contributions to the prediction, those variables may result in noises. In another line of research, a time-series prediction model was proposed so that it uses the attention-based RNN to learn the attention weights of raw time series and further enhances the ability to represent spatio-temporal features [2]. Qin et al. [12] and Yuxuan et al. [9] combined attention mechanisms with encoderdecoder models to achieve better performance in predicting one or several steps ahead. Liu et al. [11] developed Dual-Stage TwoPhase (DSTP) attention-based RNN model, by capturing correlations among multivariate explanatory variables and embedding past observations of target time series via multiple levels of attention mechanism. However, no prediction was made for multiple time-series samples with multivariate explanatory variables, in those previous studies. Moreover, no previous studies have explored on deep learning models for financial time series incorporating macroeconomic time series, such as Gross Domestic Product (GDP) and stock market indexes, to our knowledge. Also, no structure of deep learning models has been designed for focusing a specific industry, even the industry trend can be inlfuencing. This paper aims to establish a useful method for forecasting corporate financial time series by appropriately learning from multiple time-series samples with multivariate explanatory variables. We propose a new industry specific model that appropriately captures business and industry trends, as well as macroeconomic time series, in an extension of attention-based RNN. Through experiments with Japanese corporate financial time series, we demonstrate our proposed model focusing on the wholesale industry works efectively in the task of predicting the return on assets (ROA) for each company. 2
This paper attempts to predict one step ahead of corporate financial indicators by deep learning. This section consists of four topics. The first topic is about RNNs, which have been one of the most popular deep learning methods for predicting time series data. The second topic is about LSTMs, which have been extended from RNNs to capture long and short time dependencies. The third topic is on attention mechanisms, which have attracted much attention recently due to the promising prediction performance. Those topics provide basic techniques for time series analysis based on deep learning. As the last topic, we briefly review state-of-the-art related work on deep learning-based financial time series prediction. t=3
t=T t=1
t=2 Sequential data refers to any kind of data where the order of the samples is important. Especially, sequential data is also called as time series data in case that the order is based on time. It is known that prediction performance can be improved by considering the dependencies between the current samples and the past samples. One most popular method is RNN, which is an extension of feed-forward neural networks for handling sequential data [3]. Now, we suppose that the RNN receives input sequence xt for each time t and also returns one output sequence yt at the time t . At time t = n, we can assume that output yn is successfully produced from input sequence x1, x2, x3, , xn . This is because the RNN is based on a neural network with directed closed loop called ‘return path’. This structure makes it possible to store temporal information and change behaviors. Figure 1 shows the structure of the RNN and its structure expanded in time dimension.
Now, we describe the calculation process in RNNs. First, suppose xt is input to the network; ht is output from the middle layer; yt is output from the output layer; W ¹inº is input weight matrix that represents connections from the input layer to the middle layer; and W ¹out º is output weight matrix that represents connections from the middle layer to the output layer.
In the return path, the RNN returns the output of the middle layer to its own input. The RNN assumes the connection between the middle layer at time t 1 to that at time t . Therefore, a weight w is assigned to each recurrent connection from an arbitrary unit of the middle layer at time t 1 to an arbitrary unit of the middle layer at time t . We use the notation W as a recurrent weight matrix consisting of each recurrent connection’s w. Figure 2 shows one unit of middle layer in RNN.
Hidden variables in the middle layer at time t , ht , can be obtained by xt , W ¹inº, ht 1, W , activation function f and bias b, as follows:
ht = f ¹W ¹inºxt + W ht 1 + bº
Output yt are then obtained by f , W ¹inº, and ht from each unit in the middle layer:
yt = f ¹W ¹out ºht º
In this paper, we assume that activation function f in Eq. (
RNNs can capture the context of sequential data. In this case, it is
important to understand the length of past sequence that should
be captured in the model, in other words, how long past inputs
from the current time should be reflected to predict the output.
(
However, the gradient usually vanishes or explodes after a certain number of iterations during learning in case of RNNs [4, 5]. This limit is caused by the so-called gradient vanishing problem —when calculating weights, the value of the gradient explosively decreases or increases as it is propagated backward through the RNN network. To address this problem, LSTMs [6] were proposed to achieve long- and short-term memory. Compared to the RNN, each unit in the middle layer of the LSTM has a memory cell and three gates: an input gate, an output gate, and a forget gate, while the other structure is basically the same as that of the RNN. Figure 3 shows one unit of middle layer in the LSTM.
Now, let Wi , Wo , Wf , and Ws be input weight matrices, where
the subscript indicates the input gate i, output gate o, forget gate
f , or memory cell s in the LSTM network. Also, let Ui , Uo , Uf ,
and Us be recurrent weight matrices, and bi , bo , bf , and bs be
biases. Suppose σ is the sigmoid function, and tanh is the
hyperbolic tangent function. At time t , we suppose that it is the
output of the input gate; sˆt is a new candidate state of the
memory cell; ot is the output of the output gate; ft is the output of
the forget gate; st is the state of the memory cell; and ht is the
output of the memory cell. We obtain these variables, as follows:
it = σ ¹Wi xi + Ui ht 1 + bi º
sˆt = tanh¹Ws xt + Us ht 1 + bs º
ft = σ ¹Wf xt + Uf ht 1 + bf º
(
ht = ot tanh¹st º (
Now, we will discuss how the LSTM’s units work in the four steps below.
Step 1 —Update the output of the forget gate ft :
First, the model needs to determine what to forget from
the cell state, as shown in Eq. (
Step 2 —Update the output of input gate it and new
candidate state of memory cell sˆt :
Second, the model needs to determine what information
is going to be added to the cell state, as shown in Eqs. (
Step 3 —Update the state of the memory cell st :
Third, the model updates the cell state, as shown in Eq. (
Step 4 —Update the output of the memory cell ht and the
output of output gate ot :
Finally, the model determines the output unit, as shown
in Eqs. (
These structures improve the limitation of RNNs in which limited-term memory can only be captured, achieving a more accurate estimation that captures a longer-term memory. 2.3
Attention mechanisms were successfully used for LSTM [11, 12]. In time series analysis with LSTM, the attention mechanisms can simultaneously capture the spatial relationships among multiple diferent time series and the temporal structure of those time series. In the rest of this subsection, we will briefly review the attention mechanism developed by Liu et al. [11].
2.3.1 Spatial-atention LSTM. The purpose of the spatial
attention mechanism is to obtain the spatial correlations among
multivariate input time series. Given the time series (of length T )
of k-th explanatory attribute at time t , xkt = ¹xtk T +1, , xtk º 2
RT , the following attention mechanism can be used:
akt = vaT tanh¹Wa »ht 1; st 1¼ + Ua xkt + ba º
αtk =
exp¹akt º
j=1 exp¹atj º
∑n
(
T ∑ j=1
j βt hj βti =
exp¹bti º ∑T j
j=1 exp¹bt º where hi represents the i-th hidden state vector obtained in the spatial attention module mentioned previously. dt 1 2 Rp and st0 1 2 Rp are respectively the hidden state and cell state vectors of time t 1, and p is the number of hidden states of this attention module. vb , bb 2 Rp , Wb 2 Rp 2p , and Ub 2 Rp m are the parameters to learn. Next, context vector ct are defined as follows: Context vector ct represents the information of all the hidden states, representing the temporal relationships within a time window. This context vector ct is then aligned with target variable yt , as follows:
y˜t = w˜ T»yt ; ct ¼ + b˜ where w˜ 2 Rm+1 and b˜ 2 R are parameters that map the concatenation to the target variable. Aligning the target time series with the context vector makes it easier to maintain temporal relationships and makes use of the results to update the hidden state and cell state. Let ft empor al be the LSTM with the temporal attention mechanism mentioned previously. Then we can get the following equation ¹dt , st0 º = ft empor al ¹dt 1, st0 1, y˜t 1º (17)
Given T -length multivariate explanatory time series XT = ¹x1, , xt , , xT º, where xt = ¹xt1, , xtn º, and target time series yT = ¹y1, , yt , , yT º, context vector cT and hidden T T … …
Temporalattention LSTM … state vector hT are concatenated to make the final prediction of one step ahead of target variable yT +1, as follows: yˆT +1 = =
F ¹XT , yT º vyT¹Wy »dT ; cT ¼ + by º + by0 (18) (19) where F denotes the predictor. Wy 2 Rp ¹p+mº and by 2 Rp map the concatenation »hT ; cT ¼ 2 Rp+m to p-dimensional latent space. A linear function with weights vy 2 Rp and bias by0 2 R produces final prediction yˆT +1. 2.4
Quite recently, Liu et al. [11] focused on spatial correlations and
incorporated target time series to develop Dual-Stage Two-Phase
(DSTP) attention-based LSTM model. Here, we briefly introduce
the structure of their model in Figure 4. Let Xt be
multivariate explanatory time series and Yt be target time series. At the
second phase of Spatial-attention LSTM, xk in Eq. (
They enhanced the attention mechanisms to incorporate both spacial correlations and temporal relationships. However, we have three concerns when we employ the DSTP-based model to achieve our goal. First, the dataset they used was NASDAQ 100 stock data1, which involves multiple time series samples with a univariate explanatory variable, not with multivariate explanatory variables. Second, their model did not incorporate macroeconomic time series. Third, their model did not consider industrywide trends. 3
In this study, we use “Surveys for the Financial Statements Statistics of Corporations by Industry”2 collected by the Ministry of Finance Japan. The surveys are based on sampling in which target commercial corporations are general partnership companies, limited partnership companies, limited liability companies, and stock companies, all of whose head ofices are located in Japan. We excluded the companies in ‘finance and insurance’ industry 1https://cseweb.ucsd.edu/ yaq007/NASDAQ100_stock_data.html 2https://www.mof.go.jp/english/pri/reference/ssc/outline.htm for our study. The collection period is from the first quarter of 2003 to the fourth quarter of 2016. The surveys consist of annual survey and quarterly survey. The total number of companies are 57, 775 in the quarterly survey and 60, 516 in the annual survey. These surveyed items include the financial indexes shown in Table 1.
We use financial statements in the quarterly survey, and
calculate various financial ratios as explanatory and target
variables in our analysis. We need to perform preprocessing for the
use of time-series analysis. First, the survey dataset contains
both long-term and short-term companies; however, it is not
easy to include short-lived companies for time-series analysis
and thus we excluded short-term companies for our analysis.
Secondly, the survey dataset contains a number of missing
values, so we need to take care of them before calculating financial
ratios. In summary, we employed the following three steps, in
this order, for preprocessing of the data: (
Before imputation processing, we do firstly exclusion processing. This is because some of company data greatly diverge from true data even if imputation processing is performed at this step. We performed exclusion processing in the following cases: Case 1: companies that do not have all of 56 time steps. Case 2: companies of which financial statements do have no data in the entire time steps.
After the exclusion, the number of companies in the dataset for the experiments became 2296. 3.2
Before calculation processing, we do secondly imputation processing. This is because some of company data have missing values, and if so, we cannot calculate financial ratios. We show the details on the imputation processing in the appendix. 3.3
In this study, a number of financial ratios are used as explanatory variables. Each financial ratio is based on formula of the financial sales ratio in corporate enterprise statistics, as defined in Table 2. 3.4
In addition to financial ratios, we use two macroeconomic time series as external data: one is Nikkei Average closing price3 (N ) and the other is Japanese GDP4 (G). The Nikkei Average closing price (N ) is extracted from January in 2003 to December in 2016 on monthly basis. The Japanese GDP (G) is extracted from the 1st quarter of 2003 to the 4th quarter of 2016 on quarterly basis. 4
In this paper, we propose a model for predicting one step ahead of the corporate financial time series in a target industry. Figure 5 shows the model’s flow from input time series to output time series. The 1st phase aims to extract the spatial correlations 3https://indexes.nikkei.co.jp/nkave//index/profile?idx=nk225 4https://www.esri.cao.go.jp/jp/sna/data/data_list/sokuhou/files/2019/toukei_2019.html among multivariate time series. The 2nd and 3rd phases aim to make the prediction of a target variable at time T + 1, given the past explanatory and target time series of window size T . More specifically, the 2nd phase aims to extract the spatial correlations between the target time series and the multivariate explanatory time series, while the 3rd phase aims to extract temporal relationships.
1st phase. This phase captures spatial correlations in all of the multivariate time series (including both explanatory and target variables observed as time series) of the entire length. By applying the attention mechanisms to the entire time series before splitting the time-series data, the correlations between timeseries samples can be captured well. For multiple time-series samples with multivariate variables, such that each time-series sample corresponds to each company, it is not easy to well capture the spatial correlations after the data splitting, which does not distinguish which time-series slice belongs to which company.
First, Spatial Attention-based LSTM, denoted as FSpat ial , is
applied to all company time series (A) to obtain the spatial
correlations among the multivariate time series for every company,
as in Eqs. (
A´ = Fspat ial ¹Aº
We then use a linear function to aggregate all samples. This liner function summarizes all companies’ features to an aggregated feature space. Here, let W be weights and b be biases in the liner function.
A˜ = W A´ + b (21) Eqs. (20) and (21) allow the model to incorporate all financial statements. Here, A´ with shape ¹J, L, K º —J L K third-order tensor— is aggregated to A˜ with shape ¹1, L, K º, where J , L, and K indicate the number of time-series samples (or companies), the length of training data, and the number of dimensions of multivariate variables (including both explanatory and target variables), respectively. Since the closing price of Nikkei Average (N ) is on monthly basis and its length is L´ = L 3, the attention mechanism is applied to aggregate it on quarterly basis. Here, N with shape ¹1, L´, 1º is aggregated to N˜ with shape ¹1, L, 1º. More specifically, the following formula is applied when i 2 f1, 2, 3g indicates either of the first month to the third month for each quarter t : γti =
exp¹Nti º ∑3 j j=1 exp¹Nt º (22) N˜t = ¹γt1Nt1, γt2Nt2, γt3Nt3ºT (23) Then, the quarterly representations of Nikkei Average closing price, N˜ , are concatenated with the aggregated company data, A˜, as well as Japanese GDP, G, and the multivariate explanatory
A N G I Y L time series in the target industry, I . Here, I indicates all the financial ratio time series other than the target time series in its industry.
Z = »A˜; N˜ ; G; I ¼ (24) The shape of G is ¹1, L, 1º and the shape of I is ¹B, L, K 1º, where B indicates the mini-batch size for I for learning the model. Therefore, the shape of the concatenated samples Z is ¹B, L, K 2 + 1º. By applying Spatial-attention LSTM to this, we can obtain the spatial correlations between the time series.
Zˆ = Fspat ial ¹Z º In this paper, Fspat ial in Eqs. (20) and (25) are obtained independently from A and Z , respectively. As the final stage of the 1st phase, Zˆ is sliced by shifting, one by one, the time steps of length T . Therefore, the number of time series samples is B ¹L T º, and the shape is ¹B ¹L T º, T , K 2 + 1º.
2nd phase. First, Zˆ of length T and the target time series YT of the same length are concatenated at each time t 2 f1, , T g. Here, YT is obatained by slicing in the same manner of Z mentioned previously.
ZÛ = »Zˆ; YT ¼ (26) After the concatenation, the shape of the time series samples ZÛ is ¹B ¹L T º, T , K 2 + 2º. By applying Spatial-attention LSTM to this, we can obtain the spatial correlations between the target time series and the other time series.
ZˆÛ = Fspat ial ¹ZÛ º 3rd phase. At this phase, we apply Temporal-attention LSTM (denoted as Ft empor al ) to capture the temporal relationships in the spatial attentions, as in Eq. (18). In other words, it captures the spatio-temporal relationships of multiple time series starting at diferent times.
yˆT +1 = Ft empor al ¹ZˆÛ, YT º The generated yˆT +1 is the final prediction. Here, all the models in this paper use a back-propagation algorithm to train the models. During the training process, the mean squared error (MSE) between the predicted target vector yˆT +1 and the ground-truth vector yT +1 is minimized using the Adam optimizer [7]. (25) (27) (28)
Moreover, we set the other hyperparameters, as follows. Window length T = 12, the number of epochs is 500, learning rate is 0.001, and the mini-batch size B = 64. These are determined empirically.
In the setting above, we assume ‘Whole sale and trade’ as a target industry. We also assume Return on assets (ROA), denoted as X1 in Table 2, as target variable Y . As for multivariate explanatory variables, we use X = fX0, X2, X3, , X11g. Therefore, the number of the variables in time series is K = 12, including both explanatory and target variables.
In order to demonstrate the efectiveness of the proposed model, we compare the prediction performance of the proposed model to that of the DSTP-based model and simple LSTM model. The Nikkei Average closing price (N ) was converted to quarterly by averaging every three months for the DSTP-based model and simple LSTM model. We assumed that the number of LSTM layers is two.
For evaluation, we use Mean Squared Error (MSE) with the sample standard deviation. We further confirm whether the improvement of the proposed model was statistically significant via Wilcoxon signed rank testing at 0.05 level, compared with the baselines.
We show in Table 3 the evaluation results in terms of mean squared errors (MSE) and the sample standard deviations (SD) using the proposed model, DSTP-based model, and simple LSTM model in various settings. Here, I indicates a target industry data and I = »X ; Y ¼ 2 A. Also, ‘# of units’ indicates the number of the LSTM units for each model, which was determined in the validation step.
The following are discussions to clarify the contributions from the three points of view:
Using macroeconomic time series: Comparing the proposed models with macroeconomic time series (Model 1) and those without macroeconomic time series (Model 3) in Table 3, Model 1 works more efectively than Model 3, on average, by successfully capturing properties of the macroeconomic time series. We confirmed that the improvement brought by Model 1 was statistically significant via Wilcoxon signed rank test at 0.05 level, compared to Model 3.
Focusing a specific industry : When we compare the proposed model with all the time series (Model 1) and that without considering spatial correlations among all companies’ time series (Model 2), Model 1 works moderately more efectively than Model 2, on average. We confirmed that this improvement was statistically significant in the same manner mentioned previously.
Using multiple time-series samples with multivariate explanatory variables: Given all the time series, Model 1 was more efective on average (with the statistical significance) than Models 4 and 7; however, this is not the case in the other situations. It can be said that our proposed models work modestly more efective than or comparable to the DSTP-based models, depending on the situations, and that our models work greatly more efective than the simple LSTM models. More detailed evaluation is left for the future work.
In order to establish a useful method for forecasting corporate ifnancial time series data, we aimed in this paper to appropriately forecast one step ahead using multiple time-series samples of multivariate explanatory variables. For this objective, we proposed an industry specific model that simultaneously captures corporate financial time series and the industry trends. We further proposed a new model structure that appropriately captures macroeconomic time series. In particular, we showed the efectiveness of the proposing model, especially focusing on the following three points. First, the model can capture macroeconomic time series, such as GDPs, more appropriately than the DSTP- and LSTM-based models. Second, the model can be focused to a specific industry. Third, the model is designed to be learned from multiple time-series samples of multivariate explanatory variables, producing modestly more efective or comparable performance of the prediction compared to the DSTPbased model. We developed a model for predicting corporate ROA in the wholesale trade industry. Our model may be efective in forecasting other target variables in other industries; however, this extension is left for the future work.
We thank Takuji Kinkyo and Shigeyuki Hamori for valuable discussions and comments. This work was supported in part by the Grant-in-Aid for Scientific Research (#15H02703) from JSPS, Japan.
Now, we briefly describe imputation processing to handle time series with missing values. We suppose time series д in the quarterly survey dataset and time series G in the annual survey dataset with regard to each financial index. We carry out imputation processing for each financial index in the two steps below. Here, let qyj be a value as of quarter j of year y in a specific financial index time series, as shown in Table 4.
Step 1 —using annual data: – Case 4: qy1 is equal to the sum of Gy and the other fi– Case 5: qy4 is equal to the sum of Gy and the other fi
Gy + G y0 4 Step 2 — linear interpolation and extrapolation: We perform commonly-used linear interpolation and extrapolation for missing values in each financial index time series in the quarterly survey dataset. – Case 6: The sum of qy1 , q2 , qy3 and qy4 is equal to the sum y of Gy and the other financial index
The imputation in this case depends on the number of missing values in {qy1 , q2 , q3 , q4 }. We assume Cases 6-1, y y y 6-2, 6-3 and 6-4, as below, where a, b, c, d 2 f1, 2, 3, 4g: Case 6-1 —when the number of missing values in {qya , qy , qyc , qy } is one:
b y d Suppose qa is missing.
y qa = Gy + G y0 y q b y y qa = q = b
Gy + G y0 2 y qc y qc Case 6-2 —when the number of missing values in {qya , qy , qyc , qy } is two: Suppose qya adnd qy are missing.
b
b Case 6-3 —when the number of missing values in {qya , qy , qyc , qy } is three:
b y d Suppose qa , qy and qyc are missing.
b y y y qa = qb = qc =
Gy + G y0
3 Case 6-4 —when the number of missing values in {qya , qy , qyc , qy } is four: Suppobse qya , dqy , qyc and qy are missing.
b d y q d y q d y 1 2 3 4
Quarterly surveys q1y q2y q3y q4y
Annual surveys
Gy For the imputation, we make use of the annual survey dataset. When the value qy is missing, we attempt to look j up Gy in the annual survey dataset. If Gy is also missing, this step for qy is skipped. This imputation processing j consists of the following six cases, depending on financial indexes.
y – Case 1: q1 is equal to Gy .
y – Case 2: q4 is equal to Gy .
y q1 = Gy y q4 = Gy – Case 3: The sum of qy1 , qy2 , qy3 and qy4 is equal to Gy .
The imputation in this case depends on the number of missing values in {qy1 , q2 , q3 , q4 }. We assume Cases 3-1, y y y 3-2, 3-3, and 3-4, as below, where a, b, c, d 2 f1, 2, 3, 4g: Case 3-1 —when the number of missing values in {qya, qy , qyc , qy } is one:
b y d Suppose qa is missing. (29) (30) (31) (32) (33) y y qa = q = b y qc 2 Case 3-2 —when the number of missing values in {qya, qy , qyc , qy } is two: Suppose qya adnd qy are missing.
b
b Case 3-3 —when the number of missing values in {qya, qy , qyc , qy } is three:
b y d Suppose qa , qy and qyc are missing.
b Case 3-4 —when the number of missing values in {qya, qy , qyc , qy } is four: Suppobse qya , dqy , qyc and qy are missing.
b d