This study contributes to the deep learning literature by investigating the applicability of different DL models to forecasting monthly LEK/EUR exchange rate. To demonstrate the effectiveness for exchange rate forecasting of this models we examine the performance of several deep learning techniques (DFNN, LSTM and 1D-CNN). For each architecture, we have used different configuration and diverse techniques to avoid overfitting of the models. The accuracy evaluation of each model was based on the out-of-sample prediction for different horizon (3, 6 and 12 months ahead), by analyzing the model estimation for pre- and post- Covid pandemic. The comparison results show that the LSTM model with two hidden layers stands out as the best prediction model in the run-up to forecast three and six months ahead, followed by the three-layered 1DCNN model. However, they change places in the race for the 12 months horizon as the 1DCNN-3L becomes the first-best predicting model while leaving the LSTM-2L rank in second. These results demonstrate the potential of deep learning techniques, and also, they emphasize the importance of well configuring, implementing and selecting the different topologies.
Deep learning is considered a powerful tool for time series forecasting. The DL techniques are able to automatically learn linear and nonlinear relationships, extract features, handle large dataset and to capture temporal dependencies from sequential data in present of noise and missing values. However, it is important to mention that deep learning models can be computationally expensive and require large amounts of data to train effectively. Additionally, selecting the appropriate architecture, hyperparameters, and training procedure can be challenging and requires careful consideration. An exchange rate is the rate at which one currency of a country can be exchanged with the currency of another country. It is an important economic variable because most economic decisions rely on it. Thus, for easier and better decision making, it is important to know how exchange rate will change in the future. Forecasting exchange rate is a challenging task due to the fact that currency exchange rates are influenced by many political and economic factors, such as interest rate, inflation, political stability, government intervention, etc. Government agencies, financial institutions and economists are paying attention to the use of financial models to analyze the parameters of dynamic time series in order to predict exchange rate [1].
There are several methods used to forecast the exchange rate. One of them is regression analysis that tries to identify the relationship of currency exchange rates with various economic and political factors. This method is difficult to implement due to the fact that data regarding economic factors that influence exchange rate are not immediately available. Another method is time series analysis that attempts to make predictions based on historical currency exchange rate data. The time series models used for currency exchange predictions are moving averages, autoregressive integrated moving average (ARIMA) models, and exponential smoothing [1] . However, these models show problems. Moving averages and ARIMA do not work well when the time series being forecasted has a seasonality component. ARIMA models also require stationary time series and converting a non-stationary time series into a stationary one may remove some of the interesting dynamics of the time series. While the problem with exponential smoothing is that it may not capture the complexity of the time series being forecasted when there are irregular fluctuations. Recently academia and industry are demonstrating an increased interest in applying machine learning to economic and financial data [2].
This paper study the predictive performance of different deep learning models for various numbers of lags and horizon. We have selected three different forecasting univariate models, such as LSTM (Long Short-Term Memory), CNN (Convolutional Neural Network) and DFNN (Deep Feed-Forward Neural Network) for out-of-sample forecast analysis.
In this paper, we analyze the historical data on the monthly LEK/EUR exchange rate for the period from January 1992 to December 2022 (fig. 1). The dataset consists of a total 372 data points, and is divided in three sets: the training set (from January 1992 to December 2017), the validation set (which consists of 12 observation data) and for the out-of-sample analysis it was used the training set (from January 2019 to December 2022). In order to avoid overfitting during the model building we use two datasets: the training and validation set.
ALL/EUR 230.0 210.0 190.0 170.0 150.0 130.0 110.0 90.0 70.0 These prediction experiments of the exchange rate were conducted with the help of PYTHON, TensorFlow packages. We have selected three different forecasting univariate models, such as LSTM (Long Short-Term Memory), CNN (Convolutional Neural Network) and DFNN (Deep Feed-Forward Neural Network) for out-ofsample forecast analysis. Our main interest is to study the predictive performance of different deep learning models for various numbers of lags and horizon. The delayed data used as nodes value for the input layer on each model are twelve and eighteen and the predictive horizon is three, six and twelve months ahead. The predictive DL models were evaluated based on their performance on the test dataset using the root mean square error (RMSE) and mean absolute error (MAE).
By using the MAE and RMSE criteria simultaneously for the out-of-sample forecasting estimation, we can find the fluctuations in errors [2]. The RMSE can be greater than or equal to MAE. A very large difference between MAE and RMSE implies a large fluctuation in the error of the time series.2 We study the models’ performance for the all outof-sample periods. Also, for a deep analysis in our study we compare for each model configuration the model performance based on pre-Covid and post-Covid lockdown period
Deep learning is considered a powerful tool for time series forecasting. The DL techniques are able to automatically learn linear and nonlinear relationships, extract features, handle large dataset and to capture temporal dependencies from sequential data in present of noise and missing values. However, it is important to mention that deep learning models can be computationally expensive and require large amounts of data to train effectively. Additionally, selecting the appropriate architecture, hyperparameters, and training procedure can be challenging and requires careful consideration. In our work we have used three different types of DL architectures to study the Lek/Eur exchange rate forecasting: the Convolution Neural network (CNN), Deep Feed-Forword Neural Network (DFNN) and Long-Short Term Memory (LSTM). These three architectures are predominantly suited for time-series forecasting [2] [3].
The CNN model type is part of the artificial neural networks family and is widely used in image processing. The earliest presentation is LeNet and it was proposed by LeCun [4] et al. in 1998. The CNNs are designed to handle image input data efficiency but they are not limited. In order to use the CNN for sequence prediction the model needs to be a one- (1D) convolutional. CNN model is also called a deep learning model, due to the extra layers they have included in their structure called ‘convolutional’ block.
Use of the convolution block has some
advantages, such as [1]: Each of the units operates
with a vector of smaller length in order to make
the estimation for the full size of the parameters,
as was the case with DFNN. And also, the
evaluation of the convolution parameters
captures the time properties of the sequence data.
The typical architecture of 1D CNN consists of
one-dimensional convolution layers, pooling
layers, dropout layers and activation functions for
manipulating the one-dimensional input data
vector, figure 1.
The first layer that follows the input layer is the
convolution layer. This type of layer contains
filters (or also known as kernel) that are used to
calculate the dot product of the weights and the
input data by moving the filters along the input
data. The role of this layer is to detect features in
the input vector. The number and size of the
kernels are crucial for adequately capturing the
relevant features from the input data [
The pairing of a convolutional layer with a pooling layer performs smoothing of the sequence data. Because they are part of the same function that outputs predictions, by optimizing the neural network loss, one optimizes smoothing parameters directly to perform well on a prediction task [1]. For time-series forecasting it needs to be mentioned that the smoothed vector, which is the output of the 1-dimensional convolutional and pool layer, was handled by the RELU layer in order to apply non-linearity to it. The following layers then use this smoothed sequence data and deal with the crucial part of the time series forecasting problem.
All these transformations for the initial time series, in the new data it will be much easier to identify the appropriate information from the rest of the CNN layers.
Generally, in time series analysis, it is desired to
apply different types of smoothing techniques
prior to analysis. In traditional forecasting, a
moving average is commonly used to smooth the
time series, and then on the result sequence data
is applied a forecasting technique. The
convolutions layers try to mimic this process and
add more value because they perform a weighted
smoothing of the time series by establishing
automatically the ‘good’ parameters for
smoothing. [
The DFNN is a type of multilayer perceptron (MLP), composed of a sequence of layers, with a connectivity that flows in one direction from the input layer to the output layer. The layers are fully connected and do not form loops as all the information is processed in a forward way from the input layer to the output layer. The structure of DFNN consists of three different layers: the input layer, the hidden layers and the output layer. Each layer has a number of interconnected processing units called nodes. In this model, the processing units of a layer can be connected only with the nodes in the adjacent layer in forward direction.
The elements that need to be defined for the
DFNN architecture are the quantity of hidden
layers, quantity of neurons in the hidden layers,
activation function in each layer and learning
process to obtain the connecting weights. The
optimal architecture of the DFNN model was
generally determined through a trial-and-error
process, which is an exponential combinatorial
problem and a tedious task. [
The learning process for the DFNN is supervised learning in which the weights, during the training phase, are adjusted in order to reduce the difference between the real output value and the model output. The general algorithm used for weights adjustment is the backpropagation algorithm. So, reaching the more effective model for time series forecasting, is not enough to define the DFNN architecture. The learning rule applied throughout the training phase needs to guarantee improvement on the out-of-sample forecasting performance. The backpropagation algorithm, used in the type of architectures, comes across with two problems: overfitting and the vanishing gradient.
Overfitting occurs when the configured DFNN model fits exactly against its in-sample data used in the training process and performs poorly on out-of-sample forecasting. To avoid it, most research suggests the use of dropouts, in which on every epoch of model training only a percentage of nodes are used and the selection is random. The vanishing gradient is another problem that can be faced during the training phase; it happens when the backpropagation of the output error fails to reach the nodes in the input layer, thus the weights cannot be updated. Therefore, adding more hidden layers cannot improve the forecasting performance of the model.
LSTM is another architecture that can be used to
model univariate time series forecasting
problems. This architecture is a type of recurrent
neural network based on LSTM cells in which the
temporal dynamics of the input are modeled by
means of recursive connections. [1] The LSTM
are used as a solution for short term memory
learning and are one of the most successful
techniques that address vanishing gradients.
The unit in the LSTM model is a LSTM cell
(figure 3) which contains an internal state
ensuring that the gradient can pass across many
time steps without vanishing or exploding. This
connection type is characterized by the fact that
the output of a unit, after a delay, can be the input
of the same unit, along with the external inputs or
the inputs of the lower layers. [1] The important
elements on a memory cell are: a cell state (Ct-1
and Ct), hidden state (Ht-1 and Ht) and three gates
which contribute them with the power to
selectively learn, unlearn or retain information
from each of the units. [
The cell state in the LSTM model helps the information to flow through the units without being altered by allowing only a few linear interactions. Each unit has an input, output and a forget gate which can add or remove the information to the cell state. The input gate is where the information enters in the cell and is used to learn new information from it; the forget gate determine whether the information arrived from the previous timestamp is to be remembered or is irrelevant and can be forgotten, and the third, throw the output gate the cell transmit the updated information from the current timestamp to the next timestamp. For the memory cell as input at time t are the current external input vector or the inputs of the previous layers and also the output of the current memory cell. For time series forecasting problems the memory was implemented in such a way that the input units of the LSTM model perceived information from a set of past samples including a set of past delayed outputs.
The LSTM network consists of the delays and the
hidden layers’ sizes obtained from the time-series
data by applying training data. [
Although the number of tested inputs in this article is chosen rather arbitrarily, increasing the number of inputs from 12 to 18 lags reduces the forecast errors and provides clear evidence on the best prediction model for each forecast horizon and in accordance with all forecast evaluation metrics, such as RMSE and MAPE (please see Table in the Appendix). Moreover, the selected best networks for the whole out-of-sample analysis perform similarly even in the postpandemic period, particularly for horizons up to six months. For that reason, the following analysis on the performance of different types and structures of neural networks will pertain to the evaluation of models with 18 lags.
Sample Forecast Evaluation,
2019M01:2022M12 h=3 h=6 h=12
The forecast evaluation is based on the root mean square error (RMSE) metric, which is a quadratic scoring rule that measures average model prediction error. As such, lower RMSE values are an indication of better forecast ability. Graph 1 compares the performance of DFNN, DCNN and LSTM networks in predicting the lek-euro exchange rate during the whole out-of-sample period, starting from January 2019 to December 2022. It appears that none of the individual networks is shown to systematically provide superior forecasts for all of the tested horizons. The LSTM model with two hidden layers (LSTM-2L) stands out as the best prediction model in the run-up to forecast three and six months ahead, followed by the three-layered 1DCNN model (1DCNN-3L). However, they change places in the race for the 12 months horizon as the 1DCNN-3L becomes the first-best predicting model while leaving the LSTM-2L rank in second, even though the difference between their RMSEs does not seem to be significantly wide (1.98 vs 2.05, respectively). If classed together, a certain network group appears to perform better at certain forecasting horizons, but none of them stands out as the best prediction technique at every selected horizon, namely the 3, 6 and 12 months ahead. Conjointly, the couple of LSTM models with two and three hidden layers performs on average better at horizons 3 and 6, but they rank last as a pair at the one-year horizon. Also, the LSTM models show greater RMSE difference between each other if compared to the other coupled models in the DFNN and 1DCNN methods, suggesting the structure of LSTM models is more sensitive to the number of hidden layers. Finally, prediction errors of the LSTM models inflate in line with common intuition as we forecast for longer periods ahead, unlike most of the DFNN and 1DCNN models that display lower errors at the 12 months horizon than at 6 steps ahead. The forecast evaluation period (together with the validation sample) is characterized by an appreciation trend of the domestic currency. However, the strengthening of lek against euro has been quite uneven (please see Graph 2). In annual terms, the Albanian currency was enjoying an appreciation rate of 3.4% on average from January 2017 to March 2020, which gained momentum in 2018. The economic lockdown that aimed to contain the Covid-19 pandemic disrupted external trade of goods and services, including the tourism sector, thus leading to a year-on-year lek depreciation for the subsequent eleven months. Anyhow, the appreciation rate resumed again thereafter, averaging 2.2% for the period up to December 2022.
Graph 2. Year-on-year % change of the lek/eur exchange rate 4 2 0 -2 -4 -6 -8
I II III IV I II III IV I II III IV I II III IV I II III IV I II III IV 2017 2018 2019 2020 2021 2022 We are interested to analyze whether the neural networks ranking above holds for the entire period, or has changed in the post-Covid lockdown months. Graph 3 displays the root mean square errors after the pandemic lockdown months relative to the pre-Covid restriction period. A ratio of equal to, or greater than 1 indicates a deterioration in the forecast performance. Obviously, the forecast ability has improved by a large margin at every prediction horizon after July 2021. With the exception of the noticeable underperformance of the DFNN models at 3-steps ahead, the forecast gains of the others range between 32 and 78 percent, or an average ratio of 57% of the pre-Covid lockdown period. The improvements are most noteworthy for the DFNN and 1DCNN models, whose magnitudes of forecast errors are halved at 12months forecast horizon.
Graph 3. RMSEs for the post-Covid relative to the pre-Covid forecast
evaluation (excluding lockdown months) h=3 h=6 h=12 The widening differences in each model performance between sample periods have been reflected in the reconfiguration of gaps among the pairs as well as the changing of model positions at certain horizons. A quick view at Graph 4 (in comparison to Graph 1) reveals that the twolayered LSTM-2L model has confirmed its place as top-performer at horizons of 3 and 6 steps, but is ranked among the worst performers at the 12months horizon. On the other hand, the simplest DFNN-2L model has earned itself the top position in the post-Covid lockdown period for its ability to provide the best predictions at 12 steps ahead, albeit being among worst performers at shorter horizons.
Lockdown Period, 2020M07:2022M12 To summarize, the performance of our selected neural network methods and their structures varies over time and depends crucially on the selected parameters, such as the number of lags and hidden layers. The long short-term memory (LSTM) network with two hidden layers could be preferred as the best choice in predicting the lekeuro exchange rate up to six months ahead. However, the model seems to lose its appeal at longer horizons, especially in the more recent period. The LSTM technique should, nevertheless, be applied with caution as increasing the number of hidden layers is shown to deteriorate and underperform considerably in comparison to other less complicated neural networks.
On the other hand, the simplest deep feedforward neural network (DFNN), which is the quintessential deep learning method, might be the least capable at short term forecast horizons such as 3 and 6 months, but it is shown very promising at predicting 12-months ahead since the economic lockdown.
This paper investigates the applicability of different DL models to forecasting monthly LEK/EUR exchange rate. For this reason, we examine the performance of several deep learning techniques (DFNN, LSTM and 1D-CNN). For each architecture, we have used different configuration and diverse techniques to avoid overfitting of the models. The accuracy evaluation of each model was based on the outof-sample prediction for different horizon (3, 6 and 12 months ahead), by analyzing the model estimation for pre- and post- Covid pandemic. The comparison results show that the LSTM model with two hidden layers stands out as the best prediction model in the run-up to forecast three and six months ahead, followed by the three-layered 1DCNN model. However, they change places in the race for the 12 months horizon as the 1DCNN-3L becomes the first-best predicting model while leaving the LSTM-2L rank in second.
For the post-Covid lockdown period the twolayered LSTM-2L model has confirmed its place as top-performer at horizons of 3 and 6 steps, but is ranked among the worst performers at the 12months horizon. On the other hand, the simplest DFNN-2L model has earned itself the top position in the post-Covid lockdown period for its ability to provide the best predictions at 12 steps ahead, albeit being among worst performers at shorter horizons. These results demonstrate the potential of deep learning techniques, and also, they emphasize the importance of well configuring, implementing and selecting the different topologies.
[1] Claveria O., Monte E., Soric P., & Torra S., "“An application of deep learning for exchange rate forecasting”," University of Barcelona, Regional Quantitative Analysis Group, 2022,revised Jan 2022.. [2] Bandara, K., Bergmeir, C., & Smyl, S, "Forecasting across time series databases using ecurrent neural networks on groups of similar series: A clustering approach.," Expert Systems with Applications, vol. 140, no. 112896, 2020. [3] Lara-Benítez, Pedro et al., "An experimental review on deep learning architectures for time series forecasting," International Journal of Neural Systems, vol. 31, no. 2130001, 2021. [4] Lecun, Y.; Bottou, L.; Bengio, Y.; Haffner, P., "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 11, no. 86, p. 2278–2324, 1998. 1.35 1.45 1.44 1.76 1.60 1.44