If one examines the spot price series of electrical power over the course of time, it is striking that the electricity price across the day takes a course that is determined by power consumption following a day and night rhythm. This daily course changes in its height and temporal extent in both, the course of the week, as well as with the course of the year. This study deals methodologically with non-linear correlative and autocorrelative time series properties of the electricity spot price. We contribute the usage of non-fully connectionist networks in relation to fully connectionist networks to decompose non-linear correlative time series properties. Additionally, we contribute the usage of long short-term-memory network (LSTM) to discover and to deal with autocorrelation effects.
Despite all criticism of this approach, the random walk process has established itself for the modeling of stock prices. Pricing on electricity markets deviates significantly from the pricing on stock markets, as the underlying Markov property cannot be assumed for electricity markets as well. Produced electricity cannot be stored without significant losses and, accordingly, temporal arbitrage turns out to be highly inefficient. If one examines the spot price series of electrical power over the course of time, it is striking that the electricity price across the day takes a course that is determined by power consumption following a day and night rhythm. This daily course changes in its height and temporal extent in both, the course of the week, as well as with the course of the year. Accordingly, it can be concluded that the univariate time series shows non-linear correlative effects between daily, weekly, and yearly seasonal patterns as well as autocorrelative effects even without taking other explanatory variables into account.
The present study deals methodologically with non-linear correlative and autocorrelative time series properties of the electricity spot price. Correlation effects are adequately represented in classical fully connectionist networks but they cannot be meaningfully analyzed due to the high complexity of these networks. The research questions are, if the forecasting accuracy can be improved by (i) using different and complementary ANN-architectures to better reflect correlation effects and (ii) using a recurrent ANN-architecture to better reflect autocorrelation effects. To answer these questions, we contribute the usage of non-fully connectionist networks in relation to fully connectionist networks to decompose non-linear correlative time series properties.
Hence, we use (i) different ANN architectures with non-fully and fully connectionist networks to discover and to deal with correlation effects on exogenous side / input layer, (ii) using a long short-term-memory network (LSTM) to discover and to deal with autocorrelation effects, and (iii) an ARIMAX model with daily, weekly, and yearly seasonal patterns reflected as binary coded variables as a benchmark for the aforementioned models.
The paper is organized as follows: In section two, the current state of the literature is presented, and the research gap is identified. In section three, sample and methodology are introduced. In section four, the results are presented and discussed. The study closes with a conclusion. 2
Literature Review and Research Gap The number of electricity price forecasting articles has increased significantly in recent years. A particularly good overview can be obtained by Weron (2014). The author could identify 30 publications with a focus on ARIMA and its extensions. We could not identify further more recent articles in this special field of ARIMAmodelling of electricity prices. More recent electricity price forecasting literature is focused mainly on probabilistic forecasting and artificial intelligence. With regard to ANN, Weron could identify 56 publications. Subsequently, two further articles were published on electricity price forecasting using ANN that were not included in
Comparing ARIMA(X) models of the Spanish and the Californian market with and without additional explanatory variables, Contreras, Espinola, Nogales, & Conejo (2003) recognize that additional explanatory variables, such as hydropower, are only required in months of a high correlation between the explanatory variable and the price, while in months of low correlations these variables do not show significant predictive power. The authors were able to show average daily mean errors between
When forecasting with ARIMA, Conejo, Contreras, Espínola, & Plazas (2005) argue that it could be necessary to use a different notation of the model for nearly every week. Accordingly, ARIMA-models turn out to be very unstable in their predictive power over time. Especially in spring and summer where the volatility was very high the ARIMA forecast provided poor results. The authors also introduce several other techniques, e.g. an ANN with a multilayer perceptron and one hidden layer. The ARIMA model outperforms the ANN in every period except for the September. The mean week errors with ARIMA are between 6% and 27% whereas the ANN shows errors between 8% and 32%.
Garcia, Contreras, van Akkeren, & Garcia (2005) claimed that ARIMA-GARCH models show a better accuracy than seasonal ARIMA models. The authors present mean weekly errors of around 10% for relatively calm weeks. Misiorek, Trueck, & Weron (2006) compare some linear and non-linear time series models. In contrast to the aforementioned authors, the simple ARX model - the exogenous variable is the day-ahead load forecast - shows a better result than a model with an additional
Conejo, Plazas, Espinola, & Molina (2005) contributed a specified ARIMA model including wavelet transformation which was more accurate than the simple one. The wavelet transformation is applied to decompose the time series before predicting the electricity prices with ARIMA. This model outperforms the benchmark with a weekly error of 5% in winter and spring and 11% in summer and fall.
Applying a seasonal ARMA(X) process with three different explanatory variables of the temperature, Knittel & Roberts (2005) identified an inverse leverage effect with positive price reactions increasing the volatility more than negative ones. The authors further show that a higher order autocorrelation in the models is important to improve the results. The authors were able to show root mean squared errors for the out-ofsample week between 25.5 and 49.4 in the pre-crisis period and between 66.6 and 88.6 during the crises period. It is mentioned in the article that the data has a high frequency of large price deviations, which leads to these high forecast errors.
Zareipour, Canizares, Bhattacharya, & Thomson (2006) built an ARMAX and an ARX model with an average error in the 24-hour-ahead forecast of 8.1 and of 8.4 respectively, which is slightly better than the basic ARIMA model with an average of 8.8. With these models, it could be shown that market information in low-demand periods is not as useful as during high-demand periods. In general, the results have confirmed the contribution of the authors that market data improves the forecast results. Nevertheless, none of the models could forecast the extreme prices which increasingly occur in times of high-demand periods adequately.
Zhou, Yan, Ni, Li, & Nie (2006) suggested that including error correction will lead to a more accurate result in forecasting with ARIMA. Therefore, they developed an ARIMA approach which is extended by an error series. This novel method turned out to show quite good forecasting accuracies with an average error of 2% and lower despite of periods with a high price volatility.
Koopman, Ooms, & Carnero (2007) were using an ARFIMA model, which is an ARIMA model with seasonal periodic regressions, and combined it with a GARCH analysis. The authors pointed out the importance of day-of-the-week periodicity in the autocovariance function when forecasting electricity prices. Beneath the implementation of the day of the week, binaries were included for the holiday effect to consider demand variations.
With the increase in available computational power in recent years, ANN became
more and more popular in forecasting and forecasting research. Both, classical
multilayer perceptron (MLP) and recurrent networks
Recurrent ANN have the possibility to incorporate the output of latter layer units
again into earlier layer units, which is not possible in MLP networks. Commonly, the
units of all hidden layers of recurrent networks are in a chain-like informational loop.
A hidden unit can use its output as input (direct feedback), or it is connected to a
hidden unit of the preceding layer (indirect feedback), or it is connected to an unit of the
same layer (lateral feedback), or it is connected to all other hidden units (fully
recurrent). The recurrent type of LSTM is typically direct feedback
Fully connectionist MLP are the most used type of ANN for electricity price
forecasting
Weron (2014) concluded that forecasting with univariate time series models is well known in the extent literature. Accordingly, including the right external input factors into the models, as well as dealing with nonlinear dependencies between endogenous and exogenous variables and among exogenous variables will become more important. In contrast to the author, we do not see that the time for univariate time series analysis of electricity prices is already over, as we still cannot see a satisfactory approach to meaningfully deal with the time-series characteristics of electricity prices. Although the Bayes-approach offers possibilities, it is rather unsuitable for practical use due to the high load of computer capacities during simulation operations. Hence, we see a research gap in handling the non-linear correlative effects between the exogenously modelled daily, weekly, and yearly seasonal patterns as well as autocorrelative effects within the time series and among the exogenously modelled variables. Our contribution is to close this research gap by using an ANN-based methodology. We perform a time series analysis for the German EEX “Phelix” Data using (i) different ANN architectures with non-fully and fully connectionist networks to discover and to deal with correlation effects on exogenous side / input layer, (ii) using a long shortterm-memory network (LSTM) to discover and to deal with autocorrelation effects, and (iii) an ARIMAX model with time series features as binary coded variables as a benchmark for the aforementioned models. 3 3.1
Sample and Methodology
At the European Energy Exchange (EEX), electricity spot prices (EPEX Spot), as well as future contracts are traded. The vast number of German municipal utility companies, but also large industrial consumers on the demand side, and European electricity suppliers on the supply side take part at the electricity trading at the EEX. The electricity volumes can be traded on the same day (intraday) or for the following day (day-ahead). Purchase and sale orders can be placed on an hourly basis as well as for time blocks. The blocks are “baseload” (0.00am - 12.00pm) or “peakload” (8.00am - 8.00pm). These orders can be placed until 12.00pm of every trading day for the next calendar day and will be processed primarily over the internet. A computer system ensures the automatic settlement of the purchase and sale orders and the fixing of the exchange price. Finally, around 12.40pm the prices for the next day will be published via the internet and other data agencies.
The sample data used for this analysis is the EEX Phelix-DE day-ahead spot rate. It has established itself as a benchmark contract for European electricity. We considered time series data from January 1st 2015 until January 1st 2018 (Fig. 1). Each individual day has got 24 hourly price observations. The data underlying this analysis is complete
Time series for EPEX Spot ) h /MW 100 R (EU 50 e c i r 0 P y ilittrceE -001 c 2015.0 2015.5 2016.0
Since the storage of electrical power is not possible without significant efficiency losses, the price shows daily, weekly, and yearly seasonality patterns. The seasonality of the time series certainly has its origin in the electricity demand over a day and )40 h W M / R U E (e30 c ir P y iittr c c le20 E night rhythm (Fig. 2). Due to the daily, weekly and yearly seasonality patterns binary variables (“dummies”) for these categories were introduced. To capture the seasonality, our models contain 23 hour-dummies for the daily seasonality, 6 weekdaydummies for the weekly seasonality and 11 month-dummies for the yearly seasonality.
Average electricty prices per hour for selected months 0 5 10
15 Hour 20 25
Beneath seasonal and calendar day effects, the effects of wind power and solar
energy increase the volatility of the time series which is particularly challenging in the
prediction of the spot prices
In this study, our models are trained on a training data set of two years prior the predicted months. We predict the months March, June, September, and December 2017. The root mean squared error (RMSE) is selected to assess and compare the different models. In most of the extent papers, this is the standard forecasting accuracy measure (Weron, 2014). 3.2
ARIMAX-Model with Seasonality
The ARIMAX-model used in this study is an extension of the classical
ARIMAmodel, introduced by Box & Jenkins (1971). To include seasonality into the model,
the binary variables for hour, weekday, and month are applied in the X-term of the
model, which means, that these variables are supplemented as additional regressor in
the AR-Term. We used the Hyndman-Khandakar algorithm to find the best notation
for the ARIMAX model
As described above, usually
ANN are designed as fully connectio nist networks. suggest a slig htly differe nt approach to discover information about correlation of
We exogenous varia bles.
Therefore, a fo ur step approach is introduced: (i)
Sin gle
Perceptron (MLP) with hidden layer and partic ular mapping (non-fully connectio nist network), (iii)
hidden layer without partic ular mapping (fully connectio nist network), and (iv)
Memory (LSTM) with hidden layer, without partic ular mapping and direct feedback. A synopsis of the used arc hitectures is illustrated in used
The input layer is composed of units for hours, weekdays, and months as well as of units for lags.
The lag
The first three units in the hidden hidden are lag 1, lag 2 and lag 3 hours to be in line with the layers in model (ii) to (iv) consist of fo ur units. layer in (ii) are aggregated units. stands for an aggregated hourly information and gets its information corresponding hourly input units.
The second hidden unit is an aggregated weekday hidden unit and gets its information from all weekday input units.
The third
hidden The only first from unit the unit represents an aggregated month hidden unit and gets its information only from all month input units. The fourth hidden unit can be seen as an all-unit which gets its information from all input units including the lagged variables. The hidden layer in (iii) differs from (ii). Each unit in the hidden layer gets its information from all units of the input layer. There is not such a particular mapping like in (ii). The LSTM model in (iv) is equally designed as the MLP model in (iii), despite the direct feedback for each hidden unit.
The ARIMAX model, which is used as a benchmark model at this point, is able to recognize endogenous autocorrelation of the time series using the lag variables. The binary-coded seasonal variables control the seasonality as additive constants for certain hours, certain days of the week, and certain months via the ARX-term. Relationships between these seasonal components cannot be recognized by this type of model.
ANN-model (i) is essentially equivalent to OLS regression but additionally, it is able to adopt to non-linearities. It is therefore comparable to the ARX term of the ARIMAX model. If ANN model (i) should yield better results than the ARIMAX model, this is due to the ability to map nonlinear relationships as well.
The ANN model (ii) allows the explicit modeling of daily, weekly and annual seasonality through its aggregated units, but prevents the consideration of exogenous correlative effects among these seasonalities. In this respect, it serves as a benchmark for model (iii), which explicitly allows the consideration of exogenous correlative effects. If ANN-model (iii) now delivers better results than ANN-model (ii) we can assume that one reason must be unconsidered exogenous correlation, but its’ nature cannot be determined.
Expected better results of ANN-model (iv) would emerge the conclusion that the time series must have autocorrelative effects between seasonal binary variables, too. 4
The model forecasting accuracies in terms of the root mean squared error (RMSE) for the five models as well as for all tested cases can be found in Table 1. In the light of a monthly forecast horizon, the forecasting accuracies are in line with the expectation. In the extent literature, winter season is known to be more volatile and the price is more influenced by exogenous correlation effects, e.g. wind power. Accordingly, the comparatively poorer forecasting result in December is also in line with expectations from literature. It is striking, that the ARIMAX benchmark outperforms the SLP as well as both MLP in three out of four tested periods. Only in the December period all ANN models show better results than the ARIMAX model. Overall, however, the
The coefficients and model statistics of the ARIMAX model are given in Table 2 in the appendix. The coefficients show well pronounced daily, weekly and yearly seasonalities. Although the results are in line with expectation, the annual seasonality is based on only a few observations, resulting in high standard errors. The daily and weekly seasonality, on the other hand, can be described as stable, as inference is based on a large number of observations. It is evident that the daily and weekly seasonalities are far less exposed to structural changes than the annual ones. Nevertheless, the drift term of the ARIMAX model shows only low coefficients. By visual examination of the forecast, it can be seen that the ARIMAX model predicts repetitive daily patterns that oscillate slightly across the course of the week. Examples of this behavior for the months of March and December can be found in the charts in appendix 2.
The daily, weekly and annual seasonality already evident in the ARIMAX model coefficients are reflected in the weights of the SLP as well. Accordingly, it can be assumed, that due to the similarity to the ARX term of the ARIMAX, the poorer forecasting accuracy of the SLP is due to the nonexistence of the MA term in ANN or due to poorer adoption to the data as a consequence of the nonlinear activation function in the ANN. In the visual inspection, the model shows a less predictable behavior, in which daily patterns are recognizable but in a clearly distorted manner. The weaker forecasting quality is not surprising in this respect, although not in line with expectations. Which factor determines these distortions is not recognizable.
The importance of individual hidden units can be determined in ANN model (ii) by their weights to the output layer. It is striking that both, the hour unit, the day unit, and the month unit receive almost no weight and are therefore almost irrelevant to the model result. Only through the all-unit the electricity price forecast is achieved. Since the all-unit is comparable to the SLP, ANN model (ii) does not lead to a much better result than the SLP model. A possible explanation for this behavior is that only the interaction of the seasonal components and the lags provide a sufficient basis for the forecast. By visual examination of the forecast, it can be seen that the model shows a more repetitive result than the SLP, although here also unforeseeable distortions characterize the forecast.
In contrast to ANN model (ii), the weights in ANN model (iii) do no longer show seasonal structures. An interpretation of the individual weights is no longer possible. Even if there is still a strong weighting of a single hidden unit, the strongly correlative influence of the other hidden units on the forecast is clearly given. As already seen in the other MLP models, unpredictable distortions also shape the visual image of this model.
Comparable to ANN model (iii), the LSTM of ANN model (iv) shows strongly correlative influence, but strictly divided in two hidden units, whereof one hidden unit shows an excitatory and another hidden unit an inhibitory behavior. Two further hidden units show low weights, so that their influence is very limited. Due to the comparable architecture with ANN model (iii) the significantly higher forecasting accuracy is due to the ability to store the output of each unit and to feed it into this unit again (direct feedback). In other words: Not only the three lag variables fed into our model reflect autocorrelative effects but also the values stored inside the nets’ units to deal with long-term dependencies. Accordingly, the LSTM achieves significantly smoother daily patterns, similar to the ARIMAX model. Like all other models, the LSTM is also unable to predict exogenous shocks, leading to some distortions.
The superiority of the ARIMAX model and the LSTM network in comparison to the other ANN-architectures clearly shows that an additive consideration of seasonal effects for electricity prices is entirely sufficient. An alternative consideration of correlation effects does not provide improved forecasting accuracy. Thus, the problem of electricity price prediction focuses on autocorrelation effects, which can be better considered in the LSTM network than in ARIMAX.
Due to the fact, that all models are fed with the same data – including lagged variables – it is surprising, that the SLP and the MLP models are not able to smoothen the forecast. 5
The electricity price at the electricity exchange EEX shows daily, weekly, and annual seasonality patterns. Due to the cyclicality of the considered seasonal components there are non-linear correlative relationships between them. Thus, the present study deals methodologically with non-linear correlative and autocorrelative time series properties of the electricity spot price. We propose a systematic ANN-based approach to address this problem. The usage of different architectures sheds light on the strength of these relationships and their influence on electricity price prediction.
A single layer perceptron shows lower forecasting accuracy than a standard ARIMAX model with binary coded seasonalities used as a benchmark. Possible reasons for the poorer predictive quality can be specified: The non-linear activation function of the SLP and, above all, the missing MA term, which smooths the results in the
Appendix individually. The non-fully connectionist MLP shows only low correlations and a specialization of one unit considering all information. Accordingly, the forecasting accuracy cannot be better than in the single layer perceptron by large extent. This gap is closed by the fully-connectionist MLP, where all interactive relationships between these components find their way into the forecasting model. Last but not least, the long short-term memory (LSTM) model provides the most accurate forecast, which, in addition to the correlative relationships already mentioned, also included autocorrelative relationships on the endogenous side over several periods into the forecast. ) h W 40 M / R U E ( e irc P30 ity itrc c e l E 20 10 50 ) h W40 M / R U E ( e c ir30 P ity itrc c e l E20 10 )h40 W M / R U E ( e irc P ity irc20 t c e l E 0 60 40 ) h W M / R U E ( e irc20 P ity itrc c e l E 0 50 ) h W 40 M / R U E ( e irc P30 ity itrc c e l E 20 10 0 0 0 0 0 MLP (non-ful y connectionist) MLP (non-ful y connectionist) 60 ) h W M / R U E ( e irc40 P ity itrc c e l E 20 60 ) h W M / R U E ( ice40 r P ity itrc c e l E 20 60 ) h W M / R U (E40 e irc P ity itrc c e l E20 0 60 ) h SLP
SLP 50 10
150 19. Misiorek, A., Trueck, S., Weron, R.: Point and Interval Forecasting of Spot Electricity Prices: Linear vs. Non-Linear Time Series Models. Studies in Nonlinear Dynamics & Econometrics, 10(3), (2006). 20. Weron, R.: Electricity price forecasting: A review of the state-of-the-art with a look into the future. International Journal of Forecasting, 30(4), 1030-1081 (2014). 21. Zareipour, H., Canizares, C. A., Bhattacharya, K., Thomson, J.: Application of PublicDomain Market Information to Forecast Ontario’s Wholesale Electricity Prices. IEEE Transactions on Power Systems, 21(4), 1707-1717 (2006). 22. Zhang, G., Patuwo, E. B., Hu, M. Y.: Forecasting with artificial neural networks: The state of the art. International Journal of Forecasting, 14(1), 35-62 (1998). 23. Zhou, M., Yan, Z., Ni, Y. X., Li, G., Nie, Y.: Electricity price forecasting with confidenceinterval estimation through an extended ARIMA approach. IEE Proceedings - Generation,