Deep Learning Forecasts of the Electricity Price with special Consideration of the Electricity Supply

Abstract

The market price for electricity at the energy exchange is mainly determined by supply and demand. For many market participants, electricity price forecasting is an important competitive factor. Previous research primarily predicted based on demand. In times of renewable energies this mantra is no longer sustainable. This study primarily looks at the supply-side prediction of the electricity price with the help of deep learning artificial neural networks and thus makes a contribution to the literature. Autoregressive models and regressions serve as benchmarks.

Keywords

• electricity price forecasting
• ARIMAX
• regression
• artificial neural networks
• deep learning

Introduction

Due to the global increase in population and the rise in the general standard of living, a reduction in electricity demand is not foreseeable [ 2 ]. Fossil fuels continue to be the foundation of the global energy supply: coal, natural gas and crude oil. Worldwide, renewable energies such as hydropower and biomass contribute around 18% to covering electricity demand [35].

With about 70% of imported raw materials, Germany is one of the largest energy consumers worldwide. For this reason, the German government's goal is to increase the share of renewable energies in primary energy consumption from 14% (as of 2018) to 40-45% by 2025 [ 3 ]. Despite this, coal and nuclear energy have been very important energy sources for decades, even though their share in electricity generation has fallen from 84% in 1990 to currently less than 50% [33].

On the European electricity exchange (EEX), the price for electricity is determined by supply and demand. It should be noted that this is only a peak balancing, as most participants in the electricity market cover themselves by means of long-term electricity supply contracts with the power plant operators. Only short-term peaks are balanced via the electricity exchange.

It is essential for market participants in the electricity market to be able to make substantive statements about future electricity price developments. Above all producers and traders can gain great competitive advantages through accurate electricity price forecasts.

In the past, electricity supply largely followed demand. In order to ensure network stability, the amount of electricity fed into the grid must always be adapted to demand. This is ensured by using peak load power plants, such as gas turbine power plants. The latter produce comparatively expensive electricity compared to base load power plants such as coal-fired and nuclear power plants. These types of power plants can be adapted very quickly Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). to fluctuating electricity demand, whereas base-load power plants have a very high time latency. Due to the increasing use of renewable energies, such as wind and solar power, and the resulting displacement of base-load power plants, the problem of maintaining grid stability remains, but the electricity mix is constantly changing and with it the electricity price. The final consequence of this is that the supply of electricity can no longer be fully derived from the demand for electricity, and consequently the supply of electricity is a factor influencing the price of electricity. However, the latter is still not independent of the demand for electricity, so that collinearities and endogeneities arise.

The forecast of the electricity price has also been a relevant topic in the literature for many years. In this respect, a large number of authors have taken up this challenge. However, the focus of most studies is on the demand side of electricity price forecasting. A recent research has detected 105 papers that are dedicated to a demand-side electricity price forecast. In contrast, 11 papers (see Error! Reference source not found.) clearly underrepresented the supply-side electricity price forecast.

This paper examines and explains the development of electricity prices by means of a supply mix. It covers feed-in quantities of biomass, lignite, hard coal, gas, oil, oil shale, peat, geothermal energy, waste, water, solar, wind and other renewable energies. The methods used are multivariable linear regression, ARIMA(X) [ 4 ] and artificial neural networks (KNN). For KNNs, a distinction is made between classical feed-forward networks, in the form of a single-layer perceptron (SLP) and multi-layer perceptron (MLP), and recurrent networks (Hopfield 1982), or more precisely long-short term memory networks (LSTM) [ 17 ]. The two variants MLP and LSTM are designed as deep learning networks. Deep learning generally stands for a special form of KNN, which is characterized by well thoughtout successive layers with a higher number of units each [ 30, 13 ]. In MLP as well as in LSTM, the activation of units into the hidden layer is done by means of rectifiers [ 15, 16 ], which achieve better training results especially in deep networks [ 12 ]. The regression as well as the ARIMA(X) model serve as benchmarks for the comparison with the KNN.

The work is structured as follows: The second chapter reports on the state of research. The third chapter presents the sample and methodology. In the fourth chapter the models and the resulting results per model are presented and explained. The article closes with a summary. 2

Prior Research

For European markets, the electricity price is determined by supply and demand. A large number of papers already exist on demand-side electricity price forecasts. Weron [34] provides a general overview of a variety of methods and proposals as well as an outlook for the next decade. The author divides the models into five categories: multi-agent models, basic (structural) models, models in reduced form (quantitative, stochastic), statistical models and computer-aided intelligence models. With artificial neural networks, one, if not the heavyweight of artificial intelligence in electricity price forecasting is becoming increasingly important. An up-to-date and detailed overview in this respect is provided by Meier et al. [25], who take the electricity price forecast of the European market as a starting point, using different models such as ARIMAX, regressions and different KNNs, mainly using autocorrelative and non-linear correlative time series characteristics such as day, week and year related seasonalities. For probabilistic electricity price forecasts, Dudek [ 9 ] proposes an approach based on feed-forward networks. Unlike Dudek, Zhang [36] combines the advantages of a non-linear KNN and a linear ARIMA model to leverage the strengths of both methods and provide greater predictive power. In the same way, Filho et al [ 10 ] follow a hybrid approach tailored to the Brazilian market. The results of this approach are compared with classical models such as ARIMA, GARCH, Exponential Smoothing and KNN. For a forecast period of 24 and 36 weeks, the hybrid model clearly outperforms the forecast accuracy of the other models mentioned. However, the lack of general validity for other markets remains to be mentioned. Raviv et al [28] examine the forecast accuracy of uni- and multivariate models for hourly electricity prices of the North Pole market. The forecasts are based on average prices for the next 24 hours. The multivariate models perform significantly better, according to an up to 15-20% lower RMSE. Here too, the question of general transferability to other markets remains open.

Relatively few studies deal with a supply-side electricity price forecast. Huisman et al [ 19 ] deal with hourly electricity prices on several day-ahead markets. When looking at hourly electricity prices, the authors, like Raviv et al. [28], also consider the simultaneous submission of electricity prices and the formation of an average price for 24 hours on one day to be inappropriate, since hourly electricity prices do not follow a time series process. For this reason they model a panel model with 24 cross-sectional hours.

Nowotarski and Weron [26] use a quantile regression for electricity price forecasts. In contrast to other statistical analysis methods, quantile regression allows the use of many distributions without restrictions. This flexibility also makes this approach suitable for forecasts, which is reflected in low forecasting errors. Díaz et al. [ 8 ], who use quantile regression, also use an hourly electricity price forecast for the Spanish market.

Contreras et al [ 6 ] use ARIMA(X) models for price forecasts of the Spanish and Californian markets. The central result of this work is the effect of the strength of the correlation between the price and the explanatory variable on the inclusion of other explanatory variables as well as on the forecast accuracy. In case of a strong correlation, the average daily mean error is between 5% and 10%. If the correlation is weak, additional explanatory variables have no significant effect on the forecast.

For the Colombian market with hourly electricity prices, Marin et al. [24] find that ARMAX and NARX (non-linear autoregressive neural networks) lead to similar prognosis values.

ARIMA(X) models are often used in the literature as a benchmark to make the results of KNN approaches transparent. Keles et al. [ 21 ] investigate the advantages of MLP compared to ARIMA(X), whereby the KNN has lower prediction errors.

Singhal and Swarup [31] also believe that an MLP is the most appropriate means of daily electricity price forecasting. This is characterised by the mastery of complex interrelationships of given factors such as price and historical load.

Gökgöz and Filiz [ 14 ] are setting up 400 MLP models for the Turkish market, which differ in the number of units and various activation functions. The most suitable model has an average absolute percentage error (MAPE) of 9.76%.

Li et al. [23] also focus on the investigation of the properties of MLP for forecasting price time series. For shorter time intervals the KNN provide a higher accuracy of the forecast values than comparable ARIMA models. Overall, the forecast accuracy of the MLP amounts to more than 80%.

The following Error! Reference source not found.the statements once again in a clear manner. The Association of European Transmission System Operators for Electricity, ENTSO-E for short, represents 43 electricity transmission system operators from 36 countries, including Germany, Spain and France. Since the introduction of Regulation (EU) No 543/2013 of 14 June 2013, data providers and owners of data from the European Member States have been required to present information on electricity generation, use, transmission and balancing on the ENTSO-E transparency platform.

The energy quantity data required for the forecast models are taken from the ENTSO-E transparency platform. The data is derived from the query settings "Generation" and "Actual Generation per Production Type" as well as "DE-AT-LU" for the period from 01.01.2015 to 31.01.2018. The sample comprises the quarter-hourly available feed-in quantity in megawatts (MW) of 20 energy types: biomass, lignite, gas produced from coal, gas, hard coal, oil, oil shale, peat, geothermal energy, hydropump storage, run-of-river power plant, water reservoir, marine, nuclear, other, solar, waste, other renewable energies as well as wind offshore and wind onshore. The corresponding hourly electricity prices are from the European Energy Exchange EEX. Due to the different compression levels of the feed-in quantity (quarter-hour) and the electricity price (hour), the average of the quarter-hourly feed-in quantities is formed for the respective feed-in quantity per hour for the purpose of temporal adjustment on an hourly basis. To take the time component into account as a predictor, dummy values are integrated for the hour, weekday, and month.

The following Fig 1. shows the course of the electricity price.

Fig 1. Course of the electricity price of the entire data volume 3.2

(Pre-) Processing of the Data The entire data volume is first subjected to pre-processing. Missing values are imputed on the basis of spline interpolation. The time predictors month, weekday, and hour are extracted from the date and dummyfied. This results in 11 dummy variables for the month (January to November), 6 dummy variables for the weekday (Sunday to Friday) and 23 dummy variables for the hour (0:00 to 22:00). The data set is then divided into a training data set and a test data set, as usual for KNN.

For the training data set, a Kwiatkowski-Phillips-Schmidt-Shin (KPSS) test is performed, which provides the number of differentiations to establish stationarity. The electricity price of the training and test data set is differentiated once according to the result of the KPSS test. A partial autocorrelation function (PACF) analysis of the training data yields a significant number of lags (period-weighted time series) of 5. The use of Auto-ARIMA [ 20 ] confirms these results. The course of the electricity price differences can be seen in the Fig. 2.

Then both sets of data are normalized using min-max scaling to transform the values of all predictors - except the dummy variables - into the range 0 to 1. For the test data set, the scaling factors (min, max) of the training data set must be used. The dummy variables, on the other hand, are subjected to an effect coding. The usual procedure in statistics for dealing with categorical characteristics using 0/1 coded dummy variables may prove problematic for KNNs and lead to suboptimal solutions in the adaptation of a KNN. Strictly speaking, the number 0 is regarded as critical. A mathematical proof can be found in Sarle [29]. Instead, he argues for an effect coding with -1/1 coded dummy variables.

Within the framework of the investigation of the training data set for model maintenance, a k-fold cross validation [ 7, 11, 22 ] in the form of a "sliding window" is used to optimize the hyper-parameters of the CNN. In k-fold cross validation, the entire training data set is broken down into k individual, equally sized parts, the so-called folds. Usually the folders are selected randomly. In time series, however, data sets of the training data set that follow each other directly must always be combined into a fold, since the order of the data sets is decisive and must not be confused. With the Form Sliding Window, when training a KNN one fold per iteration is successively used as training data set and the immediately following fold as validation data set and the MAE (mean absolute error) per epoch is calculated. After the end of the training of a KNN, the average of all MAE values per epoch can be determined. From the visualization of these values (x-axis: epoch; y-axis: Ø-MAE), the number of epochs for which the MAE is minimal can be read. This number of epochs is considered the optimum number of epochs. Subsequent epochs should not be taken into consideration, as the increasing MAE value indicates an overfitting. The optimal number of epochs is used to generate the final or generalized model based on the total amount of training data.

The RMSE (root mean square error) is used to compare the prediction quality of the respective final models, ARIMAX, Regression, SLP, MLP and LSTM, with the test data set. This metric is considered the standard measure for metric quantities in the literature[34].

Results

The first KNN is an SLP, which can be regarded as a classical regression model substitute. The 4-fold cross-validation has resulted in an optimal epoch number of about 30. Repeated training of the SLP yields an RMSE of 4.07 for the training data set and 4.77 for the test data set. The results are shown in theFig. 3

The second KNN is a deep learning MLP with three hidden layers (128, 64 and 32 units) and respective rectifier activations. The 4-fold cross-validation has resulted in an optimal number of epochs of about 60. Repeated training of the MLP yields an RMSE of 2.88 for the training data set and 4.13 for the test data set. The results are shown in the Fig. 4.

The third KNN is a Deep Learning LSTM with also three hidden layers (128, 64 and 32 units) and respective rectifier activations. The 4-fold cross-validation has resulted in an optimal number of epochs of about 50. Repeated training of the LSTM provides an RMSE of 2.79 for the training data set and 4.09 for the test data set. The results are shown in the Fig. 5

The following Error! Reference source not found.the results and presents the RMSE values of all models used for different amounts of test data. A one-week test period is selected for each quarter to take into account seasonal variations in the test data sets. The four test datasets cover the periods 1.1.-7.1.2017, 1.4.-7.4.2017, 1.7.-7.7.2017 and 1.10.-7.10.2017.

RMSE for test periods Model Jan 2017 Apr 2017 Jul 2017 Oct 2017 total ARIMAX 4,37325 2,51524 2,24350 4,61178 Regression 4,32585 2,55180 2,23342 4,63412 SLP 4,81282 2,67128 2,38036 4,82037 4,77 MLP 4,06866 2,25496 2,27558 4,69374 4,13 LSTM 4,29037 2,36789 1,97528 4,99047 2,79 In the regression analysis, the variables lignite, gas, hard coal, oil, hydropump storage, water, solar, wind offshore, wind onshore, months 5-8, all days of the week and all hours are significant predictors for the forecast model.

In the overall view of the prognosis models, none stands out as dominant, even if MLP and LSTM deliver noticeably better results than the simpler prognosis methods, this seems to vary with the seasons, as already known from other studies. 5

Conclusion

This paper examines and explains the supply-side electricity price development, taking into account a large number of fed-in electricity quantities as well as relevant time factors in the form of month, weekday and hour. ARIMAX, regression and different KNN are used. The best forecast results are achieved by the LSTM, which, as expected, is best able to deal with time series. However, depending on the season to be forecast, it is not yet completely convincing.

Since supply-side electricity price forecasts are still very much underrepresented in the literature, the present study can serve as a basis for further replication and comparison studies. 22. Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection.

In: Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence. San Mateo, CA: Morgan Kaufmann. 2 (1995) 12, S. 1137-1143. 23. Li, G., Xu, S., Li, Z.: Short-Term Price Forecasting For Agro-products Using Artificial Neural Networks. Agriculture and Agricultural Science Procedia, International Conference on Agricultural Risk and Food Security 1 (2010), S. 278-287. 24. Marin, J. B., Orozco, E. T., Velilla, E.: Forecasting electricity price in Colombia: A comparison between Neural Network, ARMA process and Hybrid Models. In: International Journal of Energy Economics and Policy 8 (2018) 3, S. 97-106. 25. Meier, J.-H., Schneider, S., Schmidt, I., Schüller, P., Schönfeldt, T., Wanke, B.: ANN-Based Electricity Price Forecasting Under Special Consideration of Time Series Properties. In: Ermolayev, V., Suárez-Figueroa, M. C., Yakovyna, V., Mayr, H. C., Nikitchenko, M., Spivakovsky, A. (Eds.): Information and Communication Technologies in Education, Research, and Industrial Applications, Communications in Computer and Information Science. 2019.

Springer International Publishing, S. 262-275. 26. Nowotarski, J., Weron, R.: Computing electricity spot price prediction intervals using quantile regression and forecast averaging. In: Computational Statistics, 30 (2015) 3, S. 791-803. 27. Peng, L., Liu, S., Liu, R., Wang, L.: Effective long short-term memory with differential evolution algorithm for electricity price prediction. In: Energy 162 (2018), S. 1301-1314. 28. Raviv, E., Bouwman, K. E., van Dijk, D.: Forecasting day-ahead electricity prices: Utilizing hourly prices. In: Energy Economics 50 (2015), S. 227-239. 29. Sarle, W. S.: comp.ai.neural-nets FAQ, Part 2 of 7: Learning. Abruf unter http://www.faqs.org/faqs/ai-faq/neural-nets/part2/. 30. Schmidhuber, J.: Deep Learning in Neural Networks: An Overview. In: Neural Networks, 61 (2015), S. 85-117. 31. Singhal, D., Swarup, K. S.: Electricity price forecasting using artificial neural networks. In:

International Journal of Electrical Power & Energy Systems 33 (2011) 3, S. 550-555. 32. Traber, T., Kemfert, C.: Impacts of the German Support for Renewable Energy on Electricity Prices, Emissions and Profits: An Analysis Based on a European Electricity Market Model.

Discussion Papers of DIW Berlin 712, 2007, DIW Berlin, German Institute for Economic Research. 33. Umweltbundesamt: Stromerzeugung erneuerbar und konventionell. 27.09.2019. Abruf unter https://www.umweltbundesamt.de/daten/energie/stromerzeugung-erneuerbar-konventionell. 34. Weron, R.: Electricity price forecasting: A review of the state-of-the-art with a look into the future.

In: International Journal of Forecasting 30 (2014) 4, S. 1030-1081. 35. World Bank: World Development Indicators: Electricity production, sources, and access. Abruf unter http://wdi.worldbank.org/table/3.7. 36. Zhang, G. P.: Time series forecasting using a hybrid ARIMA and neural network model. In: Neurocomputing 50 (2003), S. 159-175.

References

1. Al-Saba , T. , El-Amin , I. : Artificial neural networks as applied to long-term demand forecasting . In: Artificial Intelligence in Engineering 13 ( 1999 ) 2 , S. 189- 197 .

2. BGR Energiestudie 2018 . Abruf unter: https://www.bgr.bund.de/DE/Themen/ Energie/Downloads/energiestudie_2018.html.

3. BMWi: Artikel Erneuerbare Energien. Bundesministerium für Wirtschaft und Energie . Abruf unter https://www.bmwi.de/Redaktion/DE/Dossier/erneuerbare-energien.html.

4. Box , G. E. P. , Jenkins , G. M. : Time Series Analysis: Forecasting and Control . 1970 . San Francisco: Holden Day.

5. Catalão , J. P. S. , Mariano , S. J. P. S. , Mendes , V. M. F. , Ferreira , L. A. F. M. : Short-term electricity prices forecasting in a competitive market: A neural network approach . In: Electric Power Systems Research 77 ( 2007 ) 10, S. 1297 - 1304 .

6. Contreras , J. , Espinola , R. , Nogales , F. J. , Conejo , A. J.: ARIMA models to predict next-day electricity prices . In: IEEE Transactions on Power Systems 18 ( 2003 ) 3 , S. 1014- 1020 .

7. Devijver , P. A. , Kittler , J.: Pattern Recognition : A Statistical Approach . 1982 . London: PrenticeHall.

8. Díaz , G. , Coto , J. , Gómez-Aleixandre , J.: Prediction and explanation of the formation of the Spanish day-ahead electricity price through machine learning regression . In: Applied Energy 239 ( 2019 ), S. 610 - 625 .

9. Dudek , G.: Multilayer perceptron for GEFCom2014 probabilistic electricity price forecasting . In: International Journal of Forecasting 32 ( 2016 ) 3 , S. 1057- 1060 .

10. Filho , J. C. R. , Affonso , C. de M., de Oliveira , R. C. L.: Energy price prediction multi-step ahead using hybrid model in the Brazilian market . In: Electric Power Systems Research 117 ( 2014 ), S. 115 - 122 .

11. Geisser , S.: Predictive Inference . 1993 . New York: Chapman and Hall.

12. Glorot , X. , Bordes , A. , Bengio , Y. : Deep Sparse Rectifier Neural Networks . In: Gordon, G. , Dunson , D. , Dudik , M. (Eds.): Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS) , Fort Lauderdale, FL, USA. Proceedings of Machine Learning Research (PMLR) , 15 , 2011 , S. 315 - 323 .

13. Goodfellow , I. , Bengio , Y. , Courville , A. : Deep Learning . 2016 . Cambridge, London: MIT Press.

14. Gökgöz , F. , Filiz , F. : Electricity price forecasting in Turkey with artificial neural network models . In: Investment Management and Financial Innovations 13 ( 2016 ) 3 , S. 150- 158 .

15. Hahnloser , R. H. R. , Sarpeshkar , R. , Mahowald , M. A. , Douglas , R. J. , Seung , H. S.: Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit . In: Nature , 405 ( 2000 ) 6789, S. 947 - 951 .

16. Hahnloser , R H. R. , Seung , H. S. : Permitted and Forbidden Sets in Symmetric Threshold-Linear Networks . In: Leen, T. K. , Dietterich , T. G. , Tresp , V. (Eds.): Advances in Neural Information Processing Systems 13: Proceedings of the 2000 Conference . 2001 , S. 217 - 223 . Cambridge et al.: MIT Press.

17. Hochreiter , S. , Schmidhuber , J.: Long Short-Term Memory . In: Neural Computation , 9 ( 1997 ) 8 , S. 1735- 1780 .

18. Hopfield , J. J.: Neural networks and physical systems with emergent collective computational abilities . In: Proceedings of the National Academy of Sciences , 79 ( 1982 ) 8 , S. 2554- 2558 .

19. Huisman , R. , Huurman , C. , Mahieu , R.: Hourly electricity prices in day-ahead markets . In: Energy Economics 29 ( 2007 ) 2 , S. 240- 248 .

20. Hyndman , R. J. , Athanaspopulos , G. ( 2014 ): Forecasting: Principles and Practice. 2014 . OTexts.

21. Keles , D. , Scelle , J. , Paraschiv , F. , Fichtner , W. : Extended forecast methods for day-ahead electricity spot prices applying artificial neural networks . In: Applied Energy 162 ( 2016 ), S. 218 - 230 .