The goal of this paper is to present an efficient load forecasting algorithm for large electric power systems. It uses a combination of nearest neighbor-based load profile clustering and rule-based load forecasting. The load data was sliced into daily load curves, which were K-Means-clustered, thereby compressing data and simplifying the solution. K-Means was chosen in the proof of concept phase and will be substituted with more precise solutions later. In the forecasting phase the daily load profile is predicted based on the forecast date, day type (e.g. weekday or weekend) and historical consumption data for similar days in the past. The solution was tested on a large dataset consisting of one year-long, 5-minute measurement data in a 1900-power-line system. The solution showed excellent performance in both the training and forecast phases. It produced meaningful forecasts even when the input data contained significant amounts of anomalies. An additional advantage of the presented solution is that it can be used for medium and long-term forecasting with limited and/or missing input data.
The challenge to accurately predict the power flows in today’s large electric power systems receives ample attention. Numerous papers are published in specialized smart grid journals with the promise of being able to predict the electricity consumption of single households or their groups. Others develop solutions which predict the power flows in electric power transmission systems, which span over large geographic regions, e.g. sizable parts of continental Europe or the USA. Yet another group of scientists works on data compression algorithms, with the intention to lower the communication and storage costs incurred in modern smart grids.
Within this setting, we start from the idea that the flows on the power lines in electric power transmission systems have some form of periodicity. More specifically, we will theorize that the configurations of these large systems does not change frequently, and under the same load conditions the flows will be similar on the power lines for similar days, e.g. for Wednesdays in July the load will most probably be very similar under the same loading conditions. Therefore we propose a 2-phase load forecasting algorithm, consisting of a daily load profile clustering and a load forecasting phase.
The following sections of this document contain more detailed description of each of the above steps. 2
The body of electric load forecasting knowledge is very large, with numerous papers published in all major domain-specific journals and conferences. As an extensive review of all relevant solutions would not be feasible due to the page limits, we will only refer to those research results, which specifically focus on time series clustering, smart meter big data management and the combination of solutions from these domains used in load forecasting. 2.1
Reference [
Reference [33] describes a convolutional neural
network-based time series classification solution, in which
the time series features are automatically learned instead
of handpicking. The authors describe the process of
data preparation, filtering, and the structure of the used
network. The authors of reference claim that
semisupervision can boost time series clustering performance
[
Reference [
The authors of reference [
The solution presented in reference [
A more general review of smart meter data intelligence
is provided in references [
Reference [
The authors of reference [
The authors of references [
It is necessary to develop a load forecasting solution which is capable to predict loads in extremely large, Europe-wide electric power transmission systems consisting of thousands of power lines. The input data will consist of historical measured loads with a sampling rate of 5 minutes available for at least the last 1-year period. This data will be referred to as dynamic data, due to its frequency of change. Due to data privacy limitation, the input data will not contain the complete static data model of the system under consideration. This means that there will be no data provided to build a mathematical graph consisting of the busbars (vertices) and power lines (edges) connecting them.
It is expected that the prediction horizon will be 5 minutes ahead. The solution should be extensible and be capable to provide acceptable mid- and long-term forecasts (1 day or 1 week ahead) as well. Additionally, it is necessary for the solution to handle temporary unavailability of significant amounts of measurements, when those will not be provided in a timely manner by one or more countries and/or companies in the geographical area under consideration. Optionally, the solution should be able to incorporate weather forecast and other freely available 3rd party data and thereby increase the accuracy of its outputs. The relevance of such data might vary, as the extent of data anonymization required will not allow the forecasting tool access to the geographical location of system resources (i.e. power lines). 4
We suppose that the power flows measured on the power lines in large electric power systems show some level of regularity and can be therefore classified into load profiles. Based on this assumption, we propose to create a hybrid forecasting solution which consists of two phases. In the first phase the electric power flow data is clustered into daily load profiles. In the second phase we experiment with various forecasting algorithms to predict the daily load profile for each power line based on historical data. This means that instead of predicting the expected values of power flows 5 minutes in the future, we predict the load profile for an entire day in advance.
This solution addresses most of the more complex requirements listed above, namely it is expected that it can handle missing data and create forecasts for multiple days ahead. More specifically, it tolerates the absence of significant amounts of short-term historical data and still produces meaningful forecasts based on medium- or longterm historical data (e.g. data older than a week or month). Similarly, medium- or long-term (7 days or more ahead) forecasts are feasible with this type of solution.
In the following sections we propose the load profile generation and forecasting steps. 4.1
The power flow data is sliced into daily (24h) load profiles consisting of measured flow values sampled every 5 minutes. The sliced daily load profiles are normalized. The amplitude of each daily load profile is memorized. The normalized daily flows are clustered, separately for each power line, i.e. a set of representative load profiles is calculated for each power line. For each year, month, day of the week and power line we memorize the load profile and amplitude.
The daily load profile clustering introduces some error, but significantly improves algorithm performance if it is not necessary to re-calculate the centroids too often. As the main idea is that the daily load profiles will be similar, this should not be an issue, i.e. we should re-calculate the representative daily load profiles relatively rarely, e.g. once in 15 or 30 days. 4.2
Our improved baseline load forecasting solutions is rulebased. In the predict phase it looks up the most likely historical load profile and amplitude for each of the power lines. They are used to calculate the predicted load profile for the entire prediction day and extract as many samples as required. This means that we will predict for a whole day, i.e. 288 future values with a 5-minute sampling interval.
Predictions spanning two calendar days are somewhat challenging as in their case it is necessary to either select two load profiles and stitch them together; or to select the ‘end’ and the ‘start’ of the same load profile. In the actual solution we perform the latter, simpler solution, i.e. use a single daily profile to cover both calendar days before and after midnight.
Load profile prediction The load profile selection is rulebased, and it is performed in the following steps (listed by priority): 1. if there are (past) values for the same (year, weekday, power line) tuple within the last two months, then select one; 2. if there are (past) values for the same (year-1, month, weekday, power line) tuple, then select one; 3. if there are historical values for (year – (2:N), month, weekday, power line) tuples, then select the most likely one – N is configurable and defaults to 15; or 4. if none of the above is found, do a random load profile selection.
Amplitude calculus Amplitudes are chosen as averages of historical values for similar days in the past. This part of the solution is also rule-based, and its steps/choices are very similar to the curve selection algorithm presented above (also listed by priority of choice): 1. choose non-zero amps for the same weekday within the last two calendar months and average them; 2. average last year’s amplitudes for the same month and weekday combination; 3. average the values in the longer-term history up to M years in the past, where M is configurable and defaults to 15; or 4. choose a default amplitude, which was for simplicity set to a (configurable) scalar. 5
The input dataset was loaded from Heterogeneous Data Format (HDF), version 5. We used HDFView version 3 for data exploration. The training data consisted of historical power flows for 1935 power lines over a July-toJuly one-year period expanding over two calendar years. The training data consisted of 3-month-long power flows over a July-September period immediately after the training data. The training data was split into 68 data slices ranging in length between one hour and a couple of dayslong. Both training and testing power flows were sampled every 5 minutes.
We implemented the solution in Python version 2. We used the scikit-learn library for K-Means and other necessary ML algorithms. Data visualization during experimentation was performed with matplotlib. 5.1
The number of representative load profiles (i.e. centroid count) was set to 10, which was shown to be a sufficient during data exploration. The clustering was performed by K-Means, which was chosen due to its efficiency. Each load profile was a curve consisting of 288 numerical values.
We re-trained the model if a sufficiently long time period expired since the last training performed. For simplicity we performed full re-train is runs from scratch. This design decision was acceptable as the clustering phase for the 1-year period and 1935 power lines took up to 20 minutes on a personal computer with an Intel i7 CPU, 8 GB of RAM and SSD.
Example 1-week input load profiles are shown in Figure ??. Example load profiles detected for the above single power in its corresponding 1-year-long flow data with K-Means and cluster number 5 are shown in Figure ??. Note that the cluster number of five was used here only to illustrate the clustering results in a visually pleasing diagram. Otherwise, during most experiments cluster number 10 was used. The introduction of these load profiles allowed us to significantly reduce the solution space, i.e. to replace 365 (or more if the dataset spans a longer time period) daily curves with a small set of load profiles. Daily loads in the historical data were essentially represented with tuples consisting of a load profile identifier, daily amplitude multiplier (as the curves were normalized in the (1, 1) range), year, month and weekday. We used weekdays as based on past experience, and the related works, we theorized that load profiles will be quite similar in a certain day of the week in each month of the calendar year, e.g. customer electricity use is usually similar on each Saturday in July if the weather is good.
As explained earlier, we reduced the size of the historical dataset by introducing the load profiles and thereby simplified the forecasting task. The original, vast dataset consisted of 288 daily flow measurements with a 5-minute sampling rate, collected for 365 days and 1916 power lines (i.e. 288x365x1916 = 201,409,920 values). With the load profiles we reduced the dataset to a tuple consisting of the power line identifier, date, load profile identifier and amplitude, i.e. the multiplier with which the load profile is multiplied to obtain the ‘original’ load measurements. This meant that instead of 288 floating point values for each of the day and power line combination, we received a tuple consisting of the above four elements, i.e. 4x365x1916 = 2,797,360, which was a data reduction by 72 times, i.e. almost two orders of magnitude.
We randomly selected one power line and created a plot of the assigned load profile identifiers with K-Means over a 60-day long period, starting with a Thursday (i.e. weekday identifier 3). The resulting diagram can be seen in Figure ??. The diagram covers a 60-day period during the July-August period. We can see in the diagram that for almost all weekends (for Saturdays 2, 9, 16, and Sundays, 3, 10, 17, etc.) the representative load profile was with ID = 0. Additionally, we can see that load profile 4 was very frequent for weekdays.
The power flow amplitudes for the same power line and period is shown in Figure ??. We can clearly identify an anomalous period around day 30, similarly as in the load profile diagram above. Such periods are usually related to periods with different weather conditions (e.g. colder, rainy days with less use or air-conditioning) and/or configuration changes in the power system. In this diagram we might identify dips in amplitudes during the weekends (day 0 is a Thursday), but there is no other clear regularity identifiable via visual inspection. 5.2
We transformed the date information into a tuple of three values, namely year, month and weekday. With this modification, the inputs fed into the load forecasting code were tuples representing historical daily loads in the following format: (line id, year, month, day of the week, profile id, amp) We implemented the rule-based, baseline algorithm as described above. The forecasting code expected the following inputs:
The code transformed the dates into the (year, month, day of the week) sub-tuples and subsequently looked up the most similar historical data as explained in section IV/B. The load forecasting code returned two values, namely the ‘expected’ load profile identifier and amplitude for the prediction day.
Experiment I: Short-term forecasting We compared the results of our load forecasting solution to the persistence model, which simply predicts that the N future values will be equal to the N historical values preceding them. The persistence model implementation used for testing was limited to short-term forecasts for the next N values immediately following the last time period received in the training data, i.e. it did not support time gaps between the latest training data and the forecasting period.
We experimented with N=(12, 24, 48), i.e. a 1, 2 and 4 hour short-term prediction horizons. One execution of our prediction code took around 30 seconds to execute, regardless of the selected interval length. The time to execute the same prediction task on the persistence model was quite similar, which meant that most of the time was spent on creating the resulting datasets.
The RMSE errors calculated with the persistence and our model are shown in Table ??. The error was calculated between the prediction values and measured values extracted from the adapt/test data.
Experiment II: Mid-term forecasting As explained above, the proposed solution is capable to produce meaningful mid and long-term forecasts with limited historical data availability. We tested the algorithm on the following three prediction tasks:
1 week ahead, i.e. predict the power flows for the day one week in future compared to the last training data item.
As the persistence model used during this research did not have built-in support for these types of prediction tasks, we measured the RMSE for the proposed model only. Table ?? contains the results of our measurements.
We decided to further explore the resulting predictions (i.e. load forecasts) by visually comparing the predicted load curves to the actual measurements received as part of the test/adapt data. A randomly selected load flow prediction and real daily load values are shown in Figure ??.
The values predicted one week in the future (in red color) are in a similar value range, i.e. they do not have an inverted sign or exceedingly different amplitudes. Not surprisingly the anomalous zero value in the (real) measured value is not predicted by the presented load forecasting algorithm.
In Figure ?? we present the resulting daily load curve for the same power line as in the previous example.
The relatively low accuracy levels can be improved by implementing a more accurate load profile clustering technique, instead of the relatively coarse K-Means, with a different distance measure. Such changes would allow the authors to find the most relevant daily load profiles. 6
This paper describes a power flow prediction algorithm, which relies on analyzing historical (power) flow information and creating a configurable number of representative daily load profiles for each power line. Predictions are based on high performance look-ups – a single load profile index is selected for the target prediction day, for which a (daily) load profile is calculated. The algorithm is rule-based as opposed to creating a neural network-based or other machine learning solution. It can be tweaked further, and one might expect from it to produce deterministic results for the target electric power system under consideration. It might be used as a baseline solution, against which less deterministic, machine-learning solutions can be compared and measured.
The main advantages of the presented algorithm are its training and prediction performance. It analyzes the historical flow information and creates configurable numbers of representative daily load profiles for each power line. Predictions are based on high performance look-ups – a single load profile index is selected for the prediction day, a predicted (daily) flow curve is calculated for the whole calendar day for which the prediction is initiated. The predicted values are ’stitched’ (i.e. amplitudes are augmented) to the actual input flows and the requested number of predicted samples is returned. The solution can make predictions based on (very) limited information and handle gaps in input data. It is also capable to quickly predict the most likely and meaningful flows for extended future periods, i.e. instead of covering only a couple of hours immediately after the last input (training) data received, it is capable to predict a day, week or even longer periods ahead. Prediction accuracy will obviously vary as a function of the amount of historical data, i.e. if there are historical flow values measured multiple years in the past, then we expect to obtain higher accuracy. This was not shown in our experiments though, as the training data available covered only one calendar year.
The solution was tested on dataset consisting of power flows collected over a one year, July-to-July period. The test data covered the July to September period immediately following the training data. The system under consideration consisted of 1916 power lines. The accuracy of the proposed model was compared to the persistence model. We showed that the RMSE was quite similar in short-term forecasting tasks (1 to 4 hours) to the persistence model. We measured the accuracy of our algorithm in mid-term load forecasting scenarios, whose length was set to be 1 day, 1 week, as well as 1 and 3 months in the future.
The main disadvantage of the algorithm is its reliance on historical flow information only, i.e. it does not take auxiliary information into consideration. Additionally, the algorithm does not specifically cover national holidays, which often fall on weekdays and result in weekend-like load profiles - this missing element is relevant if the algorithm is used for state-level forecasting, but has lower relevance in continent-wide (e.g. Europe) load forecasting scenarios, in which the impact of national holidays is lower. Further tuning and optimization of the load profile classification and curve/amplitude selection algorithms might further improve performance and accuracy. (2018) 861–868. [33] Ze¸bik, M., Korytkowski, M., Angryk, R., Scherer, R.: Convolutional Neural networks for time series classification. In International Conference on Artificial Intelligence and Soft Computing. Springer, Cham. (2017, June) 635–642.