This study uses data from domestic electricity smart meters to estimate annual electricity bills for a whole year. We develop a method for back-filling data smart meter for up to six missing months for users who have less than one year of smart meter data, ensuring reliable estimates of annual consumption. We identify five distinct electricity consumption user profiles for homes based on day, night, and peak usage patterns, highlighting the economic advantages of Time-of-Use (ToU) tarifs over fixed tarifs for most users, especially those with higher nighttime consumption. Ultimately, the results of this study empowers consumers to manage their energy use efectively and to make informed choices regarding electricity tarif plans.
Smart meters are advanced electronic devices designed to measure both the amount of electricity
exported to the grid and imported from the grid by a domestic or business customer. They ofer
consumers and energy providers detailed and up-to-date information on energy consumption compared
to traditional meters by eliminating the necessity for approximate meter readings [
The interval for metering and recording the consumption varies within the European Union (EU)
from 15 minutes to 2 hours, depending on the country [
As illustrated in Figure 1, thirteen member states in the EU achieved a smart meter roll-out rate
exceeding 80% in 2022, whereas ten member states had rates below 20% [
The electricity sector in Ireland has undergone a paradigm shift, transitioning from a regulated monopoly
to a deregulated competitive market [
The deregulation of the Irish electricity market was positively received by both consumers and
suppliers. This restructuring aimed to enhance consumer choice, lower retail prices, and introduce
innovative products. Consumers became more engaged leading to a better understanding of energy
costs and encouraging conservation eforts. The influx of competitive suppliers provided more choices
and attractive pricing structures, allowing them to cater to unique customer needs and diferentiate
themselves by providing innovative services and benefits. The Consumer Association of Ireland echoed
these sentiments, applauding the move as a “win-win situation for suppliers and consumers" [
The Commission for the Regulation of Utilities (CRU) functions as Ireland’s independent regulator
for energy and water, established in 1999 under the Irish Government’s policy and framework [
There is a list of all energy providers authorised by the CRU to furnish electricity and gas within the retail energy market in Ireland [12]. Presently, 13 energy suppliers serve the Irish Electricity market, catering to a population of 1.84 million households [13]. These are Arden Energy, Bord Gáis Energy, Ecopower, Energia, Electric Ireland, Flogas, Glow Power, Pinergy, PrePayPower, Yuno Energy Ltd, SSE Airtricity, Community Power, and Water Power. In total, they provide more than 60 tarifs for customers with smart meters as well as diferent fixed standing charge amounts depending on whether the household location is urban or rural.
Many studies have investigated domestic electricity consumption profiles across the world and one line of research has explored the relationship between the fundamental characteristics of households and their electricity usage patterns. Studies reported in [14, 15] utilising electricity consumption data from Danish households revealed that heat pumps, electric vehicles (EVs), and dwelling types exert significant influence on consumption levels, while socio-economic factors like occupancy, dwelling area, and income have minimal impact. Additionally, Munkhammar [16] discovered that houses equipped with EV charging predominantly see an increase in electricity usage during the evening while research in Germany [17] and Australia [18] demonstrated that households with photovoltaic (PV) systems tend to have increased electricity consumption.
The analysis of consumption data has included statistical profile analysis and clustering for profile extraction [19]. Multiple studies [20, 21, 22, 23] have evaluated various clustering algorithms and found K-means has a consistent ability to yield superior results in this context. These studies have also emphasised the importance of determining the appropriate number of clusters and we build on the lessons and conclusions from this previous work in our use of K-means to cluster our used into 5 profiles.
The data collection phase for this study was conducted from January 2024 to June 2024. During this period, two categories of data were gathered: electricity tarif plans from energy suppliers and smart meter data from households.
The tarif plan data were obtained directly from the oficial websites of 10 of the 13 energy suppliers. We excluded 3 oficial suppliers because their market penetration was small. As of June 2024, irrespective of whether the household location is urban or rural, 58 unique electricity tarif plans were then available, divided into either fixed-rate or time-of-use (ToU) tarif plans. Specifically, 17 fixed-rate plans provide a constant price for electricity regardless of the time of day, while 41 ToU plans vary the price based on the time of consumption and this includes both day/night and smart tarifs. The tarif plan details on the websites are updated periodically, influenced by the energy market thus checks on the tarifs are performed every two weeks by manually visiting the oficial websites.
ESB Networks who provide and maintain the electricity generation and distribution, allows customers to download their own smart meter data. Customers register on the ESB Networks website using their MPRN number and download a CSV file that includes all their electricity imports plus exports (if they micro-generate) from the installation date of their smart meter up to the previous day. This file is referred to as an HDF file, with a sample shown in the table. 1.
Data from smart meters installed in households and businesses was collected through a questionnaire. This included questions about respondents’ dwelling location (urban or rural) in order to determine which fixed-rate standing charge applies. Users uploaded their HDF files as downloaded from the ESB Networks website and in return received an email containing estimates of their annual bill across all available tarif plans in the market, based on their own past electricity consumption.
The service was released publicly on 31st January 2024 with ethics approval from the School of Computing ethics approval board. By 15th June 2024, it had received 193 responses and uploads, representing 113 unique users. Some users uploaded their HDF files multiple times to obtain more recent feedback as tarifs changed over time. For these users, all data entries from the same MPRN were combined by taking their earliest and latest end dates. The collected data comprises 127 million data points across these 113 distinct users.
All smart meter data from users were trimmed to fall into the period from midnight on 1st May 2023, to midnight on 1st May 2024, as there was significant variation in the duration of uploaded HDF file records for each user, ranging from over two years to less than one month.
HDF files downloaded from the ESB Networks website had occasional missing data which were sporadic and unpredictable, with the average deviation from the expected record count across all users being less than 0.43%. The irregularities resulted from factors including network transmission problems, power outages, and other unidentified issues. For monthly records, if the missing data exceeded 10% of the month’s duration, the record was excluded from our dataset.
After data cleaning, there were 107 users whose records covered at least one full calendar month. Among these, only 24 users had complete data for all 12 months because the smart meters for the others had been installed only within the previous 12 months. Such users are usually keen to use their smart meter data to find the best or cheapest tarif at the earliest possible opportunity, so are not willing to have to wait a full year after installation. Consequently, there is a real demand for predicting annual energy usage based on only partial data, as discussed later. Approximately 78% of our users have HDF data spanning more than 6 months. The distribution of data durations for these users is shown in Table 2.
Observing the average consumption of all our users over the 12 calendar months reveals a trend displayed in Figure 2. During the summer and warmer months from May to September, the consumption lfuctuates around 377 kWh per month and peaks at approximately 740 kWh in January. The cold Winter months from February to April still maintain a relatively high level of consumption, averaging around 627 kWh. The dip for February is because it has fewer days than other months.
Because of the varying durations in HDF records and the clear variation in usage based on seasonality, an annual consumption for those users who have missing HDF data as measured in months, cannot be accurately calculated as an average monthly consumption multiplied by 12. Therefore, in the following analysis of annual consumption, only the 24 users with complete data spanning the entire 12-month period are included in summary Table 3. While the sample size is relatively small, the data presented, influenced by improving living standards, challenges the accuracy of the CRU’s recommended typical annual electricity consumption figure of 4,200 kWh, which was announced in 2017 [ 24] and on which all comparator websites for energy costs, including those regulated by the CRU such as bonkers.ie and switcher.ie, are based. An analysis of data from all 107 users provides insights into the broad spectrum of electricity consumption behaviours. Figure 3 shows the ratios of electricity used across the 3 standard daily ToU timeslots for these 107 users. Here we see a notable portion (70) of these users display a trend of higher energy use during daytime, while 37 users show increased consumption patterns at night. This analysis illustrates the need for a more precise and personalised estimation of households’ electricity consumption when selecting tarif plans, rather than relying on the CRU’s stated annual domestic electricity consumption figure of 4,200 kWh.
Users with complete HDF records spanning all 12 months had their past electricity usage calculated, and their estimated electricity bills for the subsequent 12 months were computed based on all available tarif plans. Here we assumed usage patterns would not change and that consumption from the previous year would be the same as for the subsequent year. For users with HDF records covering fewer than 12 months, a methodology was devised to estimate their electricity usage for the missing months with the collected dataset. This enabled the prediction of annual bills for these users, across diferent tarif plans.
The overall methodology workflow is displayed in Figure 4, including profile extraction, electricity usage prediction, evaluation and tarif plan comparison.
Our 107 users’ HDF records were segmented into 12-month bins. For each month, the total usage was calculated for each of three diferent daily timeslots as used in ToU tarifs: day (8:00-17:00 & 19:00-23:00), night (23:00-8:00), and peak (17:00-19:00). The average monthly usage for the three time periods were then calculated across the users. The monthly usage ratios for these periods were calculated relative to the total 12-month usage creating a series of data for each user, consisting of 36 ratios that sum to 1. Users were clustered using K-means clustering based on the 36-dimension feature vectors.
As mentioned earlier the K-means [25] algorithm has proven to be the most efective for processing smart meter data. It clusters data by partitioning it into k groups of equal variance, aiming to minimise inertia criterion across clusters. K-means is both simple and eficient, scaling efectively with a large number of samples and has been widely applied across various domains. In our study, K-means used Euclidean distance between user profiles as the distance metric which can be expressed in terms of the Euclidean norm of the diference between p and q vectors, or users in our case:
To determine the optimal number of user clusters, it is important to evaluate using both inertia and the Silhouette score, which are two widely used metrics for assessing clustering algorithms. Inertia [26], also referred to as the within-cluster sum of squares, quantifies the compactness of clusters and decreases as the number of clusters increases. Silhouette score [27] assesses how similar an object is to its cluster compared to other clusters, taking into account both cohesion (similarity within the same cluster) and separation (dissimilarity between diferent clusters). Silhouette scores range from -1 to 1, with higher values indicating more well-defined clusters. Diferent numbers of clusters were tried in this investigation, and the results are shown in the elbow plot in Figure 5.
Balancing Inertia and Silhouette scores is crucial and Figure 5 suggests that using 5 or 6 clusters is the best choice. Despite the Silhouette scores being similar for 5 and 6 clusters, choosing fewer clusters is preferable, hence we chose 5 clusters. The centroid for each cluster was computed and users with missing HDF data for certain months would be assigned to the appropriate cluster by adjusting cluster centroids to match the lengths of available data for these users. This ensured accurate distance calculations to determine the closest cluster for each case of missing data.
Users with at least 12 months of smart meter data have annual bills estimated for each tarif and the procedure for back-filling electricity consumption data for users with less than 12 months of data involves several steps and leverages similarities in electricity consumption patterns from their closest user cluster. This is shown in Figure 4. In step 1, users were matched to the closest cluster profile based on only the number of months for which they have data. In step 2 the day, night, and peak consumption ratios for the missing months were determined based on that closest cluster profile. Finally, in step 3, the estimated monthly energy consumptions for each time slot in each month was back-filled using the values from the cluster and normalised by the actual consumption figures for the months for which there is data.
We assessed the accuracy of our approach to back-filling HDF data by performing an evaluation to assess clustering accuracy, verify if cluster assignments aligned with the closest cluster profile, and confirm the accuracy of back-filling for new users who submitted questionnaires after 15 th June 2024, and who have more than 12 months of data.
As indicated by the data duration analysis shown in Section 3.3, 78% of our users who uploaded their HDF files were missing up to six months of usage data from within the past year so we reconstructed where is the forecasted value, is the actual value, and is the number of forecasts.
SMAPE [28] is intended to measure the relative diference between predicted and actual values, taking into account their magnitudes in a balanced way, where 0% signifies a perfect prediction. SMAPE can efectively address the issue of having small or zero actual values in the denominator, which can lead to a very high error percentage. It ofers a more balanced and robust measure as a result [ 28], compared to Mean Absolute Percentage Error (MAPE). The SMAPEs for the three standard ToU periods are then weighted based on their respective duration within the 24 hours —- 13 hours for day, 9 hours for night, and 2 hours for peak —- resulting in an overall error value. The clustering was deemed accurate if the assigned profile exhibited the best overall performance, as indicated by the lowest weighted SMAPE value.
When evaluating the feasibility of back-filling smart meter data for between one and six months, the objective was to confirm that this extended period could produce reliable predictions, as indicated by relatively low-weighted SMAPEs. This analysis would establish confidence in our back-filling process and ensure the reliability of future forecasts.
Five consumption clusters were identified based on a K-means analysis of te 36 features of the 24 users’ electricity usage patterns, all exhibiting a similar annual trend of increased usage in winter and decreased in summer. We refer to the automatic clusters as profiles and across all profiles, users demonstrate relatively low electricity usage during peak times. However, there are diferences in their lifestyles and consumption behaviours between daytime and nighttime. The profile summaries are shown in Figures 6 and 7.
As shown in Figure 7, Profile 1, which we denote as the “day profile" , is the most prevalent, comprising 55% of households and predominantly consumes more electricity during daytime hours. In contrast, Profiles 2, 3, and 4, collectively representing 36% of users, are categorised as “night profiles" where households tend to use more energy during nighttime hours, especially during winter, possibly charging an EV or availing of cheaper energy to run a heatpump. Finally, Profile 5, which accounts for 9% of households, displays a relatively balanced electricity consumption pattern between day and night but night usage outstrips day usage during the Winter months. These distinct consumption profiles highlight a range of lifestyle and consumption habits among diferent households.
The sample users we used in this evaluation had submitted HDF files with full coverage of the investigation period and as of July 2024, there were four such users (shown as A, B, C, and D in Table 4) who met the criteria. Clustering results indicated that Users A and B were assigned to Profile 1 for each of the six back-filling durations (back-filling for only the first month, then for the first two months, ..., and finally for the first six months). User C was clustered into Profile 3 for back-filling of their first to their first four months, and into Profile 4 for back-filling for five to six months. User D was assigned to Profile 5 across all six back-filling months.
The weighted SMAPE of estimated back-filling values vs. actual values are presented in Table 4. For User A, when back-filled for one or two months, Profile 3 achieved the lowest SMAPE at approximately 15.3%. However, when back-filled for more than two months, Profile 1 consistently achieved a low SMAPE, around 14.3%, suggesting it provides the best overall accuracy. The pattern for User B is more pronounced, as the weighted SMAPEs for Profile 1 are consistently the smallest across all back-filling durations. For User C, the SMAPEs are lowest in Profile 3 when back-filling for up to five months whereas for the last back-fill duration, Profile 1 and Profile 3 have the same accuracy. The results from these sample of users validate the clustering outcomes. However, for User D, Profiles 1 and 4 show lower SMAPEs, which contrasts with our clustering which assigns User D to Profile 5. Further investigation revealed that User D’s distances to the cluster centroids were very close across all profiles except Profile 2, suggesting that the user has a stochastic electricity usage pattern that does not align closely with any of our extracted profiles.
Overall, these findings validate the clustering outcomes, highlighting the reliability of our clustering methodology in capturing stable load patterns. However, accurately predicting energy consumption at the individual level may be less feasible for users exhibiting erratic load profiles and higher variability as supported by previous research [29].
For the stable users A, B, and C, the average weighted SMAPEs across six back-filling durations are approximately 15.4%, 10.4%, and 19.3%, respectively. These weighted SMAPE values are less than 20%, which is considered indicative of good forecasting performance [28]. This confirms that back-filling for up to six months in our study is reliable despite the apparantly small amount of data used to create the clusters, while maintaining reasonable prediction accuracy and highlights our confidence in accurately estimating with six-month gaps in historical data.
Substantial savings are available in domestic energy bills when a user chooses the most economical tarif for them, which may be ToU or fixed-rate depending on the amount and the characteristics of their use. It should be stressed that this analysis is based on usage patterns from profiles and that individual users may find that their most economical tarif may be either a ToU or a fixed-rate one. For that, we direct users to URL blocked to preserve double-blind submission but available in final version where they can upload their own HDF data and where the analysis on their on data is performed directly and the results sent to them.
In this study, smart meter usage (HDF) data from more than one hundred users were gathered and analysed to determine five typical consumption profiles. Due to data collection time and acquisition channels, our sample size is relatively small compared to data gathered from national surveys. As the smart meter installation program is still ongoing in Ireland and elsewhere, many households have had them installed within the last year, leading to a limited duration usage data for such users.
For users who do not yet have a full calendar year of their own historical HDF data, we developed and evaluated a method to estimate and back-fill their usage to allow an estimation of their full annual energy bill. This was done by categorising them into one of five consumption profiles and using the usage patterns of the profile to complete the back-filling. The findings indicate that these estimates are reliable when predicting based on up to six months of missing data. Households in Ireland can choose from more than 60 tarifs from 10 suppliers but choosing the cheapest for each household can only be done using their smart meter data which captures each household’s unique usage pattern. This work allows users to use a full year of their own smart meter data to find their best energy tarif in a way that accounts for seasonal factors, even if they have only 6 months of their own actual data.
In summary, this study provides insights into an important area, namely estimating electricity consumption with a view to understanding the economic advantages of choosing the best tarif plan. It has the potential to benefit stakeholders by deepening customers’ understanding of their own energy use and helping them to choose the cheapest tarif for them. Future work will explore the economic impact of diferent tarifs from diferent energy suppliers in terms of their efects on consumer’s annual bills.
This work was partly-supported by Research Ireland under Grant Number: SFI/12/RC/2289_P2, cofunded by the European Regional Development Fund.