Many manufacturers of electrical installations in smart home environments have developed and now ofer AI solutions that record and analyze the sensor data from their products. Their goal is to monitor and forecast runtime parameters, such as the energy consumption of heat generators or the cooling performance of air conditioning systems, for predictive maintenance and to optimize the carbon footprint. The training and deployment of such AI models can, though, be costly, necessitating intelligent techniques to consolidate, i.e., aggregate models of individual installations into fewer, but larger models. The aggregation of AI models, however, poses a challenging task due to the complexity of the systems and the variability of (hidden) factors that influence the forecasts. To solve the aggregation challenge, improve the forecasting accuracies and ultimately also reduce the AI deployment costs, this paper explores the concept of consolidating similar machine learning models with a novel clustering approach. We introduce CAML, a novel technique for (C)lustering and (A)ggregating (M)achine (L)earning models with shared characteristics. The clusters efectively capture the unique features of the contained models and can be combined into fewer AI models. Our evaluation shows that the hidden parameters learned by the baseline models are key factors in achieving accurate performance, underlining the importance of these models in the clustering process. Moreover, we demonstrate that by choosing the right model architecture, cluster models ofer a higher prediction certainty while exhibiting an only slightly higher average error compared to baseline models. Our experimental results show that CAML outperforms alternative clustering techniques in terms of prediction error and variance across multiple cluster configurations.
CEUR
ceur-ws.org (T. Papenbrock) CEUR Workshop Proceedings htp:/ceur-ws.org IS N1613-073
CEUR Workshop Proceedings (CEUR-WS.org) installations and their energy forecasting models.
Accurate energy consumption forecasts are critical for various stakeholders, including utility companies, building managers, and residents, because they enable optimizations in energy usage, predictive maintenance and ultimately cost reductions. The models employed for this purpose use historical sensor data to predict future energy consumption curves. Through careful feature engineering, they are also able to reflect parameter changes (e.g. adjusting the indoor temperature) in their predictions. However, accurate forecasts depend not only on features that are captured by sensors, such as temperatures and power consumptions, but also on various hidden parameters specific to the installation environment, such as the size, age, and insulation type of the building that houses the heating system.
Heating systems usually supply temperature at a constant level, heating water to a specific temperature and circulating it to provide warmth. This approach may, however, not be the most energy-eficient method, particularly during milder weather. Modern heating systems, therefore, use a dynamic approach, adjusting the supply temperature based on the outside temperature measured by an outdoor temperature sensor. This relationship is graphically represented by a heating curve, a crucial part of a heating system’s control strategy. These curves can be tailored to the residents’ preferences, which is an important variable for the accuracy of energy consumption forecasts. Additional factors include the desired indoor temperature, which is set via a thermostat, and the application of a night setback concept. The latter lowers the temperature during unoccupied periods like nighttime to save energy. These strategies make modern heating systems usually more eficient, but the interplay of these novel features combined with the hidden housing parameters of the installations make it dificult for manufacturers to develop generalizable AI models, such as the global forecasting model shown in Figure 1a. Instead, the current practice is to train, deploy and maintain a forecasting model (with potentially diferent hyper-parameters and model architectures) for every installation, as show in Figure 1b. Although the predictions of these models are decently accurate, their deployment and maintenance is expensive.
Despite the complex feature interactions and the numerous hidden parameters, many installations and, hence, their AI models are suficiently similar, such that, as depicted in Figure 1c, a smaller set of AI models could manage their forecasts. To find these partially generalizable models, we propose to cluster similar heat generator models and consolidate every cluster into only one cluster model. These cluster models are expected to encapsulates the unique features and encoded knowledge of each cluster. In our setting, this encoded knowledge refers to constant parameters that are relevant in the learning process but not directly visible within the data set, such as the size or insulation of buildings. By consolidating individual models into cluster models with an efective clustering, the number of models necessary for deployment and serving can be reduced significantly. To find these clusterings, we evaluate three diferent approaches: a) clustering the models by their underlying time series training data; b) clustering the models by their output data, which is their predictions; and c) clustering the models by their cross performance, which is the performance on each others training data. We demonstrate that approach c) delivers the most promising results, making it our suggested solution.
More specifically, we introduce CAML, a novel technique for (C)lustering and (A)ggregating (M)achine (L)earning models of any type. CAML uses a custom cross performance similarity function and hierarchical clustering to find clusters of AI models with similar hidden features; Cloud Buildings generators. 2 2
Current 3 3 4 4 ... ...
...
Buildings 1 2 3 4
(a) Ideal. One global forecasting model for all heat (b) Current. Each heat generator correspond to one
Cloud Buildings 1 1 2 individual model.
Cloud Target 2 3 ...
4 1 1 ...
(c) Target. Each of the heat generators is assigned to one of <
diferent cluster models. the current individual models approach (1b), and the target cluster model approach (1c). Heat generator installations send their sensor recordings into the cloud and receive temperature and energy forecasts. it then trains new cluster models on the clusters’ training data to consolidate the member models into one representative model. Our evaluation demonstrates that cross performance is more efective than input- and output-based model similarities; it also shows that the cluster models are significantly more accurate than a global model and almost as accurate as the many individual models. Hence, our key contributions are as follows: 1. Cross performance: We propose an efective similarity measure for model clustering that compares two models by their pair-wise test accuracies. 2. Hierarchical clustering: We introduce an unsupervised, hierarchical clustering approach with Ward-linkage based on the cross performance similarity measure that grands the ability to configure the number of target clusters, i.e., the trade-of between deployment costs and forecasting accuracy. 3. CAML: We present an algorithm that automates the entire model consolidation pipeline, including preprocessing, matrix computation, clustering, and subsequent retraining.
Clustering is an unsupervised machine learning technique that partitions objects into (mutually
exclusive) partitions based on their similarity w.r.t. a specific similarity measure. Clustering
algorithms can be categorized into several types. These include partitioning methods that divide
the data set into a pre-set number of clusters (e.g. K-Means [
Clustering
This section introduces CAML, our proposal for a (C)lustering and (A)ggregation technique for (M)achine (L)earning models. As visualized in Figure 2, CAML comprises four steps, which are data preprocessing, distance matrix computation, hierarchical clustering, and model aggregation. To execute these steps, CAML requires only one user-defined parameter: the number of clusters that controls the granularity of the clustering. The selection of strongly depends on the desired use case. Most commonly, represents the number of cluster models the user wishes to deploy. However, an exhaustive search of can be performed to meet certain prediction accuracy goals of the resulting cluster models compared to the individual models. Because CAML embodies hierarchical clustering, which produces a linkage matrix, multiple diferent clusterings can be extracted without recomputing the actual clustering procedure. We now introduce the four steps of CAML in more detail.
Preprocessing: Time series that contain anomalous data points (e.g., due to faulty sensor readings) are harmful to any clustering attempt. The removal of outliers is an essential preprocessing step as it helps to reduce the influence of anomalous data points on subsequent steps of clustering and model aggregation. These outliers, if not addressed, can distort overall patterns and relationships within the data, leading to suboptimal clustering results. Our CAML algorithm, therefore, initially preprocesses the data on which the individual models have been trained and tested, replacing the outlier data points in the time series with the value of the next, non-anomalous data point in the series. In this context, we define an outlier as a data point where at least one of its feature values is greater or less than standard deviations from the mean of that feature. By default, we choose to be 3, because we expect that each feature follows approximately a gaussian distribution, i.e., ( − 3 ≤ ≤ + 3 ) ≈ 0.9973 . Thus, only the extreme outliers are removed, and most of the original data remains.
Next, the time series of each model is split into training and test sets. We denote the training sets of model as = ( , ) and the test sets as = ( , ), with and as features and and as labels of the model, respectively. It is important to note that these splits need to be the same as used in the training procedure of the baseline model. The test sets are used in our proposed distance function for computing a complete distance matrix of the models. Distance Matrix Computation: The hierarchical clustering step of CAML is based on a distance matrix that stores all pair-wise distances between baseline models. These pair-wise distances should measure the dissimilarity between two machine learning models and and are computed with a custom distance function, which is based on the pair-wise cross-validation of the models on each other’s test sets. In other words, the distance between model and is the mean of ’s loss on ’s test data and ’s loss on ’s test data. We calculate this loss as the Mean Absolute Error (MAE) on the respective test sets. The intuition behind this distance function is to measure how well performs in the specific environment of and vice versa. If the loss of in the setting of is close to ’s loss in its environment, can be replaced with . By performing this calculation bidirectionally, we measure the extent to which the two models can replace each other.
Formally, let ∶ × → be a prediction model, which maps data points with diferent features to predictions, ∶ × → ℝ0+ be a non-negative loss function (e.g., MAE) and = ( , ) be the test set of the features and the label of a model . The distance between any model and can be computed as: ( , ) =
Here, ( ) denotes the predicted output of model on the test set of model , and
( ( ), ) is the loss between the predicted output ( ) and the true label . The distance
function is symmetric, i.e., ( , ) = ( , ), and it is non-negative, i.e., ( , ) ≥ 0.
Hierarchical clustering: For the actual clustering of the AI models, CAML uses a hierarchical
clustering approach [
The linkage method defines the distance of any pair of clusters based on the pair-wise object distances. The distances then determine the most similar pair of clusters in each iteration. Because the choice of the linkage method can significantly influence the results of the hierarchical clustering, we considered various popular methods including single linkage, complete linkage, and average linkage, which correspond to the minimum, maximum and average distances of objects in the two compared clusters, and found Ward linkage to be the most efective method in this step.
The Ward linkage method [
is the centroid of the observations in cluster , is a distance function based on the distance matrix, and are the data points.
Model aggregation: The model aggregation takes as input the clustering hierarchy and produces as output a set of cluster models. The cluster hierarchy is stored as a linkage matrix and can be visualized as a dendrogram. At first, the aggregation process simply cuts the dendrogram at the specific depth that creates disjoint clusters. Then, it consolidates every cluster into a single cluster model by training a new model on the combined training data of all individual models of that cluster. More specifically, let = { 1, ... , } be any clustering of baseline models. The cluster model for a given cluster with size = | |, the models ,1 , ... , , and their respective test sets ,1 , ... , , , where , = ( , , , ), and training sets ,1 , ... , , , where , = ( , , , ), along with a fitting function , is computed as in Equation 3 by merging all data of the cluster.
= ( ⋃ , , ⋃ , ) =1 =1 (3)
The AI models within a cluster are similar in their behavior, but they might use very diferent
types of models (e.g. linear regression, random forest regression, LSTMs etc.) and hyperparamter
settings. Our clustering results in fact show that model architectures have little impact on the
clustering results. The consolidation, therefore, needs to find a model architecture with similar
or higher capacity than the baseline models to capture all individual properties. Because the best
model for consolidation depends on the concrete baseline models, the model aggregation needs
to test a set of diverse model types to find the most efective model architecture for the cluster
models. Within our implementation, we test five models, which are
ExtraTreesRegressor [
In this section, we describe metrics for measuring the efectiveness of our clustering-based model aggregation. With CAML, we trained new cluster models that should replace their baseline models. To measure how well a cluster model generalizes its baseline models, we need a metric, with which we can compare the efectiveness of diferent cluster model sets. To evaluate the efectiveness, we measure the accuracy of each cluster model and compare it to the unconsolidated accuracies of the respective baseline models. The performance of a cluster model is then the average decrease in loss of the trained cluster model compared to all of its baseline models. Both the cluster model and its baseline models are tested with the respective test sets of the corresponding baseline models.
First of all, we measure the accuracy of a set of predictions ̂ w.r.t. the correct test labels as the mean average error (MAE) loss:
MAE( , ̂ ) =
|| 1 | | =1 ∑ | − ̂ | (4)
Given a cluster model for cluster with 1 ≤ ≤ and cluster size | |, we calculate the efectiveness score () of on cluster as the mean MAE loss over all test sets , = ( , , , ) whose model , belongs to cluster (Equation 5). Additionally, we compute an efectiveness score () of all baseline models , of cluster for comparison. Here, we compute the mean MAE loss over all baseline models , in on their own test sets (Equation 6).
| | | | =1 1 | | 1 | | =1 () = ∑ MAE( ( , ), , ) (5) () = ∑ MAE( ( , ), , ) (6) Let = ∑
=1 | | be the number of all baseline models, we can now calculate the overall accuracy of all cluster models and all baseline models over all clusters and by aggregating the individual scores () and () to total scores and , respectively. For this, we apply the weighted mean with the cluster size | | as weight (Equations 7 and 8).
= 1 =1 ∑ | | ⋅ () (7) = 1 =1 ∑ | | ⋅ () (8)
To judge the overall efectiveness of the clustering-based model aggregation we can simply compare to or some ′. If < for some clustering , the cluster models performed better than the baseline models. The value of − can be interpreted as the additional loss one encounters when aggregating the baseline models. We also consider the spread of the consider variance of the baseline model s’ prediction error on its test set = ( , ). clustering scores as the variance 2 of () and 2 of () (Equations 9 and 10). For 2 we 2 = 1 =1 ∑ | | ⋅ ( () − ) 2 (9) ∑(MAE( ( ), ) − )2
(10) 2 = 1
=1 Because the model consolidation efectiveness depends on the clustering efectiveness and the aggregation efectiveness, we evaluate both aspects in Section 5.
In this section, we first outline our experimental setup (Section 5.1) and, then, discuss our experimental results with CAML (Section 5.2) to demonstrate the algorithms efectiveness.
Measuring Quality: To assess the quality of our clustering-based aggregation technique, we leverage of the cluster models and of the baseline models. The baseline models act as a lower bound for the MAE, because they fit the individual setups and their (hidden) hyperparameters best. As an upper bound for the MAE, we train a global model (with diferent architectures) using the entire training data set without any clustering. To evaluate the proposed aggregation technique, we also test a clustering-based approach that randomly selects one of the baseline models in every cluster as a reprepsentative.
Benchmark Approaches: To evaluate the proposed cross performance-based clustering, we
benchmark two additional clustering approaches: training data-based and output data-based.
The training data-based approach clusters the data that the models have been trained on. Here,
we use hierarchical clustering with Ward linkage with the Shape-Based Distance (SBD) derived
from the k-shape algorithm [
Technology: CAML is written in Python 3.10 and can handle any model that implements the
Scikit-learn [
C 2 C 12 10 8 6 4 2 0
2 C 40 60 Retained models (%) 80 100
Because CAML creates a hierarchical clustering, the algorithm needs to choose a number of clusters , which is the number of to-be-created cluster. Our first experiment, therefore, evaluates the influence of on the clustering scores. The results are shown in Figure 3. The -axis shows the share of retained models in percent, while the -axis shows the scores and 2; the more clusters we use, the better each aggregated model can specialize on the specific installations. The goal is to create as few clusters as possible, while keeping the clusters’ mean MAE acceptably small. The depicted curve is an efective tool to tune for a specific application. For our application with 370 models, we chose = 55 (see vertical line) with the elbow method [28] in combination with as supervised metric of the cluster models prediction error instead of the usually employed sum of squared errors. Also practical constraints on the number of deployable models from our industry partner influence the choice of .
In a second experiment, we evaluate the three clustering approaches based on (a) training data, (b) output data, and (c) cross performance (CAML) by their MAE scores. For this experiment, the cluster models are built using the original model architectures, which are either ExtraTreesRegressor or GradientBoostingRegressor, depending on which yields better accuracy; we evaluate the model selection for the cluster models in the next experiment. Figure 4 plots the measured MAE scores for the three clustering approaches. Each subfigure plots the average MAE () of the baseline models on their respective test set in each cluster ( -axis) against the average MAE () of the cluster models on all test sets in their cluster ( -axis). If a measurement point is above the diagonal line, the baseline models performed better in this cluster; if a point is below the diagonal line, the cluster model performed better. The size of the marker indicates the cluster size. For every subfigure, we also provide the scores of the entire clustering (see 4) for the baseline models ( and 2) and the cluster models ( and 2). The measurements in Figure 4 show that cross performance-based clustering (with CAML) results in a lower prediction error and variance than training data or output data-based clustering. The training data-based clustering performs worst, because the hidden features are not represented in the input data and, therefore, do not influence this clustering – if they were represented, a globally aggregated model could learn from them as well and the clustering would not have been necessary to begin with. The output data and cross performance-based clustering approaches, however, both work well for the model clustering task, because they both capture the hidden features. The cross performance-based approach, however, has a slightly lower error and more dense clustering with less variance than the output data-based approach. For this reason, we propose the former with CAML.
In a third experiment, we evaluate CAML’s model aggregation. More specifically, given a specific clustering, we evaluate diferent model architectures and how models within a cluster can be aggregated. For this, we choose the clustering of CAML with = 55 (Figure 4) and implement multiple model architectures, both as cluster models and as global models. Figure 5 shows the distribution of the MAEs of these model architectures in comparison to the baseline models. Considering only the mean performances, we see that the individual models, i.e., the baseline models perform best, the cluster models sacrificed a little bit of precision for having a much smaller overall number of models, and training only one global model provides the worst performance. The fact that the cluster models’ mean MAE is very close to the baseline’s mean MAE demonstrates the efectiveness of CAML’s clustering. The ExtraTrees and LightGBM cluster models even outperform the baseline models in terms of MAE variance, i.e., prediction certainty. Another observation that highlights the efectiveness of the proposed clustering is that the performance of a randomly selected baseline model from each cluster as cluster model ofers a similar – although slightly worse – performance than all other cluster models, which retrain on the clusters’ data set. Considering the diferent model architectures, every global model performs worse than its respective cluster model counterpart. This also shows that the clustering efectively captures some hidden features. The model architectures, however, profit diferently from the clustering. Most notably, the neuronal network-based N-Beats model hardly improves with the clustering, because it tends to overfit the training data and, hence, has the same problems on clustered and non-clustered inputs; both the cluster and global N-Beats model perform worse than all cluster models. In summary, ExtraTrees, LightGBM, and XGBoost all (a) training data (SBD) (b) output data (Euclidean) 10 (c) cross performance (MAE; CAML) 10 8 6 4 2 c 0 0 C = 2.33 C2 = 3.10 B = 0.80 B2 = 0.97 8 8 6 4 2 E40 A M 20 0
ExtraTreCeLsliughsttGeBrMMXGoBdoeolsst N-Beats Random
ExtraTreeLsighGtGlBoMbXaGlBMooostdeNl-sBeats Random performed very well as cluster models for the aggregation step.
In this paper, we introduced CAML, a novel clustering-based aggregation technique for machine learning models that have been trained in specific environments with certain hidden but constant features. Due to these hidden features, no global model can replace all individual models. The clusters of models that CAML creates efectively capture these hidden features and serve to consolidate their models into much fewer models with still very good precision. With the proposed hierarchical clustering approach, data scientists can tune the trade-of between the number of to-be-deployed models and the models’ precision. Our experimental results demonstrate that CAML outperforms all globally aggregated models as well as our benchmark approaches, which cluster the models using their training data and output data, in terms of prediction error and variance. The consolidated cluster models tend to have a slightly higher average error than the baseline models, but the proposed model architectures have an overall lower variance, i.e., prediction certainty than the baseline models. In the context of the investigated application, CAML ofers a solution for the aggregation of energy consumption models, enhancing the practical value of these models for utility companies, building managers, and consumers, while simultaneously reducing operational costs.
Future Work: We evaluated CAML on a data set with only 370 models. Due to the positive results obtained with the proposed consolidation, the next step is to deploy CAML on a data set with, at the time of writing, several hundred-thousand models and their respective data. To make this possible, further research on enhancing CAML’s performance and/or scalability is needed, given that CAML requires the computation of a distance matrix with an expensive distance function and a runtime complexity of ( 2).
Acknowledgments: This work was primarily undertaken for Viessmann IT Service GmbH. We gratefully acknowledge the management of Viessmann IT Service GmbH for their support and for allowing the presentation of the results. 23 (2022) 1–6. URL: http://jmlr.org/papers/v23/21-1177.html. [26] T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama, Optuna: A next-generation hyperparameter optimization framework, in: Proceedings of the International Conference on Knowledge discovery and data mining (SIGKDD), ???? [27] J. H. Friedman, Greedy function approximation: A gradient boosting machine., The Annals of Statistics 29 (2001) 1189–1232. doi:10.1214/aos/1013203451. [28] R. L. Thorndike, Who belongs in the family? 18 (1953) 267–276. doi:10.1007/bf02289263.