Active learning has proven to be an efective approach for reducing the amount of labeled data required in supervised learning tasks, thereby reducing data annotation costs. While ensemble-based active learning schemes have been extensively studied for classification problems, there has been limited research on their applicability to regression tasks. In this paper, a novel active learning method for regression ensemble learning is proposed, which utilizes meta-learning. The meta-learning component is employed to predict continuous utility values for candidate unlabeled data points. The sample selection process is designed to consider both ensemble accuracy and diversity simultaneously. Furthermore, the ensemble model and the meta-learner share the same features, enabling the provision of suitable explanations for selecting specific samples during the active learning procedure, thus enhancing the ensemble performance. Empirical testing of the proposed method is conducted on various real-world regression datasets, evaluating its performance and scalability. The results demonstrate its competitiveness when compared to state-of-theart approaches in active learning and ensemble learning for regression.
In supervised learning, data collection and annotation are essential stages. In passive learning,
training examples are chosen randomly from a distribution and labeled by an oracle. Usually, a
significant amount of data points is needed to train a reliable machine learning model. However,
data annotation can sometimes be associated with high costs. Henceforth, reducing the amount
of labeled data points is necessary. This is broached in the machine learning literature with
Active Learning (AL) [
a small and non-optimal training set, AL aims at iteratively selecting unlabelled data points whose inclusion in the training set improves the performance of the machine learning model. The unlabelled data points are evaluated and sorted according to a utility measure that serves as a data selection criterion. The selected data point is labeled by an oracle and then added to CEUR the training set. The entire procedure is iterated until a stopping criterion is met, e.g., a budget on the total number of points to be added or on annotation costs is consumed.
Compared to classification, active learning (AL) is less commonly used for regression tasks
[
Ensemble learning is widely known as an efective technique in machine learning for both
classification [
To do so, we introduce METAL a novel, practically useful METa-Active Learning framework
for learning regression ensembles. METAL is based on a meta-learning algorithm that learns
the utility of a set of candidate unlabelled data points to be added to the training set for learning
the ensemble model. Since diversity is a fundamental component in ensemble learning [
We note that all the experiments are fully reproducible, and the code is available under this link1. The datasets are publicly available.
Opposingly to classification, AL studies gave less attention to regression [
Meta-learning was also applied for AL by learning the utility/informativeness of an unlabelled
instance based on a set of characterizing meta-features, generally devised to take into
consideration the main learner’s performance and the characteristics of the problem. Even though
the application of meta-learning for estimating the utility of unlabelled data appears to be very
intuitive, only a few works applied it to learn AL sampling strategies and were mainly restricted
to classification [
Active sampling is also applied to learn an ensemble of classifiers. Shan et al. [
METAL combines both meta-learning and AL to reduce annotation costs and optimize the construction of an ensemble model for regression tasks. To do so, a meta-learner is trained to learn a utility measure that is devised to take into account both ensemble accuracy and diversity. The selection of unlabelled instances during the AL process is based on the expected values of the utility on the unlabelled set.
Let the dataset be defined as ( , ) ∈ ⊂ ℝ × ℝ ∶ = {1, ⋯ , } and generated by an unknown function () = , where is the number of features of a data point , and denotes a numerical response variable. We formulate a regression problem as the task of learning a function ̂ ∶ → ℝ such that
̂ ( , ) ≈ ( ) = , ∀ ∈ X, ∈ Y where ∈ ℝ is an unknown (hyper)parameters vector.
Denote with an ensemble of of regression models ̂, = {1, ⋯ , }
Formally, can be written as the convex weighted combination of the base models. on an input data point is given by: () = ∑ () =1 where , ∈ [1, ]
are the ensemble weights. The weights are constrained to be positive and sum to one. This constraint is necessary for some of the following results. For simplicity, we set the weights to be equal, i.e., =
1 ∀ ∈ [1, ] . Denote with the labeled data set, i.e., containing annotated data generated by the unknown function . We split into that is used to train the models composing the ensemble and that will be used to learn the meta-model. The meta-model is denoted by the selector in the following. Let be the unlabelled data set. Our goal is to actively sample data points from to learn the ensemble model that best approximates . (1) (2) (3) where denotes the utility measure of an unlabeled data point . However, adding the same instance to these M subsets will increase their similarity after several iterations. The base models are expected to become similar in this case, e.g., as trained on the same data, especially in the
Ensemble learning is inspired by the principle of committees. In fact, it is assumed that there is no single expert that outperforms all the others on every query. Instead, better overall performance may be obtained by combining the outputs of many experts, i.e., models. In this work, we use AL for annotation costs optimization and as an informed sampling strategy for the ensemble construction process. To do so,
diferent hypotheses are drawn from the data by means of an active sampling procedure. These hypotheses are used to generate the ensemble base models. Initially, these models are created by sampling randomly with replacement subsets , , ∈ [1, ] from with the same size. The models can create either a homogeneous ensemble if they belong to the same family of regression models or a heterogeneous ensemble if they are selected from diferent families of models. Afterward, the selector is built and trained using training The instance with the highest predicted utility ∗ is selected to be added subsequently and used afterward to predict the utility of the unlabelled instances in . The to the subsets created to build the ensemble base models.
∗ = ∈ = () case of homogeneous ensembles. This alters the diversity aspect of the ensemble. One solution to mitigate this issue is to consider the top
unlabelled instances with the highest predicted utility values from to be added to each of the subsets. This solution is inadequate when a maximum budget of annotations per iteration is set to be less than the ensemble size (i.e., <
) or in the case of the small dataset and big ensemble size. In the following subsection, we explain how the utility measure and the stopping criterion are devised to account for both accuracy and diversity of the ensemble.
Since the main learner is an ensemble model, the utility measure, which evaluates the utility of
unlabelled instances, has to be defined with respect to ensemble properties. Ensemble diversity
is considered to be one of the most important aspects of ensemble learning [
subsets , . The gain of evaluated on is giving by:
, = {1, ⋯ , } where = ∑ =1 = [ ( − )2], = ∑ =1 = [ ( − )2] and , = {1, ⋯ , } are the ensemble weights. Equation 4 separates the generalization error into two terms. The ifrst one, , is an aggregation of the base models’ errors. The second term , called ambiguity, measures the variability/ disagreement among the base models’ outputs and reflects diversity between them. It is straightforward to see that increasing the ambiguity yields a reduction in the overall ensemble error. However, since the overall error is always positive, can be seen as a lower bound of . That is why a trade-of between decreasing and increasing should be established. Denote with a given unlabelled data point and with ∪ the ensemble composed including the addition of a given instance where states for the dataset on which the ensemble error is evaluated. ∗ that maximizes the gain is simply selected.
However, to increase the ambiguity, instead of adding ∗ to all models, i.e., to the training sets of all the models, we add it to the model ∗ yielding the highest deviation from on ∗. ,∈{1,⋯,}
,∈{1,⋯,} ∗ = argmax ( ∗) = argmax ( ( ∗) − ( ∗))2 (
, ) = − ∪ ∗ = argmax ( ∈
, ) to train the base models Require: Training dataset 1: Split into , , , 2: Use , to generate ,, 3: Build the ensemble 4: for Each ∈ , 5: Calculate ( , , 6: end for 7: = {( , , 8: Train MetaM on { , 9: Return MetaM do )
and , , = {1, ⋯ , } ), = 1, ⋯ , | , , }
|}
Algorithm 1: Learning MetaM
METAL is composed of three main stages. In the first stage, a meta-model MetaM is trained after preparing a meta-dataset to learn the gain induced by the addition of a given data instance to the ensemble training set and predict the gain of unlabelled data instances in . The second stage consists of selecting the data instance from with maximal predicted gain and determining to which ensemble member should be added (See Eq.7). In the third stage, both training and unlabelled datasets are updated by adding the selected instance to the training subsets of the selected ensemble member and removing it from . The three stages are iterated until a stopping criterion is met (i.e., a maximum number of iterations in this work).
MetaM is used to learn the gain measure defined in Eq. 5. Therefore, we split the original training set into into three disjoint subsets , , , and , . , is used to create the subsets ,, , = {1, ⋯ , } to train the ensemble members as explained in subsection 3.2. , is the training dataset for the meta model MetaM. For each data instance, ∈ , , the gain achieved by adding to the training subsets ,, , = {1, ⋯ , } is evaluated using evaluation dataset , . The set of gains, = {( , , ), = 1, ⋯ , | , |} calculated for the data in , is regarded as the response variables corresponding to the explanatory variables , in , . The pairs {( , ( , , )), = 1, ⋯ , | , |} are used for training MetaM. This metalearning stage consists of learning the active learning process and is summarized in Algorithm 1.
MetaM is used afterward to predict the gain of data instances in . The sample ∗ maximizing the gain is selected and attributed to the training set of the ensemble member verifying the condition in Eq.7.
Our framework is composed of two models, a meta-model for predicting the gain of unlabelled samples and a main model for predicting the response variable. Both models share the same set of features. The gain predicted by the meta-model is related to the loss of the main model (i.e., ensemble). Therefore, the selected sample by the meta-model is also associated with the importance of the features of the sample to the main model. In this context, we use machine learning algorithms equipped with a measure of feature importance for the meta-model MetaM and the ensemble base models. The overall importance of each feature for the ensemble is assessed by computing the average of the corresponding normalized importance measures over all the base models. We monitor the change in the features’ importance and in their empirical distributions over the active learning iterations in order to explain the rationale behind sample selection in the active learning process. Concrete examples are provided in Section 4.2.
In this section, we present the experiments carried out to validate METAL and to answer the following research questions: Q1: How is the performance of METAL compared to state-ofthe-art (SoA) methods for active learning for regression problems? Q 2: What is the impact of relying only on the averaged error or the ambiguity in evaluating the gain of an unlabelled data instance? Q3: Since active learning can be viewed as an informed sampling strategy for ensemble construction, how is the performance of METAL compared to the SoA ensemble methods for regression? Q4: How is the scalability of METAL in terms of computational resources compared to SoA methods for active learning? Q5: How the meta-model and the ensemble can be used together to provide a suitable interpretation for selecting a sample in the process of active learning to improve the ensemble performance?
The methods are evaluated using the root mean squared error (RMSE). In each experiment, the data is split into 30% for training, 20% for testing, and 50% is assumed to be an unlabelled dataset. For each dataset, a cross-validation (CV) with 10 folds is performed for the evaluation with 100 repetitions for the active learning methods (i.e., diferent initialization of the labeled set). We used 15 benchmarking datasets for our experiments. They are briefly summarized in Table 1. We note that all the experiments are fully reproducible, and the code is available under this link2. The datasets are publicly available.
ID Dataset Data Source Data characteristics
Att. Ins. RT
1 2Dplanes dcc.fc.up.pt 10 40768 [-999.709,999.961]
2 Friedman Domain dcc.fc.up.pt 10 40768 [1.50,27.975 ]
3 Abalone UCI 8 4177 [1.00,29.00 ]
4 Auto-Price UCI 14 159 [9,46.60]
5 Bank8FM DELVE 8 8192 [0,0.70]
6 House (8H) DELVE 8 22784 [0,427300 ]
7 Bos.Housing UCI 13 506 [
As 30% of the total data size is kept for training data (i.e., ), this proportion is split
into 50% for , , 30% for , and 20% , . The 10 DTs are initially trained
using diferent random bootstraps of , (i.e., ,, , = {1, ⋯ , 9} ). The maximum number
of iterations in the active learning process is set up to half of the unlabelled set size. The
meta-learner is chosen to be a Random Forest (RF) [
Active Learning Methods
QBC [
M-Am: is a variant of METAL where only the ambiguity is kept in the definition of the gain: ( , ) = ( ∪ ) − ( ) . This variant is diferent from QBC since it relies on the meta-learner’s prediction of ambiguity (i.e., variance) reduction instead of a direct computation.
M-Er: is a variant of METAL where only the error term is kept in the gain: ( , ) = ( ) − ( ∪ ).
Ensemble Methods
Active learning can also be viewed as aninformed sampling strategy for optimizing ensemble construction by selecting samples that lead to enhanced accuracy and diversity. Therefore, we suggest comparing the ensemble built by METAL with ensemble methods using the whole available training set (i.e., initial labeled set and the assumed unlabelled set) (i.e., passive/ blind sampling). The hyperparameters values of the involved models are tuned with a random search over a 3-fold CV. We compare our method against the following:
RF [
ENS [
MetaBags [
Table 2 shows the performance results of METAL against SoA methods for active and ensemble learning. Results are reported in terms of RMSE. The statistical significance of the results is assessed using the Bayesian correlated -test with the significance level = 0.05 , with the null hypothesis that a given learner wins against METAL after observing the results of all repetitions.
The results presented in Table 2 show that METAL outperforms existing SoA methods for both active and ensemble learning. In addition,METAL is almost never statistically significantly worse than any active learning method. It is also highly competitive with ensemble methods trained on a larger amount of data which shows the eficiency of active learning as an informed sampling strategy for building an ensemble, supporting thus the main active learning assumption stating that a better-performing machine learning model can be built using less amount of training data points carefully selected. This can be explained by orienting the selection towards establishing a trade-of between accuracy and diversity. METAL has the lowest average rank. These results illustrate the generalization power of METAL in both cases. Comparing METAL to its diferent variants (i.e., M-Er and M-Am), we see the clear advantage of integrating both error and ambiguity components in the definition of the gain. The results are worse with M-Er, meaning that ambiguity is needed to enforce a certain degree of diversity in the ensemble. However, it is not also suficient on its own. This answers the research question Q1-Q3.
We present in Figure 1 a comparison of the averaged runtime of METAL compared to the remaining active learning methods. In METAL and GPF, the size of sub-dataset for calculating the gain is fixed, and the cost for model construction is constant (i.e., bootstrapping only in the initialization). Let be the cost for model construction in the gain calculation, be the cost for predicting one sample in the unlabelled dataset, and its size. The total cost is of order ( + ) . For QBC, we construct the committee trained with resampled datasets and predict the output for each sample in the unlabelled dataset. Then, the sample with the highest variance is selected for annotation. The computational cost for selecting a new sample is of order (( + )) , where is the number of bootstrap sampling. In EMC, in addition to the QBC prediction model, bootstrap sample is created for training GBDT models, and therefore the computational cost for EMC is of order (( + )) .
METAL and GPF, which perform model construction only a fixed number of times in each iteration, provide a clear computational advantage compared to QBC and EMC. We notice that despite their same computational order, EMC is slower than QBC. This can be explained by the
Method
EMC GPF metaAL QBC
Ran Ran metaAL
GPF
QBC
EMC manipulation of DTs in GBDT models that are required to evaluate the model change. This answers the question Q4.
Regarding question Q5, we show an example of an interpretation of sample selection for the Bos.Housing dataset at a given iteration. In this experiment, the variable MEDV (i.e., the median value of owner-occupied homes from land information) is the response variable and is predicted by our ensemble using 13 other explanatory variables, as shown in Table 4. Since MetaM is an RF, it is possible to compute variables’ importance. The main model is an ensemble of regression models, and variable importance can be obtained following the procedure detailed in Section 3.5. Figure 2a shows the importance of the normalized variables by the ensemble before the 5-ℎ samFeature CRIM ZN INDUS CHAS NOX RM Value 0.013 90 1.22 0.0 0.403 7.249 Feature DIS RAD TAX PTRATIO B LSTAT Value 8.696 5 226 17.9 395.93 4.81
AGE 21.9 ple selection. It can be seen that LSTAT, DIS, and AGE have the highest importance at this stage.
LSTAT shows the percentage of low-income inhabitants, and it is natural to assume that income is strongly related to the prices of houses. DIS presents the distances to the city center, and it naturally afects house prices. The AGE of the building also is highly correlated with its price. MetaM predicts the reduction in the ensemble error, which is reflected by the reduction of the averaged error and the increase in ambiguity once a data sample is added to the training set. We can see that LSTAT and DIS have higher importance than the other variables in learning the gain and lower importance for the AGE. Since the gain reflects a decrease in the ensemble 1.00 AGE B CHAS CRIM DIS INDUS LSTAT NOX PTRATIO RAD RM TAX ZN
AGE B CHAS CRIM DIS INDUS LSTAT NOX PTRATIO RAD RM TAX ZN (a) (b) (RF) (b) before selecting the 5-th sample. loss after a data sample is added to the training set, the variable with high importance in the ensemble also has high importance in MetaM as naturally expected. In addition, the actual selected sample is shown in Table 4. It can be seen that the values of LSTAT are 4.81, DIS 8.696, and AGE 21.9. The summary of these variables in the training data (i.e., before the data sample selection) is shown in Table 3. From this table, it can be seen that the sample is selected from low-density regions, especially for AGE and DIS. Based on this result, we can conjecture that at this stage of the learning, the information on the AGE and distance DIS of buildings together with the percentage of the population engaged in low-salary occupations LSTAT is important for the ensemble accuracy, but a suficient amount of information is not yet collected. That is why active sampling is made in favor of low-density regions.
This paper introduces METAL a novel, practically useful meta-active learning method for learning regression ensembles. This work illustrates the combination of meta and active learning for optimizing ensemble building and enhancing its performance by contributing to the accuracy-diversity trade-of. The proposed method shows that it is possible to interpret the reason for sample selection without forcing restrictions on the ensemble construction. In future work, we aim to explore heterogeneous ensembles and how diferent families of machine learning models can be combined to improve prediction accuracy and support explainability.