Machine Learning (ML) has been successfully applied to a wide range of domains and applications. One of the techniques behind most of these successful applications is Ensemble Learning (EL), the eld of ML that gave birth to methods such as Random Forests or Boosting. The complexity of applying these techniques together with the market scarcity on ML experts, has created the need for systems that enable a fast and easy drop-in replacement for ML libraries. Automated machine learning (autoML) is the eld of ML that attempts to answers these needs. We propose autoBagging, an autoML system that automatically ranks 63 bagging work ows by exploiting past performance and metalearning. Results on 140 classi cation datasets from the OpenML platform show that autoBagging can yield better performance than the Average Rank method and achieve results that are not statistically different from an ideal model that systematically selects the best work ow for each dataset. For the purpose of reproducibility and generalizability, autoBagging is publicly available as an R package on CRAN.
Ensemble learning (EL) has proven itself as one of the most powerful techniques in Machine Learning (ML), leading to state-of-the-art results across several domains. Methods such as bagging, boosting or Random Forests are considered some of the favourite algorithms among data science practitioners. However, getting the most out of these techniques still requires signi cant expertise and it is often a complex and time consuming task. Furthermore, since the number of ML applications is growing exponentially, there is a need for tools that boost the data scientist's productivity.
The resulting research eld that aims to answers these needs is Automated
Machine Learning (autoML). In this paper, we address the problem of how to
automatically tune an EL algorithm, covering all components within it: generation
(how to generate the models and how many), pruning (which technique should
be used to prune the ensemble and how many models should be discarded) and
integration (which model(s) should be selected and combined for each
prediction). We focus speci cally in the bagging algorithm [
Our proposal is autoBagging, a system that combines a learning to rank approach together with metalearning to tackle the problem of automatically generate bagging work ows. Ranking is a common task in information retrieval. For instance, to answer the query of a user, a search engine ranks a plethora of documents according to their relevance. In this case, the query is replaced by new dataset and autoBagging acts as ranking engine. Figure 1 shows an overall schema of the proposed system. We leverage the historical predictive performance of each work ow in several datasets, where each dataset is characterised by a set of metafeatures. This metadata is then used to generate a metamodel, using a learning to rank approach. Given a new dataset, we are able to collect metafeatures from it and feed them to the metamodel. Finally, the metamodel outputs an ordered list of the work ows, taking into account the characteristics of the new dataset.
Algorithm AlAgorithm AlAgorithm
a dd dd d
Estimates of Performance Metafeatures Extraction New Dataset
Metadata Learning to Rank
Metamodel Ranked Algorithms
We tested the approach in 140 classi cation datasets from the OpenML
platform for collaborative ML [
autoBagging
We approach the problem of algorithm selection as a learning to rank problem [
Furthermore, Dm = fd1; d2; :::; dmg is the set of datasets for training and di is the i-th dataset, Ai = fai;1; ai;2; :::; ai;ni g is the set of algorithms associated with dataset di and yi = fyi;1; yi;2; :::; yi;ni g is the set of labels associated with dataset di, where ni represents the sizes of Ai and yi; ai;j represents the j-th algorithm in Ai; and yi;j 2 Y represents the j-th label in yi, representing the relevance score of ai;j with respect to di. Finally, the meta-dataset is denoted as m S = f(di; Ai); yigi=1.
We use metalearning to generate the metafeature vectors xi;j = (di; ai;j ) for each dataset-algorithm pair, where i = 1; 2; :::; m; j = 1; 2; :::; ni and represents the metafeatures extraction functions. These metafeatures can describe di, ai;j or even the relationship between both. Therefore, taking xi = fxi;1; xi;2; :::; xi;ni g we can represent the meta-dataset as S0 = f(xi; yi)gim=1.
Our goal is to train a meta ranking model f (d; a) = f (x) that is able to assign a relevance score to a given new dataset-algorithm pair d and a, given x. 2.1
We approach the problem of generating metafeatures to characterize d and a
with the aid of a framework for systematic metafeatures generation [
For this task in particular, we selected a set of meta-functions that are able
to characterize the datasets as completely as possible (measuring information
regarding the target variable, the categorical and numerical features, etc) the
algorithms and the relationship between the datasets and the algorithms (who
can be seen as landmarkers [
Each meta-function is used to systematically measure information from all possible combination of input objects available for this task. We de ned the input objects available as: discrete descriptive data of the datasets, continuous descriptive data of the datasets, discrete output data of the datasets, ve sets of predictions (discrete predicted data) for each dataset (naive bayes, decision tree with depth 1, 2 and 3, and majority class).
For instance, if we take the example of using Entropy as meta-function, it is possible to measure information in discrete descriptive data, discrete output data and discrete predicted data (if the base-level problem is a classi cation task). After computing the entropy of all these objects, it might be necessary to aggregate the information in order to keep the tabular form of the data. Take for the example the aggregation required for the entropy values computed for each discrete attribute. Therefore, we choose a palette of aggregation functions to capture several dimensions of these values and minimize the loss of information by aggregation. In that sense, the post-processing functions chosen were: average, maximum, minimum, standard deviation, variance and histogram binning.
Given these meta-functions, the available input objects and post-processing functions, we are able to generate a set of 131 metafeatures. To this set we add eight metafeatures: the number of examples of the dataset, the number of attributes and the number of classes of the target variable; and ve landmarkers (the ones already described above) estimated using accuracy as error measure. Furthermore, we add four metafeatures to describe the components of each workow: the number of trees, the pruning method, the pruning cut point and the dynamic selection method. In total, autoBagging uses a set of 143 metafeatures. 2.2
In order to be able to learn a ranking meta-model f (d; a), we need to compute a metatarget that represents a score z to each dataset-algorithm pair (d; a), so that: F : (D; A) ! Z, where F is the ranking meta-models set and Z is the metatarget set.
To compute z, we use a cross validation error estimation methodology (4-fold
cross validation in the experiments reported in this paper, Section 3), in which
we estimate the performance of each bagging work ow for each dataset using
Cohen's kappa score [
Our experimental setup comprises 140 classi cation datasets extracted from
the OpenML platform for collaborative machine learning [
Regarding bagging work ows, we limited the hyperparameters of the bagging
work ows to four: number of models generated, pruning method, pruning cut
point and dynamic selection method. Speci cally, each hyperparameter could
take the following values:
{ Number of models: 50, 100 or 200. Decision trees was chosen as learning
algorithm.
{ Pruning method: Margin Distance Minimization(MDSQ) [
Pruning (BB) [
The combination of all these hyperparameters described above generated 63
valid work ows. We tested these bagging work ows in the datasets extracted
from OpenML with 4-fold cross validation, using Cohen's kappa as evaluation
metric. We used the XGBoost learning to rank implementation for gradient
boosting of decision trees [
As baselines, at the base-level, we use 1) bagging with 100 decision trees 2) the average rank method, which basically is a model that always predicts the bagging work ow with the best average rank in the meta training set and the 3) oracle, an ideal model that always selects the best bagging work ow for each dataset. As for the meta-level, we use as baseline the average rank method.
As evaluation methodology, we use an approach similar to the leave-one-out
methodology. However, each test fold consists of all the algorithm-dataset pairs
associated with the test dataset. The remaining examples are used for training
purposes. The evaluation metric at the meta-level is the Mean Average Precision
at 10 (MAP@10) and at the base-level, as mentioned before, we use Cohen's
kappa. The methodology recommended by Demsar [
Figure 2 shows a loss curve, relating the average loss in terms of performance with the number of work ows tested following the ranking suggested by each method. The loss is calculated as the di erence between the performance of the best algorithm ranked by the method in comparison with the ground truth ranking. The loss for all datasets is then averaged for aggregation purposes. We can see, as expected, that the average loss decreases for both methods as the number of work ows tested increases.
In terms of comparison between autoBagging and the Average Rank method, it is possible to visualize that autoBagging shows a superior performance for all the values of the x axis. Interestingly, this result is particularly noticeable in the rst tests. For instance, if we test only the top 1 work ow recommended by autoBagging, on average, the kappa loss is half of the one we should expect from the suggestion made by the average rank method.
0.08 0.06 % s s o L0.04 a p p a K 0.02 0.00 Method
Autobagging Average Rank 0 20
Number of Tests 40 60
We evaluated these results to assess their statistical signi cance using Demsar's methodology. Figures 3 and 4 show the Critical Di erence (CD) diagrams for both the meta and the base-level.
AutoBagging
CD 1 2 Average Rank
Oracle AutoBagging@5 AutoBagging@3 1 CD 2 3 4 5 6 Bagging AverageRank AutoBagging@1
At the meta-level, using MAP@10 as evaluation metric, autoBagging presents a clearly superior performance in comparison with the Average Rank. The difference is statistically signi cant, as one can see in the CD diagram. This result is in accordance with performance that visualized in Figure 2 for both methods.
At the base-level, we compared autoBagging with three baselines, as mentioned before: bagging with 100 decision trees, the Average Rank method and the oracle. We test three versions of autoBagging, taking the top 1, 3 and 5 bagging work ows ranked by the meta-model. For instance, in autoBagging@3, we test the top 3 bagging work ows ranked by the meta-model and we choose the best.
Starting by the tail of the CD diagram, both the Average Rank method and autoBagging@1 show a superior performance than Bagging with 100 decision trees. Furthermore, autoBagging@1 also shows a superior performance than the Average Rank method. This result con rms the indications that we visualized in Figure 2.
The CD diagram shows also autoBagging@3 and autoBagging@5 have a similar performance. However, and we must highlight these results, autoBagging@5 shows a performance that is not statistically di erent from the oracle. This is extremely promising since it shows that the performance of autoBagging excels if the user is able to test the top 5 bagging work ows ranked by the system.
This paper presents autoBagging, an autoML system that makes use of a learning to rank approach and metalearning to automatically suggest a bagging ensemble speci cally designed for a given dataset. We tested the approach on 140 classication datasets and the results show that autoBagging is clearly better than the baselines to which was compared. In fact, if the top ve work ows suggested by autoBagging are tested, results show that the system achieves a performance that is not statistically di erent from the oracle, a method that systematically selects the best work ow for each dataset. For the purpose of reproducibility and generalizability, autoBagging is publicly available as an R package.
This work was partly funded by the ECSEL Joint Undertaking, the framework programme for research and innovation horizon 2020 (20142020) under grant agreement number 662189-MANTIS-2014-1.