Feature Selection is important to improve learning performance, reduce computational complexity and decrease required storage. There are multiple methods for feature selection, with varying impact and computational cost. Therefore, choosing the right method for a given data set is important. In this paper, we analyze the advantages of metalearning for feature selection employment. This issue is relevant because a wrong decision may imply additional processing, when FS is unnecessarily applied, or in a loss of performance, when not used in a problem for which it is appropriate. Our results showed that, although there is an advantage in using metalearning, these gains are not yet su ciently relevant, which opens the way for new research to be carried out in the area.
Feature selection (FS) is a widely employed technique for reducing dimensionality
[
Recent works have approached the problem of FS method selection under the
perspective of metalearning [
However, existing approaches on metalearning for FS method selection focus their evaluation at the meta-level, e.g. whether it is possible to guess which is the best FS method for a given data set. They ignore the magnitude of the gain/loss that the method yields in terms of base-level performance (e.g., what is the di erence in classi cation performance of applying a learning algorithm with and without the selected FS method).
Thus, the purpose of this study is to extend the evaluation of metalearning approaches to FS method selection by taking into account the-base level performance.
We address two research questions. First, we investigate whether metalearning is useful for indicating when to use FS or not. The results show the advantages of using metalearning when compared to baselines as random choice and choice based on the majority class. In the second question, we analyze the gains in base-level performance associated with the use of metalearning. In this experiment, the results obtained by the metalearning strategy are very close to those obtained by an ideal meta-model (oracle), although they are also close to the recommendation based on the majority class.
This paper is organized as follows: Section 2 discusses related work. Section 3 shows our experimental analysis and results obtained. Section 4 presents our conclusions and future work. 2
In the context of FS, the use of metalearning is relatively new and there are only a
few studies that o er signi cant advances towards solving it [
Post et al. [
In this paper, we intend to increase the quality of the FS employment in classi cation
problems, extending the experiments carried out in [
The experimental setup is detailed below. 3.1
As discussed earlier, we extend the experiments carried out in [
Brie y, 394 data sets of binary classi cation problems were used, containing between 10 to 200,000 examples. One FS method (CfS-SubsetEval, a correlation-based method) and 12 machine learning algorithms (AdaBoost, Hoe dingTree, K-Nearest Neighbor, Logistic, Multilayer Perceptron, J48, JRip, Naive Bayes, Random Forest, REPTree, Stochastic Gradient Descent, Support Vector Machine) were applied to these data sets, using 10-fold-cross-validation. For each pair data set/learning algorithm the performance was evaluated with and without FS, based on the AUC measure, given the imbalanced distribution of classes in the data sets. At the meta-level, 40 meta-features were extracted from each data set (simple, statistical, informationtheoretic, landmarkers and FS landmarkers). Every instance in the meta-database are two-fold cross validation runs on a algorithm, one run with FS and one run without FS, and the meta-target is a label indicating whether the run with FS had a better performance. As meta-learner was used the Random Forest algorithm (with 100 trees).
In this work, we employ a typical leave-one-out cross-validation procedure (LOOCV) to evaluate the quality of the recommendation made by the metalearning strategy, both at the meta-level and at the base-level. 3.2
For the rst question, we carried out a set of experiments with the goal of providing a proof of concept in metalearning applied to FS recommendation. By testing whether the use of metalearning increases the quality of the decision between to use FS or not, we aim to show that using this strategy can avoid an ad hoc choice. We compare it with four distinct baselines: i) alwaysFS - a strategy that recommends the application of FS for all problems; ii) neverFS - a strategy that never recommends the use of FS; iii) random - a strategy that randomly selects whether to use FS or not; and iv) default - a strategy that selects the most frequent choice between using FS or not (majority class) in the data sets considered, for each algorithm. Table 1 shows the results of the experiment in terms of performance measured by AUC.
With these results, it is possible to see that metalearning shows a superior performance for all the learning algorithms. In ROC curves an ideal result is one that maximize the true positive rate while minimizing the false positive rate. As expected, the strategies alwaysFS, neverFS and default obtained an average performance, since they can not minimize the false positive rate. The random strategy has a balance between the true positive rate and false positive rate, which also leads to an average performance. Thus, metalearning stands out as being the best approach for decision making about the use of FS.
However, in a statistical validation we can see that, although there is a di erence
between the use of metalearning and the random strategy, this di erence is not
1 https://www.openml.org/s/15
statistically signi cant. In contrast, the di erence between using metalearning and
alwaysFS, neverFS and default is statistically relevant. These tests were carried out
using the methodology proposed by Demsar [
1
CD 2 3 4
5 metalearning random neverFS alwaysFS default
= 0:05) of the experiments at the meta-level. 3.3
In this experiment, our goal is to evaluate the performance gain obtained through the use of metalearning for FS employment at base-level, i.e, directly on the performance of the learning algorithms. For this, metalearning was applied over the 12 meta-databases and, for each data set, the recommendation to use FS or not in a given data set was scored according to the performance obtained by the corresponding class at the base-level, in terms of AUC. As baselines, we use two approaches: i) default - a strategy that select the most frequent choice between using FS or not (majority class); and ii) oracle - an ideal meta-model that always selects the best option for each data set. Table 2 presents these results.
Results show a small advantage in the average performance of metalearning relative to the default strategy. Despite being small, the advantage is consistent for all algorithms. In addition, the results also show a small di erence between the performance of metalearning and the oracle, which shows a bene t in the application of the latter strategy of recommendation. The Critical Di erence (CD) diagrams generated in the base-level experiments is presented in Figure 2.
oracle metalearning
CD 1 2 3 default
This diagram re ects the results presented in Table 2, indicating that there is no statistical di erence between metalearning and the default recommendation (majority class) in terms of performance gain. In addition, we can also note that, although the average performance of metalearning is close the oracle, statistical analysis shows that this di erence is signi cant.
However, when we take into account that the application of FS does not
significantly improve the performance of the learning algorithms for most data sets [
Figure 3 presents details of these results for the K-NN and SGD algorithms. Since the instance-based algorithms are sensitive to the curse of dimensionality, it was expected that FS would bring performance gains to the K-NN, which was con rmed in the results. In general, the application of FS provided a performance increase greater than 5% in 154 data sets (about 39%), and among these, metalearning achieved a hit rate of 83.12%. In the case of the SGD algorithm, it was expected that its performance would not be a ected by the number of features, so it did not have great advantage in the FS application. However, the results show that about 18% of the data sets bene ted from the FS application, and that metalearning obtained a 55.5% hit rate on these data.
140 120 100 ts80 s60 e a tD40 a 20 0
Frequency Hits 250 200 s150 tse taa100 D 50 0
Frequency Hits In this work we evaluate the use of metalearning to indicate when to employ feature selection in conjunction with di erent classi cation algorithms. The results showed that both at the meta-level and at the base-level, metalearning outperforms the baselines analyzed, although in the statistical test the di erence is not always signi cant. However, when we analyze the advantages of metalearning in the problems where FS causes the greatest impact in terms of performance, we can observe good results. Thus, suggestions for future works in this area indicate the possibility of extending the analysis to include new FS methods, in order to better generalize the results in relation to diversi ed inductive biases. In addition, we can develop new meta-features more appropriate for the problem.
This work is nanced by the ERDF European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme within project POCI-01-0145-FEDER-006961, and by National Funds through the FCT Fundac~ao para a Ci^encia e a Tecnologia (Portuguese Foundation for Science and Technology) as part of project UID/EEA/50014/2013. The authors also would like to thank CAPES for the nancial support through the project PDSE/88881.135787/2016-01.