In this article we investigate the combination of metalearning and optimization algorithms for parameter selection. We discuss our general proposal as well as present the recent developments and experiments performed using Support Vector Machines (SVMs). Meta-learning was combined to single and multi-objective optimization techniques to select SVM parameters. The hybrid methods derived from the proposal presented better results on predictive accuracy than the use of traditional optimization techniques.
The induction of a machine learning model with a good predictive
accuracy to solve a learning problem is influenced by a variety of
aspects, such as data pre-preprocessing, algorithm selection,
parameter optimization and training procedure. The study presented in this
paper focuses on a specific and relevant step of modeling: parameter
selection. Once a learning algorithm is chosen, the user has to
define its parameter values. Learning performance is usually affected
by a poor selection of these values. For instance, the performance of
SVMs depends on the adequate choice of its kernel function, kernel
parameters, regularization constant, among other aspects [
Parameter selection is treated by many authors as an optimization
problem in which a search technique is employed to find the
configuration of parameters which maximizes the learning performance
estimated on the problem at hand. There is an extensive literature
applying optimization algorithms for parameter selection, especially for
Artificial Neural Networks. Although it represents a systematic
approach to parameter selection, this approach can be very expensive,
since a large number of candidate parameter configurations must be
evaluated to ensure that an optimal, or at least reasonably good, set
of parameters is found [
Meta-learning, originally proposed for algorithm selection, has
also been adapted to parameter selection (e.g., for SVM [
Compared to the optimization approach, meta-learning tends to be computationally more efficient, at least at the moment when the recommendation of parameters is made. It must be observed that meta-learning however is very dependent on the quality of its metaexamples. It is usually difficult obtaining good results since metafeatures are in general very noisy and the number of problems available for meta-example generation is commonly limited.
As discussed in [
Problem MDB
Candidates
? SVM - Best
Parameters 6
Figure 1 shows the general architecture of the proposed solution. Initially, the Meta-Learner (ML) module retrieves a predefined number of past meta-examples stored in a Meta-Database (MDB), selected on the basis of their similarity to the input problem. Next, the Search module adopts as initial search points the configurations of successful parameter values on the retrieved meta-examples. In the Search module, a search process iteratively generates new candidate values for the SVM parameters. The final solution which is recommended by the system is the best one generated by the Search module up to its convergence or other stopping criteria.
In [
Hybrid−PSO Meta−Learning
PSO Default LibSVM 0.6 0.55 0.5 E SM0.45 N m u iinm 0.4 M 0.35 0.3 0.250 Figure 2 compares the minimum NMSE (averaged over the 40 problems) obtained by SVM using the parameters suggested by combining meta-learning and PSO, referred to as Hybrid-PSO (using 5 initial solutions recommended by meta-learning), and the two methods individually, PSO (with random initialization, population size = 5) and meta-learning (which recommends the best configuration of each retrieved meta-example). We also present in Figure 2 the average NMSE achieved by the default heuristic adopted by the LibSVM tool ( = inverse of the number of attributes and C=1). Finally, Figure 2 shows the average NMSE that would be achieved if the best parameter configuration had been chosen on each problem.
By comparing PSO and meta-learning, we identified a trade-off in their relative performances. Meta-learning is better than PSO for a small number of recommended parameter configurations. It is also better than the default LibSVM parameters. Hence, meta-learning alone would be indicated in situations in which the SVM user had strong resources constraints. In these situations, meta-learning could recommend a lower number of configurations with intermediate performance levels. PSO in turn is able to find better configurations along its search and then it is more adequate if a higher number of configurations can be tested.
The Hybrid-PSO was able to combine the advantages of its components. The performance of the Hybrid-PSO in the initial five recommended configurations is of course the same as the performance of meta-learning (since the initial configurations are recommended by meta-learning). From that point of the curve, the Hybrid-PSO consistently achieves better results compared to both the PSO and the
6 8 10 12 14 Number of recommended configurations 16 18 20 3
Conclusion meta-learning. It converges earlier to solutions with similar NMSE values compared to the best configurations observed in the 40 problems. There is an additional cost in recommending the configurations by the hybrid approach which is the cost of the meta-learning initialization (specially the cost of computing the meta-features). However, we deployed meta-features with a low computational cost.
In [
In our experiments, the final Pareto Fronts optimized by the MOPSO and the Hybrid MOPSO were evaluated using three metrics for MOO problems: Spacing, Hypervolume and Spread. The proposed hybrid approach was able to generate better comparative results, considering the Spacing and Hypervolume metrics. Regarding the maximum Spread, our approach lost in first generations, but was similar to MOPSO in the last generations.
The combination of meta-learning and optimization techniques showed promising results for SVM parameter values selection. The proposed approach can be easily adapted to other learning algorithms (e.g., Artificial Neural Networks). A number of aspects need to be investigated in our proposed solution such as alternative strategies to integrate meta-learning in the optimization process. For instance, not only the best solutions to similar problems can be considered, but also diverse solutions in the search space. Additionally, the limitations of the individual components (as usual in hybrid systems) need to be dealt with. For instance, new strategies to augment the number of datasets for meta-learning can improve the learning performance in our context.
Acknowledgments: The authors would like to thank CNPq, CAPES, FAPESP and FACEPE (Brazilian Agencies) and FCT (Portuguese Agency) for their financial support.