With the advent of automated machine learning, automated hyperparameter optimization methods are by now routinely used. However, this progress is not yet matched by equal progress on automatic analyses that yield information beyond performance-optimizing hyperparameter settings. Various post-hoc analysis techniques exist to analyze hyperparameter importance, but to the best of our knowledge, so far these have only been applied at a very small scale. To ll this gap, we conduct a large scale experiment to discover general trends across 100 datasets. The results in case studies with random forests and Adaboost show that the same hyperparameters typically remain most important across datasets. Overall, these results, obtained fully automatically, provide a quantitative basis to focus e orts in both manual algorithm design and in automated hyperparameter optimization.
The performance of most machine learning algorithms highly depends on their
hyperparameter settings. Various methods exist to automatically optimize
hyperparameters, including random search [
Various post-hoc analysis techniques exist that, for a given dataset and
algorithm, determine what were the most important hyperparameters; examples
include Forward Selection [
The functional ANOVA framework for assessing hyperparameter importance of algorithms is based on e cient computations of marginal performance. Various hyperparameter con gurations of an algorithm result in various performances. Functional ANOVA determines per hyperparameter how it contributes to the variance in performance.
After introducing notation, we rst brie y describe these and then summarize
how they can be plugged into the standard functional ANOVA framework to
yield the importance of all hyperparameters. For details, we refer the reader to
the original paper [
Notation. Following the notation of [
E cient marginal predictions. The marginal performance a^U ( U ) is de ned
as the average performance of all complete instantiations that agree with U
in the instantiations of hyperparameters U . We note that this marginal involves
a very large number of terms (even in the case of nite hyperparameter ranges,
it is exponential in the remaining number of hyperparameters N n U ). However,
for a tree-based model for approximating the performance of con gurations, the
average over these terms can be computed exactly by a procedure that is linear
in the number of leaves in the model [
Functional ANOVA. Functional ANOVA [
U N The components f^U ( U ) are de ned as follows: f^U ( U ) = ( ^ f; a^U ( U )
PW (U f^W ( W ) otherwise; if U = ;: where the constant f^; is the mean value of the function over its domain. Our main interest is the result of the unary function f^fjg( fjg), which captures the e ect of varying hyperparameter j, averaging across all possible values of all other hyperparameters. Additionally, the function f^U ( U ) for jU j > 1 captures the interaction e ects between all variables in U (excluding e ects of subsets (1) (2) W ( U ), but studying these higher-order e ects is beyond the scope of this preliminary work.
Given the individual components, functional ANOVA decomposes the variance V of y^ into the contributions by all subsets of hyperparameters VU : where
1 jj U jj
V = X VU ; where VU =
U N
1 jj U jj
Z
f^U ( U )2d U ; is the probability density of the uniform distribution across (3) U .
Putting it all together. To apply functional ANOVA, we rst collect perfor
K mance data h i; yiik=1 that captures the performance yi (e.g., misclassi cation rate or RMSE) of an algorithm A with hyperparameter settings i. We then t a random forest model to this data and use functional ANOVA to decompose the variance of each of the forest's trees y^ into contributions due to each subset of hyperparameters. Importantly, based on the fast prediction of marginal performance available for tree-based models, this is an e cient operation requiring only seconds in the experiments for this paper. Overall, based on the
K performance data h i; yiik=1, functional ANOVA thus provides us with the relative variance contributions of each individual hyperparameter (with the relative variance contributions of all subsets of hyperparameters summing to one). 3
We report the results of running functional ANOVA on 100 datasets for two
classi ers implemented in scikit-learn [
Both algorithms apply the following data preprocessing steps. Missing values are imputed (categorical features with the mode; for numerical features, the imputation strategy was one of the hyperparameters), categorical hyperparameters are one-hot-encoded, and constant features are removed.
For each of the 100 datasets in the OpenML-100 benchmark suite [
represents Vj =V for hyperparameter j. A high value implies that this hyperparameter accounted for a large fraction of variance, and therefore would account for high accuracy-loss if not set properly. For two datasets (`tamilnaduelectricity' and `Mice Protein') we could not perform a functional ANOVA analysis, because the random forest classi er performed very similarly across the various hyperparameter values (almost always a perfect accuracy) and functional ANOVA's internal model predicted zero variance.
The results reveal that most of the variance could be attributed to a small
set of hyperparameters: the minimal samples per leaf and maximal number of
features for determining the split were the most important hyperparameters. It
is reasonable to assume that these hyperparameters have some regions that are
clearly sub-optimal. For example, at every split point, the random forest should
always have at least some features to choose from; if this number is too low
this clearly a ects performance. This was also conjectured in earlier work, see
Figure 4:8 of [
Figure 2 shows the most important hyperparameters per dataset, plotted against the dataset dimensions (number of data points on the x-axis; number of features on the y-axis). Each data point represents a dataset, the color indicates which of the hyperparameters was most important, and the size indicates the marginal contribution (formally, Vj =V).
The results of Figure 2 add to the results presented in Figure 1: minimal samples per leaf and maximal features dominate the other hyperparameters. Only in a few cases bootstrap (`balance-scale', `credit-a', `kc1', `Australian', `profb' and `climate-model-simulation-crashes') or the split criterion (`scene') was most important. The imputation strategy was never the most important hyperparameter. This is slightly surprising, since the benchmark suite also contains datasets that have many missing values.
Figure 3 shows the result of a Nemenyi test over the average ranks of the
hyperparameters (for details, see [
Hyperparameter Importance of Adaboost Table 2 and Figures 4{6 present the results for the Adaboost case study, structured exactly like the study for random forests.
Figure 4 shows that similarly to the random forest, most of the variance could be explained by a small set of hyperparameters, in this case the maximal depth of the decision tree and, to a lesser degree, the learning rate. Figure 5 shows a similar result. The maximal depth and learning rate were the most important hyperparameters for almost all datasets; there are only a few exceptions where the boosting algorithm (`madelon' and `LED-display-domain-7digit') or the number of iterations (`steel-plates-fault') were the most important hyperparameters.
The results of the Nemenyi test (Figure 6) support the notion that the aforementioned hyperparameters indeed contributed most to the total variance. The maximal depth hyperparameter contributed signi cantly more than the other hyperparameters, followed by learning rate. Again, the imputation strategy did not seem to matter.
● imputation iterations
● learningrate
max.depth ●● algorithm 1
CD
hyperparameter values description algorithm fSAMME, SAMME.Rg Boosting algorithm to use. learning rate [0:01; 2:0] Learning rate shrinks the contribution of each classi er. max. depth [1; 10] The maximal depth of the decision trees iterations [50; 500] Number of estimators to build. imputation fmean, median, modeg Strategy for imputing numeric variables. 103 s e r u t a e F102 f o r e b m u N101 max. depth learning rate algorithm iterations 103
We note that the results presented in this paper do by no means imply that
it su ces to tune just the aforementioned hyperparameters. Contrarily, when
enough budget is available it is still advisable to tune all hyperparameters.
However, the results in [
We conducted a large scale study towards hyperparameter importance using functional ANOVA on OpenML datasets. Indeed, functional ANOVA and OpenML complement each other quite well: the experimental results available on OpenML can serve as an input for functional ANOVA (or any other model-based hyperparameter importance analysis technique) and can be used to assess which hyperparameters are most important.
To the best of our knowledge, this is the rst large scale study over many datasets assessing which hyperparameters are commonly important. We did this for two popular tree-based methods, random forests and Adaboost, resulting in quanti able measures of their hyperparameters' importance across datasets.
In future work, we aim to assess hyperparameter importance in good parts of
the space (this is already supported by functional ANOVA by assessing
improvements over a baseline [
Finally, we aim to use the knowledge obtained from hyperparameter
importance analysis to prune the hyperparameter search space (see also [