Since hyperparameter optimization is crucial for achieving peak performance with many machine learning algorithms, an active research community has formed around this problem in the last few years. The evaluation of new hyperparameter optimization techniques against the state of the art requires a set of benchmarks. Because such evaluations can be very expensive, early experiments are often performed using synthetic test functions rather than using real-world hyperparameter optimization problems. However, there can be a wide gap between the two kinds of problems. In this work, we introduce another option: cheap-to-evaluate surrogates of real hyperparameter optimization benchmarks that share the same hyperparameter spaces and feature similar response surfaces. Specifically, we train regression models on data describing a machine learning algorithm's performance under a wide range of hyperparameter configurations, and then cheaply evaluate hyperparameter optimization methods using the model's performance predictions in lieu of the real algorithm. We evaluate the effectiveness for using a wide range of regression techniques to build these surrogate benchmarks, both in terms of how well they predict the performance of new configurations and of how much they affect the overall performance of hyperparameter optimizers. Overall, we found that surrogate benchmarks based on random forests performed best: for benchmarks with few hyperparameters they yielded almost perfect surrogates, and for benchmarks with more complex hyperparameter spaces they still yielded surrogates that were qualitatively similar to the real benchmarks they model.
The performance of many machine learning methods depends
critically on hyperparameter settings and thus on the method used to set
such hyperparameters. Recently, sequential model-based Bayesian
optimization methods, such as SMAC[
Given this high overhead for studying complex hyperparameter
optimization benchmarks, most researchers have drawn on simple,
synthetic test functions from the global continuous optimization
community [
In the special case of small, finite hyperparameter spaces, a much
better alternative is simply to record the performance of every
hyperparameter configuration, thereby speeding future evaluations via a
table lookup. The result is a perfect surrogate of an algorithm’s true
performance that takes time O(1) to compute (using a hash) and that
can be used in place of actually running the algorithm and evaluating
its performance. This table-based surrogate can trivially be transported
to any new system, without the complicating factors involved in
running the original algorithm (setup, special hardware requirements,
licensing, computational cost, etc.). In fact, several researchers have
already applied this approach to simplifying their experiments: for
example, Bardenet et al. [
Of course, the drawback of this table lookup idea is that it is limited to small, finite hyperparameter spaces. Here, we generalize the idea of machine learning algorithm surrogates to arbitrary, potentially high-dimensional hyperparameter spaces (including, e.g., real-valued, categorical, and conditional hyperparameters). As in the table-lookup strategy, we first evaluate many hyperparameter configurations during an expensive offline phase. We then use the resulting performance data to train a regression model to approximate future evaluations via model predictions. As before, we obtain a surrogate of algorithm performance that is cheap to evaluate and trivially portable. However, model-based surrogates offer only approximate representations of performance. Thus, a key component of our work presented in the following is an investigation of the quality of these approximations.
We are not the first to propose the use of learned surrogate models
that stand in for computationally complex functions. In the field of
metalearning [
The surrogate benchmarks resulting from our work can be used in
several different ways. Firstly, like synthetic test functions and table
lookups, they can be used for extensive debugging and unit testing.
Since the large computational expense of running hyperparameter
optimizers is typically dominated by the cost of evaluating algorithm
performance under different selected hyperparameters, our
benchmarks can also substantially reduce the time required for running
a hyperparameter optimizer, facilitating whitebox tests of an
optimizer using exactly the hyperparameter space of the machine learning
algorithm whose performance is modelled by the surrogate. This
functionality is gained even if the surrogate model only fits algorithm
performance quite poorly (e.g., due to a lack of sufficient training
data). Finally, a surrogate benchmark whose model fits algorithm
performance very well can also facilitate the evaluation of new features
inside the hyperparameter optimizer, or even the (meta-)optimization
of a hyperparameter optimizer’s own hyperparameters (which can be
useful even without the use of surrogates, but is typically extremely
expensive [
The rest of this paper is laid out as follows. We first provide some background on hyperparameter optimization (Section 2). Then, we discuss our methodology for building surrogate benchmarks (Section 3) using several types of machine learning models. Next, we evaluate the performance of these surrogates in practice (Section 4). We demonstrate that random forest models tend to fit the data better than a broad range of competing models, both in terms of raw predictive model performance and in terms of the usefulness of the resulting surrogate benchmark for comparing hyperparameter optimization procedures. 2
The construction of machine learning models typically gives rise to two optimization problems. The first is internal optimization, such as selecting a neural network’s likelihood-maximizing weights; the second is tuning the method’s hyperparameters, such as setting a neural network’s regularization parameters or number of neurons. The former problem is closely coupled with the machine learning algorithm at hand and is very well studied; here, we consider the latter. Let 1; : : : ; n denote the hyperparameters of a given machine learning algorithm, and let 1; : : : ; n denote their respective domains. The algorithm’s hyperparameter space is then defined as = 1 n. When trained with hyperparameters 2 on data Dtrain, the algorithm’s loss (e.g., misclassification rate) on data Dvalid is L( ; Dtrain; Dvalid). Using k-fold cross-validation, the optimization problem is then to minimize:
k f ( ) = 1 X k i=1
L( ; Dtrain; Dv(ail)id): (i) (1)
A hyperparameter n can have one of several types, such as continuous, integer-valued or categorical. For example, the learning rate for a neural network is continuous; the random seed given to initialize an algorithm is integer-valued; and the choice between various preprocessing methods is categorical. Furthermore, there can be conditional hyperparameters, which are only active if another hyperparameter takes a certain value; for example, the hyperparameter “number of principal components” only needs to be instantiated when the hyperparameter “preprocessing method” is PCA.
Evaluating f ( ) for a given 2 is computationally costly, and
so many techniques have been developed to find good configurations
with few function evaluations. The methods most commonly used
in practice are manual search and grid search, but recently, it has
been shown that even simple random search can yield much better
results [
Bayesian optimization methods use a probabilistic model M to
model the relationship between a hyperparameter configuration and
its performance f ( ). They fit this model using previously gathered
data and then use it to select a next point new to evaluate, trading off
exploitation and exploration in order to find the minimum of f . They
then evaluate f ( new), update M with the new data ( new; f ( new))
and iterate. Throughout this paper, we will use the following three
instantiations of Bayesian optimization:
SPEARMINT [
Sequential Model-based Algorithm Configuration (SMAC) [
models p(f < f ), p( j f < f ), and p( j f f ), where f is defined as a fixed quantile of the function values observed so far, and the latter two probabilities are defined by tree-structured Parzen density estimators. TPE can handle continuous, categorical, and conditional parameters.
An empirical evaluation on the three methods on the HPOlib
hyperparameter optimization benchmarks showed that SPEARMINT
performed best on benchmarks with few continuous parameters and
SMAC performed best on benchmarks with many, categorical, and/or
conditional parameters, closely followed by TPE. SMAC also
performed best on benchmarks that relied on cross-validation [
We now discuss our approach, including the algorithm performance data we used, how we preprocessed the data, the types of regression models we evaluated, and how we used them to construct surrogate benchmarks. 3.1
In principle, we could construct surrogate benchmarks using algorithm performance data gathered by any means. For example, we could use existing data from a manual exploration of the hyperparameter space, or from an automated approach, such as grid search, random search or one of the more sophisticated hyperparameter optimization methods discussed in Section 2.
It is more important for surrogate benchmarks to exhibit strong predictive quality in some parts of the hyperparameter space than in others. Specifically, our ultimate aim is to ensure that hyperparameter optimizers perform similarly on the surrogate benchmark as on the real benchmark. Since most optimizers spend most of their time in high-performance regions of the hyperparameter space, and since relative differences between the performance of hyperparameter configurations in such high-performance regions tend to impact which hyperparameter configuration will ultimately be returned, accuracy in this part of the space is more important than in regions of poor performance. The training data should therefore densely sample highperformance regions. We thus advocate collecting performance data primarily via runs of existing hyperparameter optimization procedures. As an additional advantage of this strategy, we can obtain this costly performance data as a by-product of executing hyperparameter optimization procedures on the original benchmark.
Of course, it is also important to accurately identify poorly performing parts of the space: if we only trained on performance data for the very best hyperparameter settings, no machine learning model could be expected to infer that performance in the remaining parts of the space is poor. This would typically lead to underpredictions of performance in poor parts of the space. We thus also included performance data gathered by a random search. (An alternative is grid search, which can also cover the entire space. We did not adopt this approach because it cannot deal effectively with large hyperparameter spaces.) To gather the data for each surrogate benchmark in this paper, we therefore executed r = 10 runs of each of the three Bayesian optimization methods described in Section 2 (each time with a different seed), as well as random search, with each run gathering the performance of a fixed number of configurations. 3.2
For each benchmark we studied for this paper, after running the hyperparameter optimizers and random search, we preprocessed the data as follows: 1. We extracted all available configuration/performance pairs from the runs. For benchmarks that used cross-validation, we encoded the cross-validation fold of each run as an additional categorical parameter (for benchmarks without cross validation, that parameter was set to a constant). 2. We removed entries with invalid results caused by algorithm crashes. Since some regression models used in preliminary experiments could not handle duplicated configurations, we also deleted these, keeping the first occurrence. 3. For data from benchmarks featuring conditional parameters, we replaced the values of inactive conditional parameters with a default value. 4. To code categorical parameters, we used a one-hot (aka 1-in-k) encoding, which replaces any single categorical parameter with domain = fk1; : : : kng by n binary parameters, only the i-th of which is true for data points where is set to ki. 3.3
We considered a broad range of commonly used regression algorithms as candidates for our surrogate benchmarks. To keep the results comparable, all models were trained on data encoded as detailed in the previous section. If necessary for the algorithm, we also normalized the data to have zero mean and unit variance (by subtracting the mean and dividing by the standard deviation). If not stated otherwise for a model, we used the default configuration of its implementation.
Table 1 details the regression models and implementations we
used. We evaluated three different tree-based models, because SMAC
uses a random forest (RF), and because RFs have been shown to
yield high-quality predictions of algorithm performance data [
Importantly, the wall clock time to run an algorithm on XM0 is much lower than that required on X, since expensive evaluations of the machine learning algorithm underlying X are replaced by cheap model predictions. The model M is simply saved to disk and is queried when needed. We could implement each evaluation in XM0 as loading M from disk and then using it for prediction, but to avoid the repeated cost of loading M , we also allow for storing M in an independent process and communicate with it via a local socket.
To evaluate the performance of a surrogate benchmark scenario XM0 we ran the same optimization experiments as on X, using the same settings and seeds. In addition to evaluating the raw predictive performance of model M , we assessed the quality of surrogate benchmark XM0 by measuring the similarity of hyperparameter optimization performance on X and XM0 . 4
In this section, we experimentally evaluate the performance of our surrogates. We describe the data upon which our surrogates are based, evaluate the raw performance of our regression models on this data, and then evaluate the quality of the resulting surrogate benchmarks. 4.1
We collected data for four benchmarks from the hyperparameter
optimization benchmark library, HPOLIB [
The two low-dimensional benchmarks were the synthetic Branin
test function and a logistic regression [
The high-dimensional benchmarks comprised a simple and a deep
neural network, HP-NNET and HP-DBNET (both taken from [
For some model types, training with all the data from Table 2 was computationally infeasible, and we had to subsample 2 000 data points (uniformly at random3) for training. This was the case for nuSVR, SVR, and the Bayesian neural network. For the GP model, we had to limit the dataset even further to 1 500 data points. On this reduced training set, the GP model required 255 minutes to train on the most expensive data set (HP-DBNET MRBI), and the Bayesian neural networks required 36 minutes; all other models required less than one minute for training.
We used HPOLIB to run the experiments for all optimizers with a single format, both for the original hyperparameter optimization benchmarks and for our surrogates. To make our results reproducible, we fixed the pseudo-random number seed in each function evaluation to 1. The version of the SPEARMINT package we used crashed for about 1% of all runs due to a numerical problem. In evaluations where we require entire trajectories, for these crashed SPEARMINT runs, we imputed the best function value found before the crash for all evaluations after the crash. 4.2
We first studied the raw predictive performance of the models we considered on our preprocessed data. 4.2.1
To evaluate the raw predictive performance of the models listed in Table 1, we used 5-fold cross-validation performance and computed the cross-validated root mean squared error (RMSE) and Spearman’s rank correlation coefficient (CC) between model predictions and the true responses in the test fold. Here, the responses correspond to validation error rate in all benchmarks except for the Branin one (where they correspond to the value of the Branin function).
Table 3 presents these results, showing that the GP and the SVR approaches performed best on the smooth low-dimensional synthetic Branin test function, but that RF-based models are better for predicting the performance of actual machine learning algorithms. This 3 For a given dataset and fold, all models based on the same number of data points used the same subsampled data set. We note that model performance sometimes was quite noisy with respect to the pseudorandom number seed for this subsampling step and we thus used a fixed seed. strong performance was to be expected for the higher-dimensional hyperparameter space of the neural networks, since RFs perform automatic feature selection.4 The logistic regression example is rather low-dimensional, but the categorical cross-validation fold is likely harder to model for GPs than for RFs.5 Extra Trees predicted the performance of the actual machine learning algorithms nearly as good as the RF, Gradient boost was slightly worse. Bayesian neural networks, k-nearest-neighbours and our linear regression models could not achieve comparable performance. Based on these results, we decided to focus the remainder of our study on a diverse subset of models: two tree-based approaches (RFs and gradient boost), Gaussian processes, nuSVR, and, as an example of a popular, yet poorly-performing model, k-nearest-neighbours. We paid special attention to RFs and Gaussian processes, since these have been used most prominently in Bayesian hyperparameter optimization methods. 4.2.2
In practice, we will want to use our surrogate models to predict the performance of a machine learning algorithm with hyperparameter configurations selected by some new optimization method. The configurations it evaluates might be quite different from those considered by the optimizers whose data we trained on. Next to the standard crossvalidation setting from above, we therefore evaluated our models in the leave-one-optimizer-out setting, which means that the regression model learns from data drawn from all but one optimizer, and its performance is measured on the held out data.
Table 4 reports RMSE and CC analogous to those of Table 3, but for the leave-one-optimizer-out setting. This setting is more difficult and, 4 We note that RFs could also handle the categorical hyperparameters in these benchmarks natively. We used the one-hot encoding for comparability with other methods. It is remarkable that even with this encoding, they outperformed all other methods. 5 As we will see later (Figure 2), the RF-based optimizer SMAC also performed better on this benchmark than the GP-based optimizer SPEARMINT.
SPEARMINT
consequently, the results were slightly worse, but the best-performing models stayed the same: nuSVR and GP for low dimensional and RFs for higher dimensional datasets.
Figure 1 studies the predictive performance in more detail for the HP-DBNET mrbi benchmark, demonstrating that tree-based models also performed best in a qualitative sense. The figure also shows that the models tended to make the largest mistakes for the worst configurations; especially the non-tree-based models predicted some of these to be better than some of the best configurations. The same patterns also held for the logistic regression 5CV and the HP-NNET convex data (not shown). The models also completely failed to identify neural network configurations which did not converge within the time limit and therefore received an error rate of 1.0. Interestingly, the GP failed almost entirely on the high-dimensional HP-DBNET MRBI benchmark in two cases, predicting all data points around the data mean. 4.3
We now study the performance of the surrogate benchmarks XM0 obtained for random forest (RF) and Gaussian process (GP) models M . We assess the quality of XM0 by comparing the performance of various hyperparameter optimizers on XM0 and the real benchmark X. 4.3.1
We first analyzed the performance of surrogate benchmarks based on models trained on the entire data we have available. We note that in this first experiment, a surrogate that perfectly remembers the training data would achieve perfect performance, because we used the same hyperparameter optimizers for evaluation as we did to gather the training data. However, after the first imperfect prediction, the trajectories of the optimizers will diverge. Thus, since none of our models is perfect on training data, this initial experiment serves as an evaluation of surrogate benchmarks based on training data gathered through the same mechanism as at test time.
We performed experiments for our three actual hyperparameter optimization benchmarks, logistic regression, a simple and a deep neural network. For each of them, we repeated the 10 runs for TPE, SMAC and SPEARMINT we previously conducted on the real benchmarks, but now used surrogate benchmarks based on RFs and GPs, respectively.
Figure 2 shows that the surrogate benchmarks based on Gaussian process models differed substantially from the true benchmarks. The figures show the best function values found by the various optimizers over time. Visually comparing the first column (real benchmark) to the third (surrogate benchmark based on GP model), the most obvious difference is that the surrogate benchmark fails completely on HP-DBNET MRBI: since the GP model is unable to properly fit the high-dimensional data (predicting all configurations to perform roughly equally, around the data mean) all optimizers basically stay at the same performance level (the data mean). Note in the plot for the true benchmark that the GP-based optimizer SPEARMINT also performed very poorly on this benchmark.
In the other two cases (logistic regression 5CV and HP-NNET convex), the performance of the optimizers appears visually similar to the true benchmark at first glance. However, for the logistic regression 5CV the GP model predicts some parts of the hyperparameter space to be better than the actual best part of the space, leading to the final optimization results on the surrogate benchmark to appear better than optimization results on the true benchmark. Another difference is a zig-zag pattern in the trajectories for logistic regression surrogates: these are also (mildly) present in the real benchmark (mild enough to only be detectable when zooming into the figure) and are due to the slightly different performance in the 5 folds of cross validation; the impact of the folds is very small, but the GP model predicts it to be large, causing the zig-zag. Interestingly, for the GP model trained on the HP-NNET convex dataset, regions with “better” performance appear hard to find: only SMAC and TPE identified them, causing a larger gap between SPEARMINT and SMAC/TPE than on the real benchmark.
Conversely, the RF surrogates yielded results much closer to those obtained on the real benchmark. Visually, the first column (true benchmark) and second column appear very similar, indicating that the RF captured the overall pattern well. There are some differences in the details. For example, on HP-NNET convex, the surrogate does not capture that TPE finds very good configurations before SMAC and yields the overall best performance. Nevertheless, overall, our results for the RF surrogates qualitatively resemble those for the true benchmarks, and for the logistic regression example, the correspondence is almost perfect. 4.3.2
Next, we studied the use of a surrogate benchmark to evaluate a new optimizer. For each optimizer o and each of the three hyperparameter optimization benchmarks X, we trained RF and GP models M on the respective leave-one-optimizer-out training data discussed in Section 4.2.2 and compared the performance of optimizer o on X and XM0 . Figure 3 reports the results of this experiment, showing that surrogate benchmarks based on RF models qualitatively resembled the real benchmarks.
The results for the logistic regression 5CV benchmark (top row of Figure 3) show that surrogate benchmarks based on RF models mirrored the performance of each optimizer o on the real benchmark well, even when the training data did not include data gathered with optimizer o. In contrast, surrogates based on Gaussian process models performed poorly: the Gaussian process again underestimated the error, predicting better performance in some regions than possible on the real benchmark.6 Again, these regions with “better” performance appear hard to find: only SMAC and SPEARMINT found them, causing their performances on the GP-based surrogate benchmark to differ substantially from their performance on the true benchmark.
Results for HP-NNET convex were also better for the surrogate benchmark based on RFs (especially for SPEARMINT), but not as much better as for the logistic regression 5CV case. As was already the case when the surrogate was based on all training data, the RFbased surrogate benchmarks only approximately captured the strong performance TPE showed on the real benchmark.
Results on HP-DBNET MRBI show a fairly close correspondence between the real benchmark and the RF-based surrogate benchmarks. In contrast, the GP-based surrogate was dismal, once again due to the GP’s near-constant predictions (close to the data mean).
After this qualitative evaluation of the surrogate benchmarks, Table 5 offers a quantitative evaluation. We judge the quality of a surrogate benchmark XM0 by how closely it resembles the real benchmark X it was derived from, in terms of the absolute error between the best found values for our four optimizers (SMAC, TPE, SPEARMINT, and random search) after evaluating i configurations. For logistic regression 5CV, in line with our qualitative results we obtained a very small error for the RF-based surrogate. The GP-based surrogate underestimated the achievable error rates, resulting in larger differences between performances on the true and the surrogate runs. After 50 evaluations the GP-based surrogate trained on all data yielded a quite high error because it underestimated the performance for configurations selected by the optimizer SMAC. Training the GP-surrogate on the leave-one-optimizer-out dataset causes worse performance for the optimizer SPEARMINT and too much variation for SMAC resulting in a higher error as well. This misprediction decreases with more evaluated configurations.
The results for the HP-NNET convex look quite similar, with a somewhat smaller difference between RF-based and GP-based surrogates. Indeed, SMAC and TPE behaved similarly on both RF-based and GP-based surrogates as on the real benchmark; only SPEARMINT behaved very differently on the GP-based surrogate, causing an overall higher error than for the RF-based surrogates.
On the high dimensional HP-NNET mrbi the surrogates performed differently. Whereas the RF-based surrogate could still reproduce similar optimizer behavior as on the real benchmark, the GP completely failed to do so. Remarkably, overall quantitative performance was similar for surrogate benchmarks trained on all data and those trained on leave-one-optimizer-out datasets.
Overall, these results confirmed our expectation from previous findings in Section 3.3 and the raw regression model performance results in Table 3: good regression models facilitate good surrogate benchmarks. In our case, RFs performed best for both tasks. We note that using the surrogate benchmarks reduced the time requirements substantially; for example, evaluating a surrogate 100 times instead of the HP-NNET convex or HP-DBNET MRBI took less than 1 minute on a single CPU, compared to roughly 10 hours on two CPUs (HPNNET convex) and over a day on a modern GPU (HP-DBNET MRBI).7 6 We noticed similar behavior for the nuSVR, which even returned negative values for configurations and caused the optimizer to search completely different areas of the configuration space (data not shown here). 7 Of course, the overhead due to the used hyperparameter optimizer comes on top of this; e.g., SPEARMINT’s overhead for a run with 200 evaluations was roughly one hour, whereas SMAC’s overhead was less than one minute. 101 #Function evaluations 101 #Function evaluations 101 #Function evaluations
102 SMAC_REAL SPEARMINT_REAL TPE_REAL 101 #Function evaluations 102
101 #Function evaluations 102
101 #Function evaluations 102
To tackle the high computational cost and overhead of performing hyperparameter optimization benchmarking, we proposed surrogate benchmarks that behave similarly to the actual benchmarks they are derived from, but are far cheaper and simpler to use. The key idea is to collect (configuration, performance) pairs from the actual benchmark and to learn a regression model that can predict the performance of a new configuration and therefore stand in for the expensive-toevaluate algorithm. These surrogates reduce the algorithm overhead to a minimum, which allows extensive runs and analyses of new hyperparameter optimization techniques. We empirically demonstrated that we can obtain surrogate benchmarks that closely resemble the real benchmarks they were derived from.
In future work, we intend to study the use of surrogates for general
algorithm configuration. In particular, we plan to support optimization
across a set of problem instances, each of which can be described
by a fixed-length vector of characteristics, and to assess the
resulting surrogates for several problems that algorithm configuration has
tackled successfully, such as propositional satisfiability [
SPEARMINT_REAL SPEARMINT_gp SPEARMINT_rf SPEARMINT_REAL SPEARMINT_gp SPEARMINT_rf 101 #Function evaluations 0.50 ed0.45 v ihe0.40 c a ro0.35 rren0.30 itao0.25 d liva0.20 tse0.15 B 0.11000 0.90 de0.85 v ieh0.80 ca0.75 r o rr0.70 e ilittvsondaae0000....65655500 B 0.41500 101 #Function evaluations
101 #Function evaluations
101 #Function evaluations