Optimizing the hyperparameters is one of the most important and time-consuming activities to do when training machine learning models. But the lack of guidance available to optimization algorithms means that finding values for these hyperparameters is left to black-box methods. Black-box methods can be made more eficient by incorporating an understanding of where good hyperparameter values might be located for a specific model. In this paper, we visualize hyperparameter performance-landscapes in several datasets to discover how the XGBoost algorithm behaves for many combinations of hyperparameter values across these datasets. Using this knowledge, it might be possible to design more eficient search strategies for optimizing the hyperparameters of XGBoost.
Hyperparameter optimization is the task of optimizing machine learning algorithms’
performance by tuning the input parameters that influence their training procedure and model
architecture, referred to as hyperparameters. While essential to most machine learning problems,
hyperparameter optimization is a highly non-trivial task as a given algorithm can have many
hyperparameters of diferent datatypes, and efective values for these difer from one dataset to
another [
A useful method of obtaining insight into complex problems is to visualize them, as this allows them to be comprehensively presented. In this paper, we gain insight into the behavior of the hyperparameters of the XGBoost algorithm by visualizing and comparing landscapes of prediction performance generated based on hyperparameter combinations.
The remainder of the paper is structured as follows: In Section 2, we present related work relevant to the visualization of performance-landscapes. The theory behind the XGBoost algorithm is outlined in Section 3. In Section 4, we document the methodology used for generating and visualizing samples of hyperparameter-based performance-landscapes. In Section 5, we present the findings of comparing the generated landscape-samples, and we discuss these findings in Section 6. Finally, in Section 7, we conclude the paper and discuss future work.
Visualizing problem subjects can be an efective method of intuitively obtaining many types of
insight, as demonstrated with many articles within machine learning research. For instance,
Li et al. [
Despite their usefulness, research exploring visualizations directly related to
hyperparameters are quite limited. The perhaps most relevant papers are the ones that present tools using
visualizations as a method of aiding hyperparameter tuning/analysis processes [
There are, however, several interesting papers that have investigated performance-landscapes,
though not necessarily in the context of hyperparameters. Performance-landscapes are
relevant to our paper because they can be visually analyzed to obtain insights into
hyperparameters’ efects. Performance landscapes have previously been primarily investigated in the
context of neural network loss. The most prominent example of this is the earlier mentioned paper
by Li et al. [
XGBoost, developed by Chen & Guestrin [
( ) = ∑ ( , ̂
the diference between them. The that best minimizes the loss between
penalized to avoid overfitting, as denoted by the regularization term
iteration’s prediction ̂
( −1) is greedily added. Additionally, the complexity of the added is
and the previous
denote the prediction and the target, and is a loss function that measures
Much of XGBoost’s regularization and model architecture is defined through
hyperparameters. Some of the most impactful are the learning rate, the number of base learners in the
ensemble, and the base learners’ maximum depth. In this paper we refer to these as learning_rate,
n_estimators, and max_depth. The learning_rate originates from the principle of shrinkage [
The goal with this paper was to investigate how the hyperparameters of XGBoost afect its prediction performance on diferent datasets and to investigate potential similarities between these efects. Our motivation was to gain a general insight into XGBoost’s hyperparameters compared to simply analyzing datasets individually. To accomplish this, we compared 3D visualizations of performance-landscapes relative to each selected dataset, where each landscape was generated based on a combination of two hyperparameters. (1)
To visualize the hyperparameter performance-landscapes, we selected three of XGBoost’s hyperparameters and set their respective value-ranges, as tabulated in Table 1. Generated performancelandscapes were based on the combination of two of these hyperparameters at a time. This resulted in three diferent performance-landscapes for a given dataset, based on the combination of learning_rate and n_estimators, learning_rate and max_depth, and n_estimators and max_depth. For the classification datasets, the used performance metric was accuracy, while for the regression datasets, Mean Absolute Error (MAE) was used.
To reduce the amount of data needed to provide highly detailed visualizations, we developed a novel algorithm, named Adaptive Zoom, for adaptively generating more data points in regions with better predictive performance, thereby keeping computation time spent on regions of low performance minimal. The main idea behind the algorithm was to iteratively "zoom" in on the region of apparent best performance. "Zooming" here refers to adaptively determining the region of known best performance, based on pre-generated points, and generating more points within the determined region. Using this algorithm, we could eficiently obtain highly detailed landscapes by only generating high numbers of points in these specific regions while leaving the remaining regions at lower details. .
The algorithm first identifies the p% best performing hyperparameter configurations in the landscape. From this set of configurations, the region of best performance, defined by hyperparameter value-ranges, is then determined by the lowest and highest value per hyperparameter. Finally, a new landscape-sample can be generated based on the determined best performing region.
The Adaptive Zoom algorithm is visually demonstrated in Fig. 1, and its pseudocode is contained in Fig. 2.
(a) (b)
To accurately compare the performance landscapes’ characteristics, we needed the same hyperparameter combinations across all ranges. However, due to using Adaptive Zoom, a varying amount of landscapes-samples, with varying ranges, were generated for each hyperparameter ℎ ← ← ← ← end for return end procedure procedure AdaptiveZoom( , , ∈ ℕ
) sorted by performance ← points
← empty array for all ℎ ∈ ( ) do ( ) ← percentage best performing points ← values of hyperparameter ℎ ( ) ( ) (, ℎ ) (
, , ) ← generated landscape-sample based on scipy.interpolate.griddata2 with the "linear" method, which takes a list of points and a list of their corresponding values, and returns a grid with a specified resolution of interpolated values.
The landscapes, with performance values obtained through two-fold cross validation, were generated with standard values of non-investigated hyperparameters at an initial resolution of 20 x 20. Further landscape-samples based on adaptive zoom were generated at an individual resolution of 50 x 50. The number of samples for each landscape was dynamically determined by running Adaptive Zoom until the returned hyperparameter ranges were no diferent from the previous iteration. Finally, all generated landscape-samples of each hyperparameter combination were merged into singular landscapes through linear interpolation.
To explore efects on performance landscapes, we selected several datasets for both classification and regression problems, as tabulated in Table 2. The datasets were selected to be varied in dataset characteristics, such as size, and number of categorical and continuous attributes, to ensure that generated landscapes would be as varied as possible. This was important to ensure that obtained findings could be generalized and would represent the general relationship between the hyperparameters and performance as accurately as possible.
In terms of preprocessing, all datasets were randomly shufled to ensure that their contained data was dispersed, categorical features with text values were one-hot encoded, and id-columns were removed.
2https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.griddata.html
To obtain insight into the hyperparameters’ efects on performance, we analyzed the landscapes by looking at their general convexity, the compared dominance of the hyperparameters’ impact on performance, the locations of optima, the numbers of local optima, and general observations of similarities between the landscapes. The motivation behind analyzing the landscapes’ convexity and local optima was to investigate how applicable methods using gradient descent are to hyperparameter optimization. The dominance of the diferent hyperparameters was investigated to gain insight into which hyperparameters are more important to optimize to achieve the best results. Comparisons of each landscape’s optima were investigated to gain insight into the predictability and consistency of their locations.
Note that the landscapes for the regression datasets are flipped to ensure visual consistency. For this reason, the MAE-values are presented as negative.
We found that the landscapes of the learning_rate and n_estimators combination (Figure 3) were generally lacking in convexity and were instead quite flat and jagged in shape. There were, however, some exceptions to this. The landscape of the forestfires dataset had a clear convex shape in the learning_rate axis, where lower values of learning rates yielded better 3http://archive.ics.uci.edu/ml/datasets/QSAR+biodegradation 4https://archive.ics.uci.edu/ml/datasets/Contraceptive+Method+Choice 5https://archive.ics.uci.edu/ml/datasets/Soybean+(Large) 6https://datahub.io/machine-learning/vehicle 7https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic) 8https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/ 9https://archive.ics.uci.edu/ml/datasets/auto+mpg 10https://archive.ics.uci.edu/ml/datasets/Forest+Fires 11https://archive.ics.uci.edu/ml/machine-learning-databases/housing/ (d) biodegredation (e) contraceptive (d) biodegreda- (e) contraceptive tion performance than higher values. The landscapes of biodegredation, wdbc, housing, and autompg also seemed to have some convexity in this axis, though they were still relatively flat in overall shape.
For the landscapes of the learning_rate and max_depth combination (Figure 4) the convexity was quite varied from one dataset to another. The contraceptive dataset was relatively convex in the max_depth axis, which was also the case for forestfires and winequality-red. Auto-mpg and housing were somewhat convex in both the learning_rate and max_depth axis.
Most of the landscapes of the n_estimators and max_depth combination (Figure 5) were flat in overall shape. However, contraceptive, winequality-red, and perhaps forestfires, had some convexity to them.
Based on the landscapes, it seemed that n_estimators had, compared to learning_rate, a considerably larger impact on performance. However, this only seemed the case for n_estimatorsvalues lower than approximately 100, which materialized as a "wall" in the landscapes. This was apparent for all dataset-relative learning_rate and n_estimators combination landscapes (Figure 3), except for forestfires, which did not seem to contain this wall. For n_estimatorsvalues over 100, we found that the combination of learning_rate and n_estimators resulted in quite jagged landscapes, with learning_rate appearing to be the most dominant. We also observed that jaggedness in the n_estimators axis was not equal for all learning_rate values. Only the landscapes of the auto-mpg and housing datasets were lacking in visible jaggedness.
Comparing learning_rate to max_depth (Figure 2), it seems that these hyperparameters tend to have a nearly equal impact on performance except for with a few datasets, such as forestfires, contraceptive and winequality-red. For these, max_depth appeared to have a larger efect, creating a wall similar to those observed in the combinations of learning_rate and n_estimators (Figure 3). We also found that max_depth tended to stop impacting performance beyond certain values. These values seemed to never exceed max depth = 15, but were observed for several datasets as being less.
For the combination of n_estimators and max_depth (Figure 5), we found that the most dominant hyperparameter changed from one dataset to another, with there sometimes being performance walls in the n_estimators axis, sometimes in the max_depth axis, and sometimes in both. Beyond these performance walls, the two hyperparameters seemed about equally dominant.
For all combinations and datasets except vehicle, the optima were located within the learning_rate value-range of 0.1 to 1.0. For vehicle, the optima were located around the learning_rate value of 1.5. We also observed that the optima were always located between the n_estimators value-range of roughly 100 to 250.
The optima for each dataset was observed to generally be located around the same learning_rate value within all relevant hyperparameter combinations (Figure 3 and 4), with winequality being the only exception to this. For this dataset, the optimum was an entirely diferent learning_rate value for the combination of learning_rate and n_estimators (Figure 3) compared to learning_rate and max_depth (Figure 4). Relative to this, n_estimators optima seemed a bit more variable. Max_depth optima were located within values less than 15.
In terms of local optima, landscapes based on the combination of learning_rate and n_estimators (Figure 3) seemed to contain many local optima. These did, however, seem to be predominantly formed from the influence of learning_rate.
Local optima were also observed to be plentiful for the combination of learning_rate and max_depth (Figure 4). Here, it seemed that local optima were about equally based on learning_rate and max_depth, resulting in much more chaotic landscapes. This was, however, only the case for max_depth values less than 15 due to the observations outlined in Section 5.2.
For the combination of n_estimators and max_depth (Figure 5), we observed that local optima seemed less common. Most of the landscapes were relatively flat and only datasets like biodegredation, contraceptive, vehicle and winequality-red had anything that could be referred to as local optima. But even for these, local optima were small in size.
For the learning_rate and n_estimators combination (Figure 3), we found that most landscapes were quite similar in characteristics, such as general shape, convexity, and local optima. There were, however, some diferences between the landscapes of the classification and regression datasets. Most notably that regression datasets generally seemed to contain less local optima, and that forestfires’ landscape shape was completely unique compared to the others.
The diferent learning_rate and max_depth combination landscapes (Figure 4) also seemed to have reoccurring characteristics. There seemed to generally be a sharp dip in performance close to learning_rate = 2.0 and max_depth = 1. The only landscape that was lacking this dip was the one generated from the vehicle dataset. We also found that biodegreadation, soybean and wdbc had very similar landscapes for the combination of learning_rate and n_estimators, which can also be said for winequality-red and forestfires.
We also observed that several datasets resulted in similar landscapes for all hyperparameter combinations. The clearest example of this was housing and auto-mpg, which seemed nearly identical. This was also the case for soybean and wdbc. In terms of landscape characteristics, the two datasets that stuck out the most were contraceptive and forestfires. For contraceptive, max_depth seemed to have a much larger influence on performance than for other datasets, while for forestfires, n_estimators appeared to have no significant performance impact.
Our goal with this paper was to obtain insights into how and under what conditions the hyperparameters of XGBoost afect its performance by analyzing and comparing hyperparameterbased performance landscapes for various datasets. The motivation was to use such insights to aid the eficiency of hyperparameter optimization strategies for XGBoost.
The findings showed several indications that analyzing the performance-landscape visualizations helped gain insight into efective search ranges of the investigated hyperparameters; learning_rate, n_estimators, and max_depth. For learning_rate, the fact that optima were within the value-range of 0.1 to 1.0 for all datasets except one, indicates that this range might generally be optimal when searching for values of this hyperparameter. Similarly, with n_estimators, the search range of 100 to 250 may be more optimal than other alternatives under 500 estimators, based on the optima generally being located within this range. We also discovered that n_estimators values less than 100 formed a "wall" of performance-increase, most notably in landscapes of the learning_rate and n_estimators combinations (Figure 3). This could imply that n_estimators less than 100 can reasonably be excluded from hyperparameter searches. Regarding max_depth, we observed that values greater than 15 for this hyperparameter seemed not to afect XGBoost’s performance, implying that limiting the search range from 1 to 15 might be reasonable. These implications are useful for standardizing searches regarding these hyperparameters, which makes their optimization more eficient by reducing computation time.
By analyzing and comparing the general characteristics between the performance-landscapes, we also found indications that the general behavior of XGBoost’s hyperparameters is somewhat predictable. Specifically, we found that landscapes based on the same hyperparameter combination had general similarities across the datasets, and that certain datasets resulted in almost identical landscapes, optima included, across all combinations. This was observed to be the case for datasets both similar and dissimilar in general characteristics. Based on this, it is possible that performance-landscapes based on XGBoost’s hyperparameters can largely be predicted from aspects of the datasets, though any such specific aspects were not obvious from the findings. However, if such landscape predictions are possible and reliable, it could mean that efective search methods and hyperparameter values for XGBoost can potentially be made somewhat deterministic. Based on this assumption, such predictions would be a prime target to take advantage of when designing future hyperparameter optimization methods relevant to XGBoost. Worth noting, however, is that this is still unlikely to completely remove the need for black-box methods, due to the large amount of local optima and general flatness observed in the landscapes. Nevertheless, a combination of the two would likely be beneficial.
Compared to earlier studies, our visualization method seemed efective for obtaining
hyperparameter insight despite its simple premise and implementation. For instance, compared
to previously suggested tools using visualizations for hyperparameter tuning processes [
In this paper, we attempted to gain insight into how XGBoost’s hyperparameters afect performance by analyzing hyperparameter-based performance-landscape visualizations. The findings indicate that the method of visualizing performance-landscapes, based on combinations of two hyperparameters at a time, is a useful tool for gaining insight into, e.g., efective ranges of XGBoost’s hyperparameters. This was derived from, e.g., how optima were generally located between 0.1 and 1.0 learning_rate, and 100 and 250 n_estimators. Also, how max_depth was never observed to have an efect on performance for values greater than 15, and how n_estimators typically had a wall of performance within the value range of 1 to 100. We also found indications that the visualization method is efective for gaining insight into the general and specific behavior of XGBoost’s hyperparameters with diferent datasets. This was derived from how we observed that most datasets had common landscape-characteristics and how some dataset-landscapes were nearly identical regardless of similarity/dissimilarity of datasetcharacteristics. Thus, we conclude that 3D visualizing hyperparameter-based performancelandscapes is an efective tool for obtaining various types of insight into the behavior of XGBoost’s hyperparameters. And that these insights can possibly be used to design more eficient hyperparameter optimization strategies for this algorithm.
While the findings and their implications are promising, several points of investigation
should be explored in future work. For instance, larger datasets should be investigated to
ensure that the findings are generalizable. During the process of generating the visualizations,
we also noticed that the interpolation method would produce artifacts for certain landscapes.
Minor aspects of the visualizations might therefore be inaccurate and should be fixed. The
utilized visualization method is limited in that it can only visualize two hyperparameters at
a time. However, it might be possible to use hyperparameter-based vectors to generate the
visualizations, similar to how Li et al. [