Automated Machine Learning is a research area which has gained a lot of focus in the recent past. But the required components to build an autoML system is neither properly documented nor very clear due to the differences and the recentness of researches. If the required steps are analyzed and brought under a common survey, it will assist in continuing researches. This paper presents an analysis of the components and technologies in the domains of autoML, hyperparameter tuning and meta learning and, presents a checklist of steps to follow while building an AutoML system. This paper is a part of an ongoing research and the findings presented will assist in developing a novel architecture for an autoML system.
The umbrella term AutoML coined from ‘Automated Machine Learning’ [
Even though there were many promising systems emerged from these competitions
and recently we have been introduced to some commercial level AutoML systems by
Google [
In section 2, we present the methodology we used to conduct this survey and collect data. Section 3 covers the architecture designs proposed by researchers. Section 4 covers the preprocessing techniques that can be automated. Section 5 deals with algorithm selection and meta learning methods to find best candidate algorithms. Section 6 covers hyperparameter optimization techniques used in this domain and their reviews. Section 7 covers how to automate the evaluation of models to choose the best one. Section 8 deals with how to benchmark the developed autoML system and in the last chapter conclusion is provided. 2
We started gaining domain knowledge with a literature survey in the domain of autoML systems. We came across 48 such primary studies and identified the different approaches used for autoML in the available work. We short listed techniques used by the available work in order to achieve hyperparameter optimization, meta learning and algorithm selection. These techniques were selected to be discussed in this paper, according to the majority of use. 3
Throughout the researches on the autoML domain, the final goal has always been to
automate the entire pipeline of the machine learning. However, it has proved to be a
difficult task as a whole. Thus several work has been conducted in automating at least
some part of machine learning, with the intention of putting all these together at the
end. Liu [
According to him, even though most of the work available are narrow autoML, it is
eminent to achieve pivotal progresses in generalized autoML. The same concept has
been covered by Guyon et al [
An autoML system will need to automate all the parts of machine learning in its
architecture. Das and Cakmak [
Though this covers all major aspects of machine learning, few important steps like train/test splitting, models ensembling and reporting are missed in this list. Most of the architectures proposed in autoML domain (AutoWeka, Hyperopt-Sklearn, AutoSklearn, TPOT, H2O.ai) contains at least some subset of these components. Olson et al proposed an architecture [48] where these systems are arranged into different engines. Addition to visualization and graph engines that offers insights which are not listed in the above list, they also added a ‘Human Engine’, which takes human inputs for maintenance of the autoML system. With this understanding of architecture, let’s analyze how each component can be developed. 4
Raw data used in machine learning is often unclean, skewed and noisy. Cleaning data and feature transformations have proven to improve accuracy of machine learning systems substantially. Thus the first step of an AutoML system is data preprocessing. The following section covers some important preprocessing steps that can be automated at least to certain extent.
Data transformation. For numerical data, some of the common processes to automate are, scaling - standardization and normalization missing values imputation - using global constant, using mean / median, using indicator variable, predicting the most probable value or simply removing record outlier detection: univariate - interquartile range and filtering, ‘Winsorizing’ or trimming: multivariate - one class SVM, Local Outlier Factor (LOF) and isolation forest binning – equal width binning, equal frequency binning; log and power transformations identifier detection
For categorical variables, some of the common processes are, encoding - label encoding, one-hot encoding, frequency-based encoding, target mean encoding, binary encoding, hash encoding replacing missing values with the mode
Additionally, for other data types such as text or video, several other preprocessing techniques like tokenization, normalization or substitution are available. An interesting point to note is that, tree based supervised models such are random forests are able to handle feature or data abnormalities by themselves, whereas non-tree based supervised learning algorithms, are much sensitive to abnormalities.
Feature selection. Some of the feature selection that can be done before creating a model are as follows, Identifying highly correlated variables and treating them Excluding features with low variance or univariate feature selection Recursive feature elimination - Measuring information gain for the available set of features and choosing the top N features accordingly Dimensionality reduction with PCA - transforming the data in the high-dimensional space to a space of fewer dimensions.
And, if we are to do feature selection after creating baseline model, Using linear regression and selecting variables based on p values Using stepwise selection for linear regression and selecting the important variables Feature selection using random forest - Using random forest and selecting the top N important variables Feature generation - It is also possible to generate new features from the intrinsic data with techniques such as numerical feature generation, pairwise feature creation, categorical feature creation, temporal feature creation, etc.
These preprocessing steps can improve efficiency and accuracy of the subsequent machine learning workflows. Several of these preprocessing can be decided to be used based on simple statistical metrics. For example, multi-collinearity between features can be found with Pearson’s Correlation Coefficients. These variables can be treated with stepwise regression or principle component analyses. Such automatic decisionexecution pairs can help build a powerful autoML system.
Next step is to automatically find candidate algorithms suitable for the dataset. The categories of machine learning algorithms that needs to be considered are as follows,
There are other aspects data scientists worry about in algorithm selection, such as
computational complexity, differences in training and scoring time, linearity versus
nonlinearity, etc. and it’s useful if these are to be considered while automating. The main
quantitative techniques in this paradigm can be categorized as rules-based and
metalearning. The following section discusses these techniques.
In rules based systems, we try to mimic how a data scientist manually does algorithm
selection, which is a mixture of initial exploration of the dataset and his experience.
Certain characteristics of the dataset and the domain of the dataset can suggest the
possible candidates of algorithms to build machine learning experiments with. A rules
system can be implemented to reflect these characteristics with the help of many cheat
sheets of algorithms publicly available in the internet. For example, Scikit Learn python
package has a map [
Case Based Reasoning [
Algorithm selection is almost always backed by visualizing the dataset in graphical methods. These Exploratory Data Analysis (ETA) methods can be done in conjunction with other statistical methods. This helps one understand data beyond the statistical modelling or hypothesis testing procedures. The only issue with these methods are that these cannot be automated. This is done solely with the supervision of a human component. But viewing these in the AutoML system as part of the configuration step or report generation step will give additional insight to the user. The following table gives an overview of such exploratory methods.
Mainly used in data clustering, groups similar multivariate objects near each other and dissimilar objects farther. Most common ordination technique is Principle Component Analysis (PCA).
Uses the medians of the rows and columns to iteratively fit model for the data.
Very common widely used. and
pict distribution of the data. It represents the probability distribution of continuous variables, but is limited to a single variable per graph.
Used in time series data to display data in time sequence. Used as univariate graphical method.
Used to visualize high dimension geometry and multivariate data. It is closely related to time series graphs but it doesn’t have any time variables, thus do not have natural order.
Used in very complex data to find features or patterns of interest.
ness and outliers in the
data
Find density of
distributions and one of seven
tools of quality control
[
Find correlation in data and is one of seven tools of quality control.
Now that we have analyzed several algorithm selection methods, the following table gives a summary of these techniques grouped under its three types. After algorithm selection, the machine learning model and its features to learn will be customized. There are few quantitative methods data scientists use manually that can be automated as discussed below.
tive quality of models for a dataset compared to other models, thus can be used for
model selection [
Bayesian information criterion (BIC) [
Mallows's Cp calculates fit of regression models, where a model with best subset of predictors among predictor variables, available for some outcome is chosen. Small value of Cp is considered to be more precise. Cp only works well in large sample sizes and can’t handle complex collection of models.
Stepwise Regression (SR) chooses each feature in the dataset incrementally and finds
accuracy of the models [
In the entire machine learning landscape there are two types of parameters. Model parameters that are learned by the algorithm while learning, thus does not need to be automated Hyperparameters that needs to be set before beginning of the learning, thus needs to be automated
Optimizing the hyperparameter is a function with the objective of minimizing the loss / cost of the algorithm, which in turn helps keep balance between the model bias and variance. This is essential in getting a low cross-validation error at the end of the experiment. While automating machine learning process, it is also expected to automate tuning or optimizing hyperparameters that best fits the dataset. In the following section analysis of hyperparameter optimization techniques are provided. 6.1
The most trivial techniques of hyperparameter tuning is grid search and randomized search.
Grid Search. Grid search expects few set of values as parameter space and tries all combinations of these values to learn in brute force manner. Search will be guided by a metric, which is often cross validation error of the training data or evaluation on the test data. Grid search suffers from curse of dimensionality, because even when there are two hyperparameters and five distinct values of these parameters, it requires twentyfive times of modelling and evaluation. Besides, there is no feedback or adjustment mechanism, thus the algorithm is highly unintelligent.
Random Search. Random search [
fitness function is expensive and costly, these model-based systems evaluate fitness with a surrogate that is cheaper to calculate [34]. Among other options, Expected Improvement (EI) has turned out to be a good candidate for the surrogate fitness function. The concept is to use objective functions like Gaussian process to choose good hyperparameter values and then sequentially update values based on results. This makes use of the results of the previous iteration to find better hyperparameter values to try in the next iteration, thus, is considered smart.
Bayesian-based hyperparameter optimization is one of SMBO technique that is
widely used in the autoML systems. Bayesian optimization is proved to work much
better than other alternatives, as it is able to reason about the quality of runs before they
even start. It has been proven Gaussian process [
Building on top of Bayesian optimization, there has been other advancements as well. For example, the concept of meta-learning has been used to build the initial model for BO to optimize. By referring to the meta data of the hyperparameters and their performance on past similar datasets, new parameter spaces are developed that are more likely to fit well. Other than BO, there are Random Online Aggressive Racing (ROAR) as well. In evolutionary optimization [34], evolutionary algorithms follow a process inspired by biological concept of evolution. It first creates a random hyperparameter population as much as one hundred. It starts evaluating these and gets their fitness functions. Based on these relative fitness values, parameters are ranked. Worst performing tuples of parameters are replaced by new ones generated through crossover and mutation. This is repeated until the performance is not improved. Though mainly used in deep learning tasks, these have started to be used in typical machine learning as well.
In the recent times, there has been interests in developing techniques out of this standard scope, to achieve hyperparameter optimization. Genetic programming [36], transfer learning [37] and reinforcement learning [38] are some of the techniques used. Genetic programming is mainly used in neural networks and SVMs. Bandit-based approaches have been developed which uses small subsets of data to find settings space to try on complete data, making the process much more efficient. 7
Typically, a model evaluation can contain statistical methods as well as the business rules specific for the problem. While the statistical methods are general to all learning problems, business rules will be tailor made to the question at hand. Even though business rules can be skipped in the autoML system, it is essential to automate the statistical evaluation techniques. Along with it, there can also be other processes like refining models, re-training, and deployment to be automated.
The following table gives an overview of few evaluation metrics that is used while training models.
Hold-Out Validation: In a dataset with independently and identically distributed (IID) records, a small subset of random records is held out for validation. Model training is done with the large portion and the evaluation metrics are calculated with the smaller potion. A common practice is to subset 20% as validation set.
Very easy to subset, but since validation is done on If the dataset is big the smaller subset, generalization error can be less reli- enough to break into able and higher variance. subsets.
into k number of folds. Iteratively we consider each fold as the held-out validation set and training is done on the rest of the folds. The overall performance is the average of all k folds.
Much better metric than hold out validation. Can even If the dataset is very be used in hyperparameter tuning to calculate perfor- small or computer power mance of tuples. is limited.
The following table gives an overview of few evaluation metrics that is used while testing models in regression tasks. Root Mean Square Error (RMSE / MSE): The most commonly used metric. It is defined as square root of the average squared distance between the actual score and the predicted score. In other words, sample standard deviation between predicted and observed values. It gives us the sense of how far the predictions were from actual values. Lower the RMSE better the model.
Very common and widely used. Always Mean Absolute Error (MAE): Similar to RMSE, but the absolute values of distances are taken. Thus all the individual differences are weighted equally which makes it a linear score.
Easy to interpret and understand than RMSE. More If interpretation is imrobust to outliers. portant.
R Squared (R2): R2 tells how the selected independent variables explain variability in the dependent variables. Simply said, it tells how close the data are to the fitted regression line, which is also known as coefficient of determination. The higher R Squared means model fits well.
Doesn’t necessarily tell if the model is bad or good. In exploratory analysis.
Just gives the relationship.
The following table gives an overview of few evaluation metrics that is used while testing models in classification tasks. Accuracy: Tells how often classier makes correct prediction. Calculated as the ratio of number of correct predictions among total predictions.
Very simple and widely used. For simple metric.
Confusion Matrix: Shows a detailed breakdown of correct classifications in classes. Presented as table of ground truth labels and predictions.
More detailed than accuracy, thus can diag- For metric breakdown of indinose issues in dataset. vidual classes.
Logarithmic Loss: Used when the classifier gives numeric probability as output instead of class. It is considered as soft measurement as it contains details of how incorrect or how correct a prediction is and not just if it is correct.
More tolerant to confidence values. If output is numeric probability. Area Under the Curve: Shows the sensitivity of classifier by plotting the rate of true positives to the rate of false positives. Used mainly in binary classification. Greater the AUC means it’s a better model.
Hard to interpret For binary classifications
Once an autoML system is developed, it will have to be benchmarked against existing systems and manual procedures. Though it won’t be a component of the system itself and not required to be automated, it helps evaluate the built system. The following section discusses three researchers and their benchmarking methods.
Thronton et al [39] in their system Auto-WEKA, used 21 prominent bench mark datasets including 15 from the UCI repository with 70%-30% train-test splitting. Intel Xeon X5650 six-core processors, at 2.66GHz were used with RAM limit of 3GB for classification datasets to mimic typical data scientist settings. Bootstrap sampling and cross validation were used to choose best setting.
Feurer [
Allen et al [40] in their benchmarking project used mean squared error and weighted F1 score for regression and classification tasks. They choose 57 classification and 30 regression OpenML datasets. As the process was pretty extensive with as much as 10,440 compute hours, they opted for an amazon web service distributed setup. Average of several different pairwise comparisons with different seed values were considered as the final performance score. 9
This paper presented the steps to build an automated machine learning system starting from data preprocessing to model deployment. The different methods and technologies available to develop these systems and their review were also discussed. From the findings, it is clear the algorithm selection and feature preprocessing components require more refinement from research community while hyperparameter tuning and meta learning spaces are explored actively. More technologies and statistical concepts unexplored in the autoML systems will make up the majority of future efforts while the knowledge of previous efforts need to be accumulated as knowledge hubs or meta databases. So far most of the researches are in the Python and Java languages while statistical languages like R can open up more options to explore. Since execution of multiple model trainings can take up high computational power, autoML systems needs to explore distributed task offloading mechanisms actively.
With this knowledge of the autoML components, and how they can be improved, we hope to come up with an architectural style in near future, towards an efficient automated machine learning system. 34. Bergstra, J., Bardenet, R., Bengio, Y., Kegl, B.: Algorithms for Hyper-Parameter
Optimization. Curran Associates Inc. , USA ©2011. 2546–2554 (2011). 35. Rasmussen, C.E., Williams, C.K.I.: Gaussian processes for machine learning. MIT
Press, Cambridge, Mass (2006). 36. Paris, L.: Genetic Programming/Auto-ML for One-Shot Learning. 5 (2018). 37. Quanming, Y., Mengshuo, W., Hugo, J.E., Isabelle, G., Yi-Qi, H., Yu-Feng, L., Wei-Wei, T., Qiang, Y., Yang, Y.: Taking Human out of Learning Applications: A Survey on Automated Machine Learning. arXiv:1810.13306 [cs, stat]. (2018). 38. Li, Y.-F., Wang, H., Wei, T., Tu, W.-W.: Towards Automated Semi-Supervised Learning. Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence (AAAI-19). 33, 8 (2019). 39. Thornton, C., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms. arXiv:1208.3719 [cs]. (2012). 40. Balaji, A., Allen, A.: Benchmarking Automatic Machine Learning Frameworks.
arXiv:1808.06492 [cs, stat]. (2018). 41. Cashman, D., Humayoun, S.R., Heimerl, F., Park, K., Das, S., Thompson, J., Saket, B., Mosca, A., Stasko, J., Endert, A., Gleicher, M., Chang, R.: Visual Analytics for Automated Model Discovery. arXiv:1809.10782 [cs]. (2018). 42. Shen, W.: DARPA’s Data Driven Discovery of Models (D3M) and Software Defined Hardware (SDH) Programs. In: Proceedings of the 2018 on Great Lakes Symposium on VLSI - GLSVLSI ’18. pp. 1–1. ACM Press, Chicago, IL, USA (2018). 43. Bergstra, J., Yamins, D., Cox, D.D.: Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures. Proceedings of the 30 th International Conference on Ma- chine Learning, Atlanta, Georgia, USA, 2013. JMLR: W&CP. 28, 9 (2013). 44. Liu, Z., Bousquet, O., Elisseeff, A., Escalera, S., Guyon, I., Jacques, J., Pavao, A., Silver, D., Sun-Hosoya, L., Treguer, S., Tu, W.-W., Wang, J., Yao, Q.: AutoDL Challenge Design and Beta Tests-Towards automatic deep learning. CiML workshop @ NIPS2018, Dec 2018, Montreal, Canada. 7 (2018). 45. Mahpod, S., Keller, Y.: Auto-ML Deep Learning for Rashi Scripts OCR.
arXiv:1811.01290 [cs]. (2018). 46. Gijsbers, P.: Automatic construction of machine learning pipelines. 65 (2017). 47. He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., Han, S.: AMC: AutoML for Model
Compression and Acceleration on Mobile Devices. 17 (2018). 48. Olson, R.S., Sipper, M., La Cava, W., Tartarone, S., Vitale, S., Fu, W., Orzechowski, P., Urbanowicz, R.J., Holmes, J.H., Moore, J.H.: A System for Accessible Artificial Intelligence. arXiv:1705.00594 [cs]. (2017).