Despite a large amount of research devoted to improving meta-learning techniques, providing and using background knowledge for this task remains a challenge. In this paper we propose a mechanism for automatic recommendation of suitable machine learning algorithms and their parameters. We used OpenML database and use rule-based configurator to improve Auto-WEKA tool. This paper discusses the concept of our approach and the prototype tool based on the HEARTDROID rule engine being developed.
or mean attribute entropy), and model-based information (e.g. J48 or kNN AUC). Each
of such characteristics has a different non-missing value coverage that varies from 6.5%
to 100%. We choose threshold for required values coverage to 20% to leave meaningful
meta-attributes. Missing values are filled with Amelia-II algorithm [
Meta-knowledge combines meta-attributes from dataset characteristics with corresponding algorithm label or ranking. We choose only fixed number of algorithms that are taken into consideration (usually N top used in OpenML). After that we filter the results with respect to performance and leave only the set of best algorithms. Afterwards we consider meta-knowledge as labeled dataset. Using the WEKA J48 algorithm we create decision tree which is converts to the XTT2 rule representation (ang. eXtended Tabular Trees).
Making recommendation We start with computing meta-attributes of new dataset by uploading it to OpenML. Then we choose only characteristics used in created metaknowledge. In the next step we match meta-attributes of new dataset with meta-knowledge. This is done with use of meta-rules and rule-based configurator. The result consist of algorithm name or ranking and set of parameters that according to the configurator fits best the given dataset.
In the third stage we reduce Auto-WEKA’s search space only to the recommended algorithms. This is done by preparing experiment with so called XML-based BATCH file. In that file we fill path to our new dataset in ARFF format and set up list of allowed classifiers. Then we create an experiment and run optimization process. Result is in form of classifier name with single set of parameters.
Conclusion The main contribution of our work is a mechanism that allows to speedup the meta-learning task by reducing search space for Auto-WEKA software with an usage of knowledge from OpenML database. We tested our approach and the tool on 570 datasets. We built meta-knowledge using 15 most used algorithms from OpenML focused on optimizing area under ROC. We benchmarked our best recommendations against Random Forest method as standard criteria. In general for most datasets area under ROC of our recommendations were higher (for 401 datasets with avg. 0:044). For 169 datasets AUC of our suggestions were lower (avg. 0:057). It is worth to notice that after a single setup, our system makes an instant recommendation.
Our future work includes learning and gaining additional meta-knowledge during recommendation mode, adding parameter suggestion in form of value ranges, adding guidance for data preprocessing methods and including more data sources.