Due to the complexity of recommendation algorithms, experimentation on recommender systems has become a challenging task. Current recommendation algorithms, while powerful, involve large numbers of hyperparameters. Tuning hyperparameters for nding the best recommendation outcome often requires execution of large numbers of algorithmic experiments particularly when multiples evaluation metrics are considered. Existing recommender systems platforms fail to provide a basis for systematic experimentation of this type. In this paper, we describe librec-auto, a wrapper for the well-known LibRec library, which provides an environment that supports automated experimentation.
A recommender system aims to predict users' preferences and suggests desirable items to them. Due to their power and e ectiveness in generating personalized recommendations and consequently increasing companies' pro t, such systems have become an essential tool in electronic commerce systems. There are a wide variety of real world examples like video recommendation in Youtube, item recommendation in Amazon, and music recommendation in Pandora. In all these applications, speci c recommendation algorithms are employed for generating recommendations to users.
Research on recommender systems has expanded to a wide variety of topics as new problems emerged. This expansion introduced new evaluation criteria for measuring the performance of recommender systems and made recommendation algorithms, while accurate, very complicated, computationally expensive, and sensitive to parameter values. Understanding the nature of these algorithms requires extensive experiments over combination sets of parameters, multiple data sets, and di erent metrics.
Depending on the characteristics of the data set and evaluation criteria,
speci c recommendation algorithm will be needed and assignments for its
parameters may vary [
Hyperparameters play an important role on the performance of many
recommendation algorithms, requiring careful tuning. For example, the integrated
neighborhood and singular value decomposition model3 [
Moreover, reproducibility of previous experiments is key for conducting further analysis in the future. Doing this, it is necessary to save all elements of each experiment and retrieve them in the future. This is very useful particularly when analyzing recommendation results with di erent metrics is needed.
However, while a number of recommendation platforms have been developed
for recommender systems researchers: LibRec [
We have implemented librec-auto, a Python-based wrapper that
encapsulates LibRec and supports the automation of repetitive experiments. In [
In this section, we introduce some well-known recommendation tools for recommender systems research and discuss their main features along with their advantages and disadvantages. 3 SVD++ in Librec.
LibRec 2.0 [
However, LibRec does not naturally support large-scale experimentation. Similar to other recommender systems platforms, LibRec is implemented as a chained series of operations: splitting testing and training data, training an algorithm, and executing the algorithm and evaluating results.
This monolithic design creates several problems for researchers. One is that intermediate operations are not saved and repeating similar experiments entails redundant computation. It is also the case that training/testing splits are not saved when LibRec computes them. Running with the same random seed guarantees the same split will be computed, but because there is no explicit storage of the training data, it is di cult for a researcher to perform detailed analysis and debugging where knowledge of the speci c input data of each fold would be helpful.
Finally, LibRec uses a simple key-value con guration system based on the Java Properties class. The at structure requires a multi-part key-naming scheme, which places signi cant cognitive load on the experimenter to remember the meaning of each key and the appropriate set of keys required for each algorithm. 2.2
MyMediaLite 3.11 [
Even though MyMediaLite saves computed models, it does not save the intermediate les and outputs. Also, it does not have a con guration le for specifying the hyperparameters for its recommendation algorithms. 2.3
There are a wide range of other platforms for experimentations on recommender
systems: Mahout [
These platforms aim to provide a comprehensive tools for practitioners and researchers for experimenting and developing recommendation algorithms, but none provide tools for large-scale automated experimentation with a large library of state-of-the-art algorithms. 3
librec-auto is a wrapper built around the core LibRec functionality to enhance the platform's ability to support reproducible large-scale experiments. The tool is currently under development with additional features planned. The core LibRec library is used unmodi ed, with some additional Java functionality implemented in a separate set of Java classes.
Figure 1 shows the operation of librec-auto and its relationship to the core LibRec system. New elements of the system are indicated in bold font. As shown, the system reads an XML con guration le, which is converted into the properties le (or les) required by LibRec. The data from each training / split is saved to that future experiments can share a common experimental con guration. Unlike in core LibRec, recommendation generation and evaluation are separate steps, so that previously-generated recommendation results can be reused where possible.
Another key element of the librec-auto implementation is the ability (described below) to iterate over di erent parameter values in one scripted experiment. The system performs all of the necessary bookkeeping and allows the user to write scripts that summarize over all of experiments, including the creation of visualizations. 3.1
LibRec uses a simple unstructured key-value properties le to de ne the parameters related to an experimental con guration. The le format does not capture the relationships between parameters and it can be di cult to determine, for example, what parameters are appropriate for which algorithms. librec-auto uses an XML-based con guration le, with greater self-descriptive power, and the ability to perform error and consistency checking. The properties les required by LibRec are produced and managed by librec-auto, making the process less tedious, especially when multiple, similar, experiments are being performed.
Figure 2 shows a fragment of one such XML con guration le, showing how the element labels and embedding structure clarify the experimenter's intent. This part of the le de nes an algorithm to be tested, in this case ItemKNN, with its con guration settings { the number of neighbors to be used and the similarity metric. The results are to be computed in recommendation lists of size 10 and evaluated by the nDCG measure. Note that multiple values are listed for neighborhood-size. The system will therefore run an experiment for each given value. 3.2
As usual for Python programs, librec-auto provides pip style installation. It makes the installation process easy and provides access to code everywhere for running experiments.
On a machine equipped with Python and Java and with Librec and librec-auto installed, the next step is to create a con guration le. The con guration speci es the data to be operated on, the type of evaluation to be performed, the algorithms to be run, and other operations. 3.3
As the example shows, one of the important capabilities of librec-auto is its
ability to automate multiple experiments across algorithmic hyperparameters,
supporting tuning and sensitivity analysis. This type of functionality will be
familiar to users of scikit-learn's GridSearchCV model selection tool [
For each combination of parameters, the system creates a separate subdirectory with its own con guration and output les. A separate instance of LibRec runs on each con guration, allowing for parallel operation. 3.4
Some existing tools (for example, Librec and LKPy) allow parallel operation of individual experiments by computing folds of a cross-fold validation experiment separately. This is su cient when single experiments are performed. librec-auto, on the other hand, has the capability of running multiple experiments in parallel. This allows for parallelization e ciencies for a wider range of experiment types. 3.5
Because recommendation algorithms can be computationally expensive at scale, librec-auto is designed to help minimize wasted computational e ort. As much as possible, the intermediate results are stored, including training/test splits and algorithm outputs, as noted above. < alg > < class > ItemKNNRecommender </ class > < similarity > cos </ similarity > < neighborhood - size >
< value >10 </ value > < value >20 </ value > </ neighborhood - size > </ alg > < eval > < metric > NDCG </ metric > < list - size >10 </ list - size > </ eval >
Because intermediate outputs are recorded, it is possible in librec-auto to execute multiple evaluation metrics on a single set of experimental results. LibRec allows the computation of multiple metrics when the experiment is being run, but applying a di erent metric after the fact requires starting over, with potentially signi cant computational cost. Note that error-oriented and rankingoriented experiments are not compatible: the results computed for a ranking metric are recommendation lists and can be evaluated with ranking metrics such as precision. librec-auto is aware of this distinction and can warn the user if there is an attempt to apply an metric incompatible with the computed results. The multiplicity of experiments that librec-auto can perform can yield a large amount of data that researchers must collate and examine. It would be tedious to have to pore over LibRec's log les to attempt to discover the di erences between di erent parameter settings.
To make such interpretative tasks more e cient, librec-auto allows for a nal summarization phase to follow the completion of a set of experiments. This is e ectively one or more scripts that the experimenter can write to process the log le data into a useful format. Using plotting libraries such as matplotlib, it is possible to create and save visualizations that the researcher can quickly scan. 4
In this paper and our associated demo, we presented librec-auto, a tool for automating recommender systems experimentation using the LibRec 2.0 platform. Our tool retains the bene ts of working with LibRec while making large-scale experimentation easier, more e cient, and more reproducible. Some of important features of librec-auto presented in this paper are: exible con guration, installation, automation, multi-threading, intermediate results, and summarization.