Recommender Systems have shown to be a useful tool for reducing over-choice and providing accurate, personalized suggestions. The large variety of available recommendation algorithms, splitting techniques, assessment protocols, metrics, and tasks, on the other hand, has made thorough experimental evaluation extremely difficult. Elliot is a comprehensive framework for recommendation with the goal of running and reproducing a whole experimental pipeline from a single configuration file. The framework uses a variety of ways to load, filter, and divide data. Elliot optimizes hyper-parameters for a variety of recommendation algorithms, then chooses the best models, compares them to baselines, computes metrics ranging from accuracy to beyond-accuracy, bias, and fairness, and does statistical analysis. The aim is to provide researchers with a tool to ease all the experimental evaluation phases (and make them reproducible), from data reading to results collection. Elliot is freely available on GitHub at https://github.com/sisinflab/elliot.
In the last decade, Recommendation Systems (RSs) have gained momentum as the pivotal choice for personalized decision-support systems. Recommendation is essentially a retrieval task where a catalog of items is ranked in a personalized way and the top-scoring items are presented to the user [1]. Once the RSs ability to provide personalized items to clients had been demonstrated, both academia and industry began to devote attention to them [2]. This collective effort resulted in an impressive number of recommendation algorithms, ranging from memory-based [3] to latent factor-based [4,5], as well as deep learning-based methods [6]. At the same time, the RS research community realized that focusing only on the accuracy of results could be detrimental, and started exploring beyond-accuracy evaluation [7]. As accuracy was recognized as insufficient to guarantee users' satisfaction [8], novelty and diversity [9,10] came into play as new dimensions to be analyzed when comparing algorithms. However, this was only the first step in the direction of a more comprehensive evaluation. Indeed, more recently, the presence of biased [11] and unfair recommendations towards user groups and item categories [12] has been widely investigated. The abundance of possible choices has generated confusion around choosing the correct baselines, conducting the hyperparameter optimization and the experimental evaluation [13], and reporting the details of the adopted procedure. Consequently, two major concerns have arisen: unreproducible evaluation and unfair comparisons [14].
The advent of various frameworks over the last decade has improved the research process, and the RS community has gradually embraced the emergence of recommendation, assessment, and even hyperparameter tweaking frameworks. Starting from 2011, Mymedialite [15], LensKit [16], LightFM [17], RankSys [9], and Surprise [18], have formed the basic software for rapid prototyping and testing of recommendation models, thanks to an easy-to-use model execution and the implementation of standard accuracy, and beyond-accuracy, evaluation measures and splitting techniques. However, the outstanding success and the community interest in Deep Learning (DL) recommendation models, raised the need for novel instruments. LibRec [19], Spotlight1 , and OpenRec [20] are the first open-source projects that made DL-based recommenders available -with less than a dozen of available models but, unfortunately, without filtering, splitting, and hyper-optimization tuning strategies. An important step towards more exhaustive and up-to-date set of model implementations have been released with RecQ [21], DeepRec [22], and Cornac [23] frameworks. However, they do not provide a general tool for extensive experiments on the pre-elaboration and the evaluation of a dataset. Indeed, after the reproducibility hype [24,25], DaisyRec [14] and RecBole [26] raised the bar of framework capabilities, making available both large set of models, data filtering/splitting operations and, above all, hyper-parameter tuning features. Unfortunately, even though these frameworks are a great help to researchers, facilitating reproducibility or extending the provided functionality would typically depend on developing bash scripts or programming on whatever language each framework is written. This is where Elliot comes to the stage. It is a novel kind of recommendation framework, aimed to overcome these obstacles by proposing a fully declarative approach (by means of a configuration file) to the set-up of an experimental setting. It analyzes the recommendation problem from the researcher's perspective as it implements the whole experimental pipeline, from dataset loading to results gathering in a principled way. The main idea behind Elliot is to keep an entire experiment reproducible and put the user (in our case, a researcher or RS developer) in control of the framework. To date, according to the recommendation model, Elliot allows for choosing among 27 similarity metrics, defining of multiple neural architectures, and choosing 51 hyperparameter tuning combined approaches, unleashing the full potential of the HyperOpt library [27]. To enable evaluation for the diverse tasks and domains, Elliot supplies 36 metrics (including Accuracy, Error-based, Coverage, Novelty, Diversity, Bias, and Fairness metrics), 13 splitting strategies, and 8 prefiltering policies.
Elliot is an extendable framework made up of eight functional modules, each of which is in charge of a different phase in the experimental suggestion process. The user is only required to submit human-level experimental flow information via a customisable configuration file, so what happens beneath the hood (Figure 1) is transparent to them. As a result, Elliot constructs the whole pipeline. What follows presents each of Elliot's modules and how to create a configuration file.
The Data modules are in charge of handling and organizing the experiment's input, as well as providing a variety of supplementary data, such as item characteristics, visual embeddings, and pictures. The input data is taken over by the Prefiltering and Splitting modules after being loaded by the Loading module, whose techniques are described in Sect.2.1.2 and 2.1.3 respectively.
Different data sources, such as user-item feedback or side information, such as the item visual aspects, may be required for RSs investigations. Elliot comes with a variety of Loading module implementations to meet these requirements. Furthermore, the user may create computationally intensive prefiltering and splitting operations that can be saved and loaded to save time in the future. Additional data, such as visual characteristics and semantic features generated from knowledge graphs, can be handled through data-driven extensions. When a side-information-aware Loading module is selected, it filters out items that lack the needed information to provide a fair comparison.
Elliot provides data filtering procedures using two different techniques after data loading. Filter-by-rating is the first method implemented in the Prefiltering module, which removes a useritem interaction if the preference score falls below a certain level. It can be a Numerical value, such as 3.5, a Distributional information, such as the worldwide rating average value, or a user-based distributional (User Dist.) value, such as the user's average rating value. The 𝑘-core prefiltering approach eliminates people, objects, or both if there are less than 𝑘 documented interactions. The 𝑘-core technique can be used to both users and things repeatedly (Iterative 𝑘-core) until the 𝑘-core filtering requirement is fulfilled, i.e., all users and items have at least 𝑘 recorded interactions. Since reaching such condition might be intractable, Elliot allows specifying the maximum number of iterations (Iter-𝑛-rounds). Finally, the Cold-Users filtering feature allows retaining cold-users only.
Elliot implements three splitting strategies: (i) Temporal, (ii) Random, and (iii) Fix. The Temporal method divides user-item interactions depending on the transaction timestamp, either by setting the timestamp, selecting the best one [28,29], or using a hold-out (HO) mechanism. Hold-out (HO), 𝐾-repeated hold-out (K-HO), and cross-validation (CV ) are all part of the Random methods. Finally, the Fix approach leverages an already split dataset.
After data loading and pre-elaborations, the Recommendation module (Figure 1) provides the functionalities to train (and restore) both Elliot's state-of-the-art recommendation models and custom user-implemented models, with the possibility to find the best hyper-parameter setting.
To date, Elliot integrates around 50 recommendation models grouped into two sets: (i) popular models implemented in at least two of the other reviewed frameworks, and (ii) other wellknown state-of-the-art recommendation models which are less common in the reviewed frameworks, such as autoencoder-based, e.g., [6], graph-based, e.g., [30], visually-aware [31], e.g., [32], adversarially-robust, e.g., [33], and content-aware, e.g., [34,35].
According to Rendle et al. [25], Anelli et al. [36], hyper-parameter optimization has a significant impact on performance. Elliot supplies Grid Search, Simulated Annealing, Bayesian Optimization, and Random Search, supporting four different traversal techniques in the search space. Grid Search is automatically inferred when the user specifies the available hyper-parameters.
After the training phase, Elliot continues its operations, evaluating recommendations. Figure 1 indicates this phase with two distinct evaluation modules: Metrics and Statistical Tests.
Elliot provides a set of 36 evaluation metrics, partitioned into seven families: Accuracy, Error, Coverage, Novelty, Diversity, Bias, and Fairness. It is worth mentioning that Elliot is the framework that exposes both the largest number of metrics and the only one considering bias and fairness measures. Moreover, the practitioner can choose any metric to drive the model selection and the tuning.
All other cited frameworks do not support statistical hypothesis tests, probably due to the need for computing fine-grained (e.g., per-user or per-partition) results and retaining them for each recommendation model. Conversely, Elliot helps computing two statistical hypothesis tests, i.e., Wilcoxon and Paired t-test, with a flag in the configuration file.
When the training of recommenders is over, Elliot uses the Output module to gather the results. Three types of output files can be generated: (i) Performance Tables, (ii) Model Weights, and (iii) Recommendation Lists. Performance Tables come in the form of spreadsheets, including all the metric values generated on the test set for each recommendation model given in the configuration file. Cut-off-specific and model-specific tables are included in a final report (i.e., considering each combination of the explored parameters). Statistical hypothesis tests are also presented in the tables, as well as a JSON file that summarizes the optimal model parameters. Optionally, Elliot stores the model weights for the sake of future re-training.
Elliot is triggered by a single configuration file written in YAML (e.g., refer to the toy example sample_hello_world.yml). The first section details the data loading, filtering, and splitting information defined in Section 2.1. The models section represents the recommendation models' configuration, e.g., Item-𝑘NN. Here, the model-specific hyperparameter optimization strategies are specified, e.g., the grid-search. The evaluation section details the evaluation strategy with the desired metrics, e.g., nDCG in the toy example. Finally, save_recs and top_k keys detail, for example, the Output module abilities described in Section 2.4.
Elliot is a framework that looks at the recommendation process from the eyes of an RS researcher. To undertake a thorough and repeatable experimental assessment, the user only has to generate a flexible configuration file. Several loading, prefiltering, splitting, hyperparameter optimization, recommendation models, and statistical hypothesis testing are included in the framework. Elliot reports may be evaluated and used directly into research papers. We evaluated the RS assessment literature, putting Elliot in the context of the other frameworks and highlighted its benefits and drawbacks. Following that, we looked at the framework's design and how to create a functional (and repeatable) experimental benchmark. Elliot is the only recommendation framework we're aware of that supports a full multi-recommender experimental pipeline from a single configuration file. We intend to expand the framework in the near future to incorporate sequential recommendation scenarios, adversarial attacks, reinforcement learning-based recommendation systems, differential privacy facilities, sampling assessment, and distributed recommendation, among other things.