In general, recommender systems solve the problem of information overload by helping users of services to nd items in which they are interested. There are plenty of algorithms and models that can be used to build such a subset of items, and their performance may vary not only for di erent datasets but for separate parts of a single one. This issue leads to the "algorithm selection problem". Many state-of-art solutions to this problem are based on meta-learning. It associates the task features with the performance of the base-level algorithms and models. This paper presents the method of recommendation algorithm selection for particular users (or items) that uses binary representations of both explicit metadata of them and computable statistical meta-features. There are two di erent techniques within the proposed method, which are based on classi cation or clustering of such binary data, respectively. The metalearning process is almost automated. The ndings of the experiments prove that the usage of the method for recommendation algorithm selection is reasonable and e ective. In most cases, a recommender system that uses the metamodel shows lower rating prediction errors compared to any other one utilizing a single model or algorithm for all the users (or items), while in a small number of tests their performance is just the same. The detailed analysis of the evaluation results allows for a rming that the described metamodels can be used in real-world systems to improve the experience of particular users.
Numerous modern companies provide their clients with a great variety of services, products, and content. Such diversity leads to the problem of information overload that has an adverse e ect on both user experience and service growth. Recommender systems (RSs) are used to solve this problem e ectively. They automatically create suggestions of items that most likely lie in a user's area of interest.
Modern recommendation algorithms are actively used in real-world systems [
The crucial problem that arises is that the performance of a model may vary for di erent users and items within a single dataset. For example, a matrix factorization model can predict the interests of a particular service user precisely, while the k-Nearest-Neighbors algorithm is much more applicable for making recommendations for another individual. A state-of-art solution to this problem is the use of meta-learning.
Meta-learning lies in the creation of metamodels that associate task
features with algorithms' performance. Such metamodels can be separated into
the global-level (of a dataset), middle-level (of particular users or items), and
micro-level (of particular rows) ones [
In most cases, a metamodel utilizes only the data about the base models' performance. The real-world systems are generally aware of users' and items' features, even if they are not used in the recommendations generation process. The core idea behind this research is that a metamodel can take a cue from both such metadata and standard meta-features. Moreover, when a metamodel training dataset is represented in a binary format, it allows for applying the same algorithms in di erent business cases.
This paper proposes the middle-level automated meta-learning method that uses the recommendation algorithms' evaluation results as well as the binary representation of both users' (or items') metadata and their computable metafeatures.
The formal statement of the research problem is as follows. There is a dataset consisting of (u; i; rui) rows that show what rating rui has the user u given to the item i. Also, there are users' and items' metadata stored as binary vectors. Several recommendations models (base models) are trained on the part of a dataset, and evaluated on another part, which has given the predictions (u; i; rbui). The task is to create a metamodel that selects the best base model for a particular user (or item), i.e. makes a personalized selection.
It should be stated that below is the solution that utilizes users' metadata. The solution that uses items' metadata is the same.
At this stage, two di erent solutions to the problem are proposed. On the
one hand, one can take the evaluation results as the starting point and make a
dataset showing the correspondence between metadata of the user and the best
model for them. In this case, the task is to solve a classi cation problem. On the
other hand, one can start from the nature of users, group them using particular
features, and select the best model for each built group. In this case, the task is
to solve a clustering problem [
While a metamodel is being trained, its input is built from two parts. The rst one is a set of explicit users' metadata, taken from a dataset, e.g. age, sex, or profession. The second one contains meta-features computed on the basis of users' rating information, i.e., some standard values used in meta-learning.
Fig. 1 shows the proposed recommender system structure.
Both parts are represented as binary vectors. It is assumed that users' metadata are already binary, while the computable features are converted after a calculation.
Both proposed meta-learning techniques suggest the users are grouped. The best base model must be chosen for each users' group U from the whole set of models M . The selection is performed by the equation mbest = argmax( X f (m; u));
m2M u2U where f (m; u) is the utility function for the model m and the user u that grows together with the model's performance. The de nition of such a function depends on the error metric in use. Since RMSE is highly applicable to the task of rating prediction, the utility function is de ned as follows.
Let r 2 RN be the vector of actual ratings rj , rm 2 RN be the vectors of b predictions rbmj which model m has given for the row j. The vector emin 2 RN contains values ejmin = mm2iMn((rbmj rj )2) equal to minimal squares of error for each row. Thus, the utility function for the model m and the user u is computed by the formula f (m; u) =
X i.e. as the cumulative di erence between the minimal squared errors and the models' squared errors.
When one chooses an algorithm for a users' group, they should consider the fact that several base-level models can show roughly equal performance, but one of them is the best on a whole dataset, i.e., ts the task and its domain more than others. Another model may show a higher level of utility not only because of its accordance with this users' group but due to the existence of outlying values in a training dataset, causing errors in modeling of interests. It may lead to the situation when a wrong base-level model that has learned several random records, but not the actual patterns of users' behavior, is selected.
Consequently, there should be a strong reason to prefer a particular algorithm for a users' group over the overall best one. In this research, the stated problem is solved as follows. After the model mU best that has the highest utility for a group is chosen, it is calculated how much higher its utility is compared to the one of the overall best model moverall best. If this value is lower than the constant , the model moverall best is used for a users' group.
Since a utility function may take strictly non-negative values as well as strictly non-positive ones, the nal selection of a model mU for a users' group U is performed by equations: utilityU =
X f (mU best; u); utilityoverall = u2U X f (moverall best; u); u2U kimp = max (
utilityU utilityoverall ; utilityoverall );
utilityU mU = (mU best moverall best if kimp >= if kimp < : (4) (5) (6) (7)
The meta-learning techniques used within the method are described below. 2.1
Meta-features, which are commonly used in metamodel training, can be
separated into three groups [
In terms of the current research, the meta-features from the rst group are of particular interest. Base-level model features are not studied, since it is assumed that all the models are of di erent kinds, and not the ones of a single kind that di er only by hyperparameters. The usage of "landmarkers" is also not studied, as the stated task implies the hybridization of several models that are trained already.
Several statistical features have been chosen; they describe rating dataset parts that correspond to separate users. For each of these features, the algorithm of conversion to 0-or-1 columns is described. The meta-features count has been restricted to three to avoid the increase in the width of the metamodel training dataset. The meta-features are shown in Table 1. Meta-Feature Conversion to the Binary Format Relative count of user's ratings. Three columns for ranges Equals to count of user's rating divided by [0; 0:75), [0:75; 1:25), [1:25; +1). the average rating count per user in a whole dataset.
Standard deviation of user's ratings.
Two columns for ranges: [0; 0:1 of maximal possible rating), [0:1 of maximal possible rating; +1).
Two columns for positive and negative values respectively.
User's ratings skewness. 2.2
Since the input consists of binary vectors, a classi er can distinguish any two users having non-equal metadata vectors. Thus, the set U is de ned as the group of users having equal metadata vectors. The best model is selected for each set U using formulas (1) { (7). These actions build the dataset for a classi er, which associates the metadata vector with the base model as a class label. Obviously, if the dataset includes all the possible varieties of binary metadata vectors, the classi er is not needed. However, most likely, the dataset does not cover all of them.
It is considered that the model count is no less than three, so it is better to train an ensemble of binary classi ers using the "One-vs-Rest" strategy. Since the metadata consists of binary vectors, the ensemble can be e ectively founded on any decision-tree based classi ers. In this research, the random forest classi er is used. 2.3
The clustering-based technique brings a speci c problem. A clusterer can consider some users "noise" and not attach them to any group. In this case, the meta-learning process is organized as follows.
At rst, the overall set of users' metadata is clustered. As a result, several clusters are obtained, each of which can be considered as a target group U , along with the separate part of users not belonging to any cluster. The best models for clustered users are chosen using the formulas (1) { (7). For the "noise" group, a metamodel uses the overall best base model, since it is incorrect to consider these users as a neighborhood of any kind.
The task of clustering of binary vectors can be considered as the one of point
clustering in the multi-dimensional Euclidean space. It is possible if at least one
of the two following conditions is met [
Thus, the OPTICS model with cosine similarity as a distance metric has been chosen initially to perform clustering. It has been stated heuristically that the farthest two users can di er in no more than two columns and coincide in at least one another column. So, according to the distance metric, the maximal cluster size equals 1 1=(p2 p2) = 0:5.
However, the OPTICS model has a signi cant disadvantage. Its accuracy is
high, but the training process on the datasets used in the experimental study
can take an unacceptably long amount of time. This problem can be solved
by using the HDBSCAN model [
The initial pool of algorithms in this research included ve collaborative ltering
models, implemented in the Surpriselib library [
Five datasets [
Each dataset was used in the following way. It was shu ed randomly and split into three parts A, B, and C with 60, 20%, and 20% of records from the initial dataset, respectively. Each model from the initial pool was trained on part A and evaluated on part B. Based on the evaluation results, the pool of the best base models was built. Next, four metamodels were trained, each of which was classifying/clustering users/items. The constant was de ned as 1.25. During the process of classi er training, only 70% of users/items from part B were used. Finally, all the base models and created metamodels were evaluated on the part C. The RMSE was used as the evaluation metric.
The evaluation results are presented in Table 3. Additionally, the table includes the results for a perfect abstract metamodel that chooses the truly best algorithm for each user or item.
In four out of ve cases, the best metamodel had better performance than the best base-level one, since its RMSE on a whole dataset was lower. The greatest bene t from the meta-learning was taken for the Modcloth dataset. In the case of Movielens 1M dataset, the best metamodel had shown exactly the same result as the best base-level one.
The improvement achieved by applying the proposed method was di erent for di erent datasets. It is possible to suggest that it depended both on the best base-level model accuracy and on the level of maximal theoretical improvement that could be obtained by per-user (or per-item) algorithm selection. Figure 2 illustrates the relation between these values.
According to the graphs, the actual improvement that the metamodel usage made could be truly dependent on the maximal possible one. At the same time, the possibility of such dependency on the relative best base-level model error was questionable due to the results for the Rent the Runway dataset. It was important that previous experiments that had been performed without statistical meta-features and the usage of the constant showed that this dependency could exist, and the results had been generally worse. It allows for a rming that combining explicit metadata together with computable meta-features and performing a restricted selection of the best model for a users' (or items') group is reasonable and increases the accuracy of metamodels.
Both the numerical results and the graphs showed that the actual improvement was far from theoretically possible. Along with that, training accuracy was reasonably good. The graphs in Figure 3 demonstrate how often metamodels have selected the truly best algorithm for a user or an item.
30% 20% 10% 0%
MM (user class.)
MM (item class.)
MM (user clust.)
MM (item clust.) According to these measurements, classi cation and clustering models had a rather similar performance that led to the best selection rate that equaled 2428%. Worse results showed by item-based models for three out of ve datasets could be explained by relatively small item counts in them. It allows for concluding that the bottleneck lies in both dataset size and the complexity of determining the truly best model for users' or items' groups before training.
A pending question within this research is how to choose an optimal value for the constant . Figure 4 shows the ratios between the lowest metamodel RMSE and the lowest base-level model RMSE with di erent values. When is less or equal to 1, it is equivalent to to be ignored (following formulas (4) { (7) from Section 2).
In three out of ve cases, the usage of the constant was e ective. In the case of the Modcloth dataset, there was almost no di erence between the evaluated values. For the Amazon Phones dataset, the restrictions in the algorithm selection the constant brought had a ected metamodels' performance negatively. Consequently, the optimal value depended on the used dataset and performance of the base-level models. Since the evaluation of several metamodels with di erent values has a high computational cost, it is highly preferable to deter
Modcloth mine its value before the training. Possibly, this task can be done on the basis of the base-level models' errors. This problem requires additional research.
The impact of the usage of the metamodels on the time needed for recommendation generation has also been considered. Figure 5 shows the average times the base-level and meta- models need to make a single rating prediction on the used datasets. The experiments have been performed using the Intel Core i7-7700HQ CPU and DDR-4 RAM on 2133 MHz frequency. It is signi cant that the Surpriselib framework evaluates test requests individually, and not in batches. This peculiarity allows the measured time values to be as realistic as possible. 500 400 300 200 100 0
SVD
SVD++
KNN Baseline
Baseline Co-Clustering MM (user Only class.)
MM (item class.)
MM (user clust.)
MM (item clust.)
Depending on a dataset, when a metamodel was used, the average prediction time was ranged between 143 and 413 microseconds. This interval allowed an RS to process thousands of requests per second. The speed of base-level models, in some cases, was comparable to the metamodels' one, but, in some other cases, it was noticeably higher. However, the base-level models have particularly native implementation, while the metamodels are implemented in pure Python without any optimizations. Besides, it is obvious and seen from the graphs that the speed of a hybrid model depends on the speed of all the models it combines directly. Thus, the prediction time of a recommender system that uses the metamodel is enough for the production but it can be decreased by optimizations and the native implementation.
The results show that all the metamodels are equally performant. To choose one of them in real-world tasks, one should consider the number of users and items in the available data. The classi cation-based metamodels are preferred in this case, since they can be trained faster than the clustering-based ones. To employ the proposed method in an existing system that includes several recommenders, one can convert users' metadata to the binary vectors format, train the classi er using the existing recommenders' evaluation results, and redirect recommendation requests to the metamodel.
Although the improvement is moderate, the usage of the metamodel that utilizes the peculiarities of a particular user can noticeably increase the quality of their experience. To examine this assumption, one should conduct additional research that aims at the evaluation of ranking measures and uses the appropriate datasets. 4
The presented work describes a solution to the problem of recommendation algorithm selection for particular users or items within a single service. The paper proposes the middle-level automated meta-learning method that uses the rating prediction algorithms' evaluation results as well as the binary representation of both users' (or items') metadata together with their computable meta-features. Two techniques of creating the metamodel are suggested; they are based on classi cation and clustering, respectively. Both techniques are applicable for users' features as well as for an items' features.
The metamodels have been evaluated on ve open datasets, with the usage of collaborating ltering models on a base level. The experiments prove that the usage of the proposed method is reasonable and e ective. The performance in comparison to the base-level models improved up to 3.81% of RMSE value in a best-case scenario and did not decrease in any case. The measures obtained from the evaluation have been analyzed in detail, and the possible directions of future work been described, including the ones aimed at performance enhancement. Current results can be used in real-world systems; they can improve the experience of their users. The additional advantage of the method is that the process of creation of a metamodel is almost automated, and the only manual task is to convert explicit metadata to the binary format.