Recommender Systems (RS) are widely used to provide users with personalized suggestions taken from an extended variety of items. One of the major challenges of RS is the accuracy in cold-start situations where little feedback is available for a user or an item. Exploiting available user and item metadata helps to cope with this problem. We propose a hybrid training framework consisting of two predictors, a collaborative filtering instance and a metadata-based instance relying on content and demographic data. Our framework supports a wide range of algorithms to be used as predictors. The cross-training mechanism we design minimizes the weaknesses of one instance by updating its training with predicted data from the other instance. A sophisticated sampling function selects ratings to be predicted for cross-training We evaluate our framework conducting multiple experiments on the MovieLens 100K dataset, simulating diferent scenarios including user and item cold-start. Our framework outperforms stateof-the-art algorithms and is able to provide accurate predictions across all tested scenarios.
Recommender Systems (RS) are nowadays of tremendous importance across a multitude of diferent areas within the field of digital technology. While the amount of choice given to a user or customer has been growing continuously, users find it increasingly dificult to come up with a satisfactory selection without being overwhelmed by quantity.
Recommender systems help providers to ofer users personalized suggestions in order to improve user experience. A number of diferent approaches are common to generate suggestions. The two most prominent and widely used methods are Collaborative Filtering (CF) and Content-based Filtering (CN) techniques. CF algorithms only take user feedback into consideration. Feedback can be given in an explicit (e.g. movie rating 1-5 stars) or implicit 1.05 SE1.00 M R 0.95 0.90 CF MD Hybrid 0 10 20 30 40 50
60 #Ratings / User (e.g. user performed search query) way. CN algorithms mainly rely on known item properties to discover items similar to each other iftting the user’s preference.
One major dificulty of CF and CN methods is that limited amount of feedback from some users (e.g. new/inactive) usually results in poor recommendations for those users. A user who has not given any feedback has not expressed any preference, therefore it is not possible to compute personalized recommendations. This phenomenon is called the cold-start (CS) or ramp-up problem. We distinguish between user cold-start (little feedback from specific users), item cold-start (little feedback to specific items) and system cold-start (both user and item cold-start). If information about users is available, Demographic Filtering (DM) can be applied and reasonable predictions can be made in case of user cold-start. Instead of processing feedback which is not yet available for a cold-start user, DM exploits user properties available through metadata to generate recommendations based on preferences of users with a similar demographic profile. In turn, CN helps to overcome item cold-start issues by taking item metadata into consideration. Alone however CN fails to deal with a user cold-start. As CN and DM both process metadata, algorithms often combine both methods if metadata is available for users and items. We dub the combination of CN and DM Metadata-based Filtering (MD). While MD algorithms are able to produce reasonable results in case of user or item cold-start and can deal with a system cold-start as well, experiments have shown that CF usually outperforms MD as soon as some amount of feedback has been given (see Figure 1).
In order to archive optimal overall performance it is necessary to combine the results of both CF and MD. The challenge of designing hybrid systems is subject to ongoing research. We present a framework to combine the advantages of both algorithms. 1.1
We contribute a framework for training a hybrid CF-MD model consisting of one instance of each, CF and MD. The framework utilizes cross-training, an approach using prediction results of one instance to train the other and vice versa. We design a sophisticated sampling method to select specific user/item interactions for crosstraining. This enables us to increase the overall performance of both predictors over entries with little as well as a with lot of feedback. Our method is able to cope with cold-start, a major drawback of the popular CF approach. At the same time prediction results for entries with lots of feedback also improve.
We test our approach on the MovieLens [
One of the most popular CF techniques in recent research is the
latent feature based matrix factorization (MF). Koren [
We distinguish between CF and MD methods and use them in
a divided fashion in our framework. Other researchers have tried
to include both approaches into a single model. Manzato et al. [
Most factorization models are covered by the model class of
Factorization Machines (FM) [
The technique of merging multiple individual algorithms or
their results is an important challenge in RS research. Taking the
average or weighted average of results is the simplest ensemble
method available. Jahrer et al. [
Another approach of building a hybrid system was presented by
Zhang at al. [
While most contributions addressed until now measure the error
of all predictions, other researchers focus on ranking the predicted
items and evaluating the ranks. Park et al. [
In this section we describe the functionality of our framework and explain cross-training as well as the selection and generation of data for cross-training.
The basic scenario for an RS incorporates a set of users U and a set of items I. The rating rui with u ∈ U, i ∈ I refers to the explicit rating given from a user u to an item i. We call ui an index pair or tuple. All possible tuples are contained in the set L = U × I. The set K ⊂ L consists only of index pairs ui ∈ K for which the ground truth of rating rui is available in the training set. The complement of the training index-set K denotes all tuples not known: K = L \ K. The training set is defined as: T = {(ui, rui )|ui ∈ K }.
Our hybrid framework combines an instance of a CF algorithm as well as an MD algorithm. Both models act as a regression function rˆ : L 7→ R, giving their prediction rˆui for an index pair ui.
The framework is universal in the way that both algorithms, CF and MD are interchangeable and almost any regression approach relying on training through labeled data can be inserted. We specify our choice of algorithms and further beneficial properties of possible algorithms in Section 3. 2.1
Step 1: Predict ratings for index pairs from SC F (K) to construct the teaching set.
Update training of MD using the teaching set and the original training set.
Step 2: Predict ratings for index pairs from SM D (K) to construct the teaching set.
Update training of CF using the teaching set and the original training set.
Step 3: If cross-training stopping criterion is not yet reached, go back to Step 1 and repeat cross-training iteration
To ensure that reasonable results can be predicted to be used in cross-training, the first step is separate initial training. Both models are individually trained with the training dataset T.
The mechanism of cross-training shown in Figure 2 is designed to improve each model by training it with ratings predicted by the other model. We now describe the process of the first cross-training step (upper half of Figure 2).
First, we draw a number of index pairs from K which are not included in the training set. This is done by the sampling function SC F (K) ⊂ K which generates a subset of the unknown index tuples which are to be used for cross-training. As the choice of which and how many tuples are selected from K is vital for the functioning of our whole framework, we provide a detailed description in Section 2.2.
To build the teaching data, a rating rˆui is predicted for each tuple of the cross-training index set generated by SC F . All predicted ratings and their corresponding index pairs are called the teaching n o dataset: EC F = (ui, rˆuCiF )|ui ∈ SC F (K) . Teaching data and original training data are concatenated and the resulting cross-training data XC F = EC F ∪ T is then used to update the training of MD.
The second cross-training step, using data predicted from MD to train CF (lower half of Figure 2), works in the same manner. Both steps are alternated until a predefined stopping criterion is reached.
We found that a fixed number of cross-training epochs works well. See Section 4.6.3 for an analysis of performance depending on number of epochs. 2.2
The choice of how many and especially which user/item tuples are selected for cross-training is an essential part of our framework. We will now explain how we design our sampling functions.
The underlying idea of cross-training is to use the advantages of one algorithm to decrease the impact of weaknesses of the other.
Our two regression models do not provide a confidence for a predicted rating. Still, for each index tuple, we can take an educated guess whether CF or MD yield a better prediction. In the case of CF, results are poor in case of cold-start, therefore we can assume that for users and items with few ratings MD will outperform CF. In this case, providing additional training data for cold-start users and items will increase the performance of CF. Vice versa CF is able to predict high quality ratings for users and items with a lot of feedback. Cross-training additional ratings to MD will also increase its overall performance.
Selecting ratings for which one algorithm most likely outperforms the other and which will provide the greatest performance gain is the key part of our framework.
Index Tuple Sampling from CF → ←
SC F
← more - fewer →
ratings
2.2.1 Sampling Index Tuples for Predictions of CF (Figure 3).
To draw index pairs from SC F we adopt the method proposed by
Zhang at al. [
C(rˆui ) = du × di
N
Predicting the rating rˆui for a user u who has rated often and an item i which has been rated often will result in a high confidence using this measure. This reflects the experience that CF generates better predictions as soon as more feedback is available. Zhang builds a probabilistic distribution based on the confidence measure: P(u, i) = Í
C(rˆui ) (u′,i′)∈K C(rˆu′i′ )
We use this distribution in SC F and sample index pairs from K without replacement meaning no duplicates are in the set of (1) (2) sampled index tuples. Regarding the number of selected samples, we determine the number of cross-training samples depending on the size of K and the factor δ :
|SC F (K)| = ⌊δ |K |⌋ (3)
Through cross-validation we found an optimal value δ = 0.15 for the MovieLens 100K dataset.
Index Tuple Sampling from MD
SM D
Users with ← more - fewer →
ratings
2.2.2 Sampling Index Tuples for Predictions of MD (Figure 4). As described before, CF performs poorly for users and items with little feedback. We will design SM D in a way that cross-training index pairs are selected specifically among these cold-start users and items. This technique enables us to significantly improve the performance of CF through cross-training.
Experiments have shown that MD produces better results than CF for users with fewer than a certain threshold of ratings (see Figure 1 and Section 4.6.2). We select index tuples for cross-training among these users with fewer than tuser ratings. Sampling is done individually for each user to ensure a minimum number of ratings in the complete cross-train dataset for all users. For each user u we select ⌊ϵuser · max (0, tuser − du )⌋ tuples, where the factor ϵuser ∈ R+ controls the amount of tuples selected for this user. du stands for the number of ratings by user u in the training dataset. The corresponding items i sampled for the cross-training tuples are selected randomly but giving a higher probability to items with a high amount of feedback. This is done in a linear fashion according to the number of ratings of an item i with a probability P (i) = di N where N normalizes the term to a sum of 1 and di refers to the number or ratings given to item i in the training set.
In the same manner we select additional tuples for each item for which the a number of training ratings is below threshold tit em . We then go on to select ⌊ϵit em · max (0, tit em − di )⌋ tuples for each item, again using a factor ϵit em ∈ R+. The distribution to pick corresponding users u for the tuples is P (u) = du again with normalization term N. N
Tuples for cold-start users are selected favoring items with many ratings contained in the training set over others through probability P (i). This is done deliberately since the prediction of MD is more reliable if at least the item was rated, in contrast to the case in which neither user nor item were rated at all. The same logic applies to cold-start items where tuples with active users are preferred.
Again the tuples are selected without replacement, avoiding redundant index pairs in the teaching set. The total amount of cross-training samples from MD is therefore: |SM D (K)| = Õ u ∈U + Õ i ∈I ⌊ϵuser max (0, tuser − du )⌋ ⌊ϵit emmax (0, tit em − di )⌋ (4) latex The hyperparameters tuser , tit em , ϵuser and ϵit em of our framework are determined through cross-validation. 3
This section contains a detailed description of the algorithms we choose to use for the individual models in our experiments. Some constraints apply to algorithms in order to be used as part of our framework. Both models must act as regression functions, predicting a value referring to the explicit feedback based on the index tuple (u, i). Additional metadata about user and item is used by MD. Furthermore algorithms which ofer the functionality of iterative training to update their parameters do not require complete retraining during each cross-training epoch. Applying iterative updates during cross-training instead of retraining the model greatly speeds up the process. We constructed both algorithms using stochastic gradient descent as optimizer and employ iterative updates. 3.1
CF is an often used method in RS and is therefore subject to
continuous research. We use SVD++ (Koren [
The MF approach for CF was popularized by Koren and Webb
(under pseudonym of Simon Funk) [
In its simplest form, the inner product is used to predict the
explicit rating given by user u to item i: rˆui = qTi pu . Including
biases, like global rating average µ and ofsets for user bu and
item bi , has shown to improve the results. Other extensions to the
basic MF have already been mentioned in Section 1.2. The SVD++
algorithm proposed by Koren incorporates additional latent vectors
yj ∈ Rk, j ∈ I into the factorization. For the set of items N (u)
which received implicit feedback by a user u, we compute the sum
of these vectors. The complete SVD++ prediction:
For lots of implicit feedback, the impact of |N (u)|− 21 Íj ∈N (u) yj
increases since more entries are in N (u) and because the
normalization uses the square-root. The dataset we use in our experiments
does not contain implicit feedback, just explicit ratings. We
therefore define that giving a rating can also be considered as implicit
feedback. This method is also used by Koren. That way of
measuring implicit feedback is, in a way, redundant information, as ratings
are already trained. Yet SVD++ has been proven to outperform
normal MF [
All model parameters qi , yi and bi for all i ∈ I, pu and bu for all u ∈ U as well as the global average µ are learned by solving the least-squares equation through gradient descent. Regularization is applied to all parameters except µ . We use diferent regularization values for diferent parameters:
min µ ,b∗,q∗,p∗,y∗ (u,i)∈K To cope well with cold-start, relying on known ratings from the training set alone does not sufice, as no personalized predictions can be made for users who have not given any feedback yet. In this case available metadata describing users and items enables an RS to produce reasonable results. We design a custom MD model consisting of a combination of diferent biases which are motivated by possible causal links.
We assume that all users and items are described by sets of attributes. Examples for such attributes from the MovieLens dataset (Section 4.2) are gender, age and occupation of users. Gender and occupation can be considered categorical fields, those fields have a discrete value in our dataset. To be able to represent continuous fields like age as a set of possible attributes, values are discretized into age groups, each group representing a possible attribute. The set of attributes of a user u is called D(u) (Example: D(u) = {is_f emale, is_technician, is_25 − 34y/o}).
The same feature preprocessing applies to items. Continuous data has to be discretized, categorical fields are simply one-hot encoded into possible attributes. The set of attributes of item i is denoted by C(i). The used MovieLens dataset provides a classification of movies into genres. C(i) will therefore contain one or more genres as attributes (Example: C(i) = {action, adventure, romance }). (7) (8) (10) (11) In this section we will refer to an item attribute as genre since the dataset provides genres, yet item attributes are not limited to that.
To model the preference of a user accurately and to come up with a reasonable prediction for rˆui we combine a number of biases and factors to exploit multiple causal links:
3.2.1 User and Item Bias. The global average and user/item biases are the same as for CF and represents a regularized version of average rating of a user and an item:
bui = µ + bu + bi 3.2.2 Atribute Bias. For every attribute d ∈ D(u) and c ∈ C(i) we apply a bias which states whether a group of users sharing an attribute d rates better or worse than average. The bias works the same for items with attribute c. Normalization is applied according to the number of attributes per user and item.
bat t r ib = 1 3.2.3 User Atribute to Item Atribute. Now we look at the combination of attributes describing user and item. We introduce a weight hcd corresponding to an ofset in the ratings given by users with attribute d ∈ D(u) for items with attribute c ∈ C(i) w.r.t. global average µ . An example would be that users with attribute is_male tend to rate movies with attribute action a bit higher than the global rating average µ . The additional factors дc for genres and fd for user attributes can adjust the impact on hcd depending on user attribute and genre: rˆuUiA→I A = 1 1 3.2.4 User Atribute to Item. The next part of the prediction models how a homogeneous audience responds to an individual item. “Toy Story” for example might be especially liked among users of age group 0-15. The weight kid corresponds to the rating ofset of users with attribute d to the specific item i: rˆuUiA→I = 1 + λ2 © |C(i)| |D(u)| c ∈C(i) d ∈D(u) « We choose a simple thresholding method to predict ratings by the hybrid system. Hybrid results are taken from CF, except if rˆui corresponds to a user or an item with very few ratings in which case predictions are taken from MD. We define fixed the cutof thresholds cuser and cit em as hyperparameters. Predictions for users and items with fewer than cuser /cit em ratings are taken from MD. 4
We test our hybrid framework in a number of experiments to evaluate its performance. The optimal hyperparameters used in our evaluation are determined through cross-validation. Thresholding parameters like tuser are selected through observation of experiments like the one conducted in Section 4.6.2. 4.1
We implement our model using Python and Keras [
We use Keras with Theano5 as backend and were able to significantly speed up our computation by utilizing the GPU. We are able to run a complete 5-fold cross-validation of our hybrid model in under 6 minutes (using Intel® Core™ i7-2760QM / NVIDIA® Quadro® 2000M on Linux).
All ratings (1-5) are linearly transformed to a numerical range of (0.1-0.9) to represent the likelihood for a user to like an item.
It is common practice to take certain precautions to avoid overiftting. We have already described the use of regularization. In addition to this we use early stopping for the initial training. By using a small part of the available data as validation set, not including 3github.com/sbremer/hybrid_rs 4keras.io/getting-started/functional-api-guide/ 5deeplearning.net/software/theano it in the training set K, and stopping training as soon as a minimum in loss is reached in the validation set. This is accomplished by storing the learned parameters if the validation loss reaches a new minimum, stopping if no new minimum is reached after n epochs (~5) and restoring the saved parameters as they yield the best validation performance. 4.2
To test and evaluate our proposed method we use the MovieLens
100K6 dataset provided by GroupLens [
The metadata included in the dataset is not very extensive, therefore we cannot expect MD to outperform baselines by a large margin. Gathering more data as well as sophisticated feature engineering can improve results but is often dificult and costly. As the focus of this research lies on our training framework we only use the data provided in the MovieLens dataset. 4.3
We compare our hybrid algorithm against a number of baseline algorithms:
4.3.1 BiasBaseline. This approach consists only of global, user specific and item specific bias. The prediction term is: rˆui = bui with bui from Section 3.2.1.
4.3.2 SVD. SVD is a common MF approach also incorporating biases. The dot product of an item specific and a user specific latent feature vector is used to predict ratings: rˆui = bui + qTi pu 4.3.3 Metadata. Our approach processing metadata is described extensively in Section 3.2. We use this model in a standalone mode as a baseline algorithm in order to show that our framework outperforms its components individually.
4.3.4 SVD++. This corresponds to the standalone version of the CF model also used in our hybrid framework. See Section 3.1 for further reference. 4.4
We measure the performance of our results using two evaluation metrics.
4.4.1 RMSE. The root-mean-square error (RMSE) of all predicted ratings rˆui of the test set is given as:
RMSE = s 4.4.2 Precision@k. Precision, representing a ranking based measurement, is calculated for each user separately. Ratings are predicted for all index tuples of the test dataset of a user. Predicted ratings are then sorted by value and the k items with the highest predicted score are considered actual suggestions. The calculated precision represents the portion of "good" suggestions among those which were predicted top k items. We consider an item recommendation "good" if that item is also among the true top ratings of the user.
Note that the dataset’s ground truth contains only integer values as ratings. The true top ratings of a user are determined by choosing the top k ratings. As more than k items can receive a top rating by the user, we also include all tying ratings in the set of true top ratings. This means that the true top ratings for a user may contain more than k items as a user can rate more than k items with a top rating. 4.5
All given scores are the mean of results using k-fold cross-validation with k = 5. We use 3 diferent fold methods to evaluate diferent situations:
4.5.1 Normal k-fold. The standard k-fold function splits the ratings into five parts and cross-validates those. As every user in the dataset has at least 20 ratings and 80% of the ratings are assigned to the training set, it is very likely that every user has at least 10 ratings in the training set. This situation corresponds to a normal test run, without complete cold-start.
4.5.2 User Cold-Start k-fold. This evaluation randomly splits the data by users not by ratings. One fifth of the users and all their given ratings will be used as testing data while the rest is given to the algorithm as training data. Representing complete user coldstart, no ratings of any of the users of the test dataset are available for training.
4.5.3 Item Cold-Start k-fold. To simulate a complete item coldstart, we split all items, taking 80% of items with all corresponding ratings as training set. The test set consists of the last fifth of items and their ratings. The algorithm being evaluated has not seen any ratings of the items in the test set. 4.6
We conduct multiple experiments to evaluate the performance and advantages of our hybrid framework. We include results of multiple baseline algorithms described above, including our CF and MD algorithms in standalone mode. For those baselines, we measure the score of the test dataset after initial training, before applying the proposed cross-training method.
4.6.1 Non and Complete Cold-Start (Table 1). We now show our results for the 3 evaluation folding methods.
The results of competing research are not listed in Table 1 as
diferent evaluation methods were used. In addition other methods
do not consider complete user and item cold-start. Zhang et al.
[
4.6.2 Diferent Intensities of User Cold-Start (Figure 1). In addition to our three diferent cross-validation splits, we tested our approach in case of user cold-start with varying intensity. To do so, we use our user cold-start validation, splitting users into five parts and using one part as the testing dataset. This mode represents a complete cold-start as no ratings of the test set users are in the training set. To simulate cold-start with few, instead of no ratings given by test set users, we randomly take a fixed number of ratings from every user in the test set and include these ratings in the training set. The created situation models users who have given some ratings.
We can observe that for no or very few ratings (fewer than ~30) included in the training set, MD outperforms CF. With more ratings, CF is able to produce more accurate predictions than MD. Our proposed hybrid framework outperforms both its individually trained components, CF and MD, in all tested cases of user cold-start proving that cross-training improves accuracy significantly. change when applying cross-training. Figure 5 shows the absolute change of error compared to the results after initial training (epoch zero). This evaluation shows the mean values of a 5-fold cross-validation.
While initially results of both CF and MD improve significantly, a minimum of CF’s error is reached after about 7 epochs. After that we observe an increase of error in the results of CF and our hybrid system. Early stopping of cross-training is implemented to abort at a minimum. We take runtime of the algorithm into consideration when determining the epoch after which to stop, resulting in a trade-of between best improvement through cross-training and lower runtime of the framework.
Our goal was to design a hybrid framework which combines the desirable properties of CF and MD, to be able to provide accurate recommendations over a variety of scenarios including cold-start. We tested our algorithm on the MovieLens 100K dataset and compared the results to a number of baseline algorithms, including our metadata-based approach, in standalone mode.
While the proposed framework is able to yield only slightly more accurate results in case of user cold-start, we clearly outperform all baselines during normal operation, including the frameworks component algorithms. During item cold-start the framework’s results are comparable to those of standalone MD, outperforming all other baselines.
In real world application we can expect a mixture of users/items with a broad range of number of ratings per user and item as well as a constant growth of user and item base. While both CF and MD alone show clear weaknesses in some situations, our hybrid approach is superior or at least comparable over all possible cases. We therefore expect a considerable gain in overall performance.
Note that the MovieLens dataset provides little information about users and especially about items. We anticipate that additional and more meaningful metadata will increase the performance of our MD algorithm and therefore the complete framework too. 5
In this work we proposed a hybrid training framework incorporating two RS algorithms, one implementing Collaborative Filtering and one relying on metadata of users and items. Through crosstraining we were able to combine the advantages of the individual approaches. Our algorithm is able to predict accurate ratings in situations where users/items have few as well as many ratings. We are able to tackle the cold-start problem while also improving performance in the case of many ratings being available.
Possible future work includes testing our algorithm on diferent datasets, including e-commerce data from the Mercateo platform. As many components of our framework are interchangeable, we will evaluate diferent sampling functions as well as other algorithms for CF and MD. In addition we will take steps preparing deployment in online mode, including more evaluation using ranking-based metrics.
This work has been supported by the European Regional Development Fund and the Free State of Saxony via project no. - VANDA.