Recommender systems filter the items a user did not evaluate, in order to acquire knowledge on the those that might be suggested to her. To accomplish this objective, they employ the preferences the user expressed in forms of explicit ratings or of implicitly values collected through the browsing of the items. However, users have different rating behaviors (e.g., users might use just the ends of the rating scale, to expressed whether they loved or hated an item), while the system assumes that the users employ the whole scale. Over the last few years, Singular Value Decomposition (SV D) became the most popular and accurate form of recommendation, because of its capability of working with sparse data, exploiting latent features. This paper presents an approach that pre-filters the items a user evaluated and removes those she did not like. In other words, by analyzing a user's rating behavior and the rating scale she used, we capture and employ in the recommendation process only the items she really liked. Experimental results show that our form of filtering leads to more accurate recommendations.
Copyright is held by the author/owner(s). EICS’16, June 21-24, 2016, Bruxelles, Belgium.
A recommender system is designed to suggest items of
possible interest to the users [
At the moment, however, all the recommender systems
base their filtering on a unique scale of values. Therefore, if
a user is required to express a rating in a defined scale, the
system assumes that the rating behavior of the user covers
the whole scale. Instead, if the system implicitly collects the
data, a fixed cut-off value is chosen, in order to determine if
a user liked an item (e.g., Fastweb’s recommender system
collects a positive preference for a TV program if a user
watches it for at least 5 minutes [
It is widely-known that the recommendation form that generates the most accurate results is collaborative filtering and, more specifically, it is Koren’s implementation of
SV D + + [
The problem that might arise is that if users have different behaviors (both when rating the items and when browsing the Web), the system might consider as liked by a user an item with a high rating, but that actually represents the lowest rating she gave (the same problem holds in the opposite scenario, in which a user only gives low ratings and her favorite item might be misclassified if considering the system’s scale).
The intuition behind this paper is that, since SV D can detect latent spaces and work with sparse data, the algorithm might benefit from receiving less but very accurate information about what the users actually like. In this work, we first show that users have different ratings behaviors, then we propose an approach that calculates the weighted average of the user ratings and leaves in the user profile only the ratings greater or equal than this value, thus removing the other ones. By modeling the positive behavior of the users and understand what they actually like, our approach should lead to more accurate recommendations. Note that this study is based on explicitly given ratings to facilitate its validation with a public dataset, but this technique can be applied, as is, to implicitly-collected data (e.g., by removing all the items that have been browsed for a number of seconds lower than the user’s average). More formally, the problem statement is the following: Problem 1 We are given a set of users U = fu1; : : : ; uN g, a set of items I = fi1; : : : ; iM g, and a set R = [1; 5] of ratings used to express the user preferences. The set of all possible preferences expressed by the users is a ternary relation P U I R. We also denote as Iu = fi 2 Ij9(u; i; r) 2 P ^ u 2 U g the set of items in the profile of a user u. Let SV DIu denote the fact that the SVD algorithm is run with the set of preferences Iu; 8u 2 U . Our objective is to define a Weighted Cut-off Criterion (W CC) able to generate a set I^u, which considers the rating scale employed by the user and removes from the set of items positively evaluated by her (Iu) those in the lower part of her scale. The goal of this paper is to show that accuracy(SV DI^u ) > accuracy(SV DIu ).
The contributions of our work are reported in the following: • analysis of the user ratings of a real-world dataset, aimed to show the non-coincidence of the range of values adopted by the users to rate the evaluated items, with that defined by the recommender system; • formalization of a Weighted Cut-off Criterion (W CC) able to remove from the user ratings those below the weighted mean value of her preferences; • evaluation of the proposed criterion, by comparing the performance of a state-of-the-art recommendation approach, before and after the W CC process applied to the user ratings.
In the rest of this paper, we first show that users actually have different rating behaviors (Section “Analysis of the Users’ Rating Behavior"), continuing by defining our approach (Section “Approach"). Then we present the results of the performed experiments (Section “Evaluation"), the literature related with our study (Section “Background and Related Work"), concluding with some remarks (Section “Conclusions and Future Work").
In order to validate our intuition and understand if users actually have different rating behaviors or if they use the whole rating scale, in this section we are going to present the number of users who use a specific rating scale. The study has been performed on the Yahoo! Webscope
R41 dataset. It contains a large amount of data related to users preferences expressed by the Yahoo! Movies community that are rated on the base of two different scales, from 1 to 13 and from 1 to 5 (we have chosen to use the latter). The training data is composed by 7,642 users (jU j), 11,915 movies/items (jIj), and 211,231 ratings (jRj), and all users involved have rated at least 10 items and all items are rated by at least one user. The test data is composed by 2,309 users, 2,380 items, and 10,136 ratings. There are no test users/items that do not also appear in the training data. Each user in the training and test data is represented by a unique ID. Since the users in the dataset are 7; 642, only about half of them (52:51%, 4; 013 users) have given their opinion by using the whole rating scale, while the others have used a different range of values. Indeed, these users can be mostly classified in three groups: 1; 319 users (17:25%) that used the range 3 5 (i.e., by evaluating their worst experience with a minimum score of 3), 1; 315 users (17:20%) that used the range 2 5 (i.e., by evaluating their worst experience with a minimum score of 2), and 688 users (9:00%) that expressed their opinion in the range 4 5 (i.e., by keeping an high rating in all their evaluations).
These results clearly indicate that each user adopts personal criteria of evaluation. For this reason, an effective exploitation of her preferences should take into account this aspect.
1http://webscope.sandbox.yahoo.com
[
Range of V alues
Given the fact that users have different rating behaviors, in this section we present an algorithm that detects the rating scale of a user and removes from the items she evaluated those under her average rating. The algorithm performs two main steps: • Weighted Cut-off Criterion. Calculation of the average value of the ratings of each user (Weighted
Criterion (W CC), to keep only the items with a rating above the user’s average. • Item Recommendation. The state-of-the-art algorithm SV D is run with the items processed by the previous step.
The architecture of the proposed system is summarized in Figure 2. In the following, we will describe in detail how each step works.
In order to understand which item a user actually likes, it is first necessary to identify the average rating, among those User Profiles Apply WCC SVD Process SVD
Output she gave. This value, called Weighted Ratings Average (W RA), is calculated on the basis of the ratings of each user u 2 U , in the context of her profile i 2 Iu, as shown in Equation 1.
P r W RA(u) = i2Iu j Iu j Given the W RA of a user, we define the Weighted Cut-off Criterion (W CC) that allows us to filter the user ratings r 2 R in the profile Iu of a user u 2 U . We perform this operation for each item i 2 Iu, according to the criterion shown in Equation 2. (0; if r < W RA(u)
r; otherwise The output of this step is a set I^u, which contains the ratings processed through the WCC criterion. (1) (2) This filtering process is summarized in Algorithm 1. Algorithm 1 Ratings f iltering Input: Iu=User profile
^ Output: Iu = Filtered user profile 1: procedure FILTERUSERPROFILE(Iu) 2: WRA=GetWeightedRatingsAverage(Iu) 3: for each i in Iu do 4: r=GetItemRating(i) 5: if r W RA then 6: I^u (i; r) 7: end if 8: end for 9: Return I^u 10: end procedure The algorithm takes as input (step 1) the profile Iu of a user u 2 U , and provides as output (step 9) this user profile I^u filtered on the basis of the proposed W CC criterion. After the calculation of the Weighted Ratings Average (W RA) of the items in the user profile Iu, performed at the step 2, from the step 3 to step 8 we extract the rating r of each item i 2 Iu (step 4), by adding it to the set I^u, when the value of R is greater or equal the W RA value (from step 5 to step 6). The set I^u, that represents the user profile Iu filtered on the basis of the proposed criterion, is returned as output at the end of the process (step 9).
This step runs the state-of-the-art SV D algorithm, by using as input the set I^u, for each user u 2 U . In that way, the algorithm only processes the items for which a user expressed an interest above the average.
In this section, we first describe the environment and the parameters setup, then we present the strategy and the involved metric, concluding with the experimental results and their discussion. The dataset employed to validate our proposal is the same one previously presented, i.e., Yahoo! Webscope (R4).
The environment is based on the Java language, with the support of the Mahout framework2 to implement the state-of-the-art recommendation approach (i.e., SV D) and to perform the evaluation of the experimental results in terms of accuracy. The experimental framework was developed by using a machine with an Intel i7-4510U, quad core (2 GHz 4) and a Linux 64-bit Operating System (Debian Jessie) with 4 GBytes of RAM.
The optimal values of two of the three parameters needed to run the Mahout implementation of SV D (i.e., the regularization parameter lambda used to avoid overfitting and the number of training steps) have been chosen through a preliminary training (the selected values are, respectively, 0:05 and 10). The third parameter (i.e., the dimension of the feature space) was instead tested in a range from 2 to 20, during the set of experiments aimed to evaluate the accuracy of the SV D recommender approach, before and after the use of our W CC approach. This is useful to test the effectiveness of the proposed approach when the size of the latent space varies.
The accuracy of the performed recommendations was measured through the Root Mean Squared Error (RM SE). This metric considers the test set and the predicted ratings by comparing each rating rui, given by a user u for an item i and available in the test set, with the rating pui predicted by a recommender system. Its formalization is shown in the Equation 3, where n is the number of ratings available in the test set.
vuu Pn (rui RM SE = t i=0 n pui)2 (3)
We evaluate our proposal through a comparative analysis, by considering the recommendations generated by the SV D approach, before and after the proposed filtering process, based on the W CC criterion. The comparisons have been made by measuring the results accuracy through the well-known Root Mean Squared Error (RM SE) metric, described in the previous Section “Metric". In order to guarantee the repeatability of the performed experiments, according with the Mahout documentation we used in the Java code the instruction RandomUtils.useTestSeed(). The evaluation process has been performed by using the Mahout functionalities designed to perform this task (RecommenderEvaluator Java class).
We validate our proposal by running a set of experiments that measure the accuracy of the SV D recommendations, before and after the use of our W CC approach.
As shown in Figure 3, our approach gets better accuracy values along almost all the considered SV D feature space range. It means that a preliminary filtering of the user ratings leads toward a better performance in approaches of recommendation such as SV D, which are strongly based on this kind of information. We can observe how the RM SE values get worse when the latent space increases (SV DIu approach), while they remain stable in our case (SV DI^u approach), showing that the accuracy of the
1:2 E1:15 S M R 1:1 2 4 6 8 10 12 14 16 18 20
SV D F eature Space system worsens when increasing the latent space, if has been used non-filtered user ratings.
The results obtained in this study first showed the existence of the problem related with the different ways that the users adopt to evaluate the items, while the second illustrates how the proposed W CC approach is able to improve the performance of the state-of-the-art recommendation approach, SV D. Indeed, the results of the experiments show us that the range of values that the users adopt during their evaluations, are in the half of the cases different from that allowed by the system (Figure 1), and that a preliminary filtering of them by our Weighted Cut-off Criterion overcomes this problem, improving the accuracy of the recommendations (Figure 3).
In this section we briefly review some main concepts closely related with the proposed work.
User Profiling. In the e-commerce environment the
recommender systems play a determinant role, their first
implementations were based on the so-called Collaborative
Filtering approach [
Latent Factor Models. The type of data with which a
recommender system operates is typically a sparse matrix
where the rows represent the users, and the columns
represent the items. The entries of this matrix are the
interactions between users and items, in the form of ratings
or purchases. The aim of a recommender system is to infer,
for each user u, a ranked list of items, and in literature many
of them are focused on the rating prediction problem. The
most effective strategies in this field exploit the so-called
latent factor models, but especially, the matrix factorization
techniques [
User Ratings Reliability. The concept of bias, introduced
in a recommender system process as noise in user ratings,
is well known in literature since 1995, when it was cited in a
work aimed at discussing the concept of reliability of users
in terms of rating coherence [
The work presented in this paper wanted to highlight and face a problem that rises when the users assign a rating to the evaluated items, by adopting a range of values that could not cover the entire interval allowed by the system with which they interact. Through the first experiment we showed the real existence of this problem, which has been faced by introducing a novel cut-off criterion (W CC). This criterion is based on a weighted ratings average value (W RA), and its effectiveness to improve the accuracy of a recommender system at the state of the art, such a SV D, has been demonstrated in the second experiment. In summary, our contribution in this field is twofold: first, we are able to improve the performance of a recommender system based on the user ratings (almost all of the state-of-the-art approaches); second, we are also able to reduce the computational load of the recommendation process, thanks to the removal of certain items from the user profiles (i.e., those with a rating less than W RA value). It should also be noted how the proposed approach leads toward a considerable improvements of the accuracy of the recommendations, without the needing to adopt sophisticated mathematical criteria to preprocess the user ratings, with a great advantage in terms of computational complexity.
Future work will study the relations between the range of values adopted by the users to express their opinion in different domains, to model in a better way the preferences in environments that sell different types of items. (e.g., a site such as Amazon3, which sells different types of goods). Indeed, each type of item might be associated to a different rating behavior. This will allow us to generate more effective recommendations.
This work is partially funded by Regione Sardegna under project NOMAD (Next generation Open Mobile Apps Development), through PIA - Pacchetti Integrati di Agevolazione “Industria Artigianato e Servizi" (annualità 2013). 3http://www.amazon.com/ Plumbaum, Sahin Albayrak, and Christian Scheel. 2012. Estimating the magic barrier of recommender systems: a user study. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval. ACM, 1061–1062.