The success of a recommendation algorithm is typically measured by its ability to predict rating values of items. Although accuracy in rating value prediction is an important property of a recommendation algorithm there are other properties of recommendation algorithms which are important for user satisfaction. One such property is the diversity of recommendations. It has been recognized that being able to recommend a diverse set of items plays an important role in user satisfaction. One convenient approach for diversi cation is to use the rating patterns of items. However, in what sense the resulting lists will be diversi ed is not clear. In order to assess this we explore the relationship between rating similarity and content similarity of items. We discuss the experimental results and the possible implications of our ndings.
Recommender systems help users to pick items of interest
based on explicit or implicit information that users provide
to the system. One of the most successful and widely used
technique in recommender systems is called collaborative
ltering (CF) [
Copyright is held by the author/owner(s). Workshop on Recommendation
Utility Evaluation: Beyond RMSE (RUE 2012), held in conjunction with
ACM RecSys 2012. September 9, 2012, Dublin, Ireland.
they make in predicting the ratings of users for items.
Although accuracy of predictions is an important aspect of
recommender systems, it is not the only one. Recently,
increasing the diversity of recommendation lists have gained
attention among researchers in the eld [
D(R) =
1 N (N 1)
X
X i2R j2R;j6=i d(i; j); (1) where R is the recommendation list of a user and N = jRj. d(i; j) is the dissimilarity of items i and j which is de ned as one minus the similarity of items i and j.
We think that average dissimilarity is a reasonable way to measure the diversity of a list of items. However, the important part is how to de ne d(i; j), i.e., the dissimilarity of two items which is unspeci ed in equation (1). The problem is not to choose a similarity metric such as Pearson or cosine. The problem is whether we can use the rating patterns (vectors) of items in order to measure their similarity. And if we use these rating patterns, in what respect the recommendation lists will be diversi ed? For example, if it is a movie recommender system, will the recommendation lists contain more movies from di erent genres or will the content of the movies get diversi ed?
In order to answer these questions we will compare rating similarity with two types of content similarities which we will de ne below. We hope that the results we discuss will shed some light on these types of questions and stimulate discussion on diversi cation. 2.
In hybrid recommendations content information is used in
order to increase the accuracy of rating predictions especially
for items whose ratings are too sparse. For example [
Another paper which comes close to our concerns is [
In our experiments we use Movielens1 (1M) data set. In order to compare movies' rating patterns to their contents we rst need to generate movie content information. We use two sources of information to this end. One source of content information comes from Wikipedia articles corresponding to movies in the Movielens dataset. The other source of content information comes from genre information which are provided in the dataset. Details of content generation are given below. 3.1
The Movielens dataset contains 3883 distinct movies and 6040 users. Some of these movies are not rated by any user. Also some of the movies have no corresponding entries in Wikipedia. After discarding these movies we are able to fetch 3417 (approximately 88% of all movies) movie articles from Wikipedia.
In this work we only use the text of each Wikipedia article (we do not use link structure or category information of articles). The text of a Wikipedia article consists of parts such as \Plot", \Cast", and \Release". We do not include \References" and \See also" parts of the text since they may contain information which is unrelated to the content of the movies. After extracting the text of each document we apply some basic preprocessing steps such as stemming and stopwords removal. We use a vector space model to represent text documents. 3.2
As a second source of content we use the genre keywords (such as adventure, action, comedy, etc.) provided by the Movielens dataset. Each movie in the dataset is associated with one or more genre keywords. We de ne the genre similarity between two movies using the Jaccard metric given below:
J (i; j) = jGi \ Gjj jGi [ Gjj (2) where Gi and Gj are genre sets of items i and j.
In the rst set of experiments we try to understand the relation between movie rating patterns and content generated from the corresponding Wikipedia articles. We have 1http://www.grouplens.org/node/73 two matrices: one is the Movie-User matrix which holds the ratings of users on movies and the other is the MovieTFIDF matrix which holds the tf-idf weights for each document. For evaluation we use the following methodology. For each movie we nd the most similar 100 movies using the Movie-User matrix (rating neighborhood) and the most similar 100 movies using Movie-TFIDF matrix (content neighborhood). We then nd the number of common items in these two neighborhoods. It turns out that on average there are 14.74 common movies in the two neighborhoods. If we generate the neighborhoods randomly this value turns out to be around 2.80. Randomization tests show that this difference is signi cant (p < 0.01).
We run the same experiment with di erent neighborhood sizes (20 and 50) but the percentages of the number of common items in the rating and content neighborhoods turn out to be similar to the percentages we get when we use a neighborhood of size 100.
We also test whether there is a relationship between the number of ratings and the correspondence between rating and content similarity. To see this we nd the rating and content neighborhoods of those movies which have similar number of ratings. To do this we divide the movies into rating intervals according to the number of ratings they have: movies which have ratings between 1-100, between 101-200, and so on. If an interval has less than 20 movies, we merge it with the previous one in order to increase the signi cance of the results. Figure 1 shows the average number of common items in the rating and content neighborhood sets of movies as a function of rating intervals. Interestingly, Figure 1 shows a clear linear correlation, i.e., as the number of ratings increases the number of common items in the content and rating neighborhood of movies also increases. One possible explanation of this positive linear correlation might be this. Generally, there is a positive relationship between the number of ratings and the popularity of a movie. This means that popular movies receive ratings from many different people with di erent tastes. Hence the rating patterns of popular movies re ect a diverse set of characteristics. Wikipedia movie articles also have rich contents re ecting di erent characteristics of movies. This might explain why a movie's rating neighborhood approaches to its content neighborhood as the number of ratings increase.
In the next set of experiments our aim is to understand the relationship between movie rating patterns and movie genres provided in the Movielens dataset. Genre keywords provide limited information compared to Wikipedia articles. Because Wikipedia articles contain terms that give information not only about the genre of a movie but also about the director, starring, musical composition, etc.
In order to measure the relationship between movie rating patterns and genres we applied a similar methodology. For each movie m we nd the most similar 100 movies using the Movie-User matrix (that is the rating neighborhood) and nd the Jaccard similarity (as de ned in equation 2) between movie m and movies in its rating neighborhood. The average Jaccard similarity value turns out to be 0.43. If we generate the rating neighborhood randomly we nd a Jaccard value around 0.17. Randomization tests show that this di erence is signi cant (p < 0.01).
We also test whether there is a relationship between the number of ratings and genre similarity. Similar to the experiment we described above we divided the movies into rating intervals according to the number of ratings they have. Then for each movie m in a rating interval we calculate the Jaccard similarity value between the movie m and its rating neighborhood of 100 movies then calculate the averages per rating interval. Figure 2 shows these average values as a function of rating intervals. Here, we again have an interesting case. There is a negative linear correlation which means that the more a movie has ratings the more its rating similarity diverges from its genre similarity.
The reason underlying these results might be this. Movies which have limited number of ratings (unpopular movies) are generally watched by the fans of that genre. For example, a fan of sci- movies may also watch an unpopular scimovie. So, unpopular movies generally get ratings from the same set of users who are fans of that movie's genre. And this makes the rating vectors of those movies (same genre movies) similar to each other. On the other hand if a movie is popular than it gets ratings from a diverse set of users which causes their rating neighborhoods diverge from its genre.
We should note at the outset that the conclusions presented here are not conclusive. Di erent experiments on di erent datasets and with di erent item types need to be done in order to drive more rm conclusions. However, we hope that these experiments and results will stimulate discussion and further research.
In this work we examined the relationship between rating similarity and content similarity of movies in the Movielens dataset. We examined two kinds of content: one of them is the tf-idf weights of movie articles in Wikipedia and the other is the genre keywords of movies provided by the Movielens dataset.
We found that to a certain degree there is a similarity between rating similarity and Wikipedia content similarity and also between rating similarity and genre similarity. However, we leave open to discussion the magnitude of these similarities. We also found that as the number of ratings of a movie increases its rating similarity approaches to its Wikipedia content similarity whereas its rating similarity diverges away from its genre similarity.
According to these results if diversi cation is done based on the rating patterns of movies then the recommendation lists will likely be diversi ed with respect to the content of movies to some extent. So, if no content information is available or it is di cult to get it, it might be useful to use rating patterns to diversify the recommendation lists.
To this analysis we plan to add latent characteristics of
items generated by matrix factorization methods [