Motivated by the potential use of collaborative tagging systems to develop new recommender systems, we have implemented and compared three variants of user-based collaborative filtering algorithms to provide recommendations of articles on CiteULike. On our first approach, Classic Collaborative filtering (CCF), we use Pearson correlation to calculate similarity between users and a classic adjusted ratings formula to rank the recommendations. Our second approach, Neighbor-weighted Collaborative Filtering (NwCF), incorporates the amount of raters in the ranking formula of the recommendations. A modified version of the Okapi BM25 IR model over users' tags is implemented on our third approach to form the user neighborhood. Our results suggest that incorporating the number of raters into the algorithms leads to an improvement of precision, and they also support that tags can be considered as an alternative to Pearson correlation to calculate the similarity between users and their neighbors in a collaborative tagging system.
The new generation of collaborative tagging systems such as
Delicious or CiteULike presented a new challenge to researchers
and practitioners in the area of recommender systems. While both
content-based [
Web 3.0: Merging Semantic Web and Social Web at
HyperText ’09 June 29th, 2009, Torino, Italy.
different recommendation approaches. First of all,
usercontributed content is more diverse in its nature and quality than
centrally created and structured content of traditional repositories.
Second, traditional 5-10 point ratings are typically not available –
only the fact that an item was contributed or bookmarked by the
user is present in the system. At the same time, the loss of quality
control and fine-grained ratings in collaborative tagging systems
is compensated by the presence of tags and (in most systems)
explicit connections between users. It looks evident that
recommendation approaches for collaborative tagging systems
should capitalize on the success of classic recommender system,
while trying to harness the new power provided by tags and social
links. However, there is no shared understanding of how these
features have to be taken into account to improve the quality of
personalization. A few pioneer projects explored different ways to
integrate social links or social tags into collaborative
recommendation [
The rest of the paper is addressed as follows. Section 2 describes the characteristics of the data and how it was collected. Section 3 describes the three recommender approaches developed: Classic Collaborative Filtering (CCF), Neighbor-weighted Collaborative Filtering (NwCF) and BM25-based similarity (BM25). In Section 4 we describe the study conducted and present the results. Section 5 introduces relevant related work, in Section 6 we address the discussion and in Section 7 we summarize conclusions and future work.
We performed our study based on data that we crawled from CiteULike1. The daily datasets provided by CiteULike lack a lot of relevant information necessary to develop our algorithms, as the title and the authors of each article.
We selected a group of users to be our center users, i.e., those who would receive the recommendations. For each one of these center users, we crawled her posted articles (id, title, authors, post timestamp, and tags associated), the neighborhood of users who posted her same articles, and the neighborhood of users who share her same tags. To avoid limiting the neighborhood due to tag variations as hyphens, underscores and plurals, we enhanced the spreading of tags by adding stemmed tags using Krovetz algorithm, and modified tags changing hyphens and underscores to eventually be added to the set of tags to be crawled. The details of the final dataset are described in Table 1. We chose 7 center users and we crawled all their articles and tags. We chose 100 neighbors for each center user, selecting those neighbors with more shared tags in amount and frequency. There was an overlap between these neighbors, so we finally crawled 358 users, including center users and neighbors. For each of these neighbors we also crawled all their articles and tags. In Table 1, annotations correspond to tuples of the style {user, article, tag}
Table 1. Description of the dataset
Item users articles
tags annotations # of unique instances
358 186,122 51,903 902,711
To create user-based recommendations using collaborative filtering, two processes are necessary. The first one is finding the neighborhood of the center user, i.e., her most similar users. Once the most similar users are identified, the second process is to rank the articles to be recommended. These articles will be taken from the set of articles which the neighbors have rated as their favorites, yet discounting those articles that the center user already has posted.
We implemented three user-based collaborative filtering approaches: Classic Collaborative Filtering (CCF), Neighborweighted Collaborative Filtering (NwCF) and BM25-based similarity (BM25).
This approach is described in detail in [
∑i⊂CRu,n (rui − ru )(rni − rn )
∑i⊂CRu,n (rui − ru )2
∑i⊂CRu,n (rni − rn )2
In the formula, r stands for rating, u denotes the center user and n
a neighbor. CRu,n denotes the set of co-rated items between u and
n. After performing this calculation, we select the top ten most
similar users. Next, we rank the articles of these users to
recommend to the center user, using the formula of predicted
rating for user u with average adjusts described in [
This method is an enhancement of our CCF implementation. The neighborhood of ten users is obtained in exactly the same way, using the Pearson correlation. However, we have incorporated the number of raters in the calculation of the ranking of the articles. We do it due to a large amount of the articles have been rated by only one or at most two users. In this way, we push up in the recommendation list those articles rated by a larger number of neighbors. The new predicted rating is given by pred ′(u, i) = log10 (1 + nbr(i)) ⋅ pred (u, i)
BM25, also known as Okapi BM25, is a non-binary probabilistic
model used in information retrieval [
(k1 + 1)tftd
k1 ((1 − b) + b × (Ld / Lave )) + tftd
⋅
(k3 + 1)tftq (4)
k3 + tftq
In our model RSVd represents the similarity score between the
center user (the terms of the query q) and one neighbor (the terms
of the document d). This similarity is calculated as a sum over
every tag t posted by the center user. The neighbor d is
represented as her set of tags with their respective frequencies. Ld
is the document length, in our case is the sum of the frequencies
of each tag of the neighbor d. Lave is the average of the Ld of
every neighbor. The term tftd is the frequency of the tag t into the
set of tags of the neighbor d. tftq represents the frequency of the
tag t into the query, i.e., the set of tags of the center user. Finally,
k1, k3 and b are parameters that we have been set in 1.2, 1.2 and
0.8 respectively, values slightly different from those suggested by
default in [
After calculating the similarity between the center user and each neighbor, we choose the top N similar neighbors, and then we calculate the ranking of the recommended articles using the formula (3).
To perform our study, we selected seven active CiteULike users which had posted at least 50 articles each. Four of the subjects are part of the Personalized Adaptive Web Systems (PAWS) lab of the School of Information Sciences at the University of Pittsburgh. Three additional subjects were selected randomly from a list of active CiteULike users.
For each subject we generated 4 sets with 10 ranked articles each
one. The first three lists were generated using the methods CCF,
NwCF and BM25, considering 10 neighbors for each center user.
The fourth list was generated using BM25, yet considering 20
neighbors. To avoid pitfalls in the evaluation [
Another aspect considered to control the evaluation was to provide the URL on CiteULike of each article. We requested each subject to evaluate the articles based on that information or looking for the abstract on the internet, but don’t going further than the abstract.
For each subject, we calculated normalized Discounted
Cumulative Gain (nDCG) [
Figure 1 (a) shows us smooth results on different subjects and not so different results on the values of nDCG between different algorithms. However, if we compare them further, we can see that CCF performed the worst and is not so clear which one, BM25_10, BM25_20 or NwCF are significantly the best. This result suggests us that the ranking order of the recommendations, in general, is very close to the optimal one, where the most relevant articles are at the top and the less ones at the bottom. On the other hand, CCF shows in general a better level of novelty. The results on Precision_2 and Precision_2_1 do not let us infer easily some ideas, but we can see some trends. In general, CCF has the worst results, suggesting that including the amount of raters in the ranking formula is an important factor to consider in the success of these recommendations. In addition, the dissimilar results of BM25 using 10 and 20 neighbors, suggests that we should have taken a threshold to select the size of the neighborhood instead of choosing a fixed number such as 10 or 20. For example, CiteULike shows a neighborhood for each user, including just those who share at least the median number of articles of the center user.
A few pioneer projects explored different ways to integrate social
links or social tags. In [
In [
During the development of our approaches, we were stack for a while on CCF and NwCF for the low quality of the preliminary recommendations. We were using the ratings given by the users to obtain their neighborhood, which are given by 5-star scale and an “I've already read it” description. Since many users post articles without taking care of the ratings (by default it is 2 stars), and their evaluation criteria can vary a lot among different users, we decided to change the scale for a 3-point one. Afterwards, the results showed a significant improvement. We suggest paying attention to the rating scale used in recommender algorithms for social bookmarking systems, in order to diminish the impact of noise and users’ criteria.
We consider that the inclusion of the amount of raters in the ranking formula is an important contribution. The Figure 1 shows clearly that both nDCG and Precision metrics had better results for NwCF than for CCF. This result supports our claim that the “social knowledge” provided by the amount of raters helps to decrease the uncertainty implicit on items with too few ratings. However, this approach should be considered carefully depending on the user information need. CCF shows, in general, the best novelty values among the subjects, but this idea should be tested with more users to be claimed as true.
Regarding BM25-based similarity, in most cases it performs better than CCF, but with no predictable results between using 10 or 20 neighbors, which implies that a threshold based on each user characteristics should result better than a fixed number of neighbors. uncertainty produced by items with too few ratings. Third, a tagbased approach to obtain the neighborhood of a user on social tagging systems can be a suitable alternative to classical Pearson correlation. Our survey to seven users was a preliminary study and on eventual investigations we will consider more subjects to support our findings.
For our future research, we have already discussed two ideas. Firstly, we want to incorporate tags on the ranking model. On this study we used tags only to obtain the neighborhood, i.e., to perform the user-similarity calculations. We believe extending the use of tags can improve the results of precision of our BM25 approach. Secondly, we will cluster the users’ tags. Users can have more than one interest of research, which is easy to observe while examining their tags. We will implement clustering algorithms to identify the different interests of the users and we expect to provide more topic-oriented recommendations.
This material is based upon work supported by the National Science Foundation under Grant No. 0840597.