Recommender Systems (RS) algorithms are growing more and more complex to follow requirements from real-world applications. Nevertheless, the slight improvement they often bring may not compensate the considerable increase in algorithmic complexity and decrease in computational performance. Contrarily, context aspects such as social awareness are still not much explored. In view of that, this paper proposes insights on how to possibly achieve more e cient and accurate predictions for recommendations by exploring multiple dimensions of a RS architecture. A framework is designed, comprised of a Facebook application called MyPopCorn and some scenarios of user neighborhood RSs are proposed. The rst one investigates how to recommend movies based on a narrowed subset of collaborative data, extracted from the social connections of the active user. Secondly, connections between users enable a solution for the cold-start problem. Preferences from social connections are aggregated, producing a temporary pro le of the new user. Finally, a third dimension is explored regarding evaluation metrics. Results from traditional evaluation by o ine cross-validation are compared to measuring prediction accuracy of online feedback data. These insights propose how community-based RS designs might take advantage of social context features. Results show that all three proposed solutions perform better assuming some conditions. Social neighborhoods can often provide representative data for collaborative ltering user-neighborhood techniques, improving a lot the RS performance in terms of computational complexity metric without compromising prediction accuracy. Assuming a user has a dense social network, the cold-start problem can be easily tackled. Finally, rating prediction accuracy performs better when evaluated online than by o ine cross-validation.
H.4 [Information Systems Applications]: Miscellaneous; H.3.3 [Information Search and Retrieval]: CollaboraPermission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
Copyright is held by the author/owner(s). Workshop on Recommendation Utility Evaluation: Beyond RMSE (RUE 2012), held in conjunction with ACM RecSys 2012. September 9, 2012, Dublin, Ireland. .
Our generation faces several tough challenges within the
current peta-, exa- or even zettabyte information era. Every
day we deal with huge amounts of information whose
manipulation and storage struggles even on high-end computer
technologies. Shifting from the point of view of computer
capacity to an average single person, the problem gets even
worse due to human being limitations. Online services are
examples of big data resources with increasing importance in
our lives. About two years ago, Google's search engine used
to process approximately half of the entire written works of
mankind per day [
Movie domain is a great context where information
overload is a high potential pain point to be explored. Moreover,
Net ix movie streaming service is a good motivation for this
work due to two main reasons. Firstly, gures disclosed in
[
Rather than building upon complex RS methods, this paper investigates a social framework for developing state-ofthe-art RS. Aiming at current main challenges, this paper proposes contributions on how to tackle some of its most relevant issues based on possibilities enabled by social context information. The three explored RS challenges are: (i) performance issues related to scalability of recommender systems; (ii) lack of knowledge about new users, known as cold start problem; and (iii) de nition of good evaluation methods.
Some insights are discussed based on how social-graph data enable a good implementation of a user neighborhood RS algorithm, focusing not only on prediction accuracy but also on other metrics such as scalability, computational complexity and transparency. These insights lead to 3 hypotheses listed below: i. A user's social neighborhood is su ciently representative to provide e cient, in the sense of computational complexity, and e ective recommendations, in terms of prediction accuracy; ii. Social neighborhood connections can derive assumptions about new users taste, avoiding the cold-start problem; iii. Online evaluation of transparent recommendations should be a valid metric within social RSs.
In the introduction of the latest survey in RS eld, [
Scalability In real-world applications, the number of instances might often steeply increase in multiple dimensions such as number of users, items and, in turn, user-item preference signals. Despite being a good scenario for some RS algorithms to achieve better accuracy, bigger datasets may lead to a great increase in computational complexity.
[
Regarding model-based CF techniques, [
Data Sparsity It is among the main bottlenecks for RSs.
The lack of information is a big problem, especially during
rst interactions of a new user. This scenario is de ned as
the cold start or new user problem, which is traditionally
solved by requiring initial user information before any
recommendation is given. Nevertheless, this interaction is time
consuming, since the user has to look for a couple of items
to rate. To improve that, [
Besides requiring this rst interaction with the RS, one
could think of a temporary user pro le in order to enable
initial recommendations. [
Transparency Users eventually question themselves about
the reasoning behind a recommendation. They are more
inclined to accept and evaluate better once they understand
how an item has been suggested to him or her.
Nevertheless, it is not always possible to provide such a
transparent explanation. [
Evaluation One of the main modules of a RS design,
evaluation strategy is a critical and subjective aspect to be
shaped throughout the whole process of building and
maintaining a RS. Even though most papers adopt accuracy as
the most important metric, one should consider many other
evaluation criteria, as presented in [
In view of all issues previously listed and the fact some
state-of-the-art architectures might not be that attractive
for commercial purposes, this paper dives into a RS
design that is gaining special attention: Social RSs. Also
called community-based recommenders, the basic
architecture embeds context data into either collaborative ltering
or content-based algorithms, improving the RS performance.
According to [
Further than looking at social connections, the latter is
MyPopCorn
GroupLens
a trust-based RS that focuses on weighted relationships. A
clear comparison between social RS and trust-based RS is
de ned in [
As claimed in [15, pg 15], the context in which a RS is developed and its expected features determine the optimal algorithm to be adopted. Parameters such as movie domain, social community context, rating strategy and sparse data were de nitely crucial to come up with the nal architecture described hereby. A Facebook application called MyPopCorn1, the RS front-end, and a social based implementation of user neighborhood CF algorithm compose the current framework, to be presented in the two following sections. 3.1
The idea of building this movie recommender system and making it available on a social network is due to the fact social graph enables proposed recommendation experiments based on social neighborhoods. Moreover, the capability of recommending to an active user and receiving an online feedback on rating prediction accuracy on recommended items is decisive to benchmark the implemented algorithms.
MyPopCorn is a web movie recommender system. Some of its interfaces are composed as follows:
First screen presents a brief description of the main features before the user joins the application. After that, an active user can check statistics about top users and friends;
MyTaste is where a user can rate movies. Recommendationwise, this is one of the main interactions with the user, in which RS collects data;
My Friends' Taste presents a list of friends and their respective number of ratings. The more ratings each friend has, the bigger his or her basket gets. 3.1.1
The rst collaborative data with ratings over movies were taken from GroupLens 10M dataset. From that point, the database was increased with ratings from users of MyPopCorn. Information about users, friendships are also made persistent into the same database. The dataset used for the experiments is summarized in Table 1.
In a very short timeframe, the application was accepted by a good number of users. Almost 130 active users have been exploring the application during 2 months time. Figure 1 illustrates all users who contributed for the experiments carried out into this paper. The more movies a user rates, the bigger the node is represented in the social graph. The average degree of connections in this graph was 10.543.
In MyPopCorn, the user can choose a rating from 1 to 5 `stars'. Asymmetric labels were de ned for each of the 5 stars to achieve a more homogeneous judgment, namely Bad, Regular, Good, Great and Masterpiece. Test users reported good feedback on the proposed rating strategy claiming this discrete labeled design is certainly more intelligible, where users can have a hint of what each rating value may represent. While following such design, this research aims at reducing subjectivity that is intrinsic to rating process, the core interaction responsible for obtaining the main input of a Collaborative Filtering RS. This strategy also prevents the necessity of the RS to normalize user ratings. 3.1.3
Recommendations are generated from two implementations of user neighborhood recommenders, such as follows: Provided by a traditional user-based RS. The neighborhood calculated among all users in the database; Provided by a social-graph user-based RS. A social neighborhood is based on the set of active user friends, to be described in more details in the next section.
A shu ed list of recommendations generated by both RS implementations is presented to the user. Movie description and a continuous predicted value is presented. Therefore, recommendations are seen as a regression and not a classi cation problem within this framework. Finally, at the bottom of the frame one can see the explanation about each recommendation(see Figure 2). In the rst example on light blue background, a message informs the recommendation was \Based on all MyPopCorn database\. Alternatively, the second message informs that is was \Based on friends with closest taste\, followed by the list of users Friend X and Friend Y.
This system is designed to give the most transparent recommendations possible. In view of that, the reasoning behind the RS can be better understood by presenting the real number as predicted rating value. Furthermore, explaining the recommendation with a list of users will transform a formerly impersonal recommendation into a social passive interaction between friends. Due to privacy issues, presenting this list is only possible for the social neighborhood approach, where content sharing among users is agreed in advance. 3.2
The nal architecture of the social-graph recommender was developed on top of the user-based RS implementation provided in Mahout2. User neighborhood CF paradigm has close reasoning to social user behavior, being the most relevant criterion that in uenced this design choice. In possession of information about users taste, this user-centered method focus on comparing similarity among users. Furthermore, friendship data will be essential to enable modi cations on the original algorithm. Insights on how to pro t from social context information in di erent dimensions will be addressed below.
As the title suggests, solutions to the current RS challenges listed in Related Work are described in this section. Each of the following implemented scenarios tackle three main challenges previously mentioned, namely computational complexity issues of scalable user-neighborhood RSs; sparse data about new users, known as cold start problem; and de nition of optimal evaluation methods for transparent and non-transparent recommendations. 4.1
The idea of narrowing the dataset to a subset of users
aims to tackle scalability constraints and increase real-time
performance, two issues that are intrinsic to user-based RS
[
Experiments were performed in order to investigate the three insights proposed above. A standard user-based neighborhood RS setup is incrementally modi ed from the current insight until the third one. This scenario focus on predicting ratings contained in a training set comprised of 5% of all 14.367 ratings provided by MyPopCorn users. The reason for not adding any rating from GroupLens into the training set of the standard neighborhood is allow a fair comparison between both neighborhoods. By applying two strategies, namely Standard full neighborhood and hereby proposed Social one, some hypotheses are tested: (i) Real-time recommendation performance will become much more e cient while adopting social neighborhood; (ii) Rating prediction 2Apache Mahout machine learning library accuracy from social neighborhood recommendations will be as much precise as in the standard method.
For the proposed experiment methods, standard neighborhood RS performs around 70k calculations, the number of all users in the merged dataset. In the case of social neighborhood, the number of comparisons is relative to the degree of each node (user) in the social graph, which varies from 0 to 49 for MyPopCorn dataset with an average degree of 10.543. Concerning average runtime, whereas prediction process for one rating takes around 950.55 ms for standard neighborhood, after narrowing the search space to the set of social connections, it takes in average 69.975 ms, 92.63% lower. Regarding accuracy, Figure 3 presents prediction accuracy error for this new neighborhood compared to the standard implementation. Both implementations were compared by varying the size of the neighborhood k while experimenting two values of threshold t=1 and t=2. This threshold de nes the minimum number users in the neighborhood that rated a same candidate item. When t=2, the items rated by only one user in the neighborhood are not taken into account.
The minimum RMSE = 0.8385664 was obtained by Standard neighborhood (k =3,t=2). Besides that, Social (k =2,t=2) achieved RMSE = 1.018598. Surprisingly, rating prediction accuracy also improved. Except for values of k neighbors equal to 2 and 3, Social Neighborhood outperforms, in average, the standard method, con rming the rst hypothesis for this scenario. Besides that, the value of threshold t=2 performs better. The fact of accepting only items rated by at least two users might have increased the con dence on preference data, achieving better accuracy results. On the contrary, hypothesis 2 was surprisingly refuted. Instead of performing almost the same as in the original approach, Social Neighborhood can signi cantly outperform prediction accuracy for k > 3. While increasing the value of k, such social neighborhood enables a more accurate predictions and, probably, reaching higher serendipity.
Remark: This approach is not available for people with no or few friends, su ering from the cold start problem, to be solved next. 4.2
One of the main issues related to RS, the cold-start problem or new-user problem prohibit some active users to receive recommendations. In the dataset used for all experiments, 21 users out of 129 have rated less than 10 movies, while others more than a thousand. These users with few ratings are almost unable to receive any recommendation.
Instead of adopting the classic approaches such as
contentbased or presenting a list to be rated as from the rst user
interaction, this paper proposes a solution based on
socialgraph information. It is based strategy from group RS based
on aggregating user pro les. One could see this problem
following the quote \Tell me who your friends are and I will
tell you who you are\. This reasoning is also motivated by
the work carried out in [
Among some aggregation techniques mentioned in the
Related Work, Average without Misery is adopted, since it nds
a balance between the Least Misery and Average. It
preserves the main advantages of both aggregation strategies
originally applied to group RS and now re ected in the
aggregated virtual pro le to be considered by our single-user
RS. It follows the human-like reasoning in which a group of
people tend to select items that please, in average, most
persons involved. Moreover, it excludes items once rated below
a de ned threshold, as described by [
The social neighborhood method was adopted with parameters k =4 and t=1, so that the most number of predictions are enabled. The idea here is to investigate how many active users had the cold-start problem, meaning their neighborhoods were empty. While repeating the experiments from last section in 5% of MyPopCorn ratings dataset, around 103 users were in the testset. Nevertheless, RS could not estimate any rating for 13 users due to empty neighborhood issue. 6 users had no social connections, what can not be solved by the method proposed here. The remaining 7 users had their ratings predicted with accuracy error of RMSE = 1.69588.
One should raise the question that this is not much data, referring to the tiny set of 7 users. In view of that, another experiment has been run on 50% of ratings in MyPopCorn dataset. Ratings of 44 users experiencing the coldstart problem were hidden iteratively in order to be predicted by the RS. Foreach of the 44 users, the RS generated a virtual pro le based on aggregating all ratings from their friends, including those removed in order to arti cially cause the cold-start problem. Only 8 new users(18%) could not be helped by this method of aggregation due to the fact of having no social connections. Prediction accuracy error was RMSE = 1.37461.
Compared to the accuracy evaluated in the experiments of previous sections (RMSE = 1.173435 for k =4, t=1), this proposed solution to the cold-start problem has decreased performance in around 20%, considering the RMSE = 1.37461. In view of that, the proposed solution is considered to be a good alternative for social RSs. Besides not compromising the prediction accuracy signi cantly, this method should be considered in terms of how e cient the RS can deal with new users that are not interested in providing many ratings as from the rst interaction. Despite not being an objective metric, the ability of solving the cold-start should be incorporated into RS evaluation. 4.3
While the rst insight focuses on the two objective evaluation metrics, namely prediction accuracy and computational complexity, this insight focuses on transparency, a subjective metric, and evaluation methods. The most popular evaluation metric throughout RS state-of-the-art, prediction accuracy benchmark is often based on o ine cross-validation and error calculation over Root Mean Squared Error - RMSE. In view of that, this third and last section compares o ine and online methods of calculating estimation accuracy together with more transparent recommendations based on social explanation. One hypothesis is that this online method might make o ine approach suboptimal for the context of social recommenders. Instead of cross-validation, one should consider the social factor involved within online evaluation. Due to the strategy of recommending a list of movies whose predicted ratings might not be always high and to make it more transparent, the predicted value is presented to the active user. Assuming that not many people tend to converge with the RS prediction, this strategy will not bias the comparison. Actually, we believe there are people who also try to diverge from what has been predicted.
The current experiment intends to test the e ect of
explained recommendations, as previously described in [
Besides RMSE, metrics such as novelty or serendipity were taken into account while choosing higher values of k other than the ones that reached minimum accuracy, shown in Figure 3. Although the same number of recommendations with standard and social neighborhood were generated, active users gave more feedback on the social ones. 119 online feedbacks were provided, as presented in Table 2 in comparison with the traditional o ine method.
As Table 2 shows, Standard Neighborhood method achieved a prediction accuracy of 1.0646 and Social Neighborhood RS setup achieved better rating prediction accuracy of RMSE = 0.9952. Both of them presented an improvement when evaluated online other than o ine. The decrease in RMSE was of 14.16% and 6.64%.
Hypothesis was con rmed by the numbers shown in Table 2. Surprisingly, online evaluation accuracy with Standard Neighborhood improved better (14.16%) than 6.64% gain achieved by Social Neighborhood strategy. Finally, results have shown that, in average, RSs tend to present better accuracy results in online evaluations than o ine for both explained and non-explained recommendations.
This paper rst discussed the computational requirements intrinsic to user neighborhood RS, by nature a non-scalable algorithm. Based on the two most important evaluation metrics, state space reduction enabled a decrease of 92.63% in computational complexity, while not compromising accuracy. Instead, the latter also improved.
Social graph was essential to enable a solution to the coldstart problem. Tested with success in group RS, Average without Misery enabled creation of virtual pro les based on active users network. Results con rmed the proposed hypothesis, indicating this solution as a good alternative to this issue while presenting a decrease on prediction accuracy of only 20% by cross-validation.
Another important achievement was caused by transparent recommendations. Results from the third insight turn prediction accuracy by cross-validation an even more questionable benchmark method. Both neighborhood formation methods presented a considerable improvement of 6.64% and 14.12%. While choosing online evaluation methods, one could have better conclusions about the RS quality.