Recommender systems play an essential role in our digital society as they suggest products to purchase, restaurants to visit, and even resources to support education. Recommender systems based on collaborative filtering are the most popular among the ones used in e-commerce platforms to improve user experience. Given the collaborative environment, these recommenders are more vulnerable to shilling attacks, i.e., malicious users creating fake profiles to provide fraudulent reviews, which are deliberately written to sound authentic and aim to manipulate the recommender system to promote or demote target products or simply to sabotage the system. Therefore, understanding the efects of shilling attacks and the robustness of recommender systems have gained massive attention. However, empirical analysis thus far has assessed the robustness of recommender systems via simulated attacks, and there is a lack of evidence on what is the impact of fraudulent reviews in a real-world setting. In this paper, we present the results of an extensive analysis conducted on multiple real-world datasets from diferent domains to quantify the efect of shilling attacks on recommender systems. We focus on the performance of various well-known collaborative filtering-based algorithms and their robustness to diferent types of users. Trends emerging from our analysis unveil that, in the presence of spammers, recommender systems are not uniformly robust for all types of benign users.
The efect of shilling attacks on recommender systems, where malicious users create fake
profiles so that they can then manipulate algorithms by providing fake reviews or ratings, have
been long studied [
In this paper, we present an analysis conducted to understand the influence of fraudulent reviews on the recommendation process in real-world scenarios. We do this through a study of known datasets with gold standards in diferent domains and several commonly-used recommendation algorithms. Specifically, we utilize data from two widely-used e-commerce platforms, Yelp! and Amazon. Among various recommendation algorithms, we consider collaborative ifltering-based approaches as they are the most eficient and popular recommenders in such platforms. Thus, we focused our exploration on the robustness of these algorithms to shilling attacks.
The main contribution of this paper is two-fold. First, we analyze the performance of widely used five collaborative filtering-based algorithms in the presence of spammers and compared them when spammers are removed. By doing so, we seek to answer whether shilling attacks afect the robustness of the considered recommender algorithms. Second, we investigate if there is a specific user group (non-mainstream users) that are afected more than others (mainstream users) by spammers.
Our results are validated by an empirical evaluation using classical measures for evaluating predictive and top-N recommendation strategies. We show that RMSE scores decrease and NDCG@5 scores increase when removing spammers in the majority of the considered algorithms and datasets. This serves as an indication that the performances of considered collaborativeifltering-based recommender algorithms are indeed afected by spam ratings/reviews. Further, a deep investigation to quantify the efects of spammers on recommendations received by certain groups of users lead us to conclude that, for the Yelp! datasets removing spammers improves the predictive ability (RMSE) of all the considered recommender algorithms regardless of the type of users, i.e., mainstream or non-mainstream. However, in the case of Amazon datasets, we observed a trend where removing spammers lessen the predictive ability for mainstream users based on RMSE whereas improves for non-mainstream users according to both RMSE and NDCG@5. Therefore, non-mainstream users whose rating behavior does not align with the majority of the users are the most afected ones by spam ratings for Amazon datasets. Overall, we observed that 25%-29% of benign users in Amazon datasets are users who would not be equally satisfied by recommenders afected by shilling attacks. Thus, recommender algorithms are not uniformly robust for all types of benign users in the presence of spammers ratings/reviews.
The rest of this paper is organized as follows. In Section 2, we summarize related work; we then outline the dataset, algorithms, and evaluation strategies used in our empirical analysis in Section 3. In Section 4 we report on our results and, finally, conclusions are drawn in Section 5.
Collaborative filtering-based recommender systems are widely used to provide recommendations
to users in opinion-based systems, yet they are vulnerable to shilling attacks [
Most recently, the concept of diferential privacy has been explored to make matrix
factorizationbased collaborative filtering recommender algorithms more robust [
Our work add to this body of knowledge by exploring the robustness of collaborative recommender systems to shilling attacks by using real-world data with spam reviews ground truth, as opposed to attack simulation and investigating if some users are more vulnerable than others.
As previously stated, our goal is to analyze commonly-used memory-based and model-based collaborative filtering-based recommendation algorithms’ robustness to shilling attacks using a number of datasets with ground truth on spam reviews. In the rest of this section, we describe the experimental protocol for our analysis.
For analysis purposes, we rely on four datasets (described below) produced based upon data from two well-known e-commerce platforms: Yelp! and Amazon.
Yelp! We consider Yelp! reviews from two domains: hotels (YH) and restaurants (YR) [
YH YR AB AH
Amazon We also consider Amazon reviews from two domains: beauty (AB) and health
(AH) [
We provide descriptive statistics for the four datasets in Table 1. It is important to note that rating distribution is not similar across the datasets. As illustrated in Figure 1, rating trends from AH are dissimilar to the other counterparts, with a vast number of users rating only 1 item. Moreover, the rating distribution of benign users vs. spammers on attacked products (i.e., products receiving at least one spam review) is captured in Figure 2. From this figure, it emerges that benign users and spammer counterparts exhibit similar rating behavior in YR, AB, and AH, whereas in the case of YH, spammers noticeably assign ratings of ‘1’ more often than benign counterparts; the opposite is true for ratings of ‘4’.
We focus our analysis on well-known and widely-used collaborative filtering-based
recommendation algorithms implemented using Lenskit for python [
(b) Yelp!-Restaurant
(c) Amazon-Beauty
(d) Amazon-Health
Item-item [
Alternating Least Squares (ALS) [
By following the classical evaluation framework for shilling attacks on recommender systems [
Metrics. For assessment, we turn to Root Mean Square Error (RMSE) and Normalized
Discounted Cumulative Gain (NDCG), which are classical measures for evaluating predictive and
top-N recommendations. We also consider measures explicitly defined to quantify the impact of
spammer attacks on recommenders: Prediction Shift (PS), which captures the average absolute
changes in predicted ratings for attacked items and Hit Ratio (HR), which considers if attacked
items are promoted to user top-n recommendations (cf. Burke et al. [
We first examined recommender performance by considering all users in the respective
datasets. We then segmented users into fairness categories as computed by the Fairness and
Goodness algorithm described in the next paragraph in order to allow for more in-depth
explorations. By segmenting users based on fairness scores, it is possible to identify mainstream
and non-mainstream users. The latter are those whose rating patterns do not align with the
majority, i.e., liking what most people dislike and vice versa [
Fairness and Goodness The Fairness and Goodness algorithm (F&G) [
() = 1
∑︁ () × (, )
|()| ()
(1)
(2)
where (, ) is the rating given by the user to the item , () is the set of ratings given
by user , () is the set of ratings received by item , and = 4 in this case which corresponds
to the maximum error in a five-star rating system. Thus, the goodness of an item is given by
the average of its rating, where each rating is weighted by the fairness of the rater, while the
fairness of a user considers how much the ratings a user gives are far from the goodness of the
items. The higher the fairness, the more trustworthy the user is. Fairness scores of the user lie
in the [
In this section, we present our experimental evaluation of five recommendation algorithms on four datasets of diferent domains. We discuss the efect of shilling attacks on recommendations ofered to users in real-world scenarios, as opposed to simulated attacks. Specifically, we aim to answer the following research questions: RQ1 Do spammer’s ratings impact recommendations? RQ2 Who is really afected by spammers? By investigating recommender algorithm performance in the presence of spammers as well as when spammers are removed, the first question enables us to gauge the shilling attacks’ efect on the robustness of the considered recommendation algorithms. For the second question, we used the fairness metric to determine mainstream and non-mainstream users and quantify the efect of spammers on recommendations received by non-mainstream users.
To answer RQ1, we consider the performance of the recommender algorithms yielded on four
diferent datasets, as reported in Table 2. It comes across from the reported scores that removing
spammers indeed leads to lower RMSE scores, i.e., better predictions. Previous works have
shown PS values ranging from 0.5 to 1.5 when shilling attacks are simulated [
Overall, we can say that, in theory, the performance of collaborative filtering-based recommender algorithms is afected by spammers’ ratings/reviews. This is particularly noticeable for predictive recommenders (i.e., all algorithms yielded significant diferences across the datasets). In practice, however, performance improvements are in their majority barely perceptible. This leads us to question whether algorithm robustness is reflected by average metrics like RMSE or NDCG. In the end, looking at recommender performance as a whole may not clearly quantify how much spammers are able to deceit recommenders and, more importantly, if there are specific user groups that are afected more than others. With this in mind, we conduct a more thorough analysis with the aim of understanding if the aforementioned diferences in performance are more pronounced among certain types of users (i.e., non-mainstream ones).
To better understand which users are really afected by spammers, we analyzed users based on their fairness: the ability of a user to rate a product according to what it deserves. It is worth noting, however, that the rating a product deserves often aligns with what the majority of benign users (mainstream users) think about that product, as mainstream users often outnumber non-mainstream and spam users. We investigate trends according to RMSE, NDCG@5, and hit ratio. As noted in the prior subsection, prediction shift values were small, so we excluded this metric from our analysis).
Figure 3 illustrates how the RMSE varies according to the fairness of benign users; for ease of readability, we highlight statistically significant diferences in performance when spammers are excluded in Table 3. We start by observing that, regardless of the algorithm for both Yelp! datasets and with just one exception (YR, ALS, and FunkSVD, (0.4 − 0.5]), removing spammers reduces the RMSE for all users. For the Amazon datasets, instead, when the user fairness is greater than 0.4, removing spammers increases the RMSE for all users for each algorithm. We posit these results could be due to the rating distributions of spammers vs. benign users across these two platforms. As previously shown in Figure 2, spammer and benign users are more similar in Amazon than Yelp!, with the majority of ratings being 4 and 5. Therefore, removing spam could cause the recommender to lose information from mainstream users. On the other end, when fairness is less than or equal to 0.4 among Amazon users, in most cases where the diference is statistically significant, i.e., 14 out of 19 cases, removing spammers YH YR AB AH with spammers (W) across diferent user fairness ranges (* means < 0.03 and ** means ≤ 0.01). Cases where removing spammers reduces the RMSE are shaded. enables algorithms to avoid noise signals and thus perform better for these users (lower RMSE). Note that there are more cases in AH than AB (4 out of 11 vs. 1 out of 8) where removing the spammers is not beneficial for non-mainstream users. This could be due to the fact that, as shown in Figure 1, AH data is more sparse than other datasets, making the process of generating recommendations more dificult for most of the users in such a setting, independently of the presence of spam.
When we look at trends for NDCG@5, Figure 4 and Table 4 show that the quality of the generated recommendations seldom improves on the Yelp! datasets for users having fairness YH YR AB
with spammers (W) across diferent user fairness ranges (* means < 0.03 and ** means ≤ 0.01). Cases where removing spammers increases the NDCG@5 are shaded. greater than 0.5, whereas for the Amazon datasets, the value of NDCG@5 is higher when spammers are removed in the majority of the cases (28 out of 47) and independently of the user type.
Overall, our analysis reveals that removing spammers helps in reducing the number of attacked items that hit the top-5 recommendations for all the users in all the datasets (see HR@5 analysis in Table 5).1 Moreover, removing spammers in Yelp! is beneficial for all the users when considering the predictive performance of algorithms (based on RMSE); for Amazon, top-N algorithms are better (according to NDCG@5) among mainstream users, who see more tailored 1For brevity, we exclude a figure akin to those complementing Tables 3 and 4. recommended items in their top-5 item list when spammers are removed from the system. Also, we see performance improvement in terms of both RMSE and NDCG@5 scores for Amazon non-mainstream users, i.e., the ones with low fairness, hence the ones whose ratings are very far from the ones of the majority of the users. Non-mainstream users afected by spammers represent 29% (resp. 25%) of benign users in AB (resp. AH). In a real-world scenario, these percentages would translate into hundreds of thousands of users who would not be equally satisfied by recommenders that are not robust to shilling attacks.
In this paper, we have taken a deeper look into how shilling attacks afect recommender systems in a real-world scenario. For this, we conducted an in-depth exploration of the performance of ifve well-known collaborative filtering algorithms on four diferent datasets.
We saw similar trends among the performance of predictive and top-N recommenders: users are exposed to better recommendations when spammers are excluded (RQ1). This highlights the importance of further research on spammer detection and robust recommender systems. At the same time, we question if the small diferences in performance (albeit statistically significant) would be evident to recommender systems’ users and whether metrics considered for assessment which aggregate performance for all users could obfuscate users who are more deeply afected by spammers. This leads us to explore diferences in performance between mainstream vs. non-mainstream users (RQ2). We saw that Amazon non-mainstream users are the ones most afected by spam ratings according to both RMSE and NDCG@5.
Based on the findings emerging from the analysis presented in this paper, it follows that future
work will be devoted to looking at other types of recommender algorithms, beyond collaborative
ifltering-based, to see if the trends we have observed in our analysis remain. Moreover, we plan
also to test the efectiveness of adversarial training for recommender systems [
This work has been supported by the National Science Foundation under Awards no. 1943370 and 1820685.