Collaborative Filtering (CF) models lie at the core of most recommendation systems due to their state-of-the-art accuracy. They are commonly adopted in e-commerce and online services for their impact on sales volume and/or diversity, and their impact on companies' outcome. However, CF models are only as good as the interaction data they work with. As these models rely on outside sources of information, counterfeit data such as user ratings or reviews can be injected by attackers to manipulate the underlying data and alter the impact of resulting recommendations, thus implementing a so-called shilling attack. While previous works have focused on evaluating shilling attack strategies from a global perspective paying particular attention to the efect of the size of attacks and attacker's knowledge, in this work we explore the efectiveness of shilling attacks under novel aspects. First, we investigate the efect of attack strategies crafted on a target user in order to push the recommendation of a low-ranking item to a higher position, referred to as user-item attack. Second, we evaluate the efectiveness of attacks in altering the impact of diferent CF models by contemplating the class of the target user, from the perspective of the richness of her profile (i.e., slightly-active v.s. highly-active user). Finally, similar to previous work we contemplate the size of attack (i.e., the amount of fake profiles injected) in examining their success. The results of experiments on two widely used datasets in business and movie domains, namely Yelp and MovieLens, suggest that highly-active and slightly-active users exhibit contrasting behaviors in datasets with diferent characteristics.
Collaborative filtering (CF) models are a crucial component in
many real-world recommendation services due to their
state-of-theart accuracy. Considering their widespread popularity and adoption
in the industry, the output of these models can impact many
decision qualities in diferent application scenarios [
In this direction, first works [
One common characteristic of the previous literature on shilling
attacks on CF-RS is their focus on assessing the global impact of
shilling attacks on diferent CF models by examining the success of
attacks from the perspective of attacker’s knowledge and the size of
attack (i.e. the number of shilling profiles) [
The application scenario for class-based study of attacks on RS may span in diferent domains. As an example, a restaurant owner may wish to diminish the trust on a target user of a competitor by pushing a low-ranked product for the specific user . The same argument can be made for new users. An attacker may be interested in pushing or nuking, particular products with the objective of modifying the impact of a recommender system in order to afect future interactions of the new user.
The leading reserach questions of this work are then: • RQ1: From a global perspective, what is the impact of useritem attack on classes of users such as slightly-active and highly-active users? – Could attacks be tailored to have a higher impact on a particular class of users? – Which factors play a role on the impact of such an attack? • RQ2: From a local perspective, how do CF recommendation model work diferently under user-item attacks by looking to user-classes?
The remainder of the paper is structured as follows. Section 2 presents the evaluation protocol and datasets description we used in our experimental evaluation. Section 3 reports on the results and their discussion. Section 4 concludes the paper and introduces future perspectives.
In this section, we discuss our evaluation protocol for a user-item attack modeling and the corresponding evaluation setup.
In order to test the efects of a user-item attack on attacked
user classes, an extensive set of experiments has been carried out
with respect to three dimensions: (i) the attack strategy (type and
quantity of injected profiles), (ii) core CF recommendation model
and (iii) the user classes. The experimental evaluation has been
executed on two well-known datasets, MovieLens-1M (ML-1M) and
Yelp (described in Section 2.2).
2.1.1 Atack Strategies. We have implemented two attack
strategies to craft shilling profiles (SP) in order to model diferent level
of attacker’s capability. Given a user profile P (u) = {ri1 , . . . , rin }
(consisting of a set of items rated by user u), we consider the items
in P (u) in the form of: selected items (IS ), filler items (IF ), target
item (IT ) previously identified in [
BPR-SLIM [
These CF models stand for state-of-the-art models for the item
recommendation task, each using a diferent prediction concept,
allowing us to study the impact of diferent attack strategies from
multiple viewpoints.
2.1.3 User Classes. Given that CF models only rely on user
preference scores (i.e., ratings) to compute recommendation, we
hypothesize that it is relevant to investigate the impact of diferent attack
strategies with respect to the victim user’s level of activity, i.e. the
richness of her profile, calculated on the basis of the number of
ratings available in her profile. To this aim, we define two classes
of users:
• Highly-active (HA) users are defined as users who have
a number of ratings greater than the second quartile of the
number of ratings for each user in the dataset.
• Slightly-active (SA) users are defined as users who have
a number of ratings lower than the second quartile.
2.1.4 Evaluation Metric. Several metrics have already been
proposed to evaluate malicious attacks. For example, [
Definition 1. Let u be the user under attack and i be the targeted item that the attacker wants to push/appear in the top-k recommendations of u. Let topuk be the top-k recommendations of u. Let ϕ(i, topuk ) be the function to evaluate the efectiveness on an attack on (u, i). If i is pushed in the top-k then ϕ(i, topuk ) = 1 (successful attack), otherwise ϕ(i, topuk ) = 0 (unsuccessful attack). Let S be the set of (u, i) user-item pairs under attack. H R@k is defined as the fraction of successful attacks on each (u, i) ∈ S.
Í(u,i)∈S ϕ(i, topuk ) |S | where |S | is the number of (u, i) pairs over which H R@k is measured.
We conducted experiments on two well-known datasets,
MovieLens 1M [
Yelp: This dataset contains ratings of users on businesses. We used
the pre-processed version of the dataset provided by [
In order to validate the empirical impact of the under study attack types on diferent classes of users , an extensive set of experiments has been carried out with respect to the dimensions introduced in Section 2.1. The final results are presented in Table 2 and discussed from the following viewpoints: • A global analysis of the impact of attacks on user classes (cf.
Section 3.1) • A fine-grained analysis of the impact of attacks on user classes by looking into the CF models and attack types. (cf.
Section 3.2) We present each of these analysis viewpoints in the following subsections.
The goal of this analysis is to answer the first research question
related to the global assessment on the efectiveness of user-item
attack with respect to the identified users classes. We use the term
global here, since in this analysis we would like to free our attention
from the impact of attacks on CF models, attack quality (type)
and/or quantity as they have been largely addressed in previous
works [
A general observation for the results in Table 2 is that larger-size
attacks reach higher level of efectiveness on both classes of users
(highly-active and slightly-active) in comparison with smaller-size
attacks. For example, on the Yelp dataset, the average HR@10 for
UNA attack on highly-active users (across CF models) is 0.256 for
small-size attacks, while it is 0.800 for large-size attacks, a diference
of approximately three times. The same pattern of results is obtained
in other experimental cases. These results are in line with those
presented in previous works [
Our objective here is to study the impact of diferent attack strategies on user classes. For this purpose, we define the variable r = H RH A and refer to it as user-class attack impact —i.e., the
H RSA impact of an attack on highly-active users in comparison with slightly-active users. Diferent values for r are interpreted as in the following: • r = 1: the attack has an equal impact on highly-active and slightly-active users. • r > 1: the attack has an unequal impact on highly-active w.r.t slightly-active users. The impact of attack on highly-active users is relatively higher in comparison with slightly-active users.3 • r < 1: the attack has an unequal impact on highly-active v.s. slightly-active users. The impact of attack on slightly-active users is relatively higher in comparison with highly-active users.
It is obvious that the larger r deviates from the center point 1, the larger is the attack success in diferentiating highly-active with respect to slightly-active users in one of the above-mentioned directions (r < 1 or r > 1). Before starting a deeper analysis of the results we highlight that the most interesting values are in the left portion of Table 2 (small-size attacks), because when the size of attack is larger the attack reaches the maximum efectiveness, H R = 1, independently of user classes.
By looking at the results for each attack size in Table 2, we can see that the average user-class impact r¯ has a value higher than 1 for the Yelp dataset (r¯ > 1), while a value lower than 1 for the ML-1M dataset (r¯ < 1). These results show that both attack types have an unequal impact on slightly-active vs highly-active users as r , 1. However, the class of users they have a larger impact on remains largely diferent and contrasting in the two datasets .
As an example, in Yelp and for UMA, one can note that for smallsize attack r¯ = 2.393 and for large-size attack r¯ = 1.832, while the corresponding values on ML-1M are r¯ = 0.658 and r¯ = 0.909, respectively. This means that the impact of attacks on user classes is higher on highly-active users on the Yelp dataset (r¯>1), diferently from ML-1M (r¯<1).
We conjecture that the above contrasting behaviors are directly linked with the characteristics of the datasets such as their sparsity. As shown in Table 1, Yelp dataset is approximately 40 times sparser than ML-1M and we consider this diference as the main/possible cause of the contrasting outcomes in tested datasets. We try to provide a possible explanation here. In the more sparse dataset (i.e., the Yelp dataset), users with a small number of ratings (slightlyactive users) are more immune to attacks because they have a smaller support size of the user profile (i.e., the user profile is not rich enough for the attacker to be able to mimic it in a crafted way). In contrast, highly-active users are more immune to attack in ML1M with higher density, because their recommendations rely on neighbors with (very) rich user profiles. Put it simply, the crafted 3This is equal to say, slightly-active users are relatively more immune to the attack w.r.t. highly-active users. attacks need to use a large number of profiles to be able to alter recommendation for the target user.
The insight on sparsity is an important indication that data characteristics are playing a role in the efectiveness of attacks and it motivates further research in this direction.
The goal of this analysis is to study how diferent CF models behave against the attacks: which ones have similar performance and which ones have a diferent performance. This study resembles previous work on shilling attacks on CF models. However, we take into account the impact of attack on user classes in this study as well.
Instead of individual CF models performances and attack types, we compute the pairwise Pearson correlation between each pair of analyzed CF models. Figure 1 indicates a strong correlation on HR@10 between BPR-SLIM and Item-kNN (ρ = 0.960 in Figure 1a and ρ = 0.993 in Figure 1b). We justify this value by the fact that both CF models exploit the item-item similarity computation. Looking at the correlation values for User-kNN in Figure 1, one can observe a slightly lower correlation in the case of slightly-activeusers with respect to other models. We think that this phenomenon comes from the fact that tested attack are based on user preferences which gain good efect also with small-size attacks. For instance, HR@10 for Yelp on slightly-active users (0.750 and 0.850) is higher than the mean values with other models for both attack (mean = 0.288 and 0.440). We can also observe an interesting behavior when we compare ρ of BPR-MF with BPR-SLIM and Item-kNN. Figure 1 (a) and (b) show that HR@10 on both classes of attacked users is highly correlated (ρ ≥ 0.840). Finally, results in Table 2 show that BPR-MF is the model that is less influenced by user-classes because the user-impact factor is close to 1 for each class of users and attacks.
This work investigates the efect of user-item attacks on classes of users. Particularly, we investigated the efectiveness of attacks from a global and local perspective by varying the quality and quantity of attacks, the target user class and the collaborative filtering recommendation model.
Experimental results on Yelp and MovieLens datasets indicate
that for Yelp dataset slightly-active users are more immune to
shilling attacks than highly-active users, a characteristic that is in
contrast with the results on MovieLens dataset where highly-active
users are more immune than slightly-active users. As datasets have
a very diferent sparsity (Yelp is approximately 40 time more sparse
than MovieLens) we will move our future works in analyzing the
efectiveness of dataset properties under diferent attack scenarios.
From a local perspective, we evidence that BPR-MF is less
influenced than other models when varying user-class and attack types.
On the other hand, BPR-SLIM and Item-kNN have shown similar
behavior related to the efect of attacks on user classes. In future,
we also plan to extend our study by considering more datasets from
diferent domains, exploring in an extensive way the influence of
dataset properties, such as sparsity, user and item skewness, rating
variance, on the efectiveness of diferent type of attacks. Also, it
is of our interest to consider the impact of various shilling attack
types on CF models using item content as side information [
These studies give important insights on the impact of shilling attacks on recommender systems and provide clues on how to reduce their efectiveness by working on datasets characteristics.