Collaborative filtering recommender systems (CF-RSs) employ user-item feedback, e.g., ratings, purchases, or reviews, to harmonize similarities among customers and produce personalized lists of products. Being based on the benevolence of other customers, CF-RSs are vulnerable to Shilling Attacks, i.e., fake profiles injected on the platform by adversaries to hack the recommendation outcomes toward a corrupt behavior. While mainly works on shilling attacks have been conducted to propose novel methods, compare recommendation models and outputs with and without defenses, we have found a lack of study on the impact of dataset properties on the CF-RSs robustness. In this work, we present a regression model to test whether dataset characteristics can impact the robustness of CF-RSs under shilling attacks to interpret their eficacy depending on these characteristics. Obtained results can help the system designer understand the cause of CF-RSs performance variations in attack scenarios.
Collaborative filtering recommender systems (CF-RSs) are a core service in online platforms in
increasing trafic and promoting sales [
Three main directions have been explored on shilling attacks: attack designs, detection
algorithms, and defense strategies. The main shilling attack strategies are random, average,
popular, bandwagon, and love-hate [
While previous works have been orientated to “win-lose” scenarios, i.e., find an answer to questions such as “Which attack models impact more the performance of specific recommendation models? ”, “Which amount of knowledge on a specific recommendation-model is required for specific attack A to influence recommendation algorithm B? ”. No efort has been made to provide an interpretation on which dataset features can impact the efectiveness of attacks. Indeed, while it is well-known that CF-RSs are afected by the sparsity of the dataset (e.g., a denser dataset can make easier the recommendation task [17]), there are no claims in the case of shilling attacks.
In this works, we focus on a novel research question “Given popular shilling attack types and CF models already recognized by the community, which dataset characteristics can explain an observed change in the performance of recommendation?” To answer this question, we propose a regression-based model to analyze the efects of dataset characteristics on the robustness of CF-RSs, and, via a large-scale experiment on three domains, we evaluate how three classes of data characteristics —rating structure, rating value, and rating distribution— may influence the robustness of CF-RSs. This work is an extended abstract of [18] published at SIGIR 2020.
Let and denote a set of users and items in a system, and ∈ ℝ | |×| | as the complete user-item rating matrix, where each entry ∈ ℝ represents a rating assigned by user ∈ to item ∈ if it is a recorded interaction (we use to indicate the set of recorded interactions), a shilling attack consists in adding novel users as composed by the selected item set, the ifller set, the unrated-item set, and the target item set. contains items identified by the attacker to exploit the owned knowledge to maximize the efectiveness of the attack, holds randomly selected items for which rating scores are assigned to make the attack imperceptible. includes items without ratings in the fake user profile, and is the item is to push or nuke. The composition varies based on attack strategies. We study: Random [12], Love-Hate [19], Bandwagon [20], Popular [21], Average [12], and Perfect Knowledge [22]. To study the impact of characteristics on the eficacy of this class of attacks, we use an explanatory model defined as follows: Definition 1 (Framework). Let bet the set of datasets, let be the number of data characteristic factors, let X be the matrix containing the independent variables values (data characteristic values specified below), then the regression model is built using the formulation y = + 0 + X (1) where 0 represents the expected value of y (the attack performance metric under analysis), = [ 1, 2, ..., ] is the vector of the regression coeficient associated with the IVs, and the error. Independent Variables (IV) We explore three class of independent variables on the (i) structure (i.e., ,ℎ , and ), (ii) rating frequency (i.e., and ), and (iii) rating values ( ) of the user-item rating matrix. F big values might imply a higher chance of finding similar neighbor users or items. Therefore, as both attack and recommendation models rely on identifying like-minded users (neighbor users) or similarly rated items (neighbor items), we deem to be an impactful characteristic on evaluating the performance of shilling attacks. ℎ can impact the efectiveness of shilling profile injection attacks. For example, in domains where | | « | | ) there are more candidate neighbor users than candidate neighbor items for memory-based CF models. This situation might work to the advantage of user-based CF than item-based CF. Moreover, under a similar number of ratings, changing the shape implies changing the average number of ratings per item | | ÷ | | . We conjecture that this circumstance may impact the robustness of CF, since the construction of is mainly based on altering the popularity of targeted items. is a well-recognized issue in the community of RS and = 1 − . Sparser data means that the fraction of unrated items significantly exceeds the fraction of rated ones [ 23]. It can harm the performance of CF, reducing, for instance, the chance of discovering neighbors in memory-based CF, building accurate model-based CF [24]. In [25], we have already identified a potential impact of dataset density on the efectiveness of shilling attacks. and measure the concentration of items, or users’, ratings and use them to capture the rating frequency distribution. The equal popularity (e.g., all users give the same number of ratings) is represented with the value of the Gini coeficients to 0, while the total inequality (e.g., only one user has given all ratings) is represented with the value to 1. Finally, we study motivated by the connection between high rating variance and recommendation performance claimed in Herlocker et al. [26] and the linear and negative impact on the accuracy shown in [17].
Dependent Variables The dependent variable (DV) used to study the efectiveness of the attack on RS is the Incremental Overall Hit Ratio (Δ @ ). This is a stability metric that measures if the recommendation model recommends diferent products due to the attack irrespective of their actual rating value [22]. The metric has been proposed by Aggarwal [27]
We study three datasets: ML-20M [28] having movies ratings ( = 138, 493 , = 26, 744 , = 20, 000, 263, = 0.0054 ), Yelp [29] containing users’ reviews on businesses ( = 25, 677 , = 25, 778 , = 705, 994 , = 0.0010 ), and LFM-1b [30] presenting user-artist play counts( = 120, 175 , = 521, 232 , = 25, 285, 767 , = 0.0004 ). We use three CF-RSs available in [31]: User-kNN [32], Item-kNN [32], and SVD [33]. Additional reproducibility details are available in the original work [18]. Table 1 presents a snapshot of the full results for answering two research questions presented below.
RQ1. Is there an underlying relationship between the studied IVs and the DV ? The results obtained for the adjusted coeficient of determination ( . 2) show that the six dataset characteristics can explain more than 60% of the variation in Δ @ irrespective of the attack type, model, and dataset, providing empirical evidence supporting the hypothesis that the six IVs can explain a large part of the DV. The explanatory power is highest for the model-based SVD approach (when comparing the global behavior of each CF model). However, not a similar observation could be made on attacks.
RQ2. How do IVs impact the DV in terms of the significance and directionality? The significance of the computed regression coeficients for the IVs tends to vary for each IV or group of IVs. The results show that the regression coeficients computed for , ℎ , and are statistically significant. This shows enough statistical evidence to support the hypothesis that structural characteristics can explain DV variations( < 0.05, 0.01, 0.001 ). However, results for the other IVs vary depending on <attack, CF-model, dataset> triplet, or they can be insignificant as in the case of . For instance, the coeficients for Gini indices (i.e., and ) are most significant for shilling attacks against SVD, particularly for samples drawn from the Yelp and LFM-1b datasets. The coeficients for are insignificant (p-value > 0.05) in all experimented cases, except for two cases <Random/Average attack, SVD, Yelp>, implying that this dataset characteristic, which deals directly with rating values, plays a less significant role on the impact of the attack. Investigating the directionality of the coeficients, Table 1 shows that the has a negative efect on Δ @ across majority of the cases in <attacks, CF-model, dataset>. This result is consistent with RSs findings that increasing the density is suitable for the performance of CF-RSs [34, 17]. An explanation is that: if we fix the number of users and items and increase the number of genuine ratings, the accuracy of similarities is improved by using more genuine ratings. As these similarities are generally vulnerable to the insertion of fake profiles, adding more genuine feedbacks can help to decrease the impact of attacks. Additionally, we can note that has a negative impact on Δ @ in neighborhood models, which means that increasing the space size makes neighborhood models less vulnerable to attacks. Finally, and on the contrary, ℎ presents a consistently positive influence on the eficacy of the attacks. We explain it by considering the following example: increasing ℎ leads to an increased number of users with respect to items (i.e., decreasing items). In this way, it could be easier to push the target item to higher positions inside the recommendation list (i.e., fewer items have contributed).
We have provided statistical evidence to accept the hypothesis that the chosen properties account for a considerable portion of variations in attack performance. In particular, structural properties (i.e., size, shape, and density) have a significant impact on the model, distributional (i.e., Gini index) have a higher impact on memory-based models, and standard deviation does not show an efect. Novel characteristics, attacks, and models are possible extensions.
We acknowledge support of PON ARS01_00876 BIO-D, Casa delle Tecnologie Emergenti della Città di Matera, PON ARS01_00821 FLET4.0, PIA Servizi Locali 2.0, H2020 Passapartout - Grant n. 101016956, and PIA ERP4.0. [12] S. K. Lam, J. Riedl, Shilling recommender systems for fun and profit, in: WWW, ACM, 2004, pp. 393–402. [13] R. D. Burke, B. Mobasher, R. Bhaumik, C. Williams, Segment-based injection attacks against collaborative filtering recommender systems, in: ICDM, IEEE Computer Society, 2005, pp. 577–580. [14] W. Zhou, J. Wen, Q. Xiong, M. Gao, J. Zeng, SVM-TIA a shilling attack detection method based on SVM and target item analysis in recommender systems, Neurocomputing 210 (2016) 197–205. [15] M. Aktukmak, Y. Yilmaz, I. Uysal, Quick and accurate attack detection in recommender systems through user attributes, in: RecSys, ACM, 2019, pp. 348–352. [16] F. Zhang, Y. Lu, J. Chen, S. Liu, Z. Ling, Robust collaborative filtering based on non-negative matrix factorization and r1-norm, Knowl.-Based Syst. 118 (2017) 177–190. [17] G. Adomavicius, J. Zhang, Impact of data characteristics on recommender systems performance, ACM Trans. Management Inf. Syst. 3 (2012) 3:1–3:17. [18] Y. Deldjoo, T. D. Noia, E. D. Sciascio, F. A. Merra, How dataset characteristics afect the robustness of collaborative recommendation models, in: SIGIR, ACM, 2020, pp. 951–960. [19] B. Mobasher, R. Burke, R. Bhaumik, C. Williams, Toward trustworthy recommender systems: An analysis of attack models and algorithm robustness, ACM Transactions on Internet Technology (TOIT) 7 (2007). [20] M. P. O’Mahony, N. J. Hurley, G. C. M. Silvestre, Recommender systems: Attack types and strategies, in: AAAI, 2005, pp. 334–339. [21] M. P. O’Mahony, N. J. Hurley, G. C. Silvestre, An evaluation of the performance of collaborative filtering, in: 14th Irish Artificial Intelligence and Cognitive Science (AICS 2003) Conference, Citeseer, 2003. [22] M. P. O’Mahony, N. J. Hurley, N. Kushmerick, G. C. M. Silvestre, Collaborative recommendation: A robustness analysis, ACM Trans. Internet Techn. 4 (2004) 344–377. [23] Y. Moshfeghi, B. Piwowarski, J. M. Jose, Handling data sparsity in collaborative filtering using emotion and semantic based features, in: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, ACM, 2011, pp. 625–634. [24] P. Cremonesi, A. Tripodi, R. Turrin, Cross-domain recommender systems, in: ICDM
Workshops, IEEE Computer Society, 2011, pp. 496–503. [25] Y. Deldjoo, T. D. Noia, F. A. Merra, Assessing the impact of a user-item collaborative attack on class of users, in: ImpactRS@RecSys, volume 2462, CEUR-WS.org, 2019. [26] J. L. Herlocker, J. A. Konstan, J. Riedl, Explaining collaborative filtering recommendations, in: CSCW 2000, Philadelphia, PA, USA, December 2-6, 2000, 2000, pp. 241–250. URL: https://doi.org/10.1145/358916.358995. doi:1 0 . 1 1 4 5 / 3 5 8 9 1 6 . 3 5 8 9 9 5 . [27] C. C. Aggarwal, Recommender Systems - The Textbook, Springer, 2016. URL: https://doi.
org/10.1007/978-3-319-29659-3. doi:1 0 . 1 0 0 7 / 9 7 8 - 3 - 3 1 9 - 2 9 6 5 9 - 3 . [28] F. M. Harper, J. A. Konstan, The movielens datasets: History and context, TiiS 5 (2016) 19:1–19:19. [29] X. He, Z. He, X. Du, T. Chua, Adversarial personalized ranking for recommendation, in:
SIGIR, ACM, 2018, pp. 355–364. [30] M. Schedl, The lfm-1b dataset for music retrieval and recommendation, in: ICMR, ACM, 2016, pp. 103–110. [31] N. Hug, Surprise, a Python library for recommender systems, 2017. [32] Y. Koren, Factor in the neighbors: Scalable and accurate collaborative filtering, TKDD 4 (2010) 1:1–1:24. [33] Y. Koren, R. M. Bell, C. Volinsky, Matrix factorization techniques for recommender systems,
IEEE Computer 42 (2009) 30–37. [34] P. Cremonesi, Y. Koren, R. Turrin, Performance of recommender algorithms on top-n recommendation tasks, in: Proc. of the 2010 ACM Conference on Recommender Systems, RecSys 2010, 2010, pp. 39–46. URL: https://doi.org/10.1145/1864708.1864721. doi:1 0 . 1 1 4 5 / 1 8 6 4 7 0 8 . 1 8 6 4 7 2 1 .