Adversarial Attacks against Visual Recommendation: an Investigation on the Influence of Items' Popularity

Abstract

Visually-aware recommender systems (VRSs) integrate products' image features with historical users' feedback to enhance recommendation performance. Such models have shown to be very efective in different domains, ranging from fashion, food, to point-of-interest. However, test-time adversarial attack strategies have recently unveiled severe security issues on these recommender models. Indeed, adversaries can harm the integrity of recommenders by uploading item images with human-imperceptible adversarial perturbations capable of pushing a target item into higher recommendation positions. Given the importance of items' popularity on the recommendation performance, in this work, we evaluate whether there is an influence of items' popularity on the attacks' efectiveness. To this end, we perform three state-of-the-art adversarial attacks against VBPR (a standard VRS) by varying the adversary knowledge (white- vs. black- box) and capability (the magnitude of the perturbation). The results obtained evaluating attacks on two real-world datasets shed light on the remarkable eficacy of the attacks against the least popular items' when planning novel defenses.

Keywords

• eol>Adversarial Machine Learning
• Visual Recommender Systems
• Collaborative Filtering

1. Introduction

has represented a turning-point in the RecSys community, few have already considered the collateral and negative impact of adversarial attacks against deep/convolutional neural networks (DNNs/CNNs) used in visually-aware recommendation9s].[To date, there exists a plethora of adversarial attack strategies in the computer vision domain whose purpose is to perturb images and mislead the classification performance imperceptibly. In this set, FGSM10][, PGD [ 11 ], and Carlini & Wagner12[] represent the milestones in adversarial machine learning (AML). In collaborative filtering recommendations, He et al1.3[] have proposed and demonstrated the eficacy of adversarial perturbation of MF model embeddings in corrupting the model performance. Then, they have designed an adversarial training method to robustify the model performance against the previously proposed perturbations. Their experimental flow has also been applied in [ 14 ]. Here, the authors have tested the first adversarial procedures (attacks/defenses) in a visually-aware recommendation model. Indeed, they have attacked, and later defended, VBPR against adversarial perturbations applied on the visual embeddings extracted from a pretrained CNN (i.e., ResNet50 [ 15 ]). While Tang et al.1[ 4 ] have worked on feature-level perturbations, later Di Noia et al1.6[], Anelli et al.1[ 7 ] have studied and designed the first set of targeted adversarial attack methods to be directly performed against input product images (and not the visual features) to increase the recommendation probability of low-recommended categories of products bypoisoning the training data with their adversarial samples.

More recently, Liu and Larson18[] and Cohen et al.1[ 9 ] have proposed novel adversarial attack procedures that perturbs product images to push/nuke an item during the inference/testing phase (i.e.,evasive attacks). Both works have releasebdlack-box and white-box adversarial methods where, in the first scenario, the adversary does not know the recommender model, while in the second, the attacker has complete access to the model, input, and output. However, both training and testing time attacks have been evaluated considering their eficacy on pushing the target/victim items into to p- recommendation lists or increasing the preference scores without taking into account the diferent levels of items’ popularity (i.e., the number of interactions recorded on each item in the training set). Indeed, considering the influence of diferent levels of item’s popularity on the recommendation performan2c0e, [ 21, 22, 23 ], we found a lack of investigation on their potential efects on the eficacy of adversarial attacks.

In this work, motivated by the previous observations, we explore the performance of VBPR [ 4 ], a pioneering VRS, under test-time attacks. In particular, we investigate bolatchkbox and white-box settings, and we split the target items into four groups based on their popularity to understand whether there could be a connection between attacks eficacy and the number of feedbacks received by the target item. Our contributions may be summarized as follows: • we provide an extensive evaluation of three state-of-the-art adversarial attacks against visual-based recommendations in multiple settings, varying the adversary knowledge (i.e., black- and white-box), the adversarial capability (i.e., the maximum variation of each image pixel, that is∈ {4, 8, 16} ; and evaluating their performance on for groups of target items (i.e., Low Popular, Mid-Low Popular, Mid-High Popular, and High Popular); • we measure and discuss the changes in the preference scores predicted from the trained VRS according to the variations of the predicted preference scores and the fraction of times a target item has received a preference score higher than the one before the attack; • we investigate and compare the efectiveness in pushing the target items (divided again by popularity groups) in the top- position of the model generated recommendation lists.

We conduct experiments on two datasets from the Amazon domain24[ , 25 ] to validate the efectiveness of the proposed model for the task of personalized visual recommendation.

2. Related Work

Recommender Systems (RSs) may rely on additional side information (e.g., images, audio, and text) to enhance the item representation and provide more tailored recommendations. Indeed, in domains such as fashion1[], food [ 2 ], and point-of-interest3[], product images displayed on online platforms can positively drive the final users’ decision. In this respect, to extend the expressive power of RSs, visually-aware recommender systems (VRSs) have recently proposed to incorporate products’ visual appearance of items into recommenda4ti,o5n,6[ , 7, 26 ]. Given the representational power of convolutional neural networks (CNNs) in capturing highlevel images’ characteristics, state-of-the-art VRSs often integrate visual features extracted via a CNN —pre-trained, e.g.,4[ , 6, 27, 28 ], or learned end-to-end, e.g.2,9[ , 30 ]. When it comes to adversarially attacking VRSs, the literature recognizes two main approaches, by perturbing the visual appearance of items either on tfheeature-level (i.e., the visual embeddings extracted from the CNNs) or on theimage-level (i.e., the item images). On the one side, thfeeaturelevel attacks, Tang et al1.4[] designed and implemented a framework to robustify the model by He and McAuley [ 4 ] against visual feature perturbations by leveraging adversarial training, while Paul et al3.1[] proposed an aesthetic-based VRS which is adversarially defended by adopting an iterative adversarial training procedure on the aesthetic features. On the other hand, the image-level attacks, Di Noia et al1.6[], Anelli et al.1[ 7 ] run state-of-the-art adversarial attacks in computer vision for poisoning the dataset with adversarial product images to alter the training of the model in order to push the target items towards higher recommendation positions. In addition, Liu and Larso1n8][, Cohen et al. 1[ 9 ] perturbed the item images at test-time in the realistic scenario that an adversary can upload an altered version of a product picture after the training of the model. In this work, we only considimerage-level attacks, since uploading an adversarial version of product images on a platform (e.g., eBay, Amazon, and Instagram) is more realistic than having access to the model in order to modify the visual feature used in both training/prediction phases. Furthermore, we focus our investigation on test-time/evasion attacks, assuming that, for an adversary, it can be easier and more eficient to change the product image on a platform to directly increase its predicted preference score and push it in high top- recommendation positions. In particular, considering the efects of items’ popularity on the recommendation qualit3y2[ , 23, 33 ], this work explored whether test time attacks might have diferent eficacy considering the items’ popularity.

3. Methodology

In this section, we first describe some useful notations, and then briefly review and present the formalization of the three adversarial attack strategies tested in the current work.

3.1. Preliminaries and Notations

Recommendation Task. Let , ℐ , and be the set of users, items, and score-based preference feedback, where|| , |ℐ |, and | | the preference of a user∈ are the set sizes, respectively. Then, l et ∈ be on the item ∈ ℐ , assuming that = 1 when a user has previously interacted with the item(e.g. reviewed, purchased or clicked on the product). We define the recommendation task as the problem of producing a list of items that maximizes, for each user, a utility function. Moreov ê r,refers to the predicted preference score inferred from the RS trained on the set of user-item preference-feedback. A popular class of methods to learn unseen users’ preferences is based on matrix factorization 3(M4]Ft)e[chniques. The of the second is theitem latent vector ∈ ℚ|ℐ |×ℎ, whereℎ << ||, |ℐ | . training of a MF-based recommender is aimed to learn an approximate version o||f×th|ℐe | high-dimensional matrix of user-item preferences as the dot product of two low-rank matrices of latent factors. Each row of the first matrix isuaser latent vector ∈ ℙ||×ℎ , while each row Visual Feature Extraction in VRSs. When it comes to visual recommendation, a common approach is to extract high-level visual features from (pretrained) CNNs 4(e,.6g,.,2[ 8, 29 ]). We indicate with the image/photo associated with the ite m∈ ℐ . While popular VRSs leverage either visual features extracted from pretrained CNNs or end-to-end trained approaches, we focus on the former class of visual recommenders, leaving the exploration of the latter as a future research direction. In this setting, given a set of data samp(l e, s ), where is the -th image associated with the item ∈ ℐ

and is the one-hot encoded representation of ’s image category, we indicate with the DNN/CNN classifier pre-trained on al(l , ). The network is trained such that the predicted probability vector of classes associated with an image ( (

) = ̂ ) is as much as close to the one-hot encoded vector of the ground truth-cla.sSsince the DNN is composed by -layers, we indicate with () ( ), 0 ≤ ≤ − 1 , the output of the output () ( ) = as a mono-dimensional vector that will be the input to the VRS. -th layer o f given the input . The actual extraction takes place at one of the last layers of the network, i.e., () ( ), where refers to the extraction layer. In general, we define this layer A Popular Visual Recommender: VBPR. To investigate the efects of items’ popularity when afected by adversarial attacks, we considered the most popular baseline in the visuallyaware recommendation task: Visual Bayesian Personalized Ranking from Implicit Feedback (VBPR) [ 4 ]. The model improves the MF preference predictor by addingviasual contribution to the traditionaclollaborative one. Given a use r and a non-interacted ite m, the predicted preference score i ŝ

= + + , where ∈ Θ||× and ∈ Θ|ℐ |× are thevisual latent vectors of user and item , respectively (<< ||, |ℐ | ). The visual latent vector of itemis obtained as = E , where is the visual feature of image ite mextracted from a pretrained convolutional neural network (i.e., AlexN8e]ta[s in the original work), whiEleis a matrix to project the visual feature into the same space as o.f Furthermore, stands for the sum of the overall ofset, the user, item, and global visual bias.

The Failure Point of a Visual Recommender: the Visual Features. All the state-of-the ̂ ∗ the products’ images so that the newly-extracted visual feat u r∗eleads to ̂ ∗ art adversarial strategie3s5,[ 19, 18, 17 ] alter the recommendation output̂∗() by perturbing ≠ ̂ . Then, > ̂ when the adversary wants topush/increase the score predicted on the target it em for the user , while ̂ ∗ < ̂ holds innuking scenarios. To craft the adversarially perturbed version of , the adversary can either have complete knowledge of the recommender model (i.e., parameters, output, and training data) or can be completely unaware of this information. In the former case, the adversary is generally recognized to wowrkhinte-box settings, while, in the latter case, she works ibnlack-box ones. of values at the end of the attack.

Adversarial Perturbation on Images. We define an adversarial attack as the problem of ifnding the best value for a perturbation such that the attacked image ∗ = + must be visually similar to according to a certaidnistance metric, e.g., norms, and ∗ must stay within its original value range, i[.0e,.,1]for 8-bit RGB images re-scaled by a factor 255. here, the intuition is that the visual feature extracted from the network∗ (= ( ∗) ≠ = ( ∗)) will change the original behavior of the recommender towards the malicious goal. Independently on the adversary knowledge of the VRS, all the adversarial attack strategies defined in the literature and explored in this work learn the adversarial perturbation of a product image (i.e., ) by backpropagating error information v(⋅i)a. Below, we define the three strategies to evaluate : TAaMR, WB-SIGN, andINSA. Note that, ∗ pixel values are clipped in the [0, 255] range

3.2. Black-Box: Targeted Adversarial Attack against Multimedia

Recommenders (TAaMR)

The first adversarial attack strategy tested in this work is a test-time extension ofTtahregeted Adversarial Attack against Multimedia Recommenders attack (TAaMR) proposed by [ 16 ]. The strategy, originally proposed for poisoning the training set with altered items images, assumes to adversarially perturb the image such that the pretrained CNN used to extract the visual feature will misclassify the original images towards a diferent class. In particular, we use the Fast Gradient Sign Method (FGSM)1[0] attack strategy —a baseline strategy in computer vision— that generates an adversarial version of the attacked image in only one step. Given a clean input image , a target clas s , a CNN (⋅) coeficient , the targeted adversarial image∗ is: with parameters , and a perturbation ∗ ←− − ⋅ sign(∇ ℒ (, , ) (1) where∇ ℒ (, , ) is the loss function gradient o(f⋅) , and sign(⋅) is the sign function.

3.3. White-Box: Sign Method (WB-SIGN)

The second adversarial strategy is tShiegn-based White-Box Attack (WB-SIGN) method that manipulates the product image computing the partial derivatives of the preference score function (⋅) with respect to the item image used to predict the user-item preference score, and updates the pixels in that direction. Formally, we define the function of the sum of preference scores measured on all the items as: ̂ ( ) = ∑ ( ̂ ( )) = ∑ ( +

E ( ) + )

then ∗ ←− − ⋅ sign( ∈ ∈ ̂ ( )) ) (2)

3.4. White-Box: Insider Attack (INSA)

The last experimented attack is thIensider Attack (INSA) method proposed by Liu and Larson [ 18 ]. Similar toWB-SIGN, this method assumes that the adversary has full knowledge and the access to the parameters of the trained model, and uses it to modify the pixels from the product image to increase the preference scores inferred from the recommender on each target item. To this end,INSA is defined as follows: ∗ ←− − ̂ ( )) such that || || ≤ where = (3) ̂ ( )) Note that, similarly tWoB-SIGN, INSA learns to build a perturbation that seeks to maximize all the users’ scores predicted for each attacked item. However, diferently froWBm-SIGN, the INSA’s perturbation is the gradient back-propagated through the recommender and the CNN, and not an -bounded sign dependent perturbation.

4. Experiments

4.1. Experimental Setup

This section is devoted to presenting the setting we followed to run the experiments, and then discuss the obtained results.

Datasets. We perform the experiments on two recommendation datasets froAmmazon.com containing customers’ feedback and items’ images. We usAemazon Boys & Girls andAmazon Men, including images in the fashion domain. The items set depend on the images still available on the e-commerce platform since they were not available in the released reposi2t4o,r2y5][. We remove items/users with less than 5 interaction5s, 4[]. Then, Amazon Boys & Girls has 1425, 5019, and 9213, whileAmazon Men has 16278, 31750, and 113106, users, items, and feedback, respectively.

Evaluation Metrics. We evaluate the tested attacks according to the ability of the adversary to compromise the integrity of the recommendations, i.e., the eficacy inincreasing the preference score predicted by the visual recommender anpudshing the target items into the top - of each user’s recommendation list. To measure the variation of the preference score, we define the Prediction Shift (PS) as follows: where is the set of target items whose images have been perturbed in an adversarial way. Additionally, to track the occurrences of an increase in the preference score, we measure the fraction of items for which we have measured a preference score improvement. We name this metric as Improvement Fraction (IF), and we formally define it as follows,

PS = 1 | | ∈

∑( ̂∗ − ̂ ) IF = 1 | | ∈ ∑ 1[ ̂∗ − ̂ ] where IF< 0.5 means that the number of times the target items have worsened their preference score is higher than the number of times it has been improved. While the previous metrics are related to preference score predictions, similar18to,1[ 7 ], we evaluate a ranking-wise metric that measures the average number of times a target item hits the topr-ecommendation lists. This metric, named Hit Ratio (HR@K), is defined as follows, (6) where hit@ (, ) is 1 when the target item is in the to p- list of the use r .

Reproducibility. We randomly initialize the model parameters with a Gaussian distribution with a mean of 0 and standard deviation of 0.01 and set the latent factor dimension to 128 as in [ 27 ]. We explore via grid-search: the learning rat{e0.i0n001, 0.001, 0.01} and the regularizers in {0.00001, 0.001}, whereas we fix the batch size to 256. We adopt early-stopping to avoid overiftting and choose the best model configuration for each algorithm according to the Recall@100 as in [ 27 ]. After having identified the best VBPR configuration on each dataset, we randomly sample 200 items from the catalog| (| = 200 ). We attack each target item image and measure the correspondent adversarial sco r̂∗e) (for each user. To study the efects of popularity on the attack eficacy, we split the target items into four groups based on the recorded feedback in the training set (i.e., Low Popular (LP), Mid-Low Popular (MLP), Mid-High Popular (MHP), and High Popular (HP)). For each attack, we vary the perturbation budg∈e{t4, 8, 16} . For the black-box strategyT(AaMR), we select the target class as the most popular one (”running shoes”) for both datasets. All codes, datasets, and configuration files to run and evaluate the experiments are publicly available in the El1liroetproducibility framework17[ , 36 ].

4.2. Results and Discussion

In this section, we investigate the following research questions: RQ1: What is the efect of items’ popularity on the eficacy of testing time adversarial attacks with respect to increasing the inferred preference scores? Is the behavior observed on a smaller perturbation budg et=( 4 ) consistent with higher perturbation budge ts∈( {8, 16})? RQ2: While studying the adversary’s ability in pushing the target item into the topr-ecommendation list, what is the efect of items’ popularity? How much has influenced ranking-wise performance?

4.3. Analysis of Attack Performance on Increasing Preference Scores (RQ1)

This paragraph analyses the Improvement Fraction (IF) and the Prediction Shift (PS) of the tested adversarial attacks against tAhmeazon Boys & Girls and Amazon Men datasets. Table1 reports the measured performance on the four target items groups as defined in Sect4i.o1n. Average Improvement Fraction and Prediction Shift across all popularity groups and perturbation budgets. We report the configuration files used to perform the experiments.

Amazon Boys & Girls- [Elliot Configuration File: ws_attack_best_amazon_boys_girls.yml] Improvement Fraction (IF)

MLP

MHP

Prediction Shift (PS)

Before moving to the investigation of items’ popularity efects of the adversarial attack efifcacy, it is interesting to observe that the black-box strategyTA(aMR) is mostly inefective in increasing the performance of the target items independently on the tested datasets. For instance, the most powerful attack with a perturbation budget equal to 16 has still negative PS values in both datasets (i.e., -0.01750 inA<mazon Boys & Girls, LP> and -0.00803 in <Amazon Men, MHP>) and IF values lower than 0.5 in all settings. We may explain the low eficacy of this attack strategy by stating that, diferently from thTeAaMR’s proposal paper37[], we do not use the targeted adversarial strategy to poison the training procedure, but we use it in the testing phase when the model has already learned the users’ preferences. However, we may derive two additional insights. The first one states that when increasin,gthe attacks become more performant, and IF gets closer to 0.5. The second one states that the attack results reported for TAaMR have been more efective on low popular target items than the most popular ones. For instance, IF onLP items is higher thanHP ones in all attack settings (e.g., 0.49444 > 0.45004 in Amazon Boys & Girls and 0.37303 > 0.33111 inAmazon Men with = 16 ).

After having discussed the black-box attack, we move the analysis to white-box ones. Starting from WB-SIGN, it can be noticed that the adversarial strategy is more efective against low popular items than high popular ones. For instance, PS w=ith4 is 2.02115 onLP and 1.80204 on HP in Amazon Boys & Girls, while 2.19162 and 1.91346 onAmazon Men. The same result trends are confirmed forINSA, the secondly reported white-box attacks for both datasets, where, for example, 0.94537 > 0.91230 > 0.90757 > 0.90606 for the IF measured oAnmazon Men HR@50 measured before the attack (No Attacks) and after the attack on all popularity groups. For each attack values, we report the percentage variation with respect to the not attacked version.

Amazon Boys & Girls

Amazon Men 0.00075 (-45.42%) 0.00105 (+13.64%)

with = 4 . The same trends on both WB-attacks and datasets are observed when vary.ing

These empirical observations confirm thatitems’ popularity has influenced the eficacy of adversarial attack strategies, where the least popular target items are subject to an increment of the preference scores much bigger than those calculated on the most popular ones. In addition, the tendency mentioned above is consistent when varying the perturbation budget from small values (i.e., 4) to larger ones (i.e., a maximum value of 16).

4.4. Analysis of Attack Performance on Top-K Recommendation Lists (RQ2)

the execution of adversarial attacks. This paragraph seeks to verify if the higher attack eficacy on low popular items measured from a preference score point of view is consistent when analyzing the to p- recommendation lists. Diferently from the previous analysis, we should point out that most popular items may have H@R 50 values (before the attack) higher than the ones of low popular items due to the well-known popularity bias iss2u0e,s21[ , 22, 23 ]. Indeed, it can be observed that the H@R50 of HP items is more than two times higher than the one measured onLP (i.e., 0.01423 > 0.0069) inAmazon Boys & Girls, and even more than six times higher inAmazon Men (i.e., 0.00297 > 0.00044). For this reason, we report the H@R50 variation after the attack in Tabl2e.

Analyzing the variations measured under black-box attack settings, it can be noted that, consistently with the findings measured in Section4.3, low popular target items have been more afected by attacks than most popular ones. Indeed, despite the negative variations of HR@50 measured onLP items in Amazon Boys & Girls (i.e., -70.97%), the ones measured on HP are even more negative (i.e., -113.03). The same trend is also confirmed in thAemazon Men dataset independently of the perturbation budget. Extending the analysis to the whitebox adversarial strategies, and considering that PS is always greater than 1 and IF is greater than 0.5, we should expect that the percentage variations measuredLoPnand MLP items should be higher than the ones oMnHP andHP. Results in Table2 confirm that both WB-SIGN andINSA are more efective on LP items. For instance, HR@50 increases by +85.19% onLP and +74.10% on HP, when WB-SIGN with = 4 is performed onAmazon Men.

Results on the top - recommendation performance additionally confirm thaittems’ popularity afects the eficacy of attacks by making the least popular target items easier to push into higher positions than the ones already in high positions.

5. Conclusion

We examined if test-time adversarial attacks against VRSs have a distinct impact on items based on their popularity. To this end, we tested one black-box (TiA.ea.M, R) and two state-of-the-art white-box (i.e.,WB-SIGN andINSA) single-step adversarial attacks by varying three levels of perturbation budget (i.e.,∈ {4, 8, 16} ) to alter the recommendations generated by VBPR, a baseline model for visual recommendation. Indeed, after VBPR’s training on two datasets (Ai.em.,azon Boys & Girls and Amazon Men), we randomly extracted 200 target items from each catalog. Then, we divided them into four groups based on the number of ratings registered in the training set, and performed two analyses: one on the preference score and the other on the efects on top- lists. From the former, we found that items’ popularity influences the attacks’ eficacy, which is much more efective on the least popular than high popular items in incrementing the preference scores consistently at varying .oFfrom the latter, we verified that this trend is also confirmed when looking at to p- recommendation lists, with the least popular target items getting the highest pushing in ranking positions. These results open exciting challenges for developing adversarial defenses strategies, as the least popular items can be highly subjected to adversarial attacks. We propose extending the study on iterative adversarial attacks to understand if the previously identified trends are consistent with more robust strategies for future extension. Finally, we plan to extend this investigation line to examine the potential efects of users’ activeness (e.g., number of released ratings) on the attack eficacy for providing more insights into planning more powerful defense strategies and to study of the verified efects with human evaluation.

Acknowledgments

We acknowledge support of PON ARS01_00876 BIO-D, Casa delle Tecnologie Emergenti della Città di Matera, PON ARS01_00821 FLET4.0, PIA Servizi Locali 2.0, H2020 Passapartout - Grant n. 101016956, and PIA ERP4.0.

References

[1] Y. Hu , X. Yi , L. S. Davis , Collaborative fashion recommendation: A functional tensor factorization approach , in: ACM Multimedia, ACM, 2015 , pp. 129 - 138 .

[2] D. Elsweiler , C. Trattner , M. Harvey , Exploiting food choice biases for healthier recipe recommendation , in: SIGIR, ACM, 2017 , pp. 575 - 584 .

[3] S. Wang , Y. Wang , J. Tang , K. Shu , S. Ranganath , H. Liu, What your images reveal: Exploiting visual contents for point-of-interest recommendation , in: WWW, ACM, 2017 , pp. 391 - 400 .

[4] R. He , J. J. McAuley , VBPR: visual bayesian personalized ranking from implicit feedback , in: AAAI, AAAI Press, 2016 , pp. 144 - 150 .

[5] R. He , J. J. McAuley , Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering , in: WWW 2016 , 2016 .

[6] Q. Liu , S. Wu , L. Wang , Deepstyle: Learning user preferences for visual recommendation , in: SIGIR, ACM, 2017 , pp. 841 - 844 .

[7] L. Meng , F. Feng , X. He , X. Gao , T. Chua, Heterogeneous fusion of semantic and collaborative information for visually-aware food recommendation , in: ACM Multimedia, ACM , 2020 , pp. 3460 - 3468 .

[8] A. Krizhevsky , I. Sutskever, G. E. Hinton , Imagenet classification with deep convolutional neural networks , in: NeurIPS 2012 , 2012 .

[9] Y. Deldjoo , T. D. Noia , F. A. Merra , A survey on adversarial recommender systems: from attack/defense strategies to generative adversarial networks , ACM Computing Surveys (CSUR) ( 2021 ).

[10] I. J. Goodfellow , J. Shlens , C. Szegedy , Explaining and harnessing adversarial examples , in: ICLR (Poster) , 2015 .

[11] A. Madry , A. Makelov , L. Schmidt , D. Tsipras , A. Vladu , Towards deep learning models resistant to adversarial attacks , in: ICLR 2018 , 2018 .

[12] N. Carlini , D. A. Wagner , Towards evaluating the robustness of neural networks , in: SP 2017 , 2017 .

[13] X. He , Z. He , X. Du , T. Chua, Adversarial personalized ranking for recommendation , in: SIGIR, ACM, 2018 , pp. 355 - 364 .

[14] J. Tang , X. Du , X. He , F. Yuan , Q. Tian , T. Chua, Adversarial training towards robust multimedia recommender system , IEEE Trans. Knowl. Data Eng . 32 ( 2020 ) 855 - 867 .

[15] K. He , X. Zhang , S. Ren, J. Sun , Deep residual learning for image recognition , in: CVPR, IEEE Computer Society , 2016 , pp. 770 - 778 .

[16] T. Di Noia , D. Malitesta , F. A. Merra , Taamr: Targeted adversarial attack against multimedia recommender systems , in: DSN-DSML 2020 , 2020 .

[17] V. W. Anelli , Y. Deldjoo , T. Di Noia , D. Malitesta , F. A. Merra , A study of defensive methods to protect visual recommendation against adversarial manipulation of images , in: SIGIR, ACM, 2021 , pp. 1094 - 1103 .

[18] Z. Liu , M. A. Larson , Adversarial item promotion: Vulnerabilities at the core of top-n recommenders that use images to address cold start , in: WWW, ACM / IW3C2 , 2021 , pp. 3590 - 3602 .

[19] R. Cohen , O. S. Shalom , D. Jannach , A. Amir , A black-box attack model for visually-aware recommender systems , in: WSDM, ACM, 2021 , pp. 94 - 102 .

[20] D. Jannach , L. Lerche , I. Kamehkhosh, M. Jugovac , What recommenders recommend: an analysis of recommendation biases and possible countermeasures, User Model . User Adapt. Interact . 25 ( 2015 ) 427 - 491 .

[21] H. Abdollahpouri , R. Burke , B. Mobasher , Controlling popularity bias in learning-to-rank recommendation , in: RecSys, ACM, 2017 , pp. 42 - 46 .

[22] Z. Zhu , J. Wang , J. Caverlee , Measuring and mitigating item under-recommendation bias in personalized ranking systems , in: SIGIR, ACM, 2020 , pp. 449 - 458 .

[23] L. Boratto , G. Fenu, M. Marras , Connecting user and item perspectives in popularity debiasing for collaborative recommendation , Inf. Process. Manag . 58 ( 2021 ) 102387 .

[24] R. He , C. Packer , J. J. McAuley , Learning compatibility across categories for heterogeneous item recommendation , in: ICDM, IEEE Computer Society, 2016 , pp. 937 - 942 .

[25] J. J. McAuley , C. Targett , Q. Shi , A. van den Hengel , Image-based recommendations on styles and substitutes , in: SIGIR 2015 , 2015 .

[26] V. W. Anelli , A. Bellogín , A. Ferrara , D. Malitesta , F. A. Merra , C. Pomo , F. M. Donini , T. Di Noia , Elliot: A comprehensive and rigorous framework for reproducible recommender systems evaluation , in: SIGIR, ACM, 2021 , pp. 2405 - 2414 .

[27] J. Chen , H. Zhang , X. He , L. Nie , W. Liu, T. Chua, Attentive collaborative filtering: Multimedia recommendation with item- and component-level attention , in: SIGIR, ACM, 2017 .

[28] W. Niu , J. Caverlee , H. Lu , Neural personalized ranking for image recommendation , in: WSDM 2018 , 2018 .

[29] W. Kang , C. Fang , Z. Wang , J. J. McAuley , Visually-aware fashion recommendation and design with generative image models , in: ICDM, IEEE Computer Society , 2017 , pp. 207 - 216 .

[30] R. Yin , K. Li , J. Lu , G. Zhang, Enhancing fashion recommendation with visual compatibility relationship , in: WWW 2019 , 2019 .

[31] A. Paul , Z. Wu , K. Liu , S. Gong , Robust multi-objective visual bayesian personalized ranking for multimedia recommendation , Applied Intelligence ( 2021 ) 1 - 12 .

[32] R. Cañamares , P. Castells , Should I follow the crowd?: A probabilistic analysis of the efectiveness of popularity in recommender systems , in: SIGIR, ACM, 2018 , pp. 415 - 424 .

[33] E. Mena-Maldonado , R. Cañamares , P. Castells , Y. Ren , M. Sanderson , Popularity bias in false-positive metrics for recommender systems evaluation , ACM Trans. Inf. Syst . 39 ( 2021 ) 36 : 1 - 36 : 43 .

[34] Y. Koren , R. M. Bell , C. Volinsky , Matrix factorization techniques for recommender systems , Computer 42 ( 2009 ) 30 - 37 .

[35] J. Tang , X. Du , X. He , F. Yuan , Q. Tian , T. Chua, Adversarial training towards robust multimedia recommender system , IEEE Trans. Knowl. Data Eng . 32 ( 2020 ) 855 - 867 .

[36] V. W. Anelli , A. Bellogín , A. Ferrara , D. Malitesta , F. A. Merra , C. Pomo , F. M. Donini , T. Di Noia , V-elliot: Design, evaluate and tune visual recommender systems , in: RecSys, ACM, 2021 .

[37] T. Di Noia , D. Malitesta , F. A. Merra , Taamr: Targeted adversarial attack against multimedia recommender systems , in: DSN Workshops, IEEE, 2020 , pp. 1 - 8 .