Recommender systems commonly employ Collaborative Filtering to generate personalized recommendations, forming an implicit social network where users influence each other's recommendations based on their preferences. In this paper, we show that it is possible to identify users with detrimental influence-those who negatively afect the recommendations of others-and that merely removing specific detrimental users from the training data can improve system performance. We apply a Leave-one-out analysis across five datasets to capture how recommendations change if a specific user is removed. Based on that data, we quantify positive and negative influences and implement a pruning strategy to remove detrimental users. Importantly, our strategy still provides recommendations to the pruned users by recommending the most popular items. We evaluate our pruning strategy on five commonly used datasets, including MovieLens, Amazon, and LastFM. We show that pruning detrimental users increases kNN performance, achieving an average performance increase of 3% for Item-Item kNN while removing 3.56% of users from the training data. Our findings highlight the potential of influence-based pruning to enhance recommender systems by increasing performance and creating resilience against detrimental influence.
Collaborative Filtering (CF) algorithms make
recommendations based on the principle that users who agree on the
same items will do so in the future. The result is a system
in which each user’s recommendations are primarily
determined by their similarity to other users. Due to users’
influence on each other, Lathia et al. [
In the quest to increase the performance and robustness
of recommender systems, those influential users [
The potential of influential users to either enhance or
erode the quality of recommendations leads us to an
intriguing possibility. We hypothesize that there are real users
in the training data whose inclusion negatively impacts
the recommendations for other users and that removing
these users can improve overall system performance. We
call them detrimental users. Intuitively, removing many
or all detrimental users should improve recommendations.
However, it is a common assumption that less training data
decreases performance [
RQ: How can we identify and separate detrimental users? To answer our research question, we analyze the influence of every individual user in five popular datasets using three CF kNN and matrix factorization algorithms to quantify user influence on ranking predictions. We show that it is possible to identify detrimental users who negatively impact the performance of other users and that pruning the most detrimental users can improve overall recommendations.
The source code reproducing all the results presented in
this paper is available at our GitHub1.
Several aspects of user influence in recommender systems
have been the subject of previous research. Rashid et al.
[
In more recent work, Eskandanian et al. [
Wilson et al. [
Existing research indicates that just a few influential users
have the potential to considerably change
recommendations [
We examine detrimental users and their efect on
recommendations in two parts. The first consists of a user influence
analysis, which aims to identify detrimental influential users
by quantifying influence via diferent metrics. To achieve
this, we adopt the Leave-one-out (LOO) concept described
by Rashid et al. [
We use five datasets in our analysis: ML-100k, ML-1M [11], Last.FM [12], Amazon-Digital-Music and AmazonLuxury-And-Beauty [13]. We transform explicit feedback data, e.g., ratings, into implicit feedback data, e.g., interactions, treating every rating as a positive interaction and evaluate the NDCG@10 of User-User kNN, Item-Item kNN, and Alternating Least Squares (ALS) CF on each dataset. We use the algorithm implementations of LensKit [14].
The first part of our analysis follows a simple question: if one specific user is removed from the training data, how does the NDCG@10 change for every other user? To answer this question, we implement the following LOO pipeline. First, we calculate the baseline result for each algorithm, e.g., the NDCG@10 performance of each algorithm considering all users. From the obtained results, we build a vector b, with each entry representing the baseline NDCG@10 for a user . Then, we prune the user from the training and test data. We train a new model and calculate the recommendations on this pruned data. These results form a vector r for each pruned user , with , being the NDCG@10 for user calculated without user in the training data. The NDCG@10 of the pruned user is set to 0 because the user receives no recommendations. The basis of the influence analysis is the diference between the pruned results r and baseline results b. For every user , a vector ∆ r = r − b describes this diference. Pruning successively every user results in a Matrix with
= (∆ r)=1,..., where is the total number of users in the dataset. If for users and ∆ , > 0 holds, user receives better recommendations when user is not in the training data, ergo user has a detrimental influence on user . Conversely, if ∆ , < 0 holds, the existence of user in the data improves user ’s recommendations. Using , we calculate the following four normalized influence metrics.
The influence mean is the diference between the
baseline and pruned mean NDCG@10. It describes how the
overall system performance changed compared to the
baseline performance due to pruning the user . A feature of
is that it depends on user ’s baseline performance because
’s performance on the pruned dataset is 0. To address this
issue, we introduce the cleaned influence mean , which
removes the pruned user from the influence mean
calculations. Furthermore, we introduce the influence diference
, which we derive from the metric presented by
Rashid et al. [
To test whether pruning multiple detrimental users from the training data based on the acquired influence data improves recommendations, we evaluate an optimal pruning strategy on all datasets and algorithms using user-based ifve-fold cross-validation. We use random search to identify the optimal pruning threshold of each influence metric for each dataset and algorithm and prune users based on this optimal pruning strategy.
The result of pruning detrimental users is illustrated in Figure 1. Item-Item kNN benefits the most with an average performance increase of 3% while User-User kNN also shows improvements but on a lower level with around 0.2%. ALS is, on average, negatively afected by pruning detrimental users. The relative performance change for the users remaining in the training data is, on average, around 0.5 percentage points better than all users combined. This is expected since the pruned users are recommended the most popular items, which are worse than CF.
The amount of pruned users varies significantly depending on the dataset. Table 1 shows an average, with over 3.5% of users pruned for Item-Item kNN, confirming that removing multiple detrimental users can improve recommendations despite reducing the training data. We observe that the optimal influence metric and threshold depend on the dataset and algorithm. The performance increase in our experiments varies depending on the dataset, with larger datasets benefitting more. For example, we observe a three times higher relative performance improvement for ML-1M than ML-100K.
To illustrate, Figure 2 shows the user distribution in the ML-1M dataset using Item-Item kNN. The dispersion of the influence score increases with rising influence. This leads to some influential users significantly negatively afecting other users. Pruning all users below the threshold shown in Figure 2 leads to a considerable overall performance improvement of over 7% for ML-1M Item-Item kNN.
The results from our pruning analysis answer our initial research question by confirming that the influence metrics we introduce can identify and diferentiate users with positive and detrimental influences. The performance improvement we observe, especially for Item-Item kNN, demonstrates that the users we identify as detrimental harm the recommendations of other users. However, this efect depends on the algorithm, as shown by the reduced performance of ALS when pruning users. Future work should focus on understanding the characteristics of detrimental users and try to identify them based on heuristics without the need to conduct a computationally expensive LOO analysis. SIAM International Conference on Data Mining, SIAM, 2005, pp. 556–560. [7] M. A. Morid, M. Shajari, A. H. Golpayegani, Who are the most influential users in a recommender system?, in: Proceedings of the 13th international conference on electronic commerce, 2011, pp. 1–5. [8] A. M. Rashid, Mining influence in recommender systems, 2007. [9] C. E. Seminario, D. C. Wilson, Assessing impacts of a power user attack on a matrix factorization collaborative recommender system, in: The Twenty-Seventh International Flairs Conference, 2014. [10] J. Beel, V. Brunel, Data pruning in recommender systems research: Best-practice or malpractice, ACM RecSys (2019). [11] F. M. Harper, J. A. Konstan, The movielens datasets: History and context, Acm transactions on interactive intelligent systems (tiis) 5 (2015) 1–19. [12] I. Cantador, P. Brusilovsky, T. Kuflik, 2nd workshop on information heterogeneity and fusion in recommender systems (hetrec 2011), in: Proceedings of the 5th ACM conference on Recommender systems, RecSys 2011, ACM, New York, NY, USA, 2011. [13] J. Ni, J. Li, J. McAuley, Justifying recommendations using distantly-labeled reviews and fine-grained aspects, in: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), 2019, pp. 188–197. [14] M. D. Ekstrand, Lenskit for python: Next-generation software for recommender systems experiments, in: Proceedings of the 29th ACM international conference on information & knowledge management, 2020, pp. 2999–3006.