User feedback results in diferent rating patterns due to the users' preferences, cognitive diferences, and biases. However, little research has taken into account cognitive biases when building recommender systems. In this paper, we propose novel methods to take into account user polarization into matrix factorization-based recommendation systems, with the hope to produce algorithmic recommendations that are less biased by extreme polarization. Polarization is an emerging social phenomenon with serious consequences in the era of social media communication. Our experimental results show that our proposed methods outperform the widely-used methods while considering both rank-based and value-based evaluation metrics, as well as polarization-aware metrics.
Ratings are loaded with biases which are caused by factors related to users, items, and the interaction between
users and items. For example, harsh critics might give low ratings compared with the majority of users.
Similarly, the popularity of an item generally results in higher ratings for that item [
Social influence bias is defined as the ’conformity’ engendered by answers of other participants characterized
by responses similar to the community ’norm’ [
Anti-social influence bias is defined as the user’s tendency to overrate items if they are presented with low
average ratings. On Amazon 1 and Netflix 2 for example, users were found to feel the need to increase some
ratings and to want the items to receive rewards from their ratings [
Critical users tend to give low ratings to almost all items, while enthusiastic users give high ratings. Some
users are very extreme, they either give very high ratings or very low ratings, which leads to a ’U’ shape rating
distribution. We define this user pattern as ’polarized-users’, similar to the notion of ’polarized-items’ [
Previous studies in recommendation systems focused on user bias by modeling user bias as Gaussian noise.
Shan & Banerjee model user and item bias using a residual approach by removing these biases before applying
their latent factor based collaborative filtering approach [
Some researchers considered cognitive aspects of biases, which reveal diferent response styles or style biases
in rating values of recommendation systems. From these studies, various types of responses were revealed,
and researchers attempted to detect and correct bias using computational models [
In this paper we modify matrix factorization-based collaborative filtering with the aim of producing algorithmic recommendations that are less biased by extreme polarization. The rest of the paper is organized as follows: We start with a presentation of our proposed methodology in Section 2, then follow with our evaluation experiments in Section 3, and end with our conclusions in Section 4.
Our idea is inspired by the polarization phenomena studied in [
In this section, we exploit a user polarization detector in depolarizing a recommendation system. We first transform the original ratings to account for polarization. The proposed solution tries to counteract polarization by making the training data set less polarized, and employs a stochastic mapping function as defined below: : (, ) → (, , ′ ).
The transformation function aims to transform a user-item rating based on the rating itself, polarization score, and user’s current rating standard deviation, as follows
′ 1 ′ = + Φ " 1+− 1() < , = − Φ " 1+− () ⩾ (1) where is the original rating and is a threshold on ratings for user defined as the rating mid-range. is the polarization score of user . When the polarization score is 0, there is no change to the original ratings. Finally, () is the standard deviation of the user’s ratings.
The purpose of this transformation is to reduce the intensity of polarization of the ratings of the user. Figure 3 shows the efect of this transformation. The yellow bar represents all the ratings of user 16 for 100 jokes from the Jester dataset (which we use because it is a complete rating data set), while the red bar shows the transformed ratings. We also compare the polarization score before and after transformation.
Considering collaborative filtering systems, the intuition behind transforming the original ratings is to bring users closer in latent space after the transformation. Figure 2 shows the intuition of counter-polarization in matrix factorization. Sub-figure 2(a) shows the original data in latent space, while the sub-figure 2(b) shows the data after transformation. By moving a polarized user closer to other users in latent space, we obtain less polarized recommendations.
Matrix Factorization (MF)-based recommender systems have shown good performance by combining good
scalability with predictive accuracy [
∈ ∈ m,in ∑︁ ∑︁ ||′ − .||2 + (||||2 + ||||2) Here ′ is the transformed rating, while and are the latent feature vectors for users and items, respectively. Using stochastic gradient descent, we obtain the following updating rule to minimize the objective function: ← ← + (2 − 2 ) + (2 − 2 ) Here = ˆ′ − , and is the learning rate for gradient descent.
Algorithm 1: User-polarization-aware Matrix Factorization (UpaMF)
Data: Transformed rating matrix ′, , Result: Latent features and Initialize and ; Calculate error using Eq. 2; while Stopping criteria is not met do
Update using ← + (2 − 2 ) ;
Update using ← + (2 − 2 ); end
The above User-polarization-aware Recommendation System takes advantage of the polarization score obtained from the polarization detector. However, the original ratings matrix information is completely ignored. We next propose a weighted user-polarization-aware recommendation system, to allow a flexible weighting of the original ratings. The objective of the weighted user-polarization-aware matrix factorization (WUpaMF) is defined as:
∈ ∈ m,in ∑︁ ∑︁ Φ ||′ − ||2 + (1 − Φ )|| − ||2
+ (||||2 + ||||2) where Φ weights the influence of the original rating factorization, and is also the user polarization score. When Φ = 0, the objective reduces to regular matrix factorization, while it reduces to UpaMF when Φ = 1 . The updating rule, using gradient descent, for this new objective is: ← + (2((1 − Φ ) + Φ ′) − 2 ) ← + (2((1 − Φ ) + Φ ′) − 2 ) Here ′ = ′ − , = − , and is the learning rate for gradient descent.
Algorithm 2: Weighted User-polarization-aware Matrix Factorization (WUpaMF)
Data: Original rating ,Transformed rating matrix ′, , , Φ Result: Latent features and Initialize and ; Calculate error using Eq. 4; while Stopping criteria is not met do
Update using ← + (2((1 − Φ ) + Φ ′) − 2 ) Update using
← + (2((1 − Φ ) + Φ ′) − 2 ); end
We used the Jester dataset, which contains 4.1 million ratings by 73,421 users on 100 jokes, collected between
April 1999 and May 2003 [15]. Jester uses continuous ratings in the range of [
We first built a rating histogram of each user’s ratings, and labeled the ratings into Polarized or Non-Polarized. Finally, we obtained 517 polarized users. In order to have a balanced dataset, we then randomly selected 517 non-polarized users to form a combined dataset (517 polarized users and 517 non-polarized users). We randomly left out 205 users (20% of the final data) for testing the polarization detector. All in all, we ended up with a rating matrix of 1034 users by 100 items. Note that the density of those ratings is around 94%, which indicates that most of the users rated all 100 items. We then randomly split all ratings into a training (90%) and testing set (10%). For both UpaMF and WUpaWF, we set = 200, = 0.02 and = 0.002 for training in Eq. 3 and Eq. 5.
We trained our polarization detector using the Random Forest Algorithm [
Figure 4 shows that the constructed feature set can capture the polarization trends of user ratings. Figure 5 shows that the user polarization scores before the transformation are very unbalanced, and more users have very high polarization scores. On the other hand, the polarization scores after transformation are less extreme.
We evaluated the performance of our proposed method in terms of rating prediction accuracy, i.e., the Mean
Average Error (MAE) [
() = − 2(), () = ∈,̸=(, ), 0.8 re0.6 o c S n o i tza0.4 i r a l o P 0.2 where is the set of top-N recommended items for a user and (, ) is the distance between item and calculated using the cosine similarity based on the rating vector. Then, we average over all the recommended items for the user.
Another way to measure the debiasing efectiveness is to check the blind spot size (see Eq. 8). The blind spot size is defined as the number of items with predicted ratings ˆ, lower than a given threshold , i.e., Du = {i ∈ | ˆ, < }. Note that because each user has their own blind spot, we define a threshold for each user. The threshold is found based on a percentile cut-of for each user, set to 95%. Therefore, We define the blind spot for user as
Du = {i ∈ | ˆ, < () * },
where is threshold which controls the cutof. Modern recommendation systems sufer from two important
issues: filter bubbles [ 18], and blind spots [
We check how many times an item falls into the blind spot based on the predicted ratings across all users who have not rated or seen this item yet. We use ℛ = { ∈ | ℎ } as the observed ratings for user . Therefore the BL score for an an item is computed as follows: ∑︀ | ∈ ̸∈ ℛ| () = ∈ ∑︀ | ̸∈ ℛ|. ∈
The numerator is the total number of users who has not rated item and have item in their blind spot based on the prediction. The denominator is the total number of users who have not rated item . For example, if an item falls into the blind spot 100 times based on Eq. 8 across all 400 users who have not rated this item, then the BL score for this item will be 100/400 = 0.25.
We applied the polarization detector to the testing set to obtain the polarization score, then we transformed
the rating using Equation 1. We also computed the Mean Average Precision (MAP) on the top-N
recommendation list [17]. Five algorithms were applied on the transformed rating matrix or the original rating
matrix:
1. Regular Matrix Factorization (Reg MF) on riginal ratings;
2. Bias-considered matrix factorization (Bias considered MF) on original ratings (in which biases from
users, items and the overall rating are added) [
For 5), we need to compare all the above metrics on the original ratings and transformed ratings separately. We repeated the experiments 10 times and report the averaged scores.
We report all the metrics above for both polarized and non-polarized users. The WUpa MF in Eq. 4 involves both the transformed and original ratings; therefore we compare all the metrics above based on both the original ratings and transformed ratings. Figure 7 shows the MAE on both polarized users, non-polarized users and all users. Our proposed Upa MF has the lowest MAE in all groups. Figure 8 shows that our proposed Upa MF algorithm has the highest NDCG@10 score, while Bias considered Upa MF has the lowest NDCG@10 score.
We report the diversity and novelty results in Figure 9 and Figure 10. As shown in Figure 9,the Non-polarized user group has higher diversity score using Bias considered MF; but overall, Upa MF results in the highest diversity score.
We also report the (BL) score distribution for all five compared algorithms in Figure 11. Bias considered Upa has the highest BL score, indicating that this algorithm boost only a few items. On the other hand, Upa MF achieves the lowest BL score, promoting several unpopular items to be explored by users. Finally figure 10 shows that Upa MF and Bias considered MF have comparable novelty scores and they both exceed the novelty of baseline MF and bias-considered MF.
Users have been fond to exhibit diferent rating patterns in previous studies. However, few studies have taken into account the users’ polarized rating pattern. In this paper, we first adapted our previous item polarization detection technique to obtain a polarization score for each user, which was then used in a novel User-polarization-aware matrix factorization (UpaMF) algorithm to obtain less polarized recommendations. We also proposed a weighted User-polarization-aware MF algorithm. Our results show that our proposed techniques outperformed the naive MF baseline and the bias-aware Matrix Factorization in terms of both rank-based and value-based evaluation metrics, while improving the recommendation lists’ diversity and reducing the blind spots induced by the recommendations.
This work was supported by National Science Foundation grant NSF-1549981. [13] U. Chitra, C. Musco, Analyzing the impact of filter bubbles on social network polarization, in: Proceedings of the 13th International Conference on Web Search and Data Mining, 2020, pp. 115–123. [14] M. Badami, O. Nasraoui, P. Shafto, Prcp: Pre-recommendation counter-polarization., in: KDIR, 2018, pp.
280–287. [15] K. Goldberg, T. Roeder, D. Gupta, C. Perkins, Eigentaste: A constant time collaborative filtering algorithm,
Information Retrieval 4 (2001) 133–151. [16] G. Shani, A. Gunawardana, Evaluating recommendation systems, Recommender systems handbook (2011) 257–297. [17] P. Castells, S. Vargas, J. Wang, Novelty and diversity metrics for recommender systems: choice, discovery and relevance (2011). [18] T. T. Nguyen, P.-M. Hui, F. M. Harper, L. Terveen, J. A. Konstan, Exploring the filter bubble: the efect of using recommender systems on content diversity, in: Proceedings of the 23rd international conference on World wide web, ACM, 2014, pp. 677–686. [19] O. Nasraoui, P. Shafto, Human-algorithm interaction biases in the big data cycle: A markov chain iterated learning framework, arXiv preprint arXiv:1608.07895 (2016). [20] P. Shafto, O. Nasraoui, Human-recommender systems: From benchmark data to benchmark cognitive models, in: Proceedings of the 10th ACM Conference on Recommender Systems, ACM, 2016, pp. 127–130. [21] W. Sun, O. Nasraoui, P. Shafto, Evolution and impact of bias in human and machine learning algorithm interaction, Plos one 15 (2020) e0235502.