Diversity enhancement for collaborative filtering recom mendation

Abstract

interactions. To evaluate the user experience of recommendation systems in realistic and complex scenarios, the EvalRS challenge evaluates recommendation algorithms from multiple perspectives such as fairness, diversity. This paper details the diversity-enhanced collaborative filtering recommendation algorithm that won first place in the EvalRS challenge. Our proposed solution has two essential innovations. First, the importance of the user's historical behavior is ranked so as to obtain a high-ranking performance using fewer user behaviors. Second, the recommendation results are re-ranked to enhance the diversity of the recommendation results. In addition, this paper proposes a new evaluation metric, the quantile-based fairness Gini coeficient, to metric the fairness of the recommendation results, as it does not cause drastic fluctuations due to the small number of item collaborative filtering, recommendation evaluation, recommendation fairness ∗Corresponding author.

1. Introduction

2. The Challenge

Conventional recommendation algorithm evaluation met- EvalRS is one of the challenges of CIKM 2022 AnalytiCup, rics are usually ranking performance metrics such as hit which is based on the reclist[ 2 ] and aims to evaluate recrate, ndcg, MRR, etc., which may lead to recommendation ommender systems in terms of key dimensions such as systems considering only certain user preferences and ig- diversity and fairness. The dataset of this challenge is noring multiple aspects of user experience. Fairness and based on LFM-1b[ 3 ], a music dataset with about 820,000 diversity are as important as ranking performance metsongs, 110,000 users, and about 37 million user interacrics. If users frequently interact with a single type of item, tions.The goal of the challenge is to evaluate the recomthe user experience of the recommendation system will mendation system in multiple aspects, including Stanbe damaged in the long run, which will eventually lead dard recommendation system metrics, Standard metrics to user churn. Therefore, the recommendation algorithm on a per-group or slice basis, Behavioral and qualitative model needs to be evaluated from multiple perspectives. tests, etc.[ 1 ] This paper illustrates the solution of EvalRS challenge[ 1 ], including a collaborative filtering algorithm based on frequent item mining, importance ranking of user behavior based on TF-IDF, and diversity-enhanced re-ranking algorithm. The next section presents a brief introduction to this challenge, and section 3 details the solution and the fairness index based on the Gini coeficient; finally, the section 4 shows the experimental results. Furthermore, the specific solution code and documentation are publicly available on GitHub: https://github.com/lazy2panda/cikm2022_solution.

CIKM’22: Proceedings of the 31st ACM International Conference on Information and Knowledge Management Results of various experiments

Tests

MRR

3. Solution

3.1. Model Architecture

The main model used in this paper is the n-gram model, which is mainly applied in the field of natural language processing and is a statistical-based algorithm.[ 4 ] Its basic idea is that the content inside the text is operated with a sliding window of size , forming a sequence of segments of length . This method first sorts the user history sequence by time denote as and uses the ngram algorithm to process each user history sequence, where we take in n-gram as 2; that is, each user history sequence is cut into multiple subsequences of length 2, then the whole training set is 2-gram sequence sliced, and calculate the frequency of all trackid pairs; for trackid pair (, ) , its frequency is denoted as (, ) ; for track , the track with the highest co-occurrence frequency is denoted as () . Next, based on the user’s history , the similar track ()

of each user’s history is obtained, and then the recommendation result of the user is obtained by ranking all similar items according to their frequency. In order to reduce the popularity fairness of track in recommendation results, we use the TF-IDF value of track to sort the user history sequence and truncate the user history records, where TF-IDF is calculated as follows, for track , where is the number of plays of in user , and is the number of plays of in all users, where denotes the total number of users and () number of users who have interacted with . denotes the − () = ∗ ( + 1)

(1) () Finally, diversity-enhanced reranking is performed on the basis of the above recommendation results. The user’s recommendation results are denoted as , and the user’s versity is defined as ( where

) = 0.3 ∗ − 0.7 ∗ is defined as the sum of the diferences between each point in the prediction space and the mean of the prediction space, and is defined as the distance between the ground truth vector and the mean of the prediction vector.[ 1 ] In this method, we combine the MMR[ 5 ] diversity algorithm with the diversity definition of EvalRS and use the vector mean of as the ground truth vector to calculate the recommended result as follows: =

∈ ∣ arg max [0.3 ∗ diversity([, ])) − 0.7 ∗ bias([, ]))] mendation. The Gini coeficient originates from the field of economics and is often used to assess the degree of fairness of income distribution of residents. This paper uses the Gini coeficient to assess the degree of variation in the accuracy of items across quartile intervals. Compared to the standard deviation, the Gini coeficient is a more accurate reflection of the diference in fairness between two pairs of items across quartile intervals. The specific calculation is, firstly, dividing the trackid into multiple quantile intervals according to the popularity, secondly, calculating the false positive rate (FPR) of the trackid in diferent quantile intervals, and finally,the Gini coeficient is calculated as follows: =

∑=1 ∑=1 | − | 2 ∗ ̄ (3)

4. Experiments

The experiments part shows the metrics obtained after several iterations. As shown in Table 1, the results of Experiment 1 indicate that HIT_RATE and MRR are the highest, fairness and diversity metrics are the lowest, and the final AGGREGATE_SCORE is the lowest when no samples are performed on user history sequences. Experiment 2 performs TF-IDF sorting on user history sequences and uses only the top 8 tracks for inference. The results show that the fairness metrics are significantly improved compared to Experiment 1, with a 94% improvement in the MRED_TRACK_POPULARITY metric. Based on Experiment 2, Experiment 3 conducted a diversity enhancement ranking. Compared with Experiment 2, Behavioral and qualitative tests were significantly improved, with BEING_LESS_WRONG improved by 31% and LATENT_DIVERSITY improved by 46%, and , 2. The AGGREGATE_SCORE of Experiment 3 is 1.7025.

In addition, TRACK_POPULARITY_GINI is our custom test metric whose value can reflect the fairness of track popularity. Finally, since the competition requires that the submission must be run on the AWS EC2 p3.2xlarge instance within 90 minutes, with the support of JIUTIAN Artificial Intelligence Platform 1, we completed the experiments and performance tests.

5. Conclusion

(2)

This paper details the EvalRS challenge’s solution, which is based on the n-gram collaborative filtering algorithm and uses TF-IDF to rank the importance of users’ history records, followed by re-ranking for diversity, and finally achieves better results in aggregation metrics. history is recorded as . The EvalRS challenge for di- the others metrics remained the same as in Experiment

3.2. Quantile-based Gini coeficient fairness test

Based on the Gini coeficient, this paper proposes a new test to calculate the fairness of track popularity recom- 1https://jiutian.10086.cn

References

[1] J. Tagliabue , F. Bianchi , T. Schnabel , G. Attanasio, C. Greco , G. d. S. P. Moreira , P. J. Chia , Evalrs: a rounded evaluation of recommender systems , 2022 . URL: https://arxiv.org/abs/2207.05772. doi: 10 . 48550/ARXIV.2207.05772.

[2] P. J. Chia , J. Tagliabue , F. Bianchi , C. He , B. Ko , Beyond NDCG : Behavioral testing of recommender systems with RecList , in: Companion Proceedings of the Web Conference 2022 , ACM, 2022 . URL: https://doi.org/10.1145% 2F3487553 .3524215. doi: 10 . 1145/3487553.3524215.

[3] M. Schedl , The lfm-1b dataset for music retrieval and recommendation , in: Proceedings of the 2016 ACM on international conference on multimedia retrieval , 2016 , pp. 103 - 110 .

[4] W. B. Cavnar , J. M. Trenkle , et al., N-gram-based text categorization , in: Proceedings of SDAIR-94, 3rd annual symposium on document analysis and information retrieval , volume 161175 , Citeseer , 1994 .

[5] J. Carbonell , J. Goldstein, The use of mmr, diversitybased reranking for reordering documents and producing summaries , in: Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval , 1998 , pp. 335 - 336 .