A unified optimization toolbox for solving popularity bias, fairness, and diversity in recommender systems

Abstract

Historically, the main criterion for a successful recommender system was how accurate the recommendations were according to the user's taste. This emphasis on accuracy was later challenged by researchers asking for other types of metrics such as novelty, diversity, fairness of the recommendations. Researchers have proposed diferent algorithms to improve these metrics of recommendation, but the problem is that each proposed algorithm improves a certain metric (diversity, novelty, etc.) and, usually, it is dificult to improve two or more aspects simultaneously. In this paper, we unify diferent considerations into a constrained optimization framework where diferent sets of metrics can be improved by simply using diferent sets of constraints. Therefore, our framework improves the non-accuracy metrics of the recommendations by combining diferent constraints designed for separate metrics. Our biggest contribution is ofering models that are simple, easy to combine, and data independent. We create models considering popularity, fairness, and diversity metrics since they are the metrics widely investigated in the literature; however, our framework can include other metrics following the ideas proposed in this paper. Experimental results confirm that our general framework has comparable performance with the state-of-the-art methods designed for improving each individual metric, and ofers the benefit of being able to accommodate a wide range of considerations. CCS Concepts: • Information systems → Personalization; Recommender systems. Additional Key Words and Phrases: Recommender systems, optimization, popularity bias, diversity, fairness ACM Reference Format: The initial focus of recommender systems (RS) was on estimating users' preferences accurately, where measures including Root Mean Squared Error (RMSE), precision and recall were the primary objectives. Researchers later recognized the importance of other metrics such as diversity and novelty [16], fairness between multiple stakeholders [1] and so on. Various types of criteria have been recognized as important considerations for the success of a RS and for each of them numerous algorithms have been proposed. For example, for improving the fairness of the RS from the providers' perspective, algorithms such as FairRec [27] (based on fair resource allocation) and FairMatch [24] (based on a graph-based maximum flow approach), and PFAR [ 22] (based on the weighted sum of relevance and fair exposure using the Maximum Marginal Relevance approach) are proposed, each ofering a diferent approach for solving the same problem. Similarly, for mitigating the popularity bias problem Kamishima [20] uses the concept of neutrality for controlling this bias, Abdollahpouri [2] mitigates popularity bias via ensuring a balanced exposure of two groups

1 INTRODUCTION

Sinan Seymen, Himan Abdollahpouri, and Edward C. Malthouse. 2021. A unified optimization toolbox for solving popularity bias, ∗Copyright 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Presented at the MORS workshop held in conjunction with the 15th ACM Conference on Recommender Systems (RecSys), 2021, in Amsterdam, Netherlands. of popular and less popular items in each recommendation list, Vargas [32] swaps the role of items and users and change the recommendation process as if the goal is to recommend users to each item, and many others. For improving the diversity of the recommendations within each list, Zhou [33] proposes a “heat-spreading” algorithm that can be coupled in a highly eficient hybrid with a difusion-based recommendation method [ 34], Eskandanian [11] performs collaborative filtering independently on diferent segments of users based on the degree of diversity in their profiles, and Di Noia [9] uses a post-processing re-ranking technique to enhance the diversity of an initial recommendation list, and numerous other techniques.

One issue is that all the mentioned approaches for tackling diferent non-accuracy problems are implemented in isolation and cannot be easily combined to improve two or more aspects at the same time. We address this limitation by developing a constrained optimization toolkit that addresses popularity, fairness, and diversity metrics. Our framework is easy to implement and incorporate other metrics. We believe unifying all diferent non-accuracy related problems in RS under one umbrella can greatly benefit the research community and therefore we study how to create a list of items for each user, assuming the preferences of each user for each item have already been estimated using some existing RS. We show how to write various considerations (e.g., diversity, popularity, fairness) as an optimization problem, and show how it can be solved as a post-processing step. We show that our toolkit achieves a comparable performance to the best-in-class algorithms for each specific task, but our toolkit is also able to improve more than one non-accuracy aspect of the recommendations by combining diferent constraints designed for separate aspects. 2

THE CONSTRAINED OPTIMIZATION TOOLBOX

Optimization models are applied to RS in diferent forms. Rodriguez [ 29] formulates recommendations as a constrained optimization problem and proposes the TalentMatch algorithm that matches job candidates to job posts. Another early paper using optimization with RS is Ribiero [28], which searches a Pareto frontier balancing accuracy, diversity and novelty. Jugovac [19] proposes a multi-objective, post-processing model, reviews the literature on multi-objective RS, and tests diferent heuristic solutions. Sürer [ 31] proposes integer programming models to solve RS by recommending items from stakeholders (providers) in the system in a suficient amount for fairness. Antikacioglu and Ravi [ 7] use a graph optimization approach to increase diversity of the recommendation lists. Similarly, in [4], aggregate diversity is increased by graph-theoretic approach. Gogna and Majumdar [13] use regularization terms in the objective function to increase the diversity and the novelty of the solution. Other multi-objective optimization models [5, 6] are implemented to solve content recommendation problems. Works [5, 6] consider an objective function that maximizes the probability of recommendations using continuous decision variables. In another line of work, Jambor and Wang [18] propose a constrained linear optimization model for increasing the long-tail item recommendations. Most of these approaches have been tailored to solve particular RS problems, while we aim at unifying diferent non-accuracy aspects of the recommendations into a simple and flexible optimization approach.

This section describes our approach to solve diferent problems such as popularity bias, provider fairness, and diversity in RS using constrained optimization. For all problems, our technique maximizes the same objective function: the average ratings across all user and item pairs in the recommendations (i.e., the relevance of the recommendations). Diferent problems are addressed by adding diferent types of constraints. Thus, all problems have a similar structure, making it very easy to use and understand.

In the literature, some works [10, 12] have recently investigated problems including more than one non-accuracy metric. However, it is not easy to modify these models to remove some metrics and include others. Most of the time, these algorithms need significant changes to be able to incorporate diferent metrics other than ones that are already Manuscript submitted to ACM proposed. We suggest a framework that alleviates this problem, where diferent metrics can be easily mixed and matched. In other words, our approach is inspired by how one can create a diferent oatmeal each morning by simply using diferent toppings to the base oats: the relevance objective is the base and diferent types of constraints are the toppings. In the following subsections, we discuss the our optimization model in more details. 2.1

Base Top- Model

We now formalize the toolkit, beginning with notation. Let denote the set of users and be the set of items in the system. Suppose items are to be recommended to each user. We assume that the ratings have been predicted with some existing algorithm, with representing the predicted rating for user and item . Decision variables indicate which items are recommended, with = 1 if item is recommended to user , and 0 otherwise. Our base model has an objective to maximize the average predicted ratings of all recommended items, subject to the constraint that each user receives recommendations. We can write this as an optimization problem as follows:

1 Õ max

Õ | | ∈ ∈ ( 1 ) subject to: Õ = ∈ Note that equals for recommended items and 0 otherwise, and therefore their sum divided by the number of recommendations made by the system ( | |) gives the average rating. Constraint ( 2 ) forces the model to recommend items to every user . Constraint ( 3 ) forces decision variables to be binary (an item is either recommended or not). This problem can be solved eficiently by sorting items for each user in descending order of predicted ratings, and then selecting the top items for every user. Both the objective function and constraints are used in the upcoming models. Therefore, we can consider this Top- model as the base, and add constraints according to the needs of the system. 2.2

Popularity Model

Many RS have a well-known bias to recommend popular items frequently and not give enough exposure to the majority of other, less popular, items [2]. This bias can be avoided with our popularity optimization model (Pop-Opt), which extends the base by adding a constraint to limit the aggregate popularity of all recommended items to a given user. We implement this idea by putting an upper bound ( ) on the total popularity of the recommended items. We have the same objective function in ( 1 ) subject to constraints ( 2 ), ( 3 ), and Õ Õ ≤ , ( 4 ) ∈ ∈ where measures the popularity of item as the ratio of the number of ratings item received to the total number of ratings of all items in the system. Constraint ( 4 ) sums the popularity values of the recommended items and forces the sum to be at most , a tuning parameter that can be adjusted based on the needs of the system. At one extreme, if is very large then the selected items can be popular without exceeding threshold and the system can focus on maximizing the average ratings. As we decrease , the system is forced to make trade-ofs and recommend some items with equal or lower ratings that are also less popular (more novelty). Choosing values for is very intuitive. If we somewhat care about popularity, the average of times | | can be used as a starting value. Select a smaller value of to ofer less popular items. Constraint ( 4 ) is called a knapsack constraint in the literature [25], and one big advantage Manuscript submitted to ACM of using this simple structure is that of-the-shelf optimization programs such as Gurobi are very eficient in solving this common structure. We evaluate the model with the average recommended popularity over all lists ( ), and aggregate diversity (Agg. Div.), which is the number of unique recommended items [3]: = 1 Õ Õ , | | ∈ ∈ | | ( 5 )

Agg. Div. = 1

Ø , | | ∈ ( 6 ) where is the set of all items recommended to user . Smaller values of are desirable because they indicate lower popularity (more novelty). Higher values of Agg. Div is desirable because it shows the algorithm has covered a larger number of unique items in its recommendations.

2.3 Provider Fairness Model

In multi-stakeholder contexts such as a retail platform, provider fairness ensures that diferent providers (e.g., vendors) receive some minimum threshold number of recommendations. We assume that items are partitioned into groups. For example, items on a retail platform could be grouped by vendor or news articles could be grouped by publisher (e.g., Fox News, MSNBC, CNN, etc.). Let be the set of item indices in group ∈ , where is the set of all groups. Similar to Pop-Opt, provider fairness can be expressed as a constraint. Our provider fairness optimization (Fair-Opt) model uses the same objective function ( 1 ) as the base, subject to constraints ( 2 ), ( 3 ), and a new one that imposes a lower and upper bound (both can be tuned by the system designer) on the number of times items recommended from each group: ≥ Õ Õ ≥ (∀ ∈ ) ∈ ∈ ( 7 ) where = | | · | | is the fraction of items in group times the total number of items recommended. Tuning parameters |a n|d control the lower and upper bounds for the number of times items recommended from group .

The literature on fairness usually only considers the lower bound [27, 31]. Without upper bounds, however, some items can be ofered significantly more frequently than the rest, which creates an unfair distribution of recommendations across items. We choose upper bound parameter as ⌊1 + (2 − ) ⌋, which depends on . Diferent upper bounds can be selected according to the needs of the system.

We now discuss the selection of the number of groups. On one extreme, there could be one group with |1 | = | | and 1 = | |, which is the total items recommended, and the constraint would have no efect (for reasonable values of 1 and 1). The other extreme is where each item is a separate group, i.e., every provider has one item in the system. This case is called the item fairness problem, where all = , ∀.

We measure fairness with (Inequality in Producer Exposures) [27], which is defined as follows: = − Õ∈ || log| | || , ( 8 ) where = Í ∈ 1( ∈ ) is the total number of recommendations from group . This metric is 1 when every provider gets the same number of recommendations, and decreases as the disparity between providers increases. 2.4

Diversity Model

Another consideration in creating top- lists is to have diversity in that the recommended items are not highly similar to each other [35]. Our approach is to put items in distinct groups (based on their topic, genre, etc.), and constrain the number of distinct groups represented in each top- list to be at least , which is another tuning parameter under the control of the system designer. Our diversity optimization model (Div-Opt) is the same as the base model, i.e., objective ( 1 ) subject to ( 2 ) and ( 3 ), with additional constraints: Õ ≥ (∀ ∈ , ∈ ) ∈ ( 9 ) Õ ≥ ∈ We introduce a new decision variable that indicates whether any items from group are recommended to user . Constraint ( 11 ) guarantees that the new decision variable only takes values 0 or 1. Constraint ( 9 ) ensures that at least one recommendation is made from category for user , can get value of 1. Otherwise, it is always fixed to 0. Constraint ( 10 ) ensures that the top- list for each user includes at least distinct categories.

Note that in the Div-Opt formulation every item belongs to exactly one group, which we call the binarized version. This problem can be solved in an eficient manner through sorting [ 8, 23]. One solution is to sort through every category separately according to the estimated ratings of the users, and recommend at least items from separate categories. Next, we recommend items according to highest ratings until every user has exactly items in their lists. We need Div-Opt when we mix and match diferent considerations to optimize combinations of metrics at the same time.

Our metric of choice for the diversity is ILS (Intra-list similarity) [35], defined as follows:

ILS = 1 Õ Õ Õ (, ′) , ( 12 ) | | ∈ ∈ ′ ∈ | | (| | − 1) ′≠ where is the recommendation list of user , and (, ′) is the distance between two items in the same list. We slightly modify the metric in [35], to put our ILS values between 0 and 1, where low values are desirable. 2.5

Combining diferent objectives

We combine all the diferent parts of the optimization models from the previous subsections. Unlike existing approaches, the beauty of our toolbox for solving diferent non-accuracy aspects of RS is that all constraints introduced so far can be included together. Our combined model (Comb-Opt) has the same objective function in ( 1 ) subject to constraints ( 2 ), ( 3 ), ( 4 ), ( 7 ), ( 9 ), ( 10 ), and ( 11 ).

Our models use diferent ideas from the constrained optimization literature, including upper and lower bounding in Fair-Opt, weighted sums in Pop-Opt, and auxiliary variables in Div-Opt. Using these ideas, readers can include their own metrics with small modifications. Our framework can therefore accommodate diferent metrics if they can be modeled as constraints. Similarly, the metrics we have investigated can be modeled diferently. Our main consideration is to use the same objective function for all the models and keep the constraints as simple as possible.

We now discuss some shortcomings of the combined model and provide potential ways of approaching them. First, some values of parameters , , , may be infeasible, which means that there does not exist any solution satisfying all the constraints at the same time. One solution is to grid-search diferent parameters. Another solution is to penalize these constraints on the objective function whenever they are not satisfied. Second, creating decision variables for large

RESULTS

models can require a lot of memory. In that case, some sort of approximation is required to make the models smaller. One way to do this is to fix some of the decision variables to 0 or 1 according to their estimated utilities before starting the optimization. Some item-user pairs with very low predicted ratings can be ignored at the start of the process, so they basically require no memory. To keep the models and the discussion compact, we leave these for future work. In this section, we compare our optimization model for solving the non-accuracy aspects of RS (popularity bias, diversity, and provider fairness) with various other models proposed specifically to solve each individual problem. We use the MovieLens 1M data set [15] since it contains the genre of each movie, which is needed for the diversity problem. If a movie has more than one genre then we randomly assign one out of listed genres and keep the assignments consistent throughout. All the problems are solved using a laptop with Intel(R) Core (TM) i7-8750H CPU @ 2.20GHz 2.21 GHz, processor information, and with 16.0 GB of installed RAM. We use the Gurobi software [14] with optimization gap 10−4 throughout. Gurobi uses the branch & bound (B&B) algorithm, and it can have exponential time complexity [26] in worst-case scenario. However, Gurobi includes heuristics on top of B&B, and in practice the solution time performance is significantly better than exponential.

We apply the Singular Value Decomposition (SVD) [21] method implemented in the Python Surprise package [17] to estimate ratings for every user-item pair, although any RS algorithm could be used. We solve the optimization problems as a post-processing step using the predicted rating matrix. All our optimization models and benchmarks are solved using the same predicted ratings matrix. We set the size of the recommendation list given to each user to = 10. Our code is available at GitHub: Ghttps://github.com/sseymen-tech/unified_toolbox. 3.1

Popularity Bias

We compare our Pop-Opt technique with the approach proposed in [2] (we label it xQuAD), which is one of the most eficient approaches proposed in recent years for controlling popularity bias. Figure 1 shows the average popularity of the recommendations ( ) and the aggregate diversity for Pop-Opt and xQuAD for diferent values of . From left to right, changes from 0.03 to 0.16 with increments of 0.01. Larger values give the same result because Constraint ( 4 ) is satisfied trivially. We can see that, for larger values of , the Pop-Opt achieves a comparable average popularity to the xQuAD model with even slightly better average rating. Regarding aggregate diversity, we also see that our model outperforms xQuAD for larger values of which, as we mentioned before, is a tunable parameter. Thus, Pop-Opt can achieve a comparable performance with the state-of-the-art technique to mitigate popularity bias. 3.2

Provider Fairness

For provider fairness, we compare our Fair-Opt with FairRec [27], which is specifically designed for this task. Figures 2 and 3 show the fairness metric and average rating for both techniques. From left to right the parameters used in both algorithms are (0.99, 0.95, 0.9, 0.8, 0.5, 0.3, 0). We observe that for both relaxed and strict upper bound choices, Fair-Opt beats FairRec in both average rating and fairness metrics. When we compare Fair-Opt results with and without upper bounds in Figures 2 and 3, we notice that upper bounds improve fairness value without reducing the average rating of the recommendations.

Model

Pop-Opt 0.04 P0.08 R A0.12

3.3 Diversity

We compare our diversity model (Div-Opt) with one of the most common ways of improving diversity in recommendation lists that uses a simple weighted sum of relevance and Intra List Similarity (ILS) [8] (It is denoted by in the plot). In Figure 4, from left to right Div-Opt parameter takes values ( 10, 9, 8, 7, 6, 5, 0 ). From 5 to 0 solutions are the same. for = 10, we achieve ILS=0, because every user is recommended one item from every distinct category. Thus, Div-Opt can increase the number of distinct recommended genres to every user without a significant decrease in average rating. Div-Opt has achieved a comparable performance to the algorithm when = ( 9, 8, 7, 6, 5, 0 ) yet outperforms it for = 10 as its ILS is close to zero but has a higher average rating.

3.4 Combined Model Results

Table 1 exhibits the results of our Comb-Opt model. Algorithms that optimize for a specific metric should perform well on that metric, but Comb-Opt can achieve great performance on ILS, and without significantly lowering the average rating value. The results for xQuAD with lowest and highest popularity metric (highest and lowest average rating respectively) are reported. Similarly, the results for for lowest and highest ILS are reported. For the Manuscript submitted to ACM

Model

Div-Opt 0 0.03 ILS00..0069 0.12 4.55 4.60 4.65 4.70

Avg. Rating Fig. 4. Diversity Results item fairness metric, we remove upper bounds from (no parameter) both FairRec and Comb-Opt, except with the solutions with superscript ∗, which represent the solutions with upper bounds.

Model

Comb-Opt

Comb-Opt∗ Comb-Opt Comb-Opt Comb-Opt Comb-Opt Comb-Opt Comb-Opt Comb-Opt

Top- FairRec FairRec FairRec∗ xQuAD xQuAD

WS WS of FairRec∗. Comb-Opt( = 7, = 0.1, = 0.08) beats both xQuAD solutions in all metrics except average rating. Comb-Opt loses a bit of average rating but improves all the other metrics.

Recall that high , Agg. Div., and ARP values, and low and ILS values are desired. In our experiments with Comb-Opt, we see that all metrics can be improved simultaneously. If one does not care about a certain metric, the constraints for that metric can be ignored in the combined model. Parameters can overwrite each other, for example in Comb-Opt with = 1, = 0.9 and = 0.09, we have a relatively low = 0.067 when our bound is 0.09. This happens because when goes to 1, every item is recommended proportionally close to each other, which gives non-popular items to be recommended as frequently as popular ones. Therefore, the = 0.09 bound is trivially satisfied by the fairness constraints. However, this is normal since we are dealing with diferent problems at the same time, and it is expected that they have some efect on each other.

Parameter choice plays a significant role, and while optimizing more than one metric, grid-search of parameters are suggested to see the behavior of the data. Comb-Opt with parameters = 7, = 0.5, and = 0.08, for example, seem to find a good balance of all metrics.If some metrics are not important in the given problem, their corresponding constraints can be discarded. For example, if diversity is not important, Comb-Opt with parameters = 1, = 0.9, and = 0.09 finds a solution with good fairness and popularity metrics at the same time. Overall, parameter choice can be made according to the needs of the system.

Overall, Comb-Opt tends to improve all the metrics at the same time. If the RS goal is to alleviate popularity bias, fairness, and diversity metrics, we suggest adding all the constraints proposed in this paper. However, if one does not care about fairness, then it makes sense to remove the fairness constraints from the Comb-Opt. Then we immediately have a solution that can alleviate popularity bias and diversity metrics. On the other hand, the constraints proposed in this paper can be removed and new ones can be added easily, according to the needs of the specific RS. 4

CONCLUSIONS AND DISCUSSION

We propose an optimization toolbox for solving diferent types of non-accuracy problems. This method focuses on ifnding the optimal solution of the system given diferent constraints, which aim to solve various non-accuracy problems such as popularity bias, provider fairness, and diversity. We show that, all these diferent metrics can be considered at the same time while generating recommendations, and they can even beat algorithms specialized for the specific problems.

One downside of our model is its memory requirement. Therefore, for larger problems, scaling of the models is an issue. There are possible solutions to these problems, such as focusing on sub-optimal solutions which are close to optimal, fixing values to some decision variables before starting to optimization and so on. We leave the problem of ifnding good approximations of Comb-Opt for future work.

Our toolbox can be applied to many other problems. For example, retailers concerned with stock-outs can add upper bounds for how often an item is ofered. Likewise, a platform with perishable items (e.g., fresh produce or meat, hotel rooms, airline seats) could add a lower bound so they are ofered more frequently. In situations with sponsored recommendations, the manufacturer of the item may have a maximum advertising budget that it is willing to spend, which could be handled by adding upper bounds. Seymen [30] applies a similar approach to the top- list calibration problem. All these individual problems and various combinations of them and can easily be solved simply by modifying the constraints or adding new ones. Thus, post-processing optimization models ofer a flexible toolkit for managing RS involving multiple objectives and/or stakeholders.

References

[1] Himan Abdollahpouri , Gediminas Adomavicius, Robin Burke, Ido Guy, Dietmar Jannach, Toshihiro Kamishima, Jan Krasnodebski, and Luiz Pizzato . 2020 . Multistakeholder recommendation: Survey and research directions . User Modeling and User-Adapted Interaction 30 , 1 ( 2020 ), 127 - 158 .

[2] Himan Abdollahpouri , Robin Burke, and Bamshad Mobasher . 2019 . Managing popularity bias in recommender systems with personalized re-ranking . In The Thirty-Second International Flairs Conference.

[3] Gediminas Adomavicius and YoungOk Kwon . 2011 . Improving aggregate recommendation diversity using ranking-based techniques . IEEE Transactions on Knowledge and Data Engineering 24 , 5 ( 2011 ), 896 - 911 .

[4] Gediminas Adomavicius and YoungOk Kwon . 2011 . Maximizing aggregate recommendation diversity: A graph-theoretic approach . In Proc. of the 1st International Workshop on Novelty and Diversity in Recommender Systems (DiveRS 2011 ). Citeseer, 3 - 10 .

[5] Deepak Agarwal , Bee-Chung Chen , Pradheep Elango, and Xuanhui Wang . 2011 . Click shaping to optimize multiple objectives . In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining . 132 - 140 .

[6] Deepak Agarwal , Bee-Chung Chen , Pradheep Elango, and Xuanhui Wang . 2012 . Personalized click shaping through lagrangian duality for online recommendation . In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval . 485 - 494 .

[7] Arda Antikacioglu and R Ravi . 2017 . Post processing recommender systems for diversity . In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . 707 - 716 .

[8] Keith Bradley and Barry Smyth . 2001 . Improving recommendation diversity . In Proceedings of the Twelfth Irish Conference on Artificial Intelligence and Cognitive Science, Maynooth, Ireland , Vol. 85 . Citeseer , 141 - 152 .

[9] Tommaso Di Noia , Jessica Rosati, Paolo Tomeo, and Eugenio Di Sciascio. 2017 . Adaptive multi-attribute diversity for recommender systems . Information Sciences 382 ( 2017 ), 234 - 253 .

[10] Jorge Díez , David Martínez-Rego, Amparo Alonso-Betanzos, Oscar Luaces , and Antonio Bahamonde. 2019 . Optimizing novelty and diversity in recommendations . Progress in Artificial Intelligence 8 , 1 ( 2019 ), 101 - 109 .

[11] Farzad Eskandanian , Bamshad Mobasher, and Robin Burke . 2017 . A clustering approach for personalizing diversity in collaborative recommender systems . In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization . 280 - 284 .

[12] Ruoyuan Gao and Chirag Shah . 2020 . Toward creating a fairer ranking in search engine results . Information Processing & Management 57 , 1 ( 2020 ), 102138 .

[13] Anupriya Gogna and Angshul Majumdar . 2017 . DiABlO: Optimization based design for improving diversity in recommender system . Information Sciences 378 ( 2017 ), 59 - 74 .

[14] Gurobi Optimization , LLC. 2021 . Gurobi Optimizer Reference Manual . https://www.gurobi.com

[15] F Maxwell Harper and Joseph A Konstan . 2015 . The movielens datasets: History and context . Acm transactions on interactive intelligent systems (tiis) 5 , 4 ( 2015 ), 1 - 19 .

[16] Jonathan L Herlocker , Joseph A Konstan, Loren G Terveen, and John T Riedl. 2004 . Evaluating collaborative filtering recommender systems . ACM Transactions on Information Systems (TOIS) 22 , 1 ( 2004 ), 5 - 53 .

[17] Nicolas Hug . 2017 . Surprise, a Python library for recommender systems . http://surpriselib.com.

[18] Tamas Jambor and Jun Wang . 2010 . Optimizing multiple objectives in collaborative filtering . In Proceedings of the fourth ACM conference on Recommender systems. 55-62.

[19] Michael Jugovac , Dietmar Jannach, and Lukas Lerche . 2017 . Eficient optimization of multiple recommendation quality factors according to individual user tendencies . Expert Systems with Applications 81 ( 2017 ), 321 - 331 .

[20] Toshihiro Kamishima , Shotaro Akaho, Hideki Asoh, and Jun Sakuma . 2014 . Correcting Popularity Bias by Enhancing Recommendation Neutrality. . In RecSys Posters.

[21] Yehuda Koren , Robert Bell, and Chris Volinsky . 2009 . Matrix factorization techniques for recommender systems . Computer 42 , 8 ( 2009 ), 30 - 37 .

[22] Weiwen Liu and Robin Burke . 2018 . Personalizing fairness-aware re-ranking . In International Workshop on Fairness Accountability and Transpatency in Recommender Systems (FACTREC) . arXiv preprint arXiv: 1809 .02921.

[23] Edward C Malthouse , Khadija Ali Vakeel, Yasaman Kamyab Hessary, Robin Burke, and Morana Fuduric . 2019 . A Multistakeholder Recommender Systems Algorithm for Allocating Sponsored Recommendations. . In RMSE@ RecSys.

[24] Masoud Mansoury , Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke . 2020 . Fairmatch: A graph-based approach for improving aggregate diversity in recommender systems . In Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization . 154 - 162 .

[25] Silvano Martello and Paolo Toth . 1987 . Algorithms for knapsack problems . In North-Holland Mathematics Studies . Vol. 132 . Elsevier , 213 - 257 .

[26] David R Morrison , Sheldon H Jacobson, Jason J Sauppe, and Edward C Sewell. 2016 . Branch-and-bound algorithms: A survey of recent advances in searching, branching, and pruning . Discrete Optimization 19 ( 2016 ), 79 - 102 .

[27] Gourab K Patro , Arpita Biswas, Niloy Ganguly, Krishna P Gummadi, and Abhijnan Chakraborty . 2020 . Fairrec: Two-sided fairness for personalized recommendations in two-sided platforms . In Proceedings of The Web Conference 2020 . 1194 - 1204 .

[28] Marco Tulio Ribeiro , Anisio Lacerda, Adriano Veloso, and Nivio Ziviani . 2012 . Pareto-eficient hybridization for multi-objective recommender systems . In Proceedings of the sixth ACM conference on Recommender systems. 19-26.

[29] Mario Rodriguez , Christian Posse , and Ethan Zhang . 2012 . Multiple objective optimization in recommender systems . In Proceedings of the sixth ACM conference on Recommender systems. 11-18.

[30] Sinan Seymen , Himan Abdollahpouri, and Edward C Malthouse. 2021 . A Constrained Optimization Approach for Calibrated Recommendations . In Proc. of the 15th ACM Conf. on Rec. Sys. ACM.

[31] Özge Sürer , Robin Burke, and Edward C Malthouse. 2018 . Multistakeholder recommendation with provider constraints . In Proceedings of the 12th ACM Conference on Recommender Systems . 54 - 62 .

[32] Saúl Vargas and Pablo Castells . 2014 . Improving sales diversity by recommending users to items . In Proceedings of the 8th ACM Conference on Recommender systems. 145-152.

[33] Tao Zhou , Zoltán Kuscsik, Jian-Guo Liu , Matúš Medo, Joseph Rushton Wakeling, and Yi-Cheng Zhang. 2010 . Solving the apparent diversity-accuracy dilemma of recommender systems . Proceedings of the National Academy of Sciences 107 , 10 ( 2010 ), 4511 - 4515 .

[34] Tao Zhou , Jie Ren, Matúš Medo, and Yi-Cheng Zhang. 2007 . Bipartite network projection and personal recommendation . Physical review E 76 , 4 ( 2007 ), 046115 .

[35] Cai-Nicolas Ziegler , Sean M McNee , Joseph A Konstan, and Georg Lausen . 2005 . Improving recommendation lists through topic diversification . In Proceedings of the 14th international conference on World Wide Web . 22 - 32 .