RMSE: Workshop on Recommendation in Multistakeholder Environments∗ Robin Burke Himan Abdollahpouri Edward C. Malthouse Dept. of Information Science Dept. of Information Science Spiegel Research Center University of Colorado University of Colorado Northwestern University Boulder, Colorado Boulder, Colorado Evanston, Illinois KP Thai Yongfeng Zhang Squirrel AI Learning Dept. of Computer Science Shanghai, China Rutgers University New Brunswick, New Jersey ABSTRACT considerations of the retailer will also enter into the recommen- In research practice, recommender systems are typically evaluated dation function. A business may wish to highlight products that on their ability to provide items that satisfy the needs and interests are more profitable or that are currently on sale, for example. Com- of the end user. However, in many recommendation domains, the mercial recommender systems often use separate “business rules” user for whom recommendations are generated is not the only functionality to integrate such items into the personalized recom- stakeholder in the recommendation outcome. For example, fairness mendations generated through conventional means. Adding the and balance across stakeholders is important in some recommenda- retailer as a stakeholder allows such considerations to be integrated tion applications; achieving a goal such as promoting new sellers in throughout the recommendation process. a marketplace might be important in others. Such multistakeholder The workshop encouraged submissions addressing the chal- environments present unique challenges for recommender system lenges of producing recommendations in multistakeholder settings, design and evaluation, and these challenges were the focus of this including but not limited to the following topics: workshop. • The requirements of different multistakeholder applications such as: CCS CONCEPTS – Recommendation in multisided platforms • Information systems → Evaluation of retrieval results; • Social – Fairness-aware recommendation and professional topics → User characteristics. – Multi-objective optimization in Recommendation – Value-aware recommendation in commercial settings KEYWORDS – Reciprocal recommendation • Algorithms for multistakeholder recommendation including multistakeholder recommendation; fairness; discrimination; bias; multi-objective optimization, re-ranking and others. e-commerce • The evaluation of multistakeholder recommendation sys- tems. 1 WORKSHOP TOPIC • User experience considerations in multistakeholder recom- One of the defining characteristics of recommender systems is per- mendation including ethics, transparency, and interfaces for sonalization. Recommender systems are typically evaluated on their different stakeholders. ability to provide items that satisfy the needs and interests of the The RMSE 2019 workshop is a continuation of the discussion of end user. However, in many recommendation domains, the user these topics in prior RecSys workshops including Value-Aware and for whom recommendations are generated is not the only stake- Multistakeholder Recommendation (VAMS 2017) [3] and Responsi- holder in the recommendation outcome. Other users, the providers ble Recommendation (FATRec 2017 and 2018) [6, 7]. of products, and even the system’s own objectives may need to be considered when these perspectives differ from those of end users. Fairness and balance are important examples of system-level 2 WORKSHOP ORGANIZATION objectives, and these social-welfare-oriented goals may at times The program committee for the workshop included (in addition to run counter to individual preferences. Sole focus on the end user the organizers listed above): hampers developers’ ability to incorporate such objectives into • Gediminas Adomavicius, University of Minnesota recommendation algorithms and system designs. • James Caverlee, Texas A&M University In addition, in many e-commerce retail settings, recommenda- • Dietmar Jannach, University of Klagenfurt tion is viewed as a form of marketing and, as such, the economic • Toshihiro Kamishima, National Institute of Advanced Indus- trial Science and Technology (AIST) ∗ Copyright 2019 for this paper by its authors. Use permitted under Creative Commons • Rishabh Mehrotra, Spotify Research License Attribution 4.0 International (CC BY 4.0). Presented at the RMSE workshop held in conjunction with the 13th ACM Conference • Nasim Sonboli, University of Colorado Boulder on Recommender Systems (RecSys), 2019, in Copenhagen, Denmark. • Yong Zheng: Illinois Institute of Technology RecSys ’19, September 16–20, 2019, Copenhagen, Denmark Robin Burke, Himan Abdollahpouri, Edward C. Malthouse, KP Thai, and Yongfeng Zhang • Morteza Zihayat, Ryerson University for this optimization problem. The last paper was presented by [5] where authors investigated the simplification of the 3 WORKSHOP PROGRAM objective function in Groupon’s search and recommenda- tion system which is two-sided marketplace to capture the The workshop schedule included a keynote address by Ed Chi of essence of short, mid and long term benefits while preserving Google, and three paper sessions featuring 10 papers (6 long, 4 fairness and moving users forward in the customer lifecycle. short) in the following areas: Additional information about the workshop can be found at the • Fairness: defining, evaluating, and implementing different following URL: https://sites.google.com/view/rmse aspects of fairness for recommender systems. In this session, three papers were presented and discussed. In [4], Deldjoo REFERENCES et al. introduced a fairness measure based on the generalized [1] Himan Abdollahpouri and Robin Burke. 2019. Multi-stakeholder Recommenda- cross entropy to ensure the outcome of the recommendations tion and its Connection to Multi-sided Fairness. In Workshop on Recommendation in Multi-stakeholder Environments (RMSE’19), in conjunction with the 13th ACM matches a pre-defined utility for each group. Abdollahpouri Conference on Recommender Systems, RecSys’19, 2019. and Burke [1] developed a taxonomy of different types of [2] Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher. multi-stakeholder recommender systems based on the archi- 2019. The Unfairness of Popularity Bias in Recommendation. In Workshop on Recommendation in Multi-stakeholder Environments (RMSE’19), in conjunction tecture of the system. They categorized such systems into with the 13th ACM Conference on Recommender Systems, RecSys’19, 2019. 1) multi-receiver recommenders, 2) multiprovider recom- [3] Robin Burke, Gediminas Adomavicius, Ido Guy, Jan Krasnodebski, Luiz Pizzato, Yi Zhang, and Himan Abdollahpouri. 2017. VAMS 2017: Workshop on Value- menders and 3) recommenders with side stakeholders. The Aware and Multistakeholder Recommendation. In Proceedings of the Eleventh authors also showed the close connection between multi- ACM Conference on Recommender Systems. ACM, 378–379. stakeholder recommendation and multi-sided fairness. In [4] Yahshar Deldjoo, Vito Walter Anelli, Hamed Zamani, Alejandro Bellogin Kouki, and Tommaso Di Noia. 2019. Recommender Systems Fairness Evaluation via [12], Strucks et al. investigated BlurMe, a gender obfusca- Generalized Cross Entropy. In Workshop on Recommendation in Multi-stakeholder tion technique that has been shown to block classifiers from Environments (RMSE’19), in conjunction with the 13th ACM Conference on Recom- inferring binary gender from users’ profiles and proposed mender Systems, RecSys’19, 2019. [5] Joaquin Delgado, Samuel Lind, Carl Radecke, and Satish Konijeti. 2019. Simple an extension to BlurMe, called BlurM(or)e, that addresses Objectives Work Better. In Workshop on Recommendation in Multi-stakeholder the privacy issues associated with BlurMe. Environments (RMSE’19), in conjunction with the 13th ACM Conference on Recom- mender Systems, RecSys’19, 2019. • Calibration: matching recommendation output to user pref- [6] Michael D Ekstrand and Amit Sharma. 2017. FATREC Workshop on Responsible erences and examining disparities between types of users Recommendation. In Proc. ACM RecSys ’18. ACM, 382–383. https://doi.org/10. and types of items. There were four papers presented in this 1145/3109859.3109960 [7] Toshihiro Kamishima, Pierre-Nicolas Schwab, and Michael D Ekstrand. 2018. 2nd session. Tsintzou et al. [13] proposed a metric called bias FATREC workshop: responsible recommendation. In Proc. ACM RecSys ’18. ACM, disparity to measure the difference between the bias towards 516–516. https://doi.org/10.1145/3240323.3240335 different movie genres in user profiles and in recommenda- [8] Kun Lin, Nasim Sonboli, Bamshad Mobasher, and Robin Burke. 2019. Crank up the volume: preference bias amplification in collaborative recommendation. In tions. Some of the other papers in this session were inspired Workshop on Recommendation in Multi-stakeholder Environments (RMSE’19), in by this work. Mansoury et al. [11] explored how different conjunction with the 13th ACM Conference on Recommender Systems, RecSys’19, 2019. recommendation algorithms reflect the trade-off between [9] Raphael Louca, Moumita Bhattacharya, Diane Hu, and Liangjie Hong. 2019. ranking quality and bias disparity. Their experiments in- Joint Optimization of Profit and Relevance for Recommendation Systems in E- cluded neighborhood-based, model-based, and trust-aware commerce. In Workshop on Recommendation in Multi-stakeholder Environments (RMSE’19), in conjunction with the 13th ACM Conference on Recommender Systems, recommendation algorithms. Another paper in this session RecSys’19, 2019. was a work by Lin et al. [8] where authors examined bias dis- [10] Edward Malthouse, Khadija Ali Vakeel, Yasaman Hessary Kamyab, Robin Burke, parity over a range of different algorithms and for different and Morana Fuduric. 2019. A Multistakeholder Recommender Systems Algorithm for Allocating Sponsored Recommendations. In Workshop on Recommendation item categories and demonstrate significant differences be- in Multi-stakeholder Environments (RMSE’19), in conjunction with the 13th ACM tween model-based and memory-based algorithms. Finally, Conference on Recommender Systems, RecSys’19, 2019. [11] Masoud Mansoury, Bamshad Mobasher, Robin Burke, and Mykola Pechenizkiy. Abdollahpouri et al. in [2] investigated the unfairness of 2019. Bias Disparity in Collaborative Recommendation: Algorithmic Evaluation popularity bias in recommnder systems and how it affects and Comparison. In Workshop on Recommendation in Multi-stakeholder Environ- different user groups differently. They show that in many ments (RMSE’19), in conjunction with the 13th ACM Conference on Recommender Systems, RecSys’19, 2019. recommendation algorithms the recommendations the users [12] Christopher Strucks, Manel Slokom, and Martha Larson. 2019. BlurM(or)e: Revisit- get are extremely concentrated on popular items even if a ing Gender Obfuscation in the User-Item Matrix. In Workshop on Recommendation user is interested in long-tail and non-popular items showing in Multi-stakeholder Environments (RMSE’19), in conjunction with the 13th ACM Conference on Recommender Systems, RecSys’19, 2019. an extreme bias disparity. [13] Virginia Tsintzou, Evaggelia Pitoura, and Panayiotis Tsaparas. 2019. Bias Dis- • Multistakeholder Recommendation: implementing rec- parity in Recommendation Systems. In Workshop on Recommendation in Multi- stakeholder Environments (RMSE’19), in conjunction with the 13th ACM Conference ommender systems that balance interests of users and those on Recommender Systems, RecSys’19, 2019. of other stakeholders. There were three papers presented in this session. Louca et al. [9] presented an approach to jointly optimize for relevance and profit. Another paper regarding the optimization of revenue while keeping an acceptable level of accuracy for allocating sponsored content in recom- mendation was presented by Malthouse et al. [10] where they used a multi-objective binary integer programming model