A systematic review of the use of multiple criteria decision aiding methods in recommender systems: preliminary results∗ RENATA PELISSARI, School of Applied Sciences at the University of Campinas, Brazil PAULOS S.C. ALENCAR, Computer Science Department at the University of Waterloo, Canada SARAH BEN AMOR, Telfer School of Management at the University of Ottawa, Canada LEONARDO TOMAZELI DUARTE, School of Applied Sciences at the University of Campinas, Brazil Multiple Criteria Decision Making (MCDA) methods have been increasingly applied to improve recommendations when multiple criteria are considered in Recommender Systems (RSs). This study presents the preliminary results of a systematic literature review, following Kitchenham’s guidelines, regarding the application of MCDA methods in RSs over the last two decades. Based on our findings, MCDA methods can be applied in two RS phases: the preference elicitation and the recommendation phases. In the former, RSs usually have a strong interaction with the user, which results in more personalized recommendations, ensuring higher user satisfaction and contributing to address the cold-start challenge in RSs. Regarding the recommendation phase, while most RSs are based on ranking approaches, there is a trend to apply sorting methods in order to avoid an additional step involving a filtering application that selects a subset of alternatives. Future research could focus on applying preference learning combined with MCDA methods for exploring improvements in prediction and recommendation phases, and also in quality and processing time. Additional Key Words and Phrases: Multiple Criteria Decision Making, Literature review, Cold start, Preference learning 1 INTRODUCTION While the majority of existing Recommender Systems (RSs) depend only on one single criterion rating as input information, there has been an increasing interest over the last two decades in taking into consideration a rating based on multiple criteria, since the user’s preferences might cover more than one perspective [2, 3]. Thus, the recommendation process can be approached as a Multi-criteria Decision Aiding (MCDA) problem [24, 42], in which MCDA methods are used as part of the recommendation algorithm. In MCDA, alternatives are evaluated by the decision-maker (DM) according to several criteria, usually conflicting to each other, with the goal of either ranking the alternatives (ranking problematic) or sorting them into predefined and ordered categories (sorting problematic) [51]. MCDA methods are characterized mainly by proposing an aggregation procedure in which preferences of the DM are taking into account through the setting of criteria weights that represent comparative importance among the criteria. By making an analogy between MCDA and RSs, in MCDA the DM corresponds to the user, the alternatives correspond to the items, and the ranking of the alternatives corresponds to an ordered list of recommended items. RSs based on multiple criteria have been pointed out as one of the promising research areas in RSs [3]. Since then, many studies have considered merging MCDA and RSs. Towards this direction, [42] proposed a review in order to analyze and classify MCDA-based RSs. As this study was published 2007, there is an opportunity for an up-to-date review. ∗ Copyright 2022 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Presented at the MORS workshop held in conjunction with the 16th ACM Conference on Recommender Systems (RecSys), 2022, in Seattle, USA. Authors’ addresses: Renata Pelissari, School of Applied Sciences at the University of Campinas, Limeira, Brazil, renatapelissari@gmail.com; Paulos S.C. Alencar, Computer Science Department at the University of Waterloo, Waterloo, Canada, palencar@uwaterloo.ca; Sarah Ben Amor, Telfer School of Management at the University of Ottawa, Ottawa, Canada, BenAmor@telfer.uottawa.ca; Leonardo Tomazeli Duarte, School of Applied Sciences at the University of Campinas, Limeira, Brazil, leonardo.duarte@fca.unicamp.br. Manuscript submitted to ACM 1 2 Pelissari, Alencar, Ben Amor, Duarte Fig. 1. Phases of the methodology adopted for conducting the literature review. Given this context, we propose to perform a systematic literature review of the use of multiple criteria decision aiding methods in RSs. The remainder of the paper is structured as it follows. In section 2, we present the adopted methodology for conducting the literature review. The results are presented in Section 3, and the conclusions and future improvements are presented in Section 4. 2 METHODOLOGY Our study followed the guidelines for undertaking systematic reviews proposed by [35]. The set of Kitchenham’s guidelines is a well-defined approach to identify, evaluate and interpret all relevant studies regarding a particular research question, topic area or phenomenon of interest. Following Kitchenham’s guidelines, a research protocol must be defined and must contain the generic steps listed as follows: (1) definition of research questions that the review is intended to answer, an that will guide the study; (2) definition of the search strategy, including the databases and the search terms used to identify and select papers; (3) definition of the study selection criteria, determining criteria for excluding a study from the review; (4) definition of how to categorize the studies, which information shall be extracted from the studies, and how this information will be synthesized and analyzed. The protocol definition can be seen as the first phase of the literature review, which is followed by two more phases, the review conduction and the review report, as illustrated in Figure 1. 2.1 Phase 1: Protocol definition The first step of the protocol definition starts with the selection of research questions that will guide the study. This review examines the following seven sets of research questions: Manuscript submitted to ACM A systematic review of the use of multiple criteria decision aiding methods in recommender systems: preliminary results 3 RQ1. How frequently have MCDA methods been implemented in the use or research of RSs over the last 20 years? What are the trends in terms of publication venues? RQ2. What are the MCDA methods employed in RSs? And which are employed most? RQ3. What purpose do the MCDA methods serve in RSs? And what are the advantages and disadvantages of employing MCDA methods in RSs? RQ4. What are the main application domains of RSs combined with MCDAs? RQ5. What are the evaluation metrics employed in the selected studies? RQ6. What are the contributions of MCDA methods to RSs regarding some of their challenges such as cold-start, data sparsity, scalability, and privacy? RQ7. What are the trends and gaps in the use or research of MCDA methods in RSs? We focused our search on Scopus, since it is the largest curated, peer-reviewed abstract and indexing database available to academia. In order to precisely answer the defined research questions, we conducted an initial study to identify the most suitable query to be used in the paper search, by conducting pilot searches to ensure that the used keywords provided the right scope. Initially, we developed a set of keywords considering the main terms that different communities use to refer to recommender systems (“recommendation systems", “recommendation system", “recommender systems", “recommender system") and to MCDA methods (“multiple criteria decision making”, “multiple criteria decision aiding”, “multiple criteria decision analysis”, “multi-criteria decision making”, “multi criteria decision-making”, “multi criteria decision aiding”). Taking into consideration papers that apply MCDA to RSs and were already known by the authors, we realized that many important papers were not found in this initial search. Deepening our initial study, we verified that many papers have mentioned in their title, abstract or keywords, only specific words related to the MCDA method used such as “AHP” and “TOPSIS”, and not more generalized words. Therefore, we considered the acronyms of the most known MCDA methods as keywords in our search, even at the risk of not being comprehensive enough, since not all existent MCDA-method names were included. The final list of keywords used in our search is presented in Table 1. Table 1. Query used to search papers. Search query (“recommender system” OR “recommendation system” OR “recommender systems” OR “recommendation systems”) AND (“multicriteria decision making” OR “multiple criteria decision making” OR “multi-criteria decision making” OR “multicriteria decision analysis” OR “multiple criteria decision analysis” OR “multi-criteria decision analysis” OR “multicriteria decision aiding” OR “multiple criteria decision aiding” OR “multi-criteria decision aiding” OR promethee OR electre OR ahp OR vikor OR topsis OR anp OR uta OR utadis OR maut OR dematel OR “multi-attribute utility” OR “multiattribute utility” ) We were only interested in the publications of the last twenty years describing either an application or a theoretical development, once an MCDA method has been used as a core or at least as an important part of the system. Based on that, we defined the following exclusion criteria: EC1. Papers published later than 2002. EC2. Papers written in languages other than English. EC3. Non-primary studies., e.g, literature surveys. EC4. Papers whose document type is other than “Article”, e.g, conference papers, book chapters, conference reviews, etc. Manuscript submitted to ACM 4 Pelissari, Alencar, Ben Amor, Duarte EC5. Papers whose abstract does not provide enough information in order to verify whether the paper is related to the review goal. EC6. Papers that do not describe or consider an RS approach. EC7. Papers that do not describe or consider an MCDA method. EC8. Papers that do not apply MCDA methods as a core or at least as an important part of the system, e.g, application of MCDA methods for tool selection. 2.2 Phase 2: Review conduction On March 17, 2022, we queried the digital library Scopus using the search terms presented in Table 1, and our search returned 313 papers. The exclusion criteria from EC1 to EC4 were automatically applied using search resources from the Scopus database itself, reducing the number of studies from 313 to 135. Abstracts of the remaining papers were read, and the exclusion criteria from EC5 to EC8 were manually applied. In the end, 96 studies were retained. The 96 selected papers were then read in full. Throughout the reading process, papers that met at least one of the exclusion criteria EC6 to EC8, and that were not identified in the previous step, were excluded. From these, 49 studies were finally selected. 3 RESULTS In this section, we present the main results of our literature review based on the defined research questions. RQ1. How frequently have MCDA methods been implemented in the use or research of RSs over the last 20 years? What are the trends in terms of publication venues? Figure 2 shows the number of studies applying MCDA methods to RSs over the last two decades. We can see continuing growth over time in the number of published papers, with a clear increase from 2018 on. About 60% (29 papers) of the total number of selected papers have been published in the last 4 years. Fig. 2. Number of publications of studies regarding MCDA in RSs over the years. Manuscript submitted to ACM A systematic review of the use of multiple criteria decision aiding methods in recommender systems: preliminary results 5 The papers have been published in 47 different journals, indicating that there are not specific journals concentrating on the topic discussed here. The journal with the most published papers is IEEE Access, with 4 papers, followed by Expert Systems with Applications, Applied Artificial Intelligence, Electronic Commerce Research and Applications, with 2 papers each. RQ2. What are the MCDA methods employed in RSs? And which are employed most? We could verify that most of the known MCDA methods have been employed in the RS context. Table 2 shows the number of papers that applied each method and their references. As a paper may have applied more than one method, the total number of papers presented in Table 2 is greater than the number of selected papers. Table 2. MCDA methods employed in the RS context. Type of MCDA method MCMA method Number of papers References Pairwise comparison AHP 17 [5, 10–12, 16, 25, 30, 31, 39, 44–46, 50, 57, 58, 60, 61] Distance-based TOPSIS 11 [4, 7, 8, 14, 17, 20, 25, 38, 49, 53, 59] Outranking PROMETHEE 5 [6, 9, 52, 52, 54] Utility-based MAUT/ UTA 5 [1, 15, 36, 41, 43] Outranking ELECTRE 4 [27, 28, 34, 56] Utility-based Choquet integral 3 [18, 26, 28] Scoring WSM 3 [30, 33, 55] Utility-based SMARTER 1 [29] Scoring Dominance intensity 1 [40] Other Consensus method 1 [37] Distance-based VIKOR 1 [23] RQ3. What purpose do the MCDA methods serve in RSs? And what are the advantages and disadvantages of employing MCDA methods in RSs? Generally speaking, MCDA methods have been applied to RSs in order to model multiple criteria and to take user preferences into consideration, thus providing more personalized and accurate recommendations. However, depending on the applied method, other reasons were identified. The AHP method is applied mainly to estimate the users’ preferences regarding criteria weights [10–12, 16, 30, 39, 44–46, 50, 57, 58, 58, 60]. Methods such as AHP and ELECTRE-TRI-B are also apply for the purpose of organizing the criteria hierarchically [12, 13, 25, 48]. When the decision problem is based on a large set of features, it is difficult to solve the problem using a single step where all criteria are taken into account simultaneously, and organizing the criteria hierarchically is a better way to model the problem. Utility-based methods have also been highly used in RSs. For instance, [36] proposes to model user preferences together with the collaborative-filtering technique by applying the UTA method. Similar ideas are proposed in [1, 15, 41, 43]. By using these frameworks, the problem of cold-start can be addressed since preferences (utilities) of new users are learned from previous users. The main disadvantage in applying MCDA methods in RSs pointed out by many studies is related to processing and system response time, which leads to dissatisfaction due to high interaction and slowness. RQ4. What are the main application domains of RSs combined with MCDA? Manuscript submitted to ACM 6 Pelissari, Alencar, Ben Amor, Duarte Most of the analyzed studies proposed algorithms and frameworks that were then validated by different applica- tions. The main application domains are tourist recommendation (with 8 papers, including hotels and touristic sites [5, 6, 13, 19, 20, 31, 49, 56]), e-commerce recommendation (8 papers, including the recommendation of books, wines, and other products [8, 10, 29, 30, 39, 41, 43, 59]), culinary recommendation (6 papers, including food and restaurant recommendations [14, 17, 46, 48, 53, 61]), and recommendations for group-buying websites (3 papers [26–28]). RQ5. What are the evaluation metrics employed in the selected studies? Evaluation metrics typically employed in RSs are used in about 60% of the selected studies (30 papers). The combi- nation of metrics “precision, recall, F-measure, and accuracy” are among the most popular performance metrics used [4, 7, 15, 16, 26, 36, 37, 39, 41, 57]. In those cases, accuracy is mainly measured by Mean Absolute Error (MAE), Mean Average Precision (MAP) and correlation measures, such as the Spearmen’s rank correlation. Adopting only accuracy metrics is the second most frequent scenario in the analyzed studies [17, 19, 28, 34, 48, 52, 53, 59]. For the remaining papers, different combinations of metrics were employed, and among these we have many metrics related to the opinion of the user about the recommendation, such as novelty and diversity [9], coverage [43], trust [29], scalability [11] and satisfaction [22, 55]. RQ6. What are the contributions of MCDA methods to RSs regarding some of their challenges such as cold-start, data sparsity, scalability, and privacy? Scalability and privacy have not been addressed in any of the papers. More than that, scalability proved to be a problem in the analyzed RSs since many of them require high interaction with the user, which may lead to slowness. On the other hand, MCDA can contribute to the cold-start and sparsity problems. Indeed, these problems can be addressed by applying utility-based methods, since preferences (utilities) of new users are learned from previous users [29, 36]. Frameworks in which the users are invited to input their preferences are also able to deal with the cold-start problem [14, 31]. RQ7. What are the trends and gaps in the use or research of MCDA methods in RSs? As confirmed by the literature review conducted here, given the number of papers published over time, there has been an increasingly higher interest in using MCDA in RSs, showing a trend in this research field. The vast majority of MCDA methods employed in RSs are for ranking. However, producing a ranking of alternatives can be seen as a disadvantage since, in most cases, a second stage of filtering is required to finally select a subset of recommended items. In order to avoid the need of this additional phase, it seems to be more appropriate to use sorting methods that directly assign alternatives to a set of defined categories. This approach is also indicated as more scalable for large sets of alternatives and/or users. Another trend in the use of MCDA methods in RSs regards the modeling of hierarchical criteria. Despite that, few methods have been explored for this purpose so far. Exploring how MCDA methods can be used to address some of the RS challenges, such as cold-start, data sparsity, scalability, and privacy, is also an open question, which naturally open up possibilities for future studies. As already discussed, although applying MCDA methods to RSs may improve personalizing the system when taking user preferences into consideration, these methods also leads to problems regards system response time. Moreover, those systems usually ask for the user their preferences and do not take advantage of information already available on historical data. Applying preference learning combining to MCDA in RSs comes up then as an interesting possibility of future research. Indeed, the research field called “preference learning”, which can be considered a sub-field of the Manuscript submitted to ACM A systematic review of the use of multiple criteria decision aiding methods in recommender systems: preliminary results 7 machine learning research area, concerns with the acquisition of preference models from data–it involves learning from observations that reveal information about the preferences of an individual or a class of individuals, and building models that generalize beyond such training data [21]. A RSs based on preference learning and MCDA would allow at the same time to learn preferences from historical data and to use preferences established by the user, offering a large and promising scope to be explored. Bringing together contributions involving these two areas to RSs presents itself as a good solution for recommendations in high personalized learning environments. It is important to note that, although the topic preference learning has already being explored in RSs [32, 47], throughout this literature review we could see that the integration of preference learning and MCDA applied to RSs is still an unexplored topic in the literature. 4 CONCLUSION AND FUTURE IMPROVEMENTS In this paper, we have conducted a systematic literature review of the application of MCDA methods to RSs. Typically, a small sample size affects the generalizability of the research results. In order to overcome this problem, we intend to include more studies identified through a snowballing approach and take into account all the main conferences on software engineering. We also intend to extend this study in order to better discuss open issues and opportunities for future work. To do so, we want to build a novel taxonomy on multi-criteria RSs, including RSs based on MCDA methods, analyze strengths and weakness associated to each category of the taxonomy, and point to immediate future works to be developed, which could later help the research community to obtain importance advances in this research area. ACKNOWLEDGMENT This work was supported by the São Paulo Research Foundation (FAPESP), grants #2018/23447 and #2020/01089-9, and the Brazilian National Council for Scientific and Technological Development (CNPq). This project is also part of the Brazilian Institute of Data Science, grant #2020/09838-0, São Paulo Research Foundation (FAPESP). REFERENCES [1] A. Abbas, K. Bilal, L. Zhang, and S.U. Khan. 2015. A cloud based health insurance plan recommendation system: A user centered approach. Future Generation Computer Systems 43-44 (2015), 99–109. [2] G. Adomavicius and Y. Kwon. 2007. New Recommendation Techniques for Multicriteria Rating Systems. IEEE Intelligent Systems 22, 3 (2007), 48–55. [3] G. Adomavicius and Y. Kwon. 2015. Multi-Criteria Recommender Systems. Springer US, 847–880. [4] H. Al-Bashiri, M.A. Abdulgabber, A. Romli, and H. Kahtan. 2018. An improved memory-based collaborative filtering method based on the TOPSIS technique. PLoS ONE 13, 10 (2018). [5] T. Angskun and J. Angskun. 2018. A qualitative attraction ranking model for personalized recommendations. Journal of Hospitality and Tourism Technology 9 (2018), 2648352. [6] T. Arentze, A. Kemperman, and P. Aksenov. 2018. Estimating a latent-class user model for travel recommender systems. Information Technology and Tourism 19, 1-4 (2018), 61–82. [7] Y.M. Arif, S. Harini, S.M.S. Nugroho, and M. Hariadi. 2021. An Automatic Scenario Control in Serious Game to Visualize Tourism Destinations Recommendation. IEEE Access 9 (2021), 89941–89957. [8] Aleksandra Bączkiewicz, Bartłomiej Kizielewicz, Andrii Shekhovtsov, Jarosław Wątróbski, and Wojciech Sałabun. 2021. Methodical Aspects of MCDM Based E-Commerce Recommender System. Journal of Theoretical and Applied Electronic Commerce Research 16, 6 (2021), 2192–2229. [9] Z. Chai, Y. Li, and S. Zhu. 2021. P-MOIA-RS: a multi-objective optimization and decision-making algorithm for recommendation systems. Journal of Ambient Intelligence and Humanized Computing 12, 1 (2021), 443–454. [10] Abdellah Azmani Chaimae Lamaakchaoui and Mustapha El Jarroudi. 2018. The AHP Method for the Evaluation and Selection of Complementary Products. International Journal of Service Science Management Engineering and Technology 9, 3 (2018), 96695–96711. [11] D.-N. Chen, P.J.-H. Hu, Y.-R. Kuo, and T.-P. Liang. 2010. A Web-based personalized recommendation system for mobile phone selection: Design, implementation, and evaluation. Expert Systems with Applications 37, 12 (2010), 8201–8210. [12] Deng-Neng Chen, Paul Jen-Hwa Hu, Ya-Ru Kuo, and Ting-Peng Liang. 2010. A Web-Based Personalized Recommendation System for Mobile Phone Selection: Design, Implementation, and Evaluation. Expert Syst. Appl. 37, 12 (dec 2010), 8201–8210. Manuscript submitted to ACM 8 Pelissari, Alencar, Ben Amor, Duarte [13] L. Del Vasto-Terrientes, A. Valls, P. Zielniewicz, and J. Borràs. 2016. Erratum to: A hierarchical multi-criteria sorting approach for recommender systems (J Intell Inf Syst, DOI 10.1007/s10844-015-0362-7). Journal of Intelligent Information Systems 46, 2 (2016), 347–348. [14] R.K. Dewi, M.T. Ananta, L. Fanani, K.C. Brata, and N.D. Priandani. 2018. The development of mobile culinary recommendation system based on group decision support system. International Journal of Interactive Mobile Technologies 12, 3 (2018), 209–216. [15] Veer Sain Dixit, Harita Mehta, and Punam Bedi. 2014. A Proposed Framework for Group-Based Multi-Criteria Recommendations. Applied Artificial Intelligence 28, 10 (2014), 917–956. [16] F. Ebrahimi, A. Asemi, A. Nezarat, and A. Ko. 2021. Developing a mathematical model of the co-author recommender system using graph mining techniques and big data applications. Journal of Big Data 8, 1 (2021). [17] F. Effendy, B. Nuqoba, and Taufik. 2019. Culinary recommendation application based on user preferences using fuzzy topsis. IIUM Engineering Journal 20, 2 (2019), 163–175. [18] S. Fomba, P. Zarate, M. Kilgour, G. Camilleri, J. Konate, and F. Tangara. 2017. A recommender system based on multi-criteria aggregation. International Journal of Decision Support System Technology 9, 4 (2017), 1–15. [19] S. Forouzandeh, K. Berahmand, E. Nasiri, and M. Rostami. 2021. A Hotel Recommender System for Tourists Using the Artificial Bee Colony Algorithm and Fuzzy TOPSIS Model: A Case Study of TripAdvisor. International Journal of Information Technology and Decision Making 20, 1 (2021), 399–429. [20] S. Forouzandeh, M. Rostami, and K. Berahmand. 2022. A Hybrid Method for Recommendation Systems based on Tourism with an Evolutionary Algorithm and Topsis Model. Fuzzy Information and Engineering 14, 1 (2022), 26–50. [21] J. Furnkranz and E. Hüllermeier. 2011. Preference learning. 1–466 pages. [22] Asela Gunawardana and Guy Shani. 2015. Evaluating Recommender Systems. Springer US, Boston, MA, 265–308. [23] Z. Guo, C. Tang, H. Tang, Y. Fu, and W. Niu. 2018. A Novel Group Recommendation Mechanism from the Perspective of Preference Distribution. IEEE Access 6 (2018), 5865–5878. [24] Shweta Gupta and Vibhor Kant. 2020. Credibility score based multi-criteria recommender system. Knowledge-Based Systems 196 (2020), 105756. [25] Yan Hong, Xianyi Zeng, Pascal Bruniaux, Yan Chen, and Xujing Zhang. 2018. Development of a new knowledge-based fabric recommendation system by integrating the collaborative design process and multi-criteria decision support. Textile Research Journal 88, 23 (2018), 2682–2698. [26] Yi-Chung Hu. 2013. A Novel Nonadditive Collaborative-Filtering Approach Using Multicriteria Ratings. Mathematical Problems in Engineering (2013). [27] Y.-C. Hu. 2014. A multicriteria collaborative filtering approach using the indifference relation and its application to initiator recommendation for group-buying. Applied Artificial Intelligence 28, 10 (2014), 992–1008. [28] Y.-C. Hu. 2014. Nonadditive similarity-based single-layer perceptron for multi-criteria collaborative filtering. Neurocomputing 129 (2014), 306–314. [29] S.-L. Huang. 2011. Designing utility-based recommender systems for e-commerce: Evaluation of preference-elicitation methods. Electronic Commerce Research and Applications 10, 4 (2011), 398–407. [30] Wang N.-n. Zhang H. Huang, Y. and J. Wang. 2019. A novel product recommendation model consolidating price, trust and online reviews. Kybernetes 48, 6 (2019), 1355–1372. [31] Yuxia Huang and Ling Bian. 2009. A Bayesian network and analytic hierarchy process based personalized recommendations for tourist attractions over the Internet. Expert Systems with Applications 36, 1 (2009), 933–943. [32] Zhenhua Huang, Yajun Liu, Choujun Zhan, Chen Lin, Weiwei Cai, and Yunwen Chen. 2021. A Novel Group Recommendation Model With Two-Stage Deep Learning. IEEE Transactions on Systems, Man, and Cybernetics: Systems (2021), 1–12. https://doi.org/10.1109/TSMC.2021.3131349 [33] J. Iijima and S. Ho. 2007. Common structure and properties of filtering systems. Electronic Commerce Research and Applications 6, 2 (2007), 139–145. [34] C.-K. Ke and C.-M. Chang. 2020. Optimizing target selection complexity of a recommendation system by skyline query and multi-criteria decision analysis. Journal of Supercomputing 76, 8 (2020), 6453–6474. [35] B. Kitchenham and S Charters. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering. [36] Kleanthi Lakiotaki, Nikolaos F. Matsatsinis, and Alexis Tsoukiàs. 2011. Multicriteria User Modeling in Recommender Systems. IEEE Intelligent Systems 26, 2 (2011), 64–76. [37] Seok Kee Lee, Yoon Ho Cho, and Soung Hie Kim. 2010. Collaborative Filtering with Ordinal Scale-Based Implicit Ratings for Mobile Music Recommendations. Inf. Sci. 180, 11 (jun 2010), 2142–2155. [38] S.T. Li, T.T. Pham, H.C. Chuang, and Z.-W. Wang. 2016. Does reliable information matter? Towards a trustworthy co-created recommendation model by mining unboxing reviews. Information Systems and e-Business Management 14, 1 (2016), 71–99. [39] Duen-Ren Liu and Ya-Yueh Shih. 2005. Integrating AHP and data mining for product recommendation based on customer lifetime value. Information Management 42, 3 (2005), 387–400. [40] Kaur P.D. Mahajan, P. 2021. Three-tier IoT-edge-cloud (3T-IEC) architectural paradigm for real-time event recommendation in event-based social networks. J Ambient Intell Human Comput 12 (2021), 1363–1386. [41] N. Manouselis. 2008. Deploying and evaluating multiattribute product recommendation in e-markets. International Journal of Management and Decision Making 9, 1 (2008), 43–61. [42] N. Manouselis and C. Costopoulou. 2007. Analysis and classification of multi-criteria recommender systems. World Wide Web 10, 4 (2007), 415–441. [43] N. Manouselis and C. Costopoulou. 2008. marService: multiattribute utility recommendation for e-markets. International Journal of Computer Applications in Technology 33, 2-3 (2008), 176–189. Manuscript submitted to ACM A systematic review of the use of multiple criteria decision aiding methods in recommender systems: preliminary results 9 [44] O. O. Olugbara, S. O. Ojo, and M. I. Mphahlele. 2010. Exploiting image content in location-based shopping recommender systems for mobile users. International Journal of Information Technology & Decision Making 09, 05 (2010), 759–778. [45] Han-Saem Park, Moon-Hee Park, and Sung-Bae Cho. 2015. Mobile Information Recommendation Using Multi-Criteria Decision Making with Bayesian Network. International Journal of Information Technology & Decision Making 14, 02 (2015), 317–338. [46] H.-S. Park, M.-H. Park, and S.-B. Cho. 2015. Mobile information recommendation using multi-criteria decision making with bayesian network. International Journal of Information Technology and Decision Making 14, 2 (2015), 317–338. [47] G. Pigozzi, A. Tsoukiàs, and P. Viappiani. 2016. Preferences in artificial intelligence. Annals of Mathematics and Artificial Intelligence 77, 3-4 (2016), 361–401. [48] A. Pinandito, M.T. Ananta, K.C. Brata, and L. Fanani. 2015. Alternatives weighting in analytic hierarchy process of mobile culinary recommendation system using fuzzy. ARPN Journal of Engineering and Applied Sciences 10, 19 (2015), 8791–8798. [49] Y. Qin, X. Wang, and Z. Xu. 2022. Ranking Tourist Attractions through Online Reviews: A Novel Method with Intuitionistic and Hesitant Fuzzy Information Based on Sentiment Analysis. International Journal of Fuzzy Systems 24, 2 (2022), 755–777. [50] S.R. Rizvi, S. Zehra, and S. Olariu. 2019. ASPIRE: An Agent-Oriented Smart Parking Recommendation System for Smart Cities. IEEE Intelligent Transportation Systems Magazine 11, 4 (2019), 48–61. [51] B. Roy. 1996. Multicriteria methodology goes decision aiding (1 ed.). Kluwer Academic Publishers. [52] J. Serrano-Guerrero, M. Bani-Doumi, F.P. Romero, and J.A. Olivas. 2022. A fuzzy aspect-based approach for recommending hospitals. International Journal of Intelligent Systems 37, 4 (2022), 2885–2910. [53] M. Showafah, S.W. Sihwi, and Winarno. 2021. Ontology-based Daily Menu Recommendation System for Complementary Food According to Nutritional Needs using Naïve Bayes and TOPSIS. International Journal of Advanced Computer Science and Applications 12, 11 (2021), 638–645. [54] Ye Tian, Wendong Wang, Xiangyang Gong, Xirong Que, and Jian Ma. 2013. An enhanced personal photo recommendation system by fusing contextual and textual features on mobile device. IEEE Transactions on Consumer Electronics 59, 1 (2013), 220–228. [55] C. Troussas, A. Krouska, and C. Sgouropoulou. 2021. Enhancing Human-Computer Interaction in Digital Repositories through a MCDA-Based Recommender System. Advances in Human-Computer Interaction 2021 (2021). [56] L.D. Vasto-Terrientes, A. Valls, P. Zielniewicz, and J. Borràs. 2016. A hierarchical multi-criteria sorting approach for recommender systems. Journal of Intelligent Information Systems 46, 2 (2016), 313–346. [57] P. Verma, S.K. Sood, and S. Kalra. 2017. Student career path recommendation in engineering stream based on three-dimensional model. Computer Applications in Engineering Education 25, 4 (2017), 578–593. [58] Nan Wang. 2021. Ideological and Political Education Recommendation System Based on AHP and Improved Collaborative Filtering Algorithm. Scientific Programming 2021 (2021), 2648352. [59] Zhang R. Wang, L. and H. Ruan. 2014. A personalized recommendation model in E commerce based on TOPSIS algorithm. Journal of Electronic Commerce in Organizations 12, 2 (2014). [60] Linda Yang, K. Yeung, and David Ndzi. 2012. A proactive personalised mobile recommendation system using analytic hierarchy process and Bayesian network. Journal of Internet Services and Applications 3, 2 (2012), 195–214. [61] Raciel Yera Toledo, Ahmad A. Alzahrani, and Luis Martínez. 2019. A Food Recommender System Considering Nutritional Information and User Preferences. IEEE Access 7 (2019), 96695–96711. Manuscript submitted to ACM