Multiple Criteria Decision Making (MCDA) methods have been increasingly applied to improve recommendations when multiple criteria are considered in Recommender Systems (RSs). This study presents the preliminary results of a systematic literature review, following Kitchenham's guidelines, regarding the application of MCDA methods in RSs over the last two decades. Based on our ifndings, MCDA methods can be applied in two RS phases: the preference elicitation and the recommendation phases. In the former, RSs usually have a strong interaction with the user, which results in more personalized recommendations, ensuring higher user satisfaction and contributing to address the cold-start challenge in RSs. Regarding the recommendation phase, while most RSs are based on ranking approaches, there is a trend to apply sorting methods in order to avoid an additional step involving a filtering application that selects a subset of alternatives. Future research could focus on applying preference learning combined with MCDA methods for exploring improvements in prediction and recommendation phases, and also in quality and processing time. Additional Key Words and Phrases: Multiple Criteria Decision Making, Literature review, Cold start, Preference learning
While the majority of existing Recommender Systems (RSs) depend only on one single criterion rating as input
information, there has been an increasing interest over the last two decades in taking into consideration a rating based
on multiple criteria, since the user’s preferences might cover more than one perspective [
In MCDA, alternatives are evaluated by the decision-maker (DM) according to several criteria, usually conflicting to
each other, with the goal of either ranking the alternatives (ranking problematic) or sorting them into predefined and
ordered categories (sorting problematic) [
RSs based on multiple criteria have been pointed out as one of the promising research areas in RSs [
Given this context, we propose to perform a systematic literature review of the use of multiple criteria decision aiding methods in RSs. The remainder of the paper is structured as it follows. In section 2, we present the adopted methodology for conducting the literature review. The results are presented in Section 3, and the conclusions and future improvements are presented in Section 4. 2
Our study followed the guidelines for undertaking systematic reviews proposed by [
The protocol definition can be seen as the first phase of the literature review, which is followed by two more phases, the review conduction and the review report, as illustrated in Figure 1. 2.1
The first step of the protocol definition starts with the selection of research questions that will guide the study. This review examines the following seven sets of research questions:
Manuscript submitted to ACM 3 RQ1. How frequently have MCDA methods been implemented in the use or research of RSs over the last 20 years?
What are the trends in terms of publication venues? RQ2. What are the MCDA methods employed in RSs? And which are employed most? RQ3. What purpose do the MCDA methods serve in RSs? And what are the advantages and disadvantages of employing
MCDA methods in RSs? RQ4. What are the main application domains of RSs combined with MCDAs? RQ5. What are the evaluation metrics employed in the selected studies? RQ6. What are the contributions of MCDA methods to RSs regarding some of their challenges such as cold-start, data sparsity, scalability, and privacy? RQ7. What are the trends and gaps in the use or research of MCDA methods in RSs? We focused our search on Scopus, since it is the largest curated, peer-reviewed abstract and indexing database available to academia. In order to precisely answer the defined research questions, we conducted an initial study to identify the most suitable query to be used in the paper search, by conducting pilot searches to ensure that the used keywords provided the right scope. Initially, we developed a set of keywords considering the main terms that diferent communities use to refer to recommender systems (“recommendation systems", “recommendation system", “recommender systems", “recommender system") and to MCDA methods (“multiple criteria decision making”, “multiple criteria decision aiding”, “multiple criteria decision analysis”, “multi-criteria decision making”, “multi criteria decision-making”, “multi criteria decision aiding”). Taking into consideration papers that apply MCDA to RSs and were already known by the authors, we realized that many important papers were not found in this initial search. Deepening our initial study, we verified that many papers have mentioned in their title, abstract or keywords, only specific words related to the MCDA method used such as “AHP” and “TOPSIS”, and not more generalized words. Therefore, we considered the acronyms of the most known MCDA methods as keywords in our search, even at the risk of not being comprehensive enough, since not all existent MCDA-method names were included. The final list of keywords used in our search is presented in Table 1. EC5. Papers whose abstract does not provide enough information in order to verify whether the paper is related to the review goal.
EC6. Papers that do not describe or consider an RS approach.
EC7. Papers that do not describe or consider an MCDA method.
EC8. Papers that do not apply MCDA methods as a core or at least as an important part of the system, e.g, application of MCDA methods for tool selection. 2.2
On March 17, 2022, we queried the digital library Scopus using the search terms presented in Table 1, and our search returned 313 papers.
The exclusion criteria from EC1 to EC4 were automatically applied using search resources from the Scopus database itself, reducing the number of studies from 313 to 135. Abstracts of the remaining papers were read, and the exclusion criteria from EC5 to EC8 were manually applied. In the end, 96 studies were retained. The 96 selected papers were then read in full. Throughout the reading process, papers that met at least one of the exclusion criteria EC6 to EC8, and that were not identified in the previous step, were excluded. From these, 49 studies were finally selected. 3
In this section, we present the main results of our literature review based on the defined research questions.
Figure 2 shows the number of studies applying MCDA methods to RSs over the last two decades. We can see continuing growth over time in the number of published papers, with a clear increase from 2018 on. About 60% (29 papers) of the total number of selected papers have been published in the last 4 years.
The papers have been published in 47 diferent journals, indicating that there are not specific journals concentrating on the topic discussed here. The journal with the most published papers is IEEE Access, with 4 papers, followed by Expert Systems with Applications, Applied Artificial Intelligence, Electronic Commerce Research and Applications, with 2 papers each.
We could verify that most of the known MCDA methods have been employed in the RS context. Table 2 shows the
number of papers that applied each method and their references. As a paper may have applied more than one method,
the total number of papers presented in Table 2 is greater than the number of selected papers.
ELECTRE-TRI-B are also apply for the purpose of organizing the criteria hierarchically [
Utility-based methods have also been highly used in RSs. For instance, [
The main disadvantage in applying MCDA methods in RSs pointed out by many studies is related to processing and system response time, which leads to dissatisfaction due to high interaction and slowness.
Most of the analyzed studies proposed algorithms and frameworks that were then validated by diferent
applications. The main application domains are tourist recommendation (with 8 papers, including hotels and touristic sites
[
Evaluation metrics typically employed in RSs are used in about 60% of the selected studies (30 papers). The
combination of metrics “precision, recall, F-measure, and accuracy” are among the most popular performance metrics used
[
Scalability and privacy have not been addressed in any of the papers. More than that, scalability proved to be a
problem in the analyzed RSs since many of them require high interaction with the user, which may lead to slowness. On
the other hand, MCDA can contribute to the cold-start and sparsity problems. Indeed, these problems can be addressed
by applying utility-based methods, since preferences (utilities) of new users are learned from previous users [
As confirmed by the literature review conducted here, given the number of papers published over time, there has been an increasingly higher interest in using MCDA in RSs, showing a trend in this research field.
The vast majority of MCDA methods employed in RSs are for ranking. However, producing a ranking of alternatives can be seen as a disadvantage since, in most cases, a second stage of filtering is required to finally select a subset of recommended items. In order to avoid the need of this additional phase, it seems to be more appropriate to use sorting methods that directly assign alternatives to a set of defined categories. This approach is also indicated as more scalable for large sets of alternatives and/or users. Another trend in the use of MCDA methods in RSs regards the modeling of hierarchical criteria. Despite that, few methods have been explored for this purpose so far. Exploring how MCDA methods can be used to address some of the RS challenges, such as cold-start, data sparsity, scalability, and privacy, is also an open question, which naturally open up possibilities for future studies.
As already discussed, although applying MCDA methods to RSs may improve personalizing the system when taking
user preferences into consideration, these methods also leads to problems regards system response time. Moreover,
those systems usually ask for the user their preferences and do not take advantage of information already available on
historical data. Applying preference learning combining to MCDA in RSs comes up then as an interesting possibility
of future research. Indeed, the research field called “preference learning”, which can be considered a sub-field of the
Manuscript submitted to ACM
machine learning research area, concerns with the acquisition of preference models from data–it involves learning
from observations that reveal information about the preferences of an individual or a class of individuals, and building
models that generalize beyond such training data [
In this paper, we have conducted a systematic literature review of the application of MCDA methods to RSs. Typically, a small sample size afects the generalizability of the research results. In order to overcome this problem, we intend to include more studies identified through a snowballing approach and take into account all the main conferences on software engineering. We also intend to extend this study in order to better discuss open issues and opportunities for future work. To do so, we want to build a novel taxonomy on multi-criteria RSs, including RSs based on MCDA methods, analyze strengths and weakness associated to each category of the taxonomy, and point to immediate future works to be developed, which could later help the research community to obtain importance advances in this research area.
This work was supported by the São Paulo Research Foundation (FAPESP), grants #2018/23447 and #2020/01089-9, and the Brazilian National Council for Scientific and Technological Development (CNPq). This project is also part of the Brazilian Institute of Data Science, grant #2020/09838-0, São Paulo Research Foundation (FAPESP).