Research Methods for Group Recommender Systems

Abstract

In this article we argue that the research on group recommender systems must look more carefully at group dynamics in decision making in order to produce technologies that will be truly bene cial for users. Hence, we illustrate a user study method aimed at observing and measuring the evolution of user preferences and actions in a tourism decision making task: nding a destination to visit. We discuss the bene ts and caveats of such an observational study method and we present the implications that the derived data and ndings may have on the design of interactive group recommender systems.

1. INTRODUCTION

Recommender systems for groups are becoming more and more important since many information needs originate by group and social activities, like listening to music, watching movies, traveling, attending sport events, and many more. The importance of group recommender systems also has increased due to the social web, where users are not isolated but form interrelated groups. A high number of papers on group recommender systems have been published [ 13 ] but still, we believe, there is a gap between the current main focus of the research and the information search and decision making support needs of groups.

Research on group recommender systems often focuses on aggregation strategies, i.e., how to combine individual preferences, sometimes con icting preferences, into a group pro le. According to Arrow's theorem, it is clear that an optimal aggregation strategy does not exist - group recommender systems studies also con rmed that there is no ultimate winner. There are only a few studies that concentrate on decision/negotiation support in group recommender systems: Travel Decision Forum [ 12 ], Trip@dvice [ 2 ], Collaborative Advisory Travel System (CATS) [ 14 ], Choicla [ 21 ]. To our best knowledge, there are no observational studies on group decision processes in the context of group recommender systems. These types of studies are usually conducted in the social disciplines: in [ 22 ] the importance of discussions, especially with respect to information that is shared among group members is emphasized. An extensive overview of studies on group dynamics and the in uence of the di erent aspects (e.g., group structure, group decision process structure) on the group choices is presented in [ 8 ].

The main motivation of this paper is therefore to raise in the group recommender systems community the awareness of the importance of a new type of analysis: observing groups in naturalistic settings. We believe that the design of a novel and more e ective sort of group recommender systems can be initiated if one better observes and understands groups in actions, measures their behaviors, and tries to identify concrete opportunities for computerized systems to become more useful to people. In this paper we will illustrate the design, the outcome and the implications of an observational study where groups of people faced a concrete decision task - select a destination to visit as a group - and the researchers monitored the groups before, during and after the task.

Hence, our study is motivated by a range of dimensions and issues, that we list in the following.

Decision making is the ultimate motivation for a group recommender system. This is true even more than for individual recommenders which can also be used for expanding user knowledge or expressing self [ 20 ]. But if group recommenders must support decision making we must understand how this task is executed in groups and how the decision issues, the group members and the contextual situation alltogether impact on it. In the past too much attention was put on how to identify \optimal" recommendations, which in the context of groups is not even possible to correctly de ne. We believe that the application domain is crucial in a group recommender system. Recommending tourist attractions or destination for a group cannot follow the same model used to recommending movies to watch [ 24 ]. The tourism product is more complex than other types of products (i.e., it is a bundle of products and services) and in the same time it is less tangible. Moreover, traveling is an emotional experience and explicit preference characterization is problematic especially in the early phase of the travel decision-making process as di erent users usually have di erent perceptions of the features of the items. Finally, tourism products are typically experienced in groups. For that reason, we have tried to generate a decision task - destination selection - that is believable in the context of tourism decision making and we made observations for users characteristics and decision outcome that have emerged as important in tourism research on consumer behavior [ 6, 7, 23, 25 ]. Group recommendations techniques have been in uenced too strongly by social choice theory [ 13 ] and not enough by group dynamics studies [ 8 ]. It is still unclear how a recommender can identify items to suggest in a group decision making task, if the goal is not simply to aggregate the votes/preferences expressed by the group members. But we believe that studies like the presented one can help to understand the key information that groups need in order to make decisions, which could not simply be the suggested outcome of the decision. We believe that the more general concept of information recommendation, rather than product recommendation, is important to implement [ 3 ]. It is clear to us that the design of more e ective group recommender systems requires a multidisciplinary approach. In that sense the study described in this paper brings together social science and computer science scholars. Observational studies are not part of the classical research repertoire of recommender systems research methods, but, we believe that these methods are strictly required if we want to understand users in naturalistic settings and be able to generate fruitful conjectures about new and useful system functions. Another important motivation of this study is the desire to collect data about group decision making that can be exploited by several research groups. Hence, in some sense, we wanted to obtain raw data that could be used to several types of analyses, from different perspectives and with alternative motivations. We plan to make the data that we have collected, and that will also be collected in future implementations of the study, available to everyone for further analyses. Finally, we believe that the research community on group recommender systems needs to discuss and build a research agenda. We must identify critical challenges and expected results. In this study we initiate this reections by raising several issues, e.g., how to measure the collective behavior of a group, what properties of a group are more important in recommender systems and how they should be measured, how to de ne group satisfaction, how to compare and relate user preferences and group preferences.

Thus, the aim of this paper is to re ect on research methods for group recommender systems on the basis of an observational study.To present a detailed analysis of the collected data is not the focus of this paper; this was done in [ 5 ].

The rest of this paper is structured as follow: in Section 2 the study procedure is described in detail, Section 3 illustrates instruments used for the data collection, in Section 4 results of a rst analysis are summarized, followed by Section 5 where implications for recommender systems are explained. Finally, in Section 6 we discuss limitations, challenges and possible variations of the study. 2.

PROCEDURE

In order to design a new generation of more useful and e ective group recommender systems, we do not only aim at gaining insights into human behavior, but also at learning how to improve and facilitate interaction of users in a computer mediated setting. To set a basis for this, we started with an exploratory research approach that is not constrained by any pre-existing system functionality, i.e., we developed a study to collect observational data on humanto-human interactions in group decision making task. In the following we describe the procedure of this observational study in detail.

The study was initiated in a cooperation with the International Federation for Information Technologies in Travel and Tourism (IFITT) and 11 universities worldwide. The rst implementations of the study took place at the Delft University of Technology (TU Delft), the University of Klagenfurt (UNI Klagenfurt) and the University of Leiden (UNI Leiden), while an extended study was carried out at the Vienna University of Technology (TU Wien). Each implementation was conducted as a part of a regular lecture and followed a three-phases structure: pre-survey questionnaire phase, groups meeting/discussion phase and post-survey questionnaire phase (see Figure 2).

Prior to the rst study phase, an introduction with general instructions for the participants was presented. The rst task for all participant was to form groups. At TU Delft, UNI Klagenfurt and UNI Leiden, students were free to choose their group size (between two and four group members). At TU Wien students were instructed to form groups of six members and to select two students (referred to as observers) whose task was to observe and record activities of their group in the next phase. All the other group members took part in the decision making process (referred to as decision makers).

In the rst study phase, the task for the decision makers was to ll in a pre-survey online questionnaire that captures their individual pro les, preferences and dislikes. Detailed data description is provided in section 3. Also, in this phase, in Vienna, a short training for observers was organized. The purpose was to introduce them with the following study tasks and to instruct them on how to perform and record a group observation. A report template, which was constructed based on Bales's Interaction Process Analysis (IPA) [ 1 ], was provided to the observers to record the activities of the decision makers. The observers also received written instructions and during the rest of the study they were in a close contact with the study organizers. In the second study phase, the group meeting and discussion took place. The decision makers received written instructions with the following structure: 1. Ten prede ned destinations together with informational

Wiki pages; 2. Decision task scenario: Imagine that you are working on a research paper together with the other group members. Interestingly, your university o ers you the opportunity to submit this paper to a conference in Europe. If the paper gets accepted, the university will pay to each group member the trip to the conference. In addition, you will be able to spend the weekend after the conference at the conference destination. Ten conferences will take place in European capitals around the same summer period ; 3. Next, they were asked to discuss and decide which destination they would like to visit most as a group. Additionally, they also had to provide a second choice in case that the rst option would no longer be available.

Groups were not instructed on how to perform the decision making task and whether they should check the informational Wiki pages or not. This speci c design was chosen due to its simplicity. Usually, when a group is planning a trip a bundle of di erent trip aspects have to be considered, e.g., timing, budget, destination, accommodation, transport, etc. This type of task would be almost impossible to simulate in a controlled environment. Thus, we concentrated on a simple aspect to analyze the basis of group interactions and dynamics in this speci c context. At TU Wien, observers were included in the task. They audio recorded and reported the group decision process using the previously mentioned report template (details in 3).

In the third phase, the decision makers lled in an online post-survey questionnaire inquiring about the previous phase and the overall experience. During this phase, interviews with the observers were arranged in Vienna: for each group a meeting with the two observers of that group took place. Firstly, we evaluated observers' understanding of the task and the reports that they submitted, then, the observers elaborated their reports and discussed di erences between those. Furthermore, they were also queried about the behavior of the decision makers and how seriously they actually performed the task.

At each university the study implementation followed the described structure. However, still some di erences existed, they are explained in section 6. After the rst implementation round, considering all the locations where the study was conducted, the size of the collected data sample comprised 78 decision makers in all together 24 groups of two, three and four group members, plus 16 observers (two for each group) at TU Wien. At TU Delft, after a rst implementation round (referred to as TU Delft ), a second one with the same con guration (without observation) took place (referred to as TU Delft2 ). It introduced 122 new decision makers in 31 groups. Thus, currently the data sample comprises 200 decision makers in 55 groups of two, three, four and even ve group members (see Table 1) plus 16 observers.

Group size UNI Leiden UNI Klagenfurt TU Delft TU Delft2 TU Wien SUM

MEASUREMENTS

In this section we describe the data in detail as well as the instruments were used to collect it: a pre-survey questionnaire, a template for reporting the observations and a post-survey questionnaire. Each of these instruments was designed in a way that the obtained data cover di erent aspects, which might impact the group decision process and which were derived from the literature.

Accordingly, the rst data collection instrument - a presurvey questionnaire1 captured individual pro les of the participants in a similar way as the user pro le in a recom1https://survey.aau.at/2012/index.php?sid=49577&lang= en mender system would be represented. It is comprised of 68 questionnaire statements separated into four sections: 1. Demographic data and university a liation (i.e., age, gender, country of origin, university and student identi cation number);

2. 17 tourist roles and Big Five Factors:

30 questionnaire statements related to 17 tourist roles (i.e., types of touristic short term behavior) de ned in [ 10 ]; 20 questionnaire statements related to the Big Five Personality Factors (i.e., Openness to new experiences, Conscientiousness, Agreeableness, Extroversion, Neuroticism) [ 11 ]. 3. Experience and ratings/ rankings of ten prede ned destinations:

Destinations: Amsterdam (at TU Wien and UNI

Klagenfurt), Berlin, Copenhagen, Helsinki, Lisbon, London, Madrid, Paris, Rome, Stockholm and Vienna (at TU Delft and UNI Leiden);

Participants were asked how many times they have visited each destination;

Participants at the TU Wien rated, while other

participants ranked the ten destinations (implications of this distinction are discussed in section 6). 4. Ranking of decision criteria (i.e., budget, weather, distance, social activities, sightseeing and other).

A ve-point likert scale was used for the 50 questionnaire statements related to the 17 tourist roles and the Big Five Factors. To obtain the scores, i.e., the level to which a person belongs to a certain tourist role or to a certain personality trait, ratings of the statements were summed and divided by the number of related questionnaire statements. Tourist roles and personality traits are related to the user model of the picture-based recommendation engine (see section 5).

In the second phase group decision task took place. By now, only at the TU Wien, observational part of the study was implemented. The report template for the observers' recordings was designed based on the Bales's Interaction Process Analysis (IPA) (i.e., a method to study small groups and interactions among group members) [ 8 ]. Thus, the task for observers was to audio record group discussion and to ll in the provided report template. The report template consisted of the following sections: 1. Whether a plan for the group decision process was used or not and if yes the duration of the di erent decision process phases. We note that in [ 8 ] a four phases structure for the decision making process is indicated as typical: 1) Orientation, 2) Discussion, 3) Decision and 4) Implementation and evaluation of the decision; 2. Group members' roles (e.g., leader, follower, initiator, information giver, opinion seeker); 3. Group members' behavior (i.e., twelve categories of behavior: Show solidarity/ \Friendly"; Show tension release; Agree, Give suggestion/ opinion/ information; Ask for suggestion/ opinion/ information; Disagree; Show tension) - For each group member, the observers were requested to identify, record and categorize each \unit" of interaction (i.e., verbal and non verbal expressions) according to the twelve categories of behavior; 4. Social decision scheme (i.e., delegating, averaging, voting, reaching consensus or other -explanation could be provided); 5. Strength of group members' preferences (i.e., for each group member, the observers rated from 1 - Very unwilling to 5 - Very willing on how willing they were to give up on their preferred options).

Finally, a post-survey questionnaire2 was used to collect data about the participants' experience with the group decision process and the overall study. It asked for: 1. The rst and the second group choice; 2. Whether the provided information about the destinations was used during the group decision process; 3. Description of the decision process that led the group to their nal choice; 4. Overall attractiveness of the ten prede ned destinations (e.g., "Many destinations were appealing.", "I did not like any of the destinations."); 5. Satisfaction with the group choice (e.g., "I like the destination that we have chosen"); 6. Di culty of the decision process (e.g., "Eventually I was in doubt between some destinations."); 7. Participant's perceived identi cation and similarity with the other group members (e.g., "I see myself as a member of this group", etc.); 8. Assessment of the task (i.e., participants were asked to select the statements to which they agree regarding the organization of the task, their feedback and willingness to participate in the same or similar study).

A ve-point likert scale was used to assess 4., 5., 6. and 7.

The overall structure of the data is shown in Figure 2. It visualizes the data as an Entity Relationship Diagram (ERD). Di erent colors indicate di erent study phases, i.e., pink: pre-survey questionnaire, blue: groups meetings/ discussions and yellow: post-survey questionnaire. Central entity in the ERD is the group member, i.e., the decision maker who is connected to all the other data dimensions (for the observers, only the demographic data is collected). 4.

THE OUTPUT

In this section we summarize some concrete output obtained by an initial analysis of the data [ 5 ]. However, this is only one example how this type of studies can help to obtain deeper insights into the interplay of individual preferences and group processes. Various other analyses can be conducted making use of the rich information that has been (see Section 3). To facilitate this, we plan to provide the data to the research community. 2https://survey.aau.at/2012/index.php?sid=98597&lang= de (Pink - First study phase; Blue - Second study phase; Yellow - Third study phase) In a rst step, we studied whether or not the users were satis ed with the outcome of the group decision making process, and we tried to understand the impact of the initial preferences into that. The vast majority of users showed a high satisfaction for the destination chosen by the group.

Obviously this was particularly true for users, where the group selection matched their individual top choice. However, also more than two-thirds of the users, for whom the group decision was not in accordance with their most preferred destination, were satis ed with the collective choice.

To some extent this might be related to the fact that the users perceived the di erent destinations, which could be chosen for the group tour, overall as very attractive. However, our analysis clearly indicated that the group decision making process itself played a decisive role in this context: group preferences are not just an aggregation of the initial group members' preferences but are rather constructed during the process. This was also supported by the fact that common aggregation strategies in group recommender systems were hardly able to predict the outcome of the group decision making process.

Next, we studied the choice satisfaction of the users in more detail and identi ed relevant user and group characteristics in this context. We found some signi cant and moderately high correlations between the individual choice satisfaction and personality traits of a user. Also behavioral patterns during the discussion could be related to the satisfaction of a user as well as the di culty of the task. To capture the satisfaction of a group, we studied the average choice satisfaction of the group members. Statistical tests identi ed signi cant di erences between highly and less satis ed groups with respect to a number of factors. These factors captured, on the one hand, whether or not the group perceived the task as di cult. On the other hand, they were related to aggregated travel behavioral patterns as well as personality traits of the group members. Furthermore, in less satis ed group typically all members show disagreement during the decision making process.

5. IMPLICATIONS FOR RECOMMENDER

SYSTEMS

As mentioned previously, the proposed observational study is ultimately motivated by the goal of designing more e ective group recommender systems. This means that the system should better predict, and therefore recommend, which items the group will choose and will make the group members more satis ed. We will now discuss some important bene ts that we expect the analysis of the data acquired by observing users' interactions in group decision making tasks can bring to recommender systems.

First of all, group recommenders requires the design of ranking functions that can highlight which items a group must primarily look at. Ranking functions for group recommender are based on preference aggregation strategies. While we already mentioned that there is not a single best aggregation strategy that ts all recommendation tasks and decision contexts, observational study data can be used to choose and customize the aggregation function to the speci c contextual conditions of the group. We conjecture that, having a family of candidate aggregation functions, one can optimally choose the right one by tting the observation data. For instance, experimental results of the study showed that the social role and personality of the group members in uence group choices which was also con rmed in other studies [ 9 ], [ 18 ], [ 19 ]. Hence, for instance, among a family of multiplicative aggregation models one can t the importance weights of the group members depending on their roles and personality.

A second important usage of observational data is the construction of a more dynamic model of recommendation that integrate into the baseline user preference models preference information derived by the observations of the discussion process. In fact, it is clear from our study that the nal output decision is not completely determined by the initial preferences of the users. We conjecture that the observed dynamic of the users-to-users interactions must be considered in order to better predict which items may suit the group at that precise point in time. We have for instance mentioned the observed correlation between the user activity in providing information or criticizing options and the satisfaction for the nal choice. As we suggested in the paragraph above, also this data can be used to identify a better aggregation function. But, we also conjecture that this type of information can be exploited to revise the initial user models learned by the system using the historical preference data of the users. For instance, if a content based model was tted to the known ratings of a user, this model can then be revised by considering the items that the user liked or criticized. An initial prototype implementing this idea is presented in [ 17 ]. That mobile system, which is called STSGroup, allows group members to be engaged in a discussion where they can propose items that are thought to be suitable for their group and react to other group members' proposals by giving feedback such as likes, dislikes or favorites. They can also tag the proposed items with comments and emoticons as shown in Figure 3a. The interactions between the members and the system during the group discussion are monitored and taken into account in order to actively provide group members with appropriate directions and recommendations (see Figure 3b and Figure 3c). The group recommendations are built up with explanations that are computed on the base of the group members' actions and contexts.

A third, probably most fundamental issue, is related to the ultimate goals of observational data and the scope of a group recommender system. Should the recommender t the data, i.e., suggest what the users in a given context are supposed to choose, or should instead the system act as a mediator, aimed at driving the group towards a more fair choice? In the rst case, as illustrated in the two paragraphs above, the system pleases the group and let it more smoothly and e ciently converge towards the decision that the group may have taken even without the system intervention. In the second case, the system is instead assuming that the fairness of a sound aggregation strategy should prevail on the natural group dynamics and will stick to it. This contraposition is not new in recommender systems: it relates to the question whether a recommender should only suggest items predicted to be top choices for the user or inject in the recommendations items that would make the list of recommendations more diverse, novel, sustainable, or simply more trendy. In order to address these fundamental questions, and understand which role the recommender should play, live user studies are unavoidable.

A fourth, very concrete implication of the study is related to the picture-based approach introduced in [ 15, 16 ]. The pre-survey questionnaire and the picture-based approach lean upon the same dimensions when capturing a user model, i.e., 17 tourist roles and the Big Five Factors. The ndings of the observational study will be related to the picture-based approach model, which is illustrated in Figure 4, and then generalized to a group recommender system. The proposed research and related challenges are described in [ 4 ].

DISCUSSION

In this section we summarize the contributions of the paper and mention several challenges that have to be addressed when analyzing the data. Furthermore, we discuss potential variations and generalizations of the observational study.

The main contributions of the paper are:

A detailed description of the replicable study procedure and the instruments used for the data collection that can provide insights into the actual group decision making processes.

The implementation of the study procedure in a concrete context of tourism and traveling.

Experimental results showing that certain individual and group characteristics, which go beyond the initial preferences of the individuals and their straightforward aggregation, play an important role in the nal choice of the group.

The implications of the observational study for group recommender systems and di erent aspect that should be considered when building such systems.

During the initial data analysis, we encountered several challenges related to data measurements we used. These challenges are at the same time limitations of the study and need to be addressed in the future work: 1. How to aggregate di erent individual scores, e.g., personality traits, at the group level? 2. How to measure diversity among group members with respect to the di erent data dimensions? 3. How to distinguish satis ed from not so satis ed groups? 4. How to match and compare individual preferences to the preferences of the group as a whole? 5. How to address ratings/ ranking di erence in di erent study implementations? 6. How to relate participants' personalities to their preferences?

So far, we were mainly using the average of the individual scores when aggregating them at the group level [ 5 ]. However, more sophisticated approaches will be applied in future work.

Di erent dimensions of the study procedure can be varied in order to grasp diverse insights into the group dynamics in this particular context. In the following we present some of the variations and their potential implications: 1. Duration and timing of the study : In our implementations, we noticed di erent behaviors of the students in the study conducted over the three weeks period on the one hand and the study conducted in one lecture session on the other hand. In the rst case students were not explicitly referring to their initial, individual preferences, but were rather discussing their preferences in general. In the second case, students were comparing their initial preferences and their nal choice was based on these comparisons. 2. Diversity of the ten prede ned destinations (e.g., country side tourism vs. big city tourism; mountain destination vs. sea side destination; hot climate destination vs. cold climate destination): Higher diversity could generate more con icting preferences in groups and more intense discussions and decision processes. 3. Locality of the ten prede ned destinations : In our case the ten destinations (but Amsterdam) were capitals in Europe and in an hour or two ight distance. By changing the locality of the chosen destinations would there be some di erences in the observed decision process? Furthermore, the locality and overall popularity of the ten chosen destinations were related to the knowledge that the participants possessed about these destinations. But, by using less known destinations, how would the unfamiliarity with the destinations inuence the decision process? 4. Groups size: The conducted data analysis showed differences in groups' satisfaction with respect to the group size - smaller groups tend to be more satis ed with the group choice than the larger groups, which is quite intuitive. Nevertheless, varying the group size in the study can provide insights in di erent aspects that should be considered. 5. Budget : Including budget into the group discussion increases the complexity of the task for the participants and it also enables more realistic setting of the decision process in the context of traveling. 6. Group decision task : If the group were to choose a point of interest that they actually had to visit together right after the group discussion, then the group members might pursue their preferences and interests in a more natural manner and more persistently. 7. Domain: The same study could be carried out in a di erent domain, such as music, movies, restaurant, etc. In this case it would be much easier to introduce a realistic setting to participants, but the discussion process, in this case, would clearly be much di erent.

To summarize, in this paper we presented the observational study implemented at several universities, the instruments used for the data collection and described the collected data. We stressed the implications of the study for group recommender systems and our future work relying on the founding of this study. At the end, we outlined main contributions, introduced challenges and limitations detected by now.

References

[1] R. F. Bales . A set of categories for the analysis of small group interaction . American Sociological Review , 15 : 257 { 263 , 1950 .

[2] P. Bekkerman , S. Kraus , and F. Ricci . Applying cooperative negotiation methodology to group recommendation problem . In Proceedings of Workshop on Recommender Systems in 17th European Conference on Arti cial Intelligence (ECAI'06) , pages 72 { 75 , 2006 .

[3] H. Blanco and F. Ricci . Inferring user utility for query revision recommendation . In Proceedings of the 28th Annual ACM Symposium on Applied Computing, SAC '13 , Coimbra , Portugal, March 18 -22, 2013 , pages 245 { 252 , 2013 .

[4] A. Delic . Picture-based approach to group recommender systems in e-tourism domain . In Conference Proceedings of the 24th Conference on User Modeling , Adaptation, and Personalization (UMAP 2016 ), Halifax, Canada, 2016 .

[5] A. Delic , J. Neidhardt , N. Nguyen , F. Ricci , L. Rook , H. Werthner , and M. Zanker . Observing group decision making processes . In Proceedings of the tenth ACM conference on Recommender systems, RecSys'16 , 2016 .

[6] A. Delic , J. Neidhardt , and H. Werthner . Are sun lovers nervous? - research note at enter 2016 etourism conference . Bilbao, Spain, 2016 .

[7] I. Fernandez-Tob as , M. Braunhofer, M. Elahi , F. Ricci , and I. Cantador. Alleviating the new user problem in collaborative ltering by exploiting personality information . User Model. User-Adapt. Interact. , 26 ( 2-3 ): 221 { 255 , 2016 .

[8] D. Forsyth . Group Dynamics. Wadsworth Cengage Learning, 6th edition , 2014 .

[9] M. Gartrell , X. Xing , Q. Lv , A. Beach , R. Han, S . Mishra, and K. Seada . Enhancing group recommendation by incorporating social relationship interactions . In Proceedings of the 16th ACM international conference on Supporting group work , pages 97 { 106 , FL, USA, 2010 .

[10] H. Gibson and A. Yiannakis . Tourist roles: Needs and the lifecourse . Annals of tourism research , 29 ( 2 ): 358 { 383 , 2002 .

[11] L. R. Goldberg . An alternative "description of personality": the big- ve factor structure . Journal of personality and social psychology , 59 ( 6 ): 1216 , 1990 .

[12] A. Jameson . More than the sum of its members: challenges for group recommender systems . In Proceedings of the working conference on Advanced visual interfaces , pages 48 { 54 , 2004 .

[13] J. Mastho . Group recommender systems: aggregation, satisfaction and group attributes . In F. Ricci, L. Rokach , and B. Shapira, editors, Recommender Systems Handbook , pages 743 { 776 . Springer, 2015 .

[14] K. McCarthy , L. McGinty , B. Smyth , and M. Salamo . The needs of the many: a case-based group recommender system . Advances in Case-Based Reasoning , pages 196 { 210 , 2006 .

[15] J. Neidhardt , R. Schuster , L. Seyfang , and H. Werthner . Eliciting the users' unknown preferences . In Proceedings of the 8th ACM Conference on Recommender systems , pages 309 { 312 , 2645767 , 2014 . ACM.

[16] J. Neidhardt , L. Seyfang , R. Schuster , and H. Werthner . A picture-based approach to recommender systems . Information Technology & Tourism , 15 ( 1 ): 49 { 69 , 2015 .

[17] T. N. Nguyen and F. Ricci . Supporting group decision making with recommendations and explanations . In Posters, Demos, Late-breaking Results and Workshop Proceedings of the 24th Conference on User Modeling , Adaptation, and Personalization (UMAP 2016 ), Halifax, Canada, 2016 .

[18] L. Quijano-Sanchez , J. A. Recio-Garcia , B. Diaz-Agudo , and G. Jimenez-Diaz . Social factors in group recommender systems . ACM Transactions on Intelligent Systems and Technology (TIST) , 4 ( 1 ): 8 , 2013 .

[19] J. A. Recio-Garcia , G. Jimenez-Diaz , A. A. Sanchez-Ruiz , and B. Diaz-Agudo . Personality aware recommendations to groups . In Proceedings of the 3rd ACM conference on Recommender systems , pages 325 { 328 , NY, USA, 2009 .

[20] F. Ricci , L. Rokach , and B. Shapira . Recommender systems: Introduction and challenges . In Recommender Systems Handbook , pages 1 { 34 . 2015 .

[21] M. Stettinger , A. Felfernig , G. Leitner, S. Reiterer , and M. Jeran . Counteracting serial position e ects in the choicla group decision support environment . In Proceedings of the 20th International Conference on Intelligent User Interfaces , pages 148 { 157 , GA , USA, 2015 .

[22] R. S. Tindale and T. Kameda . Social sharedness as a unifying theme for information processing in groups . Group Processes and Intergroup Relations , 3 ( 2 ): 123 { 140 , 2000 .

[23] H. Werthner , A. Alzua-Sorzabal , L. Cantoni , A. Dickinger , U. Gretzel , D. Jannach , J. Neidhardt , B. Proll, F. Ricci , M. Scaglione , B. Stangl , O. Stock , and M. Zanker . Future research issues in IT and tourism . J. of IT & Tourism, 15 ( 1 ):1{ 15 , 2015 .

[24] H. Werthner and F. Ricci . E-commerce and tourism . Communications of the ACM , 47 ( 12 ): 101 { 105 , 2004 .

[25] A. Yiannakis and H. Gibson . Roles tourists play .

Annals of tourism Research , 19 ( 2 ): 287 { 303 , 1992 .