Proc. of the 16th Workshop “From Object to Agents” (WOA15) June 17-19, Naples, Italy Comparing a Social Robot and a Mobile Application for Movie Recommendation: A Pilot Study Francesco Cervone, Valentina Sica, Mariacarla Staffa, Anna Tamburro, Silvia Rossi Dipartimento di Ingegneria Elettrica e Tecnologie dell’Informazione Università degli Studi di Napoli “Federico II” via Claudio 21, 80125 Napoli, Italy {mariacarla.staffa,silvia.rossi}@unina.it Abstract—Social robots can be used as interfaces to provide real world environments [6]. Moreover, non-verbal behaviors recommendations to users. While a vast literature compares the serve important functions in affecting the trustworthiness of a user’s behavior when interacting with a robot with respect to recommendation [7]. In fact, a robot ability to build a trust re- a virtual agent, in this paper, we conduct a first evaluation on lationship depends on its capacity to help people understand it, how the user’s choices are affected if the recommendations are in part through non-verbal behavior. Emotion-related signals, provided respectively by a mobile application or by social robots with different degree of interaction capabilities. This pilot study such as those provided by voice pitch changes in speech or shows that the sole embodiment condition of the robot does not gestures are non-verbal behaviors that influence human trust imply significant changes in the users’ choices that prefer to [8]. It has been, indeed, well-documented that humans expect interact with the mobile application. However, the adoption of from humanoid robots socially intelligent responses [9]. This additional communication channels such as gestures, gaze and leaves the possibility that an agent may influence how humans voice pitch, which change accordingly to the suggested movie perceive a recommendation through the presence of more or genre, improves the users acceptability. less communication abilities. In this paper, we present a pilot study to evaluate the extent I. I NTRODUCTION of the use of a robotic system in accepting a recommendation Social robots will be used in the next future in many not with respect to a virtual agent, but to very common application domains, which span from entertainment and ed- interfaces such as mobile applications. Our experiments aim at ucation to health–care. In order to be accepted in our houses, evaluating the users’ acceptance of recommendations as well they should be perceived as trusting, helpful, reliable and as their engagement when the robot or a mobile application are engaging [1]. This is particularly important in case the robot displaying such advises. In particular, we provided the same is in charge to convey information to a person (such teaching information contents on recommended movies, but using three skills, collaborating towards a particular goal or providing different interaction conditions. In the first condition, by using advises). Social robots, as well as virtual agents, can be used as a mobile application, contents will be provided by text shown interfaces to provide recommendations. Such embodied social on the mobile screen. In the second one, the same contents agents make interaction more meaningful than it is when will be provided by using a humanoid robot interacting using provided by simple interfaces (which do not display actions speech. Finally, in the third condition the humanoid robot will or speech), because users’ attitude towards social agents is encompass both voice and genre-driven motion primitives. similar to that they show towards other people. Recommendation systems aim to provide the user with II. R ELATED W ORKS personalized advises and suggestions in many different do- mains, such as books, movies or music. Such suggestions are A vast literature compared the behavior as well as the provided according to the available information the system acceptance of robots with respect to their virtual counterparts. has on the user (e.g., his/her preferences or his/her past Embodied robots are consistently perceived as more engaging interaction with the system). Hence, recommendations can be than a character on a video display, and sometimes as engaging provided suggesting items similar to items liked by the user as a human. For example, in Kidd and Breazeals [1] work or liked by similar users. The effectiveness of the provision of subjects were instructed by an agent (either to a human, a recommendations relies itself on the concept of trust [2] with robot, or a cartoon robot), which showed only its eyes to respect to the system that proposes the recommendation. Such the subjects. All three visual presentations were accompanied trust on the recommendation depends upon machine accuracy, by the same vocal instructions. The Authors’ purpose was to predictability and dependability [3] (e.g., by recommending understand which types of interaction involved more the user items which are positively evaluated by the users). In literature, (evaluated by a questionnaire), and showed more reliability, different studies compared the impact of recommendation and usefulness and trust. The results showed that the robot, given the advises as provided by social robot with respect to virtual its physical presence, was considered as more engaging, cred- agents [4], [5], by showing that the embodiment condition, ible and informative, as well as being more pleasant as an as provided by the robot, has more impact with respect to interaction partner. As in [1], in our experiment, we provided 2D/3D virtual agents on a screen. Real robots affect subject the same information contents with very simple and controlled decision-making more effectively than computer agents in interfaces, but using different interaction modalities. 32 Proc. of the 16th Workshop “From Object to Agents” (WOA15) June 17-19, Naples, Italy How physical embodiment, as opposed to virtual pres- ence, affects human perception of social engagement with an artificial agent was addressed in [4], [6], [10]. In [6] the Authors evaluated the persuasion effects of a computer agent and of a robot in various tasks as, for example, in following indications. The results showed that the user has shown more confidence and more trust for the physical robot. User’s behavior in accepting advises was investigated also in [4]. The results showed that the user preferred to interact with the robot because it was more effective in providing recommendations. Shinozawa et al. considered the effect of persuasion in a laboratory environment comparing a robot and a computer agent (with a 2D or 3D appearance) displayed on a monitor [10]. The results showed that the geometric coherence between a social agent and the environment was an important factor in the interaction, independently whether it is 2D or 3D. Conversely to these approaches, in our study, we compared the effects of adopting for a recommending task a robot with respect to a mobile application. This is, up to our knowledge, Fig. 1. Client/Server application. the first attempt to provide such comparison. Finally, in [5] the Authors studied the impact of the robot and to populate the Ratings Repository. MovieTweetings con- size with respect to the user reactions in an advertising context. sists of movie ratings contained in well-structured tweets on The purpose was to understand which robot was more suitable the Twitter.com social network. This information is contained for interaction for advertising purposes. The results showed in three files: users.dat, ratings.dat and movies.dat, which that, in the presence of robots of different sizes, the user provide respectively the user identification number, his/her considers it easier to interact with a smaller robot. associated ratings and a list of movies. The dataset is updated every day, therefore its size is constantly changing. At the last III. S YSTEM A RCHITECTURE access, it contained about 35000 users, 360000 ratings and 20000 movies. In order to evaluate our hypothesis, we developed a client/server application, where the server provides the recom- The recommendation engine provides rating predictions mendation service and the possible clients can be a humanoid when the recommendation API is invoked. To achieve this robot or a mobile application. Clients are in charge to ask for a goal, we used item-based City Block distance, also known as list of recommendations (in particular of movies) to the server Manhattan distance. In Mahout implementation, the generic and to show them to users. The social robots and the mobile movie i is represented by a boolean vector: application will provide the same information, but in different i = [r1i , r2i , ..., rni ], ways (i.e., through different communication channels). This diversity should be reflected in a different perception of where n is the number of users in the dataset and rui = 1 if the recommendations by the users, and, presumably, it will user u rated the movie i. The distance between two movies affect their choices. In order to provide recommendations, rated by user u is the sum of the absolute value of the the Recommendation Engine needs some initial movie ratings differences of the two associated vector components. More from the users. Hence, independently from the client type formally, the distance between items i and j is: the users interact with, the initial ratings are performed by n using the mobile application, which allows users to easily X d(i, j) = |rui − ruj |. evaluate movies by means of a friendly graphical interface. u=1 The main blocks of the developed framework are detailed in the following subsections. B. Android Application A. Movie Recommendation Server On the bottom of Figure 1, the architecture design of the mobile Android application is depicted. The first duty for the The server layer of this architecture is characterized by the user, when he/she accesses the application, is to sign up/sign recommendation system (see Figure 1). It is developed in Java into the system. As explained above, when the interaction and it is hosted by a Tomcat servelet container. The core of starts, users have first to provide a certain number of movie this layer is a Web Server, used to store, process and deliver ratings (at least 20 movies). The Training module is dedicated requested content to clients, which are provided through JSON- to provide an interface to get movie lists and to store movies based API both to communicate with the humanoid robot and ratings. If a user is in the training stage, he/she can browse with the android mobile application. The module Recommen- movies by ordering them by most rated or randomly, or search dation Engine is the core of the recommendation layer. We for a movie (filtering by genre or title, see Figure 2). adopted the Apache Mahout library1 to predict the user ratings, and chosen the MovieTweetings [11] dataset to train the system After this first stage, user can get movie recommendations from the server. When the server gets the recommendation 1 http://mahout.apache.org/ request, once calculated the best movies for the user, it 33 Proc. of the 16th Workshop “From Object to Agents” (WOA15) June 17-19, Naples, Italy retrieves additional details about the film, like, for example, a) Face Detection: This module is based on a face the director, writers, actors and genres using OMDb2 web detection/recognition solution provided by OKI and included service. Fortunately, MovieTweetings data set stores, for each in the Python SDK for NAO. Such module continuously movie, its IMDb id, which can be used to address the OMDb processes frames from the NAO camera in order to detect a service. The Android application shows on the screen the human face. Once a face is detected, it provides its position. recommendations for the users through textual and graphical Moreover, in the third condition, the module continues to descriptions. provide the user position coordinates to the Motion Controller, that allows NAO to track the face by moving its head. b) Speech Recognition: This module gives to the robot the ability to recognize a predefined words list, and specifically the usernames and the acceptance/rejection of a recommenda- tion. It is based on module provided by Aldebaran, which relies on sophisticated speech recognition technologies provided by NUANCE for NAO Version 4. Before starting, the robot needs to receive the list of usernames (UsersList). Then, once the sys- tem has detected a face through Face Detection, NAO asks for a username and listens until a word is recognized. Currently, system does not provide a real authentication when interacting with the robot because the only way to communicate with NAO is the speech. Users should sign in through an input system like a keyboard or a mobile application. c) Behavior Selector: Through this module, we gener- ate all the gestures, gaze and the feedback for the user. Once a user has been recognized, a user tracking system allows the robot to track the target by moving its head. Movie information is provided to the user with the Speech Synthesis module with different speech intonations, but it can be accompanied Fig. 2. Snapshots from the movie-app training phase. with arms gestures and facial expressions (e.g., different eyes colors) generated through Motion Controller module. The Behavior Selector gets recommendations from the Web Service and related animations from the Animations Repository. The C. The Robot Client main task of this module is mapping the movie genre into a The Robot Client architecture has been designed consid- predefined set of animations and eyes colors. For example, if ering the adoption of a NAO T14 robot model, consisting in NAO recommends a drama, led eyes become red and gestures a humanoid torso with 14 degrees of freedom (2 for the head are more serious, while for a comedy led eyes become green and 12 for the arms) developed by Aldebaran Robotics3 . We and gestures are more joyous. The pitch of the voice is controlled the NAO platform by means of the standard robotic accordingly manipulated by the Speech Synthesis module. operating system (ROS) and using the Python programming language for developing the ROS nodes. NAO is endowed with IV. A P ILOT S TUDY two main sensors: a camera and a microphone through which it receives signals from the external environment. Camera The pilot study is conducted by considering three different frames are processed by a Face Detection module to detect interaction conditions, where participants receive two movie users presence into its visual field. Sounds obtained from the recommendations for each condition. microphone are processed by Speech Recognition module. A. Procedure As actuators NAO can use the following communication channels: voice, arms, head and eyes led. The Behavior The testing procedure main steps are: (i) the user provides Selector module is in charge of providing two different in- new rates for a list of movies (training phase) at the beginning teraction conditions. In the first case, it presents to users of the interaction; (ii) the recommendation system generates the recommended movies and their relative information only the top-six recommendations for each user, which will be through speech, while, in the second case, such description is shown to users through the three interaction conditions in a accompanied with gestures, gaze, eyes coloring through the random way (two for each condition); (iii) after each test Motion Controller, and pitch voice changes through Speech condition the participant has to answer to a questionnaire Synthesis. In this latter case, the Behavior Selector gets motion concerning the specific condition and at the end of the overall animations from an Animations Repository, based on the genre experiment to a general questionnaire. of the recommended movie, to execute animated speech, as will be detailed in the followings. B. Method 2 http://www.omdbapi.com - The Open Movie Database is a free web service The design of this study is a within-subjects, counterbal- to obtain movie information. anced, repeated measures experiment. The three considered 3 https://www.aldebaran.com/en interactive conditions are the following: 34 Proc. of the 16th Workshop “From Object to Agents” (WOA15) June 17-19, Naples, Italy Fig. 4. NAO and ENAO conditions. TABLE I. 18 PARTICIPANTS DATA . min max avg Age 22 55 32 Fig. 3. Snapshots from the App recommendation phase. male female Gender 72% 28% low high avg English Level 61% 39% 2.39 Condition 1 (App): in this setting, neither of the robot modali- Robotic Skills 44% 56% 3.17 ties are used. The user only interacts with the mobile applica- tion that provides to the user two different movies suggestions. For each movie the app provides the title and additional V. R ESULTS information by displaying text and images on the screen. For We hypothesized that the robot as compared with the each recommended movie, the user has to reply about his/her application will be more engaging and better liked, and hence likelihood to see it (see Figure 3). recommendation provided by the robot should be more likely Condition 2 (Nao): in this setting, the robot is located on a to be accepted. Moreover, the condition with animated motion table standing still and waiting for a person to interact with. should be more engaging and better liked with respect to the When the robot recognizes a face in its filed of view, it greets simple robot. the person, introduces itself, and asks for a username. NAO presents the two recommendations by telling the movie title A. Quantitative Analysis accompanied with the same information provided in Condition In order to evaluate the degree of acceptance of the 1 (plot, genre, actors, and so on). Finally, it asks the user if recommendations when provided by different conditions, we she/he agrees to see this movie and stores the answer. calculated the selection ratio indicating the number of accepted Condition 3 (ENao): in this setting, differently from the recommendations with respect to the total number of recom- previous condition, the robot is endowed with the motion mendations for a each specific condition. controller module. When the robot is not interacting with anyone, it simply looks around and waits for a person to talk with. In this interaction condition NAO, in addition to tell movie information, gesticulates, changes eyes led color and voice pitch according to the recommended movie genre (see Figure 4). C. Participants 18 subjects participated in this experiment with an average age of 32 years and a graduate education, for a total of 13 males and 5 females. All the participants were Italian native speakers with an average English level of 2.39 and Robotic skills of 3.17 on a likert–scale from 1 to 5. The language Fig. 5. Percentage of accepted movie recommendations for each Condition. adopted for the experiment was the English both for text description and for the robot’s voice synthesizer. The testers were not informed about the NAO interaction capabilities. In In Figure 5 the selection ratio is expressed in percentage Table I, personnel data of participants are collected. shows that there is a minimum difference in the acceptance rate 35 Proc. of the 16th Workshop “From Object to Agents” (WOA15) June 17-19, Naples, Italy between the recommendations provided by App and Nao, while likert scale from 1 to 5. Only for question 6 we explicitly there is a slightly bigger difference between App and ENao ask for a preference by the users where index from 1 to 3 conditions. This fact is in accordance with our hypothesis that represent respectively the preference for APP, NAO and ENAO. people are inclined to accept more recommendations provided The questionnaire structure is reported in Table II. through a more natural interaction, even if the sole embodi- TABLE II. Q UESTIONNAIRE . ment condition (Nao) does not imply significant changes in the testers acceptability level. However, due to the limited Section Question number of participants and recommendations provided to each Personal Age? participant these differences, evaluated using ANOVA, are not Information Gender? English level? (1 to 5) yet statistically significant, while there is a significant Pearson How familiarized are you with robotic applications? (1 to 5) correlation between App and Nao conditions (r = 0.43 with Qualitative Q1. How easy was to perform the task? (1 to 5) Questions Q2. Did the system react accordingly to your expectations? (1 to 5) p = 0.08 that is significant at p < 0.10) acceptance trends. Q3. How natural is this kind of interaction? (1 to 5) As future work we will extend such experimentation with a Q4. How satisfying do you find the interactive system? (1 to 5) greater number of users. Q5. You were sure (5) or unsure (1) of your answers? Q6. Which mode of interaction you preferred? (1 to 3). Since users selected for testing are all Italian native speak- Robot-related Q7. The agent was believable (5) or unbelievable (1). Questions Q8. The agents motions were natural (5) or unnatural (1). ers, and not all have the same level of familiarity with robotics applications, we felt it appropriate to consider data by aggre- Figure 6-(a) shows the mean value of the answers to gating the results by both the level of English proficiency (e.g., the qualitative questions for each interactive condition. Users the language used to provide recommendations) and the degree found the interaction with App easier than the interaction of experience with robots. We thus computed correlations for with Nao and ENao (Q1 in Table II). ANOVA test endorsed the acceptance ratio and among conditions couples by grouping this result by showing that differences between App and Nao users with a high (from 4 to 5) or low English level (from 1 (F = 6.48 with p = 0.02) and between App and ENao to 3) and a high or low familiarity with robots (see Section (F = 3.34 with p = 0.08) were statistically significant. A V-B): slightly preference for the interaction with the App was also • high English level: Pearson showed a negative strong shown by the answers to Q3 and Q4 questions. In this case, correlation (r = −0.76 with p < 0.05) between Nao the only statistical significant differences were between App and ENao; and Nao conditions for Q3 (F = 4.25, p = 0.05) and Q4 (F = 3.89, p = 0.06), thus the App was more natural and • low English level: there is a significant correlation satisfying than Nao interacting with speech. between App and Nao (r = 0.57 with p = 0.07); For each question, we computed correlations between App, • high familiarity: nothing significant; Nao and ENao: • low familiarity: once again we had a moderate correla- • App-Nao: we notice a moderate correlation for Q2 tion between App and Nao (r = 0.65 with p = 0.08), (r = 0.50, p = 0.03), Q3 (r = 0.44, p = 0.07) and but in this case also Nao and ENao have a moderate Q4 (r = 0.52, p = 0.03); correlation (r = 0.65 with p = 0.08). • Nao-ENao: there is a moderate correlation for Q2 (r = These results show that for testers with a low English level 0.59, p = 0.01) and Q4 (r = 0.48, p = 0.04) and a reading text from an application or hearing speech from a strong correlation for Q5 (r = 0.72, p < 0.01); robot does not have a relevant impact on the decision making, while for good English skilled participants adding an animated • App-ENao: there are no significant correlations. behavior changes the acceptance trend. This could be due to If we observe the histogram of question 6 (see Figure 6- the fact that the users attention in the first case is quite all (b)), a part from the approval ratings average of the qualitative focused on understanding the text or speech. Moreover, users questions from Q1 to Q5, it is quite evident that the major part with low familiarity in robotics shows acceptance trend similar of the users prefers to interact with the Humanoid endowed in both Nao and ENao cases. with emotion-based capabilities. This is probably due to the fact that the humanoid robot has the potential to portray a rich B. Qualitative Analysis repertoire of non–verbal behaviors that have familiar social Our evaluation also takes into account the impressions meaning for users, who perceive the interaction more natural of users with respect to the interaction with the different and engaging because of the received socially intelligent conditions. For this aim we propose a qualitative questionnaire responses by the robot. Histogram in Figure 6-(c) shows that organized in three specific sections: (i) personal information the robotic agent is perceived in the average believeble both for collecting information about the user (age, gender, english if it shows or not non-verbal feedback, and the agent motion level, and familiarity with robotics); (ii) Qualitative questions is perceived as natural. regarding the application easy of use, naturalness and satis- As for the quantitative case, for each pair of conditions, fiability consisting of 6 questions; (iii) two question specific we try to correlate answers considering only users with high for the conditions involving the humanoid, dealing with the or low English level or familiarity with robots applications: sense of trust and movements naturalness of the robot. While the general information have been asked at the beginning of • high English level: there is a moderate correlation the tests, the testers have been asked to reply to the specific between App and Nao (r = 0.53, p < 0.01), and questions at the end of each experiment. We adopted a classical Nao and ENao (r = 0.41, p < 0.01); 36 Proc. of the 16th Workshop “From Object to Agents” (WOA15) June 17-19, Naples, Italy Fig. 6. Approval ratings average with respect to the qualitative questions. • low English level: as in the previous case, App and the other two interaction modalities, and a slightly increase Nao are significantly correlated (r = 0.43, p < 0.01), in the acceptance rate (but not yet significant). In fact, when as well as Nao and ENao (r = 0.64, p < 0.01); involved in an interaction, humans expect non-verbal signals from humanoid robots as well as they did with people. Indeed, • high familiarity: Pearson shows a strong correlation when robotic emphatic responses (Nao) are absent or not between App and Nao (r = 0.75, p < 0.01) and sufficient, trust decreases. between Nao and ENao (r = 0.82, p < 0.01). There is a moderate correlation for App and ENao (r = 0.60, In most cases, there are correlations between App results p < 0.01); and Nao, and Nao and ENao, but not between App and ENao. • low familiarity: finally, we have a moderate correlation In our opinion, the leading cause of these results is due to between App and Nao (r = 0.39, p < 0.01) and the smaller difference between the interaction with the mobile between Nao and ENao (r = 0.45, p < 0.01). application and Nao condition (e.g., they provide the same content, but one with text and the other through speech), and Both for Nao (F = 3.95, p = 0.05) and ENao (F = 4.89, between Nao and Enao conditions (e.g., they share the same p = 0.03), in the case of low and high familiarity with robots, interface – an embodied agent – but with different interaction ANOVA shows significant differences between these categories capabilities). Regarding the first and the third conditions, the of users. In both cases, the mean values of answers of users large difference between the two modes of interaction implies with high familiarity is greater than other users. There are not no significant correlations between each other. significant differences in grouping per English skills. In future works we will extend the pilot study by selecting more users in order to extend our evaluation and to achieve VI. C ONCLUSION more significant results. Social robots have been used in advertisements in public spaces mainly because of their greater ability to grab customer attention with respect to displays. Previous research mainly R EFERENCES investigated the advantage of a physical body in engaging the user in an interaction with respect to its virtual counterpart. In [1] C. Kidd and C. Breazeal, “Effect of a robot on user perceptions,” in this work, we compared, in a pilot study, the effect of a social Intelligent Robots and Systems, 2004. (IROS 2004). Proceedings. 2004 IEEE/RSJ Int. Conf. on, vol. 4, Sept 2004, pp. 3559–3564 vol.4. robot with different communication channels, with respect to a well–known interface such as a mobile application in providing [2] W. Wang and I. Benbasat, “Trust in and adoption of online recom- mendation agents,” Journal of the Association for Information Systems, recommendations, and evaluated the human behavior through vol. 6, no. 3, p. 72, March 2005. quantitative and qualitative analysis. From the qualitative ques- [3] S. M. Merritt and D. R. Ilgen, “Not all trust is created equal: Dis- tionnaire it arises that the users, on average, perceive the positional and history-based trust in human-automation interactions,” interaction with the mobile application (App) easier than those Human Factors, vol. 50, no. 2, pp. 194–210, 2008. with the social robot (Nao, ENao) independently from the [4] A. Powers, S. Kiesler, S. Fussell, and C. Torrey, “Comparing a computer degree of interaction capabilities. Furthermore, App appeared agent with a humanoid robot,” in 2nd ACM/IEEE Int. Conf. on Human- more natural and satisfying than both Nao and ENao robot. Robot Interaction (HRI), March 2007, pp. 145–152. This result naturally arises from the fact that the most of the [5] M. Shiomi, K. Shinozawa, Y. Nakagawa, T. Miyashita, T. Sakamoto, users have more familiarity with mobile applications rather T. Terakubo, H. Ishiguro, and N. Hagita, “Recommendation effects of a social robot for advertisement-use context in a shopping mall,” Int. than with robots. Moreover, the sole presence of the robot Journal of Social Robotics, vol. 5, no. 2, pp. 251–262, 2013. does not provide an improvement in the acceptance rate, while [6] W. Bainbridge, J. Hart, E. Kim, and B. Scassellati, “The effect of the additional communication capabilities provided by the presence on human-robot interaction,” in The 17th IEEE Int. Symposium ENao humanoid robot generate for the users a higher level on Robot and Human Interactive Communication (RO-MAN), Aug of satisfaction with respect to the expectations compared with 2008, pp. 701–706. 37 Proc. of the 16th Workshop “From Object to Agents” (WOA15) June 17-19, Naples, Italy [7] C. M. de Melo, L. Zheng, and J. Gratch, “Expression of Moral Emotions [10] K. Shinozawa, F. Naya, J. Yamato, and K. Kogure, “Differences in in Cooperating Agents,” in Proceedings of the 9th International Confer- effect of robot and screen agent recommendations on human decision- ence on Intelligent Virtual Agents (IVA), Amsterdam, The Netherlands, making,” Int. J. Hum.-Comput. Stud., vol. 62, no. 2, pp. 267–279, Feb. Sep. 2009. 2005. [8] T. B. R and B. Ross, “Emotional expressivity and trustworthiness: The [11] S. Dooms, T. De Pessemier, and L. Martens, “Movietweetings: a movie role of nonverbal behavior in the evolution of cooperation,” in Journal rating dataset collected from twitter,” in Workshop on Crowdsourcing of Nonverbal Behavior, A. R. Library, Ed., vol. 23(3), 2003. and Human Computation for Recommender Systems, CrowdRec at [9] C. Nass and Y. Moon, “Machines and mindlessness: Social responses RecSys 2013, 2013. to computers,” Journal of Social Issues, vol. 56, pp. 81–103, 2000. 38