Online artwork recommendation has received li le a ention compared to other areas such as movies [1,10], music [4,16] or pointsof-interest [25,28,29]. e rst works in the area date from 2006-2007 such as the CHIP [2] project, which implemented traditional techniques such as content-based and collaborative ltering for artwork recommendation at the Rijksmuseum, and the m4art system by Van den Broek et al. [26], which used histograms of color to retrieve similar artworks where the input query was a painting image. More recently, deep neural networks (DNN) have been used for artwork recommendation and are the current state-of-the-art model [7,12], which is rather expected considering that DNNs are the top performing models for obtaining visual features for several tasks, such as image classi cation [15], and scene identi cation [23]. However, no user study has been conducted to validate the performance of DNNs versus other visual features. is aspect is important since past works have shown that o -line results might not always replicate when tested with actual users [14,17]. Moreover, we provide evidence of the important value of explanations in artwork recommender systems over several dimensions of user perception. Visual features obtained from DNNs are still di cult to explain to users, despite current e orts to understand them and explain them [20]. In contrast, features of visual a ractiveness could be easily explained, based on color, brightness or contrast [21]. Explanations in recommender systems have been shown to have a signi cant e ect on user satisfaction [24], and, to the best of our knowledge, no previous work has shown how to explain recommendations of images based on visual features. Hence, there is no study of the e ect on users when explaining images recommended by a Visual Content-based Recommender (Hereina er, VCBR).
Objective. In this paper, we research the e ect of explaining artistic image suggestions. In particular, we conduct a user study on Amazon Mechanical Turk under three di erent interfaces and two di erent algorithms. e three interfaces are: i) no explanations, ii) explanations based on similar images, and iii) explanations based on visual features. Moreover, the two algorithms are: Deep Neural Networks (DNN) and A ractiveness Visual Features (AVF). In our study, we used images provided by the online store UGallery (h p://www.UGallery.com/).
Research estions To drive our research, the following two questions were de ned: • RQ1. Given three di erent types of interfaces, one baseline interface without explanations and two with them, employing similar image explanations and a feature bar chart, which one is perceived as most useful? • RQ2. Furthermore, based on the visual and content-based recommender algorithm chosen, are there observable di erences in how the three interfaces are perceived?
Relevant related research is collated in two sub-sections: First, we review research on recommending artistic images to people. Second we summarize studies on explaining recommender systems.
Both are important to our problem at hand. e nal paragraph in this section highlights the di erences to previous work and our contributions to the existing literature in the area.
Recommendations of Artistic Images. e works of Aroyo et al. [2] with the CHIP project and Semeraro et al. [22] with FIRSt (Folksonomy-based Item Recommender syStem) made early contributions to this area using traditional techniques. More complex methods were implemented recently by Benouaret et al. [3], using context obtained through a mobile application, that makes a museum tour recommendation. Finally, the work of He et al. addresses digital artwork recommendations based on pre-trained deep neural visual features [12], and the work of Dominguez et al. [7] and Messina et al. [18] compared neural against traditional visual features. None of the aforementioned works performed a user study under explanation interfaces to generalize their results.
Explaining Recommender Systems. ere are some related works on explanations for recommender systems [24]. ough a good amount of research has been published in the area, to the best of our knowledge, no previous research has conducted a user study to understand the e ect of explaining recommendation of artwork images based on di erent visual features.
e closest works in this aspect are researches oriented to automatically add caption to images [9,19] or to explain image classi cations [13], but they are not directly related to personalized recommender systems.
Di erences to Previous Research & Contributions. Although we focus on artistic images, to the best of our knowledge this is the rst work which studies the e ect of explaining recommendations of images based on visual features. Our contributions are two-fold: i) we analyze and report the positive e ect of explaining artistic recommendations especially for the VCBR based on neural features, and ii) by a user study we validate o -line results stating the superiority of neural visual features compared to a ractiveness visual features over several dimensions, such as users' perception of explainability, relevance, trust and general satisfaction.
In the following section we describe in detail our study methods. First, we introduce the dataset chosen for the purpose of our study. Second we introduce the three di erent explainable visual interfaces implemented which we evaluate. ird the two algorithms chosen for our study are revealed. Finally, the user study procedure is explained.
For the purpose of our study we rely on a dataset provided by the online web store UGallery, which has been selling artwork for more than 10 years [27]. ey support emergent artists by helping them sell their artwork online. For our research, UGallery provided us with an anonymized dataset of 1,371 users, 3,490 items and 2,846 purchases (transactions) of artistic artifacts, where all users have made at least one transaction. On average, each user bought 2-3 items over recent years .
In our study we explore the e ect of explanations in visual contentbased artwork recommender systems. As such, our study contains conditions depending on how recommendations are displayed: i) no explanations, as shown in Figure 1, ii) explanations given by text and based on the top-3 most similar images a user liked in the past, as shown in Figure 2, and iii ) explanations employing a visual a ractiveness bar chart and showing the most similar image of the user's item pro le, as presented in Figure 3. In all three cases the interfaces are vertically scrollable. While Interface 1 (baseline) is able to show 5 images in a row at the same time, interfaces 2 and 3 are capable of showing one recommended image at the same time in one row to the user.
As mentioned earlier in this paper, we make use of two di erent content-based visual recommender approaches in our work.
e reason for choosing content-based methods over collaborative ltering-based methods is grounded in the fact that once an item is sold via the UGallery store, it is not available anymore (every item is unique) and hence traditional collaborative ltering approaches do not apply. DNN Visual Feature (DNN) Algorithm. e rst algorithmic approach we employed was based on image similarity, itself based on features extracted with a deep neural network. e output vector representing the image is usually called an image's visual embedding. e visual embedding in our experiment was a vector of features obtained from an AlexNet, a convolutional deep neural network developed to classify images [15]. In particular, we use an AlexNet model pre-trained with the ImageNet dataset [6]. Using the pre-trained weights, for every image a vector of 4,096 dimensions was generated with the Ca e (h p://ca e.berkeleyvision.org/) framework. We resized every image to a 227x227 image. is is the standard pre-processing needed to use the AlexNet.
Attractiveness Visual Features (AVF) Algorithm. e second content-based algorithmic recommender approach employed was a method based on visual a ractiveness features. San Pedro and Siersdorfer in [21] proposed several explainable visual features that to a great extent, can capture the a ractiveness of an image posted on Flickr. Following their procedure, for every image in our UGallery dataset we calculated: (a) average brightness, (b) saturation, (c) sharpness, (d) RMS-contrast, (e) colorfulness and (f) naturalness. In addition, we added (g) entropy, which is a good way to characterize and measure the texture of an image [11]. ese metrics have also been used in another study [8], where we show how to nudge people with a ractive images to take up more healthy recipe recommendations. To compute these features, we used the original size of the images and did not pre-process them.
Due space constrains, the details to calculate the features are described in the article by Messina et al. [18] Computing Recommendations. Given a user u who has consumed a set of artworks P u , a constrained pro le size K, and an arbitrary artwork i from the inventory, the score of this item i to be recommended to u is:
where V X z is a feature vector of item z obtained with method X , where X can be either a pre-trained AlexNet (DNN) or a ractiveness visual features (AVF). max (r ) denotes the r -th maximum value, e.g., if r = 1 it is the overall maximum, if r = 2 it is the second maximum, and so on. We compute the average similarity of the top-K most similar images because as shown in Messina et al. [18], for di erent users, the recommendations match be er using smaller subsets of the entire user pro le. Users do not always look to buy a painting similar to one they bought before, but they look for one that resembles a set of artworks that they liked. sim(V i , V j ) denotes a similarity function between vectors V i and V j . In this particular case, the similarity function used was cosine similarity:
Both methods use the same formula to calculate the recommendations. e di erence is in the origin of the visual features. For the DNN method, the features were extracted with the AlexNet [15], and in the case of AVF, the features were extracted based on San Pedro et al. [21].
To evaluate the performance of our explainable interfaces we conducted a user study in Amazon Mechanical Turk using a 3x2 mixed design: 3 interfaces (between-subjects) and 2 algorithms (withinsubjects, DNN and AVF). e interface conditions were: Interface 1: interface without explanations, as in Figure 1; Interface 2: each item recommendation is explained based on the top 3 most similar images in the user pro le, as in Figure 2; and Interface 3: only for AVF, based on a bar chart of visual features, as in Figure 3. Notice that in the condition Interface 3, for DNN we used the explanation based on top 3 most similar images, because the neural embedding of 4,096 dimensions has no human-interpretable features to show in a bar chart.
To compute the recommendations for each of the three interface conditions two recommender algorithms were chosen: one based on DNN visual features, and the other based on a ractiveness visual features (AVF). e order in which the algorithms were presented was chosen at random to diminish the chance of a learning e ect.
e full study procedure is shown in Figure 4. Participants accepted the study on Mechanical Turk (h ps://www.mturk.com) and were redirected to a web application. A er accepting a consent form, they are redirected to the pre-study survey, which collects demographic data (age, gender) and a subject's previous knowledge of art based on the test by Cha erjee et al. [5].
Table 2: Results of users' perception over several evaluation dimensions, de ned in Table 1 . Scale 1-100 (higher is better), except for Average rating (scale 1-5). DNN: Deep Neural Network, and AVF: Attractiveness visual features. e symbol ↑ 1 indicates interface-wise signi cant di erence (di erences between interfaces using the same algorithms). e * symbol denotes algorithm-wise statistical di erence (comparing a dimension between algorithms, using the same interface). Following this, they had to perform a preference elicitation task. In this step, the users had to "like" at least ten paintings, using a Pinterest-like interface. Next, they were randomly assigned to one interface condition. In each condition, they again provided feedback (rating with 1-5 scale to each image) to top ten recommendations of images with employing either the DNN or the AVF algorithm (also assigned at random as discussed before). Finally, the participants were asked to next answer a post-algorithm survey.
e dimensions evaluated in the post-algorithm survey are the same for DNN and AVF algorithms, and they are shown in Table 1. is process is repeated for the second algorithm as well. Once the participants nished answering the second post study survey, they were redirected to the nal view, where they received a survey code for later payment in Amazon Mechanical Turk.
e study was nished by in total 200 users out of which 121 were able to answer our validation questions successfully and hence were included in the results. In total, we had two validation questions set to check for a ention of our study participants. Filtering out users not responding properly to these questions allowed us to include 41 users for the Interface 1 condition, 41 users for Interface 2 condition and 39 users for Interface 3 condition. In total, participants were paid an amount of 0.40 USD per study, which took them around 10 minutes to complete.
Our subjects were between 18 to over 60 years old. 36% were between 25 to 32 years old, and 29% between 32 to 40 years old. Females made up 55.4% . 12% just nished high school, 31% had a some college degree, 57% had a bachelor's, master's or Ph.D. degree. Only 8% reported some visual impairment. W.r.t. their understanding about art, 20% had null experience, 48% had a ended 1 or 2 lessons, and 32% reported to have a ended 3 or more at high school level or above. 20% of our subjects also reported that they had almost never visited a museum or an art gallery; 36% do this once a year; and 44% do this once every 1 or 6 months.
Di erences between Interfaces. Table 2 summarizes the results of the user study. First we compared interface performance and then we looked at the algorithmic performance. e explainable interfaces (Interface 2 and 3) signi cantly improved the perception of explainability compared to Interface 1 under both algorithms.
ere is also a signi cant improvement over Interface 1 in terms of relevance and diversity, but this is only achieved by the DNN method when this is compared against the AVF method using the interface 3. Interestingly, this is the condition where the interface is more transparent, since it explains exactly what is used to recommend (brightness, saturation, sharpness, etc.). People report that they understand why the images are recommended (70.4), but since the relevance is rather insu cient (56.2), the perception of trust is reported as low (55.4).
Di erences between Algorithms. With the only exception of the dimension Diverse where AVF was signi cantly be er, DNN was perceived more positively than AVF at large. In interfaces 2 and 3, the DNN method was perceived signi cantly be er in 5 dimensions (explainability, relevance, interface satisfaction, interest for eventual use, and trust), as well as higher average rating.
Overall, the results indicate that the explainable interface based on top 3 similar images works be er than an interface without explanation. Moreover, this e ect is enhanced by the accuracy of the algorithm, so even if the algorithm has no explainable features (DNN) it could induce more trust if the user perceives a larger predictive preference accuracy.
In this paper, we have studied the e ect of explaining recommendation of images employing three di erent recommender interfaces, as well as interactions with two di erent visual content-based recommendation algorithms: one with high predictive accuracy but with unexplainable features (DNN), and another with lower accuracy but with higher potential for explainable features (AVF). e rst result, which answers RQ1, shows that explaining the images recommended has a positive e ect vs. no explanation. Moreover, the explanation based on top 3 similar images presents the best results, but we need to consider that the alternative method, explanations based on visual features, was only used with the AVF. is result is preliminary and opens a path of research in terms of new interfaces which could help to explain the features learned by a deep neural network of images.
Regarding RQ2, we see that the algorithm used plays an important role in conjunction with the interface. DNN is perceived be er than AVF in most dimensions evaluated, showing that further research should focus on the interaction between algorithm and explainable interfaces. In the future we will expand this work to other datasets, beyond artistic images, to generalize our results.
e authors from PUC Chile were funded by Conicyt, Fondecyt grant 11150783, as well as by the Millennium Institute for Foundational Research on Data (IMFD).