Explaining automatic recommendations is an active area of research since it has shown an important eect on users' acceptance over the items recommended. However, there is a lack of research in explaining content-based recommendations of images based on visual features. In this paper, we aim to ll this gap by testing three dierent interfaces (one baseline and two novel explanation interfaces) for artistic image recommendation. Our experiments with N=121 users conrm that explanations of recommendations in the image domain are useful and increase user satisfaction, perception of explainability, relevance, and diversity. Furthermore, our experiments show that the results are also dependent on the underlying recommendation algorithm used. We tested the interfaces with two algorithms: Deep Neural Networks (DNN), with high accuracy but with dicult to explain features, and the more explainable method based on Aractiveness Visual Features (AVF). e beer the accuracy performance -in our case the DNN method- the stronger the positive eect of the explainable interface. Notably, the explainable features of the AVF method increased the perception of explainability but did not increase the perception of trust, unlike DNN, which improved both dimensions. ese results indicate that algorithms in conjunction with interfaces play a signicant role in the perception of explainability and trust for image recommendation. We plan to further investigate the relationship between interface explainability and algorithmic performance in recommender systems.
Online artwork recommendation has received lile aention
compared to other areas such as movies [
Objective. In this paper, we research the eect of explaining artistic image suggestions. In particular, we conduct a user study on Amazon Mechanical Turk under three dierent interfaces and two dierent algorithms. e three interfaces are: i) no explanations, ii) explanations based on similar images, and iii) explanations based on visual features. Moreover, the two algorithms are: Deep Neural Networks (DNN) and Aractiveness Visual Features (AVF). In our study, we used images provided by the online store UGallery (hp://www.UGallery.com/).
Research estions To drive our research, the following two questions were dened: a RQ1. Given three dierent types of interfaces, one baseline interface without explanations and two with them, employing similar image explanations and a feature bar chart, which one is perceived as most useful? a RQ2. Furthermore, based on the visual and content-based recommender algorithm chosen, are there observable dierences in how the three interfaces are perceived? 2
Relevant related research is collated in two sub-sections: First, we review research on recommending artistic images to people. Second we summarize studies on explaining recommender systems. Both are important to our problem at hand. e nal paragraph in this section highlights the dierences to previous work and our contributions to the existing literature in the area.
et al. [
Explaining Recommender Systems. ere are some related
works on explanations for recommender systems [
we focus on artistic images, to the best of our knowledge this is the rst work which studies the eect of explaining recommendations of images based on visual features. Our contributions are two-fold: i) we analyze and report the positive eect of explaining artistic recommendations especially for the VCBR based on neural features, and ii) by a user study we validate o-line results stating the superiority of neural visual features compared to aractiveness visual features over several dimensions, such as users’ perception of explainability, relevance, trust and general satisfaction. 3
In the following section we describe in detail our study methods. First, we introduce the dataset chosen for the purpose of our study. Second we introduce the three dierent explainable visual interfaces implemented which we evaluate. ird the two algorithms chosen for our study are revealed. Finally, the user study procedure is explained. 3.1
For the purpose of our study we rely on a dataset provided by the
online web store UGallery, which has been selling artwork for more
than 10 years [
e Explainable Recommender Interfaces In our study we explore the eect of explanations in visual contentbased artwork recommender systems. As such, our study contains conditions depending on how recommendations are displayed: i) no explanations, as shown in Figure 1, ii) explanations given by text and based on the top-3 most similar images a user liked in the past, as shown in Figure 2, and iii ) explanations employing a visual aractiveness bar chart and showing the most similar image of the user’s item prole, as presented in Figure 3.
In all three cases the interfaces are vertically scrollable. While Interface 1 (baseline) is able to show 5 images in a row at the same time, interfaces 2 and 3 are capable of showing one recommended image at the same time in one row to the user. 3.3
As mentioned earlier in this paper, we make use of two dierent content-based visual recommender approaches in our work. e reason for choosing content-based methods over collaborative ltering-based methods is grounded in the fact that once an item is sold via the UGallery store, it is not available anymore (every item preference elicitation
Algorithm: Within subjects (repeated measures)
Swap order of algorithm randomly
DNN AVF Interface 1 No explanation No explanation Interface 2 Explanation based on Explanation based on top3 Interface:
top3 similar images similar images Bsuetbwjeecetsn Interface 3 Explanation based on Explanation based on
top3 similar images barchart of visual features pre-study survey post-DNN survey post-AVF survey is unique) and hence traditional collaborative ltering approaches do not apply.
DNN Visual Feature (DNN) Algorithm. e rst
algorithmic approach we employed was based on image similarity, itself
based on features extracted with a deep neural network. e output
vector representing the image is usually called an image’s visual
embedding. e visual embedding in our experiment was a vector
of features obtained from an AlexNet, a convolutional deep neural
network developed to classify images [
Attractiveness Visual Features (AVF) Algorithm. e
second content-based algorithmic recommender approach employed
was a method based on visual aractiveness features. San Pedro
and Siersdorfer in [
Due space constrains, the details to calculate the features are
described in the article by Messina et al. [
Computing Recommendations. Given a user u who has
consumed a set of artworks Pu , a constrained prole size K , and an
arbitrary artwork i from the inventory, the score of this item i to
be recommended to u is:
score u; i X
minrK;¶Pu ¶x
<
r 1
max r rsim ViX ; VjX x
jϵ Pu
minrK; ¶Pu ¶x
;
(1)
where VzX is a feature vector of item z obtained with method X ,
where X can be either a pre-trained AlexNet (DNN) or aractiveness
visual features (AVF). max r denotes the r -th maximum value,
e.g., if r = 1 it is the overall maximum, if r = 2 it is the second
maximum, and so on. We compute the average similarity of the
top-K most similar images because as shown in Messina et al. [
Vi Vj ½Vi ½½Vj ½ (2)
Both methods use the same formula to calculate the
recommendations. e dierence is in the origin of the visual features. For
the DNN method, the features were extracted with the AlexNet
[
To evaluate the performance of our explainable interfaces we conducted a user study in Amazon Mechanical Turk using a 3x2 mixed design: 3 interfaces (between-subjects) and 2 algorithms (withinsubjects, DNN and AVF). e interface conditions were: Interface 1: interface without explanations, as in Figure 1; Interface 2: each item recommendation is explained based on the top 3 most similar images in the user prole, as in Figure 2; and Interface 3: only for AVF, based on a bar chart of visual features, as in Figure 3. Notice that in the condition Interface 3, for DNN we used the explanation based on top 3 most similar images, because the neural embedding of 4,096 dimensions has no human-interpretable features to show in a bar chart.
To compute the recommendations for each of the three interface conditions two recommender algorithms were chosen: one based on DNN visual features, and the other based on aractiveness visual features (AVF). e order in which the algorithms were presented was chosen at random to diminish the chance of a learning eect.
e full study procedure is shown in Figure 4. Participants
accepted the study on Mechanical Turk (hps://www.mturk.com)
and were redirected to a web application. Aer accepting a consent
form, they are redirected to the pre-study survey, which collects
demographic data (age, gender) and a subject’s previous knowledge
of art based on the test by Chaerjee et al. [
Following this, they had to perform a preference elicitation task. In this step, the users had to “like” at least ten paintings, using a Pinterest-like interface. Next, they were randomly assigned to one interface condition. In each condition, they again provided feedback (rating with 1-5 scale to each image) to top ten recommendations of images with employing either the DNN or the AVF algorithm (also assigned at random as discussed before). Finally, the participants were asked to next answer a post-algorithm survey. e dimensions evaluated in the post-algorithm survey are the same for DNN and AVF algorithms, and they are shown in Table 1. is process is repeated for the second algorithm as well. Once the participants nished answering the second post study survey, they were redirected to the nal view, where they received a survey code for later payment in Amazon Mechanical Turk. 4
RESULTS e study was nished by in total 200 users out of which 121 were able to answer our validation questions successfully and hence were included in the results. In total, we had two validation questions set to check for aention of our study participants. Filtering out users not responding properly to these questions allowed us to include 41 users for the Interface 1 condition, 41 users for Interface 2 condition and 39 users for Interface 3 condition. In total, participants were paid an amount of 0.40 USD per study, which took them around 10 minutes to complete.
Our subjects were between 18 to over 60 years old. 36% were between 25 to 32 years old, and 29% between 32 to 40 years old. Females made up 55.4% . 12% just nished high school, 31% had a some college degree, 57% had a bachelor’s, master’s or Ph.D. degree. Only 8% reported some visual impairment. W.r.t. their understanding about art, 20% had null experience, 48% had aended 1 or 2 lessons, and 32% reported to have aended 3 or more at high school level or above. 20% of our subjects also reported that they had almost never visited a museum or an art gallery; 36% do this once a year; and 44% do this once every 1 or 6 months.
Dierences between Interfaces. Table 2 summarizes the results of the user study. First we compared interface performance and then we looked at the algorithmic performance. e explainable interfaces (Interface 2 and 3) signicantly improved the perception of explainability compared to Interface 1 under both algorithms. ere is also a signicant improvement over Interface 1 in terms of relevance and diversity, but this is only achieved by the DNN method when this is compared against the AVF method using the interface 3. Interestingly, this is the condition where the interface is more transparent, since it explains exactly what is used to recommend (brightness, saturation, sharpness, etc.). People report that they understand why the images are recommended (70.4), but since the relevance is rather insucient (56.2), the perception of trust is reported as low (55.4).
the dimension Diverse where AVF was signicantly beer, DNN was perceived more positively than AVF at large. In interfaces 2 and 3, the DNN method was perceived signicantly beer in 5 dimensions (explainability, relevance, interface satisfaction, interest for eventual use, and trust), as well as higher average rating.
Overall, the results indicate that the explainable interface based on top 3 similar images works beer than an interface without explanation. Moreover, this eect is enhanced by the accuracy of the algorithm, so even if the algorithm has no explainable features (DNN) it could induce more trust if the user perceives a larger predictive preference accuracy.
In this paper, we have studied the eect of explaining recommendation of images employing three dierent recommender interfaces, as well as interactions with two dierent visual content-based recommendation algorithms: one with high predictive accuracy but with unexplainable features (DNN), and another with lower accuracy but with higher potential for explainable features (AVF).
e rst result, which answers RQ1, shows that explaining the images recommended has a positive eect vs. no explanation. Moreover, the explanation based on top 3 similar images presents the best results, but we need to consider that the alternative method, explanations based on visual features, was only used with the AVF. is result is preliminary and opens a path of research in terms of new interfaces which could help to explain the features learned by a deep neural network of images.
Regarding RQ2, we see that the algorithm used plays an important role in conjunction with the interface. DNN is perceived beer than AVF in most dimensions evaluated, showing that further research should focus on the interaction between algorithm and explainable interfaces. In the future we will expand this work to other datasets, beyond artistic images, to generalize our results. 6
e authors from PUC Chile were funded by Conicyt, Fondecyt grant 11150783, as well as by the Millennium Institute for Foundational Research on Data (IMFD).