In the last years, deep learning has shown to be a game-changing technology in artificial intelligence thanks to the numerous successes it reached in diverse application fields. Among others, the use of deep learning for the recommendation problem, although new, looks quite promising due to its positive performances in terms of accuracy of recommendation results. In a recommendation setting, in order to predict user ratings on unknown items a possible configuration of a deep neural network is that of autoencoders typically used to produce a lower dimensionality representation of the original data. In this paper we present KG-AUTOENCODER, an autoencoder that bases the structure of its neural network on the semantics-aware topology of a knowledge graph thus providing a label for neurons in the hidden layer that are eventually used to build a user profile and then compute recommendations. We show the effectiveness of KG-AUTOENCODER in terms of accuracy, diversity and novelty by comparing with state of the art recommendation algorithms.
Recommender systems (RS) have become pervasive tools we experience in our everyday life. While browsing a catalog of items RSs exploit users’ past preferences in order to suggest new items they might be interested in.
Knowledge Graphs have been recently adopted to represent
items, compute their similarity and relatedness [
Knowledge-aware and Conversational Recommender Systems (KaRS) Workshop 2018 (co-located with RecSys 2018), October 7, 2018, Vancouver, Canada. 2018. ACM ISBN Copyright for the individual papers remains with the authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors..
Another technology that surely boosted the development of a
new generation of smarter and more accurate recommender
systems is deep learning [
In this paper, we show how autoencoders technology can benefit from the existence of a Knowledge Graph to create a representation of a user profile that can be eventually exploited to predict ratings for unknown items. The main intuition behind the approach is that both ANN and Knowledge Graph expose a graph-based structure. Hence, we may imagine building the topology of the inner layers in the ANN by mimicking that of a Knowledge Graph. The remainder of this paper is structured as follows: in the next section, we discuss related works on recommender systems exploiting deep learning, knowledge graphs and Linked Open Data. Then, the basic notions of the technologies we adopted are introduced in Section 3. The proposed recommendation model is described in Section 4 while in Section 5 we present the experimental setting and evaluation. Conclusions and Future Work close the paper. 2
learning techniques is for sure one of the main advances of the
last years in the field of recommender systems. In [
A pure Collaborative-Filtering (CF) model based on autoencoders
is described in [
Wang et al. [
Autoencoders. An Artificial Neural Network (ANN) is a mathematical model used to learn the relationships which underlie in a given set of data. Starting from them, after a training phase, an ANN can be used to predict a single value or a vector, for regression or classification tasks.
Basically, an ANN consists of a bunch of nodes, called neurons, distributed among three diferent kinds of layers: the input layer, one or more hidden layers, and the output layer. Typically, a neuron of a layer is connected to all the neurons of the next layer, making the ANN a fully connected network.
Autoencoders are ANNs which try to set the output values equal to the input ones, modeling an approximation of the identity function y = f (x ) = x . Roughly, they are forced to predict the same values they are fed with. Therefore, the number of output units and one of the input nodes is the same, i.e. |x | = |y |. The aim of such a task is to obtain a new representation of the original data based on the values of the hidden layer neurons. In fact, each of these layers projects the input data in a new Euclidean space whose dimensions depend on the number of the nodes in the hidden layer.
Therefore, when we use an autoencoder, we are not interested at its output, but at the encoded representation it computes: in this way, we can leverage the implicit knowledge behind the original data, performing the so-called feature extraction task. The actual meaning of each dimension (represented by hidden nodes) in the new space is unknown, but we can be sure that they are based on latent patterns binding the training cases.
KG. In 2012, Google announced its Knowledge Graph1 as a new tool to improve the identification and retrieval of entities in return to a search query. A Knowledge Graph is a form of representation of knowledge through a semantic (labelled) network that allows a system to store the human knowledge in a structured format well understandable by a computer agent.
12_Monkeys Ghostwritten e nr e g Drama
Post-apocalyptic_films
In the meantime, some communities started to build their own KGs; among them, the most important is for sure DBpedia2. We may investigate relationships among entities in the KG by exploiting graph data sources and hence discover meaningful paths within the graph.
In figure 1 we show an excerpt of the DBpedia graph, involving some entities in the movie domain. Interestingly we see that DBpedia encodes both factual information, e.g. “Cloud_Atlas_(film) has director The_Wachowskis”, and categorical one such as “Cloud_Atlas_(film) has subject Post-apocalyptic_films ”. 4
The main idea of our approach is to map the connections in a knowledge graph (KG) with those between units from layer i to layer i+1, as shown in Figure 2. There we see that we injected only categorical information in the autoencoder and we left out factual one. As a matter of fact, if we analyze these two kinds of information in DBpedia we may notice that: • the quantity of categorical information is higher than the factual one. If we consider movies, the overall number of entities they are related with is lower than the overall number of categories; • categorical information is more distributed over the items than the factual one. Going back to movies we see that they are more connected with each other via categories than via other entities.
Hence, we may argue that for a recommendation task where we are looking for commonalities among items, categorical data may result in more meaningful than the factual one. The main assumption behind this choice is that, for instance, if a user rated positively Cloud_Atlas this may be interpreted as a positive rating for the connected category Post-apocalyptic_films .
In order to test our assumption, we mapped the autoencoder network topology with the categorical information related to items rated by users. As we build a diferent autoencoder for each user depending on the items she rated in the past, the mapping with 1https://googleblog.blogspot.it/2012/05/introducing-knowledge-graph-things-not. html
a KG makes the hidden layer of variable length in the number of units, depending on how much categorical information is available for items rated by the specific user.
5.0 4.0 2.0
The Matrix The Karate Kid (film 2010) Astro Boy (film) subject
subject subject subject subject
The Matrix The Karate Kid (film 2010) Astro Boy (film) 5.0 4.0 2.0
Let n be the number of items rated by u available in the graph and
Ci = {ci1, ci2, . . . , cim } be the set of m categorical nodes associated
in the KG to the item i. Then, F u = Ðn
i=1 Ci is the set of features
mapped into the hidden layer for the user and the overall number
of hidden units is equal to |F u |. Once the neural network setup is
done, the training process takes place, feeding the neural network
with ratings provided by the user, normalized in the interval [
Nodes in the hidden layer correspond to categorical information in the knowledge graph. At every iteration of the training process, backpropagation will change weights accordingly on edges among units in the layers, such that the sum of entering edges in an output unit will reconstruct the user rating for the item represented by that unit. Regarding the nodes in the hidden layer, we may interpret the sum of the weights associated to entering edges computed at the end of the training process as the importance of that feature in the generation of an output which, in our case, are the ratings provided by the user. 4.1
Once the network converges we have a latent representation of
features associated with a user profile together with their weights.
However, very interestingly, this time the features represented by
nodes in the hidden layer also have an explicit meaning as they
are in a one to one mapping with categories in a knowledge graph.
Our autoencoder is, therefore, able to learn the semantics behind
the ratings of each user and weight them through backpropagation.
In our current implementation we used the well known sigmoid
1
σ (x ) = 1+e−x activation function since we normalized the design
matrix to be within [
Starting from the trained autoencoder, we may build a user proifle by considering the categories associated to the items she rated in the past as features and by assigning them a value according to the weights associated to the edges entering the corresponding hidden units. Given a user u, the weight associated to a feature c is then the summation of the weights wku (c) associated to the edges entering the hidden node representing the Knowledge Graph category c after training the autoencoder with the ratings of u. More formally, we have: ωu (c) = |I Õn(c)| k=1 wku (c) where In(c) is the set of the edges entering the node representing the feature c. We remember that since the autoencoder is not fully connected, |In(c)| varies depending on the related connections to the category c in the knowledge graph.
By means of the weights associated with each feature, we can now model a user profile composed by a vector of weighted categorical features. Given F u as the set of categories belonging to all the items rated by u and F = Ðu ∈U F u as the set of all features among all the users in the system we have for each user u ∈ U and for each feature c ∈ F :
P (u) = {⟨c, ω⟩ | ω = ωu (c) if c ∈ F u }
Considering that users provide a diferent number of ratings, we
have an unbalanced distribution in the dimension of user profiles.
Moreover, as a user usually rate only a small subset of the entire
catalog, we have a massive number of missing features belonging
to items not rated by u. In order to compute values associated
with missing features, we leverage an unsupervised deep learning
model inspired by the word2vec approach [
In our scenario, we may imagine replacing sentences represented by sequences of words with user profiles represented by sequences of categories in c ∈ F u and then use the word2vec approach to compute for a given user u the weight of missing features c ′ < F u .
We need to prepare the user profiles P (u) to be processed by word2vec. Hence, we first generate a corpus made of sequences of ordered features where the order is given by ω. The very preliminary step is that of selecting an order among elements c ∈ F u which results coherently for all u ∈ U thus moving from the set P (u) to a representative sequence of elements s(u).
For each ⟨c, ω⟩ ∈ P (u) we create a corresponding pair ⟨c, norm(ω)⟩ with norm being the mapping function
norm : [
Pnorm (u) = {⟨c, norm(ω)⟩ | ⟨c, ω⟩ ∈ P (u)} For each normalized user profile set Pnorm (u) we then build the corresponding sequence
s(u) = [. . . , ⟨ci , norm(ωiu )⟩, . . . ⟨cj , norm(ωuj )⟩, . . .] with ωiu ≥ ωuj .
Once we have the set S = {s(u) | u ∈ U } we can feed the word2vec algorithm with this corpus in order to find patterns of features according to their distribution across all users. In the prediction phase, by using each user’s sequence of features s(u) as input for the trained word2vec model, we estimate the probability of ⟨c ′, norm(ω ′)⟩ ∈ Ðv ∈U Pnorm (v) − Pnorm (u) to belong to the given context, or rather to be relevant for u. In other words, we compute p(⟨c ′, norm(ω ′)⟩ | s(u)).
It is worth noticing that given c ′ ∈ F u we may have multiple pairs with c ′ as first element in Ðv ∈U Pnorm (v) − Pnorm (u). For instance, given the category dbc:Kung_fu_films we may have both ⟨dbc : Kung_fu_films, 0.2⟩ and ⟨dbc : Kung_fu_films, 0.5⟩, with the corresponding probabilities p(⟨dbc : Kung_fu_films, 0.2⟩ | s(u)), p(⟨dbc : Kung_fu_films, 0.5⟩ | s(u)). Still, as we want to add the category dbc:Kung_fu_films together with its corresponding weight only once in the user profile we select only the pair with the highest probability. The new user profile is then Pˆ(u) = P (u)∪{⟨c, ω⟩ |
argmax ω ∈ {0.1, ...,1} We point out that while the original P (u) is built by exploiting only content-based information, the enhanced user profile Pˆ(u) also considers collaborative information as it based also on the set S containing a representation for the profiles of all the users in U . 4.2
Given the user profiles represented as vectors of weighted features, recommendations are then computed by using a well-known knearest neighbors approach. User similarities are found through projecting their user profile in a Vector Space Model, and then similarities between each pair of users u and v is computed using the cosine similarity.
For each user u we find the top-k similar neighbors to infer the rate r for the item i as the weighted average rate that the neighborhood gave to it: r (u, i) = Ík j=1 sim(u, vj ) · r (vj , i) Ík
j=1 sim(u, vj ) 3In our current implementation we use a standard minmax normalization. where r (vj , i) is the rating assigned to i by the user vj . We use then ratings from Equation (1) to provide top-N recommendation for each user. 5
In this section, we present the experimental evaluations performed on three diferent datasets. We first describe the structure of the datasets used in the experiments and the evaluation protocol and then we move to the metrics adopted for the evaluation and the discussion of obtained results.
Our experiments can be reproduced through the implementation available on our public repository4. 5.1
In order to validate our approach we performed experiments on the three datasets summarized in Table 1.
Amazon Digital Music LibraryThing #users
In our experiments, we referred to the freely available knowledge graph of DBpedia5. The mapping contains 22,959 mapped items for MovieLens 20M6, 4,077 items mapped for Amazon Digital Music7 and 9,926 items mapped for LibraryThing8. For our experiments, we removed from the datasets all the items without a mapping in DBpedia.
5https://dbpedia.org
6https://grouplens.org/datasets/movielens/20m/
7http://jmcauley.ucsd.edu/data/amazon/
8https://www.librarything.com
which leverages Bayesian Personalized Ranking as the objective
function. WRMF [
In terms of accuracy, we can see that KG-AUTOENCODER outperforms our baselines on both MovieLens 20M and Amazon Digital Music datasets, while on LibraryThing the achieved results are quite the same. In particular, on the LibraryThing dataset, only the fully-connected autoencoder performs better than our approach with regard to accuracy.
Concerning diversity, we get much better results on all the datasets. Furthermore, by analyzing the gathered results, it seems that our approach provides very discriminative descriptions for each user, letting us identify the most efective neighborhood and compute both accurate and diversified recommendations. As a matter of fact, we achieve the same results in terms of accuracy as the baselines by suggesting much more items.
As shown in Table 2, KG-AUTOENCODER performs better on those datasets whose items can be associated with a large amount of categorical information, which implies the usage of many hidden units. This occurs because very complex functions can be modeled by ANNs if enough hidden units are provided, as Universal Approximation Theorem points out. For this reason, our approach turned out to work better on MovieLens 20M dataset (whose related neural networks have a high number of hidden units) rather than the others. In particular, the experiments on LibraryThing dataset show that the performances get worse as the number of the neurons decreases, i.e. available categories are not enough.
Amazon Digital Music LibraryThing avg #features
In this paper, we have presented a recommendation approach that
combines the computational power of deep learning with the
representational expressiveness of knowledge graphs. As for classical
9http://mymedialite.net
10https://keras.io
applications of autoencoders to feature selection, we compute a
latent representation of items but, in our case, we attach an explicit
semantics to selected features. This allows our system to exploit
both the power of deep learning techniques and, at the same time,
to have a meaningful and understandable representation of the
trained model. We used our approach to autoencode user ratings in
a recommendation scenario via the DBpedia knowledge graph and
proposed an algorithm to compute user profiles then adopted to
provide recommendations based on the semantic features we extract
with our autoencoder. Experimental results show that we are able to
outperform state of the art recommendation algorithms in terms of
accuracy and diversity. Furthermore, we will compare our approach
with other competitive baselines as suggested in more recent works
[
Knowledge-aware and Conversational Recommender Systems (KaRS) Workshop 2018 (co-located with RecSys 2018), October 7, 2018, Vancouver, Canada.
V. Bellini et al.
nDCG
100