This work presents an empirical comparison among three widespread word embedding techniques as Latent Semantic Indexing, Random Indexing and the more recent Word2Vec. Speci cally, we employed these techniques to learn a lowdimensional vector space word representation and we exploited it to represent both items and user pro les in a content-based recommendation scenario. The performance of the techniques has been evaluated against two state-ofthe-art datasets, and experimental results provided good insights which pave the way to several future directions.
Word Embedding techniques learn in a totally unsupervised way a low-dimensional vector space representation of words by analyzing their usage in (very) large corpora of textual documents. These approaches are recently gaining more and more attention, since they showed very good performance in a broad range of natural language processingrelated scenarios, ranging from sentiment analysis and machine translation to more challenging ones as learning a textual description of a given image1.
In a nutshell, all these techniques employ a large corpora of documents to encode the co-occurences between the terms, in order to learn both linguistic regularities as well as semantic nuances, according to their usage. Next, given this huge co-occurrences matrix, each technique use a di erent approach to obtain a smaller low-dimensional representation of each word occurring in the original corpus. An important feature which is common to all these technique is that the dimension of the representation (that is to say, the size of the vectors) is just a parameter of the model, so it can be set according to speci c constraint or peculiarities of the data.
However, although the e ectiveness of such techniques (especially when combined with deep neural network architectures) is already taken for granted, just a few work inves1http://googleresearch.blogspot.it/2014/11/a-picture-isworth-thousand-coherent.html tigated how well they do perform in recommender systemsrelated tasks. To this aim, in this work we de ned a very simple content-based recommendation framework based on word embeddings, in order to assess about the e ectiveness of such techniques in these scenarios as well. Speci cally, we rst exploited word embedding techniques to represent words in vector spaces. Next, we inferred a vector-space representation of the items by summing the representation of the words occurring in the document. Similarly, user pro les are represented by summing the document representation of the items the user liked. Finally, by exploiting classic similarity measures the available items can be ranked according to their descending similarity with respect to the user pro le, and recommendations can be provided, in a typical Top-N recommendation setting.
Clearly, this is a very basic formulation, since more negrained representations can be learned for both items and users pro les. However, this work just aims to preliminarily evaluate the e ectiveness of such representations in a simpli ed recommendation framework, in order to pave the way to several future research directions in the area.
Overview of the techniques. Latent Semantic
Indexing (LSI) [
Next, Random Indexing (RI) [
Finally, Word2Vec (W2V) is a recent technique proposed
by Mikolov et al. [
W2V ral network. In the rst step, weights in the network are randomly distributed as in RI. Next, the network is trained by using the Skip-gram methodology in order to model negrained regularities in word usage. At each step, weights are updated through Stochastic Gradient Descent and a vectorspace representation of each term is obtained by extracting the weights of the network at the end of the training.
In the experimental evaluation the performance of word embedding representations were compared against two stateof-the-art datasets as MovieLens (ML) and DBbook (DB)2. Moreover, we also compared the e ectiveness of the bestperforming con gurations to some widespead baselines.
Experimental Design. Experiments were performed by adopting di erent protocols: as regards ML, we carried out a 5-folds cross validation, while a single training/test split was used for DB. Textual content was obtained by mapping items to Wikipedia pages. For each word embedding technique we compared two di erent size of learned vectors: 300 and 500. As regards the baselines, we exploited MyMediaLite library3. We evaluated User-to-User (U2U-KNN) and Item-to-Item Collaborative Filtering (I2I-KNN) as well as the Bayesian Personalized Ranking Matrix Factorization (BPRMF). U2U and I2I neighborhood size was set to 80. while BPRMF was run by setting the factor parameter equal to 100. In both cases we chose the optimal values for the parameters. Finally, statistical signi cance was assessed by exploiting Wilcoxon and Friedman tests, chosen after running the Shapiro-Wilk test which revealed the non-normal distribution of the data.
Discussion of the results. The rst six columns of Table 1 provide the results of the comparison among the word embedding techniques. As regards ML, W2V emerged as the best-performing con guration for all the metrics took into account. The gap is signi cant when compared to both RI and LSI. Moreover, results show that the size of the vectors did not signi cantly a ect the overall accuracy of the algorithms (with the exception of LSI). This is an interesting outcome since with an even smaller word representation, word embeddings can obtain good results. However, the outcomes emerging from this rst experiments are controversial, since DBbook data provided opposite results: in this dataset W2V is the best-performing con guration only for F1@5. On the other side, LSI, which performed the worst on MovieLens data, overcomes both W2V and RI on F1@10 and F1@15. On a rst sights these results indicate nongeneralizable outcomes. However, it is likely that such behavior depends on speci c pecularities of the datasets which in turn in uence the way the approaches learn their vectorspace representations. A more throrough analysis is needed to obtain general guidelines which drive the behavior of such approaches.
Next, we compared our techniques to the above described baselines. Results clearly show that the e ectiveness of word embedding approaches is directly dependent on the sparsity of the data. This is an expected behavior since contentbased approaches can better deal with cold-start situations. In highly sparse dataset as DBbook (99.13% against 93.59% of MovieLens), content-based approaches based on word embedding tend to overcome the baselines. Indeed, all the approaches overcome I2I and U2U on F1@10 and F1@15 (W2V also overcomes I2I on F1@5). Furthermore, it is worth to note that on F1@10 and F@15 word embeddings can obtain results which are comparable (or even better on F1@15) to those obtained by BPRMF. This is a very important outcome, which de nitely con rms the e ectiveness of such techniques. Conversely, on less sparse datasets as MovieLens, CF algorithms overcome their content-based counterpart.
However, the overall outcomes emerging from this preliminary investigations are very promising: given that no speci c NLP task was performed on the data, it is likely that a more thorough processing of the content can lead to even better results. Thus, this investigation showed that word embedding approaches can represent a very interesting alternative to widespread CF approaches. In the following, we will further validate our results by also further investigating the e ectiveness of novel and richer textual data silos, as those coming from the Linked Open Data cloud.