This paper presents the joint work of the Universities of Grenoble and Saint-Etienne at CLEF 2016 Social Book Search Suggestion Track. The approaches studied are based on personalization, considering the user's pro le in the ranking process. The pro le is ltered using Word Embedding, by proposing several ways to handle the generated relationships between terms. We nd that tackling the problem of \non-topical" only queries is a great challenge in this case. The o cial results show that Word Embedding methods are able to improve results in the SBS case.
This paper presents the joint participation of the MRIM group from the Laboratoire d'Informatique de Grenoble and the Laboratoire Hubert Curien de SaintEtienne at CLEF 2016 Social Book Search (SBS) Suggestion Track. The goal of this track is to evaluate book retrieval approaches using the metadata of the book and user-generated content [6]. Considering our previous participation to this track in 2015 [9], our nding was that taking into account the whole user pro le is not adequate. Hence, we believe that a selection of an adequate set of terms related both to the user's pro le and to his query could enhance the results. Such selection of terms is then matched against the documents, and fused with the query/document matching values. The work described here relies on Word Embedding (word2vec [7]) as a mean to help to select those terms.
Recent works investigate the use of Word Embedding for enhancing IR e ectiveness [1, 3, 8] or classi cation [5]. Word Embedding techniques seek to embed representations of words. For example, two vectors !t0 and !t1 , corresponding to the words t0 and t1, are close in a space of N dimensions if they have similar contexts and vice-versa, i.e. if the contexts in turn have similar words [4]. In this vector space of embedded words, the cosine similarity measure is classically used to identify words occurring in similar contexts. In the work described here, we used this approach to select words from the user's pro le that occur in the same context as the query terms. These selected words represent a user sub-pro le according to the context of his query.
More precisely, we investigate here several ways to lter these terms and their impact on the quality of the results. In a way to determine the impact of such term ltering, we also study the use of non-personalized word embeddings for the query terms, i.e., without using the user's pro le.
The rest of the paper is organized as follows: Section 2 presents the proposed approach and its four steps. The experiments and the o cial results are presented in Section 3. We conclude this work in Section 4. 2 2.1
The task described here aims to suggest books according to a user's query and to his pro le. It consists of four steps: 1. Learning of the Word Embedding : this step describes the learning of a set of word embeddings using word2vec [7] and the features applied in the training process; 2. User Model: this step describes the way user pro le is modeled; 3. Pro le Query-Filtering: this step describes the building process for the user sub-pro le based on his query; 4. Ranking Model: this step depicts the ranking model using a Pro le Query
Filtering. 2.2
We present now the notations used in this paper: { u 2 U : a user u from the whole set of users U ; { q = ft1; t2; :::; tmg: the original query of a user u. { qf = ft1; t2; :::; tog: the ltered query of a user u. { P (u) = ft1 : w1; t2 : w2; :::; tn : wng: the pro le of a user u, composed of couples <term : weight>. { T erms(u) = ft1; t2; :::; tng: a set of user terms belonging to the user pro le
P (u). { W ordEmbeddingw2v(q): a set of word embeddings (i.e. output of the section 2.3). { PF iltered(q; u): a sub-pro le of the user u according to the user query q. { PW eighted filtered(q; u): a weighted sub-pro le of the user u according to the user query q. 2.3
Word Embedding is the generic name of a set of NLP-related learning techniques to map words to low-dimensional (w.r.t. vocabulary size) vectors of real numbers. Word Embedding [7], which falls in the same category as Latent Semantic Analysis (LSA) [2], enables us to deal with a richer representation of words, namely words are represented as vectors of more elementary components or features. The similarity between vectors is supposed to be a good indicator to the `closeness ' between the respective words [11]. In addition, the arithmetic operations between vectors re ect a type of semantic-composition, e.g. bass + guitar = bass guitar [11].
In this section, we describe the process of learning word-vectors. To this end, we train word2vec [7] on the Social Book Search corpus. Word2vec represents each word w of the training set as a vector of features, where this vector is supposed to capture the contexts in which w appears. Therefore, we chose the Social Book Search corpus to be the training set, where the training set is expected to be consistent with the data on which we want to test.
To construct the training set from the Social Book Search corpus, we simply concatenate the contents of all documents in one document, without any particular pre-processing such as stemming or stop-words removal, except some text normalization and cleaning operations such as lower-case normalization, removing HTML tags, etc. The obtained document is the input training set of word2vec. The size of the training set is 12:5GB, where it contains 2:3 billions words, and the vocabulary size is about 600K. The training parameters of word2vec are set as follows: { we used the continuous bag of words model instead of the skip-gram (word2vec options: cbow=1); { the output vectors size or the number of features of resulting word-vectors is set to 500 (word2vec options: size=500); { the width of the word-context window is set to 8 (word2vec options: window=8); { the number of negative samples is set to 25 (word2vec options: negative=25).
For example, based on the output of word2vec, the 10 most similar (using cosine similarity) words of \novel " in the Social Book Search corpus are: story, novella, thriller, tale, storyline, masterpiece, plot, debut, engrossing, and genre. 2.4
We model each user u by a set of terms that describe his interests. These terms are the tags that a user u used to describe or annotate the documents (books present in his catalogue). Each user u is represented by his pro le P (u) as a vector over the set of tags as follow:
P (u) = ft1 : w1; t2 : w2; :::; tn : wng
(
As stated before, Word Embedding can potentially be helpful in the selection of terms related to the query. Despite the e ectiveness of Word Embedding to select embedded terms, it could happen that they decrease the e ectiveness of the system. Usually, embedded terms are used as extensions of a term, as query expansion. In fact, if an extended term is a noisy term (because of its ambiguity for instance, or because it is not a topical term of the query), then the set of its resulting word embeddings will increase the noise in the extended query. For example, in the queries (taken from the topics of Social Book Search 2016) \Help! i Need more books", \Can you recommend a good thesaurus?", \New releases from authors that literally make you swoon..." or \Favorite Christmas Books to read to young children", the majority of words are not useful for expansion, like \new, good, recommend, make, you ...". Therefore, we need to lter the words of the queries before the expansion process. We chose to remove from the words to be expanded all the words that are adjectives. To do that, we use an English stop-adjective list in addition to the standard English stop-word. Then, this process generates, from initial query q, a ltered query qf = ft1; t2; :::; tog that is expected to contain more relevant topical elements as output. 2.5.2
Word Embedding Selection
From each of the terms in the ltered query qf obtained by the previous step 2.5.1, we then select their set of word embeddings. For each term as an input, we obtain the top-k most related terms as an output. The metric used is the cosine similarity between the query term and all words in the training corpus. From these top-k terms, we do not want to over-emphasize variations of the query term. So, we lter out from the top-k terms the ones that have the same stem as the initial query term. Such stems are generated using the well know Porter stemmer for English.
The expanded query generated as output of this step is:
W ordEmbeddingw2v(qf ) =
(
Where W ordEmbeddingw2v(qf ) denotes the function that returns a set of word embedding for a given ltered query qf , and em tij denotes the jth element of the top-k word embeddings of ti. 2.5.3
The goal of our proposal is to construct a sub-pro le using a set of word embeddings based on a user query. Our approach here is to apply a naive method based on the Intersection between a) the query word embeddings set and b) the user pro le, as follows:
PF iltered(qf ; u) = W ordEmbeddingw2v(qf ) \ T erms(u)
(
We propose two variations of such a user query sub-pro le: { Non-Weighted Sub-Pro le (PF iltered): this user sub-pro le is represented by only the terms without weights, then all terms in the query have a same weight (i.e. corresponds to PF iltered(qf ; u) de ned in Eq. 3). { Weighted Sub-Pro le (PW eighted filtered): for this user sub-pro le, the weight for each term in the pro le is the cosine between of the embeddings of the ltered term and its respective query term. 2.6
In this part, we present the ranking used to suggest books to the user according to his query and to his pro le. We take into account two elements: if the document is related to the user query, and if the document is related to the user pro le, as follows:
Score(qf ; u; d) =
Score(qf ; d) + (
Score(qf ; d) =
ScoreBM25(qf ; d) +
ScoreLM (qf ; d)
(
ScoreBM25(W ordEmbeddingw2v(qf ); d)
ScoreLM (W ordEmbeddingw2v(qf ); d)
Where ScoreBM25(qf ; d) (resp. ScoreLM (qf ; d)) denotes a matching using the classical BM25 model (resp. language model), and , , and are parameters that determine the importance of each part of the document overall score ( + + + = 1).
The function Score(u; d) in equation (
3.1
For all the runs described below, for the BM25 runs the parameters are the defaults of Terrier [10], except for one parameter b = 0:5, according to previous experiments on SBS 2015. For the Language Models with Dirichlet smoothing, we use the default Terrier parameter = 2500.
For the re-ranking post process, we used the same protocol used in [9]. The documents that belong to the user's catalogue are removed for the result. This process is applied for each run. 3.2
1. RUN1: This run is the non-personalized approach ( = 1 in the Eq. 4). The
Score(qf ; d) is computed using Eq. 5 with the following parameters: = 0:6,
= 0:2, = 0:2, = 0:0.
2. RUN2: This run is also a non-personalized approach ( = 1 in the Eq. 4),
very similar to the run RUN1. The Score(qf ; d) is computed using Eq. 5
with the following parameters: = 0:6, = 0:2, = 0, = 0:2.
3. RUN3: This run is a personalized approach where the user u is
represented by his ltered pro le PF iltered in the Score(u; d) function (Eq.4). The
Score(qf ; d) represents the score of d according to the RUN1 above. The
score Score(u; d) of equation (
The parameter is equal to 0.7.
4. RUN4: This run is a personalized approach where the user is represented by
his ltered pro le PF iltered in the Score(u; d) function (Eq.4). The Score(qf ; d)
represents the score of d according to the RUN1 above. The score Score(u; d)
of equation (
The parameter is equal to 0.7.
6. RUN6: This run is a personalized approach where the user is represented by
his weighted pro le PW eighted filtered in the Score(u; d) function (Eq.4). The
Score(qf ; d) represents the score of d according to the RUN1 above. The score
Score(u; d) of equation (
According to table 2, we see that all of our results are quite close to each other. Even the weighted pro les compared to unweighted ones does not change results: runs RU N 6 and RU N 3 on one side and runs RU N 4 and RU N 5 lead to exactly the same result lists, and therefore the same evaluation results. This may be explained by the fact that the top-10 terms are usually very close (according to the cosine similarity) to the query terms, leading to weights close to 1:0. This has to be studied more carefully in the future. We remark that our two best runs (RU N 2 and RU N 6) include word embeddings. Both of them use Word Embedding applied on Language Models. It seems so that the integration of word embeddings behaves a bit better on language models than on BM25. Our best run, RU N 2, is not personalized. This shows that using word embeddings in the context of personalization must be further studied. We depict here a strange behavior that we faced after the release of the o cial relevance judgments: the reranking (described in Section 3.1) that removes the documents that belong to the user catalogue does lower the evaluations results. Table 3 presents the N DCG@10 values with and without reranking, with the relative improvement. This is explained by the fact that we chose to remove all the ISBN documents that correspond to the documents from the user catalogue, and not only the documents that correspond to the LT id of the books. This approach increased the quality of the results in 2015, but is clearly not adequate for this year. This may come from a di erent removal, in the relevance judgments, of the documents in the user's catalogue. We presented in this paper a joint work between the universities of Grenoble and Saint-Etienne. Our focus was to study the integration of word embeddings for the Social Book Suggestion task. We nd that the nature of the queries pose a great challenge to an e ective use of word embeddings in this context. Weighting the expansion terms does not lead to better results. Our future works may help to understand this behavior. One last point is related to the removal of the documents that belong to the user's catalogue: removing all the ISBN ids and not only the LT ids from the results, assuming that is a user already bought one edition of a book he does not want another edition, is clearly not the right choice, as such removal decreased substantially our o cial results. 7. Mikolov, T., Chen, K., Corrado, G., Dean, J.: E cient estimation of word representations in vector space. CoRR abs/1301.3781 (2013) 8. Nalisnick, E., Mitra, B., Craswell, N., Caruana, R.: Improving document ranking with dual word embeddings. In: Conference Companion on World Wide Web. pp. 83{84. WWW'16 Companion, Montreal, Quebec, Canada (2016) 9. Ould-Amer, N., Mulhem, P., Gery, M.: LIG at CLEF 2015 SBS Lab. In: Conference and Labs of the Evaluation Forum. CLEF'15, Toulouse, France (2015) 10. Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., Lioma, C.: Terrier: A High Performance and Scalable Information Retrieval Platform. In: Workshop on Open Source Information Retrieval. OSIR'06, Seattle, Washington, USA (2006) 11. Widdows, D.: Geometry and Meaning. Center for the Study of Language and Information/SRI (2004)