This paper describes the work achieved by the MRIM research group of Grenoble, using some data from the LaHC of SaintEtienne, in a way to test personalized retrieval of books for the Social Book Search Lab of CLEF 2015. Our proposal rely on a biased fusion of content-only retrieval, using BM25F and LGD retrieval models, user non-social pro le based on the catalog of the requester, and social proles using user/user links generated from their catalogs and ratings on books. The o cial results obtained show a clear positive impact of user pro le, and a small positive impact of the social elements we used. Post o cial results that present non biased fusion scores are also presented.
This paper describes our participation to INEX Social Book Search Suggestion
Track challenge. The goal of this challenge is to evaluate approaches for
supporting users in searching collections of books based on book metadata and
associated user-generated content [
From the data provided by the SBS 2015 dataset, the following elements were used at one time or another: { the elds title, summary, content and tags from the documents: all concatenated for unstructured retrieval, and separated for eld-based retrieval using BM25F; { the elds title, mediated query and narrative from the topics; { the documents and ratings from the \topic users" (a topic user is the description of the user that asks a query): used to compute \friendship" relationships between users; { the documents and ratings from the pro les of the non-topics users: used to compute \friendship" relationships between users.
The IR processes were achieved on the Terrier system3 [
The section 2 focuses on the description of the fusion that was exploited: one original point relates to the biases that we propose. Section 3 tackles multiple content-only matching for the documents, as we found out that such integration is bene cial. Then, we introduce in section 4 the use of non-social pro les, and we detail how we de ned friendship relations between users, using their catalogs and ratings, as well a the way we used the pro les of such friends when processing queries. Additional processing must be achieved on SBS data to get results. We discuss in section 5 some of these elements before depicting in section 6 the o cial, as well as some non-o cial, results obtained, before concluding in section 7. 2
Fusing scores of several IR systems is nontrivial problem. In our case, as described
in the introduction, we propose to use biased linear fusions of scores, as an
extension of the \Zero-one" normalization used by Wu, Crestani and Bi in [
Let us focus on a fusion of two results lists L1 and L2, composed of couples (doc,rsv). To be realistic, L1 and L2 are limited to the top n results. Assume that a document d has a score value of score(d; Li) in Li, with i 2 f1; 2g, and d is at rank rank(d; Li) in Li; that bi is the bias of Li; and that hi is the horizon (a rank position) above which we do not look at the in results list Li. The normalized score of d in Li is then: f (d; Li) = ( (1 vvmmaaxx(L(Li)i) svcomrien(d(L;Li)i) ) + bi if rank(d; Li) hi 0 otherwise 3 Terrier: http://www.terrier.org
with vmin and vmax the minimal and maximal value of scores in a result list.
Compared to the tting used by [
Then, the overall fusion computes a weighted average of the normalized scores
(COMB-sum from [
We discuss the impact of such biases in the section dedicated to the experiments. 3
On experiments conducted over the SBS 2014 dataset, we noticed that fusing
several content-only runs had a positive impact with a relative nDCG@10
improvement larger than 10%. That is why we propose to fuse one result coming
from BM25F [
RSVLIG 1(Q; d) =
BM25F (ScoreBM25F (Q; d) + bBM25F ) + b 0.5 0.4
What we depict here as \non-social" corresponds to the individual user data. In
our case, these data refer to the catalog of the user. We assume that Catu denotes
the catalog (list of books) of a given user u (from the corpus U of users). To
construct a user pro le, we take inspiration from Cai and Li who consider each
user pro le as a vector of tags and use a L1 normalized term frequency (NTF)
to denote the preference degree of user on a tag [
Such pro le is used as an expansion of initial query. In a way to re ect the relative importance of a term in a pro le, we de ne a function Exp(Q; u) that expands the query by a xed number of terms, and the relative importance of each term in the pro le is re ected as corresponding number of occurrences of this term in the expanded query. For instance, if the value corresponding to the term t in a user pro le accounts for 40% of the occurrences of terms in the pro le, and suppose that we x the number of terms added in the query to 100, then the query will be expanded by 0:4 100 = 40 occurrences of the term t. Then a BM25 retrieval is achieved on the documents corpus.
We noticed in SBS 2014 data that such expansion does not provides good results, but that the fusion of the results of such expanded queries and the usual content-only queries lead to better results. That is why we experimented such fusion with the parameters de ned in table 2, where N SP rof denotes the parameters related to the non social pro le fusion. The overall score for LIG 2 is:
RSVLIG 2(Q; d; u) =
BM25F (ScoreBM25F (Q; d) + bBM25F ) + + NSP rof (ScoreBM25(Exp(Q; u); d) + bNSP rof ) (2) where BM25F , LGD; NSP rof are the relative importance of BM25 result list, LGD results list, the non social pro l list respectively. Also, bBM25F , bLGD; bNSP rof are the respective bias of each results list. Moreover, the normalized scores use a horizon h of 1000, as presented in the table 2. 4.2
For the social user pro le, we choose to generate \friendship" links between topic users and the non-topic users provided by SBS. To achieve that, we assume that what makes (topic or non-topic) users similar to others is their catalog and the ratings they provide. We represent then all the non-topics users as a text document corresponding to concatenation of the document ids from the user catalog. We include the ratings (integer values) by using the ratings as the tf values for the number of occurrences of the documents ids.
To be able to nd the non-topic users similar to topic users, we describe the users topics in the same way as the non-topic users as described above. Then we used the topic-users descriptions as queries on the corpus of non-topic users using a classical BM25 matching. For rst experimentation, we lter the relationships to the top 2 most similar non-topic users for each topic user, and we plan to experiment the top k similar users in future works. 4.3
Once 2 closer friends of a topic user are obtained, we apply a process similar to section 4 to generate the non-social pro les of the friends, and then we match the topic query with the friends pro les to get documents that match the query. The matching is computed as follows:
RSVLIG 3(Q; d; u) =
The fusion parameters used for the o cially submitted run LIG 3 are given in table 3, with F ri1 and F ri2 the two friends of u.
One important point to notice is that the post processing of the obtained results have a dramatic impact on the results. For instance, as the initial corpus ids (isbn) are not the ones on which the results are evaluated (LibraryThing ids), and because of potential duplicates generated, it is not obvious to handle the translation. Our approach for such duplicate removal was the same that is provided by the organizers of SBS.
Additionally, for our runs for SBS 2015, we focused on integrating the users
examples to post-process the queries. Our idea was that a user might be
interested if he nds as answers documents that he read and that he appreciated, as
this would be an indicator that the system is providing relevant documents to
him. We declined this hypothesis in two ways:
Reranking: Achieve a reranking where the documents a user likes are boosted,
and the documents he dislikes are removed for the result. After a \Zero-one"
score normalization [
We present here two elements. First, we list the o cial results obtained for our 6 runs o cially submitted to SBS 2015. Second, we discuss additional results generated after the release of the SBS 2015 qrels, presenting the impact of the \biases" we used (see section 2) over \unbiased" results. 6.1
O
cial Results We comment here mainly the lines of table 4 corresponding to boldfaced run ids that use nor reranking neither relevance feedback. We notice then that the impact of the user non-social pro le is clearly bene cial, however the p-value of a bilateral paired Student t-test on LIG 1 versus LIG 2 equal 5.45%, thus with a signi cance threshold of 5% this di erence is not statistically signi cant. Such value is even larger between LIG 1 and LIG 3. We notice a slight improvement of nDCG@10 results when integrating the 2 best \friends", however our generation or usage of relationships between users does not seem to be e ective enough. According to what we de ned for our fusion (see section 2), we also notice that the weights assigned to the friends are very small, 0.05. With higher relative values the results degrade. So we conclude for now that our proposal does not outperform the integration of non-social user information.
As we see on table 4, the results with reranking or relevance feedback lower the quality of the results, but these elements are related to our interpretation of the catalogs and examples that are incompatible with the interpretation of the SBS organizers. In fact, our interpretation was somewhat the exact contrary of what decided the SBS organizers (they choose that the catalog + examples must not be part of the result), this explains why these additional runs behave worse than our initial runs. We describe in table 5 the impact of using the biases as de ned in section 2. To be fair compared to the o cial results, we choose to only remove the bias from the con gurations used for the o cial runs and to compare the relative gain or loss (between parentheses) with respect to the biased respective runs from table 4. In this table, the values use 3 digit precision numbers, where the percentages are computed on 4 digit precision numbers. We notice that the e ect of the bias are positive for all the measures for the runs LIG 2 and LIG 3, and have almost no e ect of the content only LIG 1 run. So, the e ect of the biases seem to be more positive as we fuse many lists. 7
We presented in this paper the experiments that were conducted for the participation of LIG to the SBS 2015 lab evaluation. Our main nding, according to our integration of non-social and social pro les, is that the use of non-social pro le has a clear positive impact on the quality of the retrieval, where the integration of generated friendship relationships does not really increase the quality of the system provided. One important conclusion that we draw from the SBS experiments is that the post processing of results has a dramatic impact on the quality of the results, and then must be carefully studied.
The experiments reported here depict our rst steps to grasp the complexity of personalized information retrieval in social context, and many e orts will focus on re ning and characterizing the numerous elements involved in such retrieval process.
This work is supported by Region Rh^one-Alpes through the ReSPIr project.