This paper describes the work of the LaHC lab of SaintEtienne for the Social Book Search lab at CLEF 2015. Our goals were i) to study a eld-based retrieval model (BM25F), exploiting various topics and documents elds, in order to build a strong baseline for further experiments, ii) to compare it with a Log logistic (LGD) retrieval model, and iii) to exploit some documents related to each topic (i.e. the documents given as negative or positive examples for a topic). The o cial results show that LGD outperforms BM25F, and that our approaches exploiting documents related to the topic requesters are based on a di erent interpretation of this additional information than the interpretation of the Social Book Search organizers.
This paper describes the work of the LaHC lab of Saint-Etienne for the
Suggestion track of the Social Book Search lab at CLEF 2015. The goal is to investigate
techniques to support users in searching and navigating the full texts of digitised
books and complementary social media as well as providing a forum for the
exchange of research ideas and contributions [
Our participation to the Social Book Search lab at CLEF 2014 [
Especially, our work for the Social Book Search lab at CLEF 2015 focuses
{ study a eld-based retrieval model (BM25F [
Our experiments were conducted using the Terrier Information Retrieval
System3 [
The paper is organized as follows: Section 2 presents brie y the Information Retrieval models used. Then, Section 3 details our approaches aiming at exploiting the positive or negative documents related to each topic. Finally, Section 4 presents the o cial results obtained, before concluding in section 5. 2
BM25F vs LGD (runs UJM 1 and UJM 2)
The Social Book Search 2015 dataset contains many various data describing or
related to the documents, the topics and the users. Among all these
information, we have used the Terrier Information Retrieval System implementation of
the eld-based models BM25F [
BM25F was used with the parameters values presented in Table 1, taken
from our participation to Social Book Search lab 2014 [
LGD is the Terrier implementation of the Log logistic model [
Documents given as \examples" (runs UJM 3, UJM 4, UJM 5, UJM 6) Our last goal was to exploit some non-social data (but nevertheless related to the users): the list of documents given as negative or positive examples for each topic. Our idea was that a user might be interested (respectively unsatis ed) if he nds as answers documents that he has read and that he appreciated (respectively disliked) and thus de ned as positive (respectively negative) example for the topic.
We implemented this hypothesis in two ways:
Re-ranking (RR): Achieve a re-ranking where the document a user likes are
boosted, and the documents he dislikes are removed for the result. After the
score normalization between 0 and 1, we add 1 to the normalized score of
document that the user likes and to set the score to 0 for the document that
he dislikes. This process is then a post-processing of an existing run. It is
worth noting that, if several retrieved documents are liked by the user, their
relative initial ranking is preserved; It has been applied on our BM25F run
UJM 1 (generating our run UJM 4) and also on our Log logistic run LGD
run UJM 2 (generating our run UJM 6);
Relevance Feedback (RF): De ne a relevance feedback, positive for the
documents that the topic user likes, and negative for the documents he does
not like. We achieved such relevance feedback on our BM25F UJM 1 run
(generating our run UJM 5), and also on our Log logistic LGD run UJM 2
(generating our run UJM 3). The relevance feedback uses all the positive
documents and selects the top 10 terms according to the default selection of
Terrier [
The Table 2 presents the o cial results obtained by our 6 runs. Log logistic (LGD) outperforms BM25F, regarding the o cial nDCG@10 measure as well as regarding the 3 other measures, despite the fact that BM25F is designed to take into account and to weight the di erent elds describing the documents.
Our approaches \Re-ranking (RR)" and \Relevance Feedback (RF)", both exploiting the list of documents given as negative or positive examples for each topic, lower the quality of the results. These approaches are based on a di erent interpretation of this list of documents than the interpretation of the Social Book Search organizers. Actually, these \examples" documents (the negative ones as well as the positive ones) are not considered as relevant by the organizers. Thus, re-ranking positively the positive examples (or using them as relevant documents in a relevance feedback process) can only lower the results. On the other hand, removing the negative examples from our runs (or using them as irrelevant documents in a relevance feedback process) may sometimes improve the results. All in all, the quality of our results is lowered.
Run
UJM 2 (LGD) UJM 6 (LGD + RR)
UJM 1 (BM25F)
UJM 3 (LGD + RF) UJM 4 (BM25F + RR) UJM 5 (BM25F + RF) nDCG@10 0.088 0.084 0.081 0.079 0.079 0.074
This paper describes the work of the LaHC lab of Saint-Etienne for the Social Book Search lab at CLEF 2015. Our quite basic experiments show that Log logistic (LGD) outperforms BM25F. Four of our six runs were based on a misinterpretation of the documents given as negative or positive examples for a topic.
Our 2015 participation allows us to build a strong basis in order to experiment in the future some more advanced some Personalized Information Retrieval approaches.
This work is supported by Region Rh^one-Alpes through the ReSPIr project.