In this paper, we describe our participation in the INEX 2015 Social Book Search(SBS) Track Suggestion Task. We try out all possible groups of XML elds to nd out the most e ective group for relevance feedback model. We investigate the contribution of user-generated data and construct a social book search system based on several important techniques. Focus on the lack of important information of the majority books, we use document expansion by crawling book information from other two web sites which can enrich the index. And then we perform re-ranking on Galago searching results on enriched XML index by 11 di erent strategies and combine the results with learning to rank. Experiments on these methods show that an enriched index and query model improves the e ectiveness. As our methods in INEX 2014 [1], reranking and Random Forests combining those re-ranking models show better performance.
In this paper, we describe our participation in the INEX 2015 Social Book Search
track suggestion task. Our goals for this task are (
The structure of this paper is as follows. We start in Section 2 by describing our methodology: pre-processing on the XML formatted documents, indexing and searching by Galago, re-ranking, combining with Learning-to-rank. In Section 3, we describe the results of our enriched index, query model and re-ranking models. Section 4 describes which runs we submitted to INEX, with the results of those runs presented in Section 5. We discuss our results and conclude in Section 6.
2.1
As we can referred to [2], there are several elds in the XML formatted documents shown meaningful numeric information which cannot be understood by searching engine, such as <tag count="3"> ction</tag> and <dewey>519</dewey>. According to the method from [2], we expand and enrich the XML formatted documents with replacing the numeric information with textual information.In this way, the XML element <tag count="3"> ction</tag> is replaced by the element <tag> ction ction ction</tag>. And the XML element <dewey>519</dewey> is replaced by the element <dewey>Probabilities & applied mathematics</dewey>.
Galago 1 is an open-source search engine. In order to improve the search effectiveness, we study two strategies to build the index. One indexing strategy is the normal indexing method describes as following. Experimentally, we nd that the elds (etc. the title, tag, content and summary) are more relevant and meaningful than others in the XML formatted documents. So we build our basic index by removing the rest useless elds content. Another strategy is to enrich the basic index. Observing the book information from the Library Thing, we nd out that there are a large proportion of books lack of the content and summary elds. Therefore, documents expansion technology is expected to utilized to enrich the basic index. Firstly, we select two web sites which contain a large amount of more useful metadata of books. The books we use are the literatures written in English in douban.com 2 and all books in lookupbyisbn.com. Then we crawl the brief introduction of douban.com and the book description eld of lookupbyisbn.com. Both web sites are available by ISBN. With the content from both web sites, we enrich six hundred thousand of books (see the examples of book document which is used for index in XML 1 and XML 2). The enriched index is based on the enriched information.
XML 1: Book document <book> <title>Mister Monday</title> <summary>So good, you can't put it down!</summary> <content>Now, I had...</content> <tag count="9">children's literature</tag> </book> 1 http://www.galagosearch.org/ 2 http://book.douban.com/
XML 2: Enriched book document <book> <title>Mister Monday</title> <summary>So good, you can't put it down!</summary> <content>Now, I had...</content> <tag count="9">children's literature</tag> <brief introduction>the content is from the douban.com</brief introduction> <description>the content is from the lookupbyisbn.com</description> </book> 2.3
(
degree = 8>4; hhasReadi = no and hsentimenti = postive: <
0; hhasReadi = yes and hsentimenti = negtive: >:1; else
Q0 =
Q + (
Where the Q is the original query vector, the title is the relevance information vector, and the degree is the relevance degree.
Another query model improvement is about the eld <narrative>. We lter out the words beginning with an upper letter which is exclude the rst word of a sentence and add them to the original galago query, such as <narrative> I love alternative histories - two great ones I've enjoyed are Robert Harris's Fatherland and Kim Stanley Robinson's Years of Rice and Salt. Any other recommendations? John <narrative>. According to our method, we transform this eld to <narrative> I've Robert Harris's Fatherland Kim Stanley Robinson's Years Rice Salt John <narrative>. 2.4
Those re-ranking methods are proposed and used by USTB at INEX2014 [1] and proposed by Toine Bogers in 2012 [3], which proved to be e ective. The re-ranking method is performed by 11 di erent models: Tag-Rerank (T ), ItemRerank (I), Deep-Rerank (D), Node-Rerank (N ), RatingBayes-Rerank (B), RatingReview-Rerank (R), Tag-Node-Rerank (T N ), Item-Tag-Rerank (IT ), Deep-TagRerank (DT ), Item-Tag-Node-Rerank (IT N ), Deep-Tag-Node-Rerank (DT N ). We use these 11 models to re-rank by the following stages:
1)Similarity Calculation. Models like T , N focus on the eld <tag> and
<BrowseNode>. We can build a feature matrix for features like T ,N . The feature
matrix of T N is the connection of two matrices. Equation (
Features like I, D focus on the eld <similar-product>, the similarities of
two documents based on the feature I is calculated by the Equation (
As we know similarity matrices SIM (I) and SIM (D) are sparse, so we use
the multi-feature like IT , DT , IT N , DT N to ll-in. For example, the similarity
based on feature IT is calculated by Equation (
simij (IT ) = (1;
simij (I) = 1:
simij (T ); else
2) Re-ranking. We re-rank the top 1000 list of initial ranking for the
abovementioned features by Equation (
simij (f ) = cos < !fi ; !fj >= ! ! fi fj ! ! j fi jjfj j simij (I) = 8>1; i is j's similar product or <
j is i's similar product >:0; else simij (D) = 81; simij (I) = 1 or > > > <
k 6= i; k 6= j; 9 > s:t: simik(I) = simjk(I) = 1: > >:0; else
) log(jreviews(i)j) where Ri is the set of all ratings given by users for the document i, and jreviews(i)j is the number of reviews.
In order to choose the most e ective strategies and select the optimized parameter , in the rst round, we train our query model on SBS 2011-13 and test on SBS2014. The results are shown in Table 1. 3 http://people.cs.umass.edu/~vdang/ranklib.html
Among all the methods, we select the best ve automatic runs to submission which are based on our query and Re-ranking Models. The rst one of these submitted runs is the result of <example> eld using as relevance feedback information. The second one is the expansion query search in the enriched index result. The third one is based on the second result with the relevance feedback method. The fourth one is applied all Re-ranking strategies and combining them by Random Forest method result. The fth one is combing the Re-ranking, Random Forest method and query model based on the eld <example>
Run 1 (example) This run takes Galago as toolkit and applies query model using the eld <example> as relevance feedback information to search.
Run 2 (Upper narrative-abstract) This run applies query model based on the eld <narrative> to search and uses the expansive index.
Run 3 (UpperNar-abs-ex) This run applies query model based on the eld <narrative> and <example> to search and uses the expansive index.
Run 4 (Rerank-RF) This run applied all Re-ranking strategies and combining them by Random Forest method.
Run 5 (Rerank-RF-example) This run applied all Re-ranking strategies and combining them by Random Forest method with query model based on the eld <example>. 5
The runs submitted to the INEX 2015 Social Book Search track are evaluated using graded relevance judgments. The relevance value were labeled manually according to the behaviors of topic creators, for example, if creator adds book to catalogue after it's suggested, the book is treated as highly relevant. A decision tree is built to help the labeling 4. All runs are evaluated using NDCG@10, MRR, MAP, R@1000 with NDCG@10 as the main metric. Table 2 shows the o cial evaluation results. 4 https://inex.mmci.uni-saarland.de/tracks/books/INEX14_SBS_results.jsp#mapping
It is obvious that among all the methods, the best-performing run on all 208 topics is run 5 with an NCDG@10 of 0.106. Run 5 uses relevance feedback based on the eld <example> to expand the query and all re-ranking models with combining them by Random Forest. Again we see that re-ranking models and relevance-feedback do improve over the initial results by searching engine. Run 5, improves over the initial ranking by about 10%. 6
On both training and the testing set the best results are from combining all re-ranking results in Random Forest and the relevance-feedback method. This shows a good use of social information can improve the results of Social Book Search. We fail to make use of the pro le of topic creators to improve the results. It is worth discussing whether the information is useful or not.