Query Type Recognition and Result Filtering in INEX 2014 Social Book Search Track

Abstract

The paper reports our system in INEX 2014 Social Book Search (SBS) track. This is the second time that we attend the SBS track. Based on our social feature re-ranking system [1], we improve our system by involving some knowledge on understanding the queries. Our baseline system is built on Lucene [6], an open source information retrieval system. The new modification is a set of rules that can filter out unnecessary books from the recommendation list. The official run results show that the system performance is much improved than the 2013 system.

Keywords

• Query type recognition
• social features
• social book search

-

The paper reports how we build a system to attend the INEX 2014 Social Book Search (SBS) track [ 10 ]. This is the second time that we attend the SBS track [ 7 ]. Based on our social feature re-ranking system [ 1 ], we improve our system by involving some knowledge on understanding the queries.

In the book search application, we believe that the result of traditional information retrieval technology is not enough for the users who need more personal recommendation. Recommendation from experienced users are more appealing; it might contain more personal feelings and cover more subtle reasons that traditional information retrieval system cannot cover. Our system integrates the social feature into the traditional information retrieval technology to give better recommendation on books. In this task, user-generated metadata is used as the social feature.

According to our observation on the topics in INEX 2012 SBS Track, we find that there are some queries that are different from others. Simply treat the keywords in the topic as search terms will not get good result. Some of them require higher level of knowledge to deal with. System needs to understand the information need behind the keyword, i.e. the knowledge on the types of literature. We analysis the topics and find several types in them. Due to the time limitation, we only implement a module to recognize one special type of topics and a filtering module to modify the recommendation result.

The structure of this paper is as follows. Section 2 is the data set description, section 3 shows our architecture and the details of our method, section 4 is the experiment results, and final section gives conclusions. 2 2.1

Dataset

Collection

The document collection in this task is provided by the INEX 2014 social book search track. The documents are in XML format, about 2.8 million books, and the size is 25.9GB. These documents are collected from Amazon.com and LibraryThing. [ 2 ] book dimensions reviews editorialreviews images creators blurbers dedications epigraphs firstwords lastwords quotations series awards browseNodes characters places subjects 2.2

Test Topic

Topics provided by INEX 2014 Social Book Search track are collected from LibraryThing. A topic describes the information needed for a user. Figure 1 and Figure 2 give partial view of an example, the XML tags used are：<topic id>, <title>, <mediated_query>, <group>, <narrative>, <catalog>, <book>, <LT_id>, <entry_date>, and <rating>.

CYUT System Methodology

System Architecture

Document

Collection

Indexing

Results

Stop words filtering

Contentbased

Retrieval

Stemming

Re-Ranking

The index and search engine in use is the Lucene system [ 6 ], which is an open source full text search engine provided by Apache software foundation. Lucene is written in JAVA and can be called easily by JAVA program to build various applications.

According to Bogers and Larsen [ 3 ], 19 tags are more useful in the social book search, they are <isbn>, <title>, <publisher>, <editorial>, <creator>, <series>, <award>, <character>, <place>, <blurber>, <epigraph>, <firstwords>, <lastwords>, <quotation>, <dewey>, <subject>, <browseNode>, <review>, and <tag>. Our system also focused on the 19 tags.

The content in the <dewey> tag is restored to strings accordint to the 2003 list of Dewey category descriptions [ 9 ] to make string matching easier. For example: <dewey>004</dewey> will be restored to <dewey>Data processing Computer science</dewey>. The content of <tag> is also expanded according to the count number to emphasize its importance. For example: <tag count="3">fantasy</tag> will be expanded as <tag>fantasy fantasy fantasy</tag>. In additional to the 19 tags, our system also indexes the content of <review> as independent indexes files and names it as reviews.

According to Koolen et al. [ 4 ], an Indri [ 5 ] based system using all the contents of <Title>, <Query>, <Group>, and <Narrative> as query terms will give better result. According to our observation on the topics in INEX 2012 SBS Track, we find that there are some queries that are different from others, we call them the Type2 queries. Type2 queries are the queries that contain the names of some books that the original users want to find similar ones. Therefore, the books in the topics should not be part of the recommendation. Since the book names are given explicitly, our system originally will find exactly the same books as the top recommendation. To filter out these ones, we define a list of phrases to identify such queries and filter out the books in the queries from the recommendation lists. The phrases are listed in the appendix in the rear of the paper. Figure 4 gives an example of Type2 queries taken from INEX 2013 SBS topics, in which contains a key phrase “I’m reading”. We find that there are 174 queries in the INEX 2013 SBS track that can be classified as Type2 queries. Therefore, this year, we add a module in our system to identify the Type2 queries and filtering out the books mentioned in the topics. The modified system flow is shown in Figure 5. 3.4

Re-ranking

The Re-ranking part is similar to that in our previous work [ 1 ]. We integrate the usergenerated metadata into the traditional content-based search result by re-ranking the results. The social features are used to give more weight on certain books, for example  User rating: users might evaluate a book from 1 to 5, the higher the better.  Helpful vote: other users might endorse one comment by voting it as helpful.  Total vote: the total number of helpful or not. Type2

Yes No

Search Engine

Search Engine

Filtering

We designed 3 different ways to use these social features in re-ranking. 1) User Rating method

Increase the weight of content-based retrieval result by adding the summation of user rating. As shown in formula (1): (1)

2) Average User Rating method Increase the weight of content-based retrieval result by adding the average of us

Re-Ranking

Result er rating. As shown in formula (2):

3) Weights User Rating method Increase the weight of content-based retrieval result by adding the book which gets more helpful votes. As shown in formula (3) and (4): (2)

(3) (4) 3.5

Find the Best α Value by Experiment

Since there is no theoretical reference on how to set the value, in our official runs, the value is selected via a series experiments that we conduct on the 2013 dataset. Table 2 shows the results, we find that the system gets the best result when is 0.95. In the official evaluation, we sent four runs. This year, we use four fields in the topics as query terms, and we filter out some book candidates for all the type2 queries. The configuration of each run is as follows. Run 1, the CYUT - Type2QTGN: without reranking. Run 2, the CYUT - 0.95AverageType2QTGN, re-ranking with Average User Rating. Run 3, the CYUT - 0.95RatingType2QTGN, re-ranking with User Rating. Run 4, CYUT - 0.95WRType2QTGN, Re-ranking with Weights User Rating.

Table 3 shows the official evaluation results of our four runs. Among them the CYUT - Type2QTGN run gives the best NDCG@10 [ 8 ] result, while the re-ranking run CYUT - 0.95AverageType2QTGN gives similar result. The other two runs give poor results due to technical errors; the system searches the document in 2013 index file. The last two runs should be better result if the system searches the document in 2014 index file. Comparing to the 2013 INEX SBS results in Table 4, our system performance improved significantly. This paper reports our system and result in INEX 2014 Social Book Search track. We sent four runs and the results are list in Table 3. In the four runs, the CYUT Type2QTGN run gives best nDCG@10, which is searching with content-based search and applying a set of filtering rules based on a list of key phrase. In the future, we will implement more modules with literature knowledge on the writers, genre of books, geometric categories of the publishers, and temporal categories of the authors that can deal with the special cases in the topics.

Acknowledgement References

This study was conducted under the "Online and Offline Integrated Smart Commerce Platform (1/4)" of the Institute for Information Industry, which is subsidized by the Ministry of Economic Affairs of the Republic of China. Appendix: The key phrases for recognizing Type2 queries. <TotalKeyWord> <keyWord>I've just finished</keyWord> <keyWord>I'm now reading</keyWord> <keyWord>I'm reading</keyWord> <keyWord>I've read</keyWord> <keyWord>I read</keyWord> <keyWord>I've ever read</keyWord> <keyWord>Any book as good as</keyWord> <keyWord>I'm not interested</keyWord> <keyWord>I already own</keyWord> <keyWord>I own</keyWord> <keyWord>picked up</keyWord> <keyWord>I can find</keyWord> <keyWord>I read</keyWord> <keyWord>I've looked through</keyWord> <keyWord>I've just found</keyWord> <keyWord>I have already read</keyWord> <keyWord>I was reading</keyWord> <keyWord>I had read</keyWord> <keyWord>to read</keyWord> <keyWord>what other</keyWord> <keyWord>I'm already completely</keyWord> <keyWord>I have already read</keyWord> <keyWord>I've started on</keyWord> <keyWord>I just finished</keyWord> <keyWord>I did enjoy</keyWord> <keyWord>something like</keyWord> <keyWord>without</keyWord> <keyWord>I am reading</keyWord> <keyWord>starting with</keyWord> <keyWord>I already have</keyWord> <keyWord>I'm thinking of</keyWord> <keyWord>I just finished reading</keyWord> <keyWord>similar</keyWord> <keyWord>I adore</keyWord> <keyWord>I tried reading</keyWord> <keyWord>I also have</keyWord> <keyWord>I've seen</keyWord> <keyWord>I recently read</keyWord> <keyWord>I discovered</keyWord> <keyWord>I have recently read</keyWord> <keyWord>have been suggested</keyWord> <keyWord>has been suggested</keyWord> <keyWord>I've enjoyed</keyWord> <keyWord>I've just completed</keyWord> <keyWord>I haven't yet read</keyWord> <keyWord>I have only found</keyWord> <keyWord>I have found</keyWord> <keyWord>I have read</keyWord> <keyWord>I am re-reading</keyWord> <keyWord>I also recently started</keyWord> <keyWord>I recently started</keyWord> <keyWord>I just re-read</keyWord> <keyWord>I've compiled</keyWord> <keyWord>I'd really like to read</keyWord> <keyWord>I've already enjoyed</keyWord> <keyWord>I can think of</keyWord> <keyWord>I was considering</keyWord> <keyWord>Currently reading</keyWord> <keyWord>Apart from</keyWord> <keyWord>I'm nearly finished</keyWord> <keyWord>have been recommended</keyWord> <keyWord>other recommendations</keyWord> <keyWord>having read</keyWord> <keyWord>on my list</keyWord> <keyWord>I've been reading</keyWord> <keyWord>I have just received</keyWord> <keyWord>finishing</keyWord> <keyWord>also read</keyWord> <keyWord>recent readings</keyWord> <keyWord>I have been reading</keyWord> <keyWord>I've recently finished</keyWord> <keyWord>other books</keyWord> <keyWord>additional resources</keyWord> <keyWord>The most recent book I haved</keyWord> <keyWord>I saw a book</keyWord> <keyWord>Thus far I</keyWord> <keyWord>what else should I</keyWord> <keyWord>Can anyone think of any more</keyWord> <keyWord>books like</keyWord> <keyWord>something elsee</keyWord> <keyWord>I thoroughly enjoyed</keyWord> <keyWord>My reading suggestions</keyWord> </TotalKeyWord>

References

1. Wei-Lun Xiao , Shih-Hung Wu , Liang-Pu Chen , Hung-Sheng Chiu , and Ren-Dar Yang , “ Social Feature R-eranking in INEX 2013 Social Book Search Track” , CLEF 2013 aE-valu tion Labs and Workshop Online Working Notes , 23 - 26 September, Valencia, Spain.

2. Marijn Koolen , Gabriella Kazai, Jaap Kamps, Michael Preminger, Antoine Doucet, and Monica Landoni, “ Overview of the INEX 2012 Social Book Search Track” , INEX'12 Workshop Pre-proceedings,P.77-P. 96 , 2012 .

3. Toine Bogers and Birger Larsen, “ RSLItS INaEX 2012 : Social Book Search Track” , INEX'12 Workshop Pre-proceedings,P.97-P. 108 , 2012 .

4. Marijn Koolen , Hugo Huurdeman and Jaap Kamps, “ Comparing Topic Representations for Social Book Search”, CLEF 2013 Evaluation Labs and Workshop Online Working Notes, 23 - 26 September, Valencia - Spain.

5. T. Strohman , D. Metzler , H. Turtle , and W. B. Croft , “ Indri: a la-mngoudaeglebased search engine for complex queries” , In Proceedings of the International Conference on Intelligent Analysis , 2005 .

6. Lucene, https://lucene.apache.org

7. Marijn Koolen , Gabriella Kazai, Michael Preminger , and Antoine Doucet, “ Overview of the INEX 2013 Social Book Search Track” , CLEF 2013 Evaluation Labs and Workshop Online Working Notes , 23 - 26 September, Valencia - Spain.

8. Järvelin , K. , Kekäläinen , “J.: Cumulated Ga-inbased Evaluation of IR Techniques” , ACM Transactions on Information Systems 20 ( 4 ) ( 2002 ) 422 - 446 .

9. 2003 list of Dewey category descriptions , https://www.library.illininois.edu/ugl/about/dewey.html

10. INEX 2013 Social Book Search Track , https://inex.mmci.uni-saarland.de/tracks/books