The Social Book Search (SBS) lab at CLEF 2016 provides a complex test collection that gives the opportunity to experiment with retrieval methods that combine various modalities in order to achieve the best possible ranked list. We show how the idea of being "characteristic", which is used as the core concept in most of the weighting schemes used for textual modalities, can be applied to non-textual modalities. Our approach re-de nes BM25 for the three non-textual modalities found in the SBS collection: ratings, price and number of pages. A fuzzy query is constructed from the user preferences inferred from the user's catalog. The results are used to re-rank a textual baseline, which signi cantly improves the retrieval e ectiveness.
The suggestion track of the INEX Social Book Search (SBS) at CLEF 2016 allows researchers to evaluate their methods on a multimodal collection with queries constructed from real LibraryThing user requests. For the books in the collection not only the book meta-information from Amazon (description, binding, number of pages, price etc.) is available but also user generated information such as book ratings. Also, the personal catalogs of the users are given and can be used to infer user preferences.
In our SBS 2015 participation [2], we found that the user preferences can be used to improve the retrieval e ectiveness, by incorporating the books read by the users in a random forest based learning to rank approach. In this participation, we focus on taking the user preferences into account using a di erent approach. BM25 is a well known weighting scheme that has widely been used in text retrieval. It was originally developed for the English language but it has proven to be useful for other languages as well as for image retrieval [3]. We show how BM25 can be applied to the modalities ratings, price and number of pages. The BM25 scores of these non-textual modalities are then used to re-rank the textual baseline to signi cantly improve it.
Textual Models Similar to our participation in 2015, we employ a textual baseline [2] as a basis for our methods. For the textual score, we merge all textual elds of the document into a single textual index eld and construct queries from the two topic elds title and request that are analogously merged into a single textual representation. Further, we use the example books mentioned by the topic creators to expand the queries with the 35 most characteristic terms. Hereby, the most characteristic terms of the example books are computed by BM25.
Additionally, we lter the books already read by the topic creator from the nal ranked list, since this is a hard criterion in the relevance assessments [1]. Hereby, we determine the read books from the catalog of the topic creator. 2.2
BM25 Model for Non-Textual Modalities BM25 can be described in terms of how it combines three components; the feature frequency ( ), the document frequency (df ) and the document length normalization component [5]. Although, it was originally developed for retrieval on English language text, it has generalized well to many related tasks, such as multilingual retrieval, multimedia retrieval and others. The and the df make sure that "characteristic" terms are weighed heavily. Hereby, a characteristic term is one that appears frequently in the document in consideration ( ) and rarely in the remainder of the collection (df ). This concept of "being characteristic" is quite general and therefore applicable to other (non)-textual modalities [4]; i.e. bag of visual words in image retrieval, locations in geographical IR or timestamps in time-aware IR.
The retrieval status value (RSV) of document dj w.r.t. query q when using BM25 is de ned as
RSVBM25(q; dj) :=
X 'k2 (q)\ (dj) w('k; dj) :=
('k; dj) k1((1
b) + b lj ) + ('k; dj) w('k; q) := ('k; q) log 0:5 + N df('k)
0:5 + df('k)
w('k; dj) w('k; q);
(
For the suggestion track of the SBS lab at CLEF 2016, we adapt BM25 for three non-textual modalities the ratings, the price and the number of pages and use it to re-rank the textual baseline. (dj) set of features representing document dj (q) set of features representing query q w('k; dj) weight of feature 'k for document dj w('k; q) weight of feature 'k for query q
('k; dj) frequency of feature 'k for document dj df('k) document frequency of feature 'k
average document length in number of tokens
Ratings For the ratings, we do not have a per-user query information, but we
assume, that in general users will prefer books with higher ratings. Therefore,
we de ne the query in the following way
(
Price For the price, we use the average price of the books that the user has already read p(q) as the query information. Since an exact match of the price is not meaningful, we use a fuzzy search with p(q) as the search parameter. We assume, that a user would also like books that are at most 20% cheaper and at most 30% more expensive than the average price of the books in his library. Although, we assume that generally a cheaper book is always acceptable, we still set a lower bound, because we assume that people tend to like similar kinds of books, that are usually in the same price range. The query's set of features and feature frequencies are de ned as ('k; q) := (q) := ]0:8
p(q); 1:3 ( 1:3
0:3 'k 1:2 0:2 p(q) 'k p(q)
p(q) p(q)
p(q)[ if 'k if 'k < p(q) p(q):
For the de nition of the df, we bin the prices into bins with a quadratically increasing width as shown in Figure 1. The bin index for the price p is de ned as bin(p) = pp : This is based on the assumption, that with increasing prices, the tolerance for two book prices to be comparable is larger. The df is then de ned as the number of documents with a price in a given bin. Since a book only has a single value for the price, the and the document length are always 1.
Binning of Prices 12 10 8 in 6 B 4 2 00
p(q) 50 100 150 200 250 300 350 400 450 500
Price Number of Pages For the number of pages of the books, we de ned the , df and the document length as well as the query, analogous to the price. 3
Our goal in the experiments is to use the information present in the non-textual modalities to re ne the result lists so that they re ect the users preferences. 3.1
Experimental Setup For the textual baseline we used Lucene3 for indexing and searching. For all modalities we used BM25 with a document length normalization parameter b = 0:75 and a saturation parameter k1 = 1:2. For the textual modalities, we used the built-in EnglishAnalyzer, which removes a small set of stopwords and stems terms using the Porter stemming algorithm. For the re-ranking, we used a linear 3 https://lucene.apache.org/core/ combination of the scores from the di erent modalities. Hereby, the weights of the linear combination sum up to one.
RSVBM25 =
In order to validate the e ectiveness of our approaches and to nd the optimal re-ranking parameters, we used the topics and relevance assessments from SBS 2015.
For our participation to INEX SBS 2016 track, we built six runs by applying di erent con gurations (the re-ranking parameters equal to zero are omitted): { Run1: Textual baseline using BM25 with example based relevance feedback using 35 terms and read book ltering with re-ranking parameters: = 1. { Run2: Textual baseline re-ranked with a query-independent BM25 model for ratings with re-ranking parameters: = 0:7818; = 0:2182. { Run3: Textual baseline re-ranked with a user catalog based BM25 model for the number of pages with re-ranking parameters: = 0:3118; = 0:6882. { Run4: Textual baseline re-ranked with a user catalog based BM25 model for the price with re-ranking parameters: = 0:2332; = 0:7668. { Run5: Textual baseline re-ranked with a user catalog based BM25 model for the price and the number of pages with re-ranking parameters: = 0:2225; = 0:3033; = 0:4742. { Run6: Textual baseline re-ranked with a query-independent BM25 model for ratings and a user catalog based BM25 model for the price and the number of pages with re-ranking parameters: = 0:265; = 0:045; = 0:225; = 0:465.
In the next section we discuss the evaluation results of our o cial submission. 3.2
Results and Discussion Table 2 summarizes our o cial results of SBS 2016 evaluated using nDCG@10 (Normalized Discounted Cumulative Gain), MRR (Mean Reciprocal Rank), MAP (Mean Average Precision) and R@1000 (Recall), with nDCG@10 being the o cial evaluation measure.
The submitted runs using all non-textual modalities to re-rank the textual baseline (Run6), the run using the price and the number of pages (Run5) as well as the run using the number of pages (Run3) signi cantly improve the nDCG@10 over the textual baseline (Run1). The signi cance is computed using a paired randomization test [7] with signi cance level = 5%. Using just the number of pages (Run3) leads to the highest nDCG@10 amongst our submitted runs. Using just the ratings for the re-ranking (Run2) increases the nDCG@10 over the textual baseline, although not signi cantly. Our re-ranking with the scores calculated based on the price (Run4) does not help to nd a better ranked list.
To further analyze the results, we also evaluated the performance of the nontextual modalities on their own. Therefore, we used the documents retrieved with the textual baseline and ranked them only based on the score of each nontextual modality. This will not lead to a fully textual baseline independent ranked list (e.g. the recall will not change), however it gives an indication how well they would perform on their own. Using this approach the nDCG@10 for the ratings is 0.0206, for the price it is 0.0258 and for the number of pages 0.0135. Surprisingly, we see that the price on its own results in the highest nDCG@10, although this is not re ected in the runs that combine the non-textual modalities with the textual baseline. We also trained the weights for modalities with the relevance assessments for the 2016 task, and found, that with the optimal weights, the textual baseline can also be improved by taking the price into account. Hence, the weights chosen based on the 2015 task, are not optimal. Nevertheless, the nDCG@10 for the runs using the number of pages (0.0706) and the ratings (0.0647) using optimal weights is still higher than for the run with the price (0.0596). This shows, that either there is a higher information overlap between the price and the textual modality than between the other modalities and the text, or the linear combination merging is not as e ective for the price as for the others. 4
In this paper, we described our participation to the suggestion track of the INEX SBS 2016 lab. We investigated how the weighting scheme BM25 can be applied to non-textual, continuous modalities. Therefore, we proposed a method to discretize the continuous modalities in order to de ne a document frequency and a fuzzy query that takes into account that the query does not require an exact match. By using our approach on the ratings, prices and number of pages, we showed that the e ectiveness of the system can be signi cantly increased over the textual baseline using a simple linear score combination. However, the performance of our random forest based learning to rank approach from 2015, can not be reached.
Our experiments, have shown that the merging the scores of the prices with the textual scores leads to a smaller improvements as could be expected based 4 We have underlined any statistically signi cant di erences in performance according to nDCG@10 to the textual baseline (Run1) resulting from a paired randomization test [7] (signi cance level = 5%). on the performance of the non-textual modalities individually. So far, we did not yet investigate the merging in more depth. It is possible that a di erent merging method could improve the merging with the price. For example, we could use a non-linear combination of the scores, or a per-query normalization strategy, like the z-score [6], to avoid that the per-query optimal weights are far apart.
Further, we would like to investigate if the function used for the fuzzy search is the best possible. We could for example use di erent parameters or a non-linear fallo of the weighting.
So far, we approximated the user preferences by the average price and number of pages of the books read by the user. However, it could also be possible to construct the query such that each price and number of pages is part of the query and therefore the loss of information due to the averaging is avoided.