In this paper we describe our participation in INEX 2012 in the Social Book Search Track and the Linked Data Track. For the Social Book Search Track we compare the impact of query- and userindependent popularity measures and recommendations based on user pro les. Book suggestions are more than just topical relevance judgements and may include personal factors such as interestingness, fun and familiarity and book-related aspects such as quality and popularity. Our aim is to understand to what extent book suggestions are related to userdependent and -independent aspects of relevance. Our ndings are that evidence that is both query- and user-independent is not e ective for improving a standard retrieval model using blind feedback. User-dependent evidence, on the contrary, is highly e ective, leading to signi cant improvements. For the Linked Data Track we compare di erent methods of weighted result aggregation using the DBpedia ontology relations as facets and values. Facets and values are aggregated using either document counts or retrieval scores. The reason to use retrieval scores for facet ranking is that we want the top retrieved results to be summarised by the top ranked facets and values. In addition, we look at the impact of taking overlap in aggregation into account. Facet values that give access to many of the same documents have high overlap. Selecting facet values that have low overlap may avoid frustrating the user.
In this paper we describe our participation in the INEX 2012 Social Book Search Track and the Link Data Track. For the Social Book Search Track we compare the impact of query- and user-independent popularity measures against recommendations based on user pro les. The web and social media have changed the way people search for books. The availability of user-reviews, ratings and tags allows users to nd out more about a book than from the traditional descriptions made by professional cataloguers. This in turn may evoke more complex information needs from users, relating to issues such as how interesting, familiar or funny, educational, engaging, well-written or popular a book is. Some of these issues are user-independent, such as the popularity of a book and to some extent its quality|in the sense of the general opinion of a whole group readers|and can be derived from data such as the number of people who reviewed, rated or tagged a book. Others are more personal, such interestingness and familiarity, and would require individual user information from user pro les or browsing and purchase history. We combine the user-dependent and -independent evidence with query-dependent evidence from a retrieval system to nd out whether book suggestion can bene t from user-dependent evidence.
For the Linked Data Track (LDT), we experiment with di erent ways of aggregating results. A standard approach is to rank facets and values using document counts. The facets and values that summarise the most retrieval results are considered the best summarisations. We compare this approach with aggregation based on retrieval scores, which prefers facet values that summarise the highest ranked documents. Assuming most of the relevant documents will be in the top ranks, result aggregation based on retrieval scores will be focused on the most relevant documents. The document collection of the LDT is rich in structure and o ers multiple ways of summarising search results. We use the DBpedia ontology relations as facets and values for summarisation.
We describe our experiments and results for the Social Book Search Track in Section 2 and for the Linked Data Track in Section 3. In Section 4, we discuss our ndings and draw conclusions. 2
The e ectiveness of user-generated content on social book search may be partly
due to its relation to popularity [
We want to compare the e ectiveness of popularity priors against recommendations based on user pro les. The goal of our experiments is to investigate whether the impact of user-dependent evidence outweighs the available evidence for popularity, which is both query- and user-independent. 2.1
From the book descriptions in the A/LT collection we can derive several indicators of popularity and quality.
We look at the following popularity priors:
{ Length: document length. Although document length is not directly related
to popularity, we assume that descriptions with many tags and reviews are
longer than descriptions with no or few tags and reviews.
{ Dirichlet: without smoothing, language models favour short documents.
Dirichlet smoothing introduces an implicit document length bias [
Next, we de ne two quality priors: { AvgRating: average rating. The arithmetic mean over all Amazon ratings for a work. { BARating: The Bayesian average rating. The Bayesian Average (BA) takes into account how many users have rated a work. As more users rates the same work, the average becomes more reliable and less sensitive to outliers. We make the BA dependent on the query, such that the BA of a book is based on books related to the query. The BA of a book b is computed as: BA(b) = n^ m^ + X r
r2R(b) n + n^ (1) where R(b) is the set of ratings for b m^ is the average unweighted rating over all books in the top 1000 results and n^ is the average number of ratings over all the books in the top 1000.
We crawled a random set of 10,000 books from LibraryThing to obtain popularity information. Each page dedicated to a book contains information on how many members have catalogued it, how popular it is (directly determined by ranking all books by the number of members who catalogued it), how many members have reviewed it and in how many forum discussions it is mentioned (derived from Touchstone mappings).
We use this set to compare the total number of tags and the frequency of the most frequent tag against the number of members who catalogued it. The correlation between these numbers indicates how well our tag-based priors re ect popularity. 2.2
We want to compare the popularity based measures against methods that take the interests and preferences of the topic creator into account. Speci cally, we want to look at collaborative ltering (CF) techniques to exploit the rich data available in the large network of users on LibraryThing. To build a recommender system based on CF, we had to obtain user pro les and personal catalogues of LibraryThing members. We started with a seed list of all the 1,104 users from the topic threads of the 211 topics of the 2011 SB task and crawled their personal catalogues and pro les. Links to other pro les (friends, members with interesting libraries) were extracted to continue the crawl. Because the members who participate in the forums may be di erent from other members, we also performed crawls based on random sets of 211, 1000 and 10,000 books. In each case, we extracted from each book page on LT the user names who have catalogued that book to generate another seed list. In total, we obtained 89,693 pro les (6% of all pro les) and 5,637,097 book ratings.
We experiment with neighbourhood-based and model-based recommendations and with rated transactions. Rated transactions indicate that a user catalogued a book and how she rated it. The k nearest neighbours ((k-NN) of a user u, denoted Ni(u), are computed using the Pearson correlation of their transaction vectors. The rating rui of an unseen item i for user u is estimated as: r^ui =
X v2Ni(u)
X v2Ni(u) wuvrvi jwuvj (2) where users v are the nearest neighbours who have rated i. For some books, none of the nearest neighbours gave a rating, and k-NN cannot make a rating prediction. In this case, the average of the ratings of all users in our crawl for this book is used. If there is only one user who has rated the book, no reliable average can be obtained and no prediction is made.
Model-based recommender systems learn a predictive model based on the
transactions of a user. The Singular Value Decomposition (SVD) method reduces
the domain complexity by reducing the number of dimensions in the item space
to a smaller set of underlying dimensions which represent the latent topics and
user preferences [
We used Indri [
The topics are taken from the LibraryThing discussion groups and contain a title eld which contains the title of a topic thread, a group eld which contains the discussion group name and a narrative eld which contains the rst message from the topic thread. In our experiments we only used the title elds of the topics as queries, which corresponds to the titles of the topic threads of the LT discussion forums. For the language model our baseline has default settings for Indri (Dirichlet smoothing with = 2500). We submitted two runs: xml social : a standard LM run on the social metadata index. xml social.fb.10.50 : a run on the social metadata index with pseudo relevance feedback using 50 terms from the top 10 results.
For the priors, each of the scores can be turned into a prior probability by dividing it by the sum of scores of all books in the collection. For instance, the document length prior probability is calculated as PLength(d) = jdj=jDj, where D is the set of all books in the collection and jDj = Pd2D jdj. The nal document score is then:
SLength(d) = P (djq) PLength(d) With some priors there are problems with zero scores. A book with no reviews would have a prior probability of zero, which would result in a score SNumReviews = 0. To solve this problem, we use the simple smoothing method known as Add-One, which adds one to the number of reviews of each book. The applies to the SumTag, MaxTag, AvgRating and BARating priors. In addition to linear prior probability, we experiment with log priors to compress the score range, thereby reducing the impact of the priors on the ranking. The log SumTag prior is calculated as:
1 + Log(1 + SumT ag(d))
PLog(SumT ag)(d) = Pd02D 1 + Log(1 + SumT ag(d0))
To rerank the retrieval results with user-dependent evidence from the Collaborative Filtering method, we use a linear combination:
SCF (d) = (1 )PRet(djq) + PCF (d) (3) (4) The relevance judgements for the SBS task are based on the book suggestions from the LT fourms, and are mapped to three di erent relevance values: irrelevant (rv=0) for suggestions made by the topic creator herself, relevant (rv=1) for suggestions by others that the topic creator did not catalogue afterwards, and highly relevant (rv=4) for suggestions that the topic creator catalogued after starting the topic. We refer to the latter as post-catalogued suggestions (PCSs).
We rst discuss the results of the o cial submissions (Table 1). Di erences between the two runs are tested for statistical signi cance using a one-tailed Bootstrap test with 100,000 resamples, at signi cance levels of 0.05 ( ), 0.01 ( ) and 0.001 ( ). The standard run on the xml social index scores 0.331 on MRR, which means the on average, the rst relevant document is found at rank 3. In the 2011 SB task, for which similar topics were used but all suggestions were considered equally relevant, a run on the same index scored 0.2913 on nDCG@10, but with this year's judgements it scores only 0.130. Either the topics this year are harder, or the impact of the di erence relevance values is big and the system fails to distinguish between the PCSs and the other suggestions. If we map the PCSs to relevance value rv = 1, the nDCG@10 score goes up from 0.130 to 0.171, and if we map all suggestions to rv = 1 (similar to operationalisation used for last year's task), it goes up to 0.224. This means that this year's topics are more di cult, but also that the distinction between PCSs and other suggestions has made the task more di cult.
The feedback run improves upon the standard run for all four measures, with signi cant improvements for MRR and P@10. Adding terms from the top 10 documents leads to a better description of the information need. However, the improvement in nDCG@10, which emphasises the suggestions that the topic creator selects to add to her catalogue, is not signi cant. For our experiments with popularity and quality priors and recommendations we use the feedback run p4.xml social.fb.10.50 as the baseline, which is the highest scoring run of all o cial submissions on nDCG@10.
The results are shown in Table 2. We start with the quality priors. The ratings have little impact on performance. All variants are able to improve MRR, but on the other measures the improvements are smaller and not signi cant. The only exception is the plain Bayesian average prior, which is more e ective than the others. This suggests that ratings are mainly useful for improving very early precision. The improvement of the BA Rating prior on nDCG@10 suggests that topic creators take ratings into account when selecting books. However, most improvements are not signi cant. Perhaps ratings do not re ect quality well, or quality is not e ective as user-independent evidence. In the latter case, it might mean that quality is perceived di erently by di erent users.
Next we discuss the popularity priors. The tag-based priors lead to signi cant drops in performance when used directly. Curbing their impact by taking the log of the MaxTag or SumTag scores is still not e ective. Only the Log(SumTag) prior leads to small but insigni cant improvements on MRR and nDCG@10. The number of reviews is more e ective. The plain NumReviews prior only improves MRR but hurts performance on the other measures. The compressed score range of the Log(NumReviews) prior is more e ective. Performance on all measures improves, with more than 11% improvements for MRR and nDCG@10. The larger improvement for nDCG@10 than for P@10 indicates the reviews are particularly useful for promoting suggestions that the topic creator decides to catalogue. Only the improvement on MRR is signi cant. This can mean that the number of reviews is a better indicator of popularity than SumTags and MaxTag, or that the topic creator tends to select books for which multiple reviews are available.
The Length prior is only e ective when logged, and only improves performance on MRR and nDCG@10, but not signi cantly. The implicit length prior of the Dirichlet smoothing parameter is more stable, and improves performance on all measures for = 10; 000. With higher values for , performance starts to drop. Completely ignoring document length and only considering term frequency and document frequency is not good for performance. Even though promoting longer document is e ective, it is still important to connect term frequency to the amount of text in a document.
Although some popularity and quality ratings can improve performance, any improvements on the o cial measure nDCG@10 are not signi cant. Evidence that is both user- and query-dependent seems not e ective for social book search.
Finally, we turn to the impact of combining retrieval with recommendation.
For the k-NN method we experimented with di erent neighbourhood sizes (25,
50, and 100 neighbours) and values. Typically, the best performance with
k-NN is achieved with 20 k 50 ([
In sum, evidence based on personal preferences of the user seems much more
e ective than user-independent evidence based on popularity and quality. The
low impact of the quality priors might indicate that quality in book search is
more user-dependent. The e ectiveness of the number of reviews may be an
indicator that popularity can be e ective, but also that forum members looking
for books only catalogue books for which reviews are available. This is in line
with our previous ndings that workers on Mechanical Turk, when judging the
relevance of books for the same LT forum topics, nd it hard to judge books
for which no reviews are available Koolen et al. [
For the Faceted Search Task of the Linked Data Track, systems are required to create a list of both facets and facet values for the explorative search queries contained in the topics of this task. The derived facets should describe relevant information for each of the queries featured in the task, preferably resulting in compact summaries of the available data. Our aim is to experiment with di erent ways of aggregating results, using either document counts or retrieval scores, and either ignoring or penalising document overlap in the ranking of facet values. The idea behind using retrieval scores for aggregation is that we want to focus on the top ranked results, as the retrieval model ranks documents by relevance, with the most relevant documents in the top of the list. Facet values that summarise many of the top documents give the user easy access to the most relevant documents.
Of course, the point of aggregation is to summarise long lists of results
effectively and e ciently, so focussing on facet values that summarise only the
top few documents de es the purpose of result aggregation. Good facet value
selection requires a careful balance between high coverage and giving access to
the most relevant documents.
We use Indri [
The Ad Hoc run is used as the basis for carrying out the Faceted Search Task. We explored possibilities to extract di erent facets and facet values from the data available in the Wikipedia-LOD collection of the Linked Data Track. The candidate facets consist of the DBpedia relations and properties for each Wikipedia article included in the collection. For our exploration, we are also using additional ontological data available from DBpedia itself.
As a basic approach to performing the selection of facets, we used the concept
of 'facet coverage' [
Using the ontology relations of DBpedia, we generated a list of all
possible facets for a topic from the available DBpedia properties contained in each
Wikipedia-LOD article in the collection. The list of facets includes the top 5
values for each facet, based on the number of documents a value covers, and
the top 5 values based on their retrieval scores (originating from the baseline
run created using Indri). To select a number of top facets out of the list of all
facets for a given query, we are using di erent methods. One way to select the
facets is based on the facet coverage. A disadvantage of this method, however,
is that this does not take the overlap between facets into account. Therefore,
a second method has been used, coverageNO, that focuses on the number of
unique documents summarized by the facets top n values (see also [
Based on a recursive selection method, it is possible to create a hierarchical list of facets and facet values. There are some issues with the available data from DBpedia, which in uenced the facet selections that we explored in our research. First of all, there is a wide range of properties that are used for DBpedia entities, but not all of them are applied consistently. Furthermore, a substantial number of the top-ranked results from our baseline run do not have DBpedia properties, except for links to other pages, and therefore are not included in the generated facets. Finally, some of the entities have incorrect properties, possibly due to the semi-automatically generated structure of DBpedia, that is based on the userauthored data of Wikipedia. To overcome these limitations, we are also exploring ways to include additional data from DBpedia in the process of selecting facets, for example the ontological structure of DBpedia.1 1 URL: http://mappings.dbpedia.org/server/ontology/classes/
In this paper we discussed our participation in the INEX 2012 Social Book Search Track and the Linked Data Track.
For the Social Book Search Track, we experimented with user-dependent and user-independent evidence in the form of document priors|length, book ratings, and numbers of tags and reviews|and user-dependent evidence in the form of recommendations from collaborative ltering approaches. We crawled a large set of user pro les and personal catalogues of LibraryThing members and experimented with neighbourhood-based and model-based recommender systems.
We found that document priors re ecting quality and popularity do not improve performance of a standard language model with blind feedback. The number of reviews of a book is the most e ective prior, but does not lead to signi cant improvements. It is not clear whether the number of reviews is e ective because it re ects popularity or because it promotes books for which the searcher can read multiple reviews and therefore make a more informed selection. Our ndings suggest that evidence that is both query- and user-independent is not e ective for social book search.
In contrast, user-dependent information from recommender systems is highly e ective. Both k-nearest neighbour and SVD approaches lead to signi cant improvements. Although the k-NN method is less complex than SVD, it is the more e ective of the two. Our ndings suggest that user-dependent evidence is more important than user-independent information.
For the Linked Data Track, our aims are to compare the e ectiveness of di erent result aggregation approach and of ignoring or penalising overlap in the results summarised by the chosen values of a selected facet. We are still implementing this model and the relevance judgements are not yet available, so we have no evaluation results yet.
Acknowledgments This research was supported by the Netherlands Organization for Scienti c Research (NWO projects # 612.066.513, 639.072.601, and 640.005.001) and by the European Communitys Seventh Framework Program (FP7 2007/2013, Grant Agreement 270404).