The goal of the SBS 2016 Suggestion Track is to evaluate approaches for supporting users in searching collections of books who express their information needs both in a query and through example books. The track investigates the complex nature of relevance in book search and the role of traditional and user-generated book metadata in retrieval. We consolidated last year's investigation into the nature of book suggestions from the LibraryThing forums and how they compare to book relevance judgements. Participants were encouraged to incorporate rich user pro les of both topic creators and other LibraryThing users to explore the relative value of recommendation and retrieval paradigms for book search. In terms of systems evaluation, the most e ective systems include ...
The goal of the Social Book Search 2016 Suggestion Track4 is to investigate techniques to support users in searching for books in catalogues of professional metadata and complementary social media. Towards this goal the track is building appropriate evaluation benchmarks, complete with test collections for social, semantic and focused search tasks. The track provides opportunities to explore research questions around two key areas:
The Social Book Search (SBS) 2016 Suggestion Track, framed within the scenario of a user searching a large online book catalogue for a given topic of interest, aims at exploring techniques to deal with complex information needs| that go beyond topical relevance and can include aspects such as genre, recency, engagement, interestingness, and quality of writing|and complex information sources that include user pro les, personal catalogues, and book descriptions containing both professional metadata and user-generated content.
The Suggestion Track has been part of the SBS Lab since 2015 and is a continuation of the INEX SBS Track that ran from 2011 up to 2014. The focus is on search requests that combine a natural language description of the information need as well as example books, combining traditional ad hoc retrieval with query-by-document. The information needs are derived from the LibraryThing (LT) discussion forums. LibraryThing forum requests for book suggestions, combined with annotation of these requests resulted in a topic set of 120 topics with graded relevance judgments. A test collection is constructed around these information needs and the Amazon/LibraryThing collection, consisting of 2.8 million documents. The Suggestion Track runs in close collaboration with the SBS Interactive Track,5 which is a user-centered track where interfaces are developed and evaluated and user interaction is analysed to investigate how book searchers make use of professional metadata and user-generated content. 2
A total of 29 organisations registered for the track, with 10 teams submitting runs (see Table 1). In 2015 25 teams registered and 10 submitted runs, so participation is stable.
3.1
The goal of the Suggestion Track is to evaluate the value of professional metadata and user-generated content for book search on the Web and to develop and evaluate systems that can deal with both retrieval and recommendation aspects, where the user has a speci c information need against a background of personal tastes, interests and previously seen books.
Through social media, book descriptions have extended far beyond what is
traditionally stored in professional catalogues. Not only are books described in
the users' own vocabulary, but are also reviewed and discussed online, and added
to online personal catalogues of individual readers. This additional information
is subjective and personal, and opens up opportunities to aid users in searching
for books in di erent ways that go beyond the traditional editorial metadata
based search scenarios, such as known-item and subject search. For example,
readers use many more aspects of books to help them decide which book to read
next
The SBS track investigates book requests and suggestions from the LibraryThing (LT) discussion forums as a way to model book search in a social environment. The discussions in these forums show that readers frequently turn to others to get recommendations and tap into the collective knowledge of a group of readers interested in the same topic.
The track builds on the INEX Amazon/LibraryThing (A/LT) collection
The SBS Suggestion Track aims to address the following research questions: { Can we build reliable and reusable test collections for social book search based on book requests and suggestions from the LT discussion forums? { Can user pro les provide a good source of information to capture personal, a ective aspects of book search information needs? { How can systems incorporate both speci c information needs and general user pro les to combine the retrieval and recommendation aspects of social book search? { What is the relative value of social and controlled book metadata for book search? 3.2
The scenario is that of a user turning to Amazon Books and LT to nd books to read, to buy or to add to their personal catalogue. Both services host large collaborative book catalogues that may be used to locate books of interest.
On LT, users can catalogue the books they read, manually index them by assigning tags, and write reviews for others to read. Users can also post messages on discussion forums asking for help in nding new, fun, interesting, or relevant books to read. The forums allow users to tap into the collective bibliographic knowledge of hundreds of thousands of book enthusiasts. On Amazon, users can read and write book reviews and browse to similar books based on links such as \customers who bought this book also bought... ".
Users can search online book collections with di erent intentions. They can search for speci c known books with the intention of obtaining them (buy, download, print). Such needs are addressed by standard book search services as o ered by Amazon, LT and other online bookshops as well as traditional libraries. In other cases, users search for a speci c, but unknown, book with the intention of identifying it. Another possibility is that users are not looking for a speci c book, but hope to discover one or more books meeting some criteria. These criteria can be related to subject, author, genre, edition, work, series or some other aspect, but also more serendipitously, such as books that merely look interesting or fun to read or that are similar to a previously read book. 3.3
The task is to reply to a user request posted on a LT forum (see Section 4.1) by returning a list of recommended books matching the user's information need. More speci cally, the task assumes a user who issues a query to a retrieval system, which then returns a (ranked) list of relevant book records. The user is assumed to inspect the results list starting from the top, working down the list until the information need has been satis ed or until the user gives up. The retrieval system is expected to order the search results by relevance to the user's information need.
The user's query can be a number of keywords, but also one or more book records as positive or negative examples. In addition, the user has a personal pro le that may contain information on the user's interests, list of read books and connections with other readers. User requests may vary from asking for books on a particular genre, looking for books on a particular topic or period or books written in a certain style. The level of detail also varies, from a brief statement to detailed descriptions of what the user is looking for. Some requests include examples of the kinds of books that are sought by the user, asking for similar books. Other requests list examples of known books that are related to the topic, but are speci cally of no interest. The challenge is to develop a retrieval method that can cope with such diverse requests.
The books must be selected from a corpus that consists of a collection of curated and social book metadata, extracted from Amazon Books and LT, extended with associated records from library catalogues of the Library of Congress and the British Library (see the next section). Participants of the Suggestion track are provided with a set of book search requests and user pro les and are asked to submit the results returned by their systems as ranked lists.
The track thus combines aspects from retrieval and recommendation. On the one hand the task is akin to directed search familiar from information retrieval, with the requirement that returned books should be topically relevant to the user's information need described in the forum thread. On the other hand, users may have particular preferences for writing style, reading level, knowledge level, novelty, unusualness, presence of humorous elements and possibly many other aspects. These preferences are to some extent re ected by the user's reading pro le, represented by the user's personal catalogue. This catalogue contains the books already read or earmarked for future reading, and may contain personally assigned tags and ratings. Such preferences and pro les are typical in recommendation tasks, where the user has no speci c information need, but is looking for suggestions of new items based on previous preferences and history. 3.4
Participants are asked to return a ranked list of books for each user query, ranked by order of relevance, where the query is described in the LT forum thread. We adopt the submission format of TREC, with a separate line for each retrieval result (i.e., book), consisting of six columns: 1. topic id: the topic number, which is based on the LT forum thread number. 2. Q0: the query number. Unused, so should always be Q0. 3. isbn: the ISBN of the book, which corresponds to the le name of the book description. 4. rank: the rank at which the document is retrieved. 5. rsv: retrieval status value, in the form of a score. For evaluation, results are ordered by descending score. 6. run id: a code to identify the participating group and the run.
Participants are allowed to submit up to six runs, of which at least one should use only the title eld of the topic statements (the topic format is described in Section 4.1). For the other ve runs, participants could use any eld in the topic statement. 4
We use and extend the Amazon/LibraryThing (A/LT) corpus crawled by the
University of Duisburg-Essen for the INEX Interactive Track
The collection consists of 2.8 million book records from Amazon, extended with social metadata from LT. This set represents the books available through Amazon. Each book is identi ed by an ISBN. Note that since di erent editions of the same work have di erent ISBNs, there can be multiple records for a single intellectual work. Each book record is an XML le with formal metadata elds like isbn, title, author, publisher, dimensions, numberofpages and publicationdate. Curated metadata comes in the form of a Dewey Decimal Classi cation in the dewey eld, Amazon subject headings in the subject eld, and Amazon category labels in the browseNode elds. The social metadata from Amazon and LT is stored in the tag, rating, and review elds. The full list of elds is shown in Table 2.
To ensure that there is enough high-quality metadata from traditional library catalogues, we extended the A/LT data set with library catalogue records from the Library of Congress (LoC) and the British Library (BL). We only use library records of ISBNs that are already in the A/LT collection. These records contain formal metadata such as title information (book title, author, publisher, etc.), classi cation codes (mainly DDC and LCC) and rich subject headings based on the Library of Congress Subject Headings (LCSH).6 Both the LoC records and the BL records are in MARCXML7 format. 6 For more information see: http://www.loc.gov/aba/cataloging/subject/ 7 MARCXML is an XML version of the well-known MARC format. See: http://www.
loc.gov/standards/marcxml/ LT users discuss their books on the discussion forums. Many of the topic threads are started with a request from a member for interesting, fun new books to read. Users typically describe what they are looking for, give examples of what they like and do not like, indicate which books they already know and ask other members for recommendations. Members often reply with links to works catalogued on LT, which, in turn, have direct links to the corresponding records on Amazon. These requests for recommendations are natural expressions of information needs for a large collection of online book records. We use a sample of these forum topics to evaluate systems participating in the Suggestion Track.
Each topic has a title and is associated with a group on the discussion forums. For instance, topic 99309 in Figure 1 has the title Politics of Multiculturalism Recommendations? and was posted in the group Political Philosophy. The books suggested by members in the thread are collected in a list on the side of the topic thread (see Figure 1). A feature called touchstone can be used by members to easily identify books they mention in the topic thread, giving other readers of the thread direct access to a book record in LT, with associated ISBNs and links to Amazon. We use these suggested books as initial relevance judgements for evaluation. In the rest of this paper, we use the term suggestion to refer to a book that has been identi ed in a touchstone list for a given forum topic. Since all suggestions are made by forum members, we assume they are valuable judgements on the relevance of books. Additional relevance information can be gleaned from the discussions on the threads. Consider, for example, topic 1299398. The topic starter rst explains what sort of books he is looking for, and which relevant books he has already read or is reading. Other members post responses with book suggestions. The topic starter posts a reply describing which suggestions he likes and which books he has ordered and plans to read. Later on, the topic starter provides feedback on the suggested books that he has now read. Such feedback can be used to estimate the relevance of a suggestion to the user.
In the following, we rst describe the topic selection and annotation procedure, then how we used the annotations to assign relevance values to the suggestions, and nally the user pro les, which were then provided with each topic.
Topic selection The topic set of 2016 is a newly selected set of topics from
the LibraryThing discussion forums. A total of 2000 topic threads were assessed
on whether they contain a book search request by four judges, with 272 threads
labelled as book search requests. To establish inter-annotator agreement, 453
threads were double-assessed, resulting a Cohen's Kappa of 0.83. Judges strongly
agree on which posts are book search requests and which are not. Of these 272
book search requests, 124 (46%) are known-item searches from the Name that
Book discussion group. Here, LT members start a thread to describe a book they
know but cannot remember the title and author of and ask others for help. In
earlier work we found that known-item topics behave very di erently from the
other topic types (
Topics are distributed to participants in XML format with as richly annotated requests. As an example, the topic in Figure 1 (topic 99309) would look like this: <topic> <topicid>99309</topicid> <query>Politics of Multiculturalism</query> <title>Politics of Multiculturalism Recommendations?</title> <group>Political Philosophy</group> <request> I'm new, and would appreciate any recommended reading on the politics of multiculturalism. <a href="/author/parekh">Parekh </a>'s <a href="/work/164382">Rethinking Multiculturalism: Cultural Diversity and Political Theory</a> (which I just finished) in the end left me unconvinced, though I did find much of value I thought he depended way too much on being able to talk out the details later. It may be that I found his writing style really irritating so adopted a defiant skepticism, but still... Anyway, I've read <a href="/author/sen">Sen</a>, <a href="/author/rawles">Rawls</a>, <a href="/author/habermas">Habermas</a>, and 8 URL: http://www.librarything.com/topic/129939
The hyperlink markup, represented by the <a> tags, is added by the Touchstone technology of LT. The rest of the markup is generated speci cally for the Suggestion Track. Above, the book with work ID 164382 is annotated as an example of what the requester is looking for.
Suggestion provided by other LT members are often marked up in touchstones so they are easy to extract and use as relevance judgements. As for the 2015 Suggestion Track, we manually annotated all book suggestions that were not marked up as touchstones by LT members to provide a more complete recall base.
Operationalisation of forum judgement labels In previous years the Suggestion Track used a complicated decision tree to derive a relevance value from a suggestion. To reduce the number of assumptions, we simpli ed the mapping of book suggestions to relevance values. By default a suggested book has a relevance value of 1. Books that the requester already has in her personal catalogue before starting the thread (pre-catalogued suggestions) have little additional value are assumed to have a relevance value of 0. On the other hand, suggestions that the requester subsequently adds to her catalogue (post-catalogued suggestions) are assumed to be the most relevant suggestions and receive a relevance value of 8, to keep that relevance level the same is in 2014 and 2015. Note that some of the books mentioned in the forums are not part of the 2.8 million books in our collection. These suggestions removed from the suggestions any books that are not in the INEX A/LT collection. The numbers reported in the previous section were calculated after this ltering step.
User pro les and personal catalogues From LT we can not only extract the information needs of social book search topics, but also the rich user pro les of the topic creators and other LT users, which contain information on which books they have in their personal catalogue on LT, which ratings and tags they assigned to them and a social network of friendship relations, interesting library relations and group memberships. These pro les may provide important signals on the user's topical and genre interests, reading level, which books they already know and which ones they like and don't like. These pro les were scraped from the LT site, anonymised and made available to participants. This allows Track participants to experiment with combinations of retrieval and recommender systems. One of the research questions of the SBS task is whether this pro le information can help systems in identifying good suggestions.
Although the user expresses her information need in some detail in the discussion forum, she may not describe all aspects she takes into consideration when selecting books. This may partly be because she wants to explore di erent options along di erent dimensions and therefore leaves some room for di erent interpretations of her need. Another reason might be that some aspects are not related directly to the topic at hand but may be latent factors that she takes into account with selecting books in general.
To anonymise all user pro les, we rst removed all friendship and group membership connections and replaced the user name with a randomly generated string. The cataloguing date of each book was reduced to the year and month. What is left is an anonymised user name, book ID, month of cataloguing, rating and tags. We distributed a set of 94,656 user pro les containing over 33 million transactions.
ISBNs and Intellectual Works Each record in the collection corresponds to an ISBN, and each ISBN corresponds to a particular intellectual work. An intellectual work can have di erent editions, each with their own ISBN. The ISBN-to-work relation is a many-to-one relation. In many cases, we assume the user is not interested in all the di erent editions, but in di erent intellectual works. For evaluation we collapse multiple ISBN to a single work. The highest ranked ISBN is evaluated and all lower ranked ISBNs of the same work ignored. Although some of the topics on LibraryThing are requests to recommend a particular edition of a work|in which case the distinction between di erent ISBNs for the same work are important|we ignore these distinctions to make evaluation easier. This turns edition-related topics into known-item topics.
However, one problem remains. Mapping ISBNs of di erent editions to a single work is not trivial. Di erent editions may have di erent titles and even have di erent authors (some editions have a foreword by another author, or a translator, while others have not), so detecting which ISBNs actually represent the same work is a challenge. We solve this problem by using mappings made by the collective work of LibraryThing members. LT members can indicate that two books with di erent ISBNs are actually di erent manifestations of the same intellectual work. Each intellectual work on LibraryThing has a unique work ID, and the mappings from ISBNs to work IDs is made available by LibraryThing.9
The mappings are not complete and might contain errors. Furthermore, the mappings form a many-to-many relationship, as two people with the same edition of a book might independently create a new book page, each with a unique work ID. It takes time for members to discover such cases and merge the two work IDs, which means that at any time, some ISBNs map to multiple work IDs even though they represent the same intellectual work. LibraryThing can detect such cases but, to avoid making mistakes, leaves it to members to merge them. The fraction of works with multiple ISBNs is small so we expect this problem to have a negligible impact on evaluation. 5
This year, 10 teams submitted a total of 46 runs (see Table 1). The evaluation results shown in Table 3. The o cial evaluation measure for this task is nDCG@10. It takes graded relevance values into account and is designed for evaluation based on the top retrieved results. In addition, P@10, MAP and MRR scores are also reported.
The best runs of the top 5 groups are described below: 1. USTB-PRIR - run1.keyQuery active combineRerank (rank 1): This run was made by a searching-re-ranking process where the ini- tial retrieval result was based on the selection of query keywords and a small index of active books, the re-ranking results based on a combination of several strategies (number of people who read the book from pro le, similar-product from amazon.com, popularity from LT forum, etc.). At indexing time, the collection is enriched with book metadata from other sources, particularly books that have little metadata from Amazon and LibraryThing. The full topic starter post in the <request> eld is ltered based key word lists from topics of the SBS Tracks of 2011{2015. The top 1000 retrieval results of a Language Model with default parameters is re-ranked using a number of query-independent features. 2. CERIST - all features (rank 7): The topic statement in the request eld is treated as a verbose query and is reduced using several features based on term statistics, Part-Of-Speech tagging, and whether terms from the request eld occur in the user pro le and example books. 3. CYUT-CSIE - 0.95Averageword2vecType2TGR (rank 11): This run uses query expansion based on word embeddings using word2vec, on top of a standard Lucene index and retrieval model. For this run, queries are represented by a combination of the title, group and request elds. Results are re-ranked using a linear combination of the original retrieval score and the average Amazon user ratings of the retrieved books.
mrr
map
Most of the top performing systems, including the top performing run preprocess the rich topic statement with the aim of reducing the request to a set of most relevant terms. Two of the top ve teams use the user pro les to modify the topic statement. This is the rst year that word embeddings are used for the Suggestion Track and not without success. Both CYUT-CSIE and MRIM found that word embeddings improved performance over con gurations without them. From these results it seems clear that topic representation is an important aspect in social book search. The longer narrative of the request eld as well as the metadata in the user pro les and example books contain important information regarding the information need, but many terms are noisy, so a ltering step is essential to focus on the user's speci c needs. 6
This was the second year of the SBS Suggestion Track as part of the SBS Lab. The overall goal remains to investigate the relative value of professional metadata, user-generated content and user pro les, but the speci c focus for this year is to construct a test collection to evaluate systems dealing with complex book search requests that combine an information need expressed in a natural language statement and through example books.
We kept the evaluation procedure the same. Like last year, we added annotated example books with each topic statement, so that participants can investigate the value of query-by-example techniques in combination with more traditional text-based queries. Whereas in 2015 we only provided the book ID of example books in the topic statement, this year we also provided book title and author name.
The evaluation has shown that the most e ective systems either adopt a learning-to-rank approach or incorporate keywords from the example books in the textual query. The e ectiveness of learning-to-rank approaches suggests the complexity of dealing with multiple sources of evidence|book descriptions by multiple authors, di ering in nature from controlled vocabulary descriptors, freetext tags and full-text reviews and information needs and interests represented by both natural language statements and user pro les|requires optimizing parameters through observing users' interactions.
Next year, we continue this focus on complex topics with example books and consider including an recommender systems type evaluation. We are also thinking of a pilot task in which the system not only has to retrieve relevant and recommendable books, but also to select which part of the book description| e.g. a certain set of reviews or tags|is most useful to show to the user, given her information need.