In this paper we present an overview of the mining track in the Social Book Search (SBS) lab 2016. The mining track addressed two tasks: (1) classifying forum posts as book search requests, and (2) linking book title mentions in forum posts to unique book IDs in a database. Both tasks are important steps in the process of solving complex search tasks within online reader communities. We prepared two data collections for the classi cation task: posts from the LibraryThing (LT) forum and a smaller number of posts from Reddit. For the linking task we used annotated LT threads. We found that the classi cation task was relatively straightforward, achieving up to 94% classi cation accuracy. The book linking task on the other hand turned out to be a di cult task: here the best system achieved an accuracy of 41% and F-score of 33.5%. Both the automatic classi cation of book search requests as the automatic linking of book mentions could next year be part of the pipeline for processing complex book searches.
The Mining track1 is a new addition to the Social Book Search (SBS) Lab in 2016. For the past ve years, the Suggestion Track has explored techniques to deal with complex information needs that go beyond topical relevance and can include other aspects, such as genre, recency, engagement, interestingness, and quality of writing. In addition, it has investigated the value of complex information sources, such as user pro les, personal catalogues, and book descriptions containing both professional metadata and user-generated content.
So far, examples of such complex search tasks have been taken from the LibraryThing (LT) discussion fora. Book search requests were manually separated from other book-related discussion threads by human annotators, and the suggestions provided by other LT users were used as relevance judgments in the automatic evaluation of retrieval algorithms that were applied to the book
search requests. If we wish to move further towards fully supporting complex book search behavior, then we should not just support the retrieval and recommendation stage of the process, but also the automatic detection of complex search needs and the analysis of these needs and the books and authors contained therein. This is the goal of the Mining Track.
The SBS 2016 Mining Track focuses on automating two text mining tasks in particular: 1. Book search request classi cation, in which the goal is to identify which threads on online forums are book search requests. That is, given a forum thread, the system should determine whether the opening post contains a request for book suggestions (i.e., binary classi cation of opening posts) 2. Book linking, in which the goal is to recognize book titles in forum posts and link them to the corresponding metadata record through their unique book ID. The task is not to mark each entity mention in the post text, but to label the post as a whole with the IDs of the mentioned books. That is, the system does not have to identify the exact phrase that refers to book, but only has to identify which book is mentioned on a per-post basis.
The suggestions that LT users provide in response to book search requests are often linked to o cial book metadata records using so-called Touchstones. Touchstones o er a wiki-like syntax for linking books (and authors) mentioned in LT threads to their o cial LT pages (and thereby the books' metadata records). All books mentioned in a thread are shown in a sidebar, so other LT users can see at a glance which books have already been suggested. Or, to quote a LT user: \The main reason I like Touchstones to work is that they allow me to scan the sidebar to see what books have already been discussed in a thread. This is particularly useful in a thread like this (in which somebody is asking for recommendations) because I can take care to mention something new without reading all previous threads (which I won't necessarily do if the thread gets really really long)."
However, not every book mentioned in LT threads is marked up using
Touchstones; previous preliminary work has shown that around 16% of all books are
not linked by LT users [
In this paper, we report on the setup and the results of the 2016 Mining Track as part of the SBS Lab at CLEF 2016. First, in Section 2, we give a brief summary of the participating organisations. Section 3 describes the two tasks in the Mining Track in more detail, along with the data used and the evaluation process. Section 4 presents the results of the participating organisations on the two tasks. We close in Section 5 with a summary and plans for 2017.
A total of 28 organisations registered for the Mining Track and 4 organisations ended up submitting a total of 34 runs. The active organisations are listed in Table 1. 3
In the following sections we describe the data collection and annotation process for both tasks in the 2016 text mining track, as well as the evaluation procedures. 3.1
Data collection For the task of classifying forum threads we created two data sets for training: one based on the LibraryThing (LT) forums and one based on Reddit. For the LT forums, we randomly sampled 4,000 threads and extracted their opening posts. We split them into a training and a test set, each containing 2,000 threads. These threads contained both positive and negative examples of book requests.
The Reddit training data was sampled from three months of Reddit threads collected in September, October, and November 2014. The set of positive book request examples comprises all threads from the suggestmeabook subreddit, whereas the negative examples comprises all threads from the books subreddit. The training set contained 248 threads in total. The Reddit test data was sampled from December 2014 and comprises 89 threads in total. Figure 1 shows an example of the training data format for the classi cation task.
Annotation The labels of the Reddit training data were not annotated manually, as they were already categorized as positive and negative by virtue of the subreddit they originated from. In the annotation process for the LT threads, positive examples of book requests consisted of all posts where the user described an explicit foreground or background information need and was searching for books to read. Examples include known-item requests, where a user is looking for a speci c book by describing plot elements, but cannot remember the title; Aix-Marseille Universite CNRS LSIS Tunis EL Manar University LIPAH Know-Center Know Radboud University Nijmegen RUN users asking for books covering a speci c topic; and users asking for books that are similar to another book they mention. Posts where users ask for new authors to explore or where they list their favorite books and ask others to do the same were not classi ed as explicit book requests.
The manual annotation of the LT data was performed by the four organizers
of the task. To get an impression of the inter-annotator agreement, a small
sample of 432 posts was labeled by two annotators. Average agreement according
to Cohen's was 0.84, averaged over the pairs of annotators, which represents
almost perfect agreement according to Cohen [
For evaluation, 1,974 out of the 2,000 threads in the LibraryThing test set were used. For the 26 remaining threads, judges were unsure whether the rst post was a request or not. The Reddit test set consisted of 89 threads with the subreddit names (books and suggestmeabook) as labels. In order to create a ground truth for the test set, two judges (track organizers) manually classi ed the 89 test threads. They discussed all disagreements and reached consensus on all 89 threads. 81 of the labels were the same as the original Reddit label; the other 8 were di erent. We used the manual labels as ground truth. Table 2 shows the proportion of positive and negative examples in the training and test sets of both data sets. Data collection Book linking through the use of Touchstones is an striking characteristic of the LT forum, and an important feature for the forum community. A Touchstone is a link created by a forum member between a book mention in a forum post and a unique LT work ID in the LT database. A single post can have zero or more di erent touchstones linked to it. Touchstones allow readers of a forum thread to quickly see which books are mentioned in the thread.
For the book linking task we created a data set based on the touchstones in the LT forum. The training data consisted of 200 threads with 3619 posts in total. The training data contains only those touchstones that had been added by the LT authors; we did not enrich the posts with more annotations. Figure 2 shows an example of the training data format for the linking task. In the example, Insomnia is the title of a book. The task is to identify the LT work ID of the corresponding book and link it to that speci c post ID.
Participants used the Amazon/LT collection for linking the book mentions to a database record. This collection originated with the Suggestion track and contains 2.8 million book metadata records along with their LT work IDs. The test data for the linking task comprised 200 LT threads. As opposed to the training data, we did make the annotations in the test data more complete by manually annotating book mentions and linking them to the book database. Annotation of test data In the annotation process, we linked books manually at the post level by their unique LT work ID. Many books are published in di erent editions throughout the years with di erent unique ISBNs, but all of these versions are connected to the same unique LT work ID. If a book occurred multiple times in the same post, only the rst occurrence was linked, so participants only need to specify each of the work IDs found in a post once. If a post mentioned a series of books, we linked this series to the rst book in the series, e.g., the \Harry Potter series" was linked to \Harry Potter and the Philosopher's Stone". In some cases, a book title was mentioned, but no suitable work ID was found in the Amazon/LT collection. In this cases, we labeled that book title as UNKNOWN.
We did not link book authors. When a book was referred to as \the Stephen King book", we did not mark this as a book title. Similarly, if a series was referred to by the name of the author, e.g., \the Stieg Larson trilogy", then the series was not labeled. We do consider these cases where the author is mentioned as borderline cases, because they point to both the author and the books that they wrote at the same time. In this data set we decided not to include them in the annotation, but we are aware that they fall in the `grey' area of unclear cases.
Another source of annotation confusion were the forum threads about short stories and collections of stories. In these cases we did not label the individual short stories (they also do not have existing LT work IDs), but only the actual book with the collection.
Other di cult cases for the manual annotation were the cases where it was not immediately clear where the book title begins and ends. For example, in (1) below, the alternative book title could have been \Bujold's Sharing Knife" instead of \Sharing Knife". Vague or partial matches were also di cult to annotate sometimes. For example, the post containing fragment (2) was not linked to a work ID because it deviated signi cantly from the actual title of the book that was mentioned (and linked) correctly in the follow up post as being \Fifteen Decisive Battles of the World from 1851".
(1) have you read Lois McMaster Bujold's Sharing Knife books? (2) I think there was a book called something like Ten Decisive Battles by a General Creasey
During manual annotation, 3 of the 220 threads were removed from the test set because of long lists of titles without context. The nal test set consists of 217 threads comprising 5097 book titles identi ed in 2117 posts.
In order to assess the di culty and subjectivity of the book linking annotation task we had 28 threads (155 posts) annotated by 2 assessors and we analyzed the di erences in annotation. We found that there was quite some disagreement between the assessors: 71 books were linked by both assessors, and 247 by only one of the two. This implies that absolute agreement is only 22%.2 There are two types of disagreement: (a) a book mention was linked by one assessor and missed/skipped by the other, and (b) a book mention was linked by both, but to di erent work ids. The most di cult were the mentions of book series. These should be linked to the rst book of the series, which is not always trivial. For example, consider this post text:
Well, I could recommend some great Batman graphic novels, only one problem. They're written for adults, and are pretty dark. Year One is an amazing version of his origin story, but it isn't exactly appropriate for a second grader. You might try some of the Tintin graphic novels. There are dozens of them, and they're great stories. I second Louis Sachar as well. You might want to try Holes. Its a great, inventive story. Plus, you can watch the movie together once he nishes the book.
Both assessors linked two series in this post. These were linked by assessor 1: { David Mazzuchelli { Batman: Year One - Deluxe Edition: Year One { Herge { Tintin in America (Tintin) and these were linked by assessor 2: { Lewis Richmond { Batman: Year One (Batman) { Herge { Tintin in the Land of the Soviets The di culty of the annotation for the linking task is a topic that should be addressed in future editions of the SBS lab. One recommendation would be to write more explicit annotation guidelines, and share those with the participants.s 3.3
For the book request classi cation task, we computed and report only accuracy, as these are binary decisions. For the linking task, we computed accuracy, precision, recall, and F-score.
Both tasks were performed and evaluated at the level of forum posts. We detected whether a forum post was a book request in the classi cation task, and whether a certain book title occurred in a post. In case the same book title was mentioned multiple times in the same post, we only counted and evaluated on one occurrence of this particular book title. Each book title is mapped to a LibraryThing work ID that links together di erent editions of the same book (with di erent ISBNs). 2 Note that Cohen's is unde ned for these data because we the number of book titles for which the assessors agree that they should not be linked is in nite.
During manual annotation, we came across several book titles for which we were unable to nd the correct LT work ID (labeled as UNKNOWN). These cases were problematic in the evaluation: just because the annotator could not nd the correct work ID does not mean that it does not exist. For that reason, we decided to discard these examples in the evaluation of the test set results. In total, 180 out of the 5097 book titles in the test set were discarded for this reason.
Similarly, during the book request classi cation task, we also found some cases in the LT data where we were unsure about categorizing them as book search requests or not. We discarded 26 such cases from the test set in the evaluation. 4
A total of 3 teams submitted 15 runs, 2 teams submitted 9 runs for the Classication task and 2 teams submitted 6 runs for the Linking task. 4.1
Baselines For the baseline system of the classi cation task, we trained separate
classi ers for the two data sets (LT and Reddit) using scikit-learn.3 We extracted
bag-of-words-features (either words or character 4-grams) from the title and the
body of the rst post, and for LT also from the category (for Reddit, the category
was the label). We used tf-idf weights for the words and the character 4-grams
from these elds. We ran 3 classi ers on these data: Multinonial Naive Bayes
(MNB), Linear Support Vector Classi cation (LinearSVC) and KNN, all with
their default hyperparameter settings in scikit-learn. The results are in Table 3.
Evaluation of submitted runs The Know team reported an interesting
experiment on the LT training data of the classi cation task [
The LIPAH team compared two types of features for the classi cation task:
(a) all nouns and verbs in the posts, and (b) compound nouns and phrases
extracted using syntactic patterns. They found that the addition of syntactic
phrases improves the classi cation accuracy [
Table 3 shows that for the LT data, the submitted runs did not beat the LinearSVC baselines. For the Reddit data however, runs by both teams were able to beat the best baseline system by a large margin. Since the Reddit dataset was much smaller than the LT dataset, the best strategy seems to be to add the LT
training data to the Reddit training data for classifying the Reddit test threads.
The best run for the Reddit data is LIPAH-submission6, which uses sequences of
words and verbs as features.
Evaluation of submitted runs The results of the book linking task can be
found in Table 4. The Know team used a list look-up system combined with
a weighting threshold in their sbs16classi cationlinking run to prevent the
overgeneration of potential book titles [
The LSIS team [
Both teams investigated the usage of author names in the proximity of potential book titles to disambiguate between potential titles and show that this is indeed a helpful feature. Both teams use complementary strategies for the book linking task as the Know systems has a higher recall while the LSIS run all achieves a better precision (as well as the highest F-score). 5
This was the rst year of the Social Book Search Mining Track. Our goal was create a benchmark data set for text mining of book related discussion forum. In this rst edition we focused on two tasks. The rst task was to automatically identify which posts in a book forum tread are actual book search requests, and the second task was to detect which book titles are mentioned in a forum post and link the correct unique book ID to the post. We had three active participants who submitted a total of 15 runs. The book search classi cation task turned out to be a relatively straightforward task, both in manual annotation and in automatic prediction. A rather simple bag-of-words baseline classi er achieved an accuracy up to 94% on the LibraryThing data. The book linking task on the other hand turned out to be a di cult task and here the best system achieved an accuracy of 41% and F-score of 33.5%.
Developing e ective algorithms for automatically detecting and linking these book mentions would be a boon to the process of supporting complex search needs. Moreover, other book discussion websites, such as GoodReads or even dedicated Reddit threads may not have Touchstone-like functionality. Here, the need for automatic book linking algorithms is even more pressing.