In this paper, we present our contribution in INEX 2016 Social Book Search Track. This year, we participate in a new track called Mining track. This track focuses on detecting and linking book titles in online book discussion forums. We propose a supervised approach based on Support Vector Machine (SVM) classi cation process combined with Conditional Random Fields (CRF) to detect book titles. Then, we use a Levenshtein distance to link books to their unique book ID.
The Social Book Search (SBS) Tracks [
We submit 5 runs in which we compare several variations of selected features provide by CRF, of the combination of detected tags (book titles and author names) and of the factor taken by the Levenshtein distance.
The rest of this paper is organized as follows. The following section describes our approach. In section 3, we describe the submitted runs. We present the obtained results in section 4. 2
In this section, we present our supervised approach dedicated to book titles detection and link to their unique book ID. Firstly, we de ne a classi cation process with Support Vector Machine (SVM). Secondly, we describe the implementation of the CRF used. Thirdly, we present the use of Levenshtein distance to link books to their unique book ID. 2.1
Retrieving posts with book titles using SVM
In context of online forums, we have a wide variety of themes that are addressed.
So, it is necessary to conduct a pre- ltering on verbose queries for identify queries
with book titles. We choose to use a supervised classi cation based on SVM. We
de ne two classes: "bibliographic eld" versus "no bibliographic eld". We
establish a manual training set extracts randomly from the threads provided for
the task. The class "bibliographic eld" contains 184 posts and the class "no
bibliographic eld" 153 posts. For the SVM implementation, we use SVMLight 2
[
Regarding the settings, we use Weka3 in order to establish a list of the most characteristic words of our classes, we use as attributes. Figure 1 shows an example of this list. The rst column designates the score of "Recursive Feature Elimination" (RFE) and the second column refers to word. This list is obtained by 2 http://svmlight.joachims.org/ 3 http://www.cs.waikato.ac.nz/ml/weka/downloading.html the algorithm InfoGainAttribute which reduces a bias of multi-valued attributes. After several tests, we decided to use 1 as a minimum occurrence frequency of terms combined with a list from which we have removed the words with score RFE equal 0. This list is composed of 2182 words. We conduct 10-fold crossvalidations to assess how the results generalize to a set of data. We obtain an accuracy of 80.69% with a recall of 100.00% and a precision of 78.45%. 2.2
Authors and book titles detection based on CRF As part of our work, we choose to use an approach based on CRF. We establish a training set extracts randomly from the threads provide for the task. We manually annotated 133 posts in which we marked both book titles and author names. In total, we annotated 264 book titles and 203 author names. For constructing our CRF, we use several features presented in tables 1 and 2. For the CRF implementation, we use the tool Wapiti 4.
Main characteristics exploited in the literature for the automatic annotation of references are based on a number of observations such as lexical or morphological characteristics, both on the elds and the words contained in the elds. Drawing a parallel between the task of named entities detection and the analysis of bibliographic references, we are able to extract more useful information in the characterization elds and words contained in the elds. As part of our work, we decide to use a typology of features inspired by the literature. - Contextual features: We add several features using the other tokens around the current one as three preceding and three following tokens. We use grammatical properties provides by a POS Tagger5. We use a natural language parser which provide structure of sentences. The latter two features, allow us to establish patterns of syntactic which de ne, independently words, potential anchors with greater portability. Table 1 describes these features. Feature category Raw input token Preceding or following tokens N-gram POS Tagger NL Parser
Description Tokenized word itself in the input string and the lowercased word Three preceding and three following tokens of current token Aggregation of preceding or following N-gram tokens Aggregation of preceding or following grammatical properties Aggregation of preceding or following syntactical properties
Table 1. Description of contextual features 4 https://wapiti.limsi.fr/ 5 http://nlp.stanford.edu/software/tagger.html - Local features: they are divided into four categories: morphological, local, lexical and syntactic characteristics. The morphological features that were selected to characterize the shape of the tokens. The locational features have been selected to de ne the position of the elds in a sequence. The lexical features have been selected to use lists of prede ned words but also linguistic category of words. And lastly, the punctuation features. Table 2 describes in contextual features.
Morphological features Feature category Feature name Number ALLNUMBERS
NUMBERS DASH
FIRSTCAP ALLSAMLL NONIMPCAP INITIAL WEBLINK ITALIC
All characters are numbers One or more characters are numbers One or more dashes are included in numbers All characters are capital letters First character is capital letter All characters are lower cased Capital letters are mixed Initialized expression Regular expression for web pages Italic characters Transformation in their radical or root Canonical form of current token form
Position is between the two-third and the end
Locational features Location Lexical features Lexicon
POSSEDITOR Possible for the abbreviation of editor POSSPAGE Possible for the abbreviation of page POSSMONTH Possible for month POSSBIBLSCOP Possible for abbreviation of bibliographic extension POSSROLE Possible for abbreviation of roles of entities SURNAMELIST Found in an external surname list FORENAMELIST Found in an external forename list PLACELIST Found in an external place list JOURNALLIST Found in an external journal list Set tags Harmonized Part of speech
Punctuation features Punctuation
POINT LINK PUNC LEADINGQUOTES ENDINGQUOTES PAIREDBRACES Table 2: Description of local features
From these characteristics we construct vectors for each word.
Following the classi cation, we get a list of posts containing book titles potentially. Then, our CRF allows us to annotate the area referring to book titles or author names. 2.3
Mapping to book Ids Once book titles or author names detection carried out, we use the Levenshtein distance for link books to their unique book ID. Just for recall, Levenshtein distance is a string metric for measuring the di erence between two sequences. Informally, the Levenshtein distance between two words is the minimum number of single-character edits required to change one word into the other. As part of our work, we use two variations of the Levenshtein distance: either the length of the shortest alignment between the sequences is taken as factor, or the length of the longer one.
For each book title found, we compare it with the whole of book titles extracts from the collection. For each book title, we obtained a list of the books sorted by normalized Levenshtein distance, so that the results of several distance measures can be meaningfully compared. Figure 2 presents the best three results obtained for the book entitled "The Old Man". For each book title, we keep the best result which is close to 1. Then, we retrieve the unique ID of the most probable best book. If an author name is located at a maximum distance of four words, we aggregate it. Figure 3 shows a query which contains both book title and author name. Then, gure presents the result obtained for the input "Timothy Findley The Last of the Crazy People". We submitted 5 runs for the linking task of Mining track. For each run, we use only the data set which contains book IDs, basic title and author metadata per book. Once the classi cation process and the annotation process done, we link books at the post level by their unique LibraryThing work ID. Concerning a book which occurs multiple times in the same post, we keep only the rst occurrence. Figure 5 shows an example of the second post of the thread 16512. For each post, we have the content of the post, the name of the user, the thread id as well as the date and time.
Figure 6 shows the result obtained for this post. The rst column corresponds to the thread id. The second column de nes the post id. The third column returns the unique LibraryThing work ID (what is shown in brackets is not present in the nal version of the results le.). - B:
After the classi cation process and the annotation process, we retrieve each book title and we compare it with the whole of the titles presents within the data set. This comparison is carried out by the Levenshtein distance set to the length of the shortest alignment between the sequences taken as factor. - B V2:
After the classi cation process and the annotation process, we retrieve each book title and we compare it with the whole of the titles presents within the data set. This comparison is carried out by the Levenshtein distance set to the length of the longer alignment between the sequences taken as factor. - BU:
For this run, we add a new feature to the CRF. This feature is to detail the punctuation marks. Once the classi cation process and the annotation process are done, we retrieve each book title and we compare it with the whole of the titles presents within the data set. This comparison is carried out by the Levenshtein distance set to the length of the shortest alignment between the sequences taken as factor. - BA V1
After the classi cation process and the annotation process, we retrieve each book title. If an author name is located at a maximum distance of four words, we aggregate it. Then, we compare the book title and the author name, if it is present, with the information present within the data set. This comparison is carried out by the Levenshtein distance set to the length of the shortest alignment between the sequences taken as factor. - BA V2:
After the classi cation process and the annotation process, we retrieve each book title. If an author name is located at a maximum distance of four words, we aggregate it. Then, we compare the book title and the author name, if it is present, with the information presents within the data set. This comparison is carried out by the Levenshtein distance set to the length of the longer alignment between the sequences taken as factor.
For evaluation, 217 threads in the test set were used, with 5097 book titles identi ed in 2117 posts. Table 3 shows 2016 o cial results for our 5 runs. Our best run is BA V2, it has classi ed the second w.r.t the measure Fscore and the rst w.r.t the measure precision the o cial evaluation measure for the workshop. The others runs have substantially similar results. However, we can see that the aggregation of author names increases performance. Compared to the best run 2016, the whole of our runs get a better precision. Several hypotheses may explain the lack of recall. Firstly, the classi cation process can occult posts containing references. Secondly, the amount of training data may not be enough to be representative of every possible case.
In this paper we presented our contribution for the INEX 2016 Social Book Search Track. In the 5 submitted runs, we tested several supervised approaches dedicated to book detection. Our results present better performance with the aggregation of author names. Moreover, the Levenshtein distance set to the length of the longer alignment between the sequences taken as factor give better results than the shortest alignment.