In this report we present our contribution to the 3rd Computational Linguistics Scienti c Document Summarization Shared Task (CL-SciSumm 2017), which poses the challenge of identifying the spans of text in a reference paper (RP) that most accurately re ect a citation (i.e. citance) from another document to the RP. In our approach, we address this challenge by applying techniques from the eld of information retrieval. Therefore we create a separate index for every RP and then transform each citance to a RP into a query. This query is subsequently used to retrieve the most relevant spans of text from the RP. Di erent ranking models and query generation strategies have been employed to alter which spans of text are retrieved from the index. Furthermore we implemented a k-nn classi cation based on our search infrastructure for assigning the cited text span to pre-de ned classes.
{ Task 1B: Classify each identi ed cited text span according to a prede ned set of facets. The elements of that set are: Implication, Method, Aim, Results, and Hypothesis. { Task 2 (optional bonus task): Generate a structured summary of the RP from the identi ed cited text spans of the RP, where the length of the summary must not exceed 250 words.
The data set provided for these tasks comprises a training set and a test set consisting of 30 and 10 RPs respectively. Each RP is associated with a set of citing papers (CP) which all contain citations to the RP. In each CP, the text spans (citances) have been identi ed that pertain to a particular citation to the RP.
To tackle the problem in Task 1A, we followed an information retrieval (IR) approach. For every RP, we created an index holding all the spans of text of that RP. A citance to a RP is transformed into a query and performed on the index associated with the RP to retrieve the most relevant spans of text.
For Task 1B, we followed a k-NN classi cation approach. Each identi ed cited text span is compared against all di erent cited text spans in the training set. Among the top ve most similar cited text spans a majority vote is used to determine the facet. 2
In this section we provide a closer look at our approach to Task 1A. We will describe how the indices to the RPs are created as well as how the citances are turned into queries and subsequently used to identify relevant spans of text in the RP.
In order to create an index to a RP, which holds all the di erent spans of text of the RP, we used the Apache Lucene text search engine library3 which features Java-based indexing and searching technology.
Taking advantage of the library's indexing technology, we created an index for every RP and added all spans of text of the RP to the index. In this scenario a single span of text can be imagined as a separate text document that is being added to a conventional index. Before we added anything to the index, however, we performed two additional preprocessing steps on every span of text. At rst, all stop words contained in a span of text were removed. The idea behind this is to tune the performance of the index (fewer terms in the index) and to obtain more relevant search results since stop words only carry little distinguishing potential [6].
To decide which words qualify as stop words and which do not, we used Apache Lucene's integrated list of stop words for the English language. As a 3 http://lucene.apache.org/ second preprocessing step, we stripped down all su xes of all words contained in a span of text in order to normalize them. This was achieved by applying Porter's stemming algorithm [3]. After the preprocessing on a span of text was completed, we moved on and added it to the index. 2.2
After all the indices for the RPs were in place, we transformed each citance to a RP into a query and ran it on the index associated with the RP in order to retrieve the most relevant spans of text of the RP for that particular citance. Since all indices were constructed with Apache Lucene, we also resorted to Apache Lucene functionality to generate the queries. There exists a broad range of di erent query types in Apache Lucene, however, after we conducted various experiments on the training set, we found that using Apache Lucene's TermQuery generated the best results.
To turn a citance into a query, we rst applied two preprocessing steps to the citance. The preprocessing steps applied, are analogous to the ones described in section 2, that is, stop words were removed from the citance and porter stemming was performed. As a next step we extracted all words from the citance and created a TermQuery for every word. This means, each TermQuery corresponds to a single word in the citance. After that, we created an Apache Lucene BooleanQuery by OR-conjuncting all TermQueries. This resulting BooleanQuery was then used to query the index associated with the RP that the citance refers to.
As a result of the query, we obtained a set of top ranked spans of text of the RP. The elements of that set are ordered according to a score, however, it depends on the ranking and retrieval model used by the index what elements are in the set and what score they are given. From all spans of text that are retrieved this way, we considered the top two as most accurately re ecting the corresponding citance. This is because considering the top two yielded the best results during experiments on the training set. 2.3
In section 2.2 we mentioned that it depends on the ranking and retrieval model that is used by the index which elements are retrieved by a query and how they are ranked. For the sake of the CL-SciSumm 2017 Shared Task, we submitted a system run using a simple vector space model (VSM) [5] based method and the popular BM25 model [4].
Vector Space Model The term frequency and inverse document frequency
(TF-IDF) weighting scheme used by Apache Lucene within the scope of the
VSM is as follows:
For the term frequency, which correlates to a term t in a document d, the formula
tft;d = pft;d
(
For the inverse document frequency, which correlates to the number of documents in which the term t appears, the formula is used, where N is the total number of documents in the index and nt is the number of documents containing the term t.
The score of a document d to a query q is calculated based on the cosine similarity and is de ned as idft = 1 + log
N nt + 1 sim(d; q) =
V (d) V (q) jV (d)jjV (q)j where V (d) V (q) is the dot product of the weighted vectors, and jV (d)j and jV (q)j are their euclidean norms.
BM25 The term frequency factors used in Apache Lucene within the scope of BM25 ranking are de ned as
Bt;d = k1 h(1 (k1 + 1)ft;d
jdj
b) + b ib2oostjdjavg
i
+ ft;d
(
The discourse facet takes one of the following values: Implication, Method, Aim, Results, and Hypothesis. To classify the spans of text, which were identi ed in Task 1A, we took the following approach: At rst we created an index which we lled with all available cited text spans of the training set plus their corresponding discourse facets. To classify which discourse facet a span of text belongs to, we then transformed the span of text into a query, analogous to the way described in 2.2, and then ran the query on the index. After that, we conducted a majority vote on the top ve retrieved results to determine the discourse facet to use. where ft;d denotes the number of times the term t occurs in document d, jdj is the length of the document d in words, jdjavg is the average document length, iboost is an index-time boosting factor, and k1 and b are parameters.
The ranking equation used in the BM25 model can then be written as sim(d; q) = X
Bt;d
log t[q;d]
N
nt + 0:5 nt + 0:5
:
Overall we submitted two system runs for Task 1. One of the system runs was conducted using the BM25 ranking model and the other using a simple vector space model (VSM). See section 2.3 for the parameters that we used for these models.
The system performance for Task 1a was determined by measuring the sentence id overlaps between the sentences identi ed by the system and the gold standard sentences created by human annotators. Based on that, precision, recall and F1 score were calculated for each system run.
The performance of Task 1b was measured by the proportion of the correctly classi ed discourse facets by the system, contingent on the expected response of Task 1a. The metrics used here are also precision, recall and F1 score.
The o cial evaluation results of our submitted system runs for Task 1a and Task 1b are shown in Table 1 and Table 2 respectively:
Judging by the o cial evaluation results we nd that our proposed approaches yield excellent results. Especially our BM25 approach seems to work very well for both Task 1a and Task 1b: An F1 score of 0:135 at Task 1a achieves the third highest result among all system runs, not far behind the winning system, which has an F1 score of 0:146. An F1 score of 0:337 at Task 1b is the eighth highest result among all system runs, with the winning system having an F1 score of 0:408. Overall 47 system runs have been submitted to Task 1. The mean F1 score at Task 1a among all system runs is 0:088 and at Task 1b 0:208. 5
In this report we described the approaches we followed to tackle the problems posed in Task 1A and Task 1B of the CL-SciSumm 2017 Shared Task. In preliminary test we found out that using a combination of stop word removal and stemming in combination with a disjunction query strategy work best. The ofcial evaluation results con rmed the success of our approach. We were able to reuse the indexing infrastructure for a classi cation task, namely assigning categories to the cited text spans. In future work we plan to make use of our infrastructure and investigate methods to enhance the process by integrating more sources of evidence. In particular, additional context information like author or venue speci c information might prove bene cial.
The Know-Center is funded within the Austrian COMET Program { Competence Centers for Excellent Technologies { under the auspices of the Austrian Federal Ministry of Transport, Innovation and Technology, the Austrian Federal Ministry of Economy, Family and Youth and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency FFG.