This report describes our contribution to the 2nd Computational Linguistics Scienti c Document Summarization Shared Task (CLSciSumm 2016), which asked to identify the relevant text span in a reference paper that corresponds to a citation in another document that cites this paper. We developed three di erent approaches based on summarisation and classi cation techniques. First, we applied a modi ed version of an unsupervised summarisation technique, TextSentenceRank, to the reference document, which incorporates the similarity of sentences to the citation on a textual level. Second, we employed classi cation to select from candidates previously extracted through the original TextSentenceRank algorithm. Third, we used unsupervised summarisation of the relevant sub-part of the document that was previously selected in a supervised manner.
Extractive summarisation of a textual document is the process of nding a
representative subset of the document text that captures as much information about
the original document as possible. A promising idea in the realm of scienti c
publications is to consider the set of sentences that cite a paper as a summary
created by the research community. Here we describe our contribution to the
2nd Computational Linguistics Scienti c Document Summarization Shared Task
(CL-SciSumm 2016)1 [
Stefan Klamp , Andi Rexha, and Roman Kern
CL Pilot Task that has been conducted as a part of the BiomedSumm Track at
the Text Analysis Conference 2014 (TAC 2014)3 [
The dataset provided for this year's task consists of a set of reference papers (RP), each of which is accompanied by a set of citing papers (CP). The goal is to identify the text span in the RP which corresponds to the citations in CP (Task 1A) as well as the discourse facet of the RP this text span belongs to (Task 1B). Human annotators have created the ground truth in the form of pairs of citation text and cited text on the granularity level of sentences.
For Task 1A, we implemented a number of approaches that employ both unsupervised and supervised techniques that di er in the way how information from the citing sentence in the CP is incorporated into the process. In total, we submitted three runs, corresponding to our three approaches for Task 1A: (i) modi ed-tsr, (ii) tsr-sent-class, and (iii) sect-class-tsr.
First, in a completely unsupervised setting, we applied a modi ed variant of
our TextSentenceRank algorithm to the RP (modi ed-tsr ). TextSentenceRank
[
As a second approach for Task 1A, we employed a supervised classi cation setting following an unsupervised preprocessing (tsr-sent-class ). Through the original version of TextSentenceRank we pre-selected candidate sentences from the RP, independently from the CP, that are potential text spans for being cited. For a given citation in the CP, we then selected the corresponding candidate through supervised classi cation.
In our third option, we took a dual approach (sect-class-tsr ). We rst used supervised learning to identify the relevant sub-part (section) of the RP that corresponds to the citing sentence in the CP. Once the relevant section has been found, we used the original version of TextSentenceRank on this sub-part, independent from the CP, to identify referenced text spans.
For Task 1B, we used a similar classi er as for identi cation of the section. We used features derived from the citing sentence as well es from the extracted text span in the reference document to determine the discourse facet. This is applied in all three approaches to Task 1A.
In principle, TextSentenceRank is able to extract multiple candidate text spans scattered across the document, but since the task description required the extraction of consecutive referenced text, we decided to output a single sentence as the extracted reference span in all of our approaches.
This report is structured as follows. In sections 2 and 3 we explain our approaches for both Task 1A and Task 1B in detail. In section 4 we individually evaluate the classi ers that we used in our approaches and present our results for the overall task. In the end, in section 5 we conclude, discuss our ndings, and give an outlook to potential future work. 3 http://www.nist.gov/tac/2014/BiomedSumm/
In this subsection we detail our three approaches for Task 1A, the identi cation of referenced text spans. We also introduce TextSentenceRank as a base method which is used in all three runs. 2.1
This approach is inspired by graph based ranking algorithms, such as Google's
PageRank [
The score s(vi) of a vertex vi is given by s(vi) = (1 d) + d X j2I(vi)
wij
P
vk2O(vj) wjk
s(vj );
(1)
where d is a parameter accounting for latent transitions between non-adjacent
vertices (set to 0.85 as in [
The TextSentenceRank algorithm is an extension to the original TextRank
algorithm [
When used for key sentence extraction, TextSentenceRank extracts the \most relevant" sentences in terms of how they are connected to other sentences in the document via co-occurring words. Here we pursue our intuition that such relevant sentences are also more likely to be cited and use TextSentenceRank as a base algorithm for extracting candidates for referenced text. 2.2
Run 1 (modi ed-tsr ) follows a completely unsupervised setting, where we applied a modi ed variant of our TextSentenceRank algorithm to the RP. The idea here 4 Here, we restrict the set of relevant tokens to adjectives, nouns, and proper nouns.
This information is obtained through part-of-speech tagging. 4
Stefan Klamp , Andi Rexha, and Roman Kern was to use a speci c weighting of the underlying graph that takes into account the information provided by the CP, in particular, the citing sentence. More precisely, the weight of an edge adjacent to a node corresponding to sentence S is modi ed by wnew = wold [1 + sim(S; C)] ; (2) where sim(S; C) is a similarity measure between sentence S of the RP and citing sentence C of the CP.
In our approach we used the Jaccard similarity [
In Run 2 (tsr-sent-class ) we employed a supervised classi cation setting following an unsupervised preprocessing. First, we pre-selected candidate sentences from the RP through the original version of TextSentenceRank. This selection is thus independent from the CP and consists of potential text spans for being cited. We then selected the corresponding candidate through supervised classi cation that takes into account the information from the citing sentence in the CP.
We used a Random Forest classi er [
In Run 3 (sect-class-tsr ), we took a dual approach to Run 2. Instead of applying an unsupervised preprocessing followed by a supervised classi cation, we rst used supervised learning to identify the relevant sub-part of the RP that corresponds to the citing sentence in the CP. This sub-part can in principle be of any granularity, but as sections are annotated in the provided dataset, we chose Identifying Referenced Text by Summarisation and Classi cation 5 the granularity of sections. Once the relevant section has been found, we used the original version of TextSentenceRank on this sub-part, independent from the CP, to identify referenced text spans.
Again, we used a Random Forest classi er, now with these features:
{ Section features: title and number of the section enclosing the candidate
sentence,
{ Tf-Idf features: information about the frequency of tokens from the citing
sentence within the section, normalized by the inverse frequency across all
the sections of the RP. This feature is motivated by the standard Tf-Idf
measure [
This classi er is a binary classi er that decides for each candidate section in the RP whether it is a section containing a referenced text span, based on information from the citing sentence. The cited text span is then selected through the original version of TextSentenceRank, applied to the sub-document spanning the selected section, independently from the CP. 3
The discourse facet takes the following values in the training set: Implication, Method, Aim, Results, and Hypothesis. We used a Random Forest classi er with section features, sentence position features, and discriminative term features to distinguish between these classes. That is, we took into account information from both the citing sentence as well es from the extracted text span in the reference document to determine the discourse facet. The same model, which was previously trained on the training set, was applied in all three runs. 4
In our evaluation, we rst determine the isolated performance of individual components that we used in our approaches. Then we present our results for the overall task. We performed these evaluations on both the provided development set and the training set, as the results on the test corpus have not yet been made available. 4.1
In our contribution we used three di erent classi ers, which we evaluate separately in this section. For the full system runs we trained the classi er on the training set, and submitted the results produced by applying the trained classiers on the test set. Here, we evaluate the classi ers on both the development set and on the training set using 10-fold cross validation.
In Run 2 of Task 1A (tsr-sent-class ) we used a binary classi er that decides for each candidate sentence whether it serves as a referenced text span for a given 6
Stefan Klamp , Andi Rexha, and Roman Kern citing sentence. Table 1 shows the cross-validation performance of this sentence classi er on both the development set and on the training set. Since it is a binary classi er, we report here the precision, recall, and F1 values with respect to the positive class, i.e., whether the sentence is classi ed as a referenced text span. A reasonable precision is achieved, which means that there are relatively few false positives, however, at the expense of low recall, indicating that many true positives are missed. Even though we pre- ltered the negative instances with TextSentenceRank, the classi cation problem is still quite unbalanced: There are about 10 times as many negative as positive examples. This unbalance might induce a certain bias in the classi er; still, the accuracy, i.e., the fraction of correctly classi ed instances, across both classes is above 90% for both datasets.
In Run 3 of Task 1A (sect-class-tsr ) we employed a binary classi er that decides for each section whether it contains a referenced text span corresponding to a given citing sentence. Table 2 shows the cross-validation performance of this section classi er on both the development set and on the training set. Again, we report here the precision, recall, and F1 values with respect to the positive class, i.e., whether the section is classi ed as containing a referenced text span. It can be seen that the performance is quite lower than for the sentence classi er, but the accuracy is still above 80%. Here the unbalance between positive and negative classes is given by the number of sections in the reference paper (on average about 5 to 6 in the training set).
In Task 1B, for the identi cation of the discourse facet, we used a multi-label classi er to categorise a referenced text span into one of the following classes: Identifying Referenced Text by Summarisation and Classi cation 7 Implication, Method, Aim, Results, and Hypothesis. Table 3 shows the crossvalidation performance of this discourse facet classi er on both the development set and on the training set. Precision and recall are given as micro-averages over all labels and lie between 60% and 70%. Classi cation accuracy is around 70% for both datasets. Table 4 shows the confusion matrix obtained by the classi er on the training set. Method is by far the most occurring label in the datasets, and the classi er might have a certain bias of generating this label, but also the quality of retrieving the label Results is reasonable. Hypothesis is the rarest label and the one with the lowest accuracy. 4.2
We evaluated all of our three approaches for Task 1A on both the development set and the training set. We compared the extracted reference spans in our system output with the corresponding reference spans provided by the human annotators in terms of overlap and distance. In Table 5 we show for each of the three runs and for each topic in the development set the number of citances for which the extracted reference span lies within 10 sentences of the true reference span and for which both reference spans actually overlap. Table 6 shows the same information for the training set. 8
Stefan Klamp , Andi Rexha, and Roman Kern
It can be seen that only in few cases a referenced text span is extracted that overlaps the text segment identi ed by the human annotator. If we allow a certain neighborhood around the true spans, considerably more matches are found. Interestingly, Run 1, the modi ed TextSentenceRank, achieves the best results, followed by Run 3, the variant with section classi cation, which slightly outperforms Run 2, the version with sentence classi cation.
The nding that the modi ed TextSentenceRank works best suggests that considering the document as a whole might be bene cial for extracting relevant key sentences. The low performance of the classi cation approaches might be due to a lack of representative features that are relevant for the task at hand. In Run 2 it is possible that the set of sentences provided by the original TextSentenceRank algorithm is already too limited before the sentence classi er can select suitable text spans. The dual approach of Run 3, rst selecting a sub-part and then applying TextSentenceRank, works slightly better.
These results also demonstrate the di culty of this task. It is worth mentioning that all our approaches are solely based on statistics of words and sentences in both the reference document and the citing sentence as well as in their comparison. Currently, we do not incorporate any semantic information, but in principle our approaches can be easily adapted through additional features for classi cation and a di erent weighting strategy for TextSentenceRank. 5
In this report we have described our contribution to the 2nd Computational Linguistics Scienti c Document Summarization Shared Task (CL-SciSumm 2016), Identifying Referenced Text by Summarisation and Classi cation 9 which asked participants to identify the relevant text span in a reference paper that corresponds to a citation in another document that cites this paper. We developed three di erent approaches based on summarisation and classi cation techniques. They employ both unsupervised and supervised techniques that differ in the way how information from the citing sentence is incorporated into the process. First, we applied a modi ed version of an unsupervised summarisation technique, TextSentenceRank, to the reference document, which incorporates the similarity of sentences to the citation on a textual level. Second, we employed classi cation to select from candidates previously extracted through the original TextSentenceRank algorithm. Third, we used unsupervised summarisation of the relevant sub-part of the document that was previously selected in a supervised manner.
We evaluated both the individual classi ers used in our approaches as well as the performance in the overall task. We believe that the performance of our systems could be improved by incorporating di erent similarity measures, e.g., measures capturing the semantic similarity of citing and cited sentences, not only in the modi ed weighting of TextSentenceRank, but also into the set of features used for classi cation. Furthermore, the relative strength of the in uence of the citing sentence can be optimised. For example, in the modi ed TextSentenceRank algorithm there could be a trade-o parameter in equation 2 that weights the relative in uences of wold and sim(S; C). Finally, the inclusion of multiple, non-consecutive sentences in the output would likely include candidate text spans of better quality.
Another aspect that likely in uences our system is the fact that the text of the given documents was extracted with OCR methods. These methods some
Stefan Klamp , Andi Rexha, and Roman Kern times yield noisy and erroneous words, and many of our methods rely on the statistics of terms within and across documents. In our experience, in some cases TextSentenceRank seems to prefer sentences containing such noisy or invalid tokens.
As a contribution to the CL-SciSumm 2016 task our work aimed at facilitating the summarisation of scienti c publications. In addition, we hope that our contribution will also provide further insight into the scienti c writing habits of researchers, both in terms of how they structure their papers and how they reference the work of others.
The Know-Center is funded within the Austrian COMET Program { Competence Centers for Excellent Technologies { under the auspices of the Austrian Federal Ministry of Transport, Innovation and Technology, the Austrian Federal Ministry of Economy, Family and Youth and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency FFG.