This paper describes the University of Houston team's e orts toward the problem of identifying reference spans in a reference document given sentences from other documents that cite the reference document. We investigated the following approaches: cosine similarity with multiple incremental modi cations and SVMs with a tree kernel. Although the best performing approach in our experiments is quite simple, it is not the best under every metric used for comparison. We also present a brief analysis of the dataset which includes information on its sparsity and frequency of section titles.
The CL-SciSumm 2016 shared task poses the problem of automatic summarization in the Computational Linguistics (CL) domain. Single text summarization is hardly new, however, in addition to the reference text to be summarized we are also given citances i.e. sentences that cite our reference text.
The shared task is broken into multiple tasks with the unifying theme of leveraging citances. Task 1a is, given a citance, to identify the span of reference text that best re ects what has been cited. Task 1b asks us to classify the cited aspect according to a prede ned set of facets: hypothesis, aim, method, results, and implication. Finally, Task 2 is generating a structured summary.
We experimented with the following approaches: SVMs with a tree kernel and cosine similarity based on TF/IDF weights for sentences. The best results when measuring by sentence inclusion are obtained by cosine similarity. However, ROUGE-L scores are better for the tree kernel approach.
We also study two characteristics of the dataset: sparsity and section
importance. We de ne section importance as the normalized frequency
of the section, i.e., the number of correct reference sentences that belong
to this section across all the citances. We found that the introduction is
the most cited section in this year's dataset. We also nd that citances
are less sparse than the average sentence within the corpus.
1.0
0.8
s0.6
d
r
o
W
%0.4
The dataset [
In Figure 1 we can see how sparse the dataset is; the quicker the decay, the greater the sparsity. Noise in the dataset is one of the factors for the sparsity. We can see that citances, seen as a corpus, are in general less sparse than the reference texts. This can be an indication that citances have some common structure or semantics. 2.2
For each cited reference sentence, we looked at the title of the section in which it appears. The titles that appeared with greatest frequency can be seen in Table 1. To extract these section titles we looked at the parent nodes of sentences within the XML document. The \title" and \abstract" sections are special since they refer to parent nodes of type other than SECTION. These results clearly show sentences that are cited are not uniformly distributed within a document.
Title introduction abstract title conclusion. the approach. potential for improvement. evaluation.
Frequency Relevant 25.55% 6.98% 5.58% 4.46% 4.05% 3.35% 3.21% In this task we are asked to identify the reference sentences referred to by a given citance. We approach the problem from two di erent perspectives. One of our systems follows the intuitive idea that the citance and the sentences to which it refers must share some similarity. Thus, we modify this system with the intent to capture more forms of similarity. Our second system abstracts further, instead of looking at the similarity between citance and reference sentences we look at the similarity between citance and reference sentence pairs, (c; r). We attempt to learn how to distinguish relevant and irrelevant pairs. We modify this system with the intent to improve the learned classi er. Context Expansion. (CE) Citances by themselves only have a limited amount of semantic information. We expand the semantic content of a citance by including the sentences that occur directly before and after the citance when constructing its word vector.
WordNet Expansion. (WN) Another method for semantic
expansion is to include the lemmas obtained from the synsets of each word
in a sentence. This applies to both citances and sentences in the
reference document. We use the Lesk algorithm, provided through the NLTK
package [
We classify the (c; r) pairs of citance and reference sentence as relevant or irrelevant. We then rank the sentences according to how con dent we are it is relevant.
SVM. The Support Vector Machine (SVM) is a classi er that learns a
hyperplane to separate two classes [
After training, an SVM will have a set of support vectors that de ne its hyperplane. The hyperplane can be thought of as a description of the boundary between classes. The output is the signed distance from the instance to this hyperplane: a positive distance implies the item belongs to the positive class; a negative distance implies the item is in the negative class.
Learning can be carried out with just the Gram Matrix, a matrix of the \distances" between pairs of data items. This is known as the kernel trick. By choosing a di erent kernel we implicitly transform the space where items reside. This broadens the utility of SVMs since the new space may allow for simpler separation between classes.
Subset Tree Kernel. Our kernel of choice is a convolutional kernel
on trees. Although there are a few di erent formulations, we use the
subset tree kernel which compares the number of common subset trees
(subtrees that can have nonterminals as leaves) [
Sentence Limiting. (SL) Similar to how we eliminated sentences from consideration in the TF/IDF approach, we do the same here. Since we randomly select negative items, it becomes of greater concern that we perform multiple runs. To get a su cient number of runs we use a narrower range to reduce classi cation time; sentences must have more than 15 and less than 35 words. This process eliminates 3462 sentences out of 6706 total, roughly 51%. Out of the 390 relevant sentences in the test set, only 215 meet the criteria.
Implementation. We utilize SVM-LIGHT-TK [
Task 2 consists in generating a summary for the reference text. For Task 2 we average the sentence scores given by each citance and extract sentences until we reach the 250 word limit. This was only performed for the Tree kernel approach and only against human summaries. Furthermore, we only evaluated the best performing run of each con guration (according to its performance in Task 1a). 5
The evaluation of our systems is done by comparison of several metrics which are detailed below. For the Tree kernel method, these values are averaged over 5 runs.
ROUGE-L. The ROUGE metrics are useful for evaluating summaries.
In particular we look at the ROUGE-L metric [
Top-3 Metrics. Our systems output the top 3 sentences, thus we compute recall, precision, and F1 score for these sentences. If a relevant sentence appears in the top-3, then it factors into recall, precision, and F1 score. Note that due to sentence limiting (SL) we impose a limit on the F1 score attainable. Since we always return the top 3 sentences, for the 279 citances of the test set we return 837 total sentences. We can calculate the maximum attainable F1 score. For the Tree kernel approach it is 35:04%. For the TF/IDF aproach it varies between 44:98% and 58:02%. Mean Average Rank. We compute the average rank by obtaining the rank of all the relevant reference sentences of each citance. These are normalized according to the total number of sentences being considered. The normalized rank is a value within the interval [0; 1]. We average these ranks among the citances for a document. Finally we nd the mean of these averages for all documents. For example, for a single reference text with two citances, each referring to a single sentence, we would average the normalized rank of these two sentences according to their appropriate citance. We then nd the mean among the documents processed. Lower is better.
TKern(1:1)+SL TKern(1:4)+SL TKern(1:8)+SL TKern(1:1)+SL+CE TKern(1:4)+SL+CE TKern(1:8)+SL+CE ROUGE-L 58.78% 57.90% 57.76% 58.84% 58.12% 57.87%
TFIDF+ST TFIDF+SL TFIDF+WN TFIDF+ST+SL TFIDF+ST+WN TFIDF+SL+WN TFIDF+ST+SL+WN
TKern(1:1)+SL 5.63% 12.10% 7.69% 0.262 (215/390) TKern(1:4)+SL 4.85% 10.41% 6.61% 0.263 (215/390) TKern(1:8)+SL 5.13% 11.02% 7.00% 0.245 (215/390) TKern(1:1)+SL+CE 5.71% 12.25% 7.79% 0.261 (215/390) TKern(1:4)+SL+CE 4.70% 10.10% 6.42% 0.262 (215/390) TKern(1:8)+SL+CE 5.11% 10.97% 6.97% 0.252 (215/390) TFIDF 7.88% 16.92% 10.75% 0.106 (298/390) TFIDF+ST 8.60% 18.46% 11.73% 0.124 (329/390) TFIDF+SL 8.72% 18.71% 11.89% 0.093 (276/390) TFIDF+WN 4.77% 10.25% 6.51% 0.154 (331/390) TFIDF+ST+SL 8.96% 19.23% 12.22% 0.112 (309/390) TFIDF+ST+WN 5.61% 12.05% 7.66% 0.164 (336/390) TFIDF+SL+WN 6.09% 13.07% 8.31% 0.137 (307/390) TFIDF+ST+SL+WN 5.61% 12.05% 7.66% 0.144 (286/390) TFIDF+CE 7.04% 15.12% 9.61% 0.159 (330/390) TFIDF+ST+CE 7.04% 15.12% 9.61% 0.167 (349/390) TFIDF+SL+CE 7.88% 16.92% 10.75% 0.137 (303/390) TFIDF+WN+CE 4.65% 10.00% 6.35% 0.216 (354/390) TFIDF+ST+SL+CE 7.76% 16.66% 10.59% 0.149 (325/390) TFIDF+ST+WN+CE 4.89% 10.51% 6.68% 0.220 (356/390) TFIDF+SL+WN+CE 5.25% 11.28% 7.17% 0.183 (326/390) TFIDF+ST+SL+WN+CE 4.77% 10.25% 6.51% 0.189 (304/390) 6 6.1
The TF/IDF approach is unexpectedly our best performing approach. Although some modi cations hurt performance, upon closer inspection we see how they might improve our results in other ways. It is, however, surprising that the minimally modi ed TFIDF+SL+ST approach has the highest F1 score. Furthermore, among the TF/IDF con gurations, the unmodi ed TF/IDF approach has the highest ROUGE-L F1 score. One of the considerations when using cosine similarity is whether or not we get any value at all due to sparsity. As we can see from Table 4, a portion of relevant sentences are indistinguishable since they have zero similarity. WordNet expansion, stemming, and context expansion provide a signi cant increase in non-zero similarities among relevant sentences. However, recall and precision decrease in turn. Sentence limits have the opposite e ect: they increase recall and precision but also decrease the number of relevant sentences we can distinguish. 6.2
For the Tree kernel approach, we expected the inclusion of more negative items to increase our recall and precision. However, the con guration with 1 : 1 ratio had the best performance. It is signi cant that this dominance occurs not only with regards to the ROUGE-L scores in Table 2 but also in terms of top-3 metrics in Table 4.
Average rank did improve with a greater number of negative items. We conjecture these negative items in training had the e ect of lowering the rank of negative items in testing, thus improving the rank of positive items. This overall improvement came at the expense of recall and precision.
The performance variation between runs was calculated for each system as the maximum F1 at top-3 score attained minus the minimum F1 at top-3 score attained in these 5 runs. The system with the largest variation was Tkern(1:1)+SL, with 2.7% di erence, whereas the system with the least variation was TKern(1:1)+SL+CE, with 0.9% di erence. The output had a curious behavior. For each reference text, the sentences chosen by the system were the same regardless of citance. Even after context expansion, the behavior persisted. Unfortunately, this did not translate into high ROUGE-L scores for Task 2. It is possible that tuning the method so this behavior does not occur would increase its performance.
Finally, it is interesting to note that the ROUGE-L scores for the Tree kernel approach were consistently higher than the scores for the TF/IDF approach. 7
We investigated the e ects of various modi cations on the performance of a simple TF/IDF approach. In addition, an SVM with a tree kernel was also employed with mixed results. Although the traditional F1 score points to one of the simplest approaches as the most e ective, it is important to remember the two approaches tackled the same problem from di erent perspectives. Whether these two perspectives { similarity between citance and reference sentence and the similarity between (c; r) pairs { complement each other is worth exploring.
The characterization of citances also warrants further research. The peculiarities apparent (and those less apparent) could lead to improvements in these tasks.