In recent years there has been an increasing interest in approaches to scienti c summarization that take advantage of the citations a research paper has received in order to extract its main contributions. In this context, the CL-SciSumm 2017 Shared Task has been proposed to address citation-based information extraction and summarization. In this paper we present several systems to address three of the CL-SciSumm tasks. Notably, unsupervised systems to match citing and cited sentences (Task 1A), a supervised approach to identify the type of information being cited (Task 1B), and a supervised citation-based summarizer (Task 2).
Although scienti c summarization has always been an important research topic
in the area of natural language processing (NLP) [
In this paper we report the systems developed at LaSTUS/TALN to
participate in CL-SciSumm 2017. They include four unsupervised systems based on
sentence similarity for Task 1A, one supervised approach for Task 1B, and one
supervised approach for Task 2. The systems for Tasks 1A and 1B follow the
approach reported in [
The organizers of the CL-SciSumm 2017 challenge provided training data structured in clusters of reference and citing papers together with manual annotations indicating, for each citance, the text span(s) in the reference paper that best represent the citance, as well as their corresponding facets. The training corpus contains 30 clusters with an average of 17 papers per cluster. For each cluster there are three manually created summaries of the reference paper: the author abstract, a community-based abstract created using citation sentences, and a human abstract created based on information from reference paper and citation sentences. The test set has 10 clusters with 11 documents each, on average.
In order to automatically process the clusters, we created, from the
documents in the training and test sets, GATE [
Based on these annotations we could easily build pairs of matching sentences (Citing Paper Sentence, Reference Paper Sentence) and associate, to each pair, the facet that the annotator considered the citation referred to (see Task 1B). 2.1
The tokenizer, sentence splitter, part-of-speech tagger, and lemmatizer available in GATE's ANNIE3 component were used to initially process the documents,
which were then further enriched with annotations obtained by means of the Dr. Inventor (DRI) Text Mining Framework [20]. In particular, we obtained a probability score for each sentence of having a particular rhetorical function in the paper's discourse, among: Approach, Background, Challenge, Outcome and FutureWork. The SUMMA library [21] was used to obtain normalized tf*idf term vectors for the sentences. For each sentence in the reference paper we computed and annotated its similarity to the sentences in the corresponding citances by means of their cosine distance. For this, we used various vectorial representations of the sentences, including SUMMA's tf*idf vectors and pretrained word embeddings. 3
In this section we present the experiments aimed at detecting which sentence(s) in the reference papers most accurately re ect each given citance. 3.1
For the rst set of experiments we used the Google News embeddings4 (three
million words in 300 dimensional vectors trained using word2vec [
We built embeddings for each citance in the citing papers by taking the centroid of the embeddings of all the word contained in it. The same procedure was used to build embeddings for each of the sentences of the reference paper. We calculated the cosine distance for each pair of embeddings: one obtained from each sentence in a reference paper and one corresponding to a citance that refers to it. We experimented with di erent combinations of embeddings: using only Google News vectors, using only ACL vectors (100 or 300 dimensions) and using the concatenation of Google News and ACL vectors (400 or 600 dimensions). We run several test considering as candidates the top two, ve, eight and ten sentences from the reference papers most similar to the given citances. Since the evaluation used for Task 1B uses the F score, we aimed at optimizing this metric. We did this by saving 10 clusters from the training data for validation and using 20 clusters for training. The best performance for the validation set was achieved
using the concatenation of Google and ACL-300 vectors and considering the two top candidate sentences from the reference papers. 3.2
We used a modi ed version of the Jaccard similarity index that takes into consideration the inverted frequency of the word in a corpus instead of just the word occurrences. For this experiment we calculated idf values of word stems using both the training set and an ACL reference corpus of around 4,000 documents. The modi ed Jaccard similarity between two text spans s1 and s2 is de ned in equation 1. Our modi cation assigns greater weight to matching word stems that are infrequent in the corpus, based on the idea that two text spans that share infrequent words are more likely to be semantically related.
M J (s1; s2) =
Pt2s1\s2 2idf(t) js1 [ s2j (1) 3.3
BabelNet [
Using the Babelfy API,5 we obtained the list of BabelNet synsets associated to each sentence of the corpus and used them to build sentence embeddings analogously as we did with the word embeddings. The BabelNet embeddings include many vectors for each synset (one for each lexicalization). We therefore calculated the centroid of all the vectors associated to each synset to generate its embedding. For this experiment we proceeded analogously to the word embeddings experiment described above: we calculated embeddings for the citances and for the sentences in the reference papers and then selected as candidates the top N sentences according to their cosine distance to the citances. Based on the tests against the validation corpus, the best results were again achieved considering the two sentences from the reference papers most similar to the citances. 3.4
We propose a system that leverages the best results obtained by the word embeddings, Modi ed Jaccard and BabelNet embeddings systems: the top ve candidates obtained for each of the systems are rst considered and then a voting process chooses as candidates all the sentences that were selected by at least two systems. If no sentence was chosen by at least two, only the top sentence selected by the Modi ed Jaccard system6 is returned. Unlike the other systems described
6 Modi ed Jaccard was chosen as default as it was the system for which the best
F-measures were obtained when run independently. for this task{where a xed number of candidate sentences are returned{in this case the number of sentences obtained is variable.
Table 1 shows the performance of the results over the validation data. The experiments are word embeddings (WE), Modi ed Jaccard (MJ), BabelNet embeddings (BN) and the voting scheme (Voting). The best results over the validation corpus are achieved by the voting system. In this section we present experiments aimed at identifying the facets the cited text spans belong to. We modeled pairs of reference and citance sentences as feature vectors, which we then used to train classi cation algorithms that determine whether a cited text span belongs to one of the prede ned facets. In the next section we describe the set of features generated. For the classi cation algorithms we relied on implementations included in the Weka machine learning framework [27]. Sentence Position Features: The sentence position in a paper can inform about the facet the sentence belongs to. For instance, sentences at the end of the document would probably belong to the Result facet. We use three features based on the location of the sentence in the reference document: { Sentence position: the position of the sentence in the reference paper; { Section sentence position: the position of the sentence in the section; { Facet position: ve binary features indicating whether the sentence is in a section whose title is associated to one of the target facets (e.g., the word \method" would indicate a section dealing with the facet Method). Text Similarity Features: The more similar a text is to another the more likely it is that they will be part of the same facet. We used two di erent tf*idf vector representations of the sentences produced by the SUMMA library{one based on word lemmas and one on BabelNet synsets{and computed their cosine similarity. We also calculated the Jaccard and Modi ed Jaccard coe cients for the lemmas, generating a total of four text similarity features.
that the DRI Framework predicts the probability of a sentence being in one of ve possible rhetorical categories. Even if they are di erent from our targeted discourse facets, we believe that these probabilities could be informative for our classi cation tasks and therefore also included them as features. Dr Inventor Sentence Related Features: Other features obtained by means of the DRI Framework that we believed could be of use in predicting a sentence belonging to a particular facet include: { Citation marker: three features to represent the number of citation markers in the reference sentence, citing sentence and the pair of sentences together; { Cause and e ect: two features to represent if the reference or citing sentence participates in one or more causal relations; { Co-reference chains: three features to represent the number of nominals and pro-nominals chained in the reference sentence, citing sentence and the pair of sentences together.
Scienti c Gazetteer Features: We generated a set of features based on Teufel's action and concept lexicon. The lexicon contains 58 lists. Each one is used to produce a feature which is the ratio of words in the sentence matching the list to the number of words in the sentence. The features are computed for the reference sentence, the citing sentence, and their combination, giving rise to 174 features. Bag-of-word Features: four string features are produced to represent the bigram lemmas, POS-tags bi-gram, lemmas and POS-tags for the combination of the reference and the citing sentences.
Based on these features we trained classi ers with 1,386 instances distributed as follows: Aim (134), Implication (150), Result (262), Hypothesis (32), Method (808). Considering the skewed distribution of the Method facet, we decided to train two models: one binary classi er to predict whether the instance is a Method or not and a multi-class classi er to identify one of the other facets in case it was previously classi ed as not-Method. We evaluated the performance of several classi cation algorithms including: SMO algorithm for support vector machines (SMO), naive Bayes (NB), K-nearest neighbors (IBk), random committee (RC), logistic regression (LR) and random forest (RF). We performed 10-fold cross validation experiments with the training data in order to decide which algorithm to use. The best results were obtained with the RF algorithm for the binary Method classi er and the SMO for the multi-class classi er representing the non-Method facet (Table 2).
The proposed summarizer is a modi ed version of our 2016 summarization
system [22] with additional features. The approach is based on a patent
summarization method [
{ Each token (i.e., lemma) is weighted by its tf*idf, where idf values are computed from training data; { For each sentence a vector of terms and normalized weights is created using the previously computed weights (SUMMA vectors); { Using the ACL word embeddings, a vector is created for each sentence in the document{average of the word embeddings of the words in the sentence (ACL vectors); { Using the Google news word embeddings, a vector is created for each sentence in the document{average of the word embeddings of the words in the sentence (Google vectors); { Using the sentence vectors (SUMMA, ACL, Google), three centroids are created for the document{each an average of the sentence vectors in the whole document; { Using the sentence vectors (SUMMA, ACL, Google), three centroids are created for the abstract of the document{each an average of the sentence vectors in the abstract; { In the citing papers, token frequency and SUMMA, ACL, Google vectors are also computed.
The features to train the linear regression algorithm are described below. Text
similarity features are the result of comparing two vectors of the same type (e.g.,
SUMMA, ACL, or Google) using the cosine similarity function implemented in
SUMMA. Therefore three di erent feature values are always generated. The
reference paper features are as follows:
{ Sentence Abstract Similarity Scores: the similarity of a sentence vector to
the author abstract vectors (for SUMMA, ACL, Google vectors);
{ Sentence Centroid Similarity Scores: the similarity (SUMMA, ACL, Google)
of a sentence to the document centroid;
{ First Sentence Similarity Scores: the similarity (SUMMA, ACL, Google) of
a sentence to the vector of the rst sentence (e.g., the title of the reference
paper);
{ Position Score: the SUMMA implementation of the position method where
sentences at the beginning of the document have high scores and sentence
at the end of the document have low scores;
{ Position in Section Score: a score representing the position of the sentence
in the section of the document. Sentences in rst section get higher scores,
sentences in last section get low scores;
{ Sentence Position in Section Score: a position method applied to sentences
in each section of the document (sentence at the beginning of the section get
higher scores and sentences at the end of the section get lower scores);
{ Normalised Cue-phrase Score: the total number of cue-words in the sentence
divided by the total number of cue-words in the document. We have relied
on [25] formulaic expressions to implement our cue-phrase gazetteer lookup
procedure;
{ TextRank Normalized Scores: the SUMMA implementation of the TextRank
algorithm [
The citing paper features are as follows: { Citing Paper Maximum Similarity Scores: each reference paper sentence vector is compared to each citance vector in each citing paper to get the maximum possible cosine similarity (for SUMMA, ACL, and Google vectors); { Citing Paper Minimum Similarity Scores: each reference paper sentence vector is compared to each citance vector in each citing paper to get the minimum possible cosine similarity (for SUMMA, ACL, and Google vectors); { Citing Paper Average Similarity Scores: each reference paper sentence vector is compared to each citance vector and the average cosine value obtained (for SUMMA, ACL, and Google vectors); The approach taken to rank sentences is to produce a cumulative value as the weighted sum of features f1; :::fn using the following formula: score(S) = n X wi fi i=0 (2) with S the sentence to score, fi the value of feature i and wi the weight assigned to feature i. The SUMMA system is used to score the sentences once the model has been created.
As we stated before, the weights for each feature are learned from training
data and although the ideal score to be learn is in principle unknown, we
approximated it with training data. By relying on the gold standard summaries{(a)
the author abstract, (b) a human-written abstract, and (c) a community-based
abstract{we created di erent target scores. We compared, using cosine
similarity, each sentence vector in the reference paper with each vector in the summary
and used the maximum similarity values as the target score for the reference
paper (e.g., score(S)) for learning. This method produced nine di erent functions
to learn: SUMMA, ACL, and Google vectors times abstract, community, human
summaries. Note that other target functions are possible but we restricted the
number of systems to nine given time constraints. The number of instances used
to train the linear regression models was 6,372.
Before submission, we carried out a preliminary evaluation of the nine models
using 20 document clusters for training and eight document clusters for testing
(we could not use two clusters due to errors generated when processing some
of the documents in them). The evaluation framework adopted was to compare
each of the summaries generated by the model (9 models times 8 clusters = 72
abstracts) against each of the summary types given by the organizers: abstract,
community, and human. The comparison was carried out using ROUGE-2 and
ROUGE-SU4 [
We submitted four runs for tasks 1A, each one applying one of the methods described in Section 3, with the results obtained by the Method /No-Method Facet Classi er described in Section 4 for Task 1B.
The organizers of the challenge evaluated Task 1A considering the overlap of the selected sentences and also calculating the resulting ROUGE-2 score. For Task 1B they considered the facet classi cation based on the sentences previously obtained for Task 1A. The results they obtained with the test set are shown in Table 4, where we include our best result{obtained with the voting system{ as well as the maximum, mean and minimum scores for all the systems submitted (macro averages). For Task 2 we submitted nine trainable systems corresponding to nine ways of interpreting the gold standard summaries: three vector representations times three gold standard summaries (system names in rst column of Table 3). The organizers of CL-SciSumm used ROUGE-2 and ROUGE-SU4 for the evaluation of the results for Task2. In Table 5 we show our results compared to the mean, minimum and maximum results obtained in the challenge. In this paper we have presented unsupervised and supervised methods to address the tasks proposed by the CL-SciSumm 2017 challenge. Our sentence-matching approach takes advantage of both discrete vector representations using terms weighted with tf*idf and continuous word representations. It optimizes several sentence similarity metrics also combining them in a voting system. The facet classi er is a SVM trained on the annotated dataset provided. It uses a set of manually engineered features informed by our previous work. Our citationbased summarization system is a linear regression based algorithm which learns to score sentences based on reference paper and citing paper(s) features, several of them based on continuous word vectors. Our long term goal is to contribute to the areas of extraction and summarization of scienti c information. Further work we are considering in this direction include the creation of linguistically enriched scienti c datasets and their exploitation to improve access to scienti c information. We are also interested in exploring other sentence matching and classi cation approaches, including some based on deep learning techniques.
This work is supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502) and by the TUNER project (TIN2015-65308-C5-5-R, MINECO/FEDER, UE). 17. Paice, C.D., Jones, P.A.: The identi cation of important concepts in highly structured technical papers. In: Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 69{78. SIGIR '93, ACM, New York, NY, USA (1993) 18. Qazvinian, V., Radev, D.R.: Scienti c paper summarization using citation summary networks. In: Proceedings of the 22Nd International Conference on Computational Linguistics - Volume 1. pp. 689{696. COLING '08, Association for Computational Linguistics, Stroudsburg, PA, USA (2008) 19. Qazvinian, V., Radev, D.R.: Identifying non-explicit citing sentences for citationbased summarization. In: ACL 2010, Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, July 11-16, 2010, Uppsala, Sweden. pp. 555{564 (2010) 20. Ronzano, F., Saggion, H.: Dr. Inventor Framework: Extracting structured information from scienti c publications. In: International Conference on Discovery Science. pp. 209{220. Springer (2015) 21. Saggion, H.: SUMMA: A robust and adaptable summarization tool. Traitement
Automatique des Langues 49(2) (2008) 22. Saggion, H., AbuRa'ed, A., Ronzano, F.: Trainable citation-enhanced summarization of scienti c articles. In: Proceedings of the Joint Workshop on Bibliometricenhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL) co-located with the Joint Conference on Digital Libraries 2016 (JCDL 2016), Newark, NJ, USA, June 23, 2016. pp. 175{186 (2016) 23. Saggion, H., Lapalme, G.: Concept identi cation and presentation in the context of technical text summarization. In: Proceedings of the 2000 NAACL-ANLP Workshop on Automatic Summarization. pp. 1{10. Association for Computational Linguistics, Stroudsburg, PA, USA (2000) 24. Saggion, H., Lapalme, G.: Generating indicative-informative summaries with SumUM. Computational Linguistics 28(4), 497{526 (2002) 25. Teufel, S., Moens, M.: Summarizing scienti c articles: Experiments with relevance and rhetorical status. Comput. Linguist. 28(4), 409{445 (Dec 2002) 26. White, L., Togneri, R., Liu, W., Bennamoun, M.: How well sentence embeddings capture meaning. In: Proceedings of the 20th Australasian Document Computing Symposium. p. 9. ACM (2015) 27. Witten, I.H., Frank, E., Hall, M.A., Pal, C.J.: Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann (2016)