In this paper we present several systems developed to participate in the 3rd Computational Linguistics Scienti c Document Summarization Shared challenge which addresses the problem of summarizing a scienti c paper taking advantage of its citation network (i.e., the papers that cite the given paper). Given a cluster of scienti c documents where one is a reference paper (RP) and the remaining documents are papers citing the reference, two tasks are proposed: (i) to identify which sentences in the reference paper are being cited and why they are cited, and (ii) to produce a citation-based summary of the reference paper using the information in the cluster. Our systems are based on both supervised (Convolutional Neural Networks) and unsupervised techiques taking advantage of word embeddings representations and features computed from the linguistic and semantic analysis of the documents.
Automatic text summarization [
In this challenge, given a cluster of n documents where one is a reference paper (RP) and the n 1 remaining documents are papers (i.e., citing papers (CPs)) citing the reference paper, participants of the challenge have to develop automatic procedures to simulate the following tasks: { Task 1A: For each citance (i.e. a reference to the RP), identify the spans of text (cited text spans) in the RP that most accurately re ect the citance. { Task 1B: For each cited text span, identify what facet of the paper it belongs to, from a prede ned set of facets namely: Aim, Hypothesis, Implication, Results or Method, { Task 2: Finally, an optional task consists on generating a structured (of up to 250 words) summary of the RP from the cited text spans of the RP.
In this system description paper, we overview the techniques we have applied to solve the proposed tasks. 2
The organizers of the CL-SciSumm 2018 challenge provide training data structured in clusters of reference and citing papers together with manual annotations indicating, for each citance, the text span(s) in the reference paper that best represent the citance, as well as their corresponding facets. The training corpus contains 40 clusters with an average of 17 papers per cluster. For each cluster there are three manually created summaries of the reference paper: the author abstract, a community-based abstract created using citation sentences, and a human abstract created based on information from reference paper and citation sentences. The test set has 20 clusters with an average of 11 papers per cluster. 2.1
In order to automatically process the clusters, we created, from the documents
in the training and test sets, GATE [
Based on these annotations we could easily build pairs of matching sentences (Citing Paper Sentence, Reference Paper Sentence) and associate, to each pair, the facet that the annotator considered the citation referred to. 2.2
The GATE system was used to tokenize, sentence split, part of speech tag,
manage gazetteers and lemmatize each document. Teufel's [
We utilized a deep-learning approach (Convolutional Neural Network (CNN)) formulating the problem of nding a set of sentences in a reference paper that best re ects a citation in a citing paper as a regression problem which uses a CNN with two inputs and one output. The rst input models the reference paper sentences as a Word2Vec representation and the second input calculates a set of features based on the pair of sentences (reference paper sentence and a citation sentence). On the other hand, the regression output represents a score for each sentence in the reference paper based on the set of citations citing the reference paper. The output score of each reference sentence is based on the distance between the sentence and a sentence that are being cited. A value of 1 is set to cited sentences and the further the sentence is from the nearest cited sentence the less score it has.
We also used the same neural network to predict the facet a cited sentence belongs to. However, for facets we formulated the problem as a classi cation problem in which the output in that case is one of the ve prede ned facets classes provided by the organizers.
We modeled each reference sentence as a Word2Vec representation from three di erent pre-trained Word2Vec models embedded in a 300 dimensional space: (1)
ACL 3[
From each reference paper we extracted all the sentences having a number of tokens in a range of 5 to 40 and we used the 300 dimensions of each of the rst 15 tokens from each sentence. In order to reduce the number of pairs of sentences to consider, we also excluded sentences which according to the analysis carried out with the Dr. Inventor library belongs to the Background or Future Work discourse facets since it is assumed those sentences will mainly refer to work carried out by other authors or still inexistent.
We ran three CNNs over each sentence embeddings in which the width is the 300 dimensions, the height is 2, 3 and 4 respectively to represent: bi, tri and quadri-grams and nally, 3 channels to present the three pre-trained models. 3.1
In addition to CNNs we calculated a set of features based on the pair of the cited sentence in the reference paper and the sentence citing it in the citing paper. Those features were modeled and motivated to identify the cited sentences and their facets. Sentence Position Features: The sentence position in a paper can identify which parts of a reference paper are mostly cited, also the facet the sentence belongs to. For instance, sentences at the end of the document would probably belong to the Result facet. We use three features based on the location of the sentence in the reference document: { Sentence position: the position of the sentence in the reference paper; { Section sentence position: the position of the sentence in the section;
WordNet Semantic Similarity Measures Features: Similarities derived
from WordNet's graph could indicate matched sentences, it could also identify
their facets, the more similar a sentence is to another sentence with a known facet
the more likely it will have the same facet. We used WS4J (WordNet Similarity
for Java) library which includes several semantic relatedness algorithms that rely
on WordNet 3.0. Given a pair of sentences (reference and citance), we retrieve
all the synsets associated to nouns and verbs in each one of them. Then, by
considering all the pairs of synsets belonging to di erent sentences, we compute
similarity values between citance sentence and reference sentence. We calculated
similarity values between each token in the citance sentence and each and every
token in the reference sentence. Finally averaging all the similarities for the given
sentence pair. The computed measures are:
{ Path similarity [
Net, the more similar they are.
{ JCN similarity [
Word2vec/tree/master/trainedmodels
4 https://code.google.com/archive/p/word2vec/
{ LCH similarity [
Scienti c Gazetteer Features: We generated two features based on Teufel's action and concept lexicon. The lexicon contains 58 lists. The features are computed based on the reference sentence. 3.2
We performed experiments using two unsupervised methods. In both
experiments, we compare all the sentences in the reference paper to the citation text
using some distance metric, then we consider the closest sentences according to
this metric as candidates. The two metrics de ned are the following:
{ Modi ed Jaccard: We used a metric similar to the Jaccard similarity
coe cient for comparing the two sentences [
The only parameter to adjust using these methods is the number of sentences to consider as candidates. In order to optimize this parameter, we tested against CL-SciSumm 2017 test data, which is also a subset of CL-SciSumm 2018 training data. The results of these experiments are shown in table 1. The best result is achieved using the BabelNet Embeddings metric, considering only the closest sentence as candidate. The best result for Modi ed Jaccard is also close in Fmeasure. We designed a voting scheme that intended to leverage the strengths of the different supervised and unsupervised approaches. We considered these four system runs for the voting scheme: { Top 10 sentences from the Convolutional Neural Network using learning rate 0.0001. { Top 10 sentences according to Modi ed Jaccard unsupervised approach. { Top 10 sentences according to BabelNet Embeddings unsupervised approach. { Top 40 sentences for each target paper according to the relevance scores described in section 4.2.
The voting scheme returns a candidate sentence if at least N of the four systems agree on that sentence. The results are ordered according to relevance score and if there are more than ve candidates, only the top ve are selected. If there are no candidates in the intersection, the top sentence according to the BabelNet embeddings approach is used as a fallback mechanism. We submitted two runs using N = 2 and N = 3.
In this section, we describe our extractive text summarization approach based on
convolutional neural networks which extends on our previous work on trainable
summarization [
In order to extract the linguistic information from both sources (RP and CPs), we developed a complex feature extraction method to characterize each sentence in the RP and its relation with the corresponding CPs.
Before the extraction of context features, we compute word vectors based on Word Embeddings and term frequencies, as described above (see Section 3). Speci cally we used Google News and ACL pre-trained word embeddings and term frequency vectors by SUMMA. Then, for each sentence in the RPs and CPs, we compute three sentence vectors based on the centroid (Google, ACL and SUMMA). In addition, for each RP, we also computed the three centroids based on the abstract and the whole article.
From these vector representations, we extracted a set of numeric features some of which are based on comparing a sentence to its (document or cluster) context: { Sentence Abstract Similarity Scores: the similarity of a sentence vector to the author abstract vectors (three features). { Sentence Centroid Similarity Scores: the similarity of a sentence vector to the article centroid (three features). { First Sentence Similarity Scores: the similarity of a sentence vector to the vector of the rst sentence, that is, the title of the RP (three features). { Position Score: a score representing the position of the sentence in the article.
Sentences at the beginning of the article have high scores and sentence at
the end of the article have low scores.
{ Position in Section Score: a score representing the position of the sentence
in the section of the article. Sentences in rst section get higher scores,
sentences in last section get low scores.
{ Position in a Speci c Section Score: a score representing the position of the
sentence in a particular section. Sentences at the beginning of the section
get higher scores and sentences at the end of the section get lower scores.
{ TextRank Normalized Scores: a sentence vector is computed to obtain a
normalized score using the TextRank algorithm [
As commented above, our CNN learns the relation between features and a score,
that is, a regression task by devising various scoring functions to represent the
likelihood of a sentence belonging to a summary (for abstract, community and
human). The nomenclature followed to symbolize a scoring function is SCSum,
where SC is the speci c scoring function (which is indicated bellow) and Sum
is any summary type: abstract (Abs), community (Com) or human (Hum). The
scoring functions are de ned bellow:
{ Cosine Distance: we calculated the maximum cosine similarity between each
sentence vector in the RP with each vector in the gold standard summaries.
This method produced three scoring functions (SUMMA (SUSum), ACL
(ACLSum), and Google (GoSum)) for each summary type.
{ ROUGE-2 Similarity: we also calculated similarities based on the overlap
of bigrams between sentences in the RP and gold standard summaries. In
this regard, each sentence in the RP is compared with each gold standard
summary using ROUGE-2 [
Finally, these computation produced eighteen di erent functions to learn: SUMMA (SU ), ACL (ACL) and Google (Go) vectors, ROUGE-2 (R2), Average (Av) and Simpli ed Average (SAv) times abstract (Abs), community (Com), human (Hum) summaries. 4.3
Basically, a CNN consists of multiple convolutional and pooling layers, with fullyconnected layers at the end. The network is fed with two di erent inputs. The inputs are composed of instances related to sentences. The rst one is based on the context features (described in the section 4.1). Speci cally, context features are introduced in the CNN within a sequential window including the context features of the 3 previous and 3 following sentences. And the second input is related to the word embedding information for each sentence. In particular, we used both word embeddings (Google and ACL) as a dual channel, which stopwords were removed, the size was xed in 15 words and they were kept static during the training.
Regarding the neural network hyperparameters, the CNN was de ned with
the Adadelta updater [
The evaluation consists of generating a 250-word summary according to the task, which are compared against each of the summary types of the gold standard: the reference papers abstract, a community summary, and a human summary. We trained and evaluated our model using the CL-SciSumm-17 dataset.
{ MJ1: unsupervised approach using Modi ed Jaccard similarity { BN1: unsupervised approach using BabelNet synset embeddings cosine similarity { 0.1CNN4: deep learning approach using CNN over the word embedding + a set of features . Learning rate: 0.1 Epoch: 50 { 0.0001CNN4: deep learning approach using CNN over the word embedding + a set of features . Learning rate: 0.0001 Epoch: 50 { Voting2: keep candidates if at least two of the systems agree (MJ, BN, CNN or top 40 sentences using task2 system) { Voting3: keep candidates if at least three of the systems agree (MJ, BN, CNN or top 40 sentences using task2 system)
For the task 2, we have submitted eighteen summaries related to each scoring function and summaries. In other words, each resulting summary is de ned by SCSum (see Section 4.2), where SC is the scoring function (SU , ACL, Go, R2, Av and SAv) and Sum is the summary type (Abs, Com and Hum). For example, submission ACL abs learns a scoring function which attempts to approximate similarity of a sentence to the abstract of the document using ACL vectors and cosine to compute similarities. 6
We have described the systems developed to participare in Tasks 1a, 1b and 2 in the CL-SciSumm 2018 summarization challenge. For Task 1a { which aimed at identifying cited sentences {, we implemented supervised and unsupervised methods. Our supervised systems are based on Convolutional Neural Networks (CNN), while the unsupervised techniques take advantage of word embedding representations and features computed from the linguistic and semantic analysis of the documents. However, as committing to only one system could result in an underperforming approach, we applied many di erent system con gurations combining them through a voting mechanism. For Task 1b we used the same CNN system of Task 1a where the output was a set of facets.
Regarding Task 2 { summarization proper {, we have developed a neural network based on convolutions to learn a speci c scoring function. The CNN model was fed by a combination of word embeddings with sentence relevance and citation features extracted from each document cluster (RP and CPs). The approach was developed and evaluated following the CL-SciSumm Shared Task 2 dataset, our approach outperformed results reported in last year CL-SciSumm17 Shared Task 2.
This work is (partly) supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM2015-0502) and by the TUNER project (TIN2015-65308-C5-5-R, MINECO/FEDER, UE).