In this paper we present several systems developed to participate in the 4th Computational Linguistics Scienti c Document Summarization Shared challenge which addresses the problem of summarizing a scienti c paper using information from its citation network (i.e., the papers that cite the given paper). Given a cluster of scienti c documents where one is a reference paper (RP) and the remaining documents are papers citing the reference, two tasks are proposed: (i) to identify which sentences in the reference paper are being cited and why they are cited, and (ii) to produce a citation-based summary of the reference paper using the information in the cluster. Our systems are based on both supervised (LSTM and convolutional neural networks) and unsupervised techniques using word embedding representations and features computed from the linguistic and semantic analysis of the documents.
Although scienti c summarization has always been an important research topic
in the area of natural language processing (NLP) [
The interest in the area has motivated the development of a series of
evaluation exercises in scienti c summarization in the Computational Linguistics (CL)
domain known as the Computational Linguistics Scienti c Document
Summarization Shared Task which started in 2014 as a pilot [
In this challenge, given a cluster of n documents where one is a reference paper (RP) and the n 1 remaining documents are papers (i.e., citing papers (CPs)) citing the reference paper, participants of the challenge have to develop automatic procedures to simulate the following tasks: given a cluster of n documents where one is a reference paper and the n 1 remaining documents are papers containing citations to it:
The challange has the following tasks: { Task 1A: For each citance in the citing papers (i.e., text spans containing a citation), identify the cited spans of text in the reference paper that most accurately re ect the citance. { Task 1B: For each cited text span, identify which discourse facet it belongs to, among: Aim, Hypothesis, Implication, Results, or Method. { Task 2: Finally, an optional task consists on generating a structured summary of the reference paper with up to 250 words from the cited text spans.
In this paper we report the systems developed at LaSTUS-TALN+INCO
to participate in CL-SciSumm 2019 [
We tried a supervised and an unsupervised approach for Task 1A. We separated
the CL SciSumm 2018 corpus of documents in 75% for training and 25% for
development evaluation. We also used the 978 documents from ScisummNet
2019 automatically annotated following [
Our supervised approach consists in a neural network architecture for nding out which sentences from the reference document are most the likely candidates for being referenced by a given citation.
Network architecture The neural networks have the following structure: { Input layer - Two sentences: the citation text and a sentence from the reference document. { Embeddings layer - We tried with two collections of embeddings: Google
News3 300 dimensions vectors and BabelNet[
We carried di erent experiments using word embeddings or BabelNet synset embeddings, the tokens in the input layer were words or synsets depending on the experiment. The LSTM layers combine up to three layers and a dense layer with sizes 150, 300, or 450. In all of our experiments we aimed to optimize against our development set, which contains 25% of the CL-SciSumm 2018 training set. Pre-training and Training We separated the training of the models in two stages: pre-training and training. The 978 clusters of documents from the Yale corpus were used to do a pre-training of the LSTM models. During pre-training, we trained the models using 70% of the Yale corpus optimizing against the remaining 30% using early stopping.
After this pre-training phase was over, we trained the resulting model using our CL-SciSumm 2018 training partition. We found out that, in general, pre-training with the Yale corpus and then training with CL-SciSumm 2019 achieved better results than only training with CL-SciSumm, even if the Yale data was automatically annotated. For the training stage, we used early stopping optimizing against 20% of our training corpus. 2.2
As in previous editions [
We submitted a voting system which considers sentences picked by two or more of the previous mentioned systems for Task 1. 2.4
In this section, we describe our extractive text summarization approach based on
convolutional neural networks which extends on our previous work on trainable
summarization [
In order to extract the linguistic information from both sources (RP and CPs), we developed a complex feature extraction method to characterize each sentence in the RP and its relation with the corresponding CPs.
We extracted a set of numeric features some of which are based on comparing a sentence to its (document or cluster) context: { Sentence Abstract Similarity Scores: the similarity of a sentence vector to the author abstract vectors (three features). { Sentence Centroid Similarity Scores: the similarity of a sentence vector to the article centroid (three features). { First Sentence Similarity Scores: the similarity of a sentence vector to the vector of the rst sentence, that is, the title of the RP (three features). { Position Score: a score representing the position of the sentence in the article.
Sentences at the beginning of the article have high scores and sentence at
the end of the article have low scores.
{ Position in Section Score: a score representing the position of the sentence
in the section of the article. Sentences in rst section get higher scores,
sentences in last section get low scores.
{ Position in a Speci c Section Score: a score representing the position of the
sentence in a particular section. Sentences at the beginning of the section
get higher scores and sentences at the end of the section get lower scores.
{ TextRank Normalized Scores: a sentence vector is computed to obtain a
normalized score using the TextRank algorithm [
As commented above, our CNN learns the relation between features and a score,
that is, a regression task by devising various scoring functions to represent the
likelihood of a sentence belonging to a summary (for abstract, community and
human). The nomenclature followed to symbolize a scoring function is SCSum,
where SC is the speci c scoring function (which is indicated bellow) and Sum
is any summary type: abstract (Abs), community (Com) or human (Hum). The
scoring functions are de ned bellow:
{ Cosine Distance: we calculated the maximum cosine similarity between each
sentence vector in the RP with each vector in the gold standard summaries.
This method produced three scoring functions (SUMMA (SUSum), ACL
(ACLSum), and Google (GoSum)) for each summary type.
{ ROUGE-2 Similarity: we also calculated similarities based on the overlap
of bigrams between sentences in the RP and gold standard summaries. In
this regard, each sentence in the RP is compared with each gold standard
summary using ROUGE-2 [
Finally, these computation produced eighteen di erent functions to learn: SUMMA (SU ), ACL (ACL) and Google (Go) vectors, ROUGE-2 (R2), Average (Av) and Simpli ed Average (SAv) times abstract (Abs), community (Com), human (Hum) summaries.
Regarding the neural network hyperparameters, the CNN was de ned with the
Adadelta updater [
{ run1: LSTM trained with Babelnet vectors with three layers of size 150.
{ run2: BabelNet centroids cosine similarity.
{ run3: LSTM trained with Google News vectors with two layers of size 150.
{ run4: Voting scheme based on [
{ Similarity with the abstract from all similarity scores except SUMMA. { Similarity with the abstract from all scores. { Rouge based score similarity with the abstract. { ACL cosine similarity based score with the abstract.
Finally, based on [
The performance of our systems for task 1 over the test set is shown in table 2.
We can see that the LSTM approached underperformed compared to their results
over the development corpus, one possible cause for this is that the systems could
have over t to the training and development data. Out of the methods we tried,
the system that performs best for task 1 is still the voting scheme based on [
We have described the systems developed to participate in Tasks 1a, 1b and 2 in the CL-SciSumm 2019 summarization challenge. For Task 1a { which aimed at identifying cited sentences {, we implemented supervised and unsupervised methods. Our supervised systems are based on LSTM neural networks, while the Run run4 Voting scheme run2 BabelNet centroids run3 Google News LSTM run1 BabelNet LSTM unsupervised techniques take advantage of BabelNet synset embedding representations. We also included a system that uses a voting scheme based on several supervised and unsupervised approaches with many di erent system con gurations.
Regarding Task 2 { summarization proper {, we have developed a neural network based on convolutions to learn a speci c scoring function. The CNN model was fed by a combination of word embedding with sentence relevance and citation features extracted from each document cluster (RP and CPs).
This work is (partly) supported by the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM2015-0502).