ATHENA@CL-SciSumm 2019: Siamese recurrent bi-directional neural network for identifying cited text spans

Abstract

In this paper we describe our participation to the Task1 of the CL-SciSumm 2019. The task is on automatic paper summarization in the research area of Computational Linguistics. Our approach is a two step binary sentence pair classi cation between the so-called citances and candidate sentences. Firstly, we classify sentences in the abstracts to prede ned classes we call \zones". These zones capture the discourse structure of a scienti c publication. We then expand these zones with additional, similar sentences which are found in the main sections of the publication body. We train a Siamese bi-directional GRU neural network with a logistic regression layer to decide if a citance alludes to a candidate sentence. The cited sentences are also assigned one or more discourse facets (i.e., categories de ned in the Task) using a multi-class SVM. We ran extensive experiments in three different datasets achieving promising results.

Keywords

• Candidate Cited Sentence Selection
• Siamese Neural Net- work
• Discourse Facet

-

Researchers are confronted with a continuously increasing volume of scienti c publications, facing difficulties to monitor and track [ 13 ]. The ability to create synopsis of the key-points, contribution and importance of a paper within an academic community is an important step [ 12 ]. This synopsis, can be created by using citation sentences (i.e., the citances) that reference a speci c paper and can be considered as a community-created summary of a topic or a paper. Scienti c summaries offer an overview of the cited paper useful to scholars, writers or literature reviewers [ 7, 10 ]. The CL-SciSumm Shared Task focuses on the scienti c summarization of papers [ 6 ], organized into two tasks. For both tasks the organizers provide several Reference Papers (RPs) called \topics". Task1A: For each citance, identify the spans of text (cited text spans) in the RP that most accurately re ect the citance. These spans are of the granularity of a sentence fragment, a full sentence, or several consecutive sentences (no more than 5).

Task1B: For each cited text span, identify what facet of the paper it belongs to, from a prede ned set of facets.

Task2 (optional): Generate a structured summary of the RP from the cited texts pans of the RP. The length of the summary should not exceed 250 words.

We participated on the tasks 1A and 1B of 2019 shared task [ 1 ] and present here our methodology. 2

Methodology

We approach Task 1A as a binary sentence pair classi cation problem. We create pairs of citance and a candidate sentence extracted from the Citing Papers (CP) and the RPs respectively. Word embeddings are used to select candidate sentences. A Siamese neural network process these pairs to decide whether or not the candidate sentence is a cited text span of the citance. For the Task 1B a multi-class SVM [ 3 ] model assigns a discourse facet to the cited text spans. 2.1

System Components Word Embeddings We use word embeddings for both the candidate sentences selection and for the embedding layer of our network. Embedding vectors are trained on the ACL Corpus dump4 using the CBOW implementation of word2vec [ 11 ] of the gensim5 tool, with negative sampling set to 5 and 100 for dimensionality of the word vectors. All words are converted to lowercase. Candidate Sentence Selection We select sentences from the RP as candidate sentences. The intuition is that not all sentences are equally important as cited text spans. Thus, we try to select sentences that are about methodology, results and conclusions and discard sentences about background and related work. This is also supported by the fact that the cited text is assigned a discourse facet. Our approach tries to eliminate sentences that potentially would be false positives.

To select the candidate sentences of the RP we split the abstract into zones [ 9 ]. Each sentence is classi ed to one of the following zones: Background, Method, Result and Conclusion. We keep only the sentences that belong to Method, Result and Conclusion zones (referred to as zone sentences ). Sentences are split into words6, punctuation and numbers are removed and each word is assigned its embedding vector. For each zone sentence and the rest of the RP sentences we calculate sentence embeddings by averaging the word embedding vectors. 4 http://acl-arc.comp.nus.edu.sg/archives/acl-arc-160301-parscit/ 5 https://radimrehurek.com/gensim/, version 3.7.3 6 Using the tokenization tools of the gensim module

Using the embedding vectors of the N zone sentences and all the other embedding vectors of the M RP sentences, we calculate a similarity matrix S 2 RN M using cosine similarity measure. To get the most similar sentences to the zone sentences we de ne a threshold ts. The RP sentences Si;j that pass the similarity threshold ts and the zone sentences are kept as candidate sentences. The decision of the ts value is discussed into section 3.

Siamese Neural Network The Siamese neural network is composed of two bi-directional GRUs (biGRU) [ 14, 2 ] and a logistic regression layer, as depicted in Figure 1. Each biGRU processes one sentence at a time. For each citance and a set of candidate sentences of the RP, the left biGRU takes as input the citance and the right biGRU takes as input one of the candidate sentences. We use w1:n to denote a sequence of words w1:n = w1; : : : wn, each with their corresponding demb dimensional word embedding ei = E[wi]. The embedding matrix E 2 RjV j demb associates words from the vocabulary V with demb dimensional dense vectors.

The left biGRU applies additive zero-centered Gaussian noise [ 4 ] to word embeddings with = 0:05 as a regularization layer at the training phase. The outputs y1b and ynf of the backward GRUb and the forward GRUf respectively are concatenated in one vector y1b = GRUb(en:1) ynf = GRUf (e1:n) xl = [y1b; ynf] y1′b = GRUb(en:1) yn′f = GRUf (e1:n) xr = [y1′b; yn′f ] We use xl to denote the output of the left input and xr of the right input and [ ; ] to denote concatenation. The two output vectors are element wise multiplied to give a vector x. A logistic regression layer (LR) with a sigmoid activation function ( ) is used to make the nal prediction y^. To summarize the architecture p(y = kjw1:n) = y^; k 2 f0; 1g y^ = LR(x); with ( ) activation x = [xl

xr]

The described model considers one sentence at a time. In order to nd if a citance references more than one sentences in the RP, we take the predictions of all the candidate sentences and keep the maximum score smax. We de ne a threshold as st = 0:98 smax. Any candidate sentence that has score s such as st s smax is selected as a cited sentence.

b y1

GRUb GRUb

GRUf GRUf

GN e1 GN e2

Citance y1b; ynf

GRUb

GRUf

GN en y^ LR y1b; ynf y1′b; yn′f f yn y1′b; yn′f y1′b

GRUb

GRUb

GRUf

GRUf

yn′f

GRUb

GRUf

e′ 1 e′ 2

e′n Candidate Sentence

Discourse Facet Task 1B asks \for each cited text span, identify what facet of the paper it belongs to, from a prede ned set of facets". The prede ned facets are Aim, Hypothesis, Method, Result and Implication. We approach this task as a multi-class classi cation problem due to the fact that some cited text spans may have up to two facets. We build a bag-of-terms representation of all n-grams with n = 1; 2; 3 and calculate their tf-idf values using L1-norm. Five one-vs-rest SVM classi ers were trained assigning a cited text span to each of the ve facets.

Experiments and Results

The dataset provided was split into a training and a development set. The training set consists of a set of 40 RPs with their CPs annotated by humans and 1000 RPs and their corresponding CPs that were automatically annotated. For our experiments we only used the human part of the training set (the TR-H set ).

As a rst step for our experiments we selected candidate sentences from the RPs. By keeping only the candidate sentences we might miss cited sentences in the RP which were not selected by our method. In Table 1, coverage is the number of RP sentences we kept and the hits metric is the number of the cited sentences in our candidate list (in percentage). Our target is to get minimum coverage with maximum hits. Minimum coverage means that we have kept all the good candidates while maximum hits denotes that the cited sentences are within our candidate list.

Table 1 displays the average coverage and hits for the 40 RPs of the training set and the 10 RPs of the development set for different thresholds. Based on the results, ts was set to 0.5. Using this threshold, we keep about 70% of the RP sentences on the training set and 60% on the development set, on average. Despite the fact that we discarded about 30% and 40% of the candidate sentences we only lose 15% and 20% of the cited text spans, respectively.

We evaluated our system in three different versions of the dataset; for each version, we used for testing the development set (Dev), the 2016 test set (2016) and the 2017 test set (2017) respectively. For training, we used the TR-H set provided that we have excluded all papers in the relevant testing set for obvious reasons. The results shown in Table 2 are comparable to those of the previous shared tasks [ 6, 5, 8 ]. 4

Conclusions and Future Work

Scienti c summarization is a challenging task as it is evident from the results of the previous shared tasks [ 6, 5, 8 ]. In our methodology we create pairs of citance and a candidate sentence extracted from the CP and the RP respectively. These pairs are classi ed from a Siamese neural network as positive if a citance indeed cites a sentences, otherwise as negative. The cited sentences are also assigned one or more discourse facets. We applied our methods on the dataset of the 2019 CLSciSumm shared task. The evaluation of our system indicates that the Siamese neural network performs comparable to other machine learning methods.

In future work we will investigate the impact of replacing the logistic regression layer with other similarity functions, such as cosine similarity. We also plan to select the best value for the st threshold via hyper-parameter tuning. Finally, we will experiment with different methods for cited sentences selection which take into account the scores of neighboring sentences. 5

Acknowledgement

We acknowledge support of this work by the Data4Impact Project which received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 770531.

References

1. Chandrasekaran , M.K. , Yasunaga , M. , Radev , D. , Kan , M.Y.: Overview and results: Cl-scisumm shared task 2019 . In: Proceedings of the 4th Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2019 ) @ SIGIR 2019 , Paris, France. ( 2019 )

2. Cho , K. , Van Merrienboer, B. , Bahdanau , D. , Bengio , Y. : On the properties of neural machine translation: Encoder-decoder approaches . arXiv preprint arXiv:1409.1259 ( 2014 )

3. Cortes , C. , Vapnik , V. : Support-vector networks . Machine learning 20(3) , 273 { 297 ( 1995 )

4. Hinton , G., Van Camp , D. : Keeping neural networks simple by minimizing the description length of the weights . In: in Proc. of the 6th Ann. ACM Conf. on Computational Learning Theory. Citeseer ( 1993 )

5. Jaidka , K. , Chandrasekaran , M.K. , Jain , D. , Kan , M.Y.: The cl-scisumm shared task 2017: Results and key insights . In: BIRNDL@ SIGIR (2) . pp. 1 { 15 ( 2017 )

6. Jaidka , K. , Chandrasekaran , M.K. , Rustagi , S. , Kan , M.Y.: Overview of the clscisumm 2016 shared task . In: Proceedings of the Joint Workshop on Bibliometricenhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL) . pp. 93 { 102 ( 2016 )

7. Jaidka , K. , Khoo , C. , Na , J.C. : Deconstructing human literature reviews{a framework for multi-document summarization . In: proceedings of the 14th European workshop on natural language generation . pp. 125 { 135 ( 2013 )

8. Jaidka , K. , Yasunaga , M. , Chandrasekaran , M.K. , Radev , D.R. , Kan , M.Y.: The cl-scisumm shared task 2018: Results and key insights . In: BIRNDL@SIGIR ( 2018 )

9. Jin , D. , Szolovits , P. : Hierarchical neural networks for sequential sentence classi - cation in medical scienti c abstracts . ArXiv abs/ 1808 .06161 ( 2018 )

10. Jones , K.S.: Automatic summarising: The state of the art . Inf. Process. Manage . 43 , 1449 { 1481 ( 2007 )

11. Mikolov , T. , Sutskever , I. , Chen , K. , Corrado , G.S. , Dean , J. : Distributed representations of words and phrases and their compositionality . In: Advances in neural information processing systems . pp. 3111 { 3119 ( 2013 )

12. Nakov , P.I. , Schwartz , A.S. , Hearst , M. : Citances: Citation sentences for semantic analysis of bioscience text . In: Proceedings of the SIGIR . vol. 4 , pp. 81 { 88 ( 2004 )

13. Qazvinian , V. , Radev , D.R. : Scienti c paper summarization using citation summary networks . In: Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1 . pp. 689 { 696 . Association for Computational Linguistics ( 2008 )

14. Schuster , M. , Paliwal , K.K.: Bidirectional recurrent neural networks . IEEE Transactions on Signal Processing 45 ( 11 ), 2673 { 2681 ( 1997 )