We present here system report for our model submitted for shared task on Computational Linguistic Scienti c-document Summarization (CL-SciSumm) 2017. We hypothesize that search and retrieval based techniques are sub-optimal for learning complex relation likes provenance. State-of-the-art information retrieval techniques using term frequency - inverted document frequency (TF-IDF) to capture surface level closeness along with di erent textual similarity features for semantic closeness are insu cient to capture implied and entailed provenance with less surface-level similarity. In our comparative studies, we nd that the provenance is relative i.e. something makes a better provenance than other based on certain linguistic cue or key information being more prominently conveyed, and hence we model the problem as pairwiseranking and not simple ranking or classi cation. To capture above points we propose a joint scoring approach weighting surface level closeness and learned semantic relation. We use TF-IDF and Longest Common Subsequence (LCS) for the syntactic score and pairwise neural network ranking model to calculate semantic relatedness score. For citation-provenance facet classi cation, we retro t the same neural network architecture on identi ed provenance, with the removed pairwise ranking component.
With the overwhelming amount of scienti c studies published every minute, it
has become very di cult to keep track of the recent advancements. Keeping this
vision in mind the BiomedSumm followed by CL-SciSumm task was proposed in
2014 [
To locate such references to the documents it's important to identify the cross-document discourse. One such structure used to capture such discourse is citation context i.e. the piece of the text in the reference paper (RP) cited by the citing paper (CP). The text in the CP is called the citance and the corresponding text in RP is called the provenance. Provenance helps in citation understanding and together with the citance can be used to understand important aspects like function, emotion, polarity etc. In the task scenario, the annotations use the statements from the provenance to create the summaries making it the performance bottleneck. Hence, in the CL-SciSumm 2017, the provenance identi cation acts at the main task.
The overall task structure is as follows: { Task 1a: Identifying the provenance of the citance { Task 1b: Classifying the identi ed provenance in on of the 6 facets { Task 2: Using the identi ed provenance make a summary of the RP (bonus task)
We present our approach for the citation contextualization (task 1a and task 1b) which has been most fundamental and di cult challenge of the complete pipeline. 2
In prior attempts, researchers have come up with features like TF-IDF, LCS,
Jaccard Similarity etc. computed over the citance and the candidates and run
simple linear regression models for getting the score. Selecting the best candidate
hence result into the provenance identi ed from human-engineered feature
without considering any comparative suitability with respect to other candidates.
The closest approach to ranking was rst used for this problem in form of linear
optimization constraints using a averaged word embedding representation [
Here, we discuss statistic of the data for the task.
{ CL-SciSumm 2017 training set compromises of 30 training documents (20
and 10 documents from CL-SciSumm 2016 training and test sets respectively
[
Provenance Detection Task 1a can be modeled in many possible ways including standard search and retrieval, sequential labeling, classi cation or ranking. Usually, a high fraction of citation (mainly for facets like Methods) shows high surface level similarity with the original citance and hence using retrieval techniques to capture high syntactic and semantic similarities give satisfactory results. However, this does not cover the cases where there is a less semantic similarity and the provenance are usually implied or entailed. Retrieval based techniques using TF-IDF like features are bound to fail in such cases. This calls for more powerful and general purpose model which has the ability to incorporate both the semantic similarity and learn the higher order relations between the texts.
To incorporate both such components in the model we propose a weighted scoring model as: score = 1(surface level closeness score) + 2(learned semantic relation score) (1)
To calculate surface level closeness a lot of features and scoring schemes has
been proposed in prior runs of the CL-SciSumm [
(2) Since the evaluation criteria use the ROUGE-SU4 and exact match for evaluation we use TF-IDF and Longest Common Subsequence (LCS) score to calculate the surface level similarity. We note that this is not the exhaustive combination of features as others features have shown to add to the performance, but we use only basic features to show the validity of our hypothesis and applicability of our method. Further, we assume that other similarity based features could be captured by the semantic relation model. For the semantic relation score, we explore deep learning based models using word embeddings as the features. We experiment with the options of classi cation and ranking.
Classi cation Versus Ranking. The classi cation model as shown in Fig. 3.1 uses RP text and CP text to form training tuples. First, vocabulary indexed text converted to word embedding passes through a Convolutional Neural Network (CNN) or Long Short Term Memory (LSTM). The CNN subpart of the network comprises convolution layer followed by a max pooling layer. The merge layer then does an element-wise multiplication of the activations thus learned for both the text. This models representation level similarity between the two texts. Finally, this representation passes through a feed forward layer which classi es it as either 1 or 0 depending on the label of the RP text. Some practical aspects of training are: { We avoid overwhelming negative samples while constructing the snippet of RP texts for training. We form samples per line rather than all possible combination of consecutive 1 to 5 lines. This helps in keeping the number of negative samples in reasonable yet varied. { Another way to keep the ratio of negative samples in check is to down-sample the RP texts. It can be done by random selection or more sophisticated way of ltering text e.g. selecting text only if the TF-IDF is greater than a certain threshold. In our experiments, this way does not give better performance as compared to just using all lines.
Another approach deploying similar architecture but with ranking ability is by incorporating modi cation as shown in 3.1. The training samples for this model are created by making tuples of two RP texts and one citance text. One of the RP text is always the correct provenance while other is sampled by one of the techniques discussed previously. The class label then predicts out of the two texts which one is better provenance for the citance text. The model predicts 0 or 1 depending on the RR text 1 or RR text 2 is the better provenance. During testing the system returns the RP text which wins the maximum number of pairwise comparison. The bene ts of this model as compared with the classi cation model are: { It solves the problem of skew class distribution by forming an exactly equal number of tuples of positive and negative samples. { It adds the ability to learn comparative features from the representations which make one text provenance as compared to other texts. For facet identi cation, we reuse the classi cation model however its trained on true provenance and citance samples and the output labels are one of the 5 classes. For balancing the class we use class weights as the log1000 of the inverse of the frequency. 4
Now we discuss the results for the di erent experimental setups we tried. All these experiments are done on the test set of CL-SciSumm 2016 while training on the train set for CL-SciSumm 2016, with no overlap among the sets. The nal submitted system uses all the documents for training. The parameters and are selected equally to sum to 1. A more sophisticated way of selecting these values is to use development set to learn the coe cients jointly during training, which can be explored in later works. All the neural networks are trained in Keras with small learning rate for 2 iterations using Adagrad optimizer. The input is padded to form a sequence of maximum length 100. GloVe word embedding of 300 dimensions is used and the size of LSTM/CNN experimented is 64. For facet identi cation, since multiple classes are allowed we pick all classes which are within a certain set as 0.05 probability score from the highest score.
The results from Table 1 shows that the classi cation model does not learn a lot compared with a model which predicts all the RP tests as non-provenance. However, the ranking model does signi cantly better possibly because of better class distribution and better modeling power of the ranking model compared to classi cation model as discussed.
The results from Table 2 shows that the proposed model for task 1a and
task 1b does not give better results as compared to already proposed traditional
syntactic similarity based features [