Cited text identification is helpful for meaningful scientific literature summarization. In this paper, we introduces our system submitted to the CLSciSumm 2019 Shared Task 1A. Our system have two stages: similarity-based ranking and supervised listwise ranking. Firstly, we select the top-5 sentences per a citation text, due to the modified Jaccard similarity. Secondly, these top-5 selected sentences are proceeded to rank by a CiteListNet (listwise ranking model based on deep learning). Our experiments showed that our proposed method outperformed other prior methods on the CL-SciSumm 2017 test dataset.
Automatic summarization of academic paper may be a very effective solution to avoid the information overload of researchers and to understand the state-of-the-art of the research topic.
The CL-SciSumm shared tasks explore the solutions for the making a comprehensible summary of an academic paper given its citation text. These tasks focuses on the sentence-level cited text information to perform the summarization of a paper. To this end, the identification of the cited text should be done. CL-SciSumm Shared Task 1A is just to identify the spans of cited text in reference paper (RP) that contain the given citation text of its citing paper (CP).
In this paper, we use various similarity metrics to evaluate the similarity between a citation text and its candidate cited sentence, and adopt listwise ranking algorithm to train our ranking model.
Most of previous studies regarded the identification of cited texts as a classification
problem and thus used some machine learning algorithms like SVM, Random Forest,
CNN to train text classifiers. To build the classifiers, various features were explored by
researchers. Ma et al. (2018) chose Jaccard similarity, cosine similarity and some
position information as features, and trained four classifiers including Decision Tree,
Logistic Regression and SVM.[
There are two main issues in the categorization-based methods: local ranking and
class-imbalanced data. On the one hand, the cited text identification problem should be
regarded as a ranking problem rather than a classification one because we only intent
to choose the sentence(s) that contains more similar content with the citation
sentence(s) compared to other sentences. On the other hand, there is only few sentences
(usually not more than five) to be cited sentences in a target paper. Sometimes the ratio
of the negative and positive sample in a corpus is even greater than 150. Ma et al. (2018)
used Nearest Neighbor (NN) rule (Wilson, 1972) to reduce data imbalance and
increased the F1-score from 11.8% to 12.5%.[
With respect to the ranking-based cited text identification, a few studies have been
done. Dipankar et al. (2017) ranked the sentences in a target paper according to the
cosine similarity between each candidate sentence and the citation sentences to select
the top five sentences as the cited sentences.[
In this study, we regarded cited text identification as a ranking problem, and proposed a ranking-based method to identify citation sentences based on deep learning. This method includes two stages of ranking: a similarity-based unsupervised ranking and a supervised listwise ranking. Since the cited text was deemed to contain more similar content with the cited text than other sentences in the same paper, we first ranked all the sentences in a reference paper according to each sentence’s similarity with a cited text. Then we choose top K sentences to create a subset of the given train corpus for the second stage ranking, while the Kth sentence obtained the best F-value according to the given training corpus. In the second stage, a listwise ranking model was trained on the subset training corpus to rank the K sentences and then top N sentences (N<K) were selected as the cited sentences. 3.1
in one of its citing paper, whereas a citation text may contain one or more sentences. We considered five kinds of similarity metrics, including TFIDF-based cosine similarity, word embedding-based cosine similarity, SVM Kernel functions-based cosine similarity, Jaccard-like similarity and BM25.
similarity metrics, N-grams were extracted from a text as its features and represented the text as a TFIDF weighted feature vector based on Vector Space Model and calculated the cosine similarity between two feature vectors. We tested two TFIDF-based cosine similarity metrics: when N=1 and when N=2 respectively for N-grams.
text was represented as a text vector based on word embedding. First, the vector of each word in a text was trained with Word2vec. Then three kinds of text vectors were calculated based on the word vectors using different weighting mechanisms, TF-weighted average of the word vectors (TF-AWV), TFIDF-weighted average (TFIDF-AWV), and T-statistics-weighted average (Tstat-AWV). Here, the T-statistic of a word refers to the T-test statistics of the hypothesis that whether the word appears in a text or not is independent to whether or not the text is a cited text.
Cosine Similarity based on SVM kernel functions. As we know, a linear inseparable sample can become linearly separable by projecting a low-dimensional space to a highdimensional one. Thus we considered to transform the TFIDF-weighted average word vector (TFIDF-AWV) to a higher-dimensional space by using three SVM kernel functions, so as to make a text more distinguishable from others. Thus three kinds of kernel functions-based cosine similarity were calculated with the following equation based on three kernel functions respectively, i.e. 2-dimension polynomial, 3-dimension polynomial function and RBF function.
( ( ), ( )) =
( )∙ ( ) √ ( )∙ ( )∙√ ( )∙ ( ) =
( , )
√ ( , )∙√ ( , )
(1)
Where (∙) denotes a mapping function, by which a vector space can be mapped to
another space, (∙,∙) refers to a kernel function that follows Mercer’s condition[
Cosine Similarity based on SVM Kernel Functions. As we know, a linear
inseparable sample can become linearly separable by projecting a low-dimensional space to a
high-dimensional one. Thus we considered to transform the TFIDF-weighted average
word vector (TFIDF-AWV[
Jaccard-like Similarity. A citation text in a citing paper and each candidate sentence in a reference paper can be regarded respectively as a set of N-grams (N=1, 2, 3). We used some Jaccard-like similarities to calculate the overlap between the N-gram set of the citation text and the one of each candidate cited sentence. In addition to the standard Jaccard similarity metric, two variations, MJS1 (see eq. 3) and MJS2 (see eq. 4) were also considered.
( , ) = || ∩∪ || =
| ∩ | | |+| |−| ∩ | 1( , ) = | ∩ |
| | 2( , ) = | ∩ | |B| (2) (3) (4) Additionally, there are two different applicable weighting methods, respectively IDFweighting and T-statistic weighting, to improve Jaccard similarity metric. B25 measure. If we make a comparison of cited text identification issue with information retrieval, citation text may correspond to query, while cited sentence to document to be retrieved. Since B25 measure is well-known metric in classical information retrieval, it may also serve as similarity metric in our work.
In total, there were 36 similarity metrics (as shown in Table 1), which were used in the first-stage similarity ranking. (5)
We used the training data of the CL-SciSumm 2017 dataset to evaluate the performance of each kind of similarity metric. We found that the 33th feature (T-statistic weighted tri-gram MJS1 similarity) is best metric for top-K selection in the first stage. From the result like as shown in Table 2, we selected top-5 selection based on T-statistic weighted tri-gram MJS1 similarity, which was also used as a baseline system. 3.3
Recently, deep learning approaches have gradually gained popularity in artificial intelligent problems. However, it is usually applied in classification problem, but not in ranking problem.
For ranking, there are three kinds of methods: pointwise, pairwise and listwise.
Pairwise ranking is almost same effect with classification, in other words, this task is
formalized as classification of object pairs (correct sample and incorrect sample), such as
RankSVM [
Therefore, we proposed a novel listwise ranking model, namely CiteListNet, based on deep learnings. Given citance and reference sentence list 1, . . . , , the our task can be formulated as following: ( , ) = ( ( , )) =
exp( ( , )) ∑ exp( ( , ))
: max ∑ , ∗ ( , ) = {
1, 0, (6) (7) (8) Where F(.) denotes a deep learning model, which can be implemented as any architecture.
In above optimization, we only consider the order of positive sample, but not negative sample. The more the score of positive sample is, the higher the order of it is. If its score is over 0.5, then its order become the first.
In addition to similarity-based metric, we also involved the section information in our feature set. Section information represents in which kind of section the reference sentence appears in the reference paper. The kind of section is decided based on rulebased method. Table 3 shows the kinds of section which frequently appear in the computational linguistics field. Introduction Related work Method Conclusion
No 7 8 9 10 11
4
The CL-SciSumm 2017 datasets contain 30 topics as training data and 10 topics as test
data [
To train the list-wise ranking model, we prepared the subset from the training data through the top-5 sentence selection as described in Section 3.2.
In our experiment, the top-2 listwise ranking showed the best performance on the training data where its F1-score is observed as 17.9%.
In order to validate the generalization ability of our method, we try to evaluate the trained model on the test data in the CL-SciSumm 2017 datasets. The result demonstrated that the top-2 listwise ranking showed a best performance and that the overfitting did not occur during our listwise ranking training.
We also compared our method with earlier approaches as shown in Table 4. To
evaluate the performance of the different systems, two kinds of metric is used: sentence ID
overlap and ROUGE scoring [
The experiment results demonstrate that our proposed method outperforms any prior approaches.
As shown in Table 5, top-2 ranking also showed a good performance on the test data
in CL-SciSumm 19 Shared Task.
1 In Table 4, the result of the prior works is from Jaidka’s report [
In this paper, we focused on the cited text identification issue and proposed a novel method, namely CiteListNet, based on listwise ranking.
The main contributions of this paper are two points: feature selection and a novel listwise ranking model.
First, we adopted new features to identify cited text. We modified Jaccard similarity to consider how much the cited sentence covers the citation text and how much the citation text covers the cited text. The former is verified to be useful for identifying cited text, while the letter is less helpful than other features. When using the Jaccard similarity, it is recommended to use N-gram language model. Moreover, we found that T-statistics that represents how much the word is probable to appear in the cited text could be used as a useful weight. The experimental result shows that T-statistics weighted Modified Jaccard Similarity feature based on tri-gram language model is the most useful feature and MJS-based top-5 sentence selection shows the comparable performance, although no training is done.
Second, we proposed a novel listwise ranking model based on deep learning CiteListNet. We found that our proposed method is stable and did not occur overfitting problem during the training process. The result of our experiment shows that our novel method outperforms other prior approaches.
In this paper, it was still not considered that cited text could be represented as a paragraph. In this case, the relationship between the cited sentence and the citation text may have a different characteristic. In the future work, we will focus this issue, and apply this research result to various bibliometric task.
Acknowledgement. This paper is one of the research outputs of the project supported by the State Key Program of National Social Science Foundation of China (Grant No. 17ATQ001).