This document demonstrates our participant system PolyU on CLSciSumm 2016. There are three tasks in CL-SciSumm 2016. In Task 1A, we apply SVM Rank to identify the spans of text in the reference paper reflecting the citance. In Task 1B, we use the decision tree to classify the facet that a citance belongs to. Finally, in Task 2, we develop an enhanced Manifold Ranking summarization model.
The CL-SciSumm Shared Task [
Our system PolyU implements all the three tasks. For Task 1A, we treat it as a
ranking problem modeled by SVM Rank [
Table 1. Overall performance (%) of PolyU. 2
2.1
Problem Transformation In this task, we need to identify the spans of text (cited text spans) in the RP that most accurately reflect the citation. Since a citation text span can be linked to many sentences in the reference paper, we firstly analyze the corresponding sentence number distribution, as shown in Fig. 1. As can be seen, a large proportion of reference spans contain more than one sentences. Therefore, the most direct approach for this task is to train a ranking model and select a series of top ranked sentences. However, this idea has two disadvantages. On the one hand, the threshold is hard to set due to the serious variation of ranking scores on different document sets. On the other hand, a reference span tends to contain adjacent sentences, which cannot be reflected by the top ranked items. The adjacency property of reference spans is presented in Fig. 2. In this figure, we count the number of reference sentences which fail to be covered by adjacent sentence chunks. We observe that most multi-sentence reference spans are just a pair of adjacent sentences. Meanwhile, when the chunk size 4, about 90% reference sentences can be covered by sentence chunks, and the uncovered ratio keeps stable.
Therefore, we simplify Task 1A into a ranking problem which just needs to select
the first item. Specifically, like n-grams, we put adjacent sentences into n-sentence
chunks. According to the data property, we set n 2 [1; ; 4]. Then, a reference paper
is represented by these n-sentence chunks. The actual ranking score of an n-sentence
chunk is the ratio of reference sentences it contains. We extract a series of features
from (CP, n-sentence chunk) pairs. Finally, we train a ranking model and choose the
top ranked n-sentence chunk as the reference span.
Since most sentence chunks have no reference sentences, the regression model trained
on this dataset tends to predict the zero score. Therefore, we adopt SVM Rank [
We analyze the model weight for each feature. The feature SIMILARITY holds the highest weight, which accords with common sense. In addition, three position features, i.e., SENT POSITION, SECTION POSITION and INNER POSITION all have relatively large negative weights. It seems sentences in front are more likely to be cited. multi-label classification task. However, from the training data, we find more than 90% of citances only belong to one facet. Therefore, we simply treat this task as the common multi-class classification problem by reserving the first facet for citances with more than one facets.
Afterwards, we analyze the facet distribution on the training set. The result is shown in Fig 3. From this figure, we find the facet distribution is extremely imbalanced. The Method facet takes about 60% proportion of the total data, and there are only 9 instances in the Hypothesis facet. Trained on the extremely biased dataset, many classifiers such as SVM and Naive Bayes tend to classify all the data into the Method facet. Although this practice achieves high overall accuracy, it is not a proper solution. Therefore, we introduce two metrics to measure the performance, i.e., the macro-averaged accuracy (AM ) as well as the micro-averaged accuracy (Am). Their formulas are as follows: Am = PPcc22CC Nrcc AM = 1 X
rc jCj c2C Nc (1) (2) where C stands for the class set, rc is the right number in the Class c, and Nc is the actual number. Just predicting the Method facet, Am = 0:59 and AM = 0:2.
We focus on the increase of the macro-averaged accuracy. To this end, we use the
decision tree to conduct hierarchical classification. The most important advantage of
the decision tree is that it has the ability to remember patterns of all the facets in the
training data. In comparison, SVM and Naive Bayes are likely to merely reserve the
patterns of the dominant class.
Since the instances for the Implication and Hypothesis facets are very limited, we only
train the classification model on the data of the other three facets. We use the tf-idf
vector of the citance as features. Notably, the vocabulary size is too large with respect
to the training data. We thereby introduce the 2-test to reserve the most significant 125
features. The learned decision tree is displayed in Fig. 4. We can find in the leaf nodes,
there is often just one support case. It seems this decision tree is still quite over-fitted.
We also consider to add more features such as the position of the citance in the citation
paper. However, the result shows these features only make the decision tree more biased
to the Method facet.
This task requires to generate a summary for the reference paper with the help of
citances. Our summarization system makes full use of the structure information in the
corpus. We regard a section of a paper as a document, since sentences in important
sections should also be important. Meanwhile, we treat the citance as a query. The idea
behind is that sentences relevant to citance may be the focus of the paper. After the
above two steps, Task 2 is converted into the query-focused multi-document
summarization problem. Then we develop an enhanced version of Manifold Ranking [
W = Winter + Wintra (3) Then, it gives different weights for these two matrices.
Wf =
1Winter + 2Wintra We fix 1 = 1, and change 2 in the experiments. If 2 < 1, inter-document links are more important than the intra-document links in the algorithm and vice versa. Note that if 2 = 1, Equation 4 reduces to Equation 3.
Subsequently, We normalize the similarity matrix Wf into a probability matrix S.
S = G 1=2WfG 1=2 where G is the diagonal matrix with (i; i)-element equal to the sum of the ith row of Wf. With the probability matrix S, we can now apply the random walk algorithm to compute the saliency scores f of the sentences: f (t + 1) =
Sf (t) + (1 )y; where is a weight parameter, and y is the prior score distribution. In Manifold Ranking, y is set as follows:
yi = 0;1=ko;theirwikse (7) It means the query-relevant sentences are most important in the prior.
We improve Manifold Ranking by modifying the prior score distribution to inspect the importance of citances. We introduce an extra parameter 2 [0; 1] to control the weight of citances, and Eq. 7 becomes:
=k; i k yi = n1 k ; otherwise Maniflod Ranking is a special case where = 1. = 0; 0:5; 1 respectively. There are three parameters in our summarization model, i.e., the intra-document weight 2, the random walk weight and the citance weight . We set = 0; 0:5; 1 respectively and conduct grid search on the development set to check the change of performance. The result is shown in Fig 5. As can be seen, = 0:5 shows the highest performance potential. Meanwhile, when 2 > 0:6, the performance is not sensitive to its change. For , a common value of 0:85 often works well. To sum up, we choose = 0:5, 2 = 0:8 and = 0:85 for the test dataset. (4) (5) (6) (8) 5
This document demonstrates our participant system PolyU on CL-SciSumm 2016. There are three tasks in CL-SciSumm 2016. In Task 1A, we apply SVM Rank to identify the spans of text in the reference paper reflecting the citance. In Task 1B, we use the decision tree to classify the facet that a citance belongs to. Finally, in Task 2, we develop an enhanced Manifold Ranking summarization model.