This paper introduces the methods and experiments applied in CIST system participating in the CLSciSumm 2016 Shared Task at BIRNDL 2016. We have participated in the TAC 2014 Biomedical Summarization Track, so we develop the system based on previous work. This time the domain is Computational Linguistics (CL). The training corpus contains 20 topics from Training-Set-2016 and Development-Set-Apr8 published by CL-SciSumm 2016. As to Task 1A and 1B, we mainly use rule-based methods with various features of lexicons and similarities; meanwhile we also have tried the machine learning method of SVM. As to Task 2, hLDA topic model is adopted for content modeling, which provides us knowledge about sentence clustering (subtopic) and word distributions (abstractiveness) for summarization. We then combine hLDA knowledge with several other classical features using different weights and proportions to evaluate the sentences in the Reference Paper from its cited text spans. Finally we extract the representative sentences to generate a summary within 250 words.
There are many researches about document summarization [
However, it has the disadvantage of relying on ideal summaries. [
In recent years, interest about information extraction and retrieval from scientific literature has increased
considerably. Some researches [
In order to identify the linkage efficiently between a paper citation and its cited text spans in the RP, we need to
catch the deep meaning of natural language sentences. In fact, digging the deep meaning of sentences also has an
important sense for information extraction and retrieval. Besides the traditional methods for sentence similarity
calculation, recently the open-source tool released by Google -- word2vec [
There are two tasks in the CL-SciSumm 2016. Testing dataset, development dataset and training dataset, each
contains 10 topics. Every topic consists of one reference paper (RP), some citing papers (CP) and one annotation file.
There are five facets pre-defined, including Aim_Citation, Method_Citation, Results_Citation, Implication_Citation
and Hypothesis_Citation. In Task 1, we need to identify the spans of text (cited text spans) in the RP that most
accurately reflect the citance (Task 1A), and what facet of the paper that each cited text span belongs to, from the
predefined set of five facets (Task 1B). Task 2 demands us to generate a structured summary of the RP from the cited text
spans of the RP. The length of the summary should not exceed 250 words. Please refer to [
We need to identify the linkage between a paper citation in the CP and its cited text spans in the RP. We think that the linkage is mainly represented by similar meaning between sentences. Hence our work is focused on sentence similarity computing based on various features.
Feature Extraction.
We have two kinds of features, one is from lexicons, and the other is from sentence similarities. Three Lexicons: 1. Reference Text high-frequency words (Lexicon 1): We picked up the words with high frequency from reference text in the training corpus artificially, and then expanded them through Wordnet and word vectors. 2. LDA lexicon (Lexicon 2): We used LDA (Latent Dirichlet Allocation) model to train the reference paper and citing papers to get a lexicon of 30 latent topics for files in every topic independently. Table 1 displays the relation of F-measure and the number of latent topics for the training corpus. We can see that the best performance is obtained when the number of latent topics is set to 30. 3. Citation Text and Reference Text co-occurrence lexicon (Lexicon 3): We obtained the co-occurrence degree between words by the word frequency statistics of citation text and its reference text from the training corpus. Similarity between two sentences: 1. Idf similarity: We add up the idf (inverse document frequency) values of the same words between two sentences.
where | | is the number of sentences, {: ∈ } is the number of the sentences which contain the word . 2. Jaccard similarity: We use the division between the intersection and the union of the words in two sentences.
where A and B represent two sentences respectively. Similarity between two contexts:
Where is the context similarity of , −1 and +1 is the imilarity of the sentence before and after it. (1) (2) (3) =
| | {: ∈ } = ∩
∪ = √ −1 ∗ +1 We also obtained two kinds of context similarities: Idf context similarity and Jaccard context similarity. Two vector similarities: 1. Word vector: We trained every word as a vector with 200 dimensions using Word2Vec. Then a sentence is: = ( , +1 , ⋯ , + ) where is the word vector and W is the vector set to represent a sentence. The vector sets of two sentences W and W can form a matrix M, : where ( , ) is the cosine similarity of and , then the similarity of and is: , = = [
⋮ + ⋯ ⋱ +
⋮ ⋯ + + 2. Doc vector: We represented every sentence as a vector with 200 dimensions by doc2vec, and used the cosine similarity between vectors to represent the sentence similarity.
We tested the performance of every feature independently for the training corpus as in Table 2. As we can see that Jaccard similarity performs the best.
125 times of the number of positive samples. So we divided the negative samples into 125 groups and train each group with all positive samples. Then using the trained 125 SVM models, we could predict the testing data and get 125 results. Finally, we choose the linkage sentences through a voting system based on these 125 results. When we tried this method on the testing data, unfortunately the performance was not so good as feature rule-based methods. As for the reason, we thought that the number of positive samples was too small, and every training set didn’t have enough data to train a good model. To find a better performance, we tried the following methods. Voting Method:
Focusing on the F-measure of every feature, we tried different weights and proportions to combine them through experiments, and then got two results through a voting system.
Jaccard Focused Method: Jaccard Cascade Method: Task 1B. Rule-based method 1. Subtitle Rule: First of all, we examine whether the subtitles of reference sentences and cite sentences contains the following facet words: Hypothesis, Implication, Aim, Results and Method. If the subtitle contains any one of these words, it will be directly classified as the corresponding facet. If it contains more than one of these words, it will be classified into all the facets. Else if it contains none of them, we just classify it as the facet of Method. 2. High Frequency Word Rule: According to the High Frequency Word Rule, we firstly count the High Frequency Word of five facets from the Training Set and the Development Set. In order to improve the coverage of sentences, we expanded the High Frequency Word to get some similar words of each facet. We set an appropriate threshold for each facet. If the number of the High Frequency Word of any facet in the sentence is more than the corresponding facet threshold, then we just use the facet whose coverage is the highest as the final class. If some facets’ coverage are same, then we just classify according to the sequence of Hypothesis, Implication, Aim，
3. Combine Subtitle and High Frequency Word Rule: We firstly use the Subtitle Rule to classify the testing set. If the results are not in the five facets of Hypothesis, Implication, Aim, Results and Method, then we use the High
SVM Classifier
We extract four features of each class. 1) Location of Paragraph: the order number of the paragraph in which the sentence is located. 2) Document Position Ratio: the ratio of sentence Sid to the total sentence number of the corresponding document. 3) Paragraph Position Ratio: the ratio of sentence Ssid to the total sentence number of the corresponding paragraph. 4) Number of Citations or References: the number of Citation Offset or Reference Offset. These features form an 8-dimension vector of a pair of reference sentence and citation sentence. We train SVM to get five classifiers. For the problem of unbalanced training data, we set different weights for different classes. If we cannot get any class of the five facets, then we classify it as Method class. Voting Method Fusion Method
We combine the results from Subtitle Rule, High Frequency Word and SVM classifier to generate the final results with most votes.
We run the above methods for each run result we obtained in Task 1A and choose a best one as the final result. Then we also tried a fusion method to combine all the run results of the above methods obtained in Task 1A. We counted the number of Method, Results, Aim, Hypothesis and Implication, and set an appropriate threshold for each facet class to get a final result of facet class. 4.2
The source documents provided by CL-SciSumm 2016 have some xml-coding errors. Besides, we need specific data format to train our hLDA feature. To obtain relatively a high-quality input dataset, we do pre-processing.
Merge Documents
Remove Stopwords Generate hLDA Input
File word Leveldistribution sentencelength sentenceposition cited-text
spans
RST-feature
1. Document merging: For each topic, we merge the reference paper and the cited text spans into a big text.
2. Stop words removing: We use the stop-word list to remove stop words. The stop-word list contains punctuation
marks and some functional words. At the same time, all capitalized characters are changed to lower case.
3. Input file generation for hLDA: For the selected words, we build a dictionary for each document, which contains
words and their corresponding frequency, and the index starts from 1 to word list size. Finally, we generate an
input file for hLDA modeling, in which each line represents a sentence presented by word index - frequency pair,
such as: [number of words in sentence i] [word-index A : frequency A] [word-index B : frequency B] …
Feature modeling.
1. hLDA-level distribution feature: To be known as exact description of latent topics over nested Chinese restaurant
process, hLDA is one of the non-parametric topic models. Given the input file, unsupervised-hLDA generates
tree-like topic structures over documents. Each sentence is assigned to a path starting from the root node to leaf
nodes. Sentences sharing the same path are similar with each other and thus constitute a subtopic of the
documents. Each node in the tree contains different words, whose distribution is referred to latent-topic. Besides,
different level contains different nodes. Each word in a sentence is randomly assigned to the node at different level
in the sentence path with some probability, and the probability to different level is the level message we would
like to exploit which can represent the abstractiveness of the topic in the document to some extent. Since we have
obtained the predefined facet information of cited test spans in Task 1B, we do not use the subtopic knowledge in
hLDA here which may not directly match the five facets as defined by CL-SciSumm 2016. [
∑=0 ( − + − ) where indicates the weight of the node level that word is assigned in, − indicates the score of level distribution of word , −
is the frequency of in current node as followed. (7) (8) (9) (10) (11) − = ∑ ( =0 ( )
) ℎ = √2 1 −(−) 2
2 = 1 − 2. Sentence-length feature: It is a balance between length and sentence meaning. Gaussian Distribution is chosen in order to get a better combination along with the level feature.
where μ indicates the expectation of word frequency, σ indicates the standard deviation of word frequency. 3. Sentence-position feature: Sentence position is widely used in the summary extraction of news and science paper.
where n is the total number of sentences, pi is the position of i − th sentence. Number i is started from 0. 4. Cited text span: A citation is a summary of other paper authors, and we got all cited text spans of each reference paper from Task 1A. Thus we use it as a weak feature, for the reason that there must be some errors. = ( ) = { 1, 0, ℎ 1
We use the Discourse Facet answer in Task 1B to provide a reference information for sentence extraction. 5. RST-feature: Rhetorical Structure Theory (RST) is the notion of rhetorical relation, a relation existing between two non-overlapping text spans called nucleus (N) and satellite (S). Text coherence in RST is assumed to arise due to a set of constraints and an overall effect that are associated with each relation. The constraints operate on the nucleus, on the satellite, and on the combination of nucleus and satellite. We call each sub-string cut by punctuations as a simple sentence. Using the RST-software we developed ourselves, we can get the RST-score of each simple sentence, then we have RST-score of a sentence as followed. where −
is the RST-score of simple sentences, is the total number of simple sentences in a sentence.
bine them by a linear formula as shown below:
With the features above, we combine them to get a better summary result. A simple and efficient way is to com = 1 − + 2 ℎ− + 3 − + 4 − + 5 − where indicates feature weight.
In Task 2, we need to generate a structured summary of the RP from the cited text spans of the RP. The length of the summary should not exceed 250 words. Besides, Task 2 pays more attention to the summary extracted from the third part, so we change the first person of the extracted sentences to third person. In order to extract a high-quality structured summary, we make full use of the prior knowledge that structured summary contains four parts: Introduction, Methods, Results and Conclusion. So we use the Discourse Facet to help extract the summary sentences. We will extract two or three sentences for each part of the summary if exists. Furthermore, we remove redundant candidate sentences. We use the following algorithm 1 to extract summary sentences.
for paper in Test-Set: stop word removing input file generation for hLDA modeling calculate scores of five features while length ≤ 250 extract sentence according to feature scoring if sentence is not redundant and length <= 250 add to candidate set person transformation convert xml formed paper to txt, including title document merging of paper and cited text spans
To find the best use of hLDA level features, we also proposed two methods to extract level feature sentences. The first one is that, we ignore the clusters of the tree, only use the level feature score to choose best-N sentences. The other one is that each step extracts one sentence from one cluster in the first half of all clusters and adds it to the summary until the summary length is 250. We call the first one as ScoreDesc, the second one as In First Half Path.
5 5.1
There are 5 runs that we submitted. From Run1 to Run4, we used the above mentioned methods respectively. Then, we got the performance of training dataset as followed in Table 7. In Run5, we used the SVM Method. And the accuracy we got in training set is 80.59% as a closed testing. We also have tested our systems on the testing dataset with the golden standard and got the results as in Table 8.
P Jaccard Cascade 0.05376 0.15831 0.08027 SVM 0.02222 0.08179 0.03495
Table 8. Task 1A results of testing dataset
From Table 7 and Table 8, we can see that there isn’t much difference among the first 4 methods. Although the performance of SVM on training dataset is good, its performance on the testing dataset is very poor. We think that the major reason comes from the shortage of training data. In spite of this, we still set it as one of our system runs. Because we thought that this is an important direction. Later maybe we can improve this method using other strategies like semi-supervised or unsupervised machine learning with more unlabeled data.
After more detailed analysis of the results, we can find two major problems. One is choosing too much sentences, in which the citation text in standard answer contains only one fitful sentence, but we chose more sentences. We got the right answer with the price of more error sentences included. The other is partial answer. For instance, there are 3 sentences in standard answer covering the two classes of features, yet we found only 1 and lost 2 of them with 3 error sentences covering only one class of features. Although there may exist some relevance between the Citation Text and Reference Text in our answers, the standard answers are much better with good precision and complete coverage. Our system cannot achieve this status. We need better methods to locate the best answer accurately and cover all aspects of the citation text. Our experiment is based on a hypothesis that the training dataset and the testing dataset have similar structures. Thus we can use the parameters learned from the training dataset to the testing dataset evaluation. After adequate experiOur system has tried to add some semantic information like word vector, doc vector and word distributions in hLDA tree to improve the citance linkage and summarization performance. Yet the result isn’t very satisfied. Our future work is to find some better ways to mine and use more semantic features for citance linkage. As to summarization, we will try to add more predefined semantic features to unsupervised hLDA model and sentences combination and compression for better summary sentences. Furthermore, we will try to find a better method to choose least sentences covering most information.
This work was supported by the National Natural Science Foundation of China under Grant 91546121, 61202247, 71231002 and 61472046; EU FP7 IRSES MobileCloud Project (Grant No. 612212); the 111 Project of China under Grant B08004; Engineering Research Center of Information Networks, Ministry of Education; Beijing Institute of