This paper introduces NJUST system which is submitted in CLSciSumm 2018 Shared Task at BIRNDL 2018 Workshop. The training corpus contains 40 articles which are created by randomly sampling documents from ACL Anthology corpus and selecting their citing papers. Overall, there are three basic tasks in CL-SciSumm 2018. Task 1A is to identify cited text spans in reference paper. Briefly, we use multi-classifiers and resemble their results via voting system. Meanwhile, we also submit results generated via single classifiers. For task 1B, which is to identify facets of cited text, except rule-based methods using human-labeled and POS dictionary, we also apply supervised topic modeling and gradient boosted decision trees. As to Task 2, after organizing identified sentences into groups based on their similarities between abstract sentences, we rank them using several features and generate a summary within 250 word by selecting the top ones.
Nowadays, increasement of publications makes researchers hard to catch up with the
progress in fields. In order to provide readers a quick overview of papers, scientific
summarization has arisen people’s attentions. Since citation sentences (citances)
usually provide useful information about reference papers, researchers were focusing on
citation-based summarization by aggregating all citances that cite one unique paper [3].
However, detailed information cannot be revealed enough in citation texts, and
viewpoints of the citing authors can also be different from each other due to citing purposes
[4]. Recently, a number of shared tasks like, TAC 2014 Biomedical Summarization
Track1, Computational Linguistics Scientific Document Summarization Shared Task
Below is the detailed information of tasks.
Given: A topic consisting of a Reference Paper (RP) and Citing Papers (CPs) that all contain citations to the RP. In each CP, the citances have been identified that pertain to a particular citation to the RP.
Task 1A: For each citance, identify the cited text span in the RP that most accurately reflect the citance. These are of the granularity of a sentence fragment, a full sentence, or several consecutive sentences (no more than 5).
Task 1B: For each cited text span, identify what facet of the paper it belongs to, from a predefined set of facets.
Task 2: Finally, generate a structured summary of the RP from the cited text spans of the RP. The length of the summary should not exceed 250 words.
Referring to our previous work in CL-SciSumm 2017 [5], multiple classifiers are integrated based on a weighted voting system to identify cited text spans. Based on that, we did some optimizations for Task 1A from aspects of feature selection, class-imbalanced data processing, voting weights allocation and parameter tuning [6]. While in system applied in CL-SciSumm 2018, we conduct the similar strategy with multi-classifiers in Task 1A, but adding new steps to process data, new features for classifiers and new classifiers as well. For Task 1B, we try to identify facet by supervised topic modeling and classifier except using built dictionaries. Final results are combined between strategies. When doing summarization in Task 2, we firstly separate sentences based on their similarity to abstracts and rank them over several features to select important ones for summary generations.
The rest of paper is organized as follows. Section 2 provides a brief review of related works. Section 3 elaborates the detailed information about our system this year. Experimental data and evaluation results on training data are given in section 4. Conclusion and direction for future research are outlined in section 5.
3 Available at: http://wing.comp.nus.edu.sg/~cl-scisumm2017/ 4 Available at: http://wing.comp.nus.edu.sg/~cl-scisumm2018/
With million publications are coming out every year [7], attention has been paid in automatic scientific summarization due to people’s demand for getting quick overviews. Recently, Computational Linguistics Scientific Document Summarization Shared Task are the first annual medium-scale shared task on scientific summarization, where summary is generated from identified cited text. This year, CL-SciSumm 2018 took place at the Joint Workshop on BIRNDL 20185 with the same goal of exploring automated summarization of scientific contributions for computational linguistics domain. Here, we do literature review of different tasks based on submitted systems in CL-SciSumm 2016 and CL-SciSumm 2017 [8].
Looking at the related work of Task 1A, most teams solved it by characterizing the
linkage between a citance in citing paper and its corresponding cited text spans in
reference paper [9]. Features are basically generated based on character-based and
semantic-based similarities. For example, in CL-SciSumm 2016, CIST System applied lexical
similarity and sentence similarity [10]. Aggarwal and Sharma [11] made use of
subsequences overlap. PolyU utilized TF-IDF cosine similarity, position of sentence chunk
and some lexical rules [12]. Other relevant features app
As for Task 1B, proportions of different discourse facet types are very imbalanced, most proposed methods are using rule-based methods, which is based on human-labeled dictionaries or some heuristics. Aggarwal and Sharma [11] identified the facet based on cited text span location, such as if cited text span lies in introduction section, beginning of abstract, then it is indicative of aim citation. CIST System took advantages of frequent word and combined it with subtitle to do judgements[10, 14]. Besides, different classifiers are also applied here, such as random forest classifier[19], SVM [14], SMO [20], convolutional neural networks [17]and so on. Except position and similarity features, new ones are proposed, like Dr inventor sentence related features and scientific gazetteer features in [15].
When doing Task 2, basically, there are two main steps. First is to cluster identified text spans to organize them into groups. Second is to rank them based on different features, which depict sentence importance in some level. CIST system calculated sentence scores of five features [10]. In order to control redundancy of summary, they used determinant point processes to enhance diversity [14]. Abura’ed, Chiruzzo [15] proposed a modified version of 2016 summarization system with additional features which are relevant with reference paper and citing paper.
As mentioned in introduction, there are two tasks. The dataset comprised 40 annotated sets of references and their citing papers from the open access research papers in the computational linguistics domain. A topic is consisted of a Reference Paper (RP) and Citing Papers (CPs) that all contain citations to the RP. In each CP, the text spans (citances) have been identified that pertain to a particular citation to the RP. 3.1 In this paper, we solve Task 1A by finding the sentence in RP that is more similar with citance. There are two main steps in our system: selecting suitable features for classifiers, integrating final results via a weighted voting system. Here are the detailed information about our system for conducting Task 1A.
Citation Text Preprocess. Since training data is labeled by human which might have some errors, we utilize two rules to expand labeled citation text in advance which can rich semantic information of citation text: First, if the next sentence behind labeled citation text contains the same author name in citation text (Example in Paper [1]), then we add this sentence into citation text. Second, if the next sentence behind labeled citation text contains demonstrative pronouns (Example in Paper [2]), then we add this sentence into citation text. We do this preprocess on training and testing data directly. For training data, there are 4,244 sentences are added into original citation texts.
Paper [1]
Like others, we have assumed lexical semantic classes of verbs as defined in Levin
(1993) (hereafter Levin), which have served as a gold standard in computational
linguistics research
To select relevant features for use in model construction, we firstly tested each feature with four classifiers, including Decision Tree (DT), Logistic Regression (LR), SVM (kernel function is linear and RBF). We select negative and positive samples in different class ratios: 1, 2, 3, 4, 5 and 6 to investigate performance stability using different training datasets. Figure 2 displays the average F1 values of different featureclassifier combinations. 0.5
In order to pick out the best feature combinations, we conduct subset selection by iteratively evaluating a candidate subset of selected features set. Based on Figure 2, for each classifiers, we choose features which are the most robust among different class ratios and have good performance to be the fixed features. Less robust features are selected to be the selected features set. We set class ratios of negative and positive samples to be 5.5. Table 2 to Table 5 shows the fixed feature and selected feature sets for each classifier and their performance of precision, recall, F1.
As we can see, Decision Tree and Logistic Regression are performing better than SVM (Linear and RBF). Therefore, when doing integrations over classifiers, we construct two voting system, one is 4-classiferis containing all classifiers, another one is 3classifiers where we remove the SVM (Linear).
Parameter Setting. In this system, voting weights of multi-classifiers and running setting are important parameters to adjust. Based on Table 2 to Table 5, we compute the average of precision, recall, F1 for each classifier and use these average values as the voting system weights. Since the SVM (Linear) behave worst among all four systems, we do another voting system which only based on the other three classifiers. Voting weights for 4-classifiers and 3-classifiers are shown in Table 6 and Table 7.
New Classifier. Except the classifiers we applied before, we also utilize a new one, called XGBOOST, which is an efficient and scalable implementation of gradient boosting framework by [21]. We use it as a single classifier with integrating into the voting system. When testing on training data, we select negative and positive samples in: 2, 3, 4 and 5. Figure 3 shows the average F1.
2 3 4
5
Therefore, we also choose the fixed feature (bigram, IDF similarity and WordNet similarity) and selected feature sets (LDA similarity and Doc2Vec similarity) for XGBOOST and test again on training data when negative/positive samples, penalty factor are 5.5, 6, 6.5 and 7. Their performance of F1 are show in Table 8 below.
In this task, for each cited text span, we need to identify what facet of the paper it belongs to. Basically, there are three components in this system to deal with Task 1B. Dictionary: We construct two kinds of dictionaries of five facets manual dictionary, and POS dictionary. The first one is made manually and latter one is made according to part-of-speech tagging results. For POS dictionary, we keep those words whose POS results are VB and JJ. In detail, method POS dictionary has words which frequency is over 5, and for the other facet POS dictionary, they has words which frequency is over 2. Supervised Topic Model: After proposing of latent sematic indexing, latent topic modeling has become very popular for topic discovery in document collections, such as Latent Dirichlet Allocation (LDA) [22]. Supervised topic model (LLDA) [23] is then followed by, which can overcome limitations of traditional ones. This model assumes availability of topic labels (keywords) and the characterization of each topic by a multinomial distribution over all vocabulary words. XGBOOST: Tree boosting is a highly effective and widely used machine learning method. Here we apply XGBOOST [24] for approximate tree learning. When training the model, there are 15 features in total. Five of them are the matched word number based on manual dictionary, five of them are the matched word number based on POS dictionary, and the left ones are position-based features mentioned in section 3.1.
Based on three components above, there are five different strategies: Manual Dictionary. Based on the five different dictionaries of five facets, if the section title or sentence content contains any one of these words in the corresponding built dictionaries, it will be directly classified as the corresponding facet. Since the manual dictionary will be more accurate than POS dictionary. We only apply this strategy using manual dictionary. When doing judgements, the first identified facet should contain more than 1( 1) word in dictionary, the second identified facet should contain more than 2( 2) words in dictionary. To find the best order of judging facets, we do the experiments over all random arrangements. In total, there are 120 sets of results, here we only show the top 20 ones based on F1 in Table 9. implication->method->result->aim->hypothesis 0.7179 method->result->hypothesis->implication->aim 0.7159 implication->method->result->hypothesis->aim 0.7179 method->result->hypothesis->aim->implication 0.7159 implication->method->hypothesis->result->aim 0.7179 method->hypothesis->result->implication->aim 0.7159 implication->method->aim->result->hypothesis 0.7162 method->hypothesis->result->aim->implication 0.7159 implication->method->aim->hypothesis->result 0.7162 implication->hypothesis->method->result->aim 0.7146 implication->method->hypothesis->aim->result 0.7162 hypothesis->implication->method->result->aim 0.7146 method->result->implication->aim->hypothesis 0.7159 method->implication->result->aim->hypothesis 0.7146 method->result->implication->hypothesis->aim 0.7159 method->implication->result->hypothesis->aim 0.7146 method->result->aim->implication->hypothesis 0.7159 method->implication->hypothesis->result->aim 0.7146 method->result->aim->hypothesis->implication 0.7159 method->hypothesis->implication->result->aim 0.7146 LLDA. For training data, we assume that each identified facet is a topic label and that each citation sentence is a mixture of the expert-assigned topics that can be learned. We firstly trained LLDA model on the training data and the dimension number is five. Then, we apply this trained model to do predictions over testing data. Here, there is no labels for testing data yet. After representing each sentence into the probability distribution over five facets, we recognize the most possible facet as its identified facet. Since some sentences might have more than one facets, we set the possibility thresholds ( 2 = 0.2 0.195) for the second possible facet. Referring to LLDA parameters, we do some adjustments on beta, where a low beta value places more weight on having each topic composed of only a few dominant words. Table 10 shows different beta settings and their corresponding F1.
0.1 0.2 XGBOOST. Here, we use the XGBOOST to do classification in this task. When choosing features, position-based features mentioned in section 3.1 are selected as selected feature set which will be evaluated using its candidate subsets. Performance of different selected feature sets are given below in Table 11. Manual dictionary + LLDA. Different from LLDA strategy, we use the manual dictionary-labeled testing data to be the testing data for LLDA prediction. Here, we also set the possibility thresholds for the second possible facet ( 2 = 0.18) and the thresholds for contained word counts of the first and second identified facet when doing different order of judgements ( 1 = 1 and 2 = 2) To find the best order of judging facets, we also do the experiments over all random arrangements. Here we only show the top 20 ones based on F1 in table 12. POS dictionary + LLDA. Similar with previous method, we use the POS dictionarylabeled testing data to be the testing data for LLDA prediction. We also set the same three parameters in this strategy, where 2 = 0.18, 1 = 3 and 2 = 8. The top 20 F1 via different judging order is give in Table 13. method->implication->result->aim->hypothesis 0.7511 method->implication->aim->result->hypothesis 0.7498 method->implication->result->hypothesis->aim 0.7511 method->implication->aim->hypothesis->result 0.7498 method->implication->hypothesis->result->aim 0.7511 method->implication->hypothesis->aim->result 0.7498 method->hypothesis->implication->result->aim 0.7511 method->aim->implication->result->hypothesis 0.7498 method->result->implication->aim->hypothesis 0.7498 method->aim->implication->hypothesis->result 0.7498 method->result->implication->hypothesis->aim 0.7498 method->aim->hypothesis->implication->result 0.7498 method->result->aim->implication->hypothesis 0.7498 method->hypothesis->result->implication->aim 0.7498 method->result->aim->hypothesis->implication 0.7498 method->hypothesis->result->aim->implication 0.7498 method->result->hypothesis->implication->aim 0.7498 method->hypothesis->implication->aim->result 0.7498 method->result->hypothesis->aim->implication 0.7498 method->hypothesis->aim->implication->result 0.7498 3.3
Summary generation is divided into two main steps. First is to group sentences into different clusters based on its similarity with different parts of abstract. Second is using several features to extract sentence from each cluster and combine them into a summary.
Normally, abstract is a complete but concise description of the work. In particular, different parts may be merged or spread among a set of sentences, like motivation, problem statement, approach, results and conclusions. Therefore, we wants to organize the abstract sentences of reference paper in advance, and group the identified cited spans based on their similarities between different parts of abstract sentences. Basically, we assume that abstract will contain motivation, approach and conclusion. In order to split them into these three group, we apply rule-based method based on writing styles. We find that when people write summaries like abstract, they will start with some fixed phrases, such as “this paper”, “in this paper” or “we”. If the first sentence doesn’t have these fixed phrases, it will be about motivation of this paper for most of the time. Meanwhile, the last sentence are usually about results or conclusions.
Therefore, we firstly split abstract sentences into groups if they follow these rules. Then, each identified text span is selected into different groups based on their similarity with the grouped abstract sentences. Here we use the linear sum of Jaccard, IDF and TFIDF similarities. After this, we rank the sentences within each group, using weighted features of those three similarities, sentence length and sentence position. Formula is shown below: (1)
Finally, for each time, we choose first one sentence from each cluster to build the summary before the length of summary exceeds 250 words. 4 4.1
When doing corpora preprocessing, we remove the stop words and stem words to base forms by Porter Stemmer algorithm6. Then, we applied Word2Vec and Doc2Vec model in Genism7 and python package of LDA8 model to represent documents. All the classifiers were done via Scikit-learn python package9. XGBOOST is obtained via a python extension package website10. Source code of our system will be made available at: https://github.com/michellemashutian/NJUST-at-CLSciSumm/tree/master/NJUST2018. 4.2
Task 1A. After using the best feature combinations on 4-classifiers and 3-classifiers, testing on different parameters, we obtain the average F1 shown in Figure 4. Proportion of negative/positive samples, penalty factor are tested on 5.5(blue cross line), 6 (red circle line), 6.5 (green triangle line) and 7 (purple square line). Thresholds range from 0.6 to 0.8, as 0.01 is the interval (x axis). 6 Available at: http://snowball.tartarus.org/algorithms/porter/stemmer.html 7 Available at: https://radimrehurek.com/gensim/ 8 Available at: https://pypi.org/project/lda/ 9 Available at: http://scikit-learn.org/stable/index.html 10 Available at: https://www.lfd.uci.edu/~gohlke/pythonlibs/#xgboost (a) Average F1 when using Precision-Oriented 3
Classifiers Voting System (b) Average F1 when using Precision-Oriented 4
Classifiers Voting System (c) Average F1 when using Recall-Oriented 3-Classi- (d) Average F1 when using Recall-Oriented 4-Clasfiers Voting System sifiers Voting System
According to Figure 4, we pick the Top 10 performance of multi-classifiers and their parameters are given in Table 14. Except voting system, we also submit another 10 running results which are obtained via single classifiers. Parameter and classifier features are given in Table 15.
DT DT LG
LG SVM(RBF) SVM(RBF) XGBOOST XGBOOST XGBOOST XGBOOST Task 1B. Referring the five strategies using dictionary, based on the performance of different judgment order (Table 9, Table 12 and Table 13), we select the specific order according to their F1 results, when they generate the same facet identification on testing data, we just move to next order which has lower F1. For LLDA strategy, we pick the top 4 results with corresponding beta settings to run on test data. For XGBOOST strategy, we also select top 4 results with corresponding feature selections to run on test data. Table 16 shows the overall parameter settings of our Task 1B submission. = 1.2, 2 = 0.2 LLDA = 1.2, 2 = 0.195 = 1.5, 2 = 0.2 = 1.5, 2 = 0.195 implication->hypothesis->method->result->aim, Manual Dictionary implication->method->result->aim->hypothesis, method->result->implication->aim->hypothesis, = 1.2, 2 = 0.18, implication->method->result->aim->hypothesis, Manual Diction- = 1.2, 2 = 0.18, ary+LLDA method->result->implication->aim->hypothesis, = 1.2, 2 = 0.18, implication->method->result->aim->hypothesis, = 1.2, 2 = 0.18, method->implication->aim->result->hypothesis, POS Diction- = 1.2, 2 = 0.18, ary+LLDA method->implication->result->aim->hypothesis, = 1.2, 2 = 0.18, method->result->implication->aim->hypothesis, sid, sid_position XGBOOST sid, inner_position sid sid, sid_position, inner_position, section_position 1 = 1 and 1 = 1 and 1 = 1 and 1 = 1 and 1 = 1 and 1 = 1 and 1 = 3 and 1 = 3 and 1 = 3 and 2 = 2 2 = 2 2 = 2 2 = 2 2 = 2 2 = 3 2 = 8 2 = 8 2 = 8 5
This document demonstrates our participant system NJUST on CL-SciSumm 2018. Compared with previous system, we has added some semantic information like WordNet and Word2Vec similarities to improve the citance linkage and summarization performance. We also optimize the bigram feature. When choosing feature and setting parameters, comparative experiments are finished systematically. New methods are proposed in this paper to deal with facet identification and automatic summarizations. In Task 1B, rule-based methods are combined with supervised topic modeling and XGBOOST. As to Task 2, we take advantages of abstract structures.
In the future work, more things can be done on these three tasks. For Task 1A and Task 1B, we can try new classifiers to see the performance. For Task 2, we need to find more features to calculate the sentence score for ranking, such as sentence position, etc. We can also make use of the results in Task 1B to generate a more reasonable summary.
This work is supported by Major Projects of National Social Science Fund (No. 17ZDA291), Fujian Provincial Key Laboratory of Information Processing and Intelligent Control (Minjiang University) (No. MJUKF201704) and Qing Lan Project. 12. 13. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24.