In this paper, we introduce the NUDT system for this year's CLSciSumm 2018 task at the BIRNDL 2018 Workshop. For task 1a, we identify the related text spans referred to the citation with random forest model, exploring multiple features. Additionally, we integrate random forest model with BM25 and VSM model and apply a voting strategy to select the most related text spans. Besides, we explore the language model with word embeddings and integrate it into the voting system to improve the performance. For task 1b, we use multi-features random forest classifier to identify the facet of the cited sentences.
The rapid growth of scientific papers and the need for a researcher to move into
another brand-new domain generate the demand of scientific summarization. Scientific
summarization has been studied for years since (Simone et al ,2002)[
The CL-SciSumm18 task can be dated back to the BiomedSumm Track at the Text Analysis Conference 2014, which concentrates on the biomedical dataset. In the next two years, the CL-SciSumm task was held respectively as part of the Joint Workshop on BIRNDL at JCDL and SIGIR.
The CL-SciSumm task of this year is also organized as part of SIGIR2018. On contrast with CL-SciSumm2017, it has increased 10 articles to the training corpus (up to 40 articles) and a new test set of 10 articles is released this year. The task description is as follows: • Task 1B: For each cited text span, identify what facet of the paper it belongs to, from a predefined set of facets • Task 2: Finally, generate a structured summary of the RP from the cited text spans of the RP. The length of the summary should not exceed 250 words.
In this paper, we describe our methods which are used to solve task 1a and 1b. As for
task 1a, we first consider to regard the task as an information retrieval problem and
draw on the method of [
There has been a large number of related works [
For the text spans identification according to citations, the methods can be
categorized into two classes, classification task and retrieval task. The former methods
include [
For facet identification, many teams used bag of words methods [
For the task of summary generation, [
In this section, we describe the method that we use to identify the related text spans in the reference paper in detail. 3.1
The official dataset1 of CL-SciSumm18 comprises 40 annotated sets of citing and
referenced papers in the training set and 10 in the test set. Since the papers are
transformed from PDF format to XML or TXT format, there exists a bunch of format
mistakes and futile characters in the dataset. Hence, it’s essential to preprocess the
sentences in the dataset before we set out to deal with the task.
• Sentence processing: we use NLTK to tag the part of speech and move
punctuations and stop words.
• Sentence filtering: based on the former step, we try to filtrate sentences which have
apparently more unreasonable characters than those that are more likely to be
candidate of the retrieved sentences. To find out the error threshold, we establish an
English word dictionary composed of 103976 English words2 and view it as a
judger to determine a word legal or not. We count the ratio of illegal words in each
of the 566 cited sentences of reference papers in the training set and choose the
error ratio threshold as 0.4, which means sentences comprise 40% or more illegal
words will be filtrated at the very beginning. Our statistics result is showed
below:
[
2 https://download.csdn.net/download/sxtuwy/9824178
In information retrieval model, we treat task 1a as a retrieval problem and refer to the
citation as query and reference text spans as documents, so we return a list of
sentences as candidates according to the query. The original model in [
The model is an improved LM that using Dirichlet smoothing and the cosine similarity of word pairs based on word embeddings taking the place of word frequencies. Where fsem is a function to measure semantic relatedness of the query term qi to the document d , C is the entire smoothing corpus, V is the vocabulary of C and the Dirichlet smoothing parameter.
fsem is defined as below: Where： fsem (qi , d ) = s(qi , d j )
d jd (e(qi ).e(d j )) , (e(qi ).e(d j )) s(qi , d j ) = 0 otherwise (1) (2) (3) (4)
Here the transformation ( ) of dot products between the word embeddings representation of query word qi and document word d is a logit function: (x) = log(
x 1− x )
As for , the value is set to be two standard deviations larger than the average value of cosine of embeddings.
We borrow ideas from the above model and present our two improved strategy.
First, we train our own word embeddings according to the AAN(ACL Anthology
Network) corpus[
4 The embeddings are trained with the setting of vector size 400, negative sampling, windows size of 5, minimum count of 5.
To validate the effectiveness of our embeddings, we also download GoogleNews embeddings and use the language model I realized according to the idea above to compare the performance of the two embeddings. Table 2 shows that our AAN embeddings performs much better than GoogleNews ones based on the test-set 2017
Second, we try to improve the performance of the language model with the section information taking into account heuristically. To validate the feasibility of the idea, we separate a reference paper by sections and apply LDA(Latent Dirichlet Allocation) and LSI(Latent Semantic Index) model to calculate the cosine value between a citation and one section respectively. We carry out the experiment on test-set 2017 and separately compute the ratio whether the section that the reference sentences locate in is in the top 2 or top 3 most similar section according to the LDA and LSI value between section and citations . Our results show that our idea with section taking part in is feasible and LSI model has the upper hand against LDA model.
Based on the above experiment, we modify the language model of (1) by adding section similarity: p(qi | d ) = fsem (qi , d ) + p(qi | C) * cosineqiq,dsection (LSI[q], LSI[section]) (5) fsem (w, d ) + wV
Compared to the former model, we integrate the cosine value of query and section in LSI space to calculate the probability of query word qi when given a document d .Here, LSI[q] means the topic distribution of query q ,where the topic number is 50. LSI[section] means the topic distribution of the given section and the topic number is the same. 3.3
In addition to the language model with word embeddings, we also implement BM25 and VSM(Vector Space Model) model, since the two models are classical retrieval model and may serve as baselines during my experiment. ─ BM25: the model is defined as follow f (q, d ) = c(w, q) wqd (k +1)c(w, d )
| d | c(w, d ) + k (1 − b + b avedl ) log(
N − nw + 0.5 nw + 0.5 ) (6)
Where q, d denotes query and document respectively, c(w, q) denotes the frequency that word w appears in q . c(w, d ) denotes the frequency that word w appears in document d . | d | is the length of document d . avedl is the average length of all the documents. N is the number of the documents and nw means the number of documents that word w appears in.
Besides, k and d are hyperparameters and the values are 1.25 and 0.75 respectively, according to our experience. ─ VSM: vector space model is another popular model to be applied in retrieval field.
We use TF-IDF(term frequency and inverse document frequency) value to constitute the vector space. 3.4
Our preceding meta-models are all unsupervised models which make full use of the sematic and lexical relevance between citations and reference papers. Since CLScisumm dataset has manual annotation in training sets, supervised approach can be a good solution to task 1a.
We apply random forest model to solve the problem, the following features are chosen: ─ Jaccard similarity: the quotient of the intersection divided by the union between the citation and the candidate reference sentence. ─ BM25 similarity: the BM25 similarity value between the citation and the candidate reference sentence as we described before. ─ Vectorized TF-IDF similarity: the cosine value between the citation and the candidate reference sentence which are represented by TF-IDF value in vector space. ─ Section similarity: the cosine value between the citation and the section that the candidate reference sentence locates in via LSI model. ─ AAN word embeddings alignment: the value is defined as follow: f (citation, sentence) =
cicitation sjsentence | sentence | f (ci , s j ) (7)
Where f (ci , s j ) is the same as our former definition in (3). ─ Average distance of AAN word embeddings: we add up all the word embeddings in the citation and the candidate reference sentence respectively, normalize the vectors and get the cosine value as the average distance.
Because of the extreme imbalance of the labels of the data, we consider oversampling strategy to deal with this situation. Here we apply SMOTE+ENN technique to increase the number of label 1.
Based on the preceding models we establish, we consider using voting method to integrate the results of the models.
Here we apply two layers of voting to select sentences as the final candidate output. The mechanism is showed below.
Since our oversampling strategy SMOTE+ENN will produce a number of positive samples in every run, the performance of the random forest is closely connected with the new samples. Hence, we consider saving the random forest models which perform well on the Test-Set 2017.
We save 50 RF models that perform well individually at first, then we determine the number of models and the threshold of voting to be 25 and 17 respectively according to Fig.2 and Table 3 in the first voting layer.
In the second voting layer, we integrate the output of voting layer 1, the top ten sentences of BM25 and the top ten sentences of VSM model to vote for the ultimate results. Only the sentences that are included in all three models will be chosen as the output sentences.
Besides, we do pruning and padding operation on the results. In case the corresponding output of a citation is nonexistent, then we return the top 2 sentences in BM25 model as the output. In case the corresponding output of a citation is more than 4 sentences, then we return the top 4 sentences in BM25 model as the output.
In addition to the above voting system, we also using another voting shown in Fig.3 and also submit a system. ─ Number of numeric character: we count the number of numeric characters for each input sentence as a feature. ─ Relative position in the section: relative position for one sentence in the section which the sentence is located. ─ Relative position in the full paper: relative position for one sentence in the full reference paper.
We train the models on the training-set 2017 and apply the following strategies to get the final forecast output. If the probability of the positive label from one classifier is over 0.5, then we return the facet correlated with the classifier. In case none of the probabilities is over 0.5, if none is over 0.2, then we identify the facet as Hypothesis. Otherwise, we identify the facet as Method. 5
For task 1a, we submit 4 systems and the settings are as follows: ─ System 1: a voting system combined 20 random forest models with the voting threshold to be 14 ─ System 2: a voting system combined 25 random forest models with the voting threshold to be 17 ─ System 3: a two-layer voting system shown in Fig 3. ─ System 4: a two-layer voting system shown in Fig 1.
We evaluate our systems using micro average metric on test-set 2017, which is part of training-set 2018.
The results are shown in table 4.
As for task 1b, because of the severe imbalance of the dataset shown in table 5. The test-set 2017 has 155 sentences in total, but 92.25% of the facets are methods, and result accounts for 7.1%. On the contrast, the facet implication and hypothesis do not appear in the dataset. We consider not evaluating the performance of identification of facets but adjust the parameters of the models according to the performance on the training set. 6
This paper has focused on our methods applied for task 1a and 1b of the CL-SciSumm 2018. For task 1a, we find the baseline BM25 model can almost achieve the best performance on test-set 2017. Although the robustness of the result is not so convincing, but the phenomenon indicates that semantic-based citation identification is the main stream of the former exploration and the popular deep learning methods do not achieve satisfactory results because of the limitation of the scale of the dataset. We also get that the voting method is an effective strategy to improve the performance of the systems. For task 1b, a valuable and heuristic conclusion is that the distribution of facets to the reference sentences according to the citations is imbalanced and the summary merely extracted from the cited spans may not comprehensive and complete. Hence, how to combine citation information and other useful information for summary generation could be a consideration when doing scientific summarization.