This report gives a brief introduction of our participation in CL-SciSumm 2017 Task 1A. We demonstrate some data analysis and point out the difficulty of this task. Then we report both unsupervised and supervised methods with their performances on 2016 and 2017 testset, from which efficiency of different features can be estimated.
Reading scientific articles is necessary but time-consuming for researchers and engineers. Although there are abstracts in papers, readers still find it difficult to understand key contributions of a paper. On one hand, original abstracts usually state in a general but less focused fashion and sometimes they do not contain all aspects of their papers. On the other hand, we might not completely believe contributions written in the abstracts, since they may be over or under-stated by authors, and may not get fully agreements from the research community. Thus, scientific paper summarization aims to automatically captures more detailed and complete contributions of a paper, objectively.
Scientific paper summarization is an NLP sub-task of automatic summarization. Different from traditional general domain such as newswire, scientific article is a special domain with extra features such as citation links, special discourse structures, etc.
To employ different aspects of a paper, [
Not long ago, [
Input citation paper “Learning Syntactic Categories Using Paradigmatic Representations of Word Context” (EMNLP 2012) Input citation sentence Input reference paper “Simple Type-Level Unsupervised POS Tagging” (Lee, Yoong Keok et al. EMNLP 2010)
Output reference sentence CL-SciSumm 2017 regards all datasets in track 2016 as the training set, and provided extra 10 papers as the evaluation set of this year. To help following researches better understand this task, some data analysis will be presented in this section. There are two annotated datasets for this task: TAC 2014 Biomedical Summarization Track 1 https://tac.nist.gov//2014/BiomedSumm/ (BioSumm 2014) and Computational Linguistic Scientific Summarization Track 2016 (CL-SciSumm 2016). Since we can only access the training data of BioSumm 2014, we will mainly focus on the CL-SciSumm 2016 dataset in the rest of this paper, and only mentioned BioSumm 2014 in the Human Performance section. 2.1
We did the analysis on CL-SciSumm 2016 dataset, which contain 20 training and development reference papers and 10 test reference papers. Detailed statistics are shown in Table 1.
Number of reference papers Number of citation sentences in total Median of citation papers per ref paper Median of citation sentences per ref paper Average length of citation sentences Average length of citation contexts into a closed name space. So, in this figure, we only choose four standard section names which occupy significant amount of portion, and let rest of names become “other” which mainly contains related work, method and experimental result sections. Notice that, the annotation rule of this track suggests annotators that, if there is no text matches, the title of the reference paper should be chosen. Fig. 4 shows the proportions of different discourse facets of citation contexts. Here, discourse facet type stands for the discourse role of a sentence in the paper. Title 3.1% Hypothesis
2.3% Implication
5.4% Results 15.0% Aim 17.9%
Method 59.3% dataset presented each citation sentence four different annotation results from four different annotators. Given a citation sentence, each annotator will find up to four related sentences from the target reference paper.
To evaluate human performance, each time we regard one annotator’s annotation as the predicted result, and the other three annotations as gold answers. Then we average four annotators’ performances as the human performance. We choose results from two recent previous work for comparison.
Models c-P c-R c-F1
Key Word Query Reformulation Method [
Human Performance 32.0 29.3 27.5
From Table 2, we can see that human performance does not reach 30% F-1 value. Since baseline methods always return top 3 sentences as results, the recall rates are even slightly higher than the human performance.
The result of low consistency among human annotator indicates the difficulty of providing one unique gold answer. It seems to be hard to clearly define the “relatedness” between reference sentence and citation sentence, and different annotators could hold different point of views.
One solution is to define “relatedness” from several more specific dimensions. Another possible solution is to change the current evalution method: For each citation sentence, rather than annotating one gold answer, it might be more reasonable to let annotators score different results from different systems. Though this evaluation method is much more expensive and time-consuming, it should be easier for annotators to distinguish between different results than to pick up answers from the whole reference paper.
For each sentence in the corpus, we first replace all reference groups using “TARGETREF” or “NORMALREF”. Here, a Reference Group is one or multiple citation markers in the same bracket. We recognize reference groups in a citation sentence using bracket pair signs like (, ) and [, ] and there also required to be at least one two digit number ranging from 00 to 99 or four digit number ranging from 1950 to 2020 inside that bracket pair.
Then, we replace all number with “NUMBER”. Finally replace all punctuations with blank space, except for segmentation punctuations such as , . ; ? !
We first employ search-based method with different features for sentence pair similarity calculation. For each citation sentence, we choose top 3 most rated sentences in the reference paper as answers.
For TF-IDF methods, we follow the Key Word configuration in [
The word embedding model refers to [
Supervised Method
we combine both training and development datasets for logistic regression learning
using sklearn. Several features are employed, such as similarity of 1-3gram TF-IDF,
similarity of 3gram character level TF-IDF, similarity of word embedding-based model [
Unigram TF-IDF (vocab = 5k)
1-3gram TF-IDF (1-3gram vocab = 200k)
3gram char TF-IDF (3gram char vocab = 5k)
(vocab = 20k, dim = 300, min freq count = 100)
Word embedding average
Word movers Distance [
Table. 3 shows performances of search-based methods with different bag-of-words
models. It shows that ngram tf-idf is slightly better than unigram tfidf. Surprisingly,
character level ngram model performs quite well. We haven’t successfully reproduced
Models Psent Rsent Fsent
All feature + logistic regression 11.6 27.5 16.2
- ngram TF-IDF (ngram vocab = 200k) 11.0 26.6 15.5
- 3gram char TF-IDF (3gram char vocab = 5k) 9.7 23.2 13.6
- Word embedding model 11.3 27.2 15.9
- Word embedding average 9.8 23.3 13.7
- section type 10.6 25.1 14.9
the results from [
Table 4 shows performance of the supervised model trained on training and
development set of CL-Scisumm 2016. Slightly different from results reported in [
Finally, we employed all hyper-parameter setups and trained the classifier on 2017 training data, and produced results on 2017 test data as our submission. The results are listed in Table 5. First two rows represent the results of unsupervised systems. And the last row stands for the supervised model.
Models Psent Rsent Fsent 1-3gram TF-IDF (vocab = 200k) 5.9 14.1 8.4 Word embedding model (vocab = 20k, dim = 300) 5.6 13.3 7.9 All feature + logistic regression 8.4 19.1 11.7 This paper is a system report on CL-SciSumm 2017 Task 1A. We analyzed annotation datasets and found out the difficulty of this task: since “relatedness” between reference and citation sentences is quite hard to define, both annotation process and question modeling become hard.
We also briefly introduced our system’s methods and their performances on CLSciSumm 2016 dataset. From results, it indicates the effectiveness of supervised methods and shows contributions of different features.