Related work section is essential in a scienti c publication, for it elaborates past studies relevant to the topic in comparison with the current one. The automatic generation of related work section in scienti c papers is a meaningful yet challenging task. While prior works have gained encouraging results, they have not fully addressed the issue of informativeness and the di culty of obtaining citation sentences due to delay of publication. In this paper, we introduce SERGE, a novel and e ective system for generating descriptive related work section automatically by sentence extraction and reordering. Our system rst employs a BERT-based ensemble model to select the most salient sentences in reference papers, and then uses a similar model to reorder these sentences for better readability. Automatic evaluation results show that SERGE signi cantly outperforms existing baselines on ROUGE metrics, gaining an improvement of 18% to 56% on recall and 4% to 33% on F-score. Human evaluation shows that SERGE gains a higher informativeness score than human-written gold standard as well as the baseline, indicating its ability to provide valuable information that matches the real interest of researchers. In contrast to existing methods, since our system is free from delayed citation problem and yields high informativeness, it shows a great potential for various applications.
Scienti c papers usually contain a related work section, which is also known as a
literature review. It summarizes previous works relevant to the research topic in
order to establish the link between existing knowledge and new ndings[
It is generally accepted that there are two distinct styles of literature reviews:
descriptive and integrative[
In this paper, we particularly focus on the generation of descriptive related
work section. On this matter, Cohan and Goharian[
To this end, in this paper, we propose a novel method for automatic generation of descriptive related work section in scienti c papers by extracting salient sentences from scienti c literature and rearranging them into a logical order. In contrast to most existing methods which su er from citation delay problem, our method does not require any citances to achieve its goal, making it applicable even when no citation data is available.
The main contributions of this paper are as follows: 1. We propose a novel and e ective approach to automatic descriptive related work section generation based on extractive document summarization techniques, including sentence extraction and reordering. 2. Our method does not need any citation data to achieve its goal, which implies that the method does not su er from the delay of citing publications or require the input of citation data. Such a characteristic o ers more potentials of our proposed method with various applications. 2
Automatic related work section generation is a special case of multi-document
summarization tailored for scienti c articles[
Automatic related work generation di ers from summarization of generic
texts in the following aspects: 1) summarization of generic texts often focus on
the content of source papers, whereas the related work section should also
delineate contributions and limitations of reference papers (i.e., cited papers) as
well as the relationship between reference papers and the current paper; 2)
compared to generic texts, scienti c articles contain more domain-speci c concepts
and technical terms, which poses great challenges for language modeling; and
3) scienti c articles are more structured than generic texts and reference prior
research[
However, we believe prior research on related work generation, with ROUGE
scores as the most popular evaluation metric, haven't fully addressed the issue
of informativeness: the property of conveying useful information[
Here, we introduce our system, SERGE, which stands for \Sentence extraction and rEordering based Related work section GEnerator". The overall architecture of SERGE is brie y illustrated in Fig. 1. Given a set of papers from which a related work section will be automatically generated, the system takes the full text of these papers as input, and generates a descriptive related work section involving all these papers as output. SERGE consists of two main parts: a classi cation model and a reordering model. The classi cation model is used to determine whether a sentence is su ciently salient that it should be included in the generated output. For each sentence as input, the classi cation model generates a probability value indicating the salience of the sentence. Then, the sentences with highest probability values are fed in to the reordering model, which determines their best order (sequence of sentences). Lastly, the sentences, sorted by the reordering model into the most sensible order, are modi ed with citation tags and proper pronouns, forming the nal output of the system.
Set of
papers
The task of classi cation model is formally stated as follows: Given a sentence s,
the model is supposed to assign a label Y (s) 2 f0; 1g to the sentence, or rather,
compute a probability value y(s) 2 [0; 1] indicating the salience of sentence s.
To accomplish the task, we adopt an ensemble model which consists of two
sub-models: a deep neural network model and a bag-of-words (BOW) sentence
matching model. The input sentence s is fed in to both models simultaneously.
The architecture of the deep neural network model follows Google's original
BERT paper[
The training of the model requires annotated data pairs. However, due to the
lack of suitable training corpus, we opt to annotate a new dataset automatically
by leveraging the ScisummNet corpus[
Linear layer C
Z
… [SEP]
s
important aspects of the cited paper and highlight its key contributions, we make
use of the citation sentences of a paper in the corpus to produce a gold label of
whether a sentence in the paper is salient. An algorithm similar to Nallapati et
al.'s paper[
By employing the greedy annotating algorithm as described above, we obtain an annotated dataset that can be used to train our neural classi cation model. The dataset includes 11954 training samples, in which 3541 are positive ones.
The bag-of-words sentence matching model simply checks whether the input
sentence s contains any of the words in a curated feature word set B. We
manually choose the words to be contained in word set B according to the ndings of
Shin[
The combined output of the ensemble model is de ned by the following equations:
8 0;
y(s) = <
Y (s) =
0; 0 y(s)
1; 0:5 < y(s)
Essentially, Eq. 1 considers a trade-o between precision and recall. By setting
L = 0:2 and H = 0:4, the ensemble model achieves the most desirable overall
performance, with a precision of 0.622 and a recall of 0.793.
The task of reordering model is formally stated as follows: Given a set of
sentences S = (s1; s2; :::; sn), the model is supposed to nd their optimal
arrangement si1 ; si2 ; :::; sin (ir 6= it; 8r 6= t; ik; k; r; t 2 f1; 2; :::; ng), where the
probability of the sequence P(si1 ; si2 ; :::; sin ) is maximized. However, considering its
sheer scale, it is practically impossible to obtain the solution of the problem
directly. Therefore, we decompose the big problem into much smaller ones using
a method similar to Chen et al.'s paper[
P(sik jsi1 ; si2 ; :::; sik 1 ) = P(sik jsik 1 ) where k 2 f2; 3; :::; ng. We also assume P(si1 ) = 1. Thus, Eq. 3 becomes P(si1 ; si2 ; :::; sin ) = It can be seen from Eq. 5 that the computation of the probability of all possible arrangements can be approached by simply computing the conditional probability of each sentence appearing after another, reducing the complexity of the problem signi cantly.
We use a MobileBERT[
To obtain the desired output, MobileBERT is ne-tuned on next sentence prediction (NSP) task, which generates a probabilistic value indicating whether the rst sentence in the input is followed by the second one in the source document. The architecture of our NSP model is identical to our neural classi cation model as is described in Section 3.2, except that the BERT layers are substituted by MobileBERT. The training data for NSP model is also extracted from ScisummNet corpus, whose writing style closely matches the expected input. To build the dataset, we pick out every pair of neighboring sentences from the 1000 papers as positive sample, combined with roughly the same amount of negative samples where the two sentences are not adjacent, yielding a total of 360509 training samples.
Finally, to nd the optimal sentence sequence, the model computes the probability of all possible arrangements of the sentences in S using Eq. 5. For the sake of the concision of the nal output, if the output of the classi cation model exceeds nmax = 3 sentences, only the ones with highest predicted value y(s) are kept, and the rest are discarded. 4
Both automatic and human evaluation approaches are employed to assess the
performance of SERGE. 10 descriptive paragraphs of related work section are
randomly extracted from 8 computer science papers published in journals or
proceedings. The papers cited in each paragraph are collected to form a set of
reference papers. MEAD[
During human evaluation, 3 computer science experts are instructed to grade the paragraphs generated by the two systems and the gold standard on three aspects: informativeness (INF), uency (FLU), and succinctness (SUC). The experts are uninformed about the authorship of the texts. The range of score is 1-5.
The result of automatic evaluation is presented in Table 1. Except for the precision of ROUGE-1, SERGE outperforms the baseline on all metrics. Notably, our system achieves a signi cant higher recall on all the three ROUGE metrics, yielding a relative gain of 20%, 56% and 18%, respectively. The result of human evaluation is presented in Table 2. SERGE gains an informativeness score of 4.23, which is about 6% higher than the baseline and 3% higher than the gold standard. SERGE also outperforms the baseline on succinctness. In short, the result above indicates that our system performs signi cantly better than previous baseline in numerous aspects, and even yields a higher informativeness than human-written gold standards. 5
In this paper, we propose a novel method for automatic generation of descriptive related work section in scienti c papers by extracting salient sentences from past literature and reordering them into a smooth paragraph, which is made possible by two BERT-based neural models. The performance of our method is evaluated by both automatic metrics and human experts. The results show that our method gains a substantial improvement compared with past baselines and achieves a high degree of informativeness comparable to human authors.
Our method addresses the problems of existing works in two ways. First, our method improves the informativeness of automatically generated related work sections signi cantly, providing more valuable information in existing literature which matches the real interest of researchers. Second, our method is immune from citation delay problem, suggesting its prospect for a wider range of applications.
There are several limitations in the current study. For example, the evaluation is not su ciently robust due to the high cost of human assessment. Also, our method is not necessarily optimal in uency and a few other metrics. These issues leave room for future exploration.
The result of this study clearly shows the e ectiveness of our novel method for related work section generation. Although the corpus used in this study is limited to the computer science eld, it is e ortless to adapt our method to other disciplines. Considering its universality and adaptiveness, our method shows a tremendous potential for becoming an intelligent and helpful tool which can increase the e ciency of researchers and boost scienti c innovations.
In the future, we may continue to explore new methods for this task via various paths, for instance, by abstractive summarization approaches or entity extraction. We are also interested in integrating summarization problem with certain knowledge bases which brings more intelligence to automatic systems.
This work was supported in part by The National Social Science Fundation of China (Number: 17BTQ066).