Query-focused summarization enables the extraction of information on a speci c \aspect" such as a \proposed approach" or \dataset used." Accordingly, it can be a powerful tool for reviewing scienti c literature. In this paper, we present a method for unsupervised extractive query-focused summarization. In our approach, we rst calculate word importance scores for each target document using a word-level random walk. Next, we optimize sentence embedding vectors using a Siamese neural network. Here, we utilize localized sentence representations obtained as the weighted average of word embeddings where the weights are determined by the word importance scores. Then, we conduct a sentence-level query-biased random walk to select a sentence to be used as a summary. In our experiments, we constructed a new evaluation dataset for query-focused summarization of scienti c papers and showed that our method achieves competitive performance compared to other sentence embeddings.
Query-focused extractive summarization is one of the key technologies needed
for nding useful information from a large number of scienti c papers [
Considering the diversity and high speci city of scienti c papers, we use a graph-based method that, unlike neural networks, does not require a large training dataset for both word- and sentence-level importance calculations. Moreover, we speci cally focus on extractive summarization because for scienti c knowledge, it is important to preserve the authors' original expressions or opinions in the generated summaries. The contributions of the present study are summarized as follows: First, we propose an unsupervised method for learning localized sentence representations that takes document-wise word importance into account. Second, we constructed an additional dataset to evaluate query-focused summarization of scienti c papers. Third, through our preliminary study, we showed that the sentence representations obtained using the proposed method achieves comparable performance with existing ones. 2
Both supervised and unsupervised methods are used for extractive
summarization. Among supervised methods, neural models are the mainstream of current
research [
Regardless of whether the learning method is supervised or not, sentence representations are generally used as inputs to summarization systems. In most recent studies, sentence representations are calculated based on the embeddings of their constituent words.
In the case of supervised methods, neural networks are often used to learn
optimal representations. Di erent types of neural architectures are applied:
recurrent neural networks [
Arora et al. proposed sentence embeddings obtained as the weighted average
of word embeddings [
Hashimoto et al. constructed a dataset of review matrices consisting of scienti c
papers submitted to NLP shared tasks [
Based on the above observation, we constructed a new dataset by (1)
collecting additional papers from later publications that report the results for the
same dataset as the submitted papers, and (2) manually selecting the most
relevant sentences for the queries1. Thus, existing system descriptions in the review
matrices can be used as the \context" for query expansion. In our dataset
construction, we selected four shared tasks, 2-3 papers and 1-2 queries, which also
appear in the review matrices dataset constructed by Hashimoto et al., for each
task. Then, we read the abstracts of the selected papers, found the most
relevant chapter for a given query, and chose the most appropriate sentence that
has adequate information about the query. The statistics of our new dataset are
listed in Table 1.
In our approach, we rst obtain word importance scores by conducting a random
walk on a word-graph for a given document. Secondly, we train a Siamese neural
network to obtain optimal sentence embeddings. Finally, we perform a
sentencelevel random walk with the acquired sentence representation and an expanded
query to select only one sentence for each pair of a paper and a query.
The word importance scores are calculated using a random walk on a graph of
all words in a document [
sim(wi ; wj ) PN
k=1 sim(wi ; wk )
where sim(wi; wj ) is the similarity score between words wi and wj , calculated
as the cosine similarity of corresponding word embeddings pretrained using
word2vec [
Our Siamese network is based on the study by Kenter et al. (2016) [
Pw2s pw(w)vw Pw02s pw(w0) : The loss function of the network is the categorical cross-entropy: L =
X
X si2D sj2fS+[S g
p(si; sj ) log(p (si; sj )) where D denotes a document, and S+ and S are positive and negative samples, respectively. The negative samples are randomly chosen from D. In Equation (4), p( ) and p ( ) are the target and the predicted probability distributions, respectively. The predicted probability distribution is obtained using a softmax function over cos(vsi ; vsj ). The target probability distribution is given as p(si; sj ) = jS1+j for sj 2 S+ and 0 for sj 2 S . 4.3
In this step, we conduct a sentence-level random walk to select the most
appropriate sentence as a summary for a given query [
t 1 Q(i; j) = Psj2D rel(sj ; q)
:
rel(si; q)
(6)
(7)
Finally, we conduct a sentence-level random walk until pts converges. The
difference between the word- and sentence-level random walks is that the latter
considers query relevance. This type of query-focused random walk can be used
for query-focused and extractive summarization [
As noted earlier, we used the word2vec toolkit to obtain word embeddings. We
trained word2vec on the scienti c papers collected from ACL Anthology2 using
SideNoter [
For the Siamese network, we trained our network with two positive and ve
negative samples for each sentence. The number of epochs for training was one.
In our query expansion, we added at most eight words from the reference
summaries. For both the word- and sentence-level random walks, the dumping factor
was set to 0.3. All these parameters other than those related to word2vec were
tuned using the review matrices dataset [
We compared our sentence representation with four other baselines.
{ TFIDF: a v-dimensional vector whose elements are the dot products of the
term frequency and the inverse document frequency of each word.
{ Average WE: obtained by simply averaging pretrained word embeddings
in a sentence.
{ Siamese CBOW: proposed by [
In the evaluation, we calculated the ROUGE-1, ROUGE-2, and ROUGE-SU4
scores which are the automatic evaluation metrics [
As shown in Table 2, surface-level word matching, used by TFIDF, is not
sufcient for our sentence similarity calculation. Among the other four
embeddingbased methods, our approach achieved the best score for all the metrics.
Although our dataset is still small and limited, the results indicate that
considering context-dependent word importance scores proved e ective in our task. In
addition, the performance of simply averaging pretrained word embeddings was
relatively low; thus, utilizing a Siamese network had a signi ant impact on this
task.
In this study, we constructed a dataset for the summarization of scienti c papers,
and evaluated the proposed approach on our dataset. In terms of mining scienti c
papers by query-focused multi-document summarization, our localized sentence
representations were proved to be useful. However, in certain cases, the output
summaries for di erent queries were the same, indicating that the dumping factor
of sentence-level random walk should be more exibly adjusted. In addition,
we observed the cases where the query expansion fails to capture the original
intension. Future work includes the improvement of the query expansion and the
relevance score calculation between a sentence and a query. We will also consider
to incorporate more advanced graph-based algorithms such as [
This work was supported by JSPS KAKENHI Grant Number 16K12546 and CREST JP-MJCR1513.