The CL-SciSumm 2018 shared task is designed to further the state-of-the-art in constructing summaries of research papers. Participating systems extract focused information from reference papers given their citing papers in order to construct a summary. To that end, given a citing sentence, we would need to know what concept (a result, method, or inference) is the citing sentence referring to in the reference paper. We explore the e cacy of word embedding based similarity functions to nd relevant sentences from reference papers that are likely being referred to by citing sentences (citances). We also classify the sentences into appropriate discourse facets. Our preliminary ndings indicate rank optimization for returning top sentences is useful for this task, and using term-IDF weights for averaging term embeddings improves performance.
An appropriate (and adequate) summary of a speci c line of research, could greatly bene t researchers: salient ndings can be quickly collated, information overload can be avoided, contrasting viewpoints can be examined, all of which help in a better understanding of the state-of-the-art of the research topic, while getting an overview of its historical development.
An obvious rst step in such an endeavor is to generate an adequate
summary of one research publication. The CL-SciSumm shared tasks [
In this paper, we explore various baseline algorithms (with some variations), for ranking the sentences of the reference paper, in order of their similarity to a given citance. We experiment with BM25 over sentences, cosine distance between sentence embeddings (constructed by averaging the word embeddings of the sentence), weighted averages for embeddings, as well as, rank optimization, and variations in word embeddings construction. Our preliminary ndings show that: { BM25 is a strong baseline for this task. { Inverse Document Frequency weighted averaging of word embeddings improves performance. { Optimizing for number of top sentences to return from ranked reference paper sentences improves performance. 1.1
The task organizers shared \topics", each consisting of a reference paper, and at least 10 citing papers. Within each citing paper, sentences that referred to the reference paper (i.e. citances) were manually annotated. Furthermore, sentence(s) in the reference paper that are most likely the subject of discourse of each citance were also manually annotated. The type of discourse (aim, hypothesis, method, implication, or result) was also noted. Gold standard summaries of the reference papers were also shared as part of the training set.
There are 3 subtasks in CL-SciSumm shared task: { Task1a: nd sentences in the reference paper that are referred to by a citance, { Task1b: classify the found sentences into appropriate discourse facets. { Task2 : construct a summary of the reference paper using the found sentences (and possibly other related content in the topic's les)
The evaluation metrics for Task1 are Precision, Recall, and F1 (the harmonic
mean of Precision and Recall). Task2 is evaluated using the ROUGE [
We downloaded the Parcit Structured XML1 [
We approach Task1a as a sentence similarity problem: if the citance text is the \query" sentence, we need to nd the sentences from the reference paper that are most similar to the query sentence. The similarity function could potentially incorporate various lexical or semantic features, and the goal then is to nd a sentence similarity function that performs best for this subtask. As the similarity function imposes an ordering over the sentences in the reference paper, we also try to optimize the number of most similar sentences to be returned. 3.1
BM25 (Vanilla) baseline (BM25) We rst build a baseline based on the BM25
similarity [
BM25 (No Author) baseline (BM25 noauth) We noticed that references of the form \(author, year)" caused those sentences to be returned (from the reference papers) that had the \author's" names present. Such sentences are typically references to an even earlier paper by the same author(s). We therefore generated a variation on the baseline wherein we removed \(author, year)" tokens from the query citance text. 3.2
We wanted to experiment with word embedding based approaches for sentence
similarity. Typically a sentence or phrase embeddings [
Average Embedding based similarity (w2v avg) Our basic word embedding based similarity function: { computes the average vector of the query citance's term embeddings giving us a query vector, { computes the average vector for each sentence in the reference paper giving us sentence vectors, { computes the cosine distance between the query vector and the sentence vectors, and { returns a ranked ordering over sentence vectors based on their cosine distance from the query vector.
this method, instead of using the regular average, we compute a weighted mean of the term embeddings. The weights are the IDF values for the terms in the sentence/citance; IDF of terms is computed using the ACL Anthology Reference Corpus (Section 2).
Smooth Inverse Frequency based similarity (w2v sif) We also
investigate the e cacy of the Smooth Inverse Frequency [
We also explore 3 di erent variations that build upon some of the methods described above.
Rank Optimization (optRank) Given that sometimes more than one sentence can be referred to from the citing paper, and that sometimes the similarity function returns the referred sentence lower in the ranked list, we can try to optimize the number of top documents to return for maximizing the F1 metric. We perform a 5-fold cross validation over the training data to determine how many top ranked sentences need to be returned on average, in order to maximize F1. We computed the mean number of top ranked sentences to be returned for each of the similarity method described in Section 3. \General" vs. \Normalized" Embeddings We observed that word2vec program3 returns a di erent vocabulary set and embeddings with di erently processed inputs. For instance, given raw document text, word2vec produces di erent embeddings for \methodically", \Methodically", \Methodically," and \methodically,". On the other hand, given lower-cased document text with all punctuations removed (normEmb), produces only one embedding for \methodically". We tried to see how (if at all) this di erence in embeddings a ects system performance . 2 https://en.wikipedia.org/wiki/Principal component analysis 3 https://github.com/tmikolov/word2vec Averaging Embeddings over a Window ( wndw avg) Given that the average length of sentences in the challenge data is 23 words, computing an average over 23 term embeddings may result in an inadequate sentence vector representation. We could try to capture phrase level concepts and use them for similarity computation. It is possible, that some phrase-level average vectors that occur early in one sentence, are most similar to phrases that occur later in another sentence. In this method: { We compute average vectors for an overlapping window of 5 terms, e.g. a sentence of 6 words would return a set of 2 averaged vectors. Thus we get a set of average vectors Aq for the query citance, and another set As for a sentence in the reference paper. { We generate a matrix Dqs of size jAqj jAsj with each element being the pairwise cosine distance between the elements of Aq and As. { We nd the maximum similarity along each row of Dqs and compute a mean of the row maximums (Rmax mean). Similarly, we compute the mean of the column maximums (Cmax mean). { The similarity function returns (Rmax mean+Cmax mean)=2 for each sentence in the reference paper.
The idea here is to capture the maximum possible similarity between the phrases (windows) of two sentences. 4
Task 1b involves identifying the Discourse Facet, of the cited text from each
reference paper, from a pre-de ned set of labels: Aim, Hypothesis, Implication,
Method, Result. We approached this task as a multi-class classi cation
problem where a trained classi er is used to predict the discourse facet label. For
each cited text sentence, we construct its corresponding feature vector , which
is an average of word2vec embedding vectors for each term in that
particular sentence. We used our word2vec embeddings obtained from ACL Anthology
Reference Corpus (as explained in Section 2) to construct our sentence
feature vectors. We then experimentally compared three classi ers: KNN classi er,
RandomForestClassi er, SVM classi er as implemented in scikit-learn [
1. The BM25 baseline is hard to beat, when comparing against basic word embedding average based similarity functions. Bold and italics values denote best performance in columns and rows respectively. 2. Optimizing number of top sentences to return almost always results in improvement in performance as measured by F1. 3. Using normalized embeddings improved performance over not using them, except in the case of smooth inverse frequency based similarity. 4. Using IDF weighted average vectors worked better than simple averaging of sentence word embeddings. 5. Window based averaging worked better than entire sentence averaging. 5.2
For our o cial submission, we experimented with the following classi ers: { K-Nearest Neighbors Classi er { RandomForest Classi er { SVM Classi er Using 2018 training data, we ran 5-fold strati ed K-fold validation to evaluate and select the classi er that gives us better performance. As observed in Table 2, majority of discourse facet instances in current training data are labeled method citation. Evaluation results for our strati ed K-fold validation are listed in Table 3. Based on the performance of our methods on the Training Set for 2018, we submit as runs the system outputs of: (i) BM25 + optRank, (ii) BM25 noauth + optRank, (iii) w2v idf avg + optRank + normEmb, (iv) w2v sif + optRank, (v) w2v wndw avg + optRank + normEmb, (vi) w2v idf wndw avg + optRank + normEmb. For each of these methods we predicted the discourse facet using the Random Forest Classi er. 6
We submitted 6 runs for Task1a of the CL-SciSumm 2018 shared task. We found that BM25 forms a good baseline for this task, and IDF weights of terms can boost e ectiveness of word embedding averaging based methods for sentence similarity. Optimizing number of sentences to return for Task1a also improves performance over the F1 metric.
In the future, for discourse facet prediction, we wish to combine word
embedding features along with word or character overlap segments between the
reference text and title text to generalize and improve prediction. In particular
for Task1a, as more data becomes available, it could be promising to use learning
to rank methods [