Finding the cited text spans of a scienti c article based on the citation text is a challenging task. In this paper, we present our novel system to identify cited sentence(s) and their residential sections in a reference paper, given a citing text. We de ne this task as a binary classi cation problem. We use domain-speci c features obtained from ACL terminology. The predictions of the system are generated by a logistic regression classi er, with additional predictions from an Adaboost-decision tree added if the logistic regression predictions do not show su cient diversity according to a threshold.
Scienti c researchers should have comprehensive knowledge of previous work
and recent advancements in their eld of interest. Considering the rapid growth
of the number of scienti c publications, it is a time-consuming and challenging
task. Hence, automatic summarization has attracted many NLP researchers in
recent years. One of the recent approaches is based on rst nding all cited
text spans that cite a paper and then creating a summary from those sentences
[
The 4th Computational Linguistics (CL) Scienti c Document Summarization Shared Task (CL-SciSumm 2018) is part of the BIRNDL workshop at the annual ACM SIGIR Conference and focuses on scienti c document summarization in the CL domain. CL-SciSumm 2018 includes three subtasks of 1A) identifying cited sentence(s) for a given citance, 1B) determining in which facet (e.g. Method or Implication) the reference paper is being cited, and 2) generating a summary. This report presents our proposed approach for tasks 1A and 1B. We have used Scikit-Learn and NLTL for our Python implementation.1 a reference o set r (a text span from the reference article), determines whether the citation sentence(s) cites the reference sentence(s).
We create the training dataset using the annotation les provided for the shared task. Every reference article has an annotation le containing all citances to it, and their cited text. Each citance c to reference article R can be paired with every sentence (cited text) in R, i.e. 8r 2 R. If the pair (c; r) exists in the corresponding annotation le, its label is positive; otherwise, its label is negative. If citance c in the annotation le is referring to multiple sentences ri::rj in R, we consider all possible pairs between c and ri::rj as positive instances. The resulting dataset is very skewed, as most reference sentences are not the ones that will end up being cited. Indeed, out of 180,867 training instances, only 753 of them are positive examples (0.4%).
For the test set, we follow the same steps to create the instances, namely pairing each citance c with every sentence r in the reference article. However, these labels are unknown and should be predicted by the trained model. 2.1
In order to characterize our (c,r) pairs we use the following features. Let W be the set of words of a certain document that appear at least 20 times in the training data and are not stop words, we de ne Wr and Wc for citation and reference respectively.
Bag of words: We build a bag of words for Wr , and another for Wc, and we calculate the size of their intersection.
Brown clusters: We construct two feature spaces, for Wr and Wc
respectively, and each one is a bag-of-words style space formed by the Brown clusters
of the words in its set. We use the ACL Brown clusters distributed with [
Embeddings: We calculate the average embedding vector of Wr and of
Wc, which yields two 100-dimensional feature spaces, and an additional numeric
feature with the cosine of the vectors for Wr and Wc. We use the ACL corpus
word embeddings distributed with [
Sentence scope and position in document: We calculate numeric features to give account for the number of year dates (e.g. 1997) in Wr, the number of capitalized words (potentially names) in Wr, the lengths of c and r, and the number of sentences in their respective o sets. We also calculate the relative position of r in its document, i.e. the index of r divided by the number of sentences in the document, as well as the relative position of r in it section. Finally, we add a small bag of words with the words in the section name for r, following the intuition that an abstract is less often cited than a methods section.
Terminology: We use the ACL terminological base provided in [
WordNet: We use features from WordNet [
Many of the features we use are bag-of-words based, and are prone to sparsity and over tting in a skewed distribution. In addition to only considering frequent terms (appearing more than 20 times), all the features that are not present in the test set are removed from the training data. We perform ve-fold cross-validation without shu ing the results to best simulate the e ect of out-of-vocabulary ratio on new data.
We have experimented with di erent simple classi ers chie y decision trees and logistic regression, and di erent ensemble methods derived thereof. Our exploration of SVM with a polynomial or RBF kernel did not outperform simpler classi ers or ensembles. Table 1 shows our two most competitive classi ers. The system marked in bold, LogregL2C10, is responsible for most of the submitted predictions, and Adaboost20 provides auxiliary predictions (cf. Section 2.4). We also report the results of applying the evaluation script provided by CLSciSumm2 for the two Logistic regression models (Table 2). As explained in Section 2, we report the sentence with the highest score as the cited text for each citance. 2 https://github.com/WING-NUS/scisumm-corpus/blob/master/2018-evaluationscript/program/task1 eval.py Our system normally chooses only 1 sentence in the reference for each citance. More speci cally, we sort the sentences of the reference article by the probability scores predicted by the classi er, and select the sentence with the highest probability score as the cited text for a given citance c.
However, some citances might cite more than one reference sentence. We achieve multiple prediction by joining results of our two candidate classi ers: if the ratio between the number of citing sentences and the number of LogregL2C10predicted reference sentences equals or exceeds 7:1, we add an additional reference sentence from the top of the Adaboost20 prediction scores. Adaboost20 is also a competitive classi er that has substantial diversity of classi cation criterion with regards to our main classi er, namely LogRecL2C10. Moreover, Adaboost20 has very high precision for the positive class and makes a good candidate for ensemble prediction. We have applied this extension of the prediction set in 4 out of the 20 reference documents that make up the test set. 3
Task 1B requires labeling positive predictions to determine its facet, which can be Aim, Hypothesis, Method, Result or Implication. We have applied a heuristic labeling based on the section of the reference text. We construct a lookup from section names to their most frequent citation label, and apply it on test. If a section name is not present in the lookup, we back o to the Method label. The macro-averaged F1 score (in percentage) for LogregL2C10 and AdaBoost20 is 8.9 and 5.3 respectively. 4
We approached the problem of identifying the cited text for a given citance as a binary classi cation task. We derived our features using a combination of content-based features and similarity measures, and performed an extensive feature and model selection. The cross validation results on the training dataset show that Logistic regression with L2 regularization outperforms more complex ensemble or kernel-SVM models.