This paper introduces the methods employed by University of Houston team participating in the CL-SciSumm 2017 Shared Task at BIRNDL 2017 to identify reference spans in a reference document given sentences from citing papers. The following approaches were investigated: structural correspondence learning, positional language models, and textual entailment. In addition, we re ned our methods from BIRNDL 2016. Furthermore, we analyzed the results of each method to nd the best performing system.
The CL-SciSumm 2017 shared task [
Three main approaches are investigated for Task 1a: positional language
models, structural correspondence learning, and textual entailment
systems. The details of each method are explained in the following sections.
Two methods are employed to address task 1b: a rule-based method
which is basically a comparison-based method augmented by WordNet
expansion and a classi cation method.
Citations are considered an important source of information in many text
mining areas [
Properly tagging/marking the actual citation has also attracted a great
deal of attention to this area of reserach. Powley and Dale [25] give
insight into recognizing text that is a citation. Siddharthan and Teufel
also introduce a new concept called \scienti c attribution" which can
help in discourse classi cation. The importance of discourse classi cation
is further developed in [
The dataset for CL-SciSumm 2017 [
This dataset contains 148,669 words and 11,114 unique words among the reference documents. There are 6,700 reference sentences and their average length is 23 words. The average reference document length is 4955 words in this dataset. Furthermore, the average number of sentences in each reference document is approximately 223 sentences.
In this task, we are asked to identify the reference sentences referred to by a given citance. In general, we rank the sentences in the reference document according to some method, then return the top 3. This year, the following new methods were attempted by our team: Positional Language Models, Structural Correspondence Learning, and Textual Entailment techniques. 4.1
Positional language model was proposed with the idea of employing proximity information in documents to retrieve better results in response to a query[17]. In task 1a, we consider each reference text a document and each citance a query. Using PLM approach, a separate language model is constructed for each position of the reference sentence and computes the score of the reference sentence based on the similarity between its positional language models and citances language model. The elements of PLM (Positional Language Model) are the propagated counts of all words within the reference sentence which are estimated using a density function. With this idea, the closer words to the position, the higher the weight of the word in the PLM. Therefore, the PLM of reference sentence d at position i is estimated as follows: p(wjd; i) = Pw0 V c0 (w0 ; i) c0 (w; i) where V is the vocabulary and c0 (w; i) is the propagated count of word w at position i from all of its occurrences in the reference sentence. Finally, PLM of each position in the reference sentence is compared with the language model of citance using KL-divergence to obtain a position speci c similarity score as follows:
S(q; d; i) =
X p(wjq) log p(wjq)
p(wjd; i) w V where p(wjq) is the language model of the citance q, p(wjd; i) is the positional language model of reference sentence d at position i and S(q; d; i) is the similarity score between the position i in the reference document and the citance. These scores are used to nd the nal similarity score of reference sentence(as a document) in response to the citance(as a query). Thus, we can apply PLM approach as a retrieval process to nd the most relevant reference sentences in response to each citance. 4.2
SCL is a method of transfer learning that attempts to learn a joint
representation for two di erent domains [
Structural Correspondence Learning seeks to nd a joint representation by focusing on pivot features, i.e. features that are frequent in occurrence in both domains. The key to SCL is to predict the occurrence of pivot features from the non-pivot features of an example. One can learn a machine learning model, such as an SVM, for this purpose. The next step is to reduce the dimensionality of these predictors; this forces some generalization. The joint representation consists of the predicted pivot features (non-pivot features are thrown away). For our purposes, these new feature vectors are used to calculate cosine similarity scores with the citance. 4.3
The property of textual entailment between two pieces of text can be described as a directional relationship which can only be True when the information contained in one text fragment is directly or indirectly derived from the other text fragment. The derived text fragment is then said to be textually entailed by the other. In Textual Entailment3, the entailing fragment is termed the text and the possibly entailed fragment is the hypothesis. For example, the following pair of text fragments demonstrate entailment:
The cat ate the rat.
The cat is not hungry.
The task of deriving inference from pairs of text is called Recognizing
Textual Entailment (RTE)4. The proposed approach uses textual
entailment as a measure of extracting the reference sentences relevant to a
given citance. We build textual pairs using the given citance (text ) and
the sentences extracted from the reference document (hypothesis). We
use an RTE system TIFMO [
TFIDF. In this method we rank sentences in the reference document according to the cosine similarity between each sentence and the citance. Both the sentences and the citance are represented as a TFIDF vector, i.e. a word vector where the weights are the TFIDF values calculated from the reference document.
Although TFIDF has been evaluated before in [21], this time we experiment with using more than just unigrams. We include variations that make use of bigrams and trigrams as well. Our naming convention for these systems includes the range of n-grams they use (for example, t df1:3 uses unigrams, bigrams, and trigrams).
LDA. Latent Dirichlet Allocation is a topic modeling method [
For our task, we represent each sentence by the topic membership vector, which assigns to the sentence a probability of membership for each topic. These vectors are then ranked by cosine similarity, similar to TFIDF. Word Embeddings. Word embeddings assign to each word a realvalued vector [18]. Through continuous iteration, the similarity between these vectors starts to approximate the similarity between the words they represent. Thus, since `dog' and `pet' are similar, their respective real-valued vectors will be similar as well.
Our task concerns the similarity between sentences, however. To generate sentence similarities from word similarities, we employ the Word Mover's Distance [13].
In addition to embeddings learned through the ACL anthology, we tested the performance of embeddings that were pretrained on the Google News corpus [18]. 4.5
The evaluation of our systems in task 1a uses the metrics: Precision@3, Recall@3 and F1-score. In addition to the results of the PLM method, two well known information retrieval methods including KL-divergence and Okapi are employed to compare with the results of PLM. In all of the retrieval methods and PLM method, reference sentences are documents and citances are queries. Okapi is known as a ranking function which is based on the probabilistic retrieval framework. KL-divergence is a language modeling retrieval approach which compares language models of document with the query and ranks them based on their KL-divergence score. The results of all three methods for task 1a on training and test set 2017 are reported in Table 1. Runs with an asterisk ( ) were submitted. As Table 1 shows, TFIDF is still a top performer. A few of the results are di erent from previous work due to the fact these results are obtained from all 30 training documents. For instance, in comparison to the results in [21], LDA and word embeddings show worse performance. It is surprising that SCL performs better than LDA. TIFMO does not perform as well as expected. Positional language model is performing better than KL-divergence and Okapi. However, none of these methods perform desirably well. One of the important reasons for this performance can be the di culty of queries which are citances in our problem de nition. Since citances may not include any of the reference text's words, it makes the retrieval process more di cult.
t df-1:1 t df-1:2 t df-1:3 t df-2:3 word2vec LDA SCL TIFMO PLM KL-div Okapi In Task 1b, for each cited text span, we pick the facet to which it belongs from a prede ned set of facets. Two di erent approaches are employed in this task: A rule-based approach and a machine learning approach. Rule-based Approach. The Rule-based approach consists of three consecutive steps. Each one is designed to nd the correct facet through some comparisons, in case a match was not found in any of the previous steps. In the rst step, citance words are compared with all ve facet labels: Method, Implication, Result, Hypothesis and Aim. If none of the words in the citance match a facet label, then we move on to the second step. In the second step of the rule-based approach, an expanded form of the citance is compared with the facet labels. We expand the citance by adding all WordNet synsets [19] of each word found in the citance. In the third step, if no matched facet label is found in steps one and two, we expand the facet labels with their synsets and once again compare with the words in the citance.
Machine Learning Approach. In this approach, each citance is
represented by a feature vector containing TFIDF values of its words and
a classi cation model is learned using our training set. Then, the trained
model is used to classify citances of the testing set. Machine learning
methods used in this approach include Support Vector Machines (SVMs)
[
As explained in section 5, we have employed two di erent approaches in Task 1b: a rule-based approach and a machine learning approach. The rule-based approach has di erent variations: 1) Rule based-V1: In this variation, all three sets of comparisons (comparing citance words with facet labels, comparing expanded form of citances with facet labels and comparing expanded form of facets with citance words) are done while non-relevant synsets of all facets are excluded. 2) Rule based-V2: In the second variation, all three sets of comparisons are done while only non-relevant synsets of \Method" facet are excluded. 3) Rule based-V3: In the third variation, only rst and second comparisons are done. The results of rst approach of Task 1b on training set 2017 and testing set 2017 are represented in Table 2. A "Method only" approach which assigns "method" to all of the citances is also employed to be compared with the rule-based approach.
Rule based-V1 Rule based-V2 Rule based-V3 Method only As Table 2 shows, the third variation of the rule-based approach outperforms other variations on both training and test sets. It means that expansion of facet labels does not help in nding the correct facet label of citances. Furthermore, the higher performance of Rule based-V2 over Rule based-V1 shows that excluding non-relevant synsets of the \Method" facet has a positive impact on the nal results of the method. It might be due to the fact that \Method" is the most frequent facet label in both the training and test set for 2017. The results of the Method-only approach also verify this fact.
Table 3 shows the results of Task 1b for machine learning methods on the training and test set. For classi cation experiments on the training set, the training set is split into two separate datasets: a subset of 20 documents is used as train data and the remaining 10 documents are used as test data. For the classi cation experiments on the test set, the whole training set is used for the learning phase.
SVM Random Forest Decision Tree MLP Adaboost Rule based-V1 Rule based-V2 Rule based-V3 Method only
F1 As Table 3 shows SVM outperforms other classi cation methods in Task 1b and the lowest results among classi cation methods belongs to Decision Tree. Furthermore, comparison between the results of Table 2 and Table 3 shows that third variation of rule-based approach is our best performing method in Task 1b among all rule-based and classi cation methods. 6
In this section, we attempt to improve on the performance of the methods found in Section 4.5 by combining them. We combine methods in three ways: 1) through a linear combination of the methods, 2) through the use of one method as a \ lter" for another, and 3) through the use of learning-to-rank algorithms that are fed the scores of our individual methods.
Linear Combination. A linear combination between two methods that tries to divide the importance given to the scores of two systems. An optimal tradeo is calculated which normally generates better rankings than either system independently. For more details see [21]. (1) Filtering. The scores for one system are used to select the top N sentences from the reference document. These N sentences are then reranked according to another system. For N = 3 there will be no di erence from the system that lters since we always return the top 3. However, as N increases, the rankings start to diverge.
Learning-to-Rank. We used a library of learning-to-rank algorithms, RankLib5, to combine the scores generated by the other methods. We construct a modi ed dataset for use with RankLib. For each citance, we construct three di erent queries by subsampling the irrelevant sentences in the reference document. Therefore, each query consists of all relevant sentences (chosen by the annotator) and 10 irrelevant sentences chosen at random. This helps emphasize learning the ranking of the relevant sentences.
The scores of the following systems were used in conjunction: t df1:1, t df-1:2, t df-1:3, t df-2:3, word2vec (ACL), word2vec (pretrained GoogleNews), SCL. These systems were chosen in an ad-hoc manner to provide a diverse set of competing rankings. Even though some of these systems underperform in general, they can occasionally provide better rankings for speci c citances. No attempt was made to tune the hyperparameters for the algorithms.
Since learning-to-rank methods had a considerable jump in performance (as can be seen in Table 4), we had to test whether over tting was occuring. We chose LambdaMART since it is similar to MART and obtained the second highest score when fed the whole training set. We sorted the training set documents by the number of annotated citances; every third document became part of a validation set. The performance gains measured were much more modest in this scenario. 6.1
The results obtained by combining individual systems are found in Table 4. Runs with an asterisk ( ) were submitted. LambdaMART was chosen as the representative for the learning-to-rank algorithms and, thus, is the only algorithm with test set results. 5 https://sourceforge.net/p/lemur/wiki/RankLib/
Method Linear Comb.
Filtering 6 LambdaMART Val. LambdaMART MART Random Forest RankBoost ListNet Coord. Ascent RankNet Linear Regres.
LambdaRank 15.51% 15.58% 28.55% 17.21% 30.59% 21.05% 15.44% 15.37% 14.92% 14.70% 14.56% 0.00% The results on the training set indicate semantic methods by themselves do not perform well, yet the test set's results directly contradict that claim: although TFIDF is the clear winner in the training set, the best method on the test set made use solely of word embeddings. Task 1b also raises questions since the skewed facet distribution of the test set exacerbates the e ectiveness of a simple baseline such as always choosing \Method". Regardless, we can choose better features for the classi ers that would permit us to reach a comparable level of performance.
In regard to the combination methods, there were less surprising results but still many unanswered questions. Our experiments with learning-torank methods hint at over tting but the test set provided no evidence it was occurring. The linear combination between two systems, explored in [21], had similar performance to ltering. The ltering method between two systems was slightly more robust. 8
We would like to investigate why there was such a drastic di erence between the performance measured in the training and test sets. A comprehensive study can be done to contrast the characteristics of the training and test sets from a linguistic and statistical point-of-view. The di erences between training and test set may reveal what type of citances bene t most from semantic information. In a similar vein, we would like to nd out why there is a considerable di erence between the performance of TFIDF and Okapi although they have similar formulations. Finally, we have not exhausted our exploration of textual entailmente and would like to investigate newer methods that have been developed. 6 Submitted run was generated erroneously, which led to an F1 score of 1.4%.
We would like to thank the NSF for Grants CNS 1319212, DGE 1433817 and DUE 1241772. 13. Matt J Kusner, Yu Sun, Nicholas I Kolkin, Kilian Q Weinberger, et al. From word embeddings to document distances. In ICML, volume 15, pages 957{966, 2015. 14. Lei Li, Liyuan Mao, Yazhao Zhang, Junqi Chi, Taiwen Huang, Xiaoyue Cong, and Heng Peng. Cist system for cl-scisumm 2016 shared task. In BIRNDL@ JCDL, pages 156{167, 2016. 15. Andy Liaw and Matthew Wiener. Classi cation and regression by randomforest. R News, 2(3):18{22, 2002. 16. Kun Lu, Jin Mao, Gang Li, and Jian Xu. Recognizing reference spans and classifying their discourse facets. In BIRNDL@ JCDL, pages 139{145, 2016. 17. Yuanhua Lv and ChengXiang Zhai. Positional language models for information retrieval. Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval SIGIR 09, page 299, 2009. 18. Tomas Mikolov, Kai Chen, Greg Corrado, and Je rey Dean. E cient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. 19. George A. Miller. Wordnet: A lexical database for english. Commun.
ACM, 38(11):39{41, 1995. 20. Saif Mohammad, Bonnie Dorr, Melissa Egan, Ahmed Hassan, Pradeep Muthukrishan, Vahed Qazvinian, Dragomir Radev, and David Zajic. Using citations to generate surveys of scienti c paradigms. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 584{592. Association for Computational Linguistics, 2009. 21. Luis Moraes, Shahryar Baki, Rakesh Verma, and Daniel Lee. Identifying reference spans: topic modeling and word embeddings help ir.
International Journal on Digital Libraries, 2017. 22. Preslav I Nakov, Ariel S Schwartz, and Marti Hearst. Citances: Citation sentences for semantic analysis of bioscience text. In Proceedings of the SIGIR, volume 4, pages 81{88, 2004. 23. Hidetsugu Nanba, Noriko Kando, and Manabu Okumura. Classication of research papers using citation links and citation types: Towards automatic review article generation. Advances in Classi cation Research Online, 11(1):117{134, 2000. 24. Tadashi Nomoto. Neal: A neurally enhanced approach to linking citation and reference. In BIRNDL@ JCDL, pages 168{174, 2016. 25. Brett Powley and Robert Dale. Evidence-based information extraction for high accuracy citation and author name identi cation. In Large Scale Semantic Access to Content (Text, Image, Video, and Sound), pages 618{632. Le Centre de Hautes Etudes Internationales D'Informatique Documentaire, 2007. 26. Vahed Qazvinian, Dragomir R Radev, Saif Mohammad, Bonnie J Dorr, David M Zajic, Michael Whidby, and Taesun Moon. Generating extractive summaries of scienti c paradigms. J. Artif. Intell.
Res.(JAIR), 46:165{201, 2013. 27. J. R. Quinlan. Induction of decision trees. MACH. LEARN, 1:81{ 106, 1986.