In this shared task, we applied “Learning to Rank” algorithm with multiple features, including lexical features, topic features, knowledge-based features and sentence importance, to Task 1A by regarding reference span finding as an information retrieval problem. Task 1B, discourse facet identifying, is treated as a text classification problem by considering features of both citation contexts and cited spans.
The 2nd CL-SciSumm Shared task follows the TAC 2014 Biomedical Summarization
Track on scientific paper summarization. An overview of the shared task, including
specific details on the dataset, the competitive results and subsequent analyses for each
task can be found in the shared task overview paper[
Task 1A We considered Task 1A as an information retrieval problem. A citance (citation context) is regarded as a query and sentences from reference paper (cited spans) are treated as candidate documents. Then, the problem becomes how to rank the sentences of the reference paper (i.e., candidate cited spans in this report) for a given citance. The most relevant sentences of the reference paper to a citation context are selected as the golden sentences of the citation context. We apply “learning to rank” algorithms to address this problem and exploit multiple features. The explored features are as follows: • Lexical Features. Bag of words is a widely used text representation method. By representing citation contexts and candidate cited spans with the bag of words model, the lexical similarities between citation contexts and candidate cited spans can be obtained. We chose four candidate lexical similarity features including Cosine similarity, Jaccard similarity, Dice similarity and LCS (longest common subsequence). When computing cosine similarity, TFIDF term weighing was applied. In the formula below, TF is the number of times a term occurring in a given sentence, while the IDF value of a term is computed from all the 437 papers in the training set (93 papers), development set (155 papers), and test set(229 papers). Thus, curs.
__ _ is 437,
which is the same for all terms. And
) is the number of the different papers that a term
ocTFIDF(Term) = TF(Term) ∗ ln
( )
(1)
• Topic Features. Bag of words model is insufficient in handling polysemy and
synonym problems. Taking topics into account can relieve this problem. Topic
modeling method[
Task 1B
Task 1B is considered as a text classification problem. The five discourse facets of a
sentence in a reference paper are Aim, Method, Result, Implication, and Hypothesis.
The features adopted by the classifiers are as follows.
• The text of the reference sentence
• The title of the section that the reference sentence belongs to
• The section type of the section that the reference sentence belongs to
• The text of the citation context
• The title of the section that the citation context belongs to
• The section type of the section that the citation context belongs to
Three classifiers from Weka[
Evaluation
Two groups of precision (P), recall (R) and F_1 measures were used to evaluate the performance in Task 1A. The first group counted the number of sentences returned by our methods that match the gold standard sentences annotated by the task organizers. = | ⋂ | | | = | ⋂ | | |
2 ∗ ∗ F1 = ( + ) (2) where G indicates the gold standard sentences, S denotes the sentences returned by our methods.
The second group of measures are ROUGE_1[
Precision (P), recall (R) and F_1 measures were calculated to evaluate the performance for each facet. For one facet, a is denoted as the number of correct predictions for this facet, b is the number of wrong predictions for this facet, and c is the number of cited sentences from gold standard sentences for this facet that are not predicted as belonging to this facet.
R = a+ac P = a+ab 1 = (2R∗R+∗PP) (3)
Then, Macro average and Micro average were used to evaluate the system on all facets. Macro average measures are computed as: Micro average measures are computed as:
Rmacro = ∑ N = ∑ N 1 = 2∗R +∗
∑ ∑ 2∗R ∗ = ∑ f+∑ f = ∑ +∑ 1 = + (4) (5) where f is the facet index, N is the number of facets (i.e., 5). A good classifier should have both high Macro and Micro average measures. 3
In this section, the results where the training set is used for training and the development set is used for testing. 3.1 Task 1A results According to Fig.1, Jaccard similarity and Dice similarity achieved better F_1 measures than Cosine similarity and LCS. Topic similarity feature (indicated as Topic_number in Fig.1) showed slight different performance among different numbers of topics. WordNet based similarity achieved best results when combining the results of standalone nouns similarities and verb similarities. TextRank showed similar results to topic similarity and WordNet based similarity, however, this feature did not improve the performance of learning to rank. Finally, Jaccard similarity, Topic similarity with 200 topics, WordNet (N,Vsep), and TextRank were kept. In the development set, RankNet and AdaRank achieved best performance with Overlap/ ROUGE_1 F_1 of 0.057/0.211 (Table 1). 0.3 0.25 0.2 0.15 0.1 0.05 0
Micro Avg Macro Avg
For the last run, we used both training set and development set to train instances. We chose Jaccard, Topic_200, WordNet (N+V) and TextRank as the final ranking features and applied AdaRank learning to rank algorithm on Task 1A. For Task 1B, Naïve Bayes was used as the classifier.
Recognizing the cited spans and determining their discourse facets are very challenging for the summarization of scientific papers. There are some issues to be addressed during our study on the tasks. First, either cited span discovery or discourse facet classification is an unbalanced problem—negative examples are more prevalent than positive examples. That would be our future work. Second, Task 1A involves much more than a similarity problem. This is because the underlying citation intentions are complex. More features that reflect the citation intentions should be explored.