We present our approach to the CL-SciSumm 2016 shared task. We propose a technique to determine the discourse role of a sentence. We differentiate between words linked to the topic of the paper and the ones that link to the facet of the scientific discourse. Using that information, histograms are built over the training data to infer a facet for each sentence of the paper (result, method, aim, implication and hypothesis). This helps us identify the sentences best representing a citation of the same facet. We use this information to build a structured summary of the paper as an HTML page.
One’s task in research is to read scientific papers to be able to compare them, to identify
new problems, to position a work within the current literature and to elaborate new
research propositions [
This implies reading many papers before finding the ones we are looking for. With the growing amount of publications, this task is getting harder. It is becoming important to have a fast way of determining the utility of a paper for our needs. A first solution is to use web sites such as CiteSeer, arXiv, Google Scholar and Microsoft Academic Search that provide cross reference citations to papers. Another approach is automatic summarization of a group of scientific papers dealing with the subject.
This year’s CL-SciSumm competition for summarization of computational linguistics papers proposes a community approach to summarization; it is based on the assumption that citances, the set of citation sentences to a reference paper, can be used as a measure of its impact. This task implies identifying the text a citance refers to in the reference paper and a facet (aim, result, method, implication and hypothesis) for the referred text.
We are building a system that given a topic, generates a survey of the topic from a set of papers. That system uses citations as the primary source of information for building an annotated summary. Our system must be able to identify the purpose/polarity/facet of a citation. to direct the reader towards the more relevant information. The summary is built by selecting sentences from the cited paper and the citations. This process uses a similarity function between sentences. The resulting summaries are presented in HTML format with their annotations and links to the original paper. The only task that is not performed by our system is finding the text referred to by the citation. We intend to use the information already found by our system (facet of citations and sentences) to complete that task.
We had already some experience in dealing with scientific papers and their references, having participated to Task2 of the Semantic Publishing Challenge of ESWC2014 (Extended Semantic Web Conference) on the extraction and characterization of citations. A short review of previous work follows in Sect. 2. We will summarize the task in Sect. 3 and the techniques for extracting information in Sect. 4. Finally, Sect. 5 will show our results. 2
There has been a growing attention towards the information carried by citations and
their surrounding sentences (citances). These contain information useful for rhetorical
classification [
Since the first works of Luhn [
Another domain for study over scientific papers is the classification of their
sentences. Teufel [
To find citations inside a paper, we need to analyse the references section.
Dominique Besagni et al. [
For this year competition we were given 30 topics, 10 for training, 10 for tuning and
10 for testing [
There are two mandatory tasks (Task 1A and Task 1B) and an optional task (Task 2)3.
Task 1A : Find the part of the RP that is indicated with each citance. This will be called the referenced text.
Task 1B : Once the referenced text is identified, we need to attribute a facet to it. A facet is one of these : result, method, aim, implication and hypothesis. Task 2 : Building a summary for the RP using the referenced text identified in Task 1A.
Both the training and the developing set of topics contain expected results for these tasks. The next section will describe how our system performs on the test set. 4
For the first task, we have to find the referenced text and its facet. We hypothesized that the referenced text should be sentences sharing the same facet as the citance. We use that fact to reduce the set of sentences to choose from for the reference. This is why we execute Task 1B on all the sentences of the RP and all the citances prior to Task 1A. We now present how we determine the facet of a citance, then the facet for sentences in the RP and finally the referenced text. 4.1
Task 1B : Facet Identification
Our goal is to be able to use our system for papers from different domains, without
having to train them again. Toward that objective, our system only uses words that are not
domain specific. Patrick Drouin [
We trained two systems, one to attribute a facet to sentences in the RP and one to attribute a facet to citances.
We determine the word distribution for each facet using an histogram. We only use words appearing in the TSL. This computation yielded a sum of each words present in all referenced text for each facet. The facet with the highest score is chosen for that sentence.
For training our system, we extract the reference sentences from each annotation with their assigned facet. Each sentence is tokenized using the NLTK library in Python. Only words from the TSL are kept. Our dataset consists of pairs of list of words with a facet : D = [(wsi; fi)].
We build a profile (hf ) for each facet using a histogram. For each word in the lexicon, we compute the number of times it appears in sentence paired with the a specific facet.
hf (w; D) = cnt(w; wsi) =
X [cnt(w; wsi) j (wsi; f ) 2 D]
X [1 j w 2 wsi] When a word appears more then once in a sentence, all its occurrences are counted.
Once the histogram is built, we use it to find the facet of new sentence. First, we extract the words that are part of the lexicon from the sentence, yielding the list of words p. Then a score sf for each facet is computed by adding the profile of each word for that facet. The facet that scored the highest value is assigned to the new sentence. sf (p; D) =
X hf (w; D) w2p
Looking closely at the results for the profile, we saw that some words have a negative effect on finding the facet. To find a better sublist of words to use within the TSL, we used a genetic algorithm that uses a population of lists of words.
A genetic algorithm starts with an initial population (set of possible solutions) and tries to find better solutions by applying small changes to existing solutions. In our case, a solution is a subset of words Li of L. The initial population is built using random subsets.
To build the next generation, we use three different techniques : 1. Adding a random word to an existing solution : L0i = Li+fwg where w 2 (L Li). 2. Removing a random word from an existing solution : L0i = Li fwg where w 2 Li. 3. Combining two subsets of existing solutions : L0i = Lj [ Lk.
Once enough solutions are built for the new population, each solution is tested using cross-validation with the histogram. The list that performed best in the task is kept for the next generation. We use the same technique over the dataset consisting of the citance texts and their facets. 4.2
Task 1A : Finding the Sentences Referred to by Citances Having determined the facet of sentences in both the RP and citances, we are now ready to assign referenced text to citances from the CPs. Our hypothesis is that a citance should have the same facet as the text it refers to. We extract Qf the subset of sentences from the RP that have the same facet f as a citance ci. To choose the sentence of RP referred to by the citance, we look for the sentence from Qf that is the most similar with the citance ci.
1 simmcs(P1; P2)= (hs(P1; P2) + hs(P2; P1)) 2
Pw2Pi ms(w; Pj ) idfw hs(Pi; Pj )= Pw2Pi idfw ms(w; Pj )=max simwup(w; v) v2Pj (1) (2) (3)
We use the similarity function simmcs defined by Mihalcea, Corley and
Strapparava [
Mihalcea et al. [
Task 2 : Summarization
Multiple source summarization adds three problems [
For Task 2, we choose to use the Maximal Marginal Relevance (MMR) proposed by
Jaime G. Carbonell et Jade Goldstein [
arg max si2RnV simmcs(si; Q) (1 ) max simmcs(si; sj ) sj2V (4)
At each iteration, their algorithm adds a sentence si to V . Sentences are choosen so that they bring new information to the summary (Points 1 and 2) and it must have a certain amount of similarity with the query (Point 2). must be adjusted to balance between adding a sentence very similar to the query and a sentence very different from the ones already in the summary V . We use the same metric (simmcs) as for task 1A to compare sentences.
We divided the summarization process in two steps : adding sentences from the citance (R = CT) and adding sentences from the paper (R = RP). In the first step, the algorithm chooses sentences from the set of citances until it reaches 150 words. For that part, we use = 0:3 to give priority to similarity with the query, trying to remove meaningless citances. Since citances have been identified as bringing new information
not present in the original paper, we believe it is important to keep them in the summary. Then, the summary is completed (to 250 words) using sentences choosen from the RP. Here, we use = 0:7. Since sentences are choosen in the RP, most of them are about the same subject, we want to give priority to sentences that are more different.
The summary is built in an XML format. Each sentence is identified with its position (the id of the paper it was extracted from, the sid and ssid attributes inside the XML source files). The citances contain the id of the referred paper. This information will enable to point a reader towards the corresponding paper.
To help analyse the summaries, our software builds an HTML page containing the extracted information (see Fig. 1). 5 5.1
Task 1 We present our results for facet attribution to citance and reference text. The set of data we receive is divided in two : the training set contains 197 sentences distributed over the citances and 247 sentences over the reference text; the development set contains 273 sentences distributed over the citances and 330 sentences over the reference text. We first train our system using the training data (T) and then we retrained it using both set training and developing set together (TD). In each case, we test the result over both sets. We show the result for simple training of the histogram and for the training using the genetic algorithm (gen_T) to select the list of words to consider. We also trained our histogram without limiting to the words in the TSL for comparison purpose.
For the genetic algorithm, we let it run over 25 generations. Each generation started with 1 000 lists of words. 9 000 lists are added using the proposed mutations, bringing the number of lists to 10 000.
The result of these experiments are presented in Table 1 and Table 2. We see that, using the training set T gives good result on itself but lower result when we apply it on the development set. After training with both set TD (Test + Development), the result over the development set raises at the expense of the result for the training set. For citance, the genetic algorithm yields better result over the training set only. It does not help to get better histograms. Considering that fact, we ask ourselves if it is possible to obtain better results using histograms, or if we have reached the limit of that technique? Limiting our choice of words to the TSL did not give lower results. It is to be tested if the histograms built with the TSL will perform better in another domain than computational linguistics.
Once we had identified the facets, we ran our script for finding the reference text. It was able to reach an F1 score of 0.095 over the training set and 0.052 over the development set (table 3). We reduced the search space for the referred text using the facet of the citance. Since the identification of the facet is not perfect, this reduction might remove a sentence we are looking for. In the future, we have to test our approach with all sentences, instead of the reduced set, to see if this reduction of space causes a problem more than helps the solution. Figure 1 shows the HTML interface we have generated for showing the result of our system. It allows for selecting different topics. The top of the page lets us choose between the different topics that were summarised. Each topic will present, on the left side, the text of each CPs and RP. The sentences have been divided and citance identified. The right side contains the different summaries that our software builds (using different values of ) and the gold standard summary. Each paper links to its pdf version on the ACL Anthology5.
On the left side of the top part of the figure we see the RP divided in sentences. On the right side, there is a summary built by choosing five sentences from the set of citances using a of 0.3. These sentences where selected to be as different as possible by the MMR algorithm. The bottom screen shoot (Fig. 1) presents one of the CP on the left. The citance and citation are colored to be easy to identify. The third sentence from the top was selected by the algorithm for the summaries.
6
We presented the use of distinguishing between topic and non-topic (TSL) words for determining the facet of sentences in a paper. This technique is useful because it lets our system work on paper in a domain independent way. We obtained good results with a simple histogram. We still have to test our histogram over other domains, to see if they also yield good results. Our experiments with a genetic algorithm to refine the list of used words did not show any improvement.
We presented our interface for browsing the results of our system. That interface presents RP, CPs and summaries with links to the original paper. This interface helps the reader browse through a topic.