The CL-SCisumm track provides a framework to evaluate systems summarising scienti c papers. It includes datasets and metrics provided by the organisers. The track comprises three tasks: (1a) identifying the spans of text in the referred document re ecting citing text spans (i.e., citances), (1b) classifying discourse facets of the cited text spans, and (2) generating a short structured summary. For the 2019 edition, we focused our work on the task 1a. This report presents our proposed approach for this task. We submitted 15 runs corresponding to di erent con gurations of the parameters involved in our approach.
Based on our hypothesis presented in the previous section we de ned a two-step approach described in the following sections.
Step 1: Identi cation of Candidate Sentences To identify candidate sentences, we converted this problem into a standard binary classi cation problem. Considering a training dataset DT , a sentence belongs to the positive class if it is targeted by at least one CP. Otherwise, the sentence belongs to the negative class. We then designed features in order to characterise candidate sentences of RPs related to four main categories: bibliographic features, conceptual features, positional features, and features based on word distribution. The features used in our approach are presented in Tab. 1.
For features based on the word distribution, the Iramuteq software [
Second, we applied a machine learning approach, i.e., XGBoost [
Step 2: Computation of Sentence{Sentence Similarities
Here, we aimed to nd the most similar sentences in the RP for each citance. We
represented sentences as vectors whose values stem from applying tf idf-based methods or applying
embedding-based methods such as Word2vec [
Simple tf idf. We rst decided to evaluate a very simple representation of candidate
sentences and citances, based on the well-known Vector Space Model. The vector representation
of a sentence is thus based on tf idf. We considered two di erent vocabularies: one composed of
all terms in RPs, and one after performing a POS-tagging using the Python library Spacy [
Embedding-based Methods. For embedding-based word representation, we trained a deep
learning model on our in-house WoS-CS corpus. This consists in textual data pulled from the
Web of Science, covering the 1.6 million abstracts (of length 140+ characters) of all 2005{2018
articles and proceeding papers published in venues listed in the following four elds of the
`Computer Science' subject unit: Information Systems, Arti cial Intelligence, Interdisciplinary
Applications, Theory & Methods. Prior to training the model, we produced a lowercased,
diacritic- and punctuation-free version of WoS-CS. We then fed it to Word2vec [
Given a sentence s, we averaged the embedding vectors of words in s. To avoid considering non-informative words i in the aggregation process, we rst performed a POS-tagging using the Python library Spacy. We then averaged vectors of nouns, adjectives, and verbs only. Note that this POS ltering is not performed for all of our runs as speci ed in Sect. 4. Matching between Citances and Candidate Sentences in the RP. Once the vector representation of sentences has been calculated, we then computed the cosine similarity for
Normalized sentence position in the paper { sentencePosition=numberOfSentences
ssid=max(ssid) for other sentences Normalized section number { 0 for titles, sectionNumber=max(sectionNumber) for other sections (the Acknowledgements section was numeroted by following the previous sections) Sentence's section label, compared to a prede ned set of labels (fabstract, introduction, model, method, results, experiments, conclusion, rw, othersg). The label is assigned regarding the keywords found in the title of the section.
Pos Pos Pos Pos Neg Neg Pos Pos Pos Pos Pos Pos Pos Pos Pos Pos Neg Neg Neg Neg each pair of (candidate target sentence, citance) and ranked the candidate target sentence in decreasing order of similarity. A maximum of n target sentences were nally selected with a similarity greater or equal to a threshold . 3
We carried out a series of preliminary experiments aiming to draw a set of e ective candidate con gurations of our system. We used the test set and the evaluation framework of the 2018 CL-SciSumm edition as well as the training set of 2019 to compare various con gurations of our system according to their F1-scores for sentence overlap on Task 1a.
One experiment intended to evaluate the interest of using automatically vs. manually annotated documents for training. As shown in Tab. 2, we experimented our system rstly training on the Training 2019 set and testing on the Training 2018 set, and secondly training on the Training 2018 set and testing on the Training 2018 with cross-validation. The comparison of the obtained evaluations did not lean towards a training on the automatically annotated documents of the Training 2019 set. As a consequence, we built runs using three training sets for the rst step: Training 2018, Training 2019, and Training 2018 + Training 2019.
Other preliminary experiments were conducted in order to identify various con gurations
of the components (embeddings generation, with or without POS-tagging) and the parameters
( , n, ) performing well in the context of the 2018 edition [
We submitted 15 runs based on our approach to address the subtask 1a. The submitted runs correspond to di erent con gurations of the components and parameters involved in our approach (Tab. 3).
On the one hand, the rst varying parameter was the training set used for the rst step among Training 2018, Training 2019, and Training 2018 + Training 2019 sets as mentioned in the previous section. The other parameters were the threshold used to select the sentences to retain as candidate for being target of citances, the threshold used to select the sentences best matching the citances, and nally the maximum number n of selected target sentences. On the other hand, the varying components were generating word embeddings or not for the vectors representing sentences and applying POS tagging or not.
A rst conclusion that can be drawn is that the simple tf-idf representation outperforms the one based on embeddings. Best results are obtained when returning 3 sentences per reference text span.
Surprisingly however, results are strictly simlilar whatever the training set used. Further investigations are needed to understand these results. We are also waiting for the ground truth to perform a failure analysis and deeply investigate on the e ectiveness of each step of our approach.