This paper introduces the National Centre for Text Mining - University of Manchester systems submitted in CL-SciSumm 2019 Shared Task at BIRNDL 2019 Workshop. CL-SciSumm shared tasks focus on the identi cation of cited passages across scienti c publications, and the subsequent summarisation of scienti c articles based on their cited extracts. More speci cally Task 1A is directed at the identi cation of cited text spans in the reference paper, based on the provided citation passages, while Task 1B concerns the classi cation of the citation passages based on their function in the text. For Task 2, the identi ed cited text spans are used in order to generate an informed summary for the reference paper. We participated in both tasks described above. We looked into supervised and semi-supervised approaches and explored the potential of adapting bidirectional transformers for each task. We further formalised Task 1A as a similarity ranking problem and implemented bilateral multi-perspective matching for natural language sentences.
In scienti c publications, citations can serve a range of di erent functions,
targeting di erent aspects of the referenced publication. For example, some
citations aim to compare to methods or results of the referenced paper, some intend
to build upon the cited methods, while others aim to corroborate or dispute a
given hypothesis or conclusion. [
The aforementioned observations kindled the interest in citation-based
summarisation methods for scienti c publications, which aim to combine information
from citing sentences in order to improve the summary of the referenced article
[
For this reason, it has been proposed that cited text spans of the referenced
article could provide less biased information to support the scienti c
summarisation task. The CL-SciSumm Shared Tasks [
More speci cally, the CL-SciSumm 2019 task [
T1A: For each citance (i.e. a citation sentence that references the RP), identify the spans of text (cited text spans) in the RP that most accurately re ect the citance.
T1B: For each cited text span, identify what facet of the paper it belongs to, from a prede ned set of facets namely: Method, Aim, Implication, Results or Hypothesis.
T2: (Optional) Generate a structured (of up to 250 words) summary of the RP.
We approached Task 1A in two di erent ways, using sentence similarity and
sentence pair classi cation methods. In the rst approach, we investigate the
potential of the BERT bidirectional transformer model [
Looking at the summarisation (Task 2), there are generally two approaches currently being adopted to create a summary from an input text: extractive and abstractive. Extractive summarisation produces a summary by choosing a subset of sentences related to the main idea of the input document. Abstractive summarisation, in contrast, generates summaries by modifying phrases and sentences from the input. It is relatively di cult to properly create attractive summaries because it requires semantic analysis. To ensure we get grammatically correct summary for Task 2, we focus on creating the summary by using extractive methods.
For this task, the system needs to generate a structured summary from the cited text spans of the input reference paper. While cited text spans capture the main points of interest for the scienti c community, the paper's full text gives more detailed information about its content, which is useful for the summary. According to our analysis on the provided training set, only a small amount of text (16%) was taken from the cited text spans while a majority of them (76%) was from the rest of the full text. Because of this, besides using cited text spans, we also employ the full text of the paper in our approach.
For Task 1, the organisers provide two di erent datasets for training: (1) a
manually annotated dataset comprising 40 articles and their respective citing papers,
which was also used in the 2018 CL-SciSumm challenge, and (2) a dataset of
1000 articles and their respective citing papers, which were automatically
annotated with a neural network approach as described in [
In order to estimate the performance of the methods discussed in Sections 3.1 and 3.2, we keep 20% of the 2018 dataset (eight randomly selected articles) on the side to be used as a development set1.
For Task 1A it was necessary to extract sentence pairs between citing sentences and text spans from the Reference Papers (RP), that would then be used as training instances for our models to learn how to classify or rank such pairs as valid or invalid citing-cited sentence pairs. We pre-processed the provided annotation les (.ann) as well as the XML les for the RP in order to extract positive and negative pairs. For the positive pairs, we used the sentences as provided in the .ann les. We applied a set of sentence reconstruction rules to sentences that were erroneously segmented by the OCR (e.g., erroneously segmented after parentheses, enumeration or abbreviations)2. The same pre-processing was applied to all sentences of the RP.
For the generation of negative pairs, each citation sentence was paired to
randomly selected sentences from the RP. The RP sentences to be used for the
negative pair generation were further processed as following: Each candidate
sentence was tokenised3, and then each token was lemmatised. Subsequently
each lemma was looked up against WordNet [
RP sentences there is a certain percentage of overlapping vocabulary4. Nearly 1 The ids of the papers used for validation are: C00-2123, C04-1089, I05-5011, J963004, N06-2049, P05-1004, P05-1053, P98-1046 2 The original sid and ssid o sets of the reconstructed sentences were indexed and restored for the nal system outputs. 3 NLTK Tokenizer was used for the tokenisation in all pre-processing steps 4 It is noted that we removed stop words and symbols when calculating overlapping vocabularies. half of the positive pairs in the 2018 training set have a vocabulary overlap of size >= 2. In an attempt to assess and control the impact of word overlap on the information learned by our models, we also experimented with the henceforth called \overlap-controlled" generation of negative pairs. In this case, negative pairs were selected so that the word overlap between the citing sentence and the reference sentence was maximised. Hence we obtained two additional datasets, the 2018 overlap-controlled (OV) dataset and the 2018-2019 overlap-controlled (OV) dataset.
The position of the RP sentence within the document (sid o set) and the document section (ssid o set) were also encoded in the generated pairs, and used as additional feature in some of the Task 1A methods (see feature-based BERT approach) as well as for Task 1B. 2.2
In order to prepare the data for Task 2, we rst lter out too long (more than 45 tokens) or too short (less than 5 tokens) sentences. Any unrelated sentences (i.e., sentences which belong to \Acknowledgment" or \References" sections) are also removed. We then tokenise the text using the stanford-corenlp toolkit5.
The provided training data (2018 dataset) was created using abstractive summarisation methods, which are not suitable to use for extractive summarisation models. To identify which sentences should be put into the extractive summary, we greedily selected sentences which can maximise the ROUGE scores to create an extractive summary version of the originally provided data. To generate training data for the classi er (described in Section 3.3), we assigned label 1 to sentences selected in the extractive summary version and 0 otherwise, thus obtaining positive and negative instances. 5 https://stanfordnlp.github.io/CoreNLP/
We considered two main approaches for the identi cation of cited text spans, both centred around the concept of identifying sentence relevance/similarity between the citing and cited text spans.
BERT-based model. In the rst approach we explore the potential of
netuning bidirectional transformers, and more speci cally the pre-trained BERT
model [
For all experiments that use BERT models, the relevant code is implemented in python and using the pytorch library. The BERT-based classi ers are built on top of BERT models as provided in pytorch by huggingface on github 6.
We used the bert-base-uncased model for the experiments, which has the following set-up: 12 layers, hidden vectors of size 768 and 12 self-attention heads. We initially ne-tuned the model trained for the next sequence classi cation task on both the 2018 dataset and the 2018-2019 dataset, as well as the respective OV versions. For the 2018-2019 dataset we used two training approaches: 1. Use the 2018-2019 dataset and randomly sample batches for all epochs. 2. Start with (1) until convergence and then continue by sampling only from the 2018 dataset for a few epochs, using weight and learning rate decay (henceforth referred to as 2018FT approach).
We also experimented with using BERT base model in a feature-based
approach, to extract features that were then used as input in a Convolutional
Neural Network (CNN). For this purpose we used the concatenation of the last
4 layers of the BERT model to extract a feature vector for each token as it has
been shown to achieve optimal performance according to [
For the feature-based approach, the position of the RP sentence in the document (sid o set) and the position of the RP sentence in the section (ssid o set) were also added as features. We call those features position features and they are concatenated with the CNN output and used as input for the linear layer. 6 https://github.com/huggingface/pytorch-pretrained-BERT version 0.4.0, which has been veri ed to reproduce the outputs of the original TensorFlow implementation.
Since BERT is pre-trained on data from the general domain, we wanted to
also experiment with models closer to the CL-SciSumm domain. For this reason
we employed two di erent approaches:
1. Fine-tuning the weights of the bace model on the ACL anthology reference
corpus (ACL-ARC) [
For ne-tuning on the ACL-ARC corpus, we aimed to ne-tune the BERT
base model weights for the next sentence prediction task. We pre-process the
corpus to lter out sentences with low OCR quality. For the sentence ltering
we rst use the OCR parsing con dence score, and reject sentences with score
<= 0:6. Subsequently we use a rule-based approach to correct sentences that
have been erroneously segmented by the OCR. We then end up with a set of 7M
sentence pairs. Of those, half are consecutive sentences and the rest randomly
chosen sentence pairs. We use the ne-tuning approach described in [
is probably some correlation between citing and cited text spans, e.g., they may
be paraphrase of each other or they may have some inference relation, we
employed Bilateral Multi-Perspective Matching model (BiMPM) [
In this work, we used Glove pre-calculated embeddings [
For Task 1B we did not use any information from the citing sentence. The features were generated exclusively based on the identi ed cited text of the RP. While it is a multi-class, multi-label problem we concluded in building separate binary classi ers for each facet label and subsequently concatenating the positive output for each label. The motivation behind this approach is the imbalance in the label proportions of Task 1B (see Figure 1).
We experimented with two approaches: (1) A Random Forest (RF) classi er
[
For the RF classi er implementation we used the scikit-learn [
For the BERT classi ers, we used the same set-up described in Section 3.1 for the feature based approach. We use the BERT sentence classi cation model provided for pytorch by huggingface7, and we train separate binary classi ers for each facet type.
Both approaches proved to be weak in identifying \Hypothesis " cases,
probably because of the very low amount of training data (only 18 instances). For
that reason we used an adaptation of the rule-based approach described in [
We formulate the summarisation task as a classi cation problem. The classi er needs to classify the sentences in the input document into two classes: included or not included in the summary. We then rank the sentences based on how likely they are to be included in the nal summary. From the ranked list, we add the sentences into the nal summary one by one ensuring that there is no trigram overlap between the current summary and the sentence to be added. The process stops when the summary reaches the maximum length (250 words in this task).
The classi er we use is similar to the one of Liu [
The small size of the CL-SciSumm dataset rendered it harder to train any neural model. To solve this, we train all of our models using the data from SciSummNet. The bene t of this approach is that we can take advantage of its large size. The drawback, however, is that all summary sentences in SciSummNet were taken from the original paper which makes them all subjective sentences. After we obtain the summary, we apply simple rule-based heuristics (for example, change \our" to \their") to convert the subjective sentence to an objective one. 4 4.1
For the evaluation results presented in this section, we generate all possible citing-cited sentence pairs for each RP and then apply the trained models on each pair. For each citing text span we choose the top three scoring pairs and return them as the predicted positives 8.
Our observations on the training set show that RP sentences are repeatedly cited from di erent CP citing sentences. Table 2 shows that half of the RP sentences are cited twice while the others are cited from 3 to 17 times in the 2018 dataset. We observed that this fact might be biasing our models in favouring speci c sentences, but it is also signi cantly a ecting the calculated performance in the case of missing highly repeated sentences.
BERT-based model. The experiments on directly ne-tuning the pre-trained BERT base model are presented in Table 3. We notice that the best performance is obtained when ne-tuning exclusively on the 2018 dataset, despite the small number of training samples. When incorporating the automatically annotated 8 The BERT-based classi ers output a score for each class [invalid, valid]. The top three scores for the \valid" annotated pairs are used. If there is no \valid" pair for a given citing text span we return the best scoring \invalid" pair 2019 data, the false positive rate increases for all classi ers. Even when using the 2018 dataset exclusively at the end of the training (2018FT), the classi er cannot exceed performance obtained on 2018 dataset. Moreover, the approach of using the overlap-controlled datasets for training yields lower performance results for all models. This could be attributed to the fact that by controlling the word overlap between candidate sentence pairs, we are implicitly reducing the vocabulary size that we ne-tune on, leading to classi ers that do not generalise well on unseen data.
In Table 4 we can see the results for the ACL model. Based on the results of previous experiments (see Table 3) we refrained from evaluating the performance on the overlap-controlled datasets. We can see that we obtain a small improvement, both on the 2018 and 2018-2019 datasets. The addition of the 2018-FT however, does not boost performance as in the case of the BERT base model. Still, we can argue that ne-tuning the pre-trained model on data from the speci c target domain can aid in improving the model.
Finally we present the results from the feature-based BERT experiments, using the BERT-base and the SciBERT models. We evaluated those models only on the 2018 dataset, as shown on Table 5. Both models reach similar performance, without a signi cant boost from the SciBERT approach. This could be attributed to the higher proportion of biomedical documents compared to computer science ones, in the training data used for SciBERT. Hence the model might be a better t for the biomedical domain. BiMPM model Table 6 reports the performance of BiMPM model on the development set. Although the 2019 dataset was automatically generated, by combining it with the golden 2018 dataset, we could obtain the best performance, which was signi cantly better than that on the 2018 dataset. Meanwhile, using the overlap-controlled strategy for generating negative pairs could slightly improve the performance on the 2018 dataset but not on the 2018-2019 dataset.
Similarly to the above-mentioned situation in the training set that RP sentences were repeatedly cited in reference papers (see Table 2), the BiMPM model also favoured some certain RP sentences. For example, with the 2018 dataset, the model could predict only 61 sentences as cited text spans for 349 CP sentences of the development set, which explains why its performance was unexpectedly low. In the case of the 2018-2019 dataset, the number of predicted RP sentences was 151, which is more diverse than that of the 2018 one. 4.2
In Table 7 we can see that the BERT classi er obtains better performance for the highly represented labels, but fails to learn the underrepresented ones. The RF classi er seems to perform better on those labels, but it should be noted that when applied on the testing data it failed to identify any \Hypothesis " citations. We use ROUGE-1, ROUGE-2, and ROUGE-L scores for evaluation. We use ScisummNet data for training and report the result on the CL-Scisumm training data 2019 as well as the CL-Scisumm test data 2016. All models use BERT base uncased model with 50,000 training steps. Table 8 show the results on four di erent settings where the model selects the sentences from.
Based on our observations, most of the summary sentences are selected from the beginning of the input document. Indeed, the abstract alone can yield the best ROUGE-2 score (25.54), although the ROUGE-1 and ROUGE-L scores are lower than the scores in our proposed method. This result may be explained by the fact that the abstract has already conveyed most of the ideas described in the paper. It is also because of the way the training data (ScisummNet) was created: the human annotators only read the abstract and the cited text spans from the paper. 5
For Task 1A we submitted 11 runs, and used the RF classi er for Task 1B. We can see that similarly to our experiments in Section 4.1 the ACL model seems to outperform other approaches. However, with the exception of the BiMPM model (run 11), most systems show a signi cant drop of performance when applied on the testing data, pointing to weak generalisation of the models. Still, the ACL model outperformed other submissions in the 2019 CL-SciSumm task.
For Task 2, we submitted only one model which augments the original abstract of the paper using sentences from the full papers to create the summary. Table 10 shows the results obtained from the submitted system on the testing data. The best score is obtained with the abstract-based evaluation, which can be explained since we opted for an abstract augmenting approach. 6
We have described the systems developed to participate in Tasks 1A, 1B and 2 in the CL-SciSumm 2019 shared task. For Task 1A we implemented two methods, which use neural networks to learn the relation between citing and cited text spans; bidirectional transformers (BERT-based) and BiLSTM networks (BiMPM-based). We showed that the BERT-based models could e ciently be trained on the manually annotated data, but could not bene t from automatically annotated one. Instead the BiMPM-based method showed signi cant improvement when trained on large data, even if it was automatically annotated (i.e., noisy). For Task 1B, we resorted to using an RF classi er over a BERT-based approach, since it could handle a smaller training dataset and under-represented labels better.
On Task 2, in order to take advantage of the informative sentences that authors provided in the abstracts, we augmented the abstract with selected sentences from the full text. The experimental results have shown that this approach outperformed the one that only used extracted sentences from full text. Acknowledgments. This work was partly supported by the EPSRC Doctoral Prize award; the HSE Discovering Safety, Lloyds Register Foundation; and Thomas Ashton Institute.