The CL-SciSumm Shared Task is the rst medium-scale shared task on scienti c document summarization in the computational linguistics (CL) domain. In 2019, it comprised three tasks: (1A) identifying relationships between citing documents and the referred document, (1B) classifying the discourse facets, and (2) generating the abstractive summary. The dataset comprised 40 annotated sets of citing and reference papers of the CL-SciSumm 2018 corpus and 1000 more from the SciSummNet dataset. All papers are from the open access research papers in the CL domain. This overview describes the participation and the o cial results of the CL-SciSumm 2019 Shared Task, organized as a part of the 42nd Annual Conference of the Special Interest Group in Information Retrieval (SIGIR), held in Paris, France in July 2019. We compare the participating systems in terms of two evaluation metrics and discuss the use of ROUGE as an evaluation metric. The annotated dataset used for this shared task and the scripts used for evaluation can be accessed and used by the community at: https://github.com/WING-NUS/scisumm-corpus.
CL-SciSumm explores summarization of scienti c research in the domain of
computational linguistics research. It encourages the incorporation of new kinds of
information in automatic scienti c paper summarization, such as the facets of
research information being summarized in the research paper. CL-SciSumm also
encourages the use of citing mini-summaries written in other papers, by other
scholars, when they refer to the paper. The Shared Task dataset comprises the
set of citation sentences (i.e., \citances") that reference a speci c paper as a
(community-created) summary of a topic or paper [19]. Citances for a reference
paper are considered a synopses of its key points and also its key contributions
and importance within an academic community [16]. The advantage of using
citances is that they are embedded with meta-commentary and o er a contextual,
interpretative layer to the cited text. Citances o er a view of the cited paper
which could complement the reader's context, possibly as a scholar [
The CL-SciSumm Shared Task is aimed at bringing together the summarization community to address challenges in scienti c communication summarization. Over time, we anticipate that the Shared Task will spur the creation of new resources, tools and evaluation frameworks.
A pilot CL-SciSumm task was conducted at TAC 2014, as part of the larger
BioMedSumm Task4. In 2016, a second CL-Scisumm Shared Task [
CL-SciSumm de ned two serially dependent tasks that participants could attempt, given a canonical training and testing set of papers.
Given: A topic consists of a Reference Paper (RP) and ten or more Citing Papers (CPs) that all contain citations to the RP. In each CP, the text spans (i.e., citances) have been identi ed that pertain to a particular citation to the RP. Additionally, the dataset provides three types of summaries for each RP: { the abstract, written by the authors of the research paper. { the community summary, collated from the reference spans of its citances. { a human-written summary, written by the annotators of the CL-SciSumm annotation e ort.
Task 1A: For each citance, identify the spans of text (cited text spans) in the RP that most accurately re ect the citance. These are of the granularity of a sentence fragment, a full sentence, or several consecutive sentences (no more than 5). Task 1B: For each cited text span, identify what facet of the paper it belongs to, from a prede ned set of facets.
Task 2: Finally, generate a structured summary of the RP from the cited text spans of the RP. The length of the summary should not exceed 250 words. This was an optional bonus task. 3
We built the CL-SciSumm corpus by randomly sampling research papers (Reference papers, RPs) from the ACL Anthology corpus and then downloading the 4 http://www.nist.gov/tac/2014 5 http://sigir.org/sigir2017/ citing papers (CPs) for those which had at least ten citations. The prepared dataset then comprised annotated citing sentences for a research paper, mapped to the sentences in the RP which they referenced. Summaries of the RP were also included.
The CL-SciSumm 2019 corpus consisted for 40 annotated RPs and their CPs.
These are the same as described in our overview paper in CL-SciSumm 2018 [
For details of the general procedure followed to construct the CL-SciSumm
corpus, and changes made to the procedure in CL-SciSumm-2016, please see
[
Annotation The rst annotated CL-SciSumm corpus was released for The CL-SciSumm 16 shared task. This was annotated based on annotation scheme from what was followed in previous editions of the task and the original BiomedSumm task developed by Cohen et. al6: Given each RP and its associated CPs, the annotation group was instructed to nd citations to the RP in each CP. Speci cally, the citation text, citation marker, reference text, and discourse facet were identi ed for each citation of the RP found in the CP.
Then CL-Scisumm-17 and CL-Scisumm-18 incrementally added more annotated RPs to its current size of 40 annotated RPs.
For CL-Scisumm-19, we augment this dataset both Task 1a and Task 2 so that they have approximately 1000 data points as opposed to 40 in previous years. Speci cally, for Task 1, we used the method proposed by [17] to prepare noisy training data for about 1000 unannotated papers. This method involves automatically matching a citance in a CP with approximately similar reference spans in its RPs. The number of reference spans per citance is a hyperparameter that can set as input. For Task 2, we used the SciSummNet corpus proposed by [23]. 4
Nine systems out of the seventeen registered systems { in Task 1 and a subset of ve also participated in Task 2 { submitted their output for evaluation. We include these system papers in the BIRNDL 2019 proceedings. We will now brie y summarise their methods and key results in lexicographic order by team name.
System 1 is from Nanjing University of Science and Technology [
Compared with previous work, this year they make new selection of features based on correlation analysis, apply similarity-based negative sampling strategy when creating training dataset and add deep learning models for classi cations. For Task 1B, they rstly calculate the probability that each word would belong to the speci c facet based on training corpus and then some prior rules are added to obtain nal result. For Task 2, to obtain a logical summary, they group sentences in two ways, rst based on their relevance between abstract segments and second arranged by recognized facet from task 1B. Then they pick out important sentences via ranking.
System 2 is from Beijing University of Posts and Telecommunications (BUPT)
[
System 3 is from University of Manchester [24]. For Task 1 they looked into supervised and semi-supervised approaches. They explored the potential of netuning bidirectional transformers for the identi cation of cited passages. They further formalised the task as a similarity ranking problem and implemented bilateral multi-perspective matching for natural language sentences. For Task 2, they used hybrid summarisation methods to create a summary from the content of the paper and the cited text spans.
System 4 is from University of Toulouse [18]. They focus on Task 1A. They rst identify candidate sentences in the reference paper and compute their similarities to the citing sentence using tf-idf and embedding-based methods as well as other features such as POS tags. They submitted 15 runs with di erent con gurations.
System 7 is from IIIT Hyderabad and Adobe Research [21]. Their architecture incorporates transfer learning by utilising a combination of pretrained embeddings which are subsequently used for building models for the given tasks. In particular, for task 1A, they locate the related text spans referred to by the citation text by creating paired text representations and employ pre-trained embedding mechanisms in conjunction with XGBoost, a gradient boosted decision tree algorithm to identify textual entailment. For task 1B, they make use of the same pretrained embeddings and use the RAKEL algorithm for multi-label classi cation.
System 8 is from Universitat Pompeu Fabra and Universidad de la
Republica [
System 9 is from Politecnico di Torino [20]. Their approach to tasks 1A and 1B relies on an ensemble of classi cation and regression models trained on the annotated pairs of cited and citing sentences. Facet assignment is based on the relative positions of the cited sentences locally to the corresponding section and globally in the entire paper. Task 2 is addressed by predicting the overlap (in terms of units of text) between the selected text spans and the summary generated by the domain experts. The output summary consists of the subset of sentences maximizing the predicted overlap score.
System 12 is from Nanjing University and Kim Il Sung University [
System 17 is from National Technical University of Athens, Athens
University of Economics and Business, and Athena Research and Innovation Center [
An automatic evaluation script was used to measure system performance for Task 1A, in terms of the sentence ID overlaps between the sentences identi ed in system output, versus the gold standard created by human annotators. The raw number of overlapping sentences were used to calculate the precision, recall and F1 score for each system. We followed the approach in most SemEval tasks in reporting the overall system performance as its micro-averaged performance over all topics in the blind test set.
Additionally, we calculated lexical overlaps in terms of the ROUGE-2 and
ROUGE-SU4 scores [
We have been reporting ROUGE scoring since CL-SciSumm 17, for Tasks 1a
and Task 2. Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is
a set of metrics used to automatically evaluate summarization systems [
Task 1B was evaluated as a proportion of the correctly classi ed discourse facets by the system, contingent on the expected response of Task 1A. As it is a multi-label classi cation, this task was also scored based on the precision, recall and F1 scores.
Task 2 was optional, and also evaluated using the ROUGE{2 and ROUGE{ SU4 scores between the system output and three types of gold standard summaries of the research paper: the reference paper's abstract, a community summary, and a human summary.
The evaluation scripts have been provided at the CL-SciSumm Github repository7 where the participants may run their own evaluation and report the results. 6
This section compares the participating systems in terms of their performance. Five of the nine system that did Task 1 also did the bonus Task 2. Following are the plots with their performance measured by ROUGE{2 and ROUGE{SU4 against the 3 gold standard summary types. The results are provided in Table 1 and Figure 1. The detailed implementation of the individual runs are described in the system papers included in this proceedings volume.
For Task 1A, the best performance was shown by System 3 (Team UoM) [24].
Their performance was closely followed by System 12 [
On the summarisation task, Task 2, System 3 (Team UoM) had the best performance against the abstract. System 2 (Team CIST-BUPT) had the best performance for community and human summaries. Again, both are deep learningbased systems. The additional 1000 summaries from SciSummnet as training data has resulted in the improved performance. System 2 was the second against abstract summaries, and system 3 was the second against human summaries. 7
For CL-SciSumm '19, we augmented the CL-SciSumm '18 training datasets for both Task 1a and Task 2 so that they have approximately 1000 data points as
Vs. Abstract
RSU{4 system 12 Run 2 0.266 system 12 Run 3 0.286 system 12 Run 4 0.315 system 17 ntua-ilsp-RUN_NN0.F013 system 17 ntua-ilsp-RUN-NN0T.016 system 2 run1-Jaccard-Casc0a.d2e7-4Voting-LSA_method_3 system 2 run10-Jaccard-Foc0u.s2e9d4-Voting-LSA_method_4 system 2 run11-Jaccard-Foc0u.s2e9d4-Voting-QD_method_1 system 2 run12-Jaccard-Foc0u.s2e9d4-Voting-SentenceVec_method_2 system 2 run13-Voting-1.1-S0u.b3t8it9leAndHfw-LSA_method_3 system 2 run14-Voting-1.1-S0u.b3t8it9leAndHfw-LSA_method_4 system 2 run15-Voting-1.1-S0u.b3t8it9leAndHfw-QD_method_1 system 2 run16-Voting-1.1-S0u.b3t8it9leAndHfw-SentenceVec_method_2 system 2 run17-Voting-2.0-T0e.x3t4C2NN-LSA_method_3 system 2 run18-Voting-2.0-T0e.x3t4C2NN-LSA_method_4 system 2 run19-Voting-2.0-T0e.x3t4C2NN-QD_method_1 system 2 run2-Jaccard-Casc0a.d2e7-4Voting-LSA_method_4 system 2 run20-Voting-2.0-T0e.x3t4C2NN-SentenceVec_method_2 system 2 run21-Voting-2.0-V0o.t3in4g1-LSA_method_3 system 2 run22-Voting-2.0-V0o.t3in4g1-LSA_method_4 Fig. 1: Performances on (a) Task 1A in terms of sentence overlap and ROUGESU4, and (b) Task 1B conditional on Task 1A Human R-2
Human R-SU4 sS4ystem 1 Run 10 Human R-20.237
Human R-S0U.1454 yssytsetemm 1 Run 120Human R-2 0.129347Human R-SU04.1514 system 1 Run 11 ystem 1 Run 1 system 1 Run 131 ystem 1 Run 10 system 1 Run 142 ystem 1 Run 1 system 1 Run 153 ystem 1 Run 12 system 1 Run 164 ystem 1 Run 13 system 1 Run 175 ystem 1 Run 14 system 1 Run 186 ystem 1 Run 15 system 1 Run 197 ystem 1 Run 16 system 1 Run 218 ystem 1 Run 17 system 1 Run 2109 ystem 1 Run 18 system 1 Run 21 ystem 1 Run 19 system 1 Run 220 ystem 1 Run 2 system 1 Run 231 ystem 1 Run 20 system 1 Run 242 ystem 1 Run 21 system 1 Run 253 ystem 1 Run 2 system 1 Run 264 ystem 1 Run 23 system 1 Run 275 ystem 1 Run 24 system 1 Run 286 ystem 1 Run 25 system 1 Run 297 ystem 1 Run 26 system 1 Run 328 ystem 1 Run 27 system 1 Run 3209 ystem 1 Run 28
3 system 1 Run 4 ystem 1 Run 29 system 1 Run 530 ystem 1 Run 3
4 system 1 Run 6 ystem 1 Run 30
5 system 1 Run 7 ystem 1 Run 4
6 system 1 Run 8 ystem 1 Run 5
7 system 1 Run 9 ystem 1 6 0.193 ds_y3s_tecmom21RmrRuununn1it-y8Jaccard-Casc0a.d2e52-94Voting-LSA_0m.1e75th14od_3_community osdy_s4te_mco21mRrRumunnu1n0i9ty-Jaccard-Foc0u.s21e152d21-Voting-LSA0._0m.090.5e91t17hod_4_community ystem 1 7 0.1 ydss_yt31set_mecamob1ms2Rtmrruauncnt81it-1yJ-aJcacacradr-dC-Faos0c.02ua.sd2e95d-84V-Vootitningg- L-QSAD0_.0m1.51e87th4od_31_caobmstmraucntity
1 oydsds_yt_1se4t_me_cmco1om2mmrumunun91nit10iyty-Jaccard-Fo0c.01u.0s2.e152d2-Voting-LQSDA0_._0mm.091e.e51tht47hoodd__14__ccoommmmuunnitiyty
R dyd_s_4t1_se_t_mcecomeo2mtmh2rmoumrdnu_1n20i1ti_y-t12ya-bJsatcrcaracdrt-dF-oFco0uc.0s2ue.05s2d.e32-d92V-oVtointign-gL-SQAeD_n_mt0em.en1etc765hteh74oVodde__c41___mccoeomtmhmomduu_nn2iti_ytyabstract
Jac escy _y1steco 2mr un1ity-J d2-Voting-QD_0m.1 5th4od_1 community escy_stmemeth2ordu_n21_2ha-aubJcsmatcracanracdrt-dF-oFcouc0sue.s2e3d9-Voting-Sent0e.n1c6e7Vec_method_2_haubsmtraanct htcysho_syodsd_tm_e1t3m_h2_aoc2brdous_rmntu2r1nam_21cu-t35oJn-maVitcymc tme aurdityFoc0u.s2e Voti
oting-1.1-S0u.b2d2t65i-t74leAndgH-Sfwen-LQt0eSD.n01A_c.51_me697mVe41et htcho_odmd_e_1t3_h_aocbdos_mt2ram_cutonmitymunity yosdt_e3m_ hsoyds_te1 m_c2hao2burmusmrnmtura1naun31cn-tVityoting-1. -S0u.b2t5i 4leAndHfw-LS0AD.01__.7m1m49ee1ththoodd__31__chaoubmsmmtraauncntity
5-Voting-1.1-S0u.b2t6it7leAndHfw-Q oehysdVosy_dtese1_ctme1_a_m2bhes2urtuhmrnauoandn1_-652V-_Vcotminmg-u1n.1it-yS0u.b2t06iet7leAndHfw-SQD0eD_.01nm_.9t1em159nte16hctoehdVo_de1_c_1a_mbhesuthmraoacdnt_2_community
1ct5 oting-1. -S0u.b2t6i 7l AndHfw-Q oyds_te1mc2omrumnu1n5i-tVy 5leAndHfw-QD0_.1m53thod_1 community esVyescte_me2thruond1_62-_oVhatoiubntsmignt-rag1an-.c1t.-1S-0uS.0b2u.t1b2it4it6leAndHfw-Se0n.t1e7n9ceVec_method_2_haubsmtraanct o3eysd_Vsy_cteseo1ctme_hm2ueum2rtunharniuotny1nd51_-672V-_oVctointmign-mg1-u.21n.-01iSt-y0uTS.0b2eu.t6xb2it750lCiet74lANenANdn-HLdSHfwAfw-_Q-mSD0e_.0t1nmh.9t1oe15nt46h_co3ed_V_ceo1cm_hmueumtnhaiotnyd_2_community
7d V4eys_eVsycteseo_ctme_ me2teuh2rtunohriuoty1_nd62_-_682Va-_oVbhtsoiuntmrigna-agc1nt-.12.-01S-0uTS.0b2eu.t4xb2it654lCiet62lANenANdn-HLdSHfwAfw-_S-meS0ne.0t1neh.07t1ne97cne97_cV4e_eVceo_cm_ meteuhtnohidoty_d2__2a_bhsutmraacnt nd
1 o.1d _ycshasoytubmsesmtemtramau2ncntriutyn17-Voting-2.0-T0e.x2t5C4N -LSA_me0th.1o7d
2 run19-Voting-2.0-T0e.x2t5C7NN-QD_meth0o.1d4_8_391__chaoubmsmmtraauncntity a_mysbhsyestuer c2t run1 -Q sthmtmaeoamdn_2_ruanb921s-90tVr-aoVctoitntign-g2-.20.-0T-0eT.0x2et.5xC2t75CN27NN-SQDeD_nm_temnteh0cto.h1d8_91
e0Vo.1de87_c971_a_mbhesuthmraoacdnt_2_abstract hmysusyemtsethatmeonmd2_r2u_rnuha1nbu92sm-0tVra-aoVnctoitntign-g2-.20.-0T-0eT.0x2et.5xC2t75CN2NN-Sentence0V.1e7c7_method_2_haubsmtraanct
-QD_m th0o.1d8_91 human _my4se_tsetmchtmeoumdn2_imt2ryu_rnuc2iot-ymJ01am-cVucoantritdny-gC-2a.s0c-0aTV.0d2eo.xt2i-tn25CVg45oN-LtNinS-gAS-_eLmnStAet_nhmcoe0deV._t1he367oc_4d_c_mo4me_tmchoumdn_imt2yu_nciotymmunity
e5 osmy n2 myestehtsetomhtmdeou_mdn2_it_2ryua_nbh2su0t2mr-a02Vac-oVnttointign-g2-.20.-0T-0eTV.0x2eot.5xtC2it2n5CNg2N-LN-S-AeSn_etmnetnectnhe0coV.e01deV.07_1c.e471_c_7m_cmeotmehtomhdou_dn2_it_2ya_bhsutmraacnt osmy run bosmytrseaomtmcdeut_mn2it_2ryucrnou2mn02m-13Vu-oVntitnyg-2.0-T0e.x2tC ye h
oting-2.0-V0o.t2in5Ng74N-LQ-SDAe_n_mtmeenet hctheo0oVd.01d_e.6_1c4378__49amcbeosmthraomcdut_n2it_ycommunity ubmysmsymtraseauntmcentmit2y run21-Voting-2.0-V0o.t2in5g4-LSA_metho0dd.0_1_.371_48_c9ahobumsmmtraauncntity
2 run23-Voting-2.0-V0o.t2in5g7-QD_metho omytsmhsymotmsedutmu_enn2mit2i_ytyrun2 n5
a2brsutnra2-c43Vt-oVtointign-g2-.20.-0V-0oV.0t2oi.t2ign52-g62L-SQAeD_n_mtemenetchtehoVodde0__c.41__75_mc76coeomtmhmomduu_nn2iti_ytyabstract uytshmrsayotascedntm_e2m2_c2ruornmu2nm32-u43Vn- Vityoting-2.0-V0o.t2ig7n25-g57Q-SQDeD_nm_temntehctoehdVo0_de.01_c.81_9a86_mb49hesuthmraoacdnt_2_community
o ing-2.0-V0o.t2in5 ymstuenmity2 2 g6-QD_m thod0_.156community tshyosdt_e2m_ha2rubrnsmutnra3a2n-c4Vt-oVtointign-g2-.20.-0V-0oV.0t2oi.nt2in5g2-SentenceVe0c.1_7m7ethod_2_haubsmtraanct ytshDasyo_sdmt_ee2m2t_hc2odrmu_2nm132_-u54Van-boVWitsyotonrtiragndc-g2t-v.20e.-0cV-0HoV.0t2-oi.Cn5t2ig7nN32-g58NQ--SDSe_unmbtetitnlhecoAedVn0_de.01Hc.81_f9h6wmu47-emQthDaon_dm_e2t_hcodm_m1_uanbitsystrauct m nte run i Dy_smtemth2odru_n125c-oWmomrdu2nviteyc- N0 8-QD_m thod_1 community seynstetenmce2Vreucn_2m6e-Wthordd_22v_ehacHub-0H-sm.C0t2-ra.Ca2nc1N2t2N-S-uSbutbitlietlAenAdn0Hd.01Hf.3w1f5w3-SentenceVec_method_2_haubsmtraanct
Fig. 2: Task 2 Performances on (a) Abstract, (b) Community and (c) Human maries. Plots correspond to the numbers in Table 2. opposed to 40 in previous years. Speci cally, for Task 1, we used the method proposed by [17] to prepare noisy training data for about 1000 unannotated papers; for Task 2, we used the SciSummNet corpus proposed by [23]. For CLSciSumm '19 we use the same blind test data used in CL-SciSumm '18.
Based on this we propose the following research questions to comparatively
analyse results from CL-SciSumm '18 with those from CL-SciSumm '19. The
research questions we have are:
RQ1. Did data augmentation help systems achieve better performance?
The best Task 1a performance (sentence overlap F1) this year is 0.126 from
System 3 [24] which is a deep learning system trained on augmented data. This
is about 0.02 lower than the best CL-SciSumm'18 system [22] which was at 0.145.
It appears that the data augmentation has helped deep learning methods. The
only fully deep learning system from CL-SciSumm '18 [
Note that there was no data augmentation to Task 1b. So, the performance of traditional methods across CL-SciSumm '18 and CL-SciSumm, '19 are largely the same.
The best on CL-SciSumm '19 Task 2 performance on human written
summaries on ROUGE-2 is 0.278 by [
RQ2. CL-SciSumm '19 encouraged participants to use deep learning based methods; do they perform better than traditional machine learning methods? In Task 1a the best performing CL-SciSumm '19 system The best performing CL-SciSumm '18 system [22] used traditional models including random forests and ranking models trained on the CL-SciSumm '18 training data. This implies that for Task 1a, traditional models trained on clean data perform better than deep learning models trained on noisy data. However, if we look at CL-SciSumm '19 systems' performances, we notice that deep learning models perform better than traditional machine learning models when trained on the augmented data.
On Task 1b, systems using traditional methods perform better than deep learning systems. Note that the winner for Task 1a, System 3, is not the best system for Task 1b although they are not far behind. We also did not add any additional training data to Task 1b. So, we cannot rule out that deep learning systems will not perform better than traditional methods when trained on enough data.
On Task 2, the best performing system on human summaries, System 2, using neural representations trained on the 1000 plus summaries, does the best with a ROUGE-2 score of 0.278. This is higher than CL-SciSumm '18 top system using traditional methods. System 3, the second best Cl-SciSumm '19 system an endend deep learning model, with a score of 0.265 is also higher than CLSciSumm '18 top system. With a score of 0.514 System 3 also improves the state-of-the-art agasint abstracts by 0.2 on ROUGE-2 score. System 3 is also the top system on community summaries with a ROUGE-2 score of 0.204.
In summary, deep learning models do well across the board for summaries. Traditional methods do better on Task 1a on small but clean training data. Deep learning methods take over on large bu tnoisy data. 8
Nine systems participated in CL-SciSumm 2019 shared tasks. The systems were provided with larger but noisy corpus with automatic annotation. Nearly all the teams had neural methods and many employed transfer learning. Participants also experimented with the use of word embeddings trained on the shared task corpus, as well as on other domain corpora. We found that data augmentation for Task 1a may have helped deep learning models but not traditional machine learning methods. It also appears that deep learning methods perform better than traditional methods across the board when they have enough training data. We will explore methods to obtain cleaner training data for Task 1 without or with minimal human annotation e ort.
We recommend that future approaches should go beyond o -the-shelf deep learning methods, and also exploit the structural and semantic characteristics that are unique to scienti c documents; perhaps as an enrichment device for word embeddings. The committee also observes that CL-SciSumm series over the past 5 years has catalysed research in the area of scienti c document summarisation. We observe that a number of papers outside of the BIRNDL workshop published at prominent NLP and IR venues evaluate on the CL-SciSumm gold standard data. To create a reference corpus for the task was a key goal of the series. We have achieved this goal now. We will consider newer tasks to push the e ort towards automated literature reviews. We will also consider switching the format of the shared evaluation from a shared task to a leaderboard to which systems can submit evaluations asynchronously throughout the year.
Acknowledgement. We would like to thank SRI International for their generous funding of CL-SciSumm '19 and BIRNDL '19. We thank ChanZuckerberg Initiative for sponsoring the invited talk. We would also like to thank Vasudeva Varma and colleagues at IIIT-Hyderabad, India and University of Hyderabad for their e orts in convening and organizing our annotation workshops in 2016-17. We acknowledge the continued advice of Hoa Dang, Lucy Vanderwende and Anita de Waard from the pilot stage of this task. We would also like to thank Rahul Jha and Dragomir Radev for sharing their software to prepare the XML versions of papers. We are grateful to Kevin B. Cohen and colleagues for their support, and for sharing their annotation schema, export scripts and the Knowtator package implementation on the Protege software { all of which have been indispensable for this shared task. Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2017) co-located with the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017), Tokyo, Japan, August 11, 2017. pp. 1{6 (2017), http://ceur-ws.org/Vol-1888/ editorial.pdf [15] Mayr, P., Frommholz, I., Cabanac, G., Wolfram, D.: Editorial for the Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL) at JCDL 2016. In: Proc. of the Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL2016). pp. 1{5. Newark, NJ, USA (June 2016) [16] Nakov, P.I., Schwartz, A.S., Hearst, M.: Citances: Citation sentences for semantic analysis of bioscience text. In: Proceedings of the SIGIR'04 workshop on Search and Discovery in Bioinformatics. pp. 81{88 (2004) [17] Nomoto, T.: Resolving citation links with neural networks. Frontiers in Research Metrics and Analytics 3, 31 (2018) [18] Pitarch, Y., Pinel-Sauvagnat, K., Hubert, G., Cabanac, G., elie FraisierVannier, O.: IRIT-IRIS at CL-SciSumm 2019: Matching Citances with their Intended Reference Text Spans from the Scienti c Literature. In: BIRNDL2019 (2019) [19] Qazvinian, V., Radev, D.: Scienti c paper summarization using citation summary networks. In: Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1. pp. 689{696. ACL (2008) [20] Quatra, M.L., Cagliero, L., Baralis, E.: Poli2Sum@CL-SciSumm 2019: identify, classify, and summarize cited text spans by means of ensembles of supervised models . In: BIRNDL2019 (2019) [21] Syed, B., Indurthi, V., Srinivasan, B.V., Varma, V.: Transfer learning for e ective scienti c research comprehension. In: BIRNDL2019 (2019) [22] Wang, P., Li, S., Wang, T., Zhou, H., Tang, J.: Nudt@ clscisumm-18. In:
BIRNDL@ SIGIR. pp. 102{113 (2018) [23] Yasunaga, M., Kasai, J., Zhang, R., Fabbri, A., Li, I., Friedman, D., Radev, D.: ScisummNet: A large annotated corpus and content-impact models for scienti c paper summarization with citation networks. In: Proceedings of AAAI 2019 (2019) [24] Zerva, C., Nghiem, M.Q., Nguyen, N.T., Ananiadou, S.: UoM@CL-SciSumm 2019. In: BIRNDL2019 (2019)