In this work we propose to tackle the limitations posed by the lack of annotated data for argument mining in scientific texts by annotating argumentative units and relations in research abstracts in two scientific domains. We evaluate our annotations by computing inter-annotator agreements, which range from moderate to substantial according to the dificulty level of the tasks and domains. We use our newly annotated corpus to fine-tune BERT-based models for argument mining in single and multi-task settings, ifnally exploring the adaptation of models trained in one scientific discipline (computational linguistics) to predict the argumentative structure of abstracts in a diferent one (biomedicine).
The accelerated pace at which scientific knowledge is produced makes its discovery and assessment a challenging task. Natural language processing (NLP) technologies, in general, and text-mining tools, in particular, have become increasingly essential to identify and characterize the most relevant information produced in a given scientific discipline.
In order to assess a research article it is necessary to consider its logic, rhetoric and dialectic
quality dimensions [
In previous works [
The SciARG corpus and the code used in the experiments described in this work are made publicly available as a contribution to the research community.1
The rest of the paper is organized as follows: in Section 2 we describe previous work aimed at identifying arguments in scientific texts. In Section 3 we describe the data used to generate the corpus, our proposed annotation scheme and the annotation process. In Section 4 we describe the experiments conducted with the generated corpus and, in Section 5, we analyze the results obtained. In Section 6, we present our conclusions and suggest potential follow-ups.
The inherent complexity and ambiguity of the scientific language makes the identification
of arguments in scientific texts a particularly challenging task [
papers and to establish connections with previous works. CoreSC, in turn, provides a readable
representation of the research process described by the paper. Diferences and similarities
between the two schemes are studied in [
In this section we describe the source data used as a basis of the SciARG corpus as well as the
annotation schema that we propose. We describe the annotation process and assess the quality
of the produced annotations by considering inter-annotator agreement measures.
3.1. Data
The SciARG corpus covers two knowledge areas: computational linguistics and biomedicine.
We refer to these sub-corpora as CL and BIO, respectively.
• CL corpus. Includes 225 computational linguistics abstracts from the ACL Anthology [
3In particular, from the Proceedings of the 2014 Conference on Empirical Methods in NLP (EMNLP).
Discourse Dependency TreeBank for Scientific Abstracts (SciDTB) [
In this work we focus on the analysis of the way in which authors logically structure information
in abstracts to persuade potential readers about the relevance and validity of their proposals. Our
annotation scheme is aimed at capturing the underlying argumentative structure departing from
its linguistic realization. It is therefore relevant to consider previous works that characterize
the diferent constituent elements of scientific abstracts. Several works have been dedicated to
the study, from a genre analysis perspective, of the rhetorical structure of scientific articles and
its parts [
Description high level description of the proposed approach/solution processes/tools/methods that are part of the proposal data obtained from experiments direct interpretation of observed data results and the means by which they were obtained high-level interpretation/generalization of results secondary methods/processes not part of the proposal known problem/limitation addressed by the proposal new ideas/paths for known problems/limitations known information to support the proposed approach additional information (definitions/examples) Coarse proposal proposal outcomes outcomes outcomes outcomes methods motivation motivation motivation other
An annotated abstract can be seen as a directed graph with the sentences as its nodes and
the relations between them as the edges. In order to gain in uniformity of the annotations,
4This allows us to continue exploring the interaction between argumentative, rhetoric and discourse annotation
levels in scientific abstracts, as originally proposed by Peldszus and Stede [
Description of the child node function
provides new supporting information/evidence for the parent
provides additional information relevant to assess/contextualize the parent
describe methods through which supporting evidence is obtained
provides information essential to understand/contextualize the parent
describes a step that comes after the step described by the parent in a process
provides non-essential information
reduce the level of ambiguity and simplify the annotation process, we only consider trees as
valid annotations graphs. In addition to types and relations, annotators were asked to identify
the unit that describe the most significant contribution of the work as the main unit. Fig. 1
shows an example of the tree resulting from annotating the abstract from [
In previous work we considered argumentative units at sub-sentence level and explored the
relation between discourse and argumentative levels [
Most sentences in computational linguistics abstracts contain one type of argumentative unit. A relatively frequent exception to this are sentences in which mentions to methods are included in the results. For this specific case we introduce in our schema the unit type results-means.8 For other cases in which more than one type of unit can be identified, our scheme allows annotators to register this information by assigning a second type to the sentence. In the annotation process annotators were asked to weight the relevance of the diferent types of information contained in the sentence to make a decision with respect to the main and secondary types.
It is frequent to find, in abstracts, that authors build up supporting evidence or justifications for implicit or explicit claims in more than one sentence. Consider the example in Fig. 1. Nodes (4) (motivation-problem) and (7) (motivation-background) provide partial information that, when considered together, contribute to justify the proposed work described in node (2). From a discourse analysis perspective, this would be represented by a multi-nuclear relation which could be annotated by introducing a diferent type of node in the argumentative tree. This, on one hand, introduces some practical dificulties in the automatic processing of the annotations, as will become evident when we describe the experiments in Section 4 and, on the other hand, does not allow to capture the hierarchical relation between nodes (4) and (7). We opt, instead, to introduce the relation info-required to account for these cases. In this example, we indicate that there is an info-required relation that goes from node (7) to node (4) and a support relation that goes from node (4) to node (2). When looking for supporting evidence for the sentence in node (2), therefore, we would consider not only their direct children but
also the chains of sentences below them linked by info-required relations. Depending on the
specific dimensions of the argumentation quality to analyze, therefore, diferent subsets of
units and relations can be considered. Most argument mining works focus on logic aspects of
argumentation and, in particular, in the arguments’ cogency [
The annotation guidelines used in this work are made available online.9 The first step of the
annotation process is to have the texts of the abstracts splitted into sentences. In the case of the
CL corpus, the source files used are already segmented into sentences and elementary discourse
units in the SciDTB corpus. For the biomedical abstracts the sentence segmentation is done by
means of the syntok tool.10 The annotation was done by means of a modified version of GraPAT
(Graph-based Potsdam Annotation Tool) [
We first developed and adjusted the annotation scheme with the CL corpus and then used it to annotate the BIO corpus. One of the goals of this work is to assess the applicability of the proposed scheme to diferent domains. In Section 3.5 we analyze the main diferences between both corpora and the resulting annotations. The annotation of the CL corpus was done by 9https://github.com/LaSTUS-TALN-UPF/SciARG/blob/main/Annotation_Guidelines_Arguments_SciDTB.pdf 10github.com/fnl/syntok three expert annotators, 1, 2, 3 (two NLP researchers and one computational linguist) in three rounds. The first two rounds were aimed at training the annotators, clarifying doubts and making the necessary adjustments to the annotation scheme and tool. As a result of the whole process 225 CL abstracts were annotated, having 30 abstracts annotated by the three annotators to compute inter-annotator agreement. For the BIO corpus only two of the CL annotators (1, 2) could participate in the annotation process. In this case no training phase was needed and there were no substantial modifications to the annotation tool or scheme. As a result of this process 285 abstracts were annotated, of which 50 were annotated by both annotators. For our experiments we split both sub-corpora into training and test sets, as described in Section 5.1.
In this section we assess the reliability of our annotations by considering inter-annotators’
agreement. Table 3 shows the agreements obtained for CL and BIO sub-corpora. In the case of
CL, where we have three annotators, we report the average of the pairwise agreements with
their corresponding standard deviations. In order to compute the agreements we consider exact
matches between pairs of labels assigned by two annotators. For the parent attachment task
the label is the absolute position of the parent sentence in the document. In addition to each
sub-task-specific agreement, we report the agreement observed when considering simultaneous
exact matches for all the tasks. It is relevant to note that annotations within one document
cannot be consider as completely independent from each other, which presents a limitation
when interpreting the significance of Cohen’s coeficient. 11 In Table 3, therefore, in addition
to Cohen’s s, we directly report the accuracy obtained for the diferent tasks without any
presupposition with respect to the independence of the annotations. When considering a pair
of annotated documents as labeled trees, the accuracy indicates the number of changes (with
respect to the number of nodes) that would be necessary to do in one tree to obtain the other one.
It can, therefore, be interpreted as an edit-distance measure that allows to estimate the degree of
agreement between two annotators with respect to the argumentative roles of the nodes when
considering the document as a whole. In all cases annotation agreements fall between moderate
and substantial levels: substantial agreements are obtained in general in the CL corpus (and
almost perfect agreement when coarse-grained types are considered), while agreements in the
identification of unit types and relations are lower in the BIO corpus. In addition to the fact
that the annotation scheme was designed and adjusted specifically for the CL domain, lower
agreements are expected in BIO as abstracts have a higher level of complexity than CL ones
in terms of their structure, the number of units that they contain and their lengths, as shown
in Section 3.5. It is also relevant to note that annotators have a high level of familiarity with
CL texts while they are not experts in the BIO domain. When analyzing discrepancies in the
annotation of the BIO corpus we observe that units of types observation and result give origin
to systematic disagreements between annotators 1 and 2. In fact, annotator 2 annotated
as observation 64% of the units that annotator 1 annotated as result, which makes us believe
that a clear distinction between these two types is dificult to establish without specific domain
11As a decision made at one node of the argumentative structure afects decisions made in other nodes. This
problem has already been observed by Marcu et al. [
Substantial diferences can be observed between the CL and BIO sub-corpora. Abstracts in BIO are, in general, longer and argumentatively more complex than those in CL (Table 4). It is frequent in BIO to find abstracts that describe a series of experiments, each one with their results. In some cases, results from one experiment are used to motivate and/or justify new ones. This level of detail is not present in CL abstracts. In general, the description of research outcomes and their interpretation is much more complex in BIO abstracts, which leads to a significant diference in the number of units of type observation, result and conclusion when compared to CL abstracts.12
The distinction between the plain report of observed data, the interpretation of results and the extraction of conclusions from them is more ambiguous in BIO than in CL and, therefore, diferentiating these types of units is more dificult. The distances between units and their parents are greater in BIO. In fact, nearly 19% of the times a unit is 5 or more units away from its parent. In CL this occurs only in 2% of the cases. In 69% of the cases CL units are only one or two units away from its parent when considering the CL corpus. In BIO this occurs only 58% of 12While in CL 3% of the units are of type observation, 19% of type result or result-means and 4% of type conclusion, in BIO there are, respectively, 18%, 26% and 11% units of these types. More details are provided in the supplemental material. the times. In both domains backward relations are more frequent than forward relations: the parent occurs before the child 68% and 66% of the times in CL and BIO, respectively.
4.1. Tasks In this section we describe the experiments carried out in order to, given a scientific abstract, predict the nodes and relations needed to represent its argumentative structure. • Unit type: Given the text of a sentence, predict its type. The class to predict in this case is one of the 11 fine-grained types described in Table 1. • Relation direction: Given two sentences, predict whether a forward or backward relation exists between them (e.g: whether the first unit is a child of the second one in the argumentative tree or vice versa). We model this task as a three-class classification task where, given two sentences, the possible classes to predict are forw, back or none, indicating, respectively, that there is a directed relation from the first to the second sentence, from the second to the first sentence, or that the two sentences are not related. • Relation type: Given a sentence, predict the label of the relation with its parent (its argumentative and/or discourse function) or none for the root node. The class to predict in this case is one of the 6 relations described in Table 2. • Main unit: Given the text of a sentence, predict whether it is the main unit.13
There are clear links between the four tasks. For instance, the main unit is, in most cases, the root of the argumentative tree. Associations can also be established between a unit’s type and its function: in the most frequent case, units of type result are used to support units of type proposal or conclusion. It is therefore natural to explore the possibility of training the tasks jointly, in a multi-task setting, which we compare to the results obtained when training the results independently, in single-task settings.
Transformer-based encoders [
When fine-tuning our models in each domain, we consider the median number of tokens in
the input sequences and set the maximum sequence length to its double. We consider
crossentropy as the loss function to optimize. We use the Adam optimizer with a learning rate of
2e-5 and a warm-up period of 10% of the learning steps. We set a dropout probability of 0.1
for multi-task settings and 0.2 for single-task ones. The batch size used is of 16 instances with
gradient accumulation of 2 batches. These hyperparameters were set based on five-fold
crossvalidation evaluations in the training set. While the general recommendation is to fine-tune
BERT for 2 to 4 epochs [
As mentioned in Sections 1.1 and 3, our annotation scheme was specifically developed to account for argumentative types and relations in computational linguistic abstracts. One of the goals of this work is to explore i) the applicability of this scheme to other scientific disciplines and ii) whether models trained in the CL domain can be easily adapted to predict the argumentative structure of abstracts in other scientific areas. In particular, in the BIO domain. We use the newly annotated set of biomedical abstracts in order to respond to both research questions.
We are also interested in exploring to what extent models trained with annotations in CL contain task-specific information that can be exploited to predict argumentative types and relations in scientific abstracts with a more complex structure and in another discipline. We therefore analyze the results obtained by keeping the weights of a model fine-tuned with the CL abstracts fixed and only training a linear classifier on top of it with the BIO abstracts.
14I.e.: if sentence 2 is a child of sentence 1 in the argumentative tree, we include the instances (2, 1, forw) and (1, 2, back) in the training set.
15While there are 3 classes and 13,874 training instances for the BIO/relation type task, we only have 1,049 training instances and 11 classes for CL/unit type.
16I.e.: "[CLS] [AFTER] [DISTANCE-1] [POS-1] This paper presents ... [SEP] We observe ..."
We present in this section the performance of the models trained in each domain (BIO, CL), as well as the adaptation of CL models to BIO by training a small number of additional parameters.
In the CL domain 30 abstracts were annotated in common by three annotators (1, 2, 3). This set is used for evaluation, while the rest of the 195 abstracts is used to train the models. We generate a set of consensus annotations by assigning, to each instance, the majority label considering the annotations by 1, 2 and 3. In the few cases in which there is total discrepancy among the three annotators we keep the label assigned by the annotator with the highest average of pairwise agreement with the other two annotators. For BIO it is not possible to do this since we only have two annotators (1, 2) that annotated 50 abstracts, while the rest of the 235 abstracts were annotated only by one annotator (1). Therefore, in these experiments we only use the annotations produced by 1. As test set, in this case, we consider the subset of 35 abstracts17 annotated by 1 with the highest levels of agreement with the annotations made by 2, keeping the remaining 250 abstracts annotated by 1 as training set.
In Table 5 we report the results obtained by the set of experiments described in Section 4 when evaluating against the consensus annotations. For the types of units and relations we use weighted-averaged F1-scores, as we want to consider the contribution of each label to the results in proportion to their frequency. For the evaluation of the direction of the relations, instead, we use macro-averaged scores, which are more sensitive to the minority classes. If we were to use micro-averaged or weighted-averaged scores in these cases we would obtain misleading high numbers for F1, given the large proportion of none labels which are correctly classified. This is also the case for the prediction of the main unit. As expected, considering the greater argumentative complexity of the BIO abstracts, which is also reflected in the lower levels of inter-annotator agreements, the performance of the models trained and evaluated with the BIO annotations is lower than the one obtained with the CL annotations (Table 5). In the BIO domain the models trained jointly in a multi-task settings tend to perform better than those in which these tasks are trained independently. In the case of CL the diference between both settings is less evident: while there is a clear advantage of the multi-task setting in the prediction of the types of units, better results are obtained for the prediction of the parent relations in a single-task setting. We also observe that the BERT models fine-tuned with the CL annotations (CL-BERT) without in-domain fine-tuning perform competitively when compared to the models in which BERT is fine-tuned with the BIO annotations.
It is relevant to note that the CL-BERT model with frozen weights performs significantly better in the prediction of the BIO annotations than the frozen SciBERT encoder that we consider as baseline. This confirms that the model fine-tuned with CL annotations is able to capture information about the argumentative structure of scientific abstracts independent of the specific discipline in which it was trained.
17The number of 35 is chosen in order to keep the training-test sets percentages similar in both domains.
In this work we propose a new sentence-level annotation scheme for the identification of argumentative units and relations in scientific abstracts, which we apply to the annotation of 510 documents in two highly specialized domains: computational linguistics and biomedicine. The resulting corpus, as well as the code used to train and evaluate models trained with it, is made publicly available. The results obtained in our experiments encourage us to think that, in spite of the fact that the annotation scheme was originally developed and refined for the CL domain, it can be successfully applied to other scientific disciplines. This work also opens up new research paths, including further exploration of domain adaptation techniques for argument mining models in challenging domains as is the case of scientific articles.
This work was (partly) supported by the Spanish Government under the María de Maeztu Units of Excellence Programme (MDM-2015-0502) and by the Research and Innovation Agency of Uruguay (ANII). We also acknowledge support from the project Context-aware Multilingual Text Simplification (ConMuTeS) PID2019-109066GB-I00/AEI/10.13039/501100011033 awarded by Ministerio de Ciencia, Innovación y Universidades (MCIU) and by Agencia Estatal de Investigación (AEI) of Spain.
proposal We present a novel approach to improve word alignment for statistical machine translation ( SMT ) . proposal-implementation We observe , identify , and detect naturally occurring signals of interestingness in click transitions on the Web between source and target documents , which we collect from commercial Web browser logs . observation Our method produces a gain of +1.68 BLEU on NIST OpenMT04 for the phrase-based system , and a gain of +1.28 BLEU on NIST OpenMT06 for the hierarchical phrase-based system . result Experimental results show statistically significant improvements of BLEU score in both cases over the baseline systems . means We conducted experiments on two standard benchmarks : Chinese PropBank and English PropBank . result-means Results on the Switchboard disfluency tagged corpus show utterance-final accuracy on a par with stateof-the-art incremental repair detection methods , but with better incremental accuracy , faster time-todetection and less computational overhead . conclusion This transfer learning approach brings a clear performance gain over features based on the traditional bag-of-visual-word approach . motivation-problem However , fundamental problems on efectively incorporating the word embedding features within the framework of linear models remain . motivation-hypothesis Combining the two tasks can potentially improve the eficiency of the overall pipeline system and reduce error propagation . motivation-background Recent work has shown success in using continuous word embeddings learned from unlabeled data as features to improve supervised NLP systems , which is regarded as a simple semi-supervised learning mechanism . information-additional The structure of argumentation consists of several components ( i.e. claims and premises ) that are connected with argumentative relations .
39 157 54 27 69 103 157 20 24