In this paper, we analyze and discuss approaches to argumentation mining from the discourse structure perspective. We chose persuasive essays and scientific articles as our example domains. By analyzing several example arguments and providing an overview of previous work on argumentation mining, we derive important tasks that are currently not addressed by existing argumentation mining systems, most importantly, the identification of argumentation structures. We discuss the relation of this task to automated discourse analysis and describe preliminary results of two annotation studies focusing on the annotation of argumentation structure. Based on our findings, we derive three challenges for encouraging future research on argumentation mining.
Argumentation mining is a recent research area which promises novel opportunities not only for information retrieval, educational applications or automated assessment tools but also aims at improving current legal information systems or policy modeling platforms. It focuses on automatically identifying and evaluating arguments in text documents and includes a variety of subtasks like identifying argument components, finding accepted arguments and discovering argumentation structures. Researchers have already investigated argumentation mining in several domains. For instance, Teufel (1999) aims at identifying rhetorical roles of sentences in scientific articles and Mochales-Palau and Moens (2011) identify arguments in legal documents. Also, Feng and Hirst (2011) investigated argumentation schemes in newspapers and court cases and Florou et al. (2013) applied argumentation mining in policy modeling.
However, current approaches mainly focus on
the identification of arguments and their
components and largely neglect the identification of
argumentation structures although an argument
consists not only of a set of propositions but also
exhibits a certain structure constituted by
argumentative relations
In this paper, we analyze several examples of argumentative discourse from the discourse structure perspective.1 We outline existing approaches on argumentation mining and discourse analysis and provide an overview of our current work on argumentation structure annotation in scientific articles and persuasive essays. We conclude this paper with a list of challenges for encouraging future 1The examples are taken from persuasive essays which are either collected from the writing feedback section of http://www.essayforum.com or from the corpus compiled by Stab and Gurevych (2014) research on argumentation mining. 2
Philosophy and Logic proposed a vast amount of argumentation theories (e.g. Toulmin (1958), Walton et al. (2008), Freeman (2011)).2 The majority of these theories generally agree that an argument consists of several argument components which can either be a premise or a claim. The simplest form of an argument includes one claim that is supported by at least one premise (figure 1).
Claim supports
Premise The claim3 is the central component of an argument that can either be true or false. Thus, the claim is a statement that should not be accepted by the reader without additional reasons. The second component of an argument, the premise4, underpins the plausibility of the claim. It is usually provided by the proponent (writer) for convincing the reader of the claim. Examples (1) and (2) illustrate two simple arguments, each containing a claim (in bold face) and a single premise (underlined): (1) “It is more convenient to learn about historical or art items online. With Internet, people do not need to travel long distances to have a real look at a painting or a sculpture, which probably takes a lot of time and travel fees.” (2) “Locker checks should be made mandatory and done frequently because they assure security in schools, make students healthy, and will make students obey school policies.”
These examples illustrate that there exist argument components both on the sentence level and on the clause level.
Argumentative relations are usually directed relations between two argument components and represent the argumentation structure. There exist different types like support or attack (Peldszus 2A review of argumentation theory is beyond the scope of this paper. A survey can be found in Bentahar et al. (2010) 3also called conclusion (Mochales-Palau and Moens, 2009)
4sometimes called support
Figure 2 shows the structure of the argument in (3). In this example, premiseb supports the claima whereas premised attacks the claima.
a supports b supports attacks c d
This example illustrates three important properties of argumentation structures: 1. Argumentative relations can hold between non-adjacent sentence/clauses, e.g. the argumentative attack relation from premised to the claima. 2. Some argumentative relations are signaled by indicators, whereas others are not. For instance, the argumentative attack relation from premised to the claima is indicated by the discourse marker ‘but’, whereas the argumentative support relation from premiseb to claima is not indicated by a discourse marker. 3. Argumentative discourse might exhibit reasoning chains, e.g. the chain constituted between argument components a, b, and c. 3
Previous approaches on argumentation mining
cover several subtasks including the separation of
argumentative from non-argumentative text units
The first step of an argumentation mining pipeline
typically focuses on the identification of
argumentative text units before analyzing the
components or the structure of arguments. This task
is usually considered as a binary classification
task that labels a given text unit as
argumentative or non-argumentative. One of the first
approaches was proposed by
Although the separation of argumentative from non-argumentative text units is an important step in argumentation mining, it merely enables the detection of text units relevant for argumentation and does not reveal the argumentative role of argument components. 3.2
The classification of argument components aims at identifying the argumentative role (e.g. claims and premises) of argument components.
One of the first approaches to identify argument
components is Argumentative Zoning proposed by
Currently, there are only few approaches aiming
at the identification of argumentation structures.
For instance, the approach proposed by
MochalesPalau and Moens (2011) relies on a manually
created context-free grammar (CFG) and on the
presence of discourse markers for identifying a
tree-like structure between argument components.
However, the approach relies on the presence of
discourse markers and exploits manually created
rules. Therefore, it does not accommodate
illformatted arguments
Another approach was presented by Cabrio and Villata (2012). They identify relations between arguments of an online debate platform for identifying accepted arguments and to support the interactions in online debates. In contrast to the work of Mochales-Palau and Moens (2011), this approach aims at identifying relations between arguments (macro-level) and not between argument components (micro-level).
Discourse analysis aims at identifying discourse
relations that hold between adjacent text units with
text units being sentences, clauses or
nominalizations
Most previous work in automated discourse
analysis is based on corpora annotated with general
discourse relations, most notably the Penn
Discourse Treebank (PDTB)
Take as an example the argumentation structure discussed in section 2.
“Everybody should study abroada. It’s an irreplaceable experience if you learn standing on your own feetb since you learn living without depending on anyone elsec. But one who is living overseas will of course struggle with loneliness, living away from family and friendsd:” Whereas the argument components b and c, as well as c and d are related through the discourse marker ‘since’ (signalling an explicit CAUSE relation) and ‘but’ (signalling an explicit CONTRAST relation), the discourse relation JUSTIFY between a and b is an implicit relation.
Existing approaches of discourse analysis
proposed different sets of discourse relations, and
there is currently no consensus in the literature
about the ‘right’ set of discourse relations. For
instance, the RST
It is still an open question how the proposed
discourse relations relate to argumentative relations.
Although, there are preliminary findings that
indicate that there are certain similarities
The difference of the relations is best illustrated
by the work of Biran and Rambow (2011), which
is to the best of our knowledge the only approach
that focuses on the identification of distinct
argumentative relations. The authors argue that
existing definitions of discourse relations are only
usable as a building block for argumentation mining
and that there are no distinct argumentative
relations included in existing approaches. Therefore,
they combine 12 relations from the RST Discourse
Treebank
There is a large body of previous research in
linguistics on the role of discourse markers,
signalling discourse relations (e.g.‘because’,
‘therefore’, ‘since’, etc.) in discourse analysis. Most
previous investigations of discourse markers are
based on the PDTB
However, a critically discussed question in this context is the definition of discourse markers. Are discourse markers in the sense of indicators marking discourse relations just words like ‘because’, ‘therefore’, ‘since’? Taboada (2006) investigates the role of discourse markers in corpora annotated with discourse relations according to the RST. In her discussion of related work on discourse markers in linguistics, she concludes that there are many lexical and linguistic devices signalling discourse relations beyond discourse markers, such as the mood (e.g. indicative or conjunctive) or the modality (e.g. possibility, necessity) of a sentence.
In particular, for argumentative discourse, the role of indicators, such as discourse markers, is not well-understood yet, which is due to the lack of corpora annotated with argumentation structures. Recently, Tseronis (2011) summarized intermediate results of a corpus-based analysis of argumentative moves, aiming at the identification of linguistic surface cues that act as argumentative markers. According to Tseronis (2011), any single or complex lexical expression can act as an argumentative marker, and it can either mark an argumentative relation (i.e., connecting two arguments or argument components) or signal a certain argumentative role, such as a claim or a premise. Moreover, he observed that also sequential patterns of argumentative markers indicate particular argumentative moves, for instance, first stating the common ground (e.g., using the marker it is understandable ...) and then presenting an attack to this common ground (e.g., using a marker such as nevertheless). 5
Our research in argumentation mining is motivated by the (1) information access and (2) computer-assisted writing perspective. Currently, we are conducting two annotation studies, focused on analyzing argumentation structures in scientific articles and persuasive essays. In the following subsections we provide an overview of the (preliminary) results. 5.1
One of the main goals of any scientific publication is to present new research results to an expert audience. In order to emphasize the novelty and importance of the research findings, scientists usually build up an argumentation structure that provides numerous arguments in favor of their results. The goal of this annotation study is to automatically identify those argumentation structures on a fine-grained level in scientific publications in the educational domain and thereby to improve information access. A potential use case could be an automated summarization system creating a summary of important arguments presented in a scientific article.
Up to now only coarse-grained approaches like
Argumentative Zoning
Due to the lack of evaluation datasets, we are performing an annotation study with four annotators, two domain experts and two annotators who developed the annotation guidelines. Our dataset consists of about 20 scientific full-texts from the educational domain. For the annotation study, we developed our own Web-based annotation tool (see figure 3 for a screenshot). The annotation tool allows to label argument components directly in the text with different colors and to add different relations (like support or attack) between argument components. The resulting argumentation structure is visualized as a graph (see figure 3).
Next, we plan to develop weakly supervised machine learning methods to automatically annotate scientific publications with argument components and the relations between them. The first step will be to distinguish non-argumentative parts (for example descriptions of the document structure) from argumentative parts (see section 3.1). The second step will be to identify support and attack relations between the argument components. In particular, we will explore lexical features, such as discourse markers (for example ‘hence’, ‘so’, ‘for that reason’, ‘but’, ‘however’, see section 4), and semantic features, such as text similarity or textual entailment.
The goal of computer-assisted writing is to
provide feedback about written language in order
to improve text quality and writing skills of
authors respectively. Common approaches are for
instance focused on providing feedback about
spelling and grammar, whereas more
sophisticated approaches also provide feedback about
discourse structures
Argumentative Writing Support is a
particular type of computer-assisted writing that aims at
providing feedback about argumentation and thus
postulates methods for reliably identifying
arguments. Besides the recognition of argument
components, the identification of the argumentation
structure is crucial for argumentative writing
support, since it would open novel possibilities for
providing formative feedback about
argumentation. On the one hand, an analysis of the
argumentation structure would enable the
recommendation of more meaningful arrangements of
argument components and a reasonable usage of
discourse markers. Both have been shown to increase
argument comprehension and recall, and thus the
quality of the text
Following this vision, we conducted an
annotation study with three annotators to model
argument components and the argumentation
structure in persuasive essays at the clause-level. The
corpus includes 90 persuasive essays which we
selected from essayforum.com. Our annotation
scheme includes three argument components
(major claim, claim and premise) and two
argumentative relations (support and attack). For defining
the annotation guidelines and the annotation
process we conducted a preliminary study on a
corpus of 14 short text snippets with five non-trained
annotators and found that information about the
topic and the author’s stance is crucial for
annotating arguments. According to these findings,
we defined a top-down annotation process
starting with the major claim and drilling-down to the
claims and the premises so that the annotators are
aware of the author’s stance and the topic before
annotating other components. Using this strategy,
we achieved an inter-rater agreement of U =
0:725 for argument components and = 0:81
for argumentative relations indicating that the
proposed scheme and annotation process successfully
guides annotators to substantial agreement. For
more details about this annotation study, we
refer the interested reader to
5We used Krippendorff’s U
6http://www.ukp.tu-darmstadt.de/data/ argumentation-mining
Existing approaches of argumentation mining mainly focus on the identification of argument components (section 3). Based on the examples analyzed in section 2 and on the experience gained in our annotation studies (section 5), we identified the following challenges for future research in argumentation mining that have not been addressed adequately by previous work.
Segmentation: Most of the existing approaches
are based on the sentence-level. However, for
analyzing arguments, a more fine-grained
segmentation is needed
(4) “Eating apples is healthy which has to do with substrates which prevent cancer and other diseases.” (5) “There are different opinions about coffee. Some people say they need it to stay awake. Other people think it’s unhealthy.”
It is an open question if existing segmentation approaches can be used for reliably identifying the boundaries of argument components. In example (4) we find two times the word “which”. This makes it hard for a segmenter to split the sentence correctly in only two parts. On the other hand, the combination of sentences (example (5)) also requires more elaborated techniques that are able to identify sentences that are related and only form in combination the support of a particular claim.
Context Dependence: The context is crucial for identifying arguments, their components and argumentation structures. As illustrated by Stab and Gurevych (2014), it is even a hard task for human annotators to distinguish claims and premises without being aware of the context. For instance, the following three argument components constitute a reasoning chain in which c is a premise for b and b a premise for a: (6) “Random locker checks should be made obligatory.a Locker checks help students stay both physically and mentally healthy.b It discourages students from bringing firearms and especially drugs.c”
In this argumentation structure, a can be classified as a claim. However, without being aware of the argument component a, b becomes a claim which is supported by premise c. The same situation can be found in example (3) in section 2. If we look at the argument components b and c in isolation, we can classify b as claim. However, looking at the whole example, the argument component a is the claim, supported by the premise b. The same holds for the argument components c and a which would be connected by a support relation if they are considered in isolation. Both examples illustrate that the context is crucial for classifying argument components as claims or premises and for identifying the argumentation structure. Although, Stab and Gurevych (2014) proposed an annotation process that facilitates these decisions in manual annotation studies of persuasive essays, it is still an open issue how to model the context in order to improve the performance of automatic argumentation mining methods.
The most important challenge for identifying argumentation structures is ambiguity, since there are often several possible interpretations of argumentation structures which makes it hard or even impossible to identify one correct interpretation. In previous examples, we have already seen that the classification of argument components depends on the context and the considered argument components respectively. However, even if we consider all components of an argument, there might be several reasonable interpretations of its structure. For instance, the structure of example (6) can be interpreted in three different ways (figure 4). In the first interpretation, the argument component c supports argument component b and argument component b supports argument component a, whereas in the second interpretation argument components b and c both support argument component a. The third interpretation contains all possible argumentative relations from the first and second interpretation combined, and thus represents a graph structure (in contrast to a tree structure).
The ambiguity of argumentation structures repb c support support
a support b support
support
c
b
a
resents a major challenge for argument
annotation studies and consequently the creation of
reliable gold standards for argumentation
mining. In all annotation studies we know, exactly
one annotation is considered to be correct which
means that other possibly correct interpretations
are considered as incorrect and therefore
downgrade the results for the inter annotator
agreement and the performance of automatic
classifiers. Consequently, it might be interesting to
explore different evaluation methods. For
instance, evaluation schemes used in automatic text
summarization could be considered as an
alternative. In text summarization, inter annotator
agreement for human-generated summaries is
particularly low, and hence, each human-generated
summary is considered valid for evaluating an
automatic summarization system
In this paper, we showed that existing approaches to argumentation mining mainly focus on the identification of argument components and largely neglect the identification of argumentation structures, although this task is crucial for many promising applications, e.g., for building novel argument related knowledge bases. By examining several examples, we derived characteristic properties of argumentation structures. We discussed the relation of discourse analysis and argumentation structure and showed that previous works in discourse analysis are not capable of identifying argumentation structures, because discourse relations do not cover all argumentative relations and are limited to relations between adjacent text units. Based on our observations, we derived three challenges for encouraging future research, i.e., (i) identifying the boundaries of argument components, (ii) modeling the context of argument components and argumentative relations, and (iii) addressing the problem of ambiguous argumentation structures. In particular, the ambiguity of argumentation structure poses an important issue for future work.