In this paper, we argue that an annotation scheme for argumentation mining is a function of the task requirements and the corpus properties. There is no one-sizefits-all argumentation theory to be applied to realistic data on the Web. In two annotation studies, we experiment with 80 German newspaper editorials from the Web and about one thousand English documents from forums, comments, and blogs. Our example topics are taken from the educational domain. To formalize the problem of annotating arguments, in the first case, we apply a Claim-Premise scheme, and in the second case, we modify Toulmin's scheme. We find that the choice of the argument components to be annotated strongly depends on the register, the length of the document, and inherently on the literary devices and structures used for expressing argumentation. We hope that these findings will facilitate the creation of reliably annotated argumentation corpora for a wide range of tasks and corpus types and will help to bridge the gap between argumentation theories and actual application needs.
Argumentation mining apparently represents an
emerging field in Natural Language Processing
(NLP) with publications appearing at mainstream
conferences, such as ACL
Our research into argumentation mining is
motivated by the information seeking perspective.
The key sources are discussions (debates) about
controversies (contentions) targeted at a particular
topic which is of the user’s interest. The scope is
not limited to a particular media type as the source
types can range from the on-line newspapers’
editorials to user-generated discourse in social
media, such as blogs and forum posts, covering
different aspects of the issues. Understanding
positions and argumentation in on-line debates helps
users to form their opinions on controversial issues
and also fosters personal and group decision
making
Although argumentation mining on the Web
has already been partly outlined in the literature
Blogs
Argument
Morover, not all of the related research works are
tightly connected to argumentation theories
In this article, we will particularly focus on bridging the gap between argumentation theories and actual application needs that has not been targeted in the relevant literature. We will support our findings by comprehensively surveying existing works and presenting results from two extensive annotation studies.
Our main findings and suggestions can be
summarized as follows: First, the use-case of any
research in argumentation mining must be clearly
stated (i.e., in terms of expected outcomes).
Second, properties of the data under investigation
must be taken into account, given the variety of
genres and registers
Research on argumentation is widely
interdisciplinary, as it spreads across philosophy and
rhetoric (Aristotle and Kennedy (translator),
1991; Perelman and Olbrechts-Tyteca, 1991;
Walton et al., 2008), informal and formal logic
For NLP research, this overwhelming amount
of related works brings many theoretical and
practical issues. In particular, there is no
one-sizefits-all argumentation theory. Even argumentation
researchers disagree on any widely-accepted
ultimate concept. For example, Luque (2011)
criticizes the major existing approaches in order to
establish a new theory which is later again severely
criticized by other in-field researches
Given the breath of argumentation mining just outlined, we would also like to discuss aspects that do not fit into our approach to argumentation mining, namely macro argumentation and evaluation using formal frameworks.
First, we treat argumentation as a product (micro argumentation or monological models), not as a process (macro argumentation or dialogical models). While dialogical models highlight the process of argumentation in a dialogue structure, monological models emphasize the structure of the argument itself (Bentahar et al., 2010, p. 215). Therefore, we examine the relationships between the different components of a given argument, not a relationship that can exist between arguments.1 Exploring how argumentation evolves between parties in time remains out of our scope.
Second, we do not tackle any logical
reasoning, defeasibility of reasoning, or evaluating
argumentation with formal frameworks in general.
Although this is an established field in informal logic
Let us set up a common terminology. Claim is
“the conclusion we seek to establish by our
arguments”
In existing works on argumentation mining of the
Web data, the connection is often made to
opinion mining
Villalba and Saint-Dizier (2012) approach
aspect-based sentiment of product reviews by
classifying discourse relations conveying arguments
(such as justification, reformulation, illustration,
and others). They build upon Rhetorical Structure
Theory (RST)
For modeling argumentation in social media,
Schneider et al. (2012) suggest using Dung’s
framework
Schneider and Wyner (2012) focus on the
product reviews domain and develops a number of
argumentation schemes (inspired by
Table 1 summarizes the previous research on
annotating argumentation. Not only it covers
related work from the NLP community but also
studies from general discourse analysis
Most studies dealing with Web data use
some kind of proprietary model without
relation to any argumentation theory
Inter-annotation agreement (IAA) that
reflects reliability of the annotated data is either
not reported
This motivates our research into annotating Web data relying on a model based on a theoretical Claim restatement = {true, false} Pre-Support Post-Support premises
Pre-Attack Post-Attack background in argumentation and reporting IAA that would confirm suitability of the model and reliability of the annotated data. 4
Up until now, we have used the terms argumentation and argument in their common meaning without any particular formal definition. We will now elaborate on annotation schemes and discuss their suitability and reliability for the Web data. 4.1
Because of the lack of a single general-purpose argumentation model (cf. discussion in x1), we present here two different schemes.2 Both are built upon foundations in argumentation theories, but they differ in their granularity, expression power, and other properties.
The Claim-Premises scheme is widely used in
previous work on argumentation mining, e.g.,
Premises either support or attack a claim, i.e., there is a support or attack relation between each 2An exhaustive overview of various argumentation models, their taxonomy, and properties can be found in (Bentahar et al., 2010). premise and a claim. The simplest way to represent the support and attack relations is to attach labels to adjacent argument components, which indicate their argumentative role. The span of argument components is left unspecified, allowing for argument components spanning a clause or one to several sentences. Using the six labels claim, restatement, pre-claim support, post-claim support, pre-claim attack and post-claim attack, a linear sequence of non-nested arguments can be represented.
While graph structures where nodes stand for
argument components, and edges for support or
attack relations are a more general way to
represent arguments (equivalent to, i.e.,
of the Claim-Premises scheme is its simplicity. Therefore, it is particularly suited for annotating arguments in long Web documents, such as news articles, editorials or blog posts. Kluge (2014) found that most documents of these text types consist of three major parts: an introductory part, summarizing the document content in one or two paragraphs, the main part, presenting a linear sequence of arguments, and an optional concluding part summarizing the main arguments.
The Claim-Premise scheme can be used to provide an overview of the claims and their supporting or attacking premises presented in a long Web document. From an information seeking perspective, arguments could be clustered by similar claims or similar premises, and then ranked in the context of a specific information need by a user. In a similar way, this scheme could be used for automatic summarization.
However, the Claim-Premises scheme does not allow to distinguish between different kinds of premises supporting the claim. Hence, finegrained distinctions of premises into specific factual evidence versus any kind of common ground can not be captured. N/A Cohen’s (0.80) not reported claimed to be small Krippendorf’s (0.37-0.56) not reported not reported Cohen’s (0.50-0.57) Cohen’s (0.68) on 10 documents N/A N/A N/A not reported not reported Fleiss’ multiple results N/A Fleiss’ (0.67) only percentage agreement reported Krippendorf’s U (0.72) Krippendorf’s (0.81) and
Stede
Freeman + RST Commentary Florou et al. (2013) none public policy making and
Stede
based on Freeman Peldszus (2013b) Sergeant (2013)
N/A Wachsmuth (2014b) Llewellyn et al. (2014) et al.
none Stab (2014) and
Gurevych proprietary, no argumentation theory Claim-Premise based on Freeman
Riot Twitter Corpus
7729 tweets student essays 90 documents
The Toulmin’s model
The roles of claim and grounds correspond
to the definitions introduced earlier (claim and
premises, respectively). The role of warrant is to
justify a logical inference from grounds to claim.
To assure the trustworthiness of the warrant,
backing provides further set of information. Qualifier
limits the degree of certainty under which the
argument should be accepted and rebuttal presents
a situation in which the claim might be defeated.
For examples of arguments based on Toulmin’s
original model see, e.g.,
Based on our experiments during annotation pre-studies, we propose an extension of the Toulmin’s model by means of (1) omitting the qualifier for stating modality, as people usually do not state the degree of cogency, (2) omitting the warrant as reasoning for justifying the move from grounds to claims is not usually explained, (3) extending the role of backing so it provides additional set of information to back-up the argument as a whole but is not directly bound to the claim as the grounds are, and (4) adding refutation which attacks the rebuttal (attacking the attack). The scheme is depicted in Figure 3.
Bentahar et al. (2010), many argumentation systems make no distinction between their premises, despite the fact that in arguments expressed in natural language we can typically observe premises playing different roles. Toulmins’ scheme allows such a distinction using the set of different components (roles). “By identifying these roles, we can present the arguments in a more readily understandable fashion, and also identify the various ways in which the argument may be accepted or attacked” (Bentahar et al., 2010, p. 216).
Toulmin’s model, as a general framework for
modeling static monological argumentation
(Bentahar et al., 2010), has been used in works on
annotating argumentative discourse
In order to examine the proposed approaches, we conducted two extensive independent annotation studies. The central controversial topics were related to education. One distinguishing feature of educational topics is their breadth, as they attract researchers, practitioners, parents, or policymakers. Since the detailed studies are being published elsewhere, we summarize only the main results and outcomes in this paper.
In the first study, we used the Claim-Premises
scheme for annotating a dataset of web documents
consisting of 80 documents from six current
topics related to the German educational system (e.g.,
mainstreaming, staying down at school), which is
described in
In the second study, the annotation was split into two stages. In the first stage, we annotated 990 English comments to articles and forums posts with their argumentativeness (persuasiveness). The source sites were identified using a standard search engine and the content was extracted manually; we chose the documents randomly without any pre-filtering. In the second stage, we applied the extended Toulmin’s scheme on 294 argumentative English comments to articles and forums posts and 57 English newspaper editorials and blog posts. The topics cover, e.g., mainstreaming,3 single-sex schools, or homeschooling, among others.
Measuring inter-annotator agreement For
any real large-scale annotation attempt, measuring
inter-annotator agreement (IAA) is crucial in
order to estimate the reliability of annotations
and the feasibility of the task itself. Both
annotation approaches share one common sub-task:
labeling spans of tokens with their corresponding
argumentation concept, the boundaries of the
spans are not known beforehand. Therefore, the
most appropriate measure here is the unitized
Krippendorf’s U as the annotators identify and
label the units in the same text
During an annotation study of 6 weeks, three annotators (one inexperienced annotator and two experts) annotated 80 documents belonging to six topics. On average, each annotator needed 23 hours to annotate the 3863 sentences. The annotators marked 5126 argument components (53% premises, 47% claims) and 2349 arguments, which is 2:2 argument components per argument. On average, 74% of the tokens in the dataset are covered by an argument component indicates that the documents are in fact highly argumentative. An average claim spans 1.1 sentences, whereas an average premise spans 2.2 sentences.
While the IAA scores appeared to be nonsubstantial, ranging from U =34:6 (distinguishing all 6 annotation classes and nonargumentative) to U =42:4 (distinguishing between premises, claims and non-argumentative), they are in line with previous results: Peldszus and Stede (2013b) report U =42:5 for their sentencelevel annotation study.
By analysing typical patterns of argument components used in arguments, Kluge (2014) found that almost three quarters of arguments (72.4%) consist of one claim and one premise. In 59.5% of these arguments, the support follows the claim, 3Discussion about benefits or disadvantages of including children with special needs into regular classes. Claim Grounds Backing Rebuttal Refutation
Comments, Forums whereas only in 11.6% of the arguments, the support precedes the claim. The corresponding patterns consisting of attack and claim are significantly less frequent: only 3.4% of the arguments consist of a claim and an attack.
Annotated examples can be found in xA.1. 5.2
In the first stage, three independent annotators labeled 524 out of 990 documents as argumentative/persuasive on the given topic. Total size of this dataset was 130,085 tokens (mean 131, std. dev. 139) and 6,371 sentences (mean 6.44, std. dev. 6.53). Agreement on the first sub-set of this dataset of 300 documents was 0.51 (Fleiss’ , three annotators per document), the second sub-set (690 documents) was then annotated by two annotators with agreement 0.59 (Cohen’s ). This stage took in total about 17 hours per annotator.
In the second phase that took about 33 hours per annotator, a collection of comments and forum posts (294 documents) was randomly chosen from the previously labeled argumentative documents from the previous stage together with 49 blog posts and 8 newspaper articles. The total size of this dataset was 345 documents, containing 87,286 tokens (mean 253.00, std. dev. 262.90) and 3,996 sentences (mean 11.58, std. dev. 11.72). Three independent annotators annotated the whole dataset in multiple phases. After each phase, they discussed discrepancies, resolved issues and updated the annotation guidelines. The inter-annotator agreement was measured on the last phase containing 93 comments and forum posts, 8 blogs, and 6 articles. During the annotations, 2 articles and 4 forum posts/comments were also discarded as non-argumentative.
Agreement (Krippendorf’s U ) varies significantly given different argumentation components and registers, as shown in Table 2. Given these results, we formulate the following conclusions.
This scheme seems to fit well short documents (forum posts and comments) as they tend to bring up one central claim with a support (grounds). Its suitability for longer documents (blogposts and editorials) is doubtful. We examined the annotation errors and found that in well-structured documents, the annotators were able to identify the concepts reliably. However, if the discussion of the controversy is complex (many sub-aspects are discussed) or follows a dialogical manner, application the Toulmin’s scheme is all but straightforward.
Furthermore, the distinction between grounds and backing also allows to capture different kinds of evidence. Authors purposely use grounds to explicitly support their claim, while backing mostly serves as an additional information (i.e., author’s personal experience, referring to studies, etc.) and the argument can be still acceptable without it. However, boundaries between these two components are still fuzzy and caused many disagreements.
We show few annotation examples (as agreed by all annotators after the study) in xA.2. 6
In this section, we would like to summarize some important findings from our annotation studies. 6.1
Variety or registers There exist many on-line registers that carry argumentation to topics under investigation, such as newspaper reports (i.e., events), editorials (opinions), interviews (single party, multiple parties), blogposts,4 comments to articles and blogs (threaded allowing explicit discussion, linear with implicit discussion by quoting and referencing), discussion forums, Twitter, etc.
ment lengths affect the style of argumentation.
Short documents (i.e., Tweets in the extreme case)
have to focus on the core of the argument. By
contrast, long documents, such as blog posts or
editorials, may elaborate various aspects of the topic
and usually employ many literary devices, such as
4In contrast to traditional publisher, bloggers do not have
to comply with strict guidelines or the use of formal language
Producing a well-understandable argument is
actually a human skill that can be acquired by
learning; many textbooks are available on that
topic, e.g.,
Non-argumentative texts Distinguishing
argumentative from non-argumentative discourse is a
necessary step that has to be undertaken before
annotating argument components. While in
newspaper editorials some parts (such as paragraphs) may
be ignored during argument annotation
Implicit argumentation components in Toulmin’s model As already reported by Newman and Marshall (1991), some argument components are not explicitly expressed. This is mostly the case of warrant in the original Toulmin’s model; we also discarded this component from our extension. However, even the claim is often not stated explicitly, as seen in example 3 (xA.2). The claim reflects the author’s stance and can be understood (inferred) by readers, but is left implicit. Other rhetorical dimensions of argument All the models for argumentation discussed so far focus solely on the logos part of the argument. However, rhetorical power of argumentation also involves other dimensions, namely pathos, ethos, and kairos (Aristotle and Kennedy (translator), 1991; Schiappa and Nordin, 2013). These have never been tackled in computational approaches to modeling argumentation. Furthermore, figurative langauge, fallacies, or narratives (see example 3 in xA.2) are prevalent in argumentation on the Web. Based on the experience from the annotation studies, we would like to conclude with the following recommendations: (1) selection of argumentation model should be based on the data at hand and the desired application; our experiments show that Toulmin’s model is more expressive than the Claim-Premise model but is not suitable for long documents, (2) annotating argumentation is timedemanding and error-prone endeavor; annotators thus have to be provided with detailed and elaborated annotation guidelines and be extensively trained (our experiments with crowdsourcing were not successful). 7
Understanding argumentation in user-generated content can foster future research in many areas. Here we present two concrete applications. 7.1
Computer-supported argumentation has been a
very active research field, as shown by Scheuer
et al. (2010) in their recent survey of
various models and argumentation formalisms from
the educational perspective. Many studies on
computer-supported collaboration and
argumentation
When summarizing argumentative discourse,
knowledge of the underlying structure of the
argument is a valuable source. Previous work in this
area includes, e.g., opinion-based summarization
of blogposts
In this article, we formulated our view on argumentation mining on the Web and identified various use-case scenarios and expected outcomes. We thoroughly reviewed related work with focus on Web data and annotation approaches. We proposed two different annotation schemes based on their theoretical counterparts in argumentation research and evaluated their suitability and reliability for Web data in two extensive independent annotation studies. Finally, we outlined challenges and gaps in current argumentation mining on the Web.
This work has been supported by the Volkswagen Foundation as part of the LichtenbergProfessorship Program under grant No. I/82806, and by the German Institute for Educational Research (DIPF).
A.1
[claim: ,,Die Umstellung zu G8 war schwierig“, sagt Diana. ] [support: In den Sommerferien nach dem Sitzenbleiben holte sie das nach, was ihr die G8er voraus hatten: Lateinvokabeln, Stochastik, Grammatik. ,,Den Vorteil, durch das Wiederholen den Stoff noch mal zu machen, hatte ich nicht.“ ] [claim: “The change [to G8] was difficult,” says Diana. ] [support: (Since) After staying down, she had to catch up with the G8 students during her summer holiday, studying Latin vocabulary, 5http://www.nist.gov/tac/publications/ 2008/papers.html stochastics, and grammar. “I did not have the advantage of reviewing previous material.” ]
[claim: Lehrer wird man, weil das ein sicherer Beruf ist. ] [support: So denken noch immer viele junge Leute, die sich fu¨r eine Pa¨dagogenlaufbahn entscheiden. Gut acht von zehn Erstsemestern, die 2009 mit einem Lehramtsstudium anfingen, war dieser Aspekt ihres ku¨nftigen Berufs wichtig oder sogar sehr wichtig. Keine andere Studentengruppe, die die Hochschul-Informations-System GmbH HIS befragte, legt so viel Wert auf Sicherheit. ] [claim: People become teachers because it is a safe job. ] [support: This is what more and more young people who decide to become a teacher think. Well over eight of 10 freshman students who started to study to become teachers in 2009 considered this an important or very important aspect. No other group of students interviewed by the HIS set that much value on safeness. ]
[claim: Fu¨r die Unis sind Doktoranden gu¨nstige Arbeitskra¨fte. ] [support: Eine Bekannte hatte mit ihrem Doktorvater zu ka¨mpfen, der versuchte, sie noch am Institut zu halten, als ihre Arbeit la¨ngst fertig war. Er hatte immer neue Ausreden, weshalb er noch keine Note geben konnte. Als sie dann auch ohne Note einen guten Job bekam, auerhalb der Uni, spielte sich eine Art Rosenkrieg zwischen den beiden ab. Bis heute verlangt er von ihr noch Nacharbeiten an der Dissertation. Sie schuftet jetzt spa¨tabends und am Wochenende fu¨r ihren Ex-Prof, der natu¨rlich immer nur an ihrem Fortkommen interessiert war. ] [claim: At university, graduate students are cheap employees. ] [support: An acquaintance struggled with her Ph.D. supervisor, who tried to keep her in his group at any rate, even though she had already completed her thesis. He pled more and more excuses for not yet grading her work. When she finally found a good job outside university even without a final grade a martial strife arose. Still today, he asks her to rework her dissertation. Now, she is drudging for her ex-supervisor, who always only wanted the best for her, late in the evening or on the weekend. ] Example 1 [.b. a..c.k.i.n.g.:. . . .I.’. m.. .a. . .re..g.u.l.a.r. .e.d. u..c.a.t.io..n. .t.e.a.c. h..e.r.. . . .I h..a.v.e. .s.t.u.d. e.n..ts.. m.. a..i n..st.r.e.a. m.. e.d.. .in..to.. .m. .y. .c.l.a.s.s. .e.v. e.r. y. y..e.a.r..]. [grounds: My opinion is that it needs to be done far more judiciously than it is done now- if six exceptional children are put in my class, that is the equivalent of putting an entire special ed classroom into my regular class.] [grounds: I personally feel like these kids are shortchangedsome of them are good kids who need an adult close by and able to give more focused attention. In a class of 30+, this isn’t going to happen consistently.] [grounds: And some of the ones who come to me have legally imposed modifications, some of which have little or no bearing on what I teach, so I am not allowed to handle my class in a way I think it should be done. That impairs my efficiency as an educator.] [grounds: Also, some have so many modifications that for all intents and purposes they are merely taking a special ed class whose physical location just happens to be in a regular classroom.] [claim: From my point of view, mainstreaming is not a terrible idea, but it is lamentable in its execution, and because of that, damaging in its results.] Comments Quite a good argument with an explicit claim, few grounds and some backing.
tara mommy: I agree with you too, which is why I said: [:r:e:b:u:t:ta:l:::::::::::a:re::o:b:v:i:o:u:s:l:y::c:a:se:s::w::h:e:re::t:h:is:
There isn’t going to work. Extreme behavioral trouble, :::::::::::::::::::::::::::::::::::::::::::: kids that just aren’t able to keep up with what :::::::::::::::::::::::::::::::::::::::::::: average classes, etc.] [claim: t:h:e:y:’:r:e:l:e:a:r:n:in:g::i:n:::::::::::::::::::: But on the whole, I like mainstreaming.]
ing grounds.
Only claim and rebuttal; no supportExample 3 l think as parents of the child you have to be certain and confident that your child is ready to mainstream. lf not, it can backfire on the child. [. b..a.c.k.i.n.g.:. . . . M.. y.. .c.h.i.l.d. . w..a.s. .i.n. . ”..p.r.e.s.c.h. o..o.l h..a.n.d.i.c.a. p..p.e.d.”. . .f.r.o. m.. . .a. g..e. . .2.-.5... . . . . .W. .e. . .t.ri.e. d.. . .t o. m..a. i.n. s.t.r.e.a. m.. .h.i.m. . i.n. .k.i.n. d..e.rg..a.r.te..n.,.b. u..t.h.e. .h.a. d.. a..h. a.r. d. t.i.m. .e. a..d.ju..s.ti. n.g.... S..o. t.h. e..s.c. h..o.o.l. g..o.t. h..i m.. .a. o..n.e. .o.n. .o.n. e. p..a.r.a. a.n..d. .it. .h.e.l.p.e. d.. a..b. i.t... 2.. g..ra..d.e.s. .la..te.r.,. .h.e. .s.ti.l.l.h. a..s a. .o. n..e. o..n. .o.n.e. .a.i.d.e. .b.u. t. .d.o.i.n. g.. E..X. .C. E..L. L..E. .N. T...]. Our goal is for him to not have a one on one by middle school. We took him off meds and we have a strong behavior plan, he sees therapists, and it is hourly teaching and redirecting with him. Truth be told College may not be in his future, but we will do everything in our power to try to get him there. Comments The claim is implicit, the author is slightly against mainstreaming. Mainly storytelling, which is not considered as grounds but as backing. The typos (using ‘l’ instead of ‘I’) are kept uncorrected.
My lo has mild autism, he has only just been diagnosed, he is delayed in some areas (but not others), he goes to ms school, and has some one to one (this should increase now, I hope). There is one TA and a full time TA who supports another child with autism. It’s a smallish school. He isn’t disruptive (well he sometimes doesn’t do as asked and can be a little akward), he has never been aggressive in anyway, he is very happy. I am worried about his future (high school)after reading this.
Sarah x Corina Andone. 2012. Bermejo-Luque, Lilian. Giving Reasons. A Linguistic-Pragmatic Approach to Argumentation Theory. Argumentation, 26(2):291–296. Aristotle and George Kennedy (translator). 1991. On Rhetoric: A Theory of Civil Discourse. Oxford University Press.
Bal Krishna Bal and Patrick Saint-Dizier. 2010. Towards Building Annotated Resources for Analyzing Opinions and Argumentation in News Editorials. In Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner, and Daniel Tapias, editors, Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), pages 1152–1158. European Language Resources Association (ELRA).
Jamal Bentahar, Bernard Moulin, and Micheline Be´langer. 2010. A taxonomy of argumentation models used for knowledge representation. Artificial Intelligence Review, 33:211–259.
text. Not an argumentative/persuasive