Argumentation Mining on the Web from Information Seeking Perspective Ivan Habernal†‡ , Judith Eckle-Kohler†‡ , Iryna Gurevych†‡ † Ubiquitous Knowledge Processing Lab (UKP-TUDA) Department of Computer Science, Technische Universität Darmstadt ‡ Ubiquitous Knowledge Processing Lab (UKP-DIPF) German Institute for Educational Research www.ukp.tu-darmstadt.de Abstract cannot feasibly process such massive amounts of data in order to reveal argumentation. Unfortu- In this paper, we argue that an annota- nately, even current Web technologies (such as tion scheme for argumentation mining is search engines or opinion mining services) are not a function of the task requirements and the suitable for such a task. This drives the research corpus properties. There is no one-size- field to the next challenge – argumentation min- fits-all argumentation theory to be applied ing on the Web. The abundance of freely available to realistic data on the Web. In two anno- (yet unstructured, textual) data and possible appli- tation studies, we experiment with 80 Ger- cations of such tools makes this task very appeal- man newspaper editorials from the Web ing. and about one thousand English docu- Our research into argumentation mining is mo- ments from forums, comments, and blogs. tivated by the information seeking perspective. Our example topics are taken from the The key sources are discussions (debates) about educational domain. controversies (contentions) targeted at a particular To formalize the problem of annotating topic which is of the user’s interest. The scope is arguments, in the first case, we apply a not limited to a particular media type as the source Claim-Premise scheme, and in the second types can range from the on-line newspapers’ ed- case, we modify Toulmin’s scheme. We itorials to user-generated discourse in social me- find that the choice of the argument com- dia, such as blogs and forum posts, covering dif- ponents to be annotated strongly depends ferent aspects of the issues. Understanding posi- on the register, the length of the document, tions and argumentation in on-line debates helps and inherently on the literary devices and users to form their opinions on controversial issues structures used for expressing argumenta- and also fosters personal and group decision mak- tion. We hope that these findings will fa- ing (Freeley and Steinberg, 2008, p. 9). The main cilitate the creation of reliably annotated task would be to identify and extract the core ar- argumentation corpora for a wide range of gumentation (its formal aspects will be discussed tasks and corpus types and will help to later) and present this new knowledge to users. bridge the gap between argumentation the- By utilizing argumentation mining methods, users ories and actual application needs. can be provided with the most relevant informa- tion (arguments) regarding the controversy under 1 Introduction investigation. Argumentation mining apparently represents an Although argumentation mining on the Web emerging field in Natural Language Processing has already been partly outlined in the literature (NLP) with publications appearing at mainstream (Schneider et al., 2012; Sergeant, 2013), the re- conferences, such as ACL (Cabrio and Villata, quirements and use-case scenarios differ substan- 2012; Feng and Hirst, 2011; Madnani et al., 2012) tially. Various tasks are being solved, most of them or COLING (Stab and Gurevych, 2014; Levy et depending on the domain, e.g., product reviews or al., 2014; Wachsmuth et al., 2014a). In particular, political contentions. As a result, different inter- there is an increasing need for tools capable of un- pretations of arguments and argumentation have derstanding argumentation on the large scale, be- been developed in NLP, and therefore, most of cause in the current information overload, humans the existing researches are not directly adaptable. News co Noroozi et al., 2013), pragmatics (Xu and Wu, Comments nt ro 2014), psychology (Larson et al., 2004), and many ve r sy others. Given so many different perspectives on Forums investigating argumentation, there is a plethora of possible interpretations of argumentation min- Blogs ing. Thus, finding a common understanding of this evolving field is a fundamental challenge. For NLP research, this overwhelming amount of related works brings many theoretical and prac- tical issues. In particular, there is no one-size- Argument fits-all argumentation theory. Even argumentation researchers disagree on any widely-accepted ulti- Figure 1: Schematic overview of argumentation mate concept. For example, Luque (2011) criti- mining on the Web cizes the major existing approaches in order to es- tablish a new theory which is later again severely Morover, not all of the related research works are criticized by other in-field researches (Andone, tightly connected to argumentation theories (de 2012; Xie, 2012). Given this diversity of perspec- Moor et al., 2004; Villalba and Saint-Dizier, 2012; tives, NLP research cannot simply adopt one par- Cabrio et al., 2013b; Llewellyn et al., 2014). How- ticular approach without investigating its theoret- ever, we feel that it is vital to ground NLP research ical background as well as its suitability for the in argumentation mining in existing work on argu- particular task. mentation. 2.1 What we do not tackle In this article, we will particularly focus on bridging the gap between argumentation theories Given the breath of argumentation mining just out- and actual application needs that has not been tar- lined, we would also like to discuss aspects that do geted in the relevant literature. We will support not fit into our approach to argumentation mining, our findings by comprehensively surveying exist- namely macro argumentation and evaluation using ing works and presenting results from two exten- formal frameworks. sive annotation studies. First, we treat argumentation as a product (mi- Our main findings and suggestions can be sum- cro argumentation or monological models), not marized as follows: First, the use-case of any re- as a process (macro argumentation or dialogical search in argumentation mining must be clearly models). While dialogical models highlight the stated (i.e., in terms of expected outcomes). Sec- process of argumentation in a dialogue structure, ond, properties of the data under investigation monological models emphasize the structure of must be taken into account, given the variety of the argument itself (Bentahar et al., 2010, p. 215). genres and registers (Biber and Conrad, 2009). Therefore, we examine the relationships between Third, an appropriate argumentation model must the different components of a given argument, be chosen according to the requirements. There- not a relationship that can exist between argu- fore, we claim that it is not possible to formulate ments.1 Exploring how argumentation evolves be- a single argumentation mining perspective that tween parties in time remains out of our scope. would be applicable to the Web data in general. Second, we do not tackle any logical reason- ing, defeasibility of reasoning, or evaluating argu- 2 Relation to Argumentation Theories mentation with formal frameworks in general. Al- though this is an established field in informal logic Research on argumentation is widely interdis- (Prakken, 2010; Hunter, 2013; Hunter, 2014), ciplinary, as it spreads across philosophy and such an approach might not be suitable directly rhetoric (Aristotle and Kennedy (translator), for Web data as it assumes that argumentation is 1991; Perelman and Olbrechts-Tyteca, 1991; Wal- logical (such a strong assumption cannot be guar- ton et al., 2008), informal and formal logic 1 (Dung, 1995; Henkemans, 2000; Stoianovici, For further discussion see, e.g., (Blair, 2004; Johnson, 2000; Reed and Walton, 2003) or Micheli (2011) who sum- 2009; Schneider et al., 2013; Hunter, 2013), edu- marizes the distinction between the process (at a pragmatic cational research (Weinberger and Fischer, 2006; level) and the product (at a more textual level). anteed). Furthermore, acceptability of arguments pus. Appropriateness of such an approach remains also touches the fundamental problem of the target questionable. On the one hand, Walton’s argumen- audience of the argument, as different groups have tation schemes are claimed to be general and do- different perceptions. Crosswhite et al. (2004) main independent. On the other hand, evidence point out that “one of the key premises from which from the field shows that schemes might not be the study of rhetoric proceeds is that influencing the best means for analyzing user-generated argu- real audiences is not simply a matter of presenting mentation. In examining real-world political ar- a set of rational, deductive arguments.” gumentation from (Walton, 2005), Walton (2012) found out that 37.1% of the arguments collected 2.2 Common terminology did not fit any of the fourteen schemes they chose Let us set up a common terminology. Claim is so they created new schemes ad-hoc. Cabrio et al. “the conclusion we seek to establish by our argu- (2013a) select five argumentation schemes from ments” (Freeley and Steinberg, 2008, p. 153) or Walton and map these patterns to discourse rela- “the assertion put forward publicly for general ac- tion categories in the Penn Discourse TreeBank ceptance” (Toulmin et al., 1984, p. 29). Premises (PDTB) (Prasad et al., 2008), but later they define are “connected series of sentences, statements, or two new schemes that they discovered in PDTB. propositions that are intended to give reasons of These findings confirm that the schemes lack cov- some kind for the claim” (Freeley and Steinberg, erage for dealing with real argumentation in natu- 2008, p. 3). ral language texts. 3 Related Work 3.2 Previous works on annotation Table 1 summarizes the previous research on an- 3.1 Opinion mining perspective notating argumentation. Not only it covers re- In existing works on argumentation mining of the lated work from the NLP community but also Web data, the connection is often made to opin- studies from general discourse analysis (Newman ion mining (Liu, 2012). From the users’ point and Marshall, 1991; Walton, 2012) and road-maps of view, opinion mining applications reveal what or position papers (Schneider and Wyner, 2012; people think about something. The key question Peldszus and Stede, 2013a; Sergeant, 2013). The which brings argumentation on the scene is why heterogeneity of used argumentation models and do they think so? – in other words, explaining the the domains under investigation demonstrates the reasons behind opinions. breath of the argumentation mining field. We iden- Villalba and Saint-Dizier (2012) approach tified the following research gaps. aspect-based sentiment of product reviews by clas- sifying discourse relations conveying arguments • Most studies dealing with Web data use (such as justification, reformulation, illustration, some kind of proprietary model without re- and others). They build upon Rhetorical Structure lation to any argumentation theory (Bal and Theory (RST) (Mann and Thompson, 1987) and Saint-Dizier, 2010; Rosenthal and McKe- argue that rhetorical elements related to explana- own, 2012; Conrad et al., 2012; Schneider tion behave as argument supports. and Wyner, 2012; Villalba and Saint-Dizier, For modeling argumentation in social media, 2012; Florou et al., 2013; Sergeant, 2013; Schneider et al. (2012) suggest using Dung’s Wachsmuth et al., 2014b; Llewellyn et al., framework (Dung, 1995) with Walton schemes 2014). (Walton et al., 2008), but do not provide evidence • Inter-annotation agreement (IAA) that re- for such a decision. They admit that “It is far flects reliability of the annotated data is either from clear how an argument [...] can be trans- not reported (Feng and Hirst, 2011; Mochales formed into a formal argumentation scheme so and Moens, 2011; Walton, 2012; Florou et that it can be reasoned in an argumentation frame- al., 2013; Villalba and Saint-Dizier, 2012), or work” (Schneider et al., 2012, p. 22). is not based on a chance-corrected measure Schneider and Wyner (2012) focus on the prod- (Llewellyn et al., 2014). uct reviews domain and develops a number of ar- gumentation schemes (inspired by (Walton et al., This motivates our research into annotating Web 2008)) based on manual inspection of their cor- data relying on a model based on a theoretical Claim premise and a claim. The simplest way to rep- restatement = {true, false} resent the support and attack relations is to attach labels to adjacent argument components, which in- Pre-Support Pre-Attack dicate their argumentative role. The span of argu- Post-Support Post-Attack ment components is left unspecified, allowing for premises argument components spanning a clause or one to several sentences. Using the six labels claim, re- Figure 2: Claim-Premise scheme. Note that the re- statement, pre-claim support, post-claim support, lations (arrows) are only illustrative; they are im- pre-claim attack and post-claim attack, a linear plicitly encoded in the roles of the particular argu- sequence of non-nested arguments can be repre- ment components. sented. While graph structures where nodes stand for background in argumentation and reporting IAA argument components, and edges for support or that would confirm suitability of the model and re- attack relations are a more general way to repre- liability of the annotated data. sent arguments (equivalent to, i.e., (Dung, 1995) or (Freeman, 1991)), it is unclear which additional 4 Annotating argumentation in Web data benefits such a more fine-grained annotation of ar- Up until now, we have used the terms argumenta- guments brings for the annotation of Web docu- tion and argument in their common meaning with- ments. In a pre-study performed by Kluge (2014), out any particular formal definition. We will now the possibility to annotate nested arguments turned elaborate on annotation schemes and discuss their out to be a drawback, rather than an advantage, be- suitability and reliability for the Web data. cause the inter-annotator agreement dropped con- siderably. 4.1 Annotation Schemes Because of the lack of a single general-purpose argumentation model (cf. discussion in §1), we Suitability of the scheme The main advantage present here two different schemes.2 Both are built of the Claim-Premises scheme is its simplicity. upon foundations in argumentation theories, but Therefore, it is particularly suited for annotating they differ in their granularity, expression power, arguments in long Web documents, such as news and other properties. articles, editorials or blog posts. Kluge (2014) found that most documents of these text types con- 4.1.1 Claim-Premises scheme sist of three major parts: an introductory part, The Claim-Premises scheme is widely used in pre- summarizing the document content in one or two vious work on argumentation mining, e.g., (Palau paragraphs, the main part, presenting a linear se- and Moens, 2009; Florou et al., 2013; Peldszus quence of arguments, and an optional concluding and Stede, 2013b). It defines an argument as con- part summarizing the main arguments. sisting of a (possibly empty) set of premises and a The Claim-Premise scheme can be used to pro- single claim; premises either support or attack the vide an overview of the claims and their sup- claim (Besnard and Hunter, 2008). We adopted porting or attacking premises presented in a long this general scheme for the purpose of annotating Web document. From an information seeking per- arguments in long Web documents (Kluge, 2014). spective, arguments could be clustered by similar According to this adopted version of the scheme, claims or similar premises, and then ranked in the claims, restatements and premises are subsumed context of a specific information need by a user. under the term argument component; a restate- In a similar way, this scheme could be used for ment of a claim is also considered as claim and is automatic summarization. part of the same argument. The scheme is depicted in Figure 2. However, the Claim-Premises scheme does not Premises either support or attack a claim, i.e., allow to distinguish between different kinds of there is a support or attack relation between each premises supporting the claim. Hence, fine- 2 grained distinctions of premises into specific fac- An exhaustive overview of various argumentation mod- els, their taxonomy, and properties can be found in (Bentahar tual evidence versus any kind of common ground et al., 2010). can not be captured. Source Arg. Model Domain Size IAA Newman and Marshall Toulmin legal domain (Peo- qualitative N/A (1991) ple vs. Carney, U.S. Supreme Court) Bal and Saint-Dizier proprietary socio-political newspa- 56 documents Cohen’s κ (2010) per editorials (0.80) Feng and Hirst (2011) Walton legal domain (Aracu- ≈ 400 arguments not reported (top 5 schemes) raria corpus, 61% sub- claimed to be small set annotated with Wal- ton scheme) Georgila et al. (2011) proprietary general discussions 21 dialogues Krippendorf’s α (negotiations between (0.37-0.56) florists) Mochales and Moens Claim-Premise legal domain (Aracu- 641 documents not reported (2011) based on Freeman raria corpus, European w/ 641 arguments Human Rights Council) (Aracuraria) 67 documents w/ 257 arguments (EHRC) Walton (2012) Walton political argumentation 256 arguments not reported (14 schemes) Rosenthal and McKe- opinionated blogposts, Wikipedia 4000 sentences Cohen’s κ own (2012) claim, sentence discussions (0.50-0.57) level Conrad et al. (2012) proprietary editorials and blogpost 84 documents Cohen’s κ (spans of arguing about Obama Care (0.68) subjectivity) on 10 documents Schneider and Wyner proprietary, ar- camera reviews N/A N/A (2012) gumentation (proposal/position schemes paper) Schneider et al. (2012) Dung + Walton unspecified social me- N/A N/A dia (proposal/position paper) Villalba and Saint- proprietary, RST hotel reviews, hi-fi 50 documents not reported Dizier (2012) products, political campaign Peldszus and Stede Freeman + RST Potsdam Commentary N/A N/A (2013a) Corpus (proposal/position paper) Florou et al. (2013) none public policy making 69 argumentative not reported segments / 322 non-argumentative segments Peldszus and Stede based on Freeman not reported, artificial 23 short documents Fleiss’ κ (2013b) documents created for multiple results the study Sergeant (2013) N/A Car Review Corpus N/A N/A (CRC) (proposal/position paper) Wachsmuth et al. none hotel reviews 2100 reviews Fleiss’ κ (2014b) (0.67) Llewellyn et al. (2014) proprietary, no ar- Riot Twitter Corpus 7729 tweets only percentage gumentation the- agreement reported ory Stab and Gurevych Claim-Premise student essays 90 documents Krippendorf’s αU (2014) based on Freeman (0.72) Krippendorf’s α (0.81) Table 1: Previous works on annotating argumentation. IAA = Inter-annotation agreement; N/A = not applicable. Backing Backing ponents (roles). “By identifying these roles, we can present the arguments in a more readily un- Grounds Claim Grounds implicit = {true, false} derstandable fashion, and also identify the various ways in which the argument may be accepted or Rebuttal attacked” (Bentahar et al., 2010, p. 216). Refutation Refutation Rebuttal Toulmin’s model, as a general framework for modeling static monological argumentation (Ben- Figure 3: Extended Toulmin’s scheme. Note that tahar et al., 2010), has been used in works on the relations (arrows) are only illustrative; they are annotating argumentative discourse (Newman and implicitly encoded in the roles of the particular ar- Marshall, 1991; Chambliss, 1995; Simosi, 2003; gument components. Weinberger and Fischer, 2006). However, its com- plexity and the fact that the description of the com- 4.1.2 Toulmin’s scheme ponents is informal and sometimes ambiguous, The Toulmin’s model (Toulmin, 1958) is a con- poses challenges for an application of the model ceptual model of argumentation, in which differ- on real-world data, especially user-generated dis- ent components play distinct roles. In the original course on the Web. Moreover, some of the com- form, it consists of six components: claim, data ponents are usually left implicit in argumentation, (grounds), warrant, backing, qualifier, and rebut- such as the warrant or even the claim (Newman tal. and Marshall, 1991). The roles of claim and grounds correspond to the definitions introduced earlier (claim and 5 Preliminary results of annotation premises, respectively). The role of warrant is to studies justify a logical inference from grounds to claim. In order to examine the proposed approaches, we To assure the trustworthiness of the warrant, back- conducted two extensive independent annotation ing provides further set of information. Qualifier studies. The central controversial topics were re- limits the degree of certainty under which the ar- lated to education. One distinguishing feature gument should be accepted and rebuttal presents of educational topics is their breadth, as they at- a situation in which the claim might be defeated. tract researchers, practitioners, parents, or policy- For examples of arguments based on Toulmin’s makers. Since the detailed studies are being pub- original model see, e.g., (Freeley and Steinberg, lished elsewhere, we summarize only the main re- 2008, Chap. 8). sults and outcomes in this paper. Based on our experiments during annotation In the first study, we used the Claim-Premises pre-studies, we propose an extension of the Toul- scheme for annotating a dataset of web documents min’s model by means of (1) omitting the qualifier consisting of 80 documents from six current top- for stating modality, as people usually do not state ics related to the German educational system (e.g., the degree of cogency, (2) omitting the warrant as mainstreaming, staying down at school), which is reasoning for justifying the move from grounds to described in (Kluge, 2014). The dataset contains claims is not usually explained, (3) extending the (newspaper) articles, blog posts, and interviews. role of backing so it provides additional set of in- It was created by Vovk (2013) who manually se- formation to back-up the argument as a whole but lected documents obtained from a focused crawler is not directly bound to the claim as the grounds and the top 100 search engine hits (per topic). are, and (4) adding refutation which attacks the In the second study, the annotation was split rebuttal (attacking the attack). The scheme is de- into two stages. In the first stage, we anno- picted in Figure 3. tated 990 English comments to articles and fo- Suitability of the scheme As pointed out by rums posts with their argumentativeness (persua- Bentahar et al. (2010), many argumentation sys- siveness). The source sites were identified using tems make no distinction between their premises, a standard search engine and the content was ex- despite the fact that in arguments expressed in nat- tracted manually; we chose the documents ran- ural language we can typically observe premises domly without any pre-filtering. In the second playing different roles. Toulmins’ scheme allows stage, we applied the extended Toulmin’s scheme such a distinction using the set of different com- on 294 argumentative English comments to arti- cles and forums posts and 57 English newspa- Argument Comments, Blogs Articles per editorials and blog posts. The topics cover, Component Forums e.g., mainstreaming,3 single-sex schools, or home- Claim 0.57 0.17 0.23 schooling, among others. Grounds 0.64 0.32 0.11 Backing 0.41 -0.16 0.28 Measuring inter-annotator agreement For Rebuttal 0.33 -0.02 0.00 any real large-scale annotation attempt, measuring Refutation 0.06 0.35 0.00 inter-annotator agreement (IAA) is crucial in order to estimate the reliability of annotations Table 2: IAA scores (Krippendorf’s αU ) from an- and the feasibility of the task itself. Both anno- notations using the Toulmin’s scheme. tation approaches share one common sub-task: labeling spans of tokens with their corresponding argumentation concept, the boundaries of the whereas only in 11.6% of the arguments, the sup- spans are not known beforehand. Therefore, the port precedes the claim. The corresponding pat- most appropriate measure here is the unitized terns consisting of attack and claim are signifi- Krippendorf’s αU as the annotators identify and cantly less frequent: only 3.4% of the arguments label the units in the same text (Krippendorff, consist of a claim and an attack. 2013). Other measures, such as Cohen’s κ or Annotated examples can be found in §A.1. Fleiss’ π, expect the units (boundaries of the argument component) to be known beforehand, 5.2 Outcomes of annotating with Toulmin’s which is not the case here. scheme In the first stage, three independent annotators la- 5.1 Outcomes of annotating with beled 524 out of 990 documents as argumenta- Claim-Premises scheme tive/persuasive on the given topic. Total size of During an annotation study of 6 weeks, three this dataset was 130,085 tokens (mean 131, std. annotators (one inexperienced annotator and two dev. 139) and 6,371 sentences (mean 6.44, std. experts) annotated 80 documents belonging to dev. 6.53). Agreement on the first sub-set of six topics. On average, each annotator needed this dataset of 300 documents was 0.51 (Fleiss’ π, 23 hours to annotate the 3863 sentences. The three annotators per document), the second sub-set annotators marked 5126 argument components (690 documents) was then annotated by two anno- (53% premises, 47% claims) and 2349 arguments, tators with agreement 0.59 (Cohen’s κ). This stage which is 2.2 argument components per argument. took in total about 17 hours per annotator. On average, 74% of the tokens in the dataset are In the second phase that took about 33 hours covered by an argument component indicates that per annotator, a collection of comments and forum the documents are in fact highly argumentative. posts (294 documents) was randomly chosen from An average claim spans 1.1 sentences, whereas an the previously labeled argumentative documents average premise spans 2.2 sentences. from the previous stage together with 49 blog While the IAA scores appeared to be non- posts and 8 newspaper articles. The total size of substantial, ranging from αU =34.6 (distin- this dataset was 345 documents, containing 87,286 guishing all 6 annotation classes and non- tokens (mean 253.00, std. dev. 262.90) and 3,996 argumentative) to αU =42.4 (distinguishing be- sentences (mean 11.58, std. dev. 11.72). Three in- tween premises, claims and non-argumentative), dependent annotators annotated the whole dataset they are in line with previous results: Peldszus and in multiple phases. After each phase, they dis- Stede (2013b) report αU =42.5 for their sentence- cussed discrepancies, resolved issues and updated level annotation study. the annotation guidelines. The inter-annotator By analysing typical patterns of argument com- agreement was measured on the last phase con- ponents used in arguments, Kluge (2014) found taining 93 comments and forum posts, 8 blogs, that almost three quarters of arguments (72.4%) and 6 articles. During the annotations, 2 articles consist of one claim and one premise. In 59.5% and 4 forum posts/comments were also discarded of these arguments, the support follows the claim, as non-argumentative. 3 Discussion about benefits or disadvantages of including Agreement (Krippendorf’s αU ) varies signifi- children with special needs into regular classes. cantly given different argumentation components and registers, as shown in Table 2. Given these narratives, quotations from sources, or direct and results, we formulate the following conclusions. indirect speech. This scheme seems to fit well short documents (forum posts and comments) as they tend to bring Well structured newspaper articles versus up one central claim with a support (grounds). poorly structured user-generated content Its suitability for longer documents (blogposts and Producing a well-understandable argument is editorials) is doubtful. We examined the annota- actually a human skill that can be acquired by tion errors and found that in well-structured doc- learning; many textbooks are available on that uments, the annotators were able to identify the topic, e.g., (Sinnott-Armstrong and Fogelin, 2009; concepts reliably. However, if the discussion of Weston, 2008; Schiappa and Nordin, 2013). the controversy is complex (many sub-aspects are Thus, it is very likely that, for example, trained discussed) or follows a dialogical manner, appli- journalists in editorials and lay people in social cation the Toulmin’s scheme is all but straightfor- media will produce very different argumentation, ward. in terms of structure, language, etc. Furthermore, the distinction between grounds and backing also allows to capture different kinds 6.2 Properties of argumentation in of evidence. Authors purposely use grounds to ex- user-generated discourse plicitly support their claim, while backing mostly serves as an additional information (i.e., author’s Non-argumentative texts Distinguishing argu- personal experience, referring to studies, etc.) and mentative from non-argumentative discourse is a the argument can be still acceptable without it. necessary step that has to be undertaken before an- However, boundaries between these two compo- notating argument components. While in newspa- nents are still fuzzy and caused many disagree- per editorials some parts (such as paragraphs) may ments. be ignored during argument annotation (Kluge, We show few annotation examples (as agreed 2014), in comments and forum posts we had to by all annotators after the study) in §A.2. perform an additional step to filter documents that do not convey any argumentation or persuasion 6 Observations (cf. §5.2 or Example 4 in §A.2). In this section, we would like to summarize some important findings from our annotation studies. Implicit argumentation components in Toul- min’s model As already reported by Newman 6.1 Data heterogeneity and Marshall (1991), some argument components are not explicitly expressed. This is mostly the Variety or registers There exist many on-line case of warrant in the original Toulmin’s model; registers that carry argumentation to topics un- we also discarded this component from our exten- der investigation, such as newspaper reports (i.e., sion. However, even the claim is often not stated events), editorials (opinions), interviews (single explicitly, as seen in example 3 (§A.2). The claim party, multiple parties), blogposts,4 comments to reflects the author’s stance and can be understood articles and blogs (threaded allowing explicit dis- (inferred) by readers, but is left implicit. cussion, linear with implicit discussion by quoting and referencing), discussion forums, Twitter, etc. Other rhetorical dimensions of argument All Short versus long documents Different docu- the models for argumentation discussed so far fo- ment lengths affect the style of argumentation. cus solely on the logos part of the argument. How- Short documents (i.e., Tweets in the extreme case) ever, rhetorical power of argumentation also in- have to focus on the core of the argument. By con- volves other dimensions, namely pathos, ethos, trast, long documents, such as blog posts or edito- and kairos (Aristotle and Kennedy (translator), rials, may elaborate various aspects of the topic 1991; Schiappa and Nordin, 2013). These have and usually employ many literary devices, such as never been tackled in computational approaches to 4 modeling argumentation. Furthermore, figurative In contrast to traditional publisher, bloggers do not have to comply with strict guidelines or the use of formal language langauge, fallacies, or narratives (see example 3 in (Santos et al., 2012). §A.2) are prevalent in argumentation on the Web. 6.3 Recommendations area includes, e.g., opinion-based summarization Based on the experience from the annotation stud- of blogposts (a pilot task in TAC 20085 ). Carenini ies, we would like to conclude with the follow- and Cheung (2008) compared extractive and ing recommendations: (1) selection of argumen- abstractive summaries in controversial documents tation model should be based on the data at hand and found out that a high degree of controver- and the desired application; our experiments show siality improved performance of their system. that Toulmin’s model is more expressive than the Similarly, presenting argumentation in a con- Claim-Premise model but is not suitable for long densed form (the large concepts of the argument documents, (2) annotating argumentation is time- are compressed or summarized) may improve demanding and error-prone endeavor; annotators argument comprehension. This approach would thus have to be provided with detailed and elab- mainly utilize tools for document compression orated annotation guidelines and be extensively (Qian and Liu, 2013). trained (our experiments with crowdsourcing were 8 Conclusions not successful). In this article, we formulated our view on argu- 7 Follow-up use cases mentation mining on the Web and identified var- ious use-case scenarios and expected outcomes. Understanding argumentation in user-generated We thoroughly reviewed related work with focus content can foster future research in many areas. on Web data and annotation approaches. We pro- Here we present two concrete applications. posed two different annotation schemes based on 7.1 Understanding argumentative discourse their theoretical counterparts in argumentation re- in education search and evaluated their suitability and reliabil- ity for Web data in two extensive independent an- Computer-supported argumentation has been a notation studies. Finally, we outlined challenges very active research field, as shown by Scheuer and gaps in current argumentation mining on the et al. (2010) in their recent survey of vari- Web. ous models and argumentation formalisms from the educational perspective. Many studies on Acknowledgments computer-supported collaboration and argumenta- tion (Noroozi et al., 2013; Weinberger and Fischer, This work has been supported by the Volks- 2006; Stegmann et al., 2007) can directly bene- wagen Foundation as part of the Lichtenberg- fit from NLP techniques for automatic argument Professorship Program under grant No. I/82806, detection, classification, and summarization. In- and by the German Institute for Educational Re- stead of relying on scripts (Dillenbourg and Hong, search (DIPF). 2008; Scheuer et al., 2010; Fischer et al., 2013) A Annotated examples or explicit argument diagramming (Scheuer et al., 2014), collaborative platforms can further provide A.1 News articles using Claim-Premises scholars with a summary of the whole argumen- scheme tation to the topic, reveal the main argumenta- Example 1 tive patterns, provide the weaknesses of other’s [claim: ,,Die Umstellung zu G8 war schwierig“, arguments, as well as identify shortcomings that sagt Diana. ] [support: In den Sommerferien nach need to be improved in the argumentative knowl- dem Sitzenbleiben holte sie das nach, was ihr die edge construction. Automatic analysis of micro- G8er voraus hatten: Lateinvokabeln, Stochastik, arguments can also help to overcome the existing Grammatik. ,,Den Vorteil, durch das Wiederholen trade-off between freedom (free-text option) and den Stoff noch mal zu machen, hatte ich nicht.“ ] guidance (scripts) (Dillenbourg and Hong, 2008). [claim: “The change [to G8] was difficult,” says 7.2 Automatic summarization of Diana. ] [support: (Since) After staying down, argumentative discourse she had to catch up with the G8 students during When summarizing argumentative discourse, her summer holiday, studying Latin vocabulary, knowledge of the underlying structure of the ar- 5 http://www.nist.gov/tac/publications/ gument is a valuable source. Previous work in this 2008/papers.html stochastics, and grammar. “I did not have the A.2 Forum posts using extended Toulmin’s advantage of reviewing previous material.” ] scheme Example 1 Example 2 [backing: . . . . . . . . . . . . . I’m . . . . .a. . .regular . . . . . . . .education . . . . . . . . . . .teacher. . . . . . . . . . . .I [claim: Lehrer wird man, weil das ein have . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .every students mainstreamed into my class ..... sicherer Beruf ist. ] [support: So denken year.] . . . . . . [grounds: My opinion is that it needs to be noch immer viele junge Leute, die sich für done far more judiciously than it is done now- if eine Pädagogenlaufbahn entscheiden. Gut acht six exceptional children are put in my class, that von zehn Erstsemestern, die 2009 mit einem is the equivalent of putting an entire special ed Lehramtsstudium anfingen, war dieser Aspekt classroom into my regular class.] [grounds: I ihres künftigen Berufs wichtig oder sogar sehr personally feel like these kids are shortchanged- wichtig. Keine andere Studentengruppe, die some of them are good kids who need an adult die Hochschul-Informations-System GmbH HIS close by and able to give more focused attention. befragte, legt so viel Wert auf Sicherheit. ] In a class of 30+, this isn’t going to happen consistently.] [grounds: And some of the [claim: People become teachers because it is a ones who come to me have legally imposed safe job. ] [support: This is what more and more modifications, some of which have little or no young people who decide to become a teacher bearing on what I teach, so I am not allowed to think. Well over eight of 10 freshman students handle my class in a way I think it should be who started to study to become teachers in 2009 done. That impairs my efficiency as an educator.] considered this an important or very important [grounds: Also, some have so many modifications aspect. No other group of students interviewed by that for all intents and purposes they are merely the HIS set that much value on safeness. ] taking a special ed class whose physical location just happens to be in a regular classroom.] [claim: Example 3 From my point of view, mainstreaming is not a terrible idea, but it is lamentable in its execution, [claim: Für die Unis sind Doktoranden günstige and because of that, damaging in its results.] Arbeitskräfte. ] [support: Eine Bekannte hatte mit ihrem Doktorvater zu kämpfen, der versuchte, Comments Quite a good argument with an ex- sie noch am Institut zu halten, als ihre Arbeit plicit claim, few grounds and some backing. längst fertig war. Er hatte immer neue Ausreden, Example 2 weshalb er noch keine Note geben konnte. Als sie dann auch ohne Note einen guten Job bekam, tara mommy: auerhalb der Uni, spielte sich eine Art Rosenkrieg I agree with you too, which is why I said: zwischen den beiden ab. Bis heute verlangt er von [rebuttal: There are obviously cases where this :::::::::::::::::::::::::::::::::::::::::::: ihr noch Nacharbeiten an der Dissertation. Sie isn’t going to work. Extreme behavioral trouble, :::::::::::::::::::::::::::::::::::::::::::: schuftet jetzt spätabends und am Wochenende für kids that just aren’t able to keep up with what :::::::::::::::::::::::::::::::::::::::::::: ihren Ex-Prof, der natürlich immer nur an ihrem they’re learning in average classes, etc.] [claim: :::::::::::::::::::::::::::::::::::: Fortkommen interessiert war. ] But on the whole, I like mainstreaming.] [claim: At university, graduate students are Comments Only claim and rebuttal; no support- cheap employees. ] [support: An acquaintance ing grounds. struggled with her Ph.D. supervisor, who tried to keep her in his group at any rate, even though Example 3 she had already completed her thesis. He pled l think as parents of the child you have to be more and more excuses for not yet grading her certain and confident that your child is ready work. When she finally found a good job outside to mainstream. lf not, it can backfire on the university even without a final grade a martial child. [backing:. . . . . . . . . . . . .My. . . . .child . . . . . .was . . . . . in . . . ”preschool ........... strife arose. Still today, he asks her to rework handicapped” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .tried from age 2-5. We . . . . . . .to. her dissertation. Now, she is drudging for her mainstream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . a. .hard him in kindergarten, but he had .... ex-supervisor, who always only wanted the best time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .one adjusting. So the school got him a one on ... for her, late in the evening or on the weekend. ] para . . . . . and . . . . .it. .helped . . . . . . . a. .bit. . . . . 2. . grades . . . . . . . .later, . . . . . .he . . .still . . . .has ... a. .one . . . . on . . . .one . . . .aide . . . . .but . . . .doing . . . . . .EXCELLENT.] ................ Philippe Besnard and Anthony Hunter. 2008. El- Our goal is for him to not have a one on one by ements of argumentation, volume 47. MIT press Cambridge. middle school. We took him off meds and we have a strong behavior plan, he sees therapists, and it is Douglas Biber and Susan Conrad. 2009. Register, hourly teaching and redirecting with him. Truth be Genre, and Style. Cambridge Textbooks in Linguis- told College may not be in his future, but we will tics. Cambridge University Press. do everything in our power to try to get him there. J. Anthony Blair. 2004. Argument and its uses. Infor- mal Logic, 24:137151. Comments The claim is implicit, the author is slightly against mainstreaming. Mainly story- Elena Cabrio and Serena Villata. 2012. Combin- telling, which is not considered as grounds but as ing textual entailment and argumentation theory for supporting online debates interactions. In Proceed- backing. The typos (using ‘l’ instead of ‘I’) are ings of the 50th Annual Meeting of the Association kept uncorrected. for Computational Linguistics (Volume 2: Short Pa- pers), pages 208–212, Jeju Island, Korea, July. As- Example 4 sociation for Computational Linguistics. My lo has mild autism, he has only just been di- Elena Cabrio, Sara Tonelli, and Serena Villata. agnosed, he is delayed in some areas (but not oth- 2013a. From Discourse Analysis to Argumen- ers), he goes to ms school, and has some one to tation Schemes and Back: Relations and Differ- ences. In João Leite, Tran Cao Son, Paolo Torroni, one (this should increase now, I hope). There is Leon Torre, and Stefan Woltran, editors, Proceed- one TA and a full time TA who supports another ings of 14th International Workshop on Computa- child with autism. It’s a smallish school. tional Logic in Multi-Agent Systems, volume 8143 He isn’t disruptive (well he sometimes doesn’t do of Lecture Notes in Computer Science, pages 1–17. as asked and can be a little akward), he has never Springer Berlin Heidelberg. been aggressive in anyway, he is very happy. Elena Cabrio, Serena Villata, and Fabien Gandon. I am worried about his future (high school)after 2013b. A support framework for argumentative reading this. discussions management in the web. In Philipp Cimiano, Oscar Corcho, Valentina Presutti, Laura Sarah x Hollink, and Sebastian Rudolph, editors, The Se- mantic Web: Semantics and Big Data, volume 7882 Comments Not an argumentative/persuasive of Lecture Notes in Computer Science, pages 412– text. 426. Springer Berlin Heidelberg. Giuseppe Carenini and Jackie Chi Kit Cheung. 2008. Extractive vs. NLG-based abstractive summariza- References tion of evaluative text: The effect of corpus contro- Corina Andone. 2012. Bermejo-Luque, Lilian. Giving versiality. In Proceedings of the Fifth International Reasons. A Linguistic-Pragmatic Approach to Argu- Natural Language Generation Conference, INLG mentation Theory. Argumentation, 26(2):291–296. ’08, pages 33–41, Stroudsburg, PA, USA. Associ- ation for Computational Linguistics. Aristotle and George Kennedy (translator). 1991. On Marilyn J. Chambliss. 1995. Text cues and strate- Rhetoric: A Theory of Civil Discourse. Oxford Uni- gies successful readers use to construct the gist of versity Press. lengthy written arguments. Reading Research Quar- terly, 30(4):778–807. Bal Krishna Bal and Patrick Saint-Dizier. 2010. To- wards Building Annotated Resources for Analyz- Alexander Conrad, Janyce Wiebe, and Rebecca Hwa. ing Opinions and Argumentation in News Editorials. 2012. Recognizing arguing subjectivity and ar- In Nicoletta Calzolari, Khalid Choukri, Bente Mae- gument tags. In Roser Morante and Caroline gaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Sporleder, editors, Proceedings of the Workshop on Mike Rosner, and Daniel Tapias, editors, Proceed- Extra-Propositional Aspects of Meaning in Compu- ings of the Seventh International Conference on tational Linguistics, pages 80–88, Jeju Island, Ko- Language Resources and Evaluation (LREC’10), rea. Association for Computational Linguistics. pages 1152–1158. European Language Resources Association (ELRA). Jim Crosswhite, John Fox, Chris Reed, Theodore Scalt- sas, and Simone Stumpf. 2004. Computational Jamal Bentahar, Bernard Moulin, and Micheline models of rhetorical argument. In Chris Reed and Bélanger. 2010. A taxonomy of argumentation Timothy J. Norman, editors, Argumentation Ma- models used for knowledge representation. Artifi- chines, volume 9 of Argumentation Library, pages cial Intelligence Review, 33:211–259. 175–209. Springer Netherlands. Aldo de Moor, Lilia Efimova, and Aldo De Moor. Anthony Hunter. 2014. Probabilistic qualification 2004. An Argumentation Analysis of Weblog Con- of attack in abstract argumentation. International versations. In Proceedings of the 9th International Journal of Approximate Reasoning, 55(2):607–638, Working Conference on the Language-Action Per- January. spective on Communication Modelling (LAP 2004), volume 197, pages 1–16. Ralph H Johnson. 2000. Manifest rationality: A prag- matic theory of argument. Routledge. Pierre Dillenbourg and Fabrice Hong. 2008. The me- chanics of CSCL macro scripts. International Jour- Roland Kluge. 2014. Automatic Analysis of Ar- nal of Computer-Supported Collaborative Learning, guments about Controversial Educational Topics 3(1):5–23. in Web Documents, Master Thesis, Ubiquitious Knowledge Processing Lab, TU Darmstadt. Phan Minh Dung. 1995. On the acceptability of ar- Klaus Krippendorff. 2013. Content Analysis: An In- guments and its fundamental role in nonmonotonic troduction to Its Methodology. Thousand Oaks, CA: reasoning, logic programming and n-person games. Sage Publications, 3rd edition. Artificial Intelligence, 77(2):321 – 357. Meredith Larson, M. Annae Britt, and Aaron Larson. Vanessa Wei Feng and Graeme Hirst. 2011. Classi- 2004. Disfluencies in comprehending argumentative fying arguments by scheme. In Proceedings of the texts. Reading Psychology, 25:205–224. 49th Annual Meeting of the Association for Com- putational Linguistics: Human Language Technolo- Ran Levy, Yonatan Bilu, Daniel Hershcovich, Ehud gies - Volume 1, HLT ’11, pages 987–996, Strouds- Aharoni, and Noam Slonim. 2014. Context depen- burg, PA, USA. Association for Computational Lin- dent claim detection. In Proceedings of the 25th In- guistics. ternational Conference on Computational Linguis- tics (COLING 2014), August. To appear. Frank Fischer, Ingo Kollar, Karsten Stegmann, and Christof Wecker. 2013. Toward a script theory of Bing Liu. 2012. Sentiment analysis and opinion min- guidance in computer-supported collaborative learn- ing. Synthesis Lectures on Human Language Tech- ing. Educational Psychologist, 48(1):56–66. nologies, 5(1):1–167. Eirini Florou, Stasinos Konstantopoulos, Antonis Clare Llewellyn, Claire Grover, Jon Oberlander, and Koukourikos, and Pythagoras Karampiperis. 2013. Ewan Klein. 2014. Re-using an Argument Cor- Argument extraction for supporting public policy pus to Aid in the Curation of Social Media Col- formulation. In Proceedings of the 7th Workshop lections. In Proceedings of the Ninth International on Language Technology for Cultural Heritage, So- Conference on Language Resources and Evaluation cial Sciences, and Humanities, pages 49–54, Sofia, (LREC’14), pages 462–468. Bulgaria. ACL. Lilian Bermejo Luque. 2011. Giving Reasons: Austin J. Freeley and David L. Steinberg. 2008. Ar- A Linguistic-Pragmatic Approach to Argumenta- gumentation and Debate. Cengage Learning, Stam- tion Theory, volume 20 of Argumentation Library. ford, CT, USA, 12th edition. Springer Netherlands. Nitin Madnani, Michael Heilman, Joel Tetreault, and James B Freeman. 1991. Dialectics and the Martin Chodorow. 2012. Identifying high-level macrostructure of arguments: A theory of argument organizational elements in argumentative discourse. structure, volume 10 of Trends in Linguistics. De In Proceedings of the 2012 Conference of the North Gruyter. American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Kallirroi Georgila, Ron Artstein, Angela Nazarian, NAACL HLT ’12, pages 20–28, Stroudsburg, PA, Michael Rushforth, David Traum, and Katia Sycara. USA. Association for Computational Linguistics. 2011. An annotation scheme for cross-cultural ar- gumentation and persuasion dialogues. In Proceed- William C. Mann and Sandra A. Thompson. 1987. ings of the SIGDIAL 2011 Conference: the 12th An- Rhetorical structure theory: A theory of text organi- nual Meeting of the Special Interest Group on Dis- zation. Technical report, Information Sciences Insti- course and Dialogue, pages 272–278, Portland, Ore- tute, University of Southern California, Marina del gon. Association for Computational Linguistics. Rey, CA, USA. A. Francisca Snoeck Henkemans. 2000. State-of-the- Raphaël Micheli. 2011. Arguing Without Trying to art: The structure of argumentation. Argumentation, Persuade? Elements for a Non-Persuasive Definition 14(4):447–473. of Argumentation. Argumentation, 26(1):115–126, September. Anthony Hunter. 2013. A probabilistic approach to modelling uncertain logical arguments. Interna- Raquel Mochales and Marie-Francine Moens. 2011. tional Journal of Approximate Reasoning, 54(1):47– Argumentation mining. Artificial Intelligence and 81, January. Law, 19(1):1–22, April. S. Newman and C. Marshall. 1991. Pushing Toulmin Rodrygo LT Santos, Craig Macdonald, Richard MC Too Far: Learning From an Argument Representa- McCreadie, Iadh Ounis, Ian Soboroff, et al. 2012. tion Scheme. Technical report, Xerox Palo Alto Re- Information retrieval on the blogosphere. Founda- search Center 3333 Coyote Hill Road, Palo Alto, CA tions and Trends in Information Retrieval, 6(1):1– 94034. 125. Omid Noroozi, Armin Weinberger, Harm J.a. Biemans, Oliver Scheuer, Frank Loll, Niels Pinkwart, and Martin Mulder, and Mohammad Chizari. 2013. Bruce M. McLaren. 2010. Computer-supported ar- Facilitating argumentative knowledge construction gumentation: A review of the state of the art. In- through a transactive discussion script in CSCL. ternational Journal of Computer-Supported Collab- Computers & Education, 61:59–76, February. orative Learning, 5(1):43–102. Raquel Mochales Palau and Marie-Francine Moens. Oliver Scheuer, BruceM. McLaren, Armin Weinberger, 2009. Argumentation mining: The detection, clas- and Sabine Niebuhr. 2014. Promoting critical, elab- sification and structure of arguments in text. In orative discussions through a collaboration script Proceedings of the 12th international conference on and argument diagrams. Instructional Science, artificial intelligence and law, pages 98–107, New 42(2):127–157. York, NY, USA. ACM. Edward Schiappa and John P. Nordin. 2013. Argumen- Andreas Peldszus and Manfred Stede. 2013a. From tation: Keeping Faith with Reason. Pearson UK, 1st Argument Diagrams to Argumentation Mining in edition. Texts:. International Journal of Cognitive Informat- ics and Natural Intelligence, 7(1):1–31, January. Jodi Schneider and Adam Wyner. 2012. Identifying Consumers’ Arguments in Text. In Diana Maynard, Andreas Peldszus and Manfred Stede. 2013b. Rank- Marieke van Erp, and Brian Davis, editors, Semantic ing the annotators : An agreement study on argu- Web and Information Extraction SWAIE 2012, pages mentation structure. In Proceedings of the 7th Lin- 31–42. guistic Annotation Workshop & Interoperabilty with Discourse, pages 196–204. Association for Compu- Jodi Schneider, B Davis, and Adam Wyner. 2012. Di- tational Linguistics. mensions of argumentation in social media. In Lec- ture Notes in Computer Science, volume 7603, pages Chaim Perelman and Lucie Olbrechts-Tyteca. 1991. 21–25. Springer Berlin Heidelberg. The New Rhetoric. University of Notre Dame Press. Jodi Schneider, Tudor Groza, and Alexandre Passant. 2013. A review of argumentation for the social se- Henry Prakken. 2010. An abstract framework for ar- mantic web. Semantic Web, 4(2):159–218. gumentation with structured arguments. Argument & Computation, 1(2):93–124, June. Alan Sergeant. 2013. Automatic Argumentation Ex- traction. In ESWC 2013, pages 656–660. Springer- Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt- Verlag Berlin Heidelberg. sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. Maria Simosi. 2003. Using Toulmin’s frame- In Nicoletta Calzolari, Khalid Choukri, Bente Mae- work for the analysis of everyday argumentation: gaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Some methodological considerations. Argumenta- and Daniel Tapias, editors, Proceedings of the Sixth tion, 17:185–202. International Conference on Language Resources and Evaluation (LREC’08), pages 1–4. European Walter Sinnott-Armstrong and Robert J. Fogelin. 2009. Language Resources Association (ELRA). Understanding Arguments: An Introduction to In- formal Logic. Cengage Learning, 8 edition. Xian Qian and Yang Liu. 2013. Fast joint compres- sion and summarization via graph cuts. In Proceed- Christian Stab and Iryna Gurevych. 2014. Annotat- ings of the 2013 Conference on Empirical Methods ing argument components and relations in persua- in Natural Language Processing, pages 1492–1502, sive essays. In Proceedings of the 25th International Seattle, Washington, USA, October. Association for Conference on Computational Linguistics (COLING Computational Linguistics. 2014), August. To appear. Chris Reed and Douglas Walton. 2003. Argumenta- Karsten Stegmann, Armin Weinberger, and Frank Fis- tion schemes in argument-as-process and argument- cher. 2007. Facilitating argumentative knowl- as-product. In Proceedings of the conference cele- edge construction with computer-supported collab- brating informal Logic, volume 25. oration scripts. International Journal of Computer- Supported Collaborative Learning, 2(4):421–447. Sara Rosenthal and Kathleen McKeown. 2012. De- tecting Opinionated Claims in Online Discussions. Dragan Stoianovici. 2009. Formal Logic vs. Philo- In 2012 IEEE Sixth International Conference on Se- sophical Argument. Argumentation, 24(1):125–133, mantic Computing, pages 30–37. IEEE, September. January. Stephen Toulmin, Richard Rieke, and Allan Janik. 1984. An Introduction to Reasoning. Macmillan, 2nd edition. Stephen E. Toulmin. 1958. The Uses of Argument. Cambridge University Press. Maria Paz Garcia Villalba and Patrick Saint-Dizier. 2012. Some Facets of Argument Mining for Opin- ion Analysis. In Bart Verheij, Stefan Szeider, and Stefan Woltran, editors, Proceedings of Fourth In- ternational Conference on Computational Models of Argument, COMMA 2012. Artem Vovk. 2013. Discovery and Analysis of Public Opinions on Controversial Topics in the Educatio- nal Domain, Master Thesis, Ubiquitious Knowledge Processing Lab, TU Darmstadt. Henning Wachsmuth, Martin Trenkmann, Benno Stein, and Gregor Engels. 2014a. Modeling Review Argu- mentation for Robust Sentiment Analysis. In Pro- ceedings of the 25th International Conference on Computational Linguistics COLING 2014, page To appear, Dublin, Ireland. Henning Wachsmuth, Martin Trenkmann, Benno Stein, Gregor Engels, and Tsvetomira Palakarska. 2014b. A Review Corpus for Argumentation Analysis. In Alexander Gelbukh, editor, 15th International Con- ference on Intelligent Text Processing and Compu- tational Linguistics (CICLing 14), pages 115–127. Springer. Douglas Walton, Christopher Reed, and Fabrizio Macagno. 2008. Argumentation Schemes. Cam- bridge University Press. Douglas Walton. 2005. Fundamentals of Critical Ar- gumentation. Critical Reasoning and Argumenta- tion. Cambridge University Press, 1 edition. Douglas Walton. 2012. Using Argumentation Schemes for Argument Extraction: A Bottom-Up Method. International Journal of Cognitive Infor- matics and Natural Intelligence, 6(3):33–61. Armin Weinberger and Frank Fischer. 2006. A frame- work to analyze argumentative knowledge construc- tion in computer-supported collaborative learning. Computers & Education, 46(1):71–95, January. Anthony Weston. 2008. A Rulebook for Arguments. Hackett Pub Co., 4 edition. Yun Xie. 2012. Review of Giving Reasons. Informal Logic, 32(4). Cihua Xu and Yicheng Wu. 2014. Metaphors in the perspective of argumentation. Journal of Pragmat- ics, 62:68–76, February.