Mining Italian Short Argumentative Texts Ivan Namor Pietro Totis Università degli studi di Padova KU Leuven ivan.namor@studenti.unipd.it pietro.totis@cs.kuleuven.be Samuele Garda Manfred Stede Potsdam Universität Applied Computational Linguistics garda@uni-potsdam.de University of Potsdam, Germany stede@uni-potsdam.de Abstract This task can be decomposed into several subtasks: segmentation of the text in elementary We present the first model for argumentation mining discourse units (EDUs), identification of for Italian short argumentative texts. We adapted to argumentative discourse units (ADUs), Italian the software developed by (Peldszus and classification of argumentative discourse units, Stede, 2015) and built a suitable corpus of Italian "microtexts" by semi-automatically translating the identification of the relations between original English corpus. Our results are comparable argumentative discourse units and classification to those of (Peldszus and Stede, 2015), which proves of these relations. The argumentation structure that their model is applicable successfully to of a text can be presented as a tree structure, languages other than English and German.1 with a node for each argumentative discourse unit and different edges between nodes 1. Introduction representing the different types of relations. There are many simple models that recognize In recent years, argumentation mining (Lippi automatically the argumentation structure of a and Torroni, 2016) has become an area of big micro-text. interest in the field of natural language Our starting point is the model by (Peldszus processing. Argumentation mining seeks to and Stede, 2015), who developed a software to automatically recognize the structure of the automatically mine the argumentation structure argumentation in a text by identifying, of short texts for English and German. In this classifying and connecting the central claim of a paper we perform argumentation mining on a text, supporting premises, possible objections corpus of short Italian argumentative texts. To and counter-objection. Argumentation mining transfer the approach to Italian, we assembled a has many possible applications in very different suitable corpus by semi-automatically translating fields. Recognizing automatically the the original German corpus and we adapted the argumentative structure of a text can be useful as features used by the software, by assembling a an extension of opinion mining, in retrieval of list of Italian connectives necessary to fulfill the court decisions from databases (Palau and task. Moens, 2011), in automatic document summarization (Teufel and Moens, 2002), in Our results are slightly lower than the ones analysis of scientific papers as in biomedical text for German and English, but they demonstrate mining (Teufel, 2010; Liakata et al., 2012) in that the model can be considered valid also for essay scoring, and more. Italian. Besides, a major contribution of this paper is the free availability of the annotated Italian corpus.2 1 Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License 2 Attribution 4.0 International (CC BY 4.0). https://github.com/PietroTotis/evidencegraph 2. Related works corpus of 113 short texts and a total of 576 ADUs (Peldszus and Stede 2015). The corpus is (Peldszus and Stede, 2016) collected the arg- made by 90 short texts collected in a controlled microtext corpus, a freely available parallel text generation experiment and by 23 written corpus of 112 texts with 576 argumentative directly by Andreas Peldszus, mainly in order to ADUs (argumentative discourse units). It differs teach and test the probands of the experiment. from other web-text corpora collected for The texts are short but at the same time argumentation mining purposes, such as the “complete” and the underlying argumentation Internet Argument Corpus (Abbott et al., 2016) structure is relatively clear. The probands were and the ABCD corpus (Rosenthal and asked to first gather a list with the pros and cons McKeown, 2015), because the texts have been of the trigger question, then take stance for one collected in a controlled text generation side and argue for it in a short argumentative experiment. text, which had to be at least five segments long (Peldszus and Stede, 2013) proposed an with each segment argumentatively relevant, had annotation scheme, which has been based on to contain at least one objection and finally had Freeman’s theory of argumentation structures to be understandable without having its trigger (Freeman, 2011) and has been used to annotate question as a headline. All of the microtexts the arg-microtext corpus. This annotation were originally written in German and have been scheme has been proven to yield reliable successively professionally translated in English. structure in annotation and classification experiments (Peldszus and Stede, 2015; Potash 3.2 Annotation scheme et al., 2017). One of a few similar approaches is that of The annotation scheme we used for our (Stab and Gurevych, 2017), who introduced a corpus is the same used for the original corpus, corpus of persuasive essays annotated with developed by Peldszus and Stede on the basis of argumentation structures related to the arg- different ideas from literature about microtexts and presented a similar approach for argumentation structures (Peldszus and Stede, parsing argumentation structures. 2013). Two important steps in the development of a theory of argumentation are Toulmin’s An example of argumentation mining for influential analysis of argument (Toulmin, 1958) Italian is presented in (Basile et al., 2016), where and Grewendorf’s dialog-oriented diagram the researchers tested their method on a corpus method (Grewendorf, 1980). of user comments to online newspaper articles. The annotation scheme used for the arg- microtexts corpus is based mainly on Freeman’s 3. Original Corpus theories, which integrate Toulmin’s ideas into The interest in argumentation-oriented corpora the argument diagraming techniques of the of monologue text is rising, but most of the informal logic tradition (Freeman, 1991, 2011). present data are not suitable for these operations. The central claim of Freeman’s theory is that the For this reason it is necessary to have well- different ways in which premises and formed and controlled corpora of short conclusions combine to form larger complexes, argumentative texts. can be modeled as a hypothetical dialectical exchange between a proponent and an opponent. 3.1 Data collection An argument is a non-empty set of premises supporting some conclusion. The argumentation In order to provide a corpus of Italian short structure of a text is defined as a graph with the argumentative texts we translated to Italian the text segments as nodes. Each node is associated arg-microtexts corpus, a freely available3 parallel with a specific argumentative role: the “proponent”, who presents and supports a central claim, and the “opponent”, who questions the proponent’s claims. Argumentative 3 https://github.com/peldszus/arg-microtexts relations are represented by the edges between foreseeable, a lot of words were translated with the nodes and have a specific argumentative the most common Italian translation, but not the function, which can be “support” or “attack”. most appropriate. All the microtexts have been Support relations can be of different types: basic, thereby post edited in order to look as they were linked, multiple, serial and the example relation. generated directly in Italian. Connectives have a Attack relations can target both premises or fundamental role in the identification of conclusions and can be of two different types: function, role and attachments of a sentence. We they are a “rebut” if they target another node or therefore dedicated special attention to this “undercut” if they target an edge between two aspect; in the automatic translation, many nodes. different original forms converged to the most common connective in the target language. For example, almost all the connectives expressing similarity were translated with “e” (“and”) and most of the connectives expressing contrast were translated with “ma” (“but”). In order to have a more realistic corpus we tried to use a more various set of connectives, comparable to the set used in the original corpus. 4.2 Projection annotations The annotated graph structures are stored in Figure 1: An example text (micro_b037) and its reduced XML format. The main advantage of translating argumentation structure: texts segments, proponent and the arg-microtexts corpus was that it was not opponent nodes (rounds and boxes), supporting, attacking necessary to make the annotations from scratch. and undercutting relations (arrow-head, circle-head and As expected, there was a one by one square-head). correspondence between original sentences in German and the translations in Italian. In order 4. Translation to have Italian annotated graph structures it was only necessary to automatically substitute every The choice of translating into Italian the arg- German sentence in the XML file with the microtexts corpus, likewise it was previously corresponding Italian sentence. In case a done for English, is motivated by the controlled sentence contained more ADUs, it has been setting of the experiment. The translation process divided manually. had two phases. In the first phase we automatically translated the entire corpus using 5. Software DeepL Translator4, a free and multilingual translation service. In the second phase, all the The code for computing the tree predictions translations have been manually checked and, if have been taken over from the work of Peldszus needed, post-edited. and Stede (Peldszus and Stede, 2015). 4.1 Post-editing 5.1 Original model Some corrections were necessary in almost In order to recognize the argumentation every microtext: from a syntactic point of view structure, the model considers not only the the translator respected most of the probability of attachment of each segment pair, dependencies, losing however accuracy with but also the probabilities of role, function and of increasingly complex syntactic structures As . being the central claim. In order to do so it is necessary to predict probabilities for each argumentative unit on different levels: 4 https://www.deepl.com/translator attachment, central claim, role (proponent or 6. Results opponent) and function (supporting or attacking). The metrics to evaluate our adaptation are The first step is to build a fully connected Macro F1 and Micro F1 for each sub-task: multigraph that connects every segment pair central claim, role, function and attachment with as many edges as the function types. In detection. The results are reported in Table 1. order to get central claim, role, function and Compared to the results obtained in the attachment probabilities, the model uses experiment with the English and the German different classifiers and then jointly combines corpus (Peldszus and Stede, 2015), the results these probabilities in a single edge score, defined for Italian are slightly lower. The results are as the weighted sum of the level specific edge almost the same for central claim and attachment scores, on which it is possible to apply a MST detection and lower in function and role (minimum spanning tree) algorithm (Chu and classification. The most significant drop of the Liu, 1965; Edmonds, 1967). F1 scoring regards the task of function The result represents the best global classification. Nonetheless, the overall attachment structure for the text. This model performances are sufficient to confirm the outperformed other baseline and simpler models validity of the model for Italian. The smaller size when tested on the German and English parallel of the Italian model provided by spaCy might corpus (Peldszus and Stede, 2015). explain the gap in performance with the other two languages. 5.2 Adaptation to Italian cc ro fu at In order to run the original experiments on Macro F1 0.813 0.724 0.413 0.690 the Italian corpus, we adapted the sections of the Micro F1 0.883 0.811 0.593 0.792 code related to the corpus and the NLP tools. Table 1: Results for Italian The latter represents the major divergence from the original setting, since it entailed upgrading cc ro fu at the spaCy package, along with its language Macro F1 0.825 0.765 0.431 0.706 models. This also involved upgrading other Micro F1 0.888 0.841 0.618 0.796 packages and porting the whole project to Table 2: Results for English Python 3.x, but these were minor modifications cc ro fu at that should not have a meaningful impact on the Macro F1 0.817 0.750 0.671 0.663 performances. Table 3: Results for English (Peldszus and Stede, 2015) A language-specific set of connectives is essential for the classification of the relations 6.1 Error analysis between ADUs. For this purpose, we used LiCo5, a lexicon of Italian connectives (Feltracco et al. We investigated the reason for the lower 2016). The connectives are stored in XML performances in the task of function format, each entry contains: classification: Figure 2 and 3 show an example - Part type (phrasal or single). of misclassification. The prediction for the - Syntactic type (preposition, adverb, microtext mistakenly detects an attacking and an coordinating conjunction, subordinating undercutting relation in place of two supporting conjunction). relations. Wrong function classification of some argumentative unit can be found in most of the - Relation type (as cause, concession, outputs of the corpus. contrast, purpose). Another common error is the wrong - An example of use in a sentence. attachment: Figure 3 and 4 present an interesting error for this task. In place of an “attach to first” structure, which is typical of the English style of 5 http://connective-lex.info/ essay writing and can be used as baseline, our model has attached all the argumentative units to the preceding segment, which is also a typical baseline in discourse parsing (Muller et al., 2012). We investigated the role of connectives in the attachment prediction and ran the same experiment on a less specific list of connectives, i.e. with more general relation types. With this simplified version of the connectives, the classifier achieved lower results in all the tasks. This suggests that specificity is not the reason behind these errors and at the same time proves Figure 3: micro_b033 expected output the central role of the connectives in the recognition of an argumentation structure. 7. Conclusion We presented, to our knowledge, the first model that transfers on an Italian microtexts corpus the approach developed by (Peldszus and Stede, 2015). We ran the experiment on an Italian corpus obtained by translating the original German one and by designing a suitable list of connectives. We adapted the code by changing the sections related to the corpus and the NLP tools. Our results are comparable to those of Peldszus and Stede, which proves that their model is applicable successfully to languages other than English and German. Figure 4: micro_b031 wrong output Figure 2: micro_b033 wrong output Figure 5: micro_b031 expected output References [Peldszus and Stede2013] Andreas Peldszus and Manfred Stede. 2013. From argument diagrams to argumentation mining in texts: a survey. Universität Potsdam. [Abbott et al.2016] Rob Abbott, Brian Ecker, Pranav Anand, and MarilynWalker. 2016. Internet Argument Corpus 2.0: An SQL schema for dialogic social media [Peldszus and Stede2015] Andreas Peldszus and Manfred and the corpora to go with it. In Proc. Language Stede. 2015. Joint prediction in MST-style discourse Resources and Evaluation, pages 4445–4452. parsing for argumentation mining. In Proceedings of the 2015 Conference on Empirical Methods in Natural [Basile et al.2016] Pierpaolo Basile, Valerio Basile, Elena Language Processing (EMNLP), 938–948. Lisbon, Cabrio and Serena Villata. 2016. Argument Mining on Portugal. Italian News Blogs. [Peldszus and Stede2016] Andreas Peldszus and Manfred [Chu and Liu1965] Yoeng-Jin Chu and Tseng-Hong Liu. Stede. 2016. An annotated corpus of argumentative 1965. On the shortest arborescence of a directed graph. microtexts. In First European Conference on In Science Sinica, 14:1396–1400. Argumentation: Argumentation and Reasoned Action, Portugal, Lisbon. [Edmonds1967] Jack Edmonds. 1967. Optimum Branchings. In Journal of Research of the National [Potash et al.2017] Peter Potash, Alexey Romanov and Bureau of Standards, 71B:233–240. Anna Rumshisky. 2017. Here’s my point: Joint pointer architecture for argument mining. In Proceedings of the [Feltracco, Jezek, Magnini and Stede2016] Anna Feltracco, 2017 Conference on Empirical Methods in Natural Elisabetta Jezek, Bernardo Magnini and Manfred Stede. Language Processing, pages 1375–1384, Copenhagen, 2016. Lico: a lexicon of italian connectives. In Denmark. Association for Computational Linguistics. Proceedings of the 3rd Italian Conference on Computational Linguistics (CLiC-it). Napoli, Italy. [Rosenthal and McKeown2015] Sara Rosenthal and Kathy McKeown. 2015. I couldn’t agree more: The role of [Freeman1991] James B. Freeman. 1991. Dialectics and the conversational structure in agreement and disagreement Macrostructure of Argument. Foris, Berlin. detection in online discussions. In Proceedings of the 16th Annual Meeting of the Special Interest Group on [Freeman2011] James B. Freeman. 2011. Argument Discourse and Dialogue, pages 168–177, Prague, Structure: Representation and Theory. Argumentation Czech Republic. Association for Computational Library (18). Springer. Linguistics. [Grewendorf1980] Günther Grewendorf. 1980. [Stab and Gurevych2016] Christian Stab and Iryna Argumentation in der Sprachwissenschaft. In Zeitschrift Gurevych. 2016. Parsing Argumentation Structures in f¨ur Literaturwissenschaft und Linguistik, Persuasive Essays. In Computational Linguistics. 43. 10(38/39):129–150. 10.1162/COLI_a_00295. [Liakata et al.2012] Maria Liakata, Shyamasree Saha, [Teufel and Moens2002] Simone Teufel and Marc Moens. Simon Dobnik, Colin R. Batchelor and Dietrich 2002. Summarizing scientific articles: Experiments Rebholz-Schuhmann. 2012. Automatic recognition of with relevance and rhetorical status. Computational conceptualization zones in scientific articles and two Linguistics, 28(4):409–445, December. life science applications. Bioinformatics, 28(7):991– 1000. [Teufel2010] Simone Teufel. 2010.The Structure of Scientific Articles: Applications to Citation Indexing [Lippi and Torroni2016] Marco Lippi and Paolo Torroni. and Summarization. In CSLI Studies in Computational 2016. Argumentation Mining: State of the Art and Linguistics. CSLI Publications. Emerging Trends. In ACM Transactions on Internet Technology. 16. 1-25. 10.1145/2850417. [Toulmin1958] Stephen Toulmin. 1958. The Uses of Argument. Cambridge University Press, Cambridge. [Muller et al.2012] Philippe Muller, Stergos Afantenos, Pascal Denis, and Nicholas Asher. 2012. Constrained decoding for text-level discourse parsing. In Proceedings of COLING 2012, pages 1883–1900, Mumbai, India, December. The COLING 2012 Organizing Committee. [Palau and Moens2011] Raquel Mochales Palau and Marie- Francine Moens. 2011. Argumentation mining. In Artificial Intelligence and Law, 19(1):1–22.