We formulate a problem of an assessment of argumentation validity based on rhetorical analysis of text. Argumentation structure can be detected in text in the form of discourse trees extended with edge labels for communicative actions. Extracted argumentation structure is represented as a defeasible logic program and is subject to dialectical analysis to establish the validity of the arguments for the main claim being communicated. We evaluate the accuracy of argument mining and then argument validation as well as an overall performance of an end-to-end argumentation system.
In this study we focus on validating claims of human agent expressed in text. In non-trivial cases, claim validation relies on an analysis of arguments. When domain knowledge is available and formalized, truthfulness of a claim can be validated directly. However, in most text analysis environments such knowledge is unavailable and other implicit means need to come into play, such as writing style and writing logic, in particular, used argumentation patterns. In this study we employ the discourse analysis in our end-to-end argument validation system for texts and explore which discourse features can be leveraged for argumentation validity analysis.
When an author attempts to provide an argument for something, a number of
argumentation patterns can be employed. The basic points of argumentation are
reflected in the rhetorical structure of text where an argument is present
Most of the modern techniques treat computational argumentation as specific
discourse structures and perform detection of arguments of various sorts in text, such as
classifying a text paragraph as argumentative or non-argumentative
To address this shortcoming, in this project, we build an end-to-end argumentation system, augmenting an argument extraction from text with its logical analysis. To represent the linguistic features of text, we use the following sources: 1) Rhetoric relations between the parts of the sentences, obtained as a discourse tree (DT). Discourse trees encode rhetorical relations such as Cause, Contrast, Condition, Attribution which are correlated with argumentation attack relation. 2) Speech acts and communicative actions, obtained as verbs from the VerbNet resource.
To assess the logical validity of an extracted argument, we apply the Defeasible Logic
Program
We start with a political domain and give an example of conflicting agents providing their interpretation of certain events. These agents provide argumentation for their claims; we will observe how formed rhetoric structures correlate with their argumentation patterns. We focus on the Malaysia Airlines Flight 17 example with the agents exchanging arguments: Dutch investigators, The Investigative Committee of the Russian Federation, and the self-proclaimed Donetsk People's Republic. It is a controversial conflict where each agent attempts to blame its opponent. To sound more convincing, each agent postulates its claim in a way to attack the claims of its opponents, matching their argumentation styles and trying to defeat their claims. “Dutch accident investigators say that strong evidence points to pro-Russian rebels as being fully responsible for shooting down plane. The report indicates where the missile was fired from and identifies who was in control of the territory and pins the downing of MH17 on the pro-Russian rebels.” (Fig. 1a).
“The Investigative Committee of the Russian Federation believes that the plane was hit by a missile, which could not be produced in Russia. The committee cites an investigation that established the type of the missile and disagrees with Dutch accident investigators.”(Fig. 1b)
“Rebels, the self-proclaimed Donetsk People's Republic, deny that they controlled the territory from which the missile was allegedly fired. They confirm that it became possible only after three months after the tragedy to say if rebels controlled one or another town and the claim of Dutch accident investigators is flawed”(Fig. 1c).
To show the structure of arguments one needs to merge discourse relations with information from speech acts. We need to know the discourse structure of interactions between agents, and what kinds of interactions they are. For argument identification, we do not need to know the domain of interaction (here, aviation), the subjects of these interaction, what are the entities, but we need to take into account mental, domain-independent relations between them. We accomplish this by introducing the concept of Communicative Discourse Tree (CDT).
CDT is a DT with labels for edges that are the VerbNet expressions for verbs
(which are communicative actions,
Notice that in the CDTs for three paragraphs expressing the views of conflicting
parties (Figs 1a, 2b and 2c), communicative actions with their subjects contain the
main claims of the respective party, and the DTs without these labels contain
information on how these claims are logically packaged. To summarize, a typical
CDT for a text with argumentation includes rhetoric relations other than Elaboration
and Join, and a substantial number of communicative actions. However, these rules
are complex enough so that the structure of CDT matters and tree-specific learning is
required
Argumentation analysis needs a systematic approach to learn associated discourse structures. The features of CDTs could be represented in a numerical space so that argumentation detection can be conducted; however, structural information on DTs would not be leveraged. Also, features of argumentation can potentially be measured in terms of maximal common sub-DTs, but such nearest neighbor learning is computationally intensive and too sensitive to errors in DT construction. Therefore, a CDT-kernel learning approach is selected which applies a support vector machine (SVM) learning to the feature space of all sub-CDTs of the CDT for a given text where an argument is being detected.
Tree Kernel (TK) learning for strings, parse trees and parse thickets is a wellestablished research area nowadays. The CD-TK counts the number of common subtrees as the discourse similarity measure between two DTs. In this study, we extend the TK definition for the CDT, augmenting DT kernel by the information on CAs. TK-based approaches are not very sensitive to errors in parsing (syntactic and rhetoric) because erroneous sub-trees are mostly random and will unlikely be common among different elements of a training set.
A CDT can be represented by a vector V of integer counts of each sub-tree type (without taking into account its ancestors):
V( ) = (# 1, … , # , … , #
). Given two tree segments CDT1 and CDT2 , the tree kernel function is defined:
(CDT1, CDT2) = <V(CDT1), V(CDT2) > = Σi V(CDT1)[i], V(CDT1)[i] =
Σn1Σn2 Σi Ii(n1)* Ii(n2), where 1∈ 1 , n2∈ 2 and 1 and N2 are the sets of all
nodes in CDT1 and CDT2 , respectively; i( ) is the indicator function:
i( ) = {1 iff a subtree of type occurs with a root at a node; 0 otherwise}. Further
details for using TK for paragraph-level and discourse analysis are available in
Only the arcs of the same type of rhetoric relations (presentation relation, such as antithesis, subject matter relation, such as condition, and multinuclear relation, such as List) can be matched when computing common sub-trees. We use N for a nucleus or situations presented by this nucleus, and S for a satellite or situations presented by this satellite. Situations are propositions, completed actions or actions in progress, and communicative actions and states (including beliefs, desires, approve, explain, reconcile and others). Hence we have the following expression for RSTbased generalization ‘^’ for two texts text1 and text2 :
text1 ^ text2 = ∪i,j (rstRelation1i, (…,…) ^ rstRelation2j (…,…)), where I ∈ (RST relations in text1), j ∈ (RST relations in text2). Further, for a pair of RST relations their generalization looks as follows: rstRelation1(N1, S1) ^ rstRelation2 (N2, S2) = (rstRelation1^ rstRelation2 )( N1^N2, S1^S2).
We define CA as a function of the form verb (agent, subject, cause), where verb
characterizes some type of interaction between involved agents (e.g., explain,
confirm, remind, disagree, deny, etc.), subject refers to the information transmitted or
object described, and cause refers to the motivation or explanation for the subject. To
handle meaning of words expressing the subjects of CAs, we apply word2vec models
We combined Stanford NLP parsing, coreferences, entity extraction, DT
construction
https://github.com/bgalitsky/relevance-based-on-parse-trees.
To convince an addressee, a message needs to include an argument and its structure
needs to be valid. Once an argumentation structure extracted from text is represented
via CDT, we need to verify that the main point (target claim) communicated by the
author is not logically attacked by her other claims. To assess the validity of the
argumentation, a Defeasible Logic Programming (DeLP) approach is selected. It is an
argumentative framework based on logic programming
A DeLP is a set of facts, strict rules Π of the form (A:-B), and a set of defeasible rules Δ of the form A-<B, whose intended meaning is “if B is the case, then usually A is also the case”. Let P=(Π, Δ) be a DeLP program and L a ground literal. Let us now build an example of a DeLP for legal reasoning about facts extracted from text (Fig. 2). A judge hears an eviction case and wants to make a judgment on whether rent was provably paid (deposited) or not (denoted as rent_receipt). An input is a text where a defendant is expressing his point. Underlined words form the clause in DeLP, and the other expressions formed the facts.
The complaint is as follows: The landlord contacted me, the tenant, and the rent was requested. However, I refused the rent since I demanded repair to be done. I reminded the landlord about necessary repairs, but the landlord issued the three-day notice confirming that the rent was overdue. Regretfully, the property still stayed unrepaired.
Defeasible Rules Prepared In Advance rent_receipt -< rent_deposit_transaction. rent_deposit_transaction -< contact_tenant. ┐rent_deposit_transaction -<contact_tenant,
three_days_notice_is_issued. ┐rent_deposit_transaction -< rent_is_overdue. ┐repair_is_done -< rent_refused, repair_is_done. repair_is_done -< rent_is_requested. ┐rent_deposit_transaction -<
tenant_short_on_money, repair_is_done. ┐repair_is_done -< repair_is_requested. ┐repair_is_done -<rent_is_requested. ┐repair_is_requested -< stay_unrepaired. ┐repair_is_done -< stay_unrepaired.
? - rent_receipt
repair_is_done -< rent_refused.
contact_tenant. rent_is_requested. rent_refused. remind_about_repair. three_days_notice_is_issued.
rent_ is_overdue. stay_unrepaired.
We outline the algorithm for validation of a domain-specific claim for arguments extracted from text: 1. Build a DT from input text; 2. Attach communicative actions to its edges to form CDT; 3. Detect argumentation from this CDT using SVM learning; Stop if not detected. 4. Extract subjects of communicative actions attached to CDT and add to ‘Facts’ section (Fig. 3 on the left); 5. Extract the arguments for rhetoric relation contrast and communicative actions of the class disagree and add to ‘Clauses Extracted FromText’ section of Fig. 2; 6. Add a domain-specific section to DeLP; 7. Having the DeLP formed, build a dialectical tree and assess the claim (Fig. 3 on the right).
We use the Tweety (2017) system for DeLP implementation
This dataset is used for both argument detection (first step) and argument validity (second step) tasks. For argument detection, we attempt to identify if a given paragraph of text has contains an argument, in a domain-independent manner. For argument validation, in the second step, if we detected an argument in the first step, we try to validate it having the domain-ontology built in a given vertical domain such as landlord-tenant dispute. If an argument has not been detected in the first step, we have nothing to validate.
For the negative dataset, only for the argument detection task, we used Wikipedia,
factual news sources, and also the component of (Lee, 2001) dataset that includes
such sections of the corpus as: instructions for how to use software, hardware,
presentations of a news article in an objective, independent manner, and others.
Further details on the data set are available in
Each row indicates a method used to detect a presence of argumentation in a
paragraph. We start with baseline methods, based on keywords and their frequencies
(second and third row on the top, Table 1). Second column shows precision (P), third
– recall and the fourth – F1 measure. Frequently, a coordinated pair of communicative
actions (so that at least one has a negative sentiment polarity related to an opponent)
is a hint that logical argumentation is present. This naïve approach is outperformed by
the top performing TK learning CDT approach by 29%. SVM TK of CDT
outperforms SVM TK for RST+CA and RST + full parse trees
In our validity assessment, we focus on target features (claims) related to how a given complaint needs to be handled, such as compensation_required, proceed_with_eviction, rent_receipt and others. System decision is determined by whether claim is validated or not: if it is validated, then the decision support system demands compensation, and if not validated, decides that compensation should not be demanded (for the compensation_required claim). 18.7 24.8 Four and above specific relations or communicative 86.3 actions R Validity assessment results are shown in Table 2. These results are computed together for detection and validation steps. In the first and second rows, we show the results of the simplest complaint with a single rhetoric relation such as contrast with a single CA indicating an extracted argumentation attack relation respectively. In the third row we assess complaints of average complexity, and in the bottom row, the most complex, longer complaints in terms of their CDTs. The third column shows detection accuracy for invalid argumentation in complaints in a stand-alone argument validation system. Finally, the fourth column shows the accuracy of the integrated argumentation extraction and validation system.
In these results recall is low because in the majority of cases the invalidity of claims is due to factors other than being self-defeated. Precision is relatively high since if a logical flaw in an argument is established, most likely the whole claim is invalid because other factors besides argumentation (such as false facts) contribute as well. As complexity of a complaint and its discourse tree grows, F1 first improves since more logical terms are available and then goes back down as there is a higher chance of a reasoning error due to a noisier input.
For decision support systems, it is important to maintain a low false positive rate. It is acceptable to miss invalid complaints, but for a detected invalid complaint, confidence should be rather high. If a human agent is recommended to look at a given complaint as invalid, her expectations should be met most of the time. Although F1measure of the overall argument detection and validation system is low in comparison with modern recognition systems, it is still believed to be usable as a component of a CRM decision-support system.
We observed that by relying on discourse tree data, one can reliably detect patterns of logical argumentation. Communicative discourse trees become a source of information to form a defeasible logic program to validate an argumentation structure. Although the performance of the former being about 80% is significantly above that of the latter (29%), the overall pipeline can be useful for detecting cases of invalid argumentation, which is important in decision support for CRM.
To the best of our knowledge, this is the first study building a whole argument validity pipeline in the industrial setting. Hence although the overall detection rate for invalid argument is fairly low, there is no existing system to compare this performance against. All detected cases with invalid claims are very valuable for a business or a legal case. In this paper we attempted to combine the best of both worlds, argumentation mining from text and reasoning about the extracted argument. Whereas applications of either technology are limited, the whole argumentation validation system is expected to find a broad range of applications. In this work, we focused on a very specific legal area such as customer complaints, but it is easy to see a decision support system employing the proposed argumentation pipeline in other domains of CRM.
The article was prepared within the framework of the Basic Research Program at the National Research University Higher School of Economics (HSE) and supported within the framework of a subsidy by the Russian Academic Excellence Project '5-100'.