We describe a pilot study in using argumentation frameworks obtained from an online debate to evaluate positions expressed in the debate. This pilot study aims at exploring the richness of Computational Argumentation methods and techniques for evaluating arguments to reason with the output of Argument Mining. It uses a hand-generated graphical representation of the debate as an intermediate representation from which argumentation frameworks can be extracted, but richer than any existing argumentation framework. The intermediate representation can provide insights for benchmark sets derived from online debates.
1. Introduction
Computational Argumentation (CompArg) is a branch of AI aiming at providing
computational models of argumentation; see [
This paper describes a pilot study in comparing those different frameworks, building
on top of an Argument Mining (ArgMin) exercise. ArgMin is an emerging field aiming to
automatically extract argumentation structures from natural language texts; see [
Despite a large theoretical investigation on the semantic intertranslatability of
frameworks [
The starting point of the experiment was an excerpt from an online for/against debate
taken from www.createdebate.com. The excerpt is given in Table 1.1 We then:
1. mapped the debate onto a hand-annotated graphical representation, identifying
annotations dynamically as demanded by the features of the debate and the
opinions expressed therein; this resulted in a rich annotation scheme with five types
of nodes and six types of edges;
2. mapped the hand-annotated graphical representation onto an Abstract
Argumentation Framework and determined the dialectical acceptability of opinions in the
debate by determining the grounded labelling [
The pilot study raises a number of questions, from both the ArgMin and CompArg perspectives. In particular, for CompArg: • whether our hand-annotated graphical representation can be used as a tool for producing cross-framework benchmarks; • whether the Argumentation Frameworks and tools considered are sufficiently general to serve as a target for reasoning automatically with debates; • whether other existing Argumentation Frameworks and tools may be more suitable for the task at hand.
The paper is organised as follows. Section 2 first presents in full the debate used as
a starting point for our experiment, then continues with the hand-annotated graphical
representation of the debate. We additionally discuss the relationships between
usergenerated annotations to the dialogue and those from expert annotators. Section 3 shows
the mapping onto Abstract Argumentation, Section 4 onto QuAD, and Section 5 onto
AFRA. In Section 6 we conclude.
2. A Graphical Analysis of the Input Debate
We identified statements in the dialogue as well as the relationships between them via the
means of a graph-based representation. Although this representation has been influenced
by other works, notably the Argument Interchange Format (AIF) [
We linked the nodes using six types of edges, each taking one of several different possible values, as follows: • answer-to-question, from one answer node to a question node (directed edges); • standard-explicit, from one standard statement node to another, or to a distractor node, a partial statement node, or an answer node (directed edges), with possible values attack/support/neither; • standard-implicit, from one partial statement to any statement or answer node (directed edges), with possible values attack/support; • meta, from any statement to any statement (directed edges), with possible values attack/support; • node-to-edge, from standard statements to edges (directed), with possible values attack/support; • expansion, amongst any statements (undirected edges).
This analysis resulted in the graph shown in Fig. 1, with Q denoting the question node, and Y and N denoting the answer nodes. Moreover, we label each node in the graph with the identifier of the statements made in the debate (e.g., “#2” in Table 1). Some identifiers in Fig. 1 have a superscript (i.e., 2) to indicate that they actually represent multiple (i.e., two) statements.
For example, we made the following mapping choices in deriving the graph: • #4 is a partial statement as it lacks an explicit conclusion; • #3 is a distractor statement as it is a sort of distraction from the main point of the debate, although still “on topic”, and could be interpreted as intended to promote conflict; edges onto distractor statements are neither attacks nor supports; • #2 and #6 form an expansion statement because #6 fills in some of the details omitted in #2; • #12 criticises #11 at the dialectical (meta) level as well as the content (standard) level; • #8 criticises the attack by #7 on #4.
Comments on the Annotations. Once a debate has started in the system, users may posit arguments in the form of short textual posts as seen in Table 1. However, as shown in the expert annotation presented in Fig. 1, some of these posts contain more than a single argument, which poses the challenge of splitting posts into atomic arguments.
Furthermore, each user of the debate platform is required to explicitly define how her posts correspond and relate to the existing posts that were already presented in the debate. Specifically, the user is required to choose whether her post supports, disputes,
Y + #4 - + or clarifies an existing post. We consider these as non-expert annotations of the relations between the presented posts.
In the debate on which we focus, all posts were annotated as dispute posts by their authors. Namely, all posts were annotated as disputing other posts that had already been presented in the debate. However, in our post factum annotation in Fig. 1, we show that sometimes it is the support and clarification relations that were actually intended.
In some cases, dispute annotations can be interpreted as attack annotations. For example, post #2 was designated as disputing post #1 by its author and indeed argument #2 attacks argument #1 (see Fig. 2). Yet, this is not always the case. For example, post #6 was designated as disputing post #5, however we did not find any significant relation between these posts.
Overall, it is our opinion that having non-expert annotations generated by debaters
can be useful as a rough starting point for expert or automated annotation of the
relation between arguments. Nevertheless, one needs to keep in mind that these non-expert
annotations are biased and imperfect.
3. From the Graphical Analysis to an Abstract Argumentation Framework
We mapped the hand-annotated graphical representation given in Fig. 1 onto an Abstract
Argumentation Framework in order to determine the dialectical acceptability of opinions
in the debate. To this aim, we had to identify the two main components of an Abstract
Argumentation Framework, namely the set of arguments and the set of attacks. In fact,
an Abstract Argumentation Framework [
Since the notion of argument is now overloaded with different meanings, in this and in the following section, argument stands for formal abstract argument, i.e., an element of a mathematical theory of computational argumentation. We refer to the pieces of texts considered in the annotation process as statements.
To compute the dialectical acceptability of opinions in the debate, it was necessary both to include the two possible outcomes of the dialogue—i.e., whether a player would answer Yes (Y) or No (N) to the question—and to link arguments to the statements put forward in the dialogue. In particular, we needed to identify atomic statements—as each player might put forward multiple atomic statements in a single claim. We then aggregated atomic statements into arguments.
The first step is to identify the atomic statements in the dialogue. According to Fig. 1, nodes #5 and #7 contain two statements each: • #5a: It is true that women have the right to choose what they wish to do with their bodies,. . . ; • #5b: . . . but they have absolutely no power to force insurance companies to pay for them. That should be left up to the insurance company, and not the woman.; • #7a: That is a much more logical argument that the user Sitara. Allow me to continue. I agree with you.; • #7b: I do not think Viagra should be covered by insurance though and thus do not believe that contraception should be provided by insurance companies. It should not be covered due to it’s initial purpose, to prevent pregnancy, hence the name “birth control.
The other statements require no further analysis and thus are treated as atomic.
We then aggregated atomic statements into arguments by exploiting both implicit and explicit support links, as well as expansion links. Therefore, #2 and #6 together form the argument #2#6, and similarly #4 together with #5a and #8. However, expansion and support play different roles: an expansion should be interpreted as a single argument that spans multiple atomic statements. Support should rather be seen as a combination of two sub-arguments. We chose to also represent sub-arguments in the Abstract Argumentation Framework, and thus #4#8 should be considered as an additional argument, as well as #5a alone.
To simplify the discussion, we assumed that the arguments Y and N are mutually exclusive, and thus attacking each other. Therefore an implicit or explicit support to a positive answer to the question (respectively a negative answer to the question) is transformed into an attack to the negative answer (respectively the positive answer). Since both #5b and #1 support (cf. Fig. 1) the negative answer to the question, they now both attack the
Y argument. Similarly, #4 supports a positive answer to the question and thus it attacks the N argument.
Moreover, we also considered attacks derived from the attacking links, either explicit or implicit, depicted in Fig. 1. Therefore, the argument #2#6 attacks #1, and similarly the argument #4#8 (and clearly its super-argument comprising #4, #8 and #5a) attacks #1.
Finally, #7b attacks the argument #2#6, while #7a is a self-defeating argument that also undermines #7b.
The analysis depicted in Fig. 1 requires a language much richer than just abstract arguments and attacks. The links marked with a question mark as well as those denoting meta-information are rather complicated to represent in the abstract formalism. In this pilot study we chose to ignore them instead of enforcing a specific semantics that—in our opinion—is still unclear.
Similarly, Fig. 1 includes edges pointing to other edges, potentially implying other
sorts of meta-information. Although there are proposals for encompassing recursive
attacks on Abstract Argumentation Frameworks [
Consequently, #3, #9, and #10 become unconnected arguments. Similarly, #12 attacks #11, but together they are detached from the rest of the graph. Since they cannot have any effect whatsoever on the dialectical acceptability of opinions for this dialogue, in particular they cannot influence the acceptability of arguments Y or N, we chose to filter them out from the final Abstract Argumentation Framework depicted in Fig. 2.
Once the Abstract Argumentation Framework depicted in Fig. 2 is obtained, we can
evaluate the dialectical acceptability of each argument by identifying positions—i.e., sets
of arguments—that together stand against critiques and form a coherent point of view.
In [
Both Y and N are OUT as a combined effect of #5b and the argument comprising #4, #8, and #5a. Although inconclusive, it allows participants in the dialogue to strategically focus their attention. Indeed, let us assume that participants are supporting the Yes answer, then they should focus on arguing against #5b as it is the only argument undermining the Y argument. 4. From the Graphical Analysis to a QuAD Framework
As a next step, we mapped the hand-annotated graphical representation given in Fig. 1
onto a QuAD Framework [
We followed the same approach described in Section 3.1 to identify arguments. Unlike Abstract Argumentation Frameworks, though, QuAD Frameworks allow both attack and support relationships between arguments to be represented explicitly, by assigning “types” to arguments (as pros or cons or answers). Thus, arguments #4#8 and #5a can keep their separate identities in the resulting QuAd Framework, and #5a is no longer “isolated”. Moreover, QuAD Frameworks, when visualised as graphs, are acyclic, with the result that neither the mutual attack between the Y and N arguments nor the selfattack by argument #7a can be represented directly in the resulting QuAD Framework (see Fig. 3, where pros arguments are indicated with ‘+’, cons arguments are indicated as ‘-’ and answer arguments are indicated by a blue light-bulb/mushroom). Note that, since arguments have a single “type” in QuAD Frameworks, if an argument simultaneously attacks one argument and supports another, it (and all its descendants, if any) needs to be duplicated, as in the case of argument #4#8 in Fig. 3.
Note also that converting the original graphical analysis in Fig. 1 to the QuAD Framework in Fig. 3 required simplifications similar to those for converting to Abstract Argumentation (Section 3.3).
In QuAD Frameworks, arguments are assigned a dialectical strength, from which, in particular, a ranking amongst answer arguments is determined. Note that ranking answers amounts to seeing them as “incompatible”; thus the lack of mutual attacks between answers is not a genuine limitation of QuAD Frameworks.
In order to determine the strength of arguments in QuAD Frameworks, they need to
have a base score to start with (seen as an intrinsic strength, prior to any debate about
the arguments). Note that the self-attacking argument #7a in the original Fig. 1 can be
thought of as having a base score of 0, because of the self-attack, amounting to its
computed strength being also 0 (by using, for example, the methods for computing strength
in [
We experiment with two different policies for assigning base scores to the arguments
included in Fig. 3, leading to different rankings of the answers using the method in [
Choosing base scores 0.9 for #2#6 and 0.1 for #4#8 results in No being ranked
higher than Yes (with computed strengths, respectively, of 0.811875 and 0.775).
Thus, the use of base scores in QuAD Frameworks can accommodate information (e.g.,
meta-edges) playing no role in Abstract Argumentation Frameworks. Morever, the use
of dialectical strength instead of dialectical acceptability of arguments can help better
discriminate amongst arguments, but is highly sensitive to the choice of underlying base
score. Indeed, it is clear that the choice of base scores influences the final outcome from
the system.
5. From the Graphical Analysis to an AFRA
5.1. Identification of Arguments and Attacks
As the final step, we mapped the hand-annotated graphical representation (Fig. 1) onto
AFRA [
We followed the same approach described in Section 3.1 to identify arguments.4
Unlike Abstract Argumentation Frameworks, though, AFRA allows attacks to be in turn
the object of other attacks. Therefore, we are now able to represent the attacks from #52
to the attacks between #4#8#5a and #1, (similarly between #4#8 and #1), and between
#4#8#5a and N. The resulting framework is depicted in Fig. 4. Please note that attacks to
supports such as the one from #52 against the support from #4 to Y (Fig. 1) becomes an
attack on the attack from #5a#8#4 against N (Fig. 2, see discussion in Section 3.2).
5.2. Dialectical Strength of Arguments
The semantic notions of AFRA are derived from those that apply for Dung’s Abstract
Argumentation Framework [
Fig. 4 also depicts the restriction of the grounded extension to the set of arguments only. The ⇣ attack, in particular, is pivotal in defending the argument N from the attack it received from #4#8#5a, which is instead effective when using only the Dung’s framework (Section 3.1). 6. Conclusion In this paper we discuss a pilot study for comparing different argumentation frameworks on the basis of the same annotation resulting from an analysis of an online debate. The analysis suggests that the information captured by the original annotation scheme (Fig. 1) is much richer than what can be represented in some of the current state-of-the-art frame3https://www.dbai.tuwien.ac.at/proj/argumentation/systempage/ 4Although AFRA allows to represent more interactions than Dung’s AF, e.g., #11 attacking the support from #4 to Y, we chose to consider the same set of arguments identified in Section 3 in order to facilitate the comparison among the different formalisms. works and tools. We also lack a ground truth (for assessing which position debated is strongest) to assess which tool is better equipped for the task of analysing the specific dialogue we considered in this pilot study.
In the case at hand, increasing the elements of the original annotation schema included in the formal analysis, i.e., the case of AFRA, leads to a less undecided situation w.r.t. the outcome of the dialogue. Moreover, the use of graded semantics as in QuAD allows a much more fine-grained analysis and shows how initial assumptions on the base score of each argument might have a sensible effect on the outcome of the dialogue.
Apart from highlighting differences between Abstract Argumentation, QuAD, and AFRA, this pilot study shows how the proposed annotation scheme (Fig. 1) seems well equipped to represent the complexity of online debates, and that it could be used to produce a set of benchmarks for a variety of frameworks. In fact, most—if not all—of the process described in Sections 3, 4, and 5 can be easily automatised. The foremost issues are determining the arguments (cf. Section 3.1.1), and the relationships among them, especially considering that non-expert annotations are of little help (cf. Section 2).
This pilot study may also help in linking the two research areas of Argument Mining
(ArgMin) and of Computational Argumentation (CompArg). In particular, we showed
that the output of a potential ArgMin process—namely, the graphical analysis in Fig. 1—
may become the input to tools developed in the CompArg community for determining
the dialectical acceptability or strength of opinions in debates. Moreover, this mapping
may provide valuable feedback to debaters, for example, to inform strategies regarding
which aspects to focus on in order to modify the outcome of debates, or to make
decisions based on debates. Concretely, we mapped a naturally-occurring multi-party debate
from a debate website onto a hand-annotated graphical representation, and then: (a) onto
an Abstract Argumentation Framework to determine the dialectical acceptability of
opinions [
Future work will include evaluating other frameworks proposed in CompArg, e.g.,
ADF [
This pilot study raises a number of questions also for the ArgMin community, while at the same time shedding some light on the applicability of the approach taken. For instance it would be interesting to study whether any of the existing NLP methods and tools could be deployed to support the automatic generation of the initial graphical representation and annotation scheme. Moreover, it would be interesting to study other debates to ascertain the generality or otherwise of the annotation scheme we identified. Acknowledgements The input debate was suggested by Adam Wyner and Ivan Habernal, as part of the Dagstuhl seminar on “Natural Language Argumentation: Mining, Processing, and Reasoning over Textual Arguments”. We also thank Ivan Habernal for helpful feedback during the preliminary graphical analysis of the debate described in Section 2.