Argumentation theory is the interdisciplinary study of how conclusions can be supported or undermined by premises through logical reasoning. It is an essential part of our daily human reasoning, interaction and communication with one another. Argumentation is often used in decision-making problems, resolving conflicts of opinion through negotiation and deliberation processes, and influencing the thoughts of others through persuasion. The complex and dynamic nature of argumentation is studied in various disciplines.
It stands as an important domain of Artificial Intelligence (AI), especially in knowledge representation and reasoning. Indeed, the use of AI allows for the development of computational models for the exchange of arguments among various agents, facilitating the derivation of valid conclusions from incomplete, inconsistent or conflicting information. Within AI, computational argumentation focuses on creating formal models that support decision-making through the construction and evaluation of arguments. Many works in computational arguments are based on Dung's seminal work [1] where he proposed the abstract argumentation framework, which is a general tool for studying relations between the arguments such as attacks (and, more recently, supports [2,3] or sets of attacking arguments [4,5]).
On a given abstract argumentation framework, one can apply many different semantics to compute sets of acceptable arguments (conflict-free, admissible, . . . ) [1,6] or rank arguments from the strongest to the weakest (using ranking-based semantics) [7]. Those semantics are based on many intuitions or normative principles, defined in the literature, which characterise their behaviour. While some work has studied the link between basic intuitions from the argumentation theory (e.g., reinstatement) and human reasoning [8,9,10], it is still unknown if most of the principles are intuitive or used by humans.
A recent work by Vesic, Yun, and Teovanovic [11] studied some principles for ranking-based semantics (anonymity, void precedence, maximality, and independence) and showed that the graphical representation influences the normativity of participant responses. Namely, there was a higher level of compliance with the principles when the participants were shown the graphical representation of the arguments. Moreover, they showed that anonymity between tasks principle was not followed by the participants. In other words, for two different sets of textual arguments with same structure (the graphs are isomorphic), the participants differed in their evaluation of the arguments, although this effect was dampened for participants who were shown the graphical representation. This demonstrates that both the content of the arguments and the graphical representation have a significant effect on the evaluation task.
The latter result opens up several important questions that were not studied before. In what way does the content of the arguments (e.g., language, bias-provoking content) affect the principle compliance? Can graphical representations consistently serve as a prescriptive tool for enhancing rationality in individuals?
In this exploratory paper, we will focus on a restricted setting. We will study whether participants comply with the simple reinstatement principle and how this is affected by the naturalness bias and the graphical representation.
On the one hand, the simple reinstatement principle states that in a configuration with three arguments 𝐴, 𝐵, and 𝐶 such that 𝐶 attacks 𝐵 and 𝐵 attacks 𝐴 (see Figure 1), roughly speaking "both 𝐶 and 𝐴 must be stronger than 𝐵". On the other hand, the naturalness bias (also known as appeal-to-nature) is a cognitive bias where people have a preference for things perceived as "natural" over those perceived as "unnatural" or artificial. This bias can influence attitudes and decisions in various domains, including food, medicine, or lifestyle choices, among others.
To experimentally check the impact of the content (e.g. bias-provoking text) and the graph, we set up an experiment featuring three hypotheses. Firstly, we expected participants to behave more rationally in the absence of the naturalness bias, regardless of graph presence. Next, we predicted that the graph would make participants more rational (as shown in previous studies) and we also expected that the effect of the graph would be stronger in the presence of naturalness bias than in its absence.
This paper is structured as follows. In Section 2, we recall the computational argumentation background on extension and ranking-based semantics needed to motivate the principles. In Section 3, we formulate our aims and hypothesis with respect to the principles, the graph, and the naturalness bias. In Section 4, we describe the design of our study, the sampling plan, and the instruments. Lastly, we present our results in Section 5 and conclude in Section 6.
We start this section by recalling the definition of an argumentation framework as defined by Dung in his seminal paper [1] (as an example, see Figure 1).
). An argumentation framework is a pair ℱ = (𝒜, 𝒞), where 𝒜 is a finite set of arguments and 𝒞 ⊆ 𝒜 × 𝒜 is a set of binary attacks between arguments. The set of attackers of 𝑎 ∈ 𝒜 is 𝐴𝑡𝑡(𝑎) = {𝑏 ∈ 𝒜 | (𝑏, 𝑎) ∈ 𝒞}.
In the following sub-sections, we explain what is considered as rational with respect to extension-based semantics and ranking-based semantics.
Given an argumentation framework ℱ = (𝒜, 𝒞), we say that a set 𝑆 ⊆ 𝒜 is conflict-free iff there is no 𝑥, 𝑦 ∈ 𝑆 such that (𝑥, 𝑦) ∈ 𝒞. A set 𝑆 ⊆ 𝒜 defends 𝑎 ∈ 𝒜 iff for every 𝑏 ∈ 𝒜 such that (𝑏, 𝑎) ∈ 𝒞, there exists 𝑠 ∈ 𝑆 such that (𝑠, 𝑏) ∈ 𝒞. A set 𝑆 ⊆ 𝒜 is admissible iff it is conflict-free and defends every argument in 𝑆. A set 𝑆 is preferred iff it is a maximal (for set inclusion) admissible set.
The preferred semantics is the function that computes all the preferred sets from an argumentation framework. There are many other acceptability semantics, which we do not introduce in this paper as the details are not of practical interest for the rest of our cognitive study1 .
Example 1. In Figure 1, there are three admissible sets, which are ∅, {𝐶}, and {𝐶, 𝐴}, but there is only one preferred set which is {𝐶, 𝐴}.
In Example 1, the preferred extension shows that both 𝐴 and 𝐶 are better than 𝐵 (as 𝐴 and 𝐶 are in the preferred set but 𝐵 is not). In this case, we can consider that 𝐴 and 𝐶 should be equally acceptable. Thus, we define the extension-based simple reinstatement, inspired by [12,13], as follows.
Definition 2 (Extension-based simple reinstatement). In a configuration with three arguments 𝐴, 𝐵, and 𝐶 such that 𝐶 attacks 𝐵 and 𝐵 attacks 𝐴, the extension-based simple reinstatement is satisfied iff 𝐶 is equally strong as 𝐴, 𝐶 is stronger than 𝐵 and 𝐴 is stronger than 𝐵.
While extension can extract sets of acceptable arguments, an orthogonal approach consists in ranking-arguments from the stronger to the weakest one.
Another family of argumentation semantics is called ranking-based. They do not calculate extensions; instead, they rank the arguments from the strongest to the weakest one. In this paper, we focus on a particular family of ranking-based semantics, called gradual semantics, which associates to each argument, a number from 0 (weakest) to 1 (strongest). Those scores naturally induce an order on the set of arguments.
For the rest of our study, we do not need particular semantics. However, just for the sake of illustration, we introduce one semantics, called h-categorizer [14]. We refer the reader to the work of [15] for a comparative study on ranking-based semantics.
The ℎ-categorizer semantics is a ranking-based semantics such that for every ℱ = (𝒜, 𝒞), for every argument 𝑎 ∈ 𝒜 we have
It was shown [16] that ℎ-categorizer is well-defined for every argumentation framework (i.e., the semantics converge to unique values for each argumentation framework). To illustrate, the values assigned by ℎ-categorizer to the arguments from Figure 1
, and 𝐷𝑒𝑔 ℎ ℱ (𝐶) = 2 3 . Many principles were defined for characterising the behaviour of ranking-based semantics. For the purpose of this paper, we do not need to recall them all. We just formalize the rankingbased simple reinstatement and the void precedence principles.
In a configuration with three arguments 𝐴, 𝐵, and 𝐶 such that 𝐶 attacks 𝐵 and 𝐵 attacks 𝐴, the ranking-based simple reinstatement is satisfied iff 𝐶 is stronger than 𝐴 and 𝐴 is stronger than 𝐵.
We consider that both extension-based and ranking-based are valid and rational points of view. In the rest of the paper, we allow the rational agent to act based on either of them. Hence, we will say that the reinstatement is satisfied iff either ranking-based simple reinstatement or extension-based simple reinstatement is satisfied. Definition 6 (Simple reinstatement). In a configuration with three arguments 𝐴, 𝐵, and 𝐶 such that 𝐶 attacks 𝐵 and 𝐵 attacks 𝐴, the simple reinstatement is satisfied iff 𝐶 is equal or stronger than 𝐴 and 𝐴 is stronger than 𝐵.
Void precedence states that a non-attacked argument is stronger than an attacked one.
). We say that a ranking-based semantics 𝜎 satisfies void precedence iff for every argumentation graph ℱ = (𝒜, 𝒞) and 𝑎, 𝑏 ∈ 𝒜 such that 𝐴𝑡𝑡(𝑎) = ∅ and
Now that we have defined what is considered rational in abstract argumentation, we formally describe our aims and hypotheses.
In the current study, we examined if people comply with normative principles more readily when presented with arguments with neutral content in comparison to arguments that contain biasprovoking content. To do so, we explored people's response patterns in a simple reinstatement scenario. As an example of bias-provoking content, we relied on the naturalness bias i.e., the tendency to prefer natural things over artificial ones, all other things being equal. Furthermore, we aimed to replicate previous findings that graphical representation of arguments leads to higher compliance with normative principles i.e., the presence of an argumentation graph facilitates rational inference.
We expected participants to comply more with normative principles in the neutral condition than in the naturalness bias condition, regardless of graph presence (H1). Moreover, we expected participants to comply more with normative principles in the graph condition relative to the no-graph condition, regardless of content (H2). Finally, we expected an interaction between the two factors, i.e., that the effect of the graph will be stronger in the naturalness bias condition relative to the neutral condition (H3).
On an exploratory level, we also checked whether people tend to follow extension-based or ranking-based argumentation semantics more, depending on the rating of different arguments (the non-attacked one and the reinstated one). The extension-based semantics predict that both arguments will have the same acceptability level, whereas most of the ranking-based semantics predict that the non-attacked one will be stronger.
The materials and data to reproduce the findings of this study can be found on the project OSF page: https://osf.io/kce7q/.
We employed a 2x2 design with two independent factors: content (neutral vs. bias-provoking) and graph (present vs. absent), resulting in four experimental groups of participants.
The sample size was based on an a priori power analysis. Since we considered several possible effects, we opted to base the power analysis on the smallest effect size of interest -i.e., a small interaction effect size (Cohen's 𝜔 = .10). For power .80, the apriori analysis indicated we would need 𝑛 = 197 per group (788 in total) to detect this effect size. For power .90, for the same effect size, we would need 𝑛 = 263 per group (1052 in total). For further details on the power analysis, see preregistration -https://aspredicted.org/9T9_NG4.
As per preregistration, we included an attention check ("Please choose Completely agree to indicate you are paying attention"), and all participants who failed the attention check were automatically excluded from the sample and did not count towards the final sample size.
The final sample size consisted of a total of 𝑁 = 1048 UK and US Prolific participants. All respondents were compensated for their participation using the standard Prolific 9 £/hour rate. Most participants were from the UK (85.6%), and 61.8% were female, with an average age of 𝑀 𝑎𝑔𝑒 = 41.45 (𝑆𝐷𝑎𝑔𝑒 = 13.71) 2 . Due to a technical error in the questionnaire setup, there was an unequal distribution between the four experimental groups (116 for neutral content without a graph, 398 for neutral content with a graph, 400 for bias-provoking content without a graph, and 133 for bias-provoking content with a graph).
The questionnaire was administered online via Prolific and hosted on the SoSci Survey platform [17]. The participants were randomly allocated first to either the neutral or biasedcontent condition, and then to either the graph or no-graph condition. The participants then saw the arguments (and the graph, depending on the condition) and rated them, after which they answered an additional question on their strategy when rating the arguments. The planned duration of the study was four minutes, but on average, participants completed it in 100 seconds (𝑆𝐷 = 43.3𝑠).
Consider the arguments below.
• Argument A: It doesn't matter if you drink natural or lab-produced water.
• Argument B: Natural things are healthier than lab-produced ones, so you should drink natural water. • Argument C: They are chemically identical, so they have the same impact on your health. Consider the arguments below.
• Argument A: It's not going to rain today.
• Argument B: But look at all these clouds, it will rain.
• Argument C: Those clouds are Cumulus, hence they do not produce rain.
As mentioned in Section 4.2, we had four experimental groups of participants (neutral vs. bias-provoking / graph vs. no-graph).
Each participant was first shown three arguments in their textual forms. Depending on their group (neutral vs. bias-provoking), the textual content was different (see Figures 2 and 3). We can observe that in both cases, argument B attacks argument A which consists only of a claim, argument C attacks argument B since C attacks B' s premise.
The textual content of the arguments was inspired by structured argumentation which allows for constructing arguments from a knowledge base using a formal language. In this context, arguments are characterized as structured because they reveal their underlying premises and conclusions clearly, the connection between them is formally established, and attacks among them are formally defined. There exist several approaches in structured argumentation for formalizing arguments such as the ASPIC + framework [18], deductive argumentation [19] and the assumption-based framework [20]. We represented arguments in their simplest forms which consists of a premise that supports a conclusion (claim) using an inference rule. In some cases, the premise can be hidden (e.g., in argument A), hence the argument consists only of a claim. Based on the internal structure of the arguments, we can have different types of attack relations between the arguments. For example, an argument can attack another argument' s premise; this type of attack is called undermining attack (e.g., attack from 𝐶 to 𝐵). Another type of attack, called rebutting attack, is an attack on the conclusion of an argument (e.g., attack from 𝐵 to 𝐴).
Only for the participants in the graph groups, we displayed the graphical visualisation of the arguments and attacks (see Figure 1).
Lastly, the participants were asked to assess the strength of each argument using a 5-point Likert scale from 1 (Weak) to 5 (Strong), with 3 being "Neutral".
Overall, participants complied with simple reinstatement to a low degree (see Figure 4), with the percentage of participants who followed these principles in either of the groups ranging from 10.3% to 16.5%. We conducted a logistic regression analysis to investigate how well the two factors (content and graph) and their interaction predict compliance with simple reinstatement. The goodnessof-fit of the logistic regression model was assessed using a likelihood ratio test, which yielded a non-significant chi-square statistic, 𝜒 2 (3) = 2.96, 𝑝 = .398. Additionally, the 𝑅 2 value indicated that the model accounts for only 0.3% of the variability in compliance with simple reinstatement. This suggests that the model with argument content and graphical representation did not provide a significantly better fit to the data than a model without them, as shown in Table 1.
Regarding void precedence, the participants in our sample followed this principle to a larger degree (see Figure 5) -the lowest percentage in either of the groups was 44.4% while the highest was 69.8%.
To explore if the two factors and their interaction predict compliance with void precedence, we conducted another logistic regression analysis. The goodness-of-fit of the model was assessed using a likelihood ratio test, which yielded a significant chi-square statistic, 𝜒 .001, and 𝑅 2 value indicated that the model accounts for 5% of the variability in compliance with void precedence. As shown in Table 2, content influenced rational behavior, while the graph and interaction terms were non-significant. Presenting arguments with naturalness bias-provoking content decreased compliance with void precedence by 71% (𝑒𝑥𝑝(𝐵) = 0.29, 95%𝐶𝐼 = [0.11, 0.77]).
Further analyses revealed that the effect of content is due to higher ratings of argument 𝐵 (𝑡(1057) = 2.66, 𝑝 = .008, 𝑑 = 0.16) and especially lower ratings of argument 𝐶 (𝑡(1057) = 7.69, 𝑝 < .001, 𝑑 = 0.47) in the group presented with bias-provoking content, regardless of the presence of graph (see Table 3). Ratings for argument 𝐴 did not differ significantly between bias-provoking and neutral content groups (𝑝 = .28).
Mean ratings (with standard errors) for three arguments in four experimental conditions.
We now summarize and discuss all of our hypotheses and results. First, we expected participants to comply more with normative principles in the neutral condition than in the naturalness bias condition, regardless of graph presence (H1). For this hypothesis, our results are mixed. The experiment showed that the change in content (i.e. neutral vs biased) did not impact the degree of compliance with the simple reinstatement principle. However, the content influenced the degree to which participants complied with the void precedence principle. Namely, in the presence of arguments incorporating naturalness bias, the ratings of the non-attacked argument 𝐶 were very low while the ratings of the non-defended argument 𝐵 were high (probably due to the naturalness bias increasing participants' ratings of 𝐵).
Second, we expected participants to comply more with normative principles in the graph condition relative to the no-graph condition, regardless of content (H2). This hypothesis was disconfirmed -our experiment showed that the graph did not provide a significantly better fit to the data. This contrasts the recent results [11] where the presence of the graphical representation was shown to enhance compliance with principles for graded argumentation semantics. Note, however, that the participants in that study went through a tutorial and several tasks before evaluating the strength of the arguments. This hints that a tutorial may be necessary to teach participants the meaning of the graphical representation and how it is linked to argument ratings.
Third, we expected an interaction between the two factors, i.e., that the effect of the graph will be stronger in the naturalness bias condition relative to the neutral condition (H3). Contrary to our hypothesis, the experiment showed that the interaction effect was not significant in our model.
In our experiments, compliance with the reinstatement principle was very low (less than 20%). Our results differ from the existing empirical work on reinstatement [21]. However, the task that participants were presented with is also different. While we study the individual satisfaction of the principle considering all arguments simultaneously, they compared the average ratings of argument 𝐴 (before 𝐵 is introduced, after 𝐵 is introduced, and then when 𝐶 is introduced).
Note that according to the average scores of arguments in our experiment, 𝐶 is stronger than 𝐵 and 𝐵 is stronger than 𝐴. One of the possible explanations for this is that the argument 𝐴 is just a statement and contains no justification. Argument 𝐵 has a general justification (e.g., there are clouds so it will rain) and might be seen as more acceptable for this reason. Finally the argument 𝐶 has the strongest hypothesis that fully justifies its conclusion and might seem difficult to attack (e.g., the cumulus clouds do not produce rain).
Another reason why compliance with simple reinstatement might be low is that participants' intuition about the notion of defense diverges significantly from that of the researchers. More precisely, it might be that the participants misunderstood the meaning of attacks and associated attacks solely with a reduction in argument 𝐵 strength, ignoring the increase in strength brought by reinstatement for argument 𝐴.
Importantly, several factors limit the generalizability of our findings. Firstly, due to a programming error, the four groups in our study were not equal in size, somewhat reducing statistical power. Next, since participants only responded to one presented scenario, it is difficult to say how they would respond to other scenarios, be it simple reinstatement scenarios with different content or more complex scenarios. Finally, we presented all arguments at the same time, which may have an impact on participants' perception of attacks and, consequently, their ratings of arguments' strength.
In future work, we plan to introduce the arguments step-by-step (as it was done by Rahwan et al. [21]) while monitoring the ratings of all arguments (instead of only 𝐴). We believe that this will give us a better insight on the factors at play behind the participants' ratings of the arguments.
This work benefited from the support of the project SATTORI in the framework of the Pavle Savic -Hubert Curien funding scheme, and from the support of the project AGGREEY ANR-22-CE23-0005 of the French National Research Agency (ANR).
(S. Vesic) https://lira.f.bg.ac.rs/marija-petrovic-en/ (M. Petrović); https://lira.f.bg.ac.rs/phd-predrag-teovanovic/ (P. Teovanović); https://lira.f.bg.ac.rs/phd-danka-puric/ (D. Purić); https://liris.cnrs.fr/page-membre/bruno-yun (B. Yun); https://www.cril.univ-artois.fr/~alanaissy/ (C. A. Anaissy); https://www.cril.fr/~konieczny/ (S. Konieczny); https://www.cril.univ-artois.fr/~vesic/ (S.