While it is unrealistic to assume users of online debate platforms to read and interpret all the arguments available, it is important to understand how positions will emerge on the basis of a fraction of those arguments. What arguments exactly will be accessed by users depend on assumptions on the platform design or on the readers' behaviours. Young et al. were the rst to explore this question and report results in the context of an underlying extension-based semantics. We undertake a similar study in the context of gradual semantics, using a more comprehensive set of metrics, testing a larger number of behaviours, and come to di erent conclusions. We show in particular that a reading behaviour balancing supports and attacks provides interesting results.
The Internet allows people to express their opinions by participating in online discussions, sometimes involving many users and comments. These debates can provide a wealth of relevant information for users curious about the subjects under discussion. The main obstacle that such users may encounter is the sheer volume of comments exchanged, making it di cult for a human being to read all the arguments of a debate and to assess the relevance of each point of view in a reasonable amount of time. Indeed, under time constraints, the reading behaviour, i.e. the way users read the arguments of the debate, a ects the subset of arguments they are exposed to, and consequently, their assessment of the acceptability or strength of those arguments. In this paper, we explore several “natural” reading behaviours and experimentally investigate how these behaviours in uence the user’s perception of a debate.
Online debates often take the form of an initial topic, to which many participants have responded with comments, comments which themselves have responses, and so on. These debates can therefore be represented in the form of graphs, more speci cally trees, where the nodes are the comments and the edges are the attack or support relationships between two comments. In short, these debates are well suited to the use of argumentation theory. In this paper, we focus on the Kialo platform1 and we note that the number of arguments can be high. It is unlikely that an average user will have the time to read all the arguments.
The question investigated here is to study how the order in which the arguments are
considered can a ect the view readers have about the acceptability (as de ned in the extension-based
semantics introduced by Dung [
We take inspiration from this study, but we depart from the methodology used by Young et al. for the following reasons. First, the attening approach they use might lead to information loss. Let us illustrate this on an example with x being the central proposition attacked by one argument a and supported by two arguments b and c. The algorithm used by Young et al. to atten this graph results here in ignoring the supports (they will be deleted). In our work, we propose a completely distinct way to take into account the supports (namely by directly using bipolar gradual semantics, and not a attening which is inappropriate in the case of online debates) to take them into account properly.
Another issue with the work of Young et al. is that they use grounded semantics. As previously
emphasized, extension-based semantics su er from a lack of smoothness [
Recently, behavioural studies have become more popular as a way to assess formal
argumentation frameworks [
The remainder of this paper is as follows. In Section 2 we present Kialo and the data. Section 3 provides the background on gradual semantics. We detail the reading behaviours and the metrics used in Section 4 and Section 5. The experimental results are reported in Section 6.
The full code together with a notebook allowing to explore many other parameters, semantics and reading behaviours is available on our Git4 repository.
Kialo is an online platform for structured debates. When a user initiates a debate, she puts an initial claim which stands for the central question. Users can then add claims that respond directly to the question if it is closed-ended, or to one of the existing alternatives if the question is open-ended, or to another claim already in the discussion. Claims are classi ed as “PRO” and “CON” depending on whether they support or attack the claim to which they are attached. The moderation of the platform allows moderators to rephrase claims, to move or merge claims or to delete claims that do not ful ll the requirements of the platform. Finally, users can vote on the claims by choosing a score from 0 to 4 (integers only).
Example 1 (A debate of Kialo). One of Kialo’s biggest debates (in terms of number of claims) is “The Ethics of Eating Animals: Is Eating Meat Wrong?”. The central proposition debated is “Humans should stop eating animal meat.” An example of a “PRO” claim for this proposition is “Eating meat, in the majority of cases, involves the cruel and immoral treatment of animals.” An example of a “CON” claim is “The taste of meat is delicious and brings many people pleasure in a manner that vegetarian food cannot fully imitate.” This claim received 185 votes, distributed as follows: 66 users voted for 0, 31 users for 1, 26 users for 2, 17 users for 3, and 45 users for 4.
Kialo debates are “clean” thanks to the moderation carried out by users: there are no insults, claims must be concise, be based on logic and facts, and present a single point related to the theme of the debate. Claims can therefore be considered as arguments. Furthermore, claims must not duplicate other claims. Finally, Kialo has been designed in such a way that the most general claims are the closest to the central question, allowing users to dive into more details as they move down the graph. 4https://gitlab.com/jthieyre/reassessing-the-impact-of-reading-behaviour-in-online-debates-under-the-lens-ofgradual-semantics
Firstly, we used Kialo’s API (Application Programming Interface) to retrieve the list of public debate IDs. These IDs are then used to ask the API for the data relating to the debates, i.e. the ID, the text, the votes, the date of creation and of last modi cation of the claim, and the ID of its author; the ID, the type of relation (attack or support), the date of creation and of last modi cation of the relation between this claim and another, and the ID of its author. Of this large amount of data, we have only kept the IDs, dates and votes of the claims, and the dates and relationships between claims with their types, to protect the privacy of users as much as possible. In total, we recovered data from 2,959 public debates. We analysed the data collected and observed that a large proportion of the data in each debate corresponded to “archived” data, in other words, outdated claims or relationships. Using a depth- rst search (DFS) algorithm, we were able to isolate the claims that corresponded to those displayed by the Kialo front-end from those that are archived. The outcome is 377,182 claims spread over 2,959 cleaned debates.
The distribution of claims per debate is illustrated in Figure 1a and shows that only a small number of debates have a signi cant number of claims. This is consistent with the distribution of maximum depths depicted in Figure 1c, which shows that half of the debate graphs have a maximum depth of 5 or less. All of Kialo’s claims received a total of 590,261 votes. The distribution of votes by claim is depicted in Figure 1b and shows a very uneven distribution. The closed-question debates were relatively balanced, as shown in Figures 1d and 1e: Figure 1d gives the di erence between the number of PRO claims and the number of CON claims, as a proportion of the number of claims in the debate. For example, 25% of debates have more CON claims than PRO claims, so that the di erence between the two is at least 9% of the total number of claims in these debates. Figure 1e depicted the distribution of the di erence between the number of claims which contribute to support the central question and the number of claims which help to attack it, as a proportion of the number of claims in the debate. In the same way as illustrated in Figure 1d, we can see that 10% of debates have a greater number of claims supporting the central proposition than attacking it, so that the di erence between the two is at least 33% of the number of claims in these debates. Figure 1f illustrates the distribution of incoming degrees, and shows that at least half of the claims have neither an attacker nor a supporter.
As stated in Section 1, the Kialo debates are well suited to the use of argumentation theory.
Abstract argumentation theory was rst introduced in [
3,539
978
s
e
t
o
v
f
o
r
e
b
m
u
N 0 1 4 8 16 58
225
709
o
it
a
r
e
c
n
a
l
a
B
h
t
p
e
d
x
M 1 3 5 7 9 11
a
e
e
r
g
e
d
n
I
frameworks [
De nition 1. A Bipolar Argumentation Framework (BAF) is a triple (A, R , R+) such that: • A is a • R ✓
nite set of arguments, A ⇥ A is an acyclic binary relation on A describing attack relations, • R+ ✓ A ⇥ A is an acyclic binary relation on A describing support relations. R (a) hence denotes the set of direct attackers of an argument a and R+(a) its set of direct supporters. In the following, we will generalize the equations by using the notation R⇤ which can designate either R+ or R .
The rst semantics we studied, DF-QuAD [
Y (1 b2 R⇤ (a) sf (b)) where sf is the function used to calculate the nal score (see below), andf ⇤ stands for f + (respectively f ) if we consider the set of supporters (respectively attackers). Note that R⇤ stands for R+ or R (but not both at the same time). By convention, if R⇤ (a) = ; , Q (1 sf (b)) = 1. b2 R⇤ (a) The function sf computing the nal score of an argument is de ned as: De nition 3 (Euler-based semantics). The function sf computing the nal score of an argument is de ned as:
sf : A ! [0, 1[ sf (a) =1 1
si(a)2 1 + si(a)eE(a) where E(a) =
X b2 R+(a) sf (b)
X b2 R (a) sf (b)
Finally, the last semantics we considered, is the Quadratic Energy Model (QuEM) [
h : R ! [
max(e, 0)2 1 + max(e, 0)2 The function sf used to calculate the nal score of an argument is de ned as:
All studied semantics need to de ne an initial score si for each argument. In this paper, we
chose to leverage the votes on arguments provided by Kialo debates to initialize the scores.
As mentioned before, on the Kialo platform, each participant can vote for an argument by
choosing an integer score in [0 · · · 4]. For each argument a, the votes are summarized by a
vector v(a) 2 R5 of size 5 indicating how many participants voted for each score. We de ne
the initial score of each argument a by performing a normalized weighted average as proposed
by Yang et al. [
v(a)·w Pk2 v(a) k if v(a) = [0, 0, 0, 0, 0] otherwise
As explained by Young et al., we can interpret the ve vote values as representing a “rational” reader’s belief in the truth of a claim, with 0 for those thinking it is false and 4 for those believing it is true. Then, we use weights w = [0, 0.25, 0.5, 0.75, 1] for score initialization, where each step re ects an equal increase in con dence.
The semantics presented in the previous section propose di erent ways to update the score
of an argument. Before studying di erent reading behaviours, we wanted to emphasize how
these di erent semantics a ect the nal scores obtained in debates. As discussed before, the
axiomatic analysis of gradual semantics may provide precious insights. For instance, the notion
of open-mindedness has been proposed by Potyka [
DF-QuAD
Semantics
the initial score of all arguments in the Kialo dataset as a function of the semantics used, when
these di erences are di erent from 0. In fact, the majority of arguments keep the same score
value. This is due to the fact that they have neither attacker nor supporter, as can be seen in
Figure 1f. Besides that, Figure 2 shows that the Euler-based semantics modi es the score less
than the other semantics, but is biased towards positive variation, while the other semantics
show a symmetric behaviour. This asymmetric behaviour of the Euler-based semantics is in
line with axiomatic analysis [
Secondly, to get a better intuition regarding the impact of debates’ depth on the scores of arguments for each semantics, we illustrate how they vary on single-path attack-only debates of increasing lengths. As depicted in Table 1, we iteratively derive from an argumentation graph G1 containing a single argument (a), 5 other graphs (G2 to G6) organized as a line. A node corresponds to an argument and arrows represent attack relations between two arguments. From each graph Gi, we build Gi+1 by adding an attacking argument to the last added node. Table 1 reports the variation of the nal score of argument a for the di erent graphs. Note that we assume that each argument has an initial score of 0.5. Graph G1 is not mentioned since the score of a remains the same (0.5) for each semantics (a has no attack nor support). For all studied semantics, the variation of the nal score of a decreases as the depth of the graph i.e. the length of the attack path, increases. Due to the way we extend the graphs, the sign of the variation is alternating: the last argument attacks a or attacks an attack on a. Nonetheless, we can see that the relation between the depth and impact of arguments is very di erent among semantics. We observe that the Euler-based semantics leads to almost no variation in the score of argument a already from depth 3. On the other hand, the largest variations are observed for DF-QuAD.
Debates can involve a large number of arguments leading to an information overload, so that
users are rarely able to read all the arguments. In this section, we introduce various debate
reading behaviours that seem natural for humans. In particular, we assume that the reader starts
with the central question being debated and the arguments directly related to it. As reported by
[
Since Kialo debates are represented as trees, each behaviour is based on a tree traversal method combined with (or without) the exploitation of information about the arguments (number of votes, chronological order, PRO/CON classi cation).
We considered the following methods for traversing the debate tree: • Depth First Search (DFS), • Breadth First Search (BFS), • Hybrid Traversal (HT ): based on a DFS but forcing to explore all the child nodes of an argument before exploring in depth the sub-tree of one of these children, • No traversal (NT ), which does not exploit the structure of the tree.
We combine these traversal methods with a ranking method for ordering the nodes (i.e. arguments) within the same level: • Chronological Order (CO): oldest to newest (based on the timestamps of the arguments), • Descending likes (DL): arguments are ranked based on their initial score si, • Descending likes with PRO-CON Diversity (PCD): the PRO argument with the highest like value is ranked rst, then the CON argument with the highest like value, then the PRO argument with the second highest like value, and so on.
We cannot present all the combinations of traversing and ranking methods but we focus on
the following reading behaviours5:
1. Behaviour NT+CO: chronological order with no traversal,
2. Behaviour DFS+CO: depth rst search + chronological order,
3. Behaviour DFS+DL: depth rst search + descending likes (introduced as “likes” policy by
[
Example 2. We illustrate the reading order of the arguments obtained by the di erent reading behaviours on the graph below. Argument a is the source node. Numerical values correspond to the like value of each node. Arguments are assumed to be labeled in chronological order (a is the oldest one and k is the newest). Solid edges represent attacks and dashed ones represent supports. PRO (resp. CON) arguments are depicted in green (resp. red).
The question is now how to compare the di erent reading behaviours presented in the previous
section. Recall that the objective is to compare the situation where someone would have entirely
read the debate, with the situation where someone has partially read the debate. In [
RM SE = utuv X (sf (a, rb(F, n)) n
sf (a, rb(F, N ))2
a2 rb(F,n)
• the Kendall Tau (KT) (normalized) distance between the rankings of the arguments induced
by the scores, as was done in [
KT = 2|{(i, j) : i < j, B(i, j) _ C(i, j)}|
n(n 1) where B(i, j) = (Sn(i) < Sn(j)) ^ (SN (i) > SN (j)) and C(i, j) = (Sn(i) > Sn(j)) ^ (SN (i) < SN (j)). KT counts the number of pairwise disagreements between two ranking lists. Thus, B(i, j) checks whether the i-th argument is ranked before the j-th in Sn, and after in SN . Similarly, C(i, j) checks whether the i-th argument is ranked after the j-th in Sn, and before in SN . • the Jaccard coe cient (JC) between the sets of accepted arguments, where arguments are considered as accepted in our setting when their score reaches a prede ned threshold (set to 0.5 in this paper):
J C = |Dn T DN | |Dn S DN | where
Dn = {a 2 rb(F, n)|sf (a, rb(F, n)) > 0.5}
DN = {a 2 rb(F, n)|sf (a, rb(F, N )) > 0.5} JC measures similarity between Dn, the set of accepted arguments when we consider the partially read graph, and DN , the set of accepted arguments when we consider the entire graph and when we keep only the n rst arguments given by rb. Indeed, when a reader has seen n arguments, she cannot expect to have seen more than n accepted arguments.
By doing this, JC is not biased when n is less than the nal number of accepted arguments. Example 3. Let s = h0.8, 0.2, 0.7, 0.6, 0.1i and s0 = h0.6, 0.4, 0.7, 0.1, 0.2i be two score vectors for some arguments a, b, c, d, e. The RSME between these two vectors s and s0 of argument scores is 0.261. The induced rankings are respectively a c d b e and c a b e d, yielding a KT distance of 0.3. Finally, the set of accepted arguments of s is {a, c, d} while it is {a, c} for s0, hence the JC is 2/3.
Furthermore, we will parametrize our metrics in such a way that the focus of the study can be
put on speci c arguments of the debates. For instance, top-k focused metrics are only concerned
with the central question, together with arguments up to depth k. Technically, it su ces to
weight the di erent metrics introduced above. As mentioned in Section 4, because stakeholders
of participatory platforms “are mainly looking for assessments and arguments related to their
proposal”[
0 0.15
Observation 1. Whatever the metric or semantics used, behaviour BFS+PCD followed by MDHT+PCD outperform all the others.
Observation 2. Under BFS+PCD, the true ranking can be retrieved for all semantics when reading at least 17% in the best case and in the worst case 50% of the graph.
Observation 3. In trend, values of the metrics decrease for all the reading behaviours except for NT+CO under the DF-QuAD semantics for which the error rst increases and then decreases.
We have investigated the di erences in errors between the behaviours and highlighted 3 main explanations: 1. Traversal direction: adding an argument in breadth rather than in depth leads to a stronger reduction of the median error. This can be observed when comparing DFS+PCD, HT+PCD, MDHT+PCD and BFS+PCD. HT has a behaviour close to a DFS, while MDHT is close to a BFS. This coincides with Figure 3 where we can see that the results of these behaviours can be ranked (from best to worst) as: BFS+PCD MDHT+PCD HT+PCD DFS+PCD. Finally, when we analyse the type of traversal that NT+CO produces at the start of reading, we realise that it is similar to a DFS.
2. Respecting the nal attack and support balance: let a be one of the arguments considered by our metrics, i.e. with a weight di erent from 0. When we analyse the factors in uencing the evaluation of the nal score of a, we see that the balance between the attackers and the supporters of a plays an important role. Indeed, the nal score depends on the number of attackers and supporters, and their respective scores. Therefore, any behaviour that leads to a balance, when n arguments have been read, that is too di erent from the one when all the arguments have been read, will produce a greater di erence between the nal scores of a, and therefore a greater error. In practice, this result can be observed when we compare DFS+CO, DFS+DL and DFS+PCD: the ranking method PCD produces a more faithful balance (by alternating between PRO and CON arguments and placing the strongest arguments rst) throughout the reading, and so it performs better. For instance, at the maximum RMSE for DF-QuAD semantics, i.e. when a reader has read between 11% and 12% of the graph, 62% of the arguments added by NT+CO are PRO arguments compared with 54% for BFS+PCD. However, we did not observe any signi cant di erence between the average scores for the arguments added by these two behaviours for this example.
3. The semantics used: although the relative performance is preserved from one semantics to another, we can see that the error values di er. In addition, we can see that the DF-QuAD semantics is more sensitive to the two error explanation factors that we described above, than the Euler-based and QuEM semantics.
One important nding is thus that NT+CO produces an unbalanced and in-depth reading of the debate. From a behavioural point of view, this can be explained by the fact that on Kialo, the rst arguments are added by the creator of the debate who is more likely to be biased by her own opinion on the issue and therefore add, at the rst stage of the debate, arguments that support the central question, and also arguments that support her other arguments. Oldest arguments are hence more likely to support the central question.
Even though our approach di ers from [
In this paper we performed a new assessment of the impact of the behaviour of users when
reading through online debates, following the methodology proposed in [
This work has been supported by ANR (french National Research Agency) project AGGREEY (ANR-22-CE23-0005).