argumentation provides a formal framework for modeling and analyzing argumentative reasoning processes. As the eld progresses, the need for benchmarks to evaluate and compare di erent algorithmic approaches becomes increasingly important. However, the process of generating suitable benchmarks for abstract argumentation is not without its challenges. This paper aims to explore the key challenges encountered in benchmark generation for abstract argumentation. In particular, we address the task of skeptical acceptability w.r.t. preferred semantics and describe a benchmark generator designed for this speci c problem.
As a central aspect of human communication, the concept of argumentation has been adopted in
the area of Arti cial Intelligence in various forms. The principle of abstract argumentation [
One notable initiative in advancing the evaluation of argumentation systems is the
International Competition on Computational Models of Argumentation (ICCMA)1. ICCMA serves
as a platform for researchers and practitioners to showcase their systems and compare their
performance on a common set of benchmarks. While ICCMA has signi cantly contributed to
the evaluation of argumentation systems, the process of generating benchmarks for abstract
argumentation remains a perpetual task. For instance, w.r.t. a set of ICCMA’17 benchmarks, it
was recently pointed out that a majority of arguments that are skeptically accepted under
preferred semantics (a task which is 2P -complete [
An (abstract) argumentation framework (AF) [
Further, a set E Arg is con ict-free if E EF+ = . If a set E Arg is con ict-free and
each a E is defended by E, we call E admissible (ad). We call sets of jointly acceptable
arguments extensions, which can be de ned under various semantics. The classical semantics,
following the seminal work by Dung [
Arg is grounded (gr) i E is complete and -minimal.
Arg is preferred (pr) i E is complete and -maximal.
Arg is stable (st) i it is complete and E EF+ = Arg.
In addition, we de ne a set E Arg to be an ideal (id) extension [
Typical problems in the eld of abstract argumentation involve the enumeration of one extension (or all extensions), or deciding whether a given argument is included in one extension (or all extensions) w.r.t. a given semantics. Let = {co, gr, pr, st, id}. An argument is credulously accepted w.r.t. if it is included in at least one extension, and it is skeptically accepted if it is included in all extensions. We denote the computational problem of credulous acceptability regarding a semantics as DC , and the problem of skeptical acceptability regarding as DS .
In the following, we provide a brief overview of existing benchmarks (which were used in past editions of ICCMA) and subsequently discuss challenges that arise in the development of new benchmark generation techniques.
E.
Existing benchmarks for abstract argumentation can be roughly categorized into the following
groups:
• Random graphs. Some benchmark instances, e.g., those provided by AFBenchGen [
When creating a benchmark data set, the overall goal should be to obtain a diverse set of argumentation frameworks. Since the term “diversity” allows for multiple perspectives, we discuss some key aspects in the following.
Graph-Theoretical Features In order to ensure diversity in a graph-theoretical sense, benchmarks for abstract argumentation should encompass a wide range of graph properties and characteristics. This includes various properties, for example the node degree, the occurrence and number of (odd) cycles, variations in connectivity patterns, such as di erent levels of connectedness, etc. When creating new benchmarks, an analysis regarding such graph properties is valuable in order to check how newly generated AFs di er from existing benchmarks from a graph-theoretical perspective. New graph generators may also o er the possibility of parameterizing a number of graph features (which is already possible, to a degree, with most random graph generators). On the other hand, this may not be applicable in some scenarios (e.g., when dealing with real-world data).
Relation to Real-World Scenarios Creating benchmarks that re ect real-world argumentation scenarios can be challenging. Abstract argumentation frameworks might abstract from the complexity of real-world arguments and their relationships. Moreover, di erent domains, such as law, politics, healthcare, or ethics, have unique argumentation characteristics and requirements. Incorporating domain-speci c considerations in benchmark generation allows for more targeted evaluations and comparisons of argumentation systems within their intended domains.
Semantic Aspects Benchmarks should also be geared towards evaluating solution approaches
to the di erent problems related to abstract argumentation. Some benchmarks are already
designed for such purposes (such as SemBuster, which is aimed at problems related to the
semistable semantics), however, there are still numerous problems that have not been speci cally
addressed yet. As an example, it was recently demonstrated that in most ICCMA’17 instances, a
majority of skeptically accepted arguments w.r.t. pr were also included in the grounded and the
ideal extension. Since the computational complexity of deciding DSpr is 2P -complete [
In the previous section, we identi ed a number of challenges that occur in the generation of
benchmarks for abstract argumentation. Since it is not reasonable to address all concerns within
one graph generator, we focus on a speci c semantic aspect as an example, namely the issue
that solving the 2P -complete problem DSpr can often be bypassed by checking if the given
argument is accepted w.r.t. gr or id. Note that another “easy case” regarding DSpr occurs when
arguments are attacked by some admissible set—such arguments are never skeptically accepted
w.r.t. pr—and deciding this is a problem in NP. In [
We developed the KWT generator, which takes as parameters • numargs: the total number of arguments, • numpa: the number of arguments to be skeptically accepted under preferred semantics, • numcred: the number of arguments to be contained in at least one preferred extension, • numpref: the number of preferred extensions, • numideal: the number of arguments in the ideal extension, and further parameters that control the probability of attacks between di erent sets of arguments. More precisely, these parameters set the probabilities of arguments in the ideal extension to be attacked and to attack back, respectively, the probabilities of credulously accepted arguments to be attacked and to attack back, the probabilities of skeptically accepted arguments that are not contained in the ideal extension to be attacked and to attack back, and the probability of further random attacks between unaccepted arguments. Given these parameters, a random AF F is generated as follows: 1. The set Arg of numargs arguments is created and arguments are associated to sets Spa (skeptically accepted arguments w.r.t. preferred semantics), Sideal (arguments in the ideal extension), Scred (arguments that are credulously accepted w.r.t. preferred semantics), Sunacc (arguments that are not credulously accepted w.r.t. preferred semantics), such that Sideal Spa Scred, Scred Sunacc = Arg, and the corresponding cardinalities are respected. Finally, sets E1, . . . , Enumpref (the preferred extensions) are created by adding all arguments from Spa and randomly drawn arguments from Scred \ Spa. 2. For every argument a Sideal, random attackers from Sunacc are sampled. For each of these attackers b, another argument from Sideal is sampled that attacks b. This ensures that the grounded extension will be empty and that the ideal extension is capable of defending itself (thus forming an admissible set). 3. For every argument a Spa \ Sideal, attacks from unaccepted arguments are sampled in a similar way (to ensure an empty grounded extension). Furthermore, every such argument a must be defended by each preferred extension. Thus, for each preferred extension E, some arguments are sampled to defend a. 4. For every preferred extension E and a E \ Spa, attackers for a are sampled from Arg \ E and corresponding defenders are de ned within E. 5. Additional random attacks are added between arguments in Sunacc. 6. In order to avoid having stable extensions (which may also ease computation of arguments that are skeptically accepted under preferred semantics, since every stable extension is also preferred), we add another self-attacking argument and some attacks between this argument and arguments from Sunacc.
Note that due to the random approach of generating an argumentation graph, it may not
necessarily be the case that the number of skeptically/credulously accepted arguments (w.r.t.
preferred semantics) as well as the number of arguments in the ideal extension exactly match
the given parameters. However, our experiments in [
The graph generator2 and an example demonstrating its usage3 we used can be found online.
Throughout this paper we discussed how the generation of new benchmarks for abstract argumentation problems can be challenging from multiple perspectives. Although existing benchmarks for abstract argumentation already provide valuable resources for evaluating and comparing di erent frameworks and algorithms, they may not adequately capture the challenges and requirements posed by recent developments in the eld. Moreover, existing benchmarks may have limitations in terms of the problem space they cover (e.g. concerning characteristics of di erent graph properties).
Overall, we would like to highlight that new benchmarking techniques should yield AFs that are indeed novel in some regard, i.e., which di er from existing data—for instance, in terms of graph-theoretical properties, by addressing previously little considered semantic aspects, or by incorporating new real-world problems. Combining all of these di erent facets in one 2http://tweetyproject.org/r/?r=kwt_gen 3http://tweetyproject.org/r/?r=kwt_gen_ex single generator is presumably rather di cult, however, considering them individually might already lead to new insights. As an example, we presented the KWT generator, which generates particularly challenging AFs for the task of deciding skeptical acceptability w.r.t. preferred semantics.
The research reported in this work was supported by Deutsche Forschungsgemeinschaft under grant 375588274. [14] B. Yun, M. Croitoru, Benchmark on logic-based argumentation framework with datalog±, 2019.