The third International Competition on Computational Models of Argumentation (ICCMA'19) focuses on reasoning tasks in Abstract Argumentation. Submitted solvers are tested on a selected collection of benchmark instances, including arti cially generated argumentation frameworks and some frameworks formalizing real-world problems. In this paper we introduce the testing environment set for the competition, including its problems and participants.
First of all, the competition features a new track concerning dynamic
Argumentation Frameworks (AFs, see Sect. 2), which can measure the e ciency of
solvers in recomputing extensions with small modi cations to a starting AF. In
this track, dedicated to dynamic solvers and approaches [
a platform-as-a-service software that uses operating-system level virtualization to deliver software in packages called containers. In this case, our purpose is to encourage the development of solves that can easily run everywhere, so to ease the evaluation phase and allow for the recomputation of the competition results.
The paper is structured as follows: in Sect. 3 we describe the main novelties
introduced in ICCMA'19 with respect to previous editions. In Sect. 4 we describe
the tasks we tested in the competition. Section 5 shows the input and output
formats of abstract argumentation frameworks that solvers were required to deal
with. Finally, Sect. 6 outlines ICCMA'19 participants and benchmarks, and how
results were evaluated in order to obtain a nal ranking. Section 7 reports the
nal conclusions and future improvements.
2
An Abstract Argumentation Framework (AF, for short) [
The collective acceptability of arguments depends on the de nition of di erent
semantics. Four of them are proposed by Dung in his seminal paper [
Semantics determine sets of jointly acceptable arguments, called extensions,
by mapping each F = (A; !) to a set (F ) 2A, where 2A is the power-set of A,
and parametrically stands for any of the considered semantics. The extensions
under complete, preferred, stable, semi-stable [
set inclusion) among all complete extensions in F , { E 2 ST(F ) i E 2 CO(F ) and E [ E+ = A, { E 2 STG(F ) i E is con ict-free in F and E [ E+ is maximal (w.r.t. set inclusion) among all con ict-free sets in F , { E 2 GR(F ) i E 2 CO(F ) and there is no E0 2 CO(F ) s.t. E0 E, { E 2 ID(F ) if and only if E is admissible, E T PR(F ) and there is no admissible E0 T PR(F ) s.t. E0 E.
For a more detailed view on these semantics please refer to [
In Sect. 3.1 and Sect. 3.2 we respectively introduce the Docker environment we used to execute and test solvers, and the literature about dynamic solvers. 3.1
The Docker Platform Docker is an open-source implementation of operating-system-level virtualisation, also known as containerisation. It can be used for developing, shipping, and running applications, separating applications from infrastructure with the purpose to deliver software quickly.
Docker is primarily developed for Linux, where it uses the resource isolation features of the Linux kernel such as cgroups and kernel namespaces, and a union-capable le system, to allow independent \containers" to run within a single Linux instance. The main aim is to avoid the overhead of starting and maintaining virtual machines. Docker allows applications to use the same Linux kernel as the system that they are running on and only requires applications to be shipped with things not already running on the host computer. The same container can also be executed on di erent operating systems: besides di erent Linux distros such as Debian, Fedora, and Ubuntu, there also exist Docker engines for MacOS, Windows, Amazon Web Services, and Microsoft Azure, which allow for directly moving an application into the cloud without modi cations.
The Docker Engine is a client-server application. The rst component is a server, i.e., dockerd. It listens for Docker API requests and manages Docker objects such as images, containers, networks, and volumes. The second component is a REST API which speci es interfaces that programs can use to talk to the daemon and instruct it what to do. Finally, the third component is a command line interface (CLI) client: the CLI uses the Docker REST API to control or interact with the Docker daemon through scripting or direct CLI commands.
We require each solver to be submitted to ICCMA'19 to be packaged in a Docker container. To do so, a participant needs three les at least: i) a Docker le, ii) a solver interface.sh le, and iii) the solver itself. The Docker le de nes the environment in the container. The access to resources like networking interfaces is virtualised inside this environment. We suggested to package solvers by using
3.2
Motivations to Dynamic Frameworks In previous ICCMA editions, all the frameworks in each database were considered static, in the sense that all the AFs were sequentially passed as input to solvers, representing di erent and independent problem instances: all tasks were computed from scratch without taking any potentially useful knowledge from previous runs into account.
However, in many practical applications, an AF represents only a temporary
situation: arguments and attacks can be added/retracted to take into account
new knowledge that becomes available. For instance, in disputes among users of
online social networks [
For this reason, ICCMA'19 also featured additional tracks to evaluate solvers on dynamic Dung's frameworks. The aim was to test those solvers dedicated to e ciently recompute a solution after a small change in the original AF. In this case, a problem instance consists of an initial framework (as for classical tracks) and an additional le storing a sequence of additions/deletions of attacks on the initial framework, that is a list of modi cations. This le is provided through a simple text format, e.g., a sequence of +att(a; b) (attack addition) or att(d; e) (attack deletion). The nal single output needs to report the solution for the initial framework and as many outputs as the number of changes.
The dynamics of frameworks has attracted recent and wide interest in the
Argumentation community. We describe some related work, which also points
to the research groups interested in the organisation of such a track. In [
A work that tests complete, preferred, stable, and grounded semantics on
an AF and a set of updates is [
Modi cations of AFs are also studied in the literature as a base to compute
robustness measures of frameworks [
ICCMA'19 let solvers participate in 7 classical tracks, exactly the same tracks as in ICCMA'17. The tracks are named along the name of semantics, thus we have a track for each 2 fCO; PR; ST; SST; STG; GR; IDg.
The tasks are characterised by a problem and the semantics with which the problem is solved. The considered problems are: SE- : Given F = (A; !), return some set E A that is a -extension of F . EE- : Given F = (A; !), enumerate all sets E A that are -extensions of
F .
DC- : Given F = (A; !) and a 2 A, decide whether a is credulously accepted in F under .
DS- : Given F = (A; !) and a 2 A, decide whether a is skeptically accepted in F under .
For single-status semantics (GR and ID) the problem EE is equivalent to SE, and DS is equivalent to DC. Also note that the DC problem returns the same results when computed for CO and PR, but in order to allow the participation in the PR track without implementing tasks on the CO semantics (or vice versa), both tasks are maintained. Hence, the tasks in ICCMA'19 were: CO: Complete Semantics (SE, EE, DC, DS); PR: Preferred Semantics (SE, EE, DC, DS); ST: Stable Semantics (SE, EE, DC, DS); SST: Semi-stable Semantics (SE, EE, DC, DS); STG: Stage Semantics (SE, EE, DC, DS); GR: Grounded Semantics (only SE and DC); ID: Ideal Semantics (only SE and DC).
The combination of problems and semantics amounts to a 24 tasks
overall. In addition, 4 new tracks were dedicated to the solution of problems over
dynamic frameworks, this time using the semantics originally proposed in [
In this case, the combination of problems with semantics amounts to a total 14 tasks. Tasks in dynamic tracks are invoked by appending \D" at the end of the intended task: for instance, EE-PR-D points to the enumeration task with the preferred semantics.
In total, ICCMA'19 was composed of 11 tracks that collect 38 di erent tasks. Each participating solver could compete in an arbitrary set of tasks. If a solver supported all the tasks of a track (e.g., the track on complete semantics), it also automatically participated in the corresponding track. 5
In the following of this section we rst describe the two le format taken as input by solvers. Benchmarks in ICCMA'19 were available in both the formats, in order to allow participating solvers to choose their preferred during the competition.10 Moreover, we also shortly describe the required output. In this case the format has to be standard in order to evaluate the answers returned by solvers. 5.1
Input Format Each benchmark instance, that is each AF, is represented in two di erent le formats: trivial graph format (tgf) and aspartix format (apx). We now represent F = (A; !), where A = fa1; a2; a3g and != f(a1; a2); (a2; a3); (a2; a1)g, in each of these two formats. 10 Solvers that can use the two formats were required to select the one they wanted to be tested on in ICCMA'19.
tgf11 is a simple text-based adjacency list le format for describing graphs. The format consists of a list of node labels, followed by a list of edges, which specify node pairs and an optional edge label: in the above example we follow the format 1 2 3 # 1 2 2 3 2 1.
The apx format is instead described in [
Both the tgf and apx formats have been used during the previous editions of ICCMA. The novelty is instead represented by formats for dynamic AFs. For each (dynamic) problem instance, a solver requires to take as input two les: the initial framework (either in apx or tgf format) and a text le with a list of changes to be applied to it. The le with changes has to report a list of modi cations (one per line) over the initial framework. The format of the le with changes has to follow the same format of the original le (either in apxm or tgfm format, see in the following of this section). Let us introduce an example in apx.
Example 1. The initial framework is provided in a le named, for example, myFile.apx: arg(a1). arg(a2). arg(a3). att(a1,a2). att(a2,a3). +att(a3,a2). -att(a1,a2). +att(a1,a3).
The second le is a text le containing the list of modi cations to be sequentially performed on the starting le, one after another. The name of this le has to be the same as the starting framework, with extension .apxm instead of .apx. For this example, myFile.apxm is:
Applying these changes over the initial le corresponds in practice to three full frameworks (besides the initial one), which are represented in Figure 2:
We propose a second example by using the same framework expressed in the tgf format this time.
Example 2. The text le with modi cation needs to have the same name of the the initial framework, with su x .tgfm instead of .tgf. Hence, in this case myFile.tgfm is: 11 Trivial graph format: http://en.wikipedia.org/wiki/Trivial_Graph_Format.
As for the previous example, even in this case we obtain three di erent frameworks, in this case represented in tgf. Such AFs are in Figure 3.
We required ICCMA'19 benchmark generators to produce modi cations where attacks have to be added only between existing arguments: no new argument can be introduced. In case all the attacks connected to an argument were removed, such an argument is not removed from the framework.
Moreover, benchmarks and benchmark generators needed to provide both the initial framework and the modi cation le for each instance. For each initial framework, a modi cation le with at least 15 attack additions/deletions was required.
Therefore, if a modi cation le had n changes (one per text line), a solver had to run n di erent problems by applying such modi cations in sequence (from the top of the le).12 5.2
Output Format Concerning the same tasks tested in previous ICCMA editions, the format that the output needs to follow does not change. For DC and DS tasks, the printed 12 We asked the participants to nd solutions sequentially, one modi cation after another, even if such changes are all in the same le. standard output was required to be either YES or NO. The SE task had to output the list of arguments belonging to the returned extension: for example, [a1; a2; a4]. Finally, EE had to return the list of extensions in the form [[a1; a2] [a1; a3] [a2; a3]].
Concerning the dynamic tracks instead, all the answers were in the form of a list where the rst element represents the solution of the required task on the initial framework; each following element in this list is the answer returned on the (i + 1)th framework obtained by sequentially applying the rst i changes in the modi cation le: i 2 [1::n] and n is the total number of changes in the modi cation le. DC- -D and DS- -D returns a list of YES or NO, one for each modi cation including the initial framework: for example, [[YES]; [YES]; [NO]; : : : ]. In SE- -D we have a list of extensions: [[a1; a3][a1][a1; a2]]. Finally, the result of EE- -D tasks corresponded to lists of extensions, one for each obtained AF: for instance, [[[a1; a3]] [[a1; a3]] [[a1][a1; a3] [a1; a2]] [[a1; a2]]]. 6 6.1
Participants The competition received 9 solvers from research groups in Austria, Finland, France, Germany, Italy, Romania and the UK: see Table 1. Among them, 3 were submitted to all the tracks (including dynamic tracks). The authors of the solvers used di erent techniques to implement their tools. In particular, 4 were based on the transformation of argumentation problems to SAT, 1 on transformation to ASP, 1 relied on machine learning, and 3 were built on tailor-made algorithms.
Table 2 reports every single track each solver participated in.
. s r e v l o s y b d e t r o p p u s s k s a T . 2 e l b a T
ID SECD
E STG SSED
C D E
E SST SSED
C
D RG SECD 33 33 33 33 33
E c CD 3 3 3 3 3 i m a n y D
Q YG SPA san RD oC P A oY r f e r a i s k gp rgqA to cea A E - M 9 r e v o r P / 4 6.2
Benchmarks Each solver had 4GByte of RAM to compute the results of tasks in both the classical and dynamic track. 326 argumentation framework instances were selected among those used in previous competitions as well as 2 new benchmarks submitted for ICCMA'1913.
The ranking of solvers for a track was based on the sum of scores over all tasks of the track. Ties were broken by the total time it took the solver to return correct results. Note that in order to make sure that each task had the same impact on the evaluation of the track, all tasks for one semantics had the same number of instances. Each solver participating in a task was queried with a xed number of instances corresponding to the task with a timeout of 10 minutes each. For each instance, a solver got (0; 1] points if it delivered the correct result (it may have been incomplete); 5 points if it delivered an incorrect result; 0 points if the result was empty (e.g., the timeout was reached without answer) or if it was not parsable (e.g., some unexpected error message). In case of testing SE, DC, and DS, the assigned score was 1 if the solver returned the correct answer (respectively \yes", \no", or just an extension). In case of EE, a solver received a (0; 1] fraction of points depending on the percentage of found enumerated extensions (1 if it returned all of them). 6.3
Evaluation The timeout to compute an answer for dynamic tracks was 5 minutes for each framework/change (half of the time in the classical track for a single instance). For the solvers participating in the dynamic tracks, a result was considered correct and complete if, for n changes, n + 1 correct and complete results were given. The score for a correct and complete result was 1. A partial (incomplete) result was considered correct if it gave less than n + 1 answers, but each of the given answers was correct and complete (with respect to the corresponding static tasks). These rules hold for all the problems (SE, DC, DS, EE) in the dynamic tracks. A correct but incomplete result scored a value in (0; 1], depending on the rate of correct sub-solutions given. There was an exception in the case the considered dynamic task involved enumeration (i.e., EE): if the last solution a solver provided was correct but partial, then the whole answer was evaluated as the last problem was not solved at all, considering the answer as partial and correct, and fraction of 1=n points, depending on the percentage of returned enumerated extensions was assigned. If any of the sub-solution was incorrect, then the overall output was considered incorrect ( 5 points). Otherwise, in case no answer was given, 0 points were assigned (for sintance, due to a timeout). In the nal ranking, ties were broken by the total time it took the solver to return correct results for all the considered frameworks (initial framework and successive changes). 13 ICCMA'19 benchmarks: https://iccma2019.dmi.unipg.it/submissions.html.
The third International Competition on Computational Models of Argumentation (ICCMA'19) focuses on reasoning tasks in abstract argumentation frameworks. Its aim is to provide a forum for empirical comparison of solvers, to highlight challenges to the community, to propose new directions for research and to provide a core of common benchmark instances and a representation formalism that can aid in the comparison and evaluation of solvers.
We have described the environment in which we performed ICCMA19. Using Docker made the competition easy to recompute, allowing submitters and independent parties to easily reproduce our results and build on them to advance the state of the art. As a second improvement, we also organized a track completely dedicated to dynamic solvers, where previous results can be used to rapidly reach a solution on a slightly modi ed AF, instead of solving the whole problem from scratch.
As future work, we will provide insights on the results obtained by solvers; for a rst summary of nal solver-rankings, the interested reader can check them from the main Website of ICCMA.14 In the non-dynamic tracks, -toksia is the overall winner, followed by CoQuiAAs and Aspartix19, while in the dynamic track, the rst two positions are the same, and then we have Pyglaf.
The rst and third author have been supported by projects Argumentation 360 (\Ricerca di Base" 2017{2019) and Rappresentazione della Conoscenza e Apprendimento Automatico (RACRA) (\Ricerca di base" 2018{2020). 14 Rankings: http://argumentationcompetition.org/2019/results.html.