We address the task of selecting the fastest algorithm, in terms of runtime, for determining skeptical acceptance under preferred semantics in abstract argumentation frameworks out of a set of multiple algorithms by means of machine learning. To be precise, we examine four “classical” machine learning techniques, as well as three graph neural networks, and compare all of these approaches with regard to both prediction accuracy and the total amount of time the selected algorithms require to solve a given test set in an experimental analysis. Our set of algorithms includes three solvers from the International Competition on Computational Models of Argumentation. Our results demonstrate that graph neural networks are a promising method for algorithm selection in abstract argumentation, as two out of three neural network models outperform all four classical machine learning approaches.
Approaches to formal argumentation [
In recent years, there has been an increased effort to develop algorithms and systems to solve
these high-complexity problems [
The remainder of this paper is structured as follows. In Section 2, we discuss the relevant preliminaries with regard to abstract argumentation, as well as both classical machine learning and graph neural network techniques. Section 3 comprises an overview of our approach and methodology. In Section 4, we present an experimental analysis, and we conclude in Section 5.
In the following, we provide an overview of the fundamentals of abstract argumentation on the one hand, and of classical machine learning as well as graph neural network methods on the other hand.
An abstract argumentation framework is a tuple AF = (A, R) where A is a set of arguments and R is a relation R ⊆ A × A. For two arguments a, b ∈ A the relation aRb means that argument a attacks argument b. For a ∈ A define a− = {b | bRa} and a+ = {b | aRb}. We say that a set S ⊆ A defends an argument b ∈ A if for all a with aRb then there is c ∈ S with cRa.
Semantics are given to abstract argumentation frameworks by means of extensions [
2. E defends every a ∈ E, and it is called complete (CO) if, additionally, it satisfies
Different types of classical semantics can be phrased by imposing further constraints. In particular, a complete extension E • is grounded (GR) if and only if E is minimal, • is preferred (PR) if and only if E is maximal, and • is stable (ST) if and only if A = E ∪ {b | ∃a ∈ E : aRb}.
All statements on minimality/maximality are meant to be with respect to set inclusion. Note that
the grounded extension is uniquely determined and that stable extensions may not exist [
E1 = {a1} E2 = {a1, a3}
E3 = {a1, a4} E1 is also grounded and E2 and E3 are both stable and preferred.
In this work we only consider the task of skeptical acceptance under preferred semantics, which we denote as DSPR.
The task considered in this paper is the selection of the fastest solver for a given problem instance. Thus, we are dealing with a classification problem: our goal is to classify which one of the algorithms at hand is most suitable to solve DSPR wrt. a given AF and a corresponding query argument a ∈ AF.
There exists a plethora of machine learning (ML) approaches which solve different types of classification problems. The overall goal of an ML method is to “learn” from given data (training data) in order to apply this “knowledge” on unknown data (test data). For this work, we selected a total of four supervised machine learning techniques, namely k-nearest neighbor, naive Bayes, random forest, and support vector machine, which will be explained in more detail in the following. The term supervised refers to the fact that all labels of the training data are known at all times during training, meaning that in our application scenario, for each instance in the training dataset (i.e., for each AF and corresponding query argument), the fastest solver is known. k-Nearest Neighbor The k-Nearest Neighbor (KNN) approach [19, 20] is based on the idea of classifying a datapoint according to its nearest k neighbors. More specifically, given a datapoint Y we aim to classify, we calculate the distance of all datapoints in the training data to Y . Then we select the k datapoints with the shortest distance to Y (i.e., the k nearest neighbors). Finally, we assign Y the class which is most frequently found among the nearest neighbors, using a voting rule.
Naive Bayes According to Bayes’s well-known Theorem (based on [21]), the probability of a datapoint Y = (y1, . . . , yn) belonging to class cd ∈ {c1, . . . , cm} is p(cd | Y ) = p(Y | cd)p(cd) , p(Y ) where p is a probability function. The naive Bayes (NB) classifier is built on the “naive” assumption that the value of a certain feature is independent of any other feature (i.e., given cd, y1 is independent of y2, and so forth), given the value of the class variable. Because of this independence assumption, we can use p(Y | cd) = p(y1 | cd) · . . . · p(yn | cd), and get p(cd | Y ) = p(y1 | cd) · . . . · p(yn | cd) · p(cd) .
p(y1) · . . . · p(yn) Since the denominator is constant (it is the same for each class) we can ignore it. Finally, we can classify Y by determining the class with the highest probability (cmax): n cmax = argmax p(c j) ∏ p(yi | c j)
j∈{1,...,m} i=1 Random Forest A Random Forest (RF) is an ensemble of decision trees which vote for the most popular class [22]. A decision tree is, as the name suggests, a tree structure in which the inner nodes are essentially test nodes, and the leaf nodes correspond to class labels. A test node checks certain feature values of a given sample and computes some outcome which is associated with one of the node’s subtrees. To classify a datapoint, we start at the root of the decision tree and propagate from test node to test node, until we reach a leaf node—which contains a class label, i.e., the classification result [ 23]. To construct an RF, for each individual decision tree, we randomly select a number of samples from the training set, and we randomly select a number of features to be considered [22].
Support Vector Machine The underlying principle of a Support Vector Machine (SVM) is that we view training samples as vectors in a vector space, which can be separated by hyperplanes, according to their class assignment. If the classes of the data at hand are not linearly separable (which is usually the case), we can apply a kernel function, which essentially transfers the training data to a higher dimension. If the dimension is high enough, the data become linearly separable [24].
We select a total of three different Graph Neural Network (GNN) architectures, namely Graph Isomorphism Network, Graph Convolutional Network, and GraphSage, which will be explained in more detail in the following. The core idea of message passing GNNs is to learn node or graph representations by iteratively aggregating local neighborhood information of a node (messages or embeddings) using non-linear transformations. Varying definitions of how the embeddings of the neighborhood nodes are aggregated (aggregate function) and how they are combined with the node embeddings from previous iterations (combine function) lead to different GNN architectures. After the final iteration, the embeddings encapsulate structural information of a node, respectively graph. These generated embeddings can then be used for downstream prediction tasks. For node classification tasks, the embedding of the final iteration is used for prediction. For graph classification tasks, a so-called readout function is used to aggregate node embeddings to obtain a representation of the entire graph.
Graph Isomorphism Network The Graph Isomorphism Network (GIN) [14] models the Weisfeiler-Lehman graph isomorphism test [25] in a neural network. It implements the aggregate and combine functions as the sum of the node embeddings and a multi-layer perceptron (MLP) [26] with non-linearity. For graph-level readout, the node embeddings of every layer are summed up and concatenated to get the final graph representation.
Graph Convolutional Network Graph Convolutional Networks (GCNs) [13], are initially motivated by sprectral graph convolutions [27, 28]. Following the definition in [ 14], they integrate the aggregation and combine step as an element-wise mean pooling, followed by a ReLU [29] non-linearity.
GraphSage The GraphSage [15] model was proposed with three different aggregation functions: (1) a mean aggregator, (2) a Long Short-Term Memory (LSTM) [30] aggregator and (3) a max-pooling aggregator. In this work, we consider the mean aggregator variant of GraphSage. The combination function is a concatenation followed by a linear mapping.
Various works in automated reasoning have investigated the concept of generating models that allow for identifying the most appropriate—or best—algorithm for an instance of a particular (computationally complex) problem. Empirical predictive models (EPMs) have been employed in many areas of Artificial Intelligence, such as the Satisfiability Problem (SAT) or Answer Set Programming (ASP), with great success [31, 32]. A basic distinction is made between two approaches: classification approaches and regression approaches. Classification approaches assign any given instance a single category corresponding to the algorithm, which is predicted to be the fastest. Regression approaches, on the other hand, try to predict the actual runtime of each algorithm under consideration. The algorithm with the lowest predicted runtime is then selected.
In this work, we consider algorithm selection for the skeptical acceptance wrt. preferred
semantics as a classification problem. Let S = {s1, ..., sn} be a set of solvers and let A be the set
of all argumentation frameworks. Conceptually, a classifier C is a mapping C : A → S, where any
AF ∈ A is assigned a solver s ∈ S that solves this instance the fastest. These mappings can be
learned using ML methods. In order to achieve that, classical supervised ML approaches need
some numerical representation of the instance in question, mainly referred to as features. In [
In this section, we present the results of an experimental analysis, in which we (1) investigate the applicability of different GNN architectures to select the most appropriate solver given an AF and (2) compare them to “classical” machine learning approaches. The analysis aims to give an overview of whether and to what extent GNNs are suitable for algorithm selection in abstract argumentation, and how they differ from classical methods in terms of performance. Below, we describe the experimental setup and subsequently discuss our findings.
In this work, we consider three SAT-based approaches for solving the problem of skeptical acceptance under preferred semantics: ArgSemSAT, Fudge, and µ-toksia.
ArgSemSAT The ArgSemSAT solver [16] is the winner of the preferred semantics track at the 2017 International Competition on Computational Models of Argumentation (ICCMA’17). It iteratively calls a SAT solver to compute complete labelings and encoding constraints to drive the search towards the solution of decision and enumeration problems. It is written in C++ and can be used with the Minisat [33] or the Glucose [34] SAT solver. For our experiments we use ArgSemSAT with Glucose.
Fudge The Fudge solver [17] tightly integrates satisfiability solving technology to solve a series of abstract argumentation problems. While most of the encodings used by Fudge derive from standard translation approaches, Fudge makes use of completely novel encodings to solve the skeptical reasoning problem wrt. preferred semantics. It is written in C++ and uses the satisfiability solver CaDiCaL 1.3.13 1. µ-toksia The µ-toksia solver [18] ranked first in all reasoning tasks of the ICCMA’19. It is a “purely” SAT-based system that is heavily based on the incremental use of SAT solving. This means, for iterative calls, the state of the SAT solver is maintained. By that, only a single SAT solver is instantiated during a single program run. It is implemented in C++ and includes interfaces to the Glucose [34] and CryptoMiniSAT [35] SAT solvers. We used µ-toksia with CryptoMiniSAT as an underlying SAT solver for our experiments. It should be noted that we considered various other solvers such as Pyglaf [36], Heureka [37], or ConArg2 [38] in our initial experiments. However, finding a fruitful mix of solvers to select from is a challenge in itself. We analyzed different combinations of solvers with regard to two criteria: (1) the number of instances for which each solver achieved the best performance and (2) the differences in runtimes (compared to the other solvers) wrt. the best-solved instances of each solver. The second criterion, in particular, is essential, since significant differences in the runtimes of solvers are fundamental in order for a given instance to benefit from the selection of a certain solver. This is also reflected in the fact that all machine learning-based approaches for algorithm selection introduce some overhead, for example, for calculating instance features. If there is no significant difference in execution times, the potential time savings of selecting the fastest solver get nullified due to the mentioned overhead. Our analysis showed that the combination of the selected solvers yielded the best-balanced ratio of the best-solved instances per solver and exhibited significant differences in their runtimes. It should also be noted that the goal of the evaluation is to compare the machine learning algorithms for algorithm selection, and not the selected algorithms themselves.
To collect sufficient training and test data, we randomly generated a total of 6200 argumentation frameworks using three different generators of the ICCMA’17: AFBenchGen2, SccGenerator, and StableGenerator.
The AFBenchGen2 [39] generator was used to create instances of (1) Erdös-Renyi [40] and (2) Watts-Strogatz [41] graphs. The SccGenerator aims to generate AFs with many strongly connected components, whereas the StableGenerator aims to generate AFs with many stable extensions (and therefore many preferred extensions). A detailed description of these generators can be found in [42]. All generators have been parameterized as described in [43]. We randomly selected a query argument for the DSPR task for each generated instance. A cutoff value of 600 seconds (10 minutes) per instance was imposed. For each solver, we recorded: (1) the number of solved instances, (2) the number of timed-out instances, (3) the number of crashed instances, and (4) the execution time per instance. The runtime of unsolved instances—timedout or crashed—was set equal to the cutoff. A total of 785 instances were excluded because all solvers failed to solve them within the given time frame. Due to memory limitations of the experimental environment, another 230 (very large) instances had to be excluded, as they could not be processed. No further systematic exclusions were made. We randomly divided the remaining 5185 instances into separate training and test data following an 80%/20% train/test split2. Table 1 shows the characteristics of the considered AFs.
For each instance, the fastest solver determined its ground-truth label. We ran the experiments 2Download: https://fernuni-hagen.sciebo.de/s/UkL9WRjegFlGyhk on a virtual machine running Ubuntu 20.04 with a 2.5 GHz Intel(R) Xeon(R) E5-2680 CPU and 60 GB of RAM.
We trained and evaluated four different supervised machine learning techniques: k-Nearest
Neighbor (KNN), Naive Bayes (NB), Random Forest (RF), and Support Vector Machine (SVM)
(see Section 2.2), using scikit-learn3 [44], a machine learning framework for Python4. As input for
the classifiers, we use the three best features that Vallati et al. [
Further, we trained and evaluated three widely adopted GNN models: Graph Isomorphism Networks (GIN) [14], Graph Convolutional Networks (GCN) [13], and GraphSage [15] (see Section 2.3). Each model features one pre-message passing layer (256-dim MLP), three message passing layers (determined by the respective GNN model) and two post-message passing layers (256-dim MLP). Each model is trained for 1000 epochs using the Adam [45] optimizer with a learning rate of 0.01 and batch normalization [46]. Results are reported for the final epoch. All models were trained without additional pre-calculated node, graph, or edge features. For the training and evaluation we use GraphGym [47], a platform for designing and evaluating GNNs.
To compare the performance of the approaches—both GNNs and classical ML methods—we use accuracy, precision, and recall to measure how accurate the predictions are. For all approaches the precision and recall are reported for each individual solver, following the definitions Precisions = Recalls =
T Ps T Ps + FPs
T Ps
,
T Ps + FNs where s denotes the class of the corresponding solver. True Positive (T P) are the elements that have been labeled as positive by the model and are actually positive, while False Positive (FP) are the elements labeled as positive, but they are actually negative. False Negative (FN) elements have been labeled as negative but are actually positive. The accuracy is defined as follows: Accuracy =
T P + T N
T P + T N + FP + FN where T N denotes the True Negative elements, i.e., the elements that the model correctly labeled as negative5. In addition, we measure the overhead (CPU-time) of calculating the features (which is only necessary for the classical ML approaches), and the actual classification time. For the GNN models, we also measure the prediction times when utilizing a GPU instead of a CPU. All models were trained and evaluated on a virtual machine running Ubuntu 20.04 with a 3.7 GHz AMD Ryzen 5 5600X 6-Core Processor, 32 GB of RAM and a NVIDIA GeForce RTX 3070 GPU with 8 GB of RAM. In order to accelerate the training process of the GNNs, training was carried out on the GPU. 3https://scikit-learn.org/stable/index.htm 4The KNN classifier was parameterized with n_neighbors = 17 and the RF classifier with random_state = 11, min_samples_split = 30, and min_samples_leaf = 10. The remaining classifiers (NB and SVM) were used with their default configurations. 5Note that, unlike precision and recall, the accuracy is not calculated by class but across all classes.
To begin with, we examine how accurate the predictions of both the classical ML methods and
the GNN approaches are. For this purpose, we first draw a comparison to the work by Vallati
et al. [
With regard to the GNN methods, our experiments reveal similar results: again, ArgSemSAT tends to have the hightest precision, µ -toksia has rather low and Fudge a rather high recall, and a large number of µ -toksia instances are classified as Fudge (see the lower part of Table 2). However, GraphSage exhibits a slightly different behavior: here, the precision is higher wrt. µ -toksia (0.90) than wrt. Fudge (0.66). The overall accuracy values of the GNN approaches lie between 0.63 (GIN) and 0.71 (GraphSage), and are consequently a bit higher on average than those of the classical ML approaches.
The second aspect we aim to examine, besides classification accuracy, is the overall solving time. For each approach, we sum up the solving time of each individual test instance with regard to the respective predicted solver, to calculate the total solving time.
For comparison we consider the time required by each solver when solving all test instances (see Table 3). Note that Fudge is clearly overall the fastest solver with 24,248 seconds, compared to 45,248 seconds (ArgSemSAT) and 77,734 seconds (µ -toksia). The results concerning the classical ML techniques are presented in the upper part of Table 4. We see that the overall solving time is quite similar with regard to KNN, RF, and SVM (between 24,507 and 24,538 seconds), only NB is considerably slower (27,794 seconds). The former three classifiers also perform an algorithm selection which results in a total solving time that is shorter than that of using Fudge (i.e., the fastest individual solver) for all instances. Only the NB classifier yields an algorithm selection which results is a longer solving time. However, NB also exhibited the lowest accuracy (0.60), which could explain this outcome. On the other hand, we notice that the SVM classifier produces the shortest total solving time, even though it did not feature the highest accuracy.
ArgSS
Fudge µ-toksia
ArgSS
Fudge
µ-toksia Precision Likewise, this effect can be observed wrt. the GNN methods. The lower part of Table 4 shows that GIN, which has the lowest accuracy of all GNN methods (0.63), produces a slightly lower total solving time than GraphSage, which has a significantly higher accuracy of 0.71. Overall, the GNN methods perform superior to the classical ML methods. Only the “fastest” classical ML method (SVM) accomplishes a slightly superior algorithm selection than the “slowest” GNN method (GCN).
As of yet, we only considered the solving time required by the selected algorithms. However, we additionally need to consider the time required for each prediction, and in the classical ML case also the time required to compute the features of each test instance. The latter amounts to an average of 0.0077 seconds per instance (i.e., 8.02 seconds for the entire test set). The time needed to predict all test instances is < 0.5 seconds for KNN, NB, and SVM, only the RF takes longer (5.2 seconds in total). The GNN approaches require significantly more time for the prediction process—they take between 58.4 seconds (GIN) and 68.2 seconds (GCN). However, these values were measured when the predictions we conducted on the CPU. When using the GPU, we can drastically reduce the prediction time to values between 0.9 seconds (GIN) and 1.3 seconds (GCN). Moreover, even if there was no GPU available, both GIN and GraphSage would still outperform all classical ML models. For instance, wrt. GIN, we have 24,410.0 seconds of total solving time plus 58.4 seconds of prediction time, i.e., a total of 24,468.4 seconds, which is already less than the shortest total solving time wrt. the classical ML methods (24,506.7 seconds), regardless of the additional time required for prediction and feature generation.
In the scope of this work, we examined different ML approaches for the task of algorithm
selection in the field of abstract argumentation. We followed up on a study by Vallati et al. [
One noteworthy result of our experiments is that µ-toksia instances were often classified as Fudge instances, and that this effect did not only occur with the classical ML methods, which are feature-based, but also with the GNN methods, which are not given any pre-calculated features explicitly6. Nevertheless, our results demonstrated that neural networks are generally a useful approach for the task of algorithm selection. In particular, GIN and GraphSage performed superior to all classical ML approaches in terms of the total solving time of the predicted algorithms, even including a rather lengthy prediction time when no GPU is available. However, there is still room for improvement—if we always used the fastest solver, the total solving time would be 14,430 seconds, which is 40.9% less than our best result (see Table 4).
Furthermore, we examined the prediction accuracy of the different approaches and discovered that a higher accuracy does not automatically lead to a shorter overall solving time. This suggests that in some cases the classifier did not only fail to predict the fastest solver, but also failed to 6We also conducted an experiment in which we additionally fed the neural networks the same features that were used with the other ML methods. However, this did not improve our results. predict the second fastest (i.e., it predicted the slowest one). Hence, one idea to consider in future work is to introduce some sort of weighting which penalizes the prediction of a slower solver more than the prediction of a faster one during training. Regarding the classical ML methods, one could use a different set of features which is more suitable for the newly considered solvers (Fudge and µ -toksia). However, this might lead to an increase in feature generation time which must always be weighed against the potential reduction in solving time. Further issues which could be addressed in future work are the consideration of other argumentation semantics and tasks, as well as other GNN architectures. [12] D. Craandijk, F. Bex, Enforcement heuristics for argumentation with deep reinforcement learning, in: Proceedings of the 36th AAAI Conference on Artificial Intelligence, 2022. [13] T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, in: International Conference on Learning Representations (ICLR), 2017. [14] K. Xu, W. Hu, J. Leskovec, S. Jegelka, How powerful are graph neural networks?, in:
International Conference on Learning Representations, 2019. [15] W. Hamilton, Z. Ying, J. Leskovec, Inductive representation learning on large graphs,
Advances in neural information processing systems 30 (2017). [16] F. Cerutti, M. Giacomin, M. Vallati, How we designed winning algorithms for abstract argumentation and which insight we attained, Artificial Intelligence 276 (2019) 1–40. URL: https://www.sciencedirect.com/science/article/pii/S0004370218302650. doi:https: //doi.org/10.1016/j.artint.2019.08.001. [17] M. Thimm, F. Cerutti, M. Vallati, Fudge: A light-weight solver for abstract argumentation based on sat reductions, in: The Fourth International Competition on Computational Models of Argumentation (ICCMA’21), 2021. [18] A. Niskanen, M. Järvisalo, µ-toksia: An Efficient Abstract Argumentation Reasoner, in: Proceedings of the 17th International Conference on Principles of Knowledge Representation and Reasoning, 2020, pp. 800–804. URL: https://doi.org/10.24963/kr.2020/82. doi:10.24963/kr.2020/82. [19] E. Fix, J. L. Hodges, Discriminatory analysis. nonparametric discrimination: Consistency properties, International Statistical Review/Revue Internationale de Statistique 57 (1989) 238–247. [20] P. Cunningham, S. J. Delany, k-nearest neighbour classiefirs-a tutorial, ACM Computing
Surveys (CSUR) 54 (2021) 1–25. [21] T. Bayes, Lii. an essay towards solving a problem in the doctrine of chances. by the late rev. mr. bayes, frs communicated by mr. price, in a letter to john canton, amfr s, Philosophical transactions of the Royal Society of London (1763) 370–418. [22] L. Breiman, Random forests, Machine learning 45 (2001) 5–32. [23] J. R. Quinlan, Learning decision tree classifiers, ACM Computing Surveys (CSUR) 28 (1996) 71–72. [24] C. J. Burges, A tutorial on support vector machines for pattern recognition, Data mining and knowledge discovery 2 (1998) 121–167. [25] N. Shervashidze, P. Schweitzer, E. J. Van Leeuwen, K. Mehlhorn, K. M. Borgwardt,
Weisfeiler-lehman graph kernels., Journal of Machine Learning Research 12 (2011). [26] S. Haykin, Neural networks: a comprehensive foundation, Prentice Hall PTR, 1994. [27] D. K. Hammond, P. Vandergheynst, R. Gribonval, Wavelets on graphs via spectral graph theory, Applied and Computational Harmonic Analysis 30 (2011) 129–150. [28] M. Defferrard, X. Bresson, P. Vandergheynst, Convolutional neural networks on graphs with fast localized spectral filtering, Advances in neural information processing systems 29 (2016). [29] A. F. Agarap, Deep learning using rectified linear units (relu), arXiv preprint arXiv:1803.08375 (2018). [30] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (1997) 1735–1780. [31] L. Xu, F. Hutter, H. H. Hoos, K. Leyton-Brown, Satzilla: portfolio-based algorithm selection for sat, Journal of artificial intelligence research 32 (2008) 565–606. [32] M. Gebser, B. Kaufmann, A. Neumann, T. Schaub, clasp: A conflict-driven answer set solver, in: International Conference on Logic Programming and Nonmonotonic Reasoning, Springer, 2007, pp. 260–265. [33] N. Eén, N. Sörensson, An extensible sat-solver, in: International conference on theory and applications of satisfiability testing, Springer, 2003, pp. 502–518. [34] G. Audemard, L. Simon, Lazy clause exchange policy for parallel sat solvers, in: International Conference on Theory and Applications of Satisfiability Testing, Springer, 2014, pp. 197–205. [35] M. Soos, K. Nohl, C. Castelluccia, Extending sat solvers to cryptographic problems, in: International Conference on Theory and Applications of Satisfiability Testing, Springer, 2009, pp. 244–257. [36] M. Alviano, The pyglaf argumentation reasoner, in: Technical Communications of the 33rd International Conference on Logic Programming (ICLP 2017), Schloss Dagstuhl-LeibnizZentrum fuer Informatik, 2018. [37] N. Geilen, M. Thimm, Heureka: a general heuristic backtracking solver for abstract argumentation, in: International Workshop on Theorie and Applications of Formal Argumentation, Springer, 2017, pp. 143–149. [38] S. Bistarelli, F. Rossi, F. Santini, Conarg 2 : A constraint-based tool for abstract argumentation, 2015. [39] F. Cerutti, M. Giacomin, M. Vallati, Generating structured argumentation frameworks:
Afbenchgen2., in: COMMA, 2016, pp. 467–468. [40] P. Erdo˝s, A. Rényi, et al., On the evolution of random graphs, Publ. Math. Inst. Hung. Acad.
Sci 5 (1960) 17–60. [41] D. J. Watts, S. H. Strogatz, Collective dynamics of ‘small-world’networks, nature 393 (1998) 440–442. [42] M. Thimm, S. Villata, The first international competition on computational models of argumentation: Results and analysis, Artificial Intelligence 252 (2017) 267–294. URL: https://www.sciencedirect.com/science/article/pii/S0004370217301030. doi:https: //doi.org/10.1016/j.artint.2017.08.006. [43] S. A. Gaggl, T. Linsbichler, M. Maratea, S. Woltran, Benchmark selection at iccma’17 (2018). [44] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830. [45] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014). [46] S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: F. Bach, D. Blei (Eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, PMLR, Lille, France, 2015, pp. 448–456. URL: https://proceedings.mlr.press/v37/ioffe15. html. [47] J. You, R. Ying, J. Leskovec, Design space for graph neural networks, in: NeurIPS, 2020.