The paper presents the technical details of the ArgEML system α-version, which implements a general argumentation-based framework and methodology for Explainable Machine Learning. ArgEML is based on a novel approach that integrates sub-symbolic methods with logical methods of argumentation to provide explainable solutions to learning problems. Figure 1: ArgEML Methodology
ArgEML is motivated by several works in the literature that explore the potential of the strong connection of argumentation with learning in the context of explainability. Some of these works have studied how to learn argumentation frameworks from data, abstract frameworks, [1], [2],[3],[4],[5] or structured frameworks, [6], [7], [8], [9], [10], [11]. Other interesting works can be found in [12], [13], [14], [15],[16], [17], [18], [19].
The ArgEML learning methodology is a case of symbolic supervised learning, that can be also applied in a hybrid mode on top of other symbolic or non-symbolic learners that would generate an initial learning theory. The methodology is outlined in Figure 1 and briefly explained in paragraph 1.1.
• Step 1: decides the language (relevant features / predictors) of the learning problem in a similar way to the data processing step in a standard machine learning pipeline. • Step 2: identifies the basic contexts of the problem domain by selecting a compact set of arguments with high coverage to initialize the theory.
Both steps (1) and (2) can be executed automatically or in a hybrid mode by calling onto a subsymbolic or symbolic existing learner. •
Step 3: involves a repeated learning process to produce an argumentation theory as the final output of the learner. At each iteration step two main operators are considered: a mitigation of errors in the definite prediction of (some part of) the current theory and an operator for resolving conflicts in the ambiguity of the current theory. The step is guided by a learning assessment (metric) that measures the quality of a theory as a trade-off between accuracy and ambiguity.
The resulting explainable model is an argumentation theory that supports the conclusions (labels) of a target variable (classification problem case). To generate a prediction for an input case the theory is queried against all possible conclusions. If exactly one conclusion can be derived then the prediction is considered definite, otherwise, the conclusion forms a dilemma within the theory. Moreover, a definite prediction can be correct or wrong, that is definite correct or definite wrong. The learning assessment metric, which is a generalization of the standard classification accuracy, is defined as: LA includes a weighted element wd that reflects the weakness of dilemmas of the theory, e.g. for binary classification learning problem this factor can be chose to be one-half.
System components and its main functions are discussed in chapters 2.1 and 2.2 respectively, whereas in chapter 2.3 we explain the evaluation (system verification) process followed. Details of the ArgEML theory and learning method can be found in [20]. Figure 2 shows two screenshots of the system, an ArgEML run on the left, and an ArgEML output on the right.
The ArgEML system is a Java application that integrates with Gorgias [21], a structured argumentation framework, for the development and evaluation of the argumentation theory it generates. In the automatic mode of operation, the application accepts as input a dataset (examples + feature set), while in the hybrid mode of operation, the system also accepts as input the results of an external ML model’s execution on the input dataset. The current implementation can process the results of the inTrees [22] library. The application interacts with the SWI-Prolog2 component for the evaluation of the Gorgias argumentation theories learned. This interaction is achieved via the JPL API3. 2 A versatile implementation of the Prolog language. https://www.swi-prolog.org/ 3 A library that provides a bidirectional interface between Java and Prolog. https://jpl7.org/ .
The system accepts as input a dataset in the form of a csv file (feature set is automatically derived from the file), a set of decision-rules in a predefined format as a csv file, and a set of parameters that control the learning process. In the automatic mode of operation, the learning process starts from exploring the input feature set, to construct the initial set of arguments. In the hybrid mode, additional knowledge is provided as input to the system, in the form of association rules between input features and the target feature.
The output of the system is a Gorgias argumentation theory that we can use like any other ML model to generate predictions for new inputs with the corresponding explanations. The execution of the system is highly parametric allowing the end user to fine tune the execution of the process. The basic parameters are shown in Table 1.
Values Description args-np / Strategy to define type of initial arguments as args-wp /mixed general or with premises or both.
percentage The target value for Definite errors metric. percentage The target value for Ambiguity metric. percentage It defines the percentage above which a class of data
is considered as a majority class. percentage It defines the range up to which a class distribution
is considered balanced. integer It defines the maximum number of iterative learning
steps. percentages It defines the percentages for splitting the data into
train and test. integer It defines the maximum number of conditions for
rules selected by the hybrid process of step 2. decimal<1 It defines the acceptable performance loss during
the iterative learning process. args-np:arguments without premises. args-wp:arguments with premises.
• Parameters fine tuning: The user can experiment with various parameter values to understand under which configuration the system performs better for their problem. • Explanations (system output): Explanations of a prediction are provided in a natural form containing also a contrastive element against other possible predictions. An example of explanation is shown in Table 2. In this example the system learns an argumentation theory from an artificial dataset with 10 binary features that supports scenarios for “staying at home” or “going to work”.
The system can also use the argumentation-based explanations to partition the problem-space into different sub-groups, examples are show in Table 3.
The system can use these sub-groups to provide a grading a confidence for new predictions depending on the group that a new case may fall. Also, the identification of the dilemma groups can guide us to look for new data (to help resolve these).
Currently, the ArgEML system supports classification problems on datasets with categorical features. The ArgEML system is under continuous evaluation on different learning problems through which we get feedback that can help us tune and improve the approach. We present the results of our experimentation on three datasets, (1) an artificial dataset, (2) a standard dataset from a ML repository, and (3) a real-life image dataset. We compare the results with Random Forest (RF) models in Table 4.
Train set(80%) Test set(20%) RF ArgEML RF ArgEML CA DA LA CA DA LA Artificial Dataset (120) {args-wp, 0%, 0%, n/a} 1 1 1 n/a n/a n/a IRIS (150) {args-np, 0%, 0%, n/a} 0.96 0.96 0.96 0.90 0.93 0.93 ACSRS (200) {hybrid, 5%, 10%, 2} 0.90 0.94 0.84 0.78 0.77 0.71 a:{initialize-theory, definite-errors-threshold, ambiguity-threshold, rules-complexity}.
All experiments run with majority-class=60%, balanced-distribution=20%, iterative-learning-steps=10. The experiment on the artificial dataset run with 100% on the train set. RF: Random Forest. CA: Classification Accuracy. LA: Learning Assessment. A: Definite Accuracy.
The comparison shown in Table 4 is between the metric of Definite Accuracy, defined as (definite correct predictions) / (definite predictions), for the ArgEML theories, and Classification Accuracy for the RF models. We are also currently experimenting by running ArgEML in hybrid mode on top of standard explainability systems, such as LIME [23], SHAP [24] and GLocalX [25].
The related material and the codebase of the system, together with the example data sets used in the demo are available on GitHub (github.com/nicolepr/argeml). The release of ArgEML α-version will provide the research community with another xAI tool for learning, experimentation and development of explainable solutions for decision support. We look forward to collaborate with the community to improve ArgEML and also work on new ideas. An important case of this is to examine how ArgEML can be used to enhance post-hoc explainability layer for opaque black-box learned models.