TED has been proposed as a simple framework for using any supervised classi cation approach to create Explainable AI models [3]. It requires manual annotation of training data with explanations. The authors of TED report that it yields results as good or better than conventional supervised learning. This paper shows that one possible reason for this is that the extension of training data can introduce bias by changing the class distribution. Experiments on the Iris dataset and a synthetic dataset show that TED performs signi cantly worse than initially reported when using class weights during training to o set class imbalance in training data.
Explainable Arti cial Intelligence as a eld of research is becoming more popular. While black-box approaches like deep learning have been very successful, good performance alone is not always enough. As AI becomes more widely used it also becomes more important to understand what it is actually doing. A decision may be good if it is correct, but it is better if it is correct and understandable.
There are good, old-fashioned AI approaches, like Inductive Logic
Programming (ILP), which are inherently white-box and thus interpretable. This can,
for example, be used to generate human-readable representations of the
decision making of a system. Dare2Del [
LIME [
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Another approach has a similar goal, but a very di erent design: TED
(Teaching Explanations for Decisions ) is a simple, high-level framework . The basic idea
is to use any supervised classi cation approach to learn not only labels, but also
explanations. This approach depends heavily on the quality of the training data.
The authors explicitly state the importance of good explanations provided by
domain experts [
Its high-level, semi-manual nature makes this approach very exible. The
explanations are part of the training data, therefore they can be adapted to a
variety of domains and target groups. The authors argue that the requirement
to manually create training data with explanations can also help reduce bias,
since adding a biased example would be hard to justify [
However, bias is not always obvious when considering single examples. Class imbalance is one type of bias that can occur in training data. When classes are not equally distributed, trained classi ers may inadvertently favor more frequent classes when making predictions.
This paper investigates this problem in the context of TED. In section 2 the
design of the TED framework is described and the original evaluation by [
TED, unlike traditional classi cation approaches, learns class labels and
explanations from examples [
Y a class label for X E an explanation for Y The form and content of explanations are not speci ed and may contain arbitrary values. The only constraint is that they must be enumerable.
Because explanations are meant to be provided as part of the training data,
the application of TED is extremely simple. Instead of learning only labels from
examples (X ! Y ), TED learns labels and explanations (X ! Y E). The
Cartesian product instantiation described in [
The Cartesian product instantiation of TED has been evaluated on two datasets. The Tic-Tac-Toe example is based on the game of tic-tac-toe. The legal non-terminal board positions are labeled with the optimal move to make and an explanation based on a small set of rules for playing the game. For any move, the explanation can be (in order of preference):
2. block (if the move prevents the other player from winning) 3. threat (if the game could be won with the following move) 4. empty (otherwise).
This results in a dataset of 4,520 examples. There are 36 possible combinations of label and explanation, since each rule can apply to 9 di erent moves.
The Loan Repayment example contains 10,000 applications for credit with
two labels (good or bad payment performance). Explanations are not included in
the original dataset. In [
Table 1 shows the accuracy in the Tic-Tac-Toe and Loan Repayment
examples reported in [
In the Cartesian product instantiation the result is simply a larger number of classes and therefore a smaller number of examples per class compared to the same dataset without added explanations. This means that the underlying classi er in a simple TED instance has to learn more from less. Intuitively there should be a decrease in performance.
The addition of explanations to an existing dataset can change the e ective class balance. Even if the original class labels are left untouched, it is possible to have a di erent number of explanations for each class. In that case the combined Y E learned by the classi er will not have the same distribution of classes as the original dataset without explanations. This can introduce or change existing bias in the training data.
It seems plausible that the improved performance which TED shows (in comparison to an equivalent classi er trained without explanations) is at least partially due to this altered class balance. Section 4 attempts to test this idea. 4
In this section TED is evaluated on two additional datasets: the well-known Iris
ower dataset [
TED is included in IBM AIX360 [
Three classi ers were chosen for these experiments: a Support Vector Machine (SVM) classi er, a Decision Tree classi er, and a DummyClassi er. The DummyClassi er is used with the strategy parameter set to \strati ed". It makes predictions by choosing a label randomly, considering the class distribution of the training data. This makes bias through class imbalance obvious, since the DummyClassi er does not have any other biases.
All experiments in this section were executed using AIX360 version 0.2 and scikit-learn version 0.21. 4.1
The Iris ower dataset consists of 150 examples of 3 types of owers (iris setosa, iris versicolor, iris viginica ). The 3 classes are equally distributed, i. e. there are 50 examples per class. There are 4 real-valued features for petal/sepal length/width.
Proper manual annotation would require domain knowledge. Since this requirement could not be ful lled for this paper, a decision tree (standard parameters, no maximum depth) was trained on the full dataset. This over tted decision tree perfectly predicts every example in the dataset. The leaves (identi ed by the node number) of this tree were used as explanations (see gure 1).
This results in a total of 9 explanations: { 1 for iris setosa { 3 for iris versicolor { 5 for iris virginia These are also all possible Y E combinations.
node #4
petal width (cm) <= 1.65
samples = 48
value = [
node #0 petal width (cm) <= 0.8
samples = 150 value = [50, 50, 50]
class = setosa True
False node #1 samples = 50 value = [50, 0, 0] class = setosa setosa, versicolor, and virginica samples in the sub-tree.
Unlike the labels, the combined labels and explanations are not equally distributed (see table 2).
An SVM classi er (default parameters), a decision tree (maximum depth of
2 to prevent over tting), and a DummyClassi er were trained using a 33% test
data split (i. e. 100 training examples). The results of the classi ers without
balanced class weights can be seen in table 3. The accuracy of the SVM and decision
tree are consistent with the evaluation in [
The DummyClassi er shows a surprising increase in accuracy. Without explanations, labels are predicted with 34% accuracy, which is to be expected from a classi er that chooses randomly from 3 equally distributed labels. With explanations, however, the performance increases by over 10 percentage points. With balanced class weights, the SVM and decision tree yield noticably different results (see table 4). Accuracy without explanations is the same. With explanations the prediction performance for both labels and explanations decreases signi cantly. Label accuracy drops by 12{30 percentage points, and the prediction of explanations is only about two-thirds as good.
The Iris dataset is admittedly quite small. The next section repeats the same experiments on a larger, synthetic dataset. 4.2
The TED source code in the IBM AIX360 repository (https://github.com/IBM /AIX360/) includes a synthetic dataset that is used in the Proactive Retention tutorial. The intended use case is nding employees who are at risk of leaving a company.
The data is generated according to distribution functions. There are 8 features: Each feature has a non-uniform distribution from which the values are randomly sampled.
There are two labels, Yes and No, expressing whether proactive retention is necessary for a given employee or not. Explanations are generated by applying 25 rules based on the features. If any of these rules applies to an example, it is labeled Yes with the rule number as explanation. If no rule applies, the example is labeled No. The dataset in the TED source code contains 10,000 examples of which 33.8% are labeled Yes.
The SVM classi er was trained with default parameters. The decision tree was trained with a maximum depth of 5.
The results are similar to the ones in section 4.1 with regard to class balance (see table 5), but more in line with the expectation stated in section 3. Without balanced class weights, the TED-augmented versions of the classi ers perform worse than their counterparts without explanations by about 6 percentage points. The DummyClassi er improves very slightly in label prediction. The accuracy for explanations is at roughly the same level as for labels.
With balanced class weights, however, both the SVM and the decision tree
again perform much worse. Label prediction performance decreases by 18{40
percentage points.
This paper performs additional experiments based on the work by [
The experiments show that class imbalance can be a problem. If training data, especially small datasets, is not carefully created or extended, the distribution of classes can become an issue. The simple Cartesian product instantiation of TED can be very sensitive to training data. Especially when augmenting existing data with explanations to make it compatible with TED, it can be easy to overlook the introduced or changed bias.
Using balanced class weights to counteract the bias in the class distribution of the training data results in a large decrease in performance in both experiments (see section 4). This shows that at least some of the accuracy increases in label prediction that TED achieves compared to equivalent classi ers trained without explanations may be based on the aforementioned bias.
Since manual work on datasets is time-consuming, methods that
automatically generate explanations for existing datasets or create synthetic datasets
from scratch are very appealing. Such methods have been used in the original
work by [
Prof. Dr. Ute Schmid, head of Cognitive Systems Group at University of Bamberg, held a seminar on Explainable AI during the winter semester 2019/2020 out of which this paper emerged.