In AI and law, systems that are designed for decision support should be explainable when pursuing justice. In order for these systems to be fair and responsible, they should make correct decisions and make them using a sound and transparent rationale. In this paper, we introduce a knowledge-driven method for model-agnostic rationale evaluation using dedicated test cases, similar to unit-testing in professional software development. We apply this new method in a set of machine learning experiments aimed at extracting known knowledge structures from arti cial datasets from ctional and non- ctional legal settings. We show that our method allows us to analyze the rationale of black box machine learning systems by assessing which rationale elements are learned or not. Furthermore, we show that the rationale can be adjusted using tailor-made training data based on the results of the rationale evaluation.
In AI and Law, explainability is a key requirement in system design, due to
the need for the justi cation of decisions. For machine-supported decisions, this
is nowadays encoded in the GDPR's right to explanation. Four types of
explanations are distinguished [Miller, 2019] and have been applied in AI and
Law [At
This requirement of explainability is problematic for the application of
central machine learning techniques in law. Neural networks, for example, are known
to perform well, but behave like a black box algorithm. Hence, explanation
techniques have been developed to `open the black box'
In this paper, we expand upon the method introdu
To measure and possibly improve rationale discovery, we create dedicated test datasets, on which a machine learning system can only perform well if it has learned a particular component of the knowledge structure that de ned the data. The idea is similar to how unit testing works in professional software development: we de ne a set of cases, targeting a speci c component, in which we know what the answer should be, and compare that to the output that the system gives.
To be able to focus on what is methodologically feasible, we do not use
natural language corpora
Our work builds on a study investigating whether neural networks are able
to tackle open
In the upcoming sections,
For each of the three domains considered in this paper, this section describes the underlying knowledge structure using logic, from which we will generate datasets to train a series of neural networks. These networks will subsequently be analysed using a method we propose for assessing the quality of their rational discovery. To this end we need two types of datasets for the purpose of testing. The rst are standard test sets sampled from the complete domain to evaluate the accuracy of the networks. The second type is a dedicated test set designed to target a speci c aspect of the domain knowledge. This section describes all datasets we use. 2.1
Welfare bene t domain This ctional domain in
(Gender(x) = male ^ Age(x) 65) C2(x) () jCon1(x); Con2(x); Con3(x); Con4(x); Con5(x)j 4 C3(x) () Spouse(x) C4(x) () :Absent(x) C5(x) () :Resources(x) 3000 C6(x) () (T ype(x) = in ^ Distance(x) < 50)_
(T ype(x) = out ^ Distance(x) 50) That is, a person is eligible i he/she is of pensionable age (60 for a woman, 65 for a man), paid four out of the last ve contributions Coni, is the patient's spouse, is not absent from the UK, has capital resources not amounting to more than £3,000, and lives at a distance of less than 50 miles from the hospital if the relative is an in-patient, or beyond that for an out-patient.
The six independent conditions for eligibility are de ned in terms of 12 variables, which are the features of the generated datasets. These features and their possible values are shown in Table 1. In addition to these 12 features, the datasets will contain 52 noise features unrelated to eligibility, just as in the original experiment, giving a total of 64 features plus an eligibility label for each instance. All datasets are valid in the sense that the given eligibility labels follow from evaluating the 6 conditions above.
Simpli ed domain Experiments with di erent models for the welfare domain
all concluded that it was not possible to extract all six conditions for eligibili
Eligible(x) ()
C1(x) ^ C6(x) Eligibility is thus determined through a combination of a XOR-like function (C6) and a nuanced threshold function (C1).
Tort law domain Our third domain concerns Dutch tort law: articles 6:162 and 6:163 of the Dutch civil code that describe when a wrongful act is committed and resulting damages must be repaired. This `duty to repair' (dut) can be formalised as follows: dut(x) () c1(x) ^ c2(x) ^ c3(x) ^ c4(x) ^ c5(x) c1(x) () cau(x) c2(x) () ico(x) _ ila(x) _ ift (x) c3(x) () vun(x) _ (vst (x) ^ :jus(x)) _ (vrt (x) ^ :jus(x)) c4(x) () dmg (x) c5(x) () :(vst (x) ^ :prp(x)) where the elementary propositions are provided alongside an argumentative model of the law in Figure 1 [Verheij, 2017], and conditions c2 and c3 capture the legal notions of unlawfulness (unl ) and imputability (imp), respectively.
Compared to the ctional welfare domain, the Dutch tort law domain is captured in 5 conditions for duty to repair (dut ), based upon 10 Boolean features. Each condition is a disjunction of one or more features, possibly with exceptions. The feature capturing a violation of a statutory duty (vst ) is present in both condition c3 and c5, rendering these dependent. 2.2
For each experiment, we generate datasets of di erent types, for di erent
purposes3 For most types of datasets, the generating process is at least partly
stochastic and repeated for every repetition of an experiment. Using the same
3 The Jupyter notebooks used for data generation can be found in a Github repository:
https://github.com/CorSteging/DiscoveringTheRationaleOfDecisions
type of dataset, for example in training and testing a neural network, does
therefore not mean using the exact same dataset. Table 2 shows an overview of the
domains and their datasets, and illustrates their di erences and similarities.
Welfare bene t datasets Within this domain, four types of datasets are
generated, following the original s
Type A datasets are generated with either 2,400 instances or 50,000 instances. Exactly half of the instances are eligible, creating a balanced label distribution, as is common practice in machine learning problems. For the eligible instances, feature values are generated (randomly where possible) such that they satisfy the conditions C1 C6. For each condition, 61 th of the ineligible instances is designed to fail on that speci c condition; where possible the values of the features involved are generated randomly such that the condition fails. All remaining features in these instances are generated randomly across their full range of values (see Table 1); as a result, it is possible for ineligible instances to fail on multiple conditions, and some conditions will fail more often than others.
In the original 1993 study it was argued that it was too easy to achieve high accuracy scores with networks trained and tested on type A datasets, which contained an average of 4.1 conditions that were not satis ed for ineligible cases. Using only 4 out of 6 conditions was shown to be su cient for classifying 98.95% of the instances correctly.
Type B datasets were subsequently introduced to make the problem more challenging. These datasets di er from type A datasets only in that ineligible instances fail on exactly one condition, rather than at least one condition; the other ve conditions are always satis ed. Type B datasets again contain either 2,400 instances or 50,000 instances.
The original study investigated whether \an acceptable rationale can be
uncovered by an examina
The Age-Gender datasets are generated by sampling the age and gender features across their full range of values, this time considering only multiples of 5 for age. The values for the other features are generated such that every condition is satis ed except for the age-gender condition (C1). As a result, the eligibility of an instance in these datasets is solely determined by whether or not condition C1 is satis ed. The Age-Gender sets contain 40,000 instances, with every possible combination of values for age and gender occurring a 1000 times. This gives a slightly unbalanced label distribution with 42.5% of the instances being eligible, and 57.5% ineligible. Because the dataset is only used to test the networks, rather than to train the network, this is not an issue.
The Patient-Distance datasets are similarly generated by sampling the distance and patient type features across their full range of values, this time considering only multiples of 5 for distance. The eligibility of an instance in these datasets is thus determined by whether or not condition C6 is satis ed. The Patient-Distance datasets also contain 40,000 instances, with every possible combination of values for patient type and distance occurring a 1000 times. In these datasets, exactly 50% of the instances is eligible.
Simpli ed datasets For the simpli ed welfare domain, the same type of datasets are generated as above, with the same properties except that all noise features and 8 of the 12 actual features are excluded. For type B datasets, this means that ineligible instances fail on either C1 or C6, but not on both, while in type
A datasets the ineligible instances can fail on both conditions. Moreover, in the Age-Gender dataset the patient-distance condition C6 is always satis ed, and in the Patient-Distance dataset the age-gender condition C1 is always satis ed. Type A and type B datasets again contain 50,000 instances each. The AgeGender dataset now contains only two features and 4,242 instances, that is, one unique instance for every possible combination of age and gender. Likewise, the Patient-Distance dataset contains 3,234 unique instances.
Tort law datasets With 10 Boolean features there are 210 = 1024 possible unique cases that can be generated from the argumentation structure of the tort law domain in Figure 1. Each case has a corresponding outcome for dut, indicating whether or not there is a duty to repair someone's damages. We will again consider four types of datasets.
The unique dataset contains these 1024 unique instances for the 10 features plus the label. In this dataset, there are 912 instances where dut is false and 112 instances where dut is true (11%).
The regular type datasets are generated such that dut is true in exactly half of the instances. The sets are regular in the sense that balanced label distributions are common in machine learning problems. These regular datasets are generated by sampling uniformly from the subset of cases from the unique dataset, such that each possible case is represented equally within the 50/50 label distribution. In practice, only a subset of the possible cases is typically available and presented to a network, upon which the network will have to learn to generalize to all possible cases. In addition to generating regular type datasets with 5,000 cases, we therefore also generate smaller regular type datasets with only 500 instances; the latter contains 35.35% of the unique instances.
In the tort law domain we focus on the notions of unlawfulness (c2) and imputability (c3) to assess whether the networks are able to discover conditions in the data. For each of the two conditions, we again create a dedicated dataset.
The Unlawfulness dataset is the subset of the unique dataset in which the features for the unlawfulness condition c2 can take on any of their values, while the other features have values that are guaranteed to satisfy the remaining conditions.Whether or not there is a duty to repair is therefore solely determined by whether or not condition c2 is satis ed. All combinations of values of the other features are considered. The Unlawfulness dataset therefore consists of 168 unique instances, of which 66.66% have a positive dut value.
The Imputability dataset is a similar subset of the unique dataset, but now the features for the imputability condition (c3) can take on any value, provided that the value of vst is such that condition c5 is satis ed. The value of dut(x) now completely depends on whether or not condition c3 evaluates to true. Due to the interdependency of conditions c3 and c5, the Imputability dataset only has 128 unique instances, with 87.5% of them having a positive dut value.
Simpli ed welfare bene t Tort law
Regular (5,000 instances) Regular (500 instances)
In this section we describe and motivate the experiments we performed and
report on their results.
In order to demonstrate our method for assessing and improving rationale
discovery of models learned from data, we rst need such models. Though our
method is model agnostic, we choose to use neural ne
In this paper we will use a similar set-up and network architecture for all
three domains. The output is always a single node, representing either eligibility
or duty to repair, depending on the domain under consideration. The number of
input nodes corresponds to the number of features and is therefore dependent on
the domain (see Table 2). More speci cally, the welfare bene t domain will have
64 input nodes, the simpli ed domain will have 4, and the tort law domain will
have 10 input nodes. The node con guration (i.e. number of nodes per layer) of
each network is as follows, where input represents the number of nodes in the
input layer:
{ One hidden layer network: input -12-1
{ Two hidden layer network: input -24-6-1
{ Three hidden layer network: input -24-10-3-1
In
Training and performance testing The three types of neural networks will be trained and tested on a combination of di erent datasets, from each of the three domains. A complete overview of the datasets used in the experiments is shown in Table 3. This table shows the datasets that the networks will train on, and the datasets that the networks will be tested with. For each domain, every combination of training dataset and testing dataset is evaluated in terms of the accuracy of the resulting network on the test data. Because some of the datasets are stochastic (each generated dataset is slightly di erent), the whole process of data generation, training and testing is repeated 50 times. The mean classi cation accuracies along with their standard deviations will be reported.
To assess the rationale discovery capabilities of all the trained networks, we study their performance on the dedicated test sets for the age-gender, patientdistance, unlawfulness and imputability conditions. Performance will be measured both quantitatively, using standard accuracy, and qualitatively by a more detailed comparison of actual and expected outcomes. 3.2
We will rst report the accuracy scores for all combinations of training and testing datasets in the di erent domains and subsequently focus on rationale discovery. Results will be discussed in detail in Section 4. Accuracy Tables 5 { 8 show the mean classi cation accuracies over 50 runs, together with their standard deviations, for the di erent combinations of training and testing sets in the three domains. These tables include the quantitatively measured performance on the various dedicated test sets. Tables 5 and 6 present the results for the replication experiment with the original sizes of the Type A and Type B datasets and for the replication experiment with more data, respectively. The accuracies as reported in the orginal paper are provided in Table 4 [Bench-Capon, 1993]). Results for the simpli ed welfare bene t domain are shown in Table 7, and for the tort law domain in Table 8. Rationale discovery Each dedicated test dataset is designed to measure how well a model has learned a speci c condition from the domain. Since performance on these test sets in terms of accuracy is comparable for the di erent neural network architectures used, we present the results for the qualitative evaluation of their rationale discovery capabilities for what is in theory the most sophisticated one: the models with 3 hidden layers.
In the welfare bene t domain, the Age-Gender datasets are used to measure how well condition C1 is learned. In addition to measuring accuracy on these dedicated datasets, we can plot the actual output of the neural network, which should be 1 (eligible) for an individual of pensionable age and 0 otherwise, against age, for both values of gender. Such plots, showing the mean output of the networks over 50 runs, are shown in Figure 2 for both the welfare bene t domain and its simpli ed version, for networks trained on each of the training sets under consideration.
Similarly, the Patient-Distance datasets are used to evaluate how well condition C6 is learned, which can be assessed by plotting the networks' output against the distance to hospital, for both in-patients and out-patients. The plots showing the mean network output over 50 runs for both the welfare bene t do(A) Replication (B) More data (C) Simpli ed (D) Replication (E) More data (F) Simpli ed Fig. 3: For all training sets from (simpli ed) welfare domain: mean network output vs distance on Patient-Distance test set when trained on type A training sets (A-C) and on type B training sets (D-F). main and its simpli ed version, and for networks trained on each of the training sets under consideration, are shown in Figure 3.
In the tort law domain, we can similarly evaluate how well conditions c2 (unlawfullness) and c3 (imputability) are learned. For these conditions, the network should output 1 in cases of the Unlawfulness dataset where the case is unlawful (c2), or in the Imputability dataset where the case can be imputated to a person (c3); otherwise the output should be 0. Since the tort law domain only contains Boolean features, the outputs of the networks are presented in tables rather than plots. The mean output over 50 runs for the two training sets on the Unlawfulness and Imputability datasets is presented in Table 9. 4
In this section we discuss in detail the results we found and the conclusions we can draw from them. We separately focus on standard classi cation accuracy and on rational discovery capabilities. We conclude by introducing the approach we took as a general knowledge-driven method for model-agnostic rationale evaluation. Standard accuracy is measured to see whether the learned models are able to solve the classi cation problem, regardless of whether or not they discovered the rationale underlying the data.
Welfare bene t The accuracies obtained in the replication experiment (Table 5) di er from those in the original study (Table 4), but show similar trends. Originally, networks trained on a type A training set performed well on type A test sets (around 99%), but much worse on the type B test set (around 7076%). When trained on a type B training set, the accuracies on test set A in the original study stayed the same, with accuracies on test set B increasing to around 98%. In the replication experiment, training on a type B test set slightly decreases accuracies on type A test sets, while the accuracy on type B test sets increased less substantially than in the original experiment. In both cases, changing the distribution of the training data from type A to type B served to increase performance on test sets of the latter type while hardly a ecting performance on test sets of the former type. Since type B datasets exploited some knowledge of the domain (i.e. bene t is typically denied due to failure on only a single condition), this suggests that overall performance can be improved using tailor made training sets.
Using more data, we nd higher accuracies (Table 6). This is not surprising, as more training data generally leads to a better performance. Still, for networks trained on type A data sets, accuracies on a type B test sets (below 85%) are much lower than on type A test sets (around 99.8%). Training the networks on type B training sets with more data shows signi cantly better results than with fewer datapoints. Accuracies on type A test sets then are still around 99%, whereas the accuracies on type B test sets are around 98%. The above observations therefore still hold, even with more data and modern machine learning methods.
Simpli ed In the simpli ed domain, high accuracies are found across all datasets, averaging out at around 99% (Table 7). Accuracies on type B test sets are only slightly lower than on type A test sets, unlike in the other welfare domain experiments. The networks do seem to perform slightly better on both types of test sets when trained on a type B training set as compared to a type A training set. However, this di erence is much more nuanced than in the regular welfare bene t dataset. This can be explained by the fact that cases in type A data sets can now fail on at most 2 conditions, rather than the original 6, which is only 1 more than the single failed condition in the type B datasets.
Tort law In the Tort law domain we nd accuracies of 100% or near 100% for networks trained on all instances (see Table 8). When presented with all unique instances, the networks with one and two hidden layers are able to perfectly predict the outcome from the Dutch tort law, and the network with three hidden layers can create a very close approximation.
Presenting a neural network with all available cases is in practice often infeasible. If it is possible, then a simple lookup table rather than a neural network would most likely su ce. For this reason, we also trained the networks on a subset of only around 35% of the unique instances (see Table 8). As expected, the accuracies of the networks on the general test sets drop, but only slightly (to 98-99%). Even on the unique test set, accuracies remain around 96%. This suggests that it is possible to approximate tort law with a small subset of the unique cases. 4.2
Looking at the performance of the networks on the dedicated test sets partially exposes the rationale captured by the network. We designed these test sets such that each one targets a single condition from the domain. In addition to considering the accuracy on these dedicated test sets, we qualitatively evaluate the rational discovery capabilities of the networks by comparing their outputs with the actual outputs we would ideally expect for the di erent domains. Welfare bene t In the welfare bene t domain, the Age-Gender dataset is used to measure how well condition C1 is learned, that is, whether the networks output 1 if the individual is of pensionable age (male and over the age of 65 or female and over the age of 60), and output 0 otherwise. Plotting the age of the individuals from the Age-Gender dataset against the output of the network, for each gender, should ideally result in the graph on the left side of Figure 4. Here the output of the network spikes instantly from 0 to 1 at the age of 60 for women and 65 for men.
Similarly, for cases from the Patient-Distance dataset the networks should only output 1 (eligible) if the relative is an in-patient and the distance to the hospital is less than 50 miles, or if the relative is an out-patient and the distance to the hospital is further than 50 miles (condition C6). Plotting the distance against the output of the network for both types of patients would ideally result in the graph shown on the right in Figure 4.
In our replication experiment the output graphs show a similar pa
Upon repeating the replication experiment with more data, this indeed increases the performance of the networks signi cantly, but we still nd performance for networks trained on type B datasets to be better than that of networks trained on type A datasets. Output patterns more closely resemble the ideal ones after training on type B datasets (see Figure 2(B) versus (E) and Figure 3(B) versus (E)) and accuracies also increase (see Table 6). Interestingly, the turning point for condition C1 does occur at the right place when training on more type B training data: at 60 for females and 65 for males (see Figure 2(E)). Simpli ed The simpli ed domain consists of only the two conditions C1 and C6, without any other conditions or noise variables. In this less complex version of the domain, overall performance is much higher, and conditions C1 and C6 are learned quite successfully. Figures 2(C) and (F), and 3(C) and (F), respectively, are very close to the ideal output graphs, with turning points in the correct places. This is also re ected in near perfect accuracy scores on Age-Gender and Patient-Distance datasets in Table 7. As argued before, the di erence between type A and type B datasets is much smaller than in the original domain, hence the results found for these to datasets are now quite similar.
Tort law Recall that in the tort law domain, on the Imputability dataset, networks should output 1 if the case can be imputated to the person, and 0 otherwise; on the Unlawfulness dataset, the networks should output 1 if the case is unlawful, and 0 otherwise. Table 9 shows well the networks were able to internalize the notions of unlawfulness and imputability. When trained on all instances, the mean output of the networks is 0 if a case is not unlawful, and 1 if it is, which is exactly what it should do. Networks trained on all instances attain a perfect score on the Imputability dataset as well. This can also be seen in Table 8, where the networks score 100% accuracy on the Unlawfulness and Imputability datasets after training on all instances.
With less data, however, accuracies drop to around 92-95% for the Unlawfulness dataset and 91-94% for the Imputability dataset. This accuracy may still seem high, but we should take into account the label distributions (66.6733.33% and 87.5-12.5%, respectively). Table 9 shows that networks still perform perfectly on cases in which the unlawfulness and imputability conditions evaluate to true. When the conditions are false, however, mistakes are made. The average output of networks on the Unlawfulness dataset increases to 0.018, which should be 0, meaning that it classi es some lawful cases as unlawful. In the Imputability dataset, the mean output increased more drastically to 0.875 when imputability is false. Meaning that in 87.5% of the instances in which the case cannot be imputed to a person, the network incorrectly decided that it should. This means that despite its high accuracy on the general test set, the networks largely ignored the concept of imputability. 4.3
Although our experiments and discussion focused on speci c example domains and neural networks, our approach for rationale evaluation can be seen as a general method independent of the machine learning algorithm applied. This paper therefore proposes a knowledge-driven method for model-agnostic rationale evaluation, consisting of three distinct steps: 1. Measure the accuracy of a trained system, and proceed if the accuracy is su ciently high; 2. Design dedicated test sets for rationale evaluation targeting selected rationale elements based on expert knowledge of the domain; 3. Evaluate the rationale through the performance of the trained system on these dedicated test sets.
The rst step is based on the assumption that e orts for assessing and possibly improving the rationale discovery capabilities of a learned model are only taken if the general performance of the model is already considered good enough. Here we assume performance is measured using accuracy, but other measures can be employed as well and the threshold of what is considered good enough may vary per domain and application.
The second step in our method depends on domain knowledge. Hence the method e ectively is a quantitative human-in-the-loop solution for rationale evaluation.
In the third step, performance is again evaluated, by now not only considering accuracy but also examining model output and expected output in terms of the dedicated test sets. Our examples have shown that the latter depend on the type of features involved.
Subsequently, the information gained by using this rationale evaluation method can be used to improve the rationale of the system by adjusting the training data accordingly, imposing sound rationale discovery.
The method does not currently specify how the dedicated test sets are constructed. We aim to further operationalize the rationale evaluation method by using information about the knowledge in the domain, and the distribution of examples, for instance building on Bayesian networks.
The work in this paper was inspired by Bench-Capon's 1993 paper that
investigated whether neural networks are able to tackle open texture problems.
The conclusions were that neural networks can perform very well on such
problems in terms of accuracy, even if some conditions from the domain are no
In this paper we rst replicated the original experiments as closely as possible
to verify that we can reproduce
We investigated three legal domains, in which neural networks were trained
on labelled cases and tasked with predicting unlabelled cases. We started o
with an arti cial domain from the literature, followed by a simpli ed adaptation
of that domain. Lastly, we investigated a real life domain as well. The results
indicate that the network are able to achieve high accuracies in each of the
three domains. The networks are therefore able to make the right decisions in
most cases, with accuracies averaging around 99% on type A or regular test
sets. This is how machine learning problems are usually evaluated. Using our
approach of rationale evaluation, however, we show that the networks do not
necessarily learn the conditions, despite their high accuracy scores. Performance
on the dedicated test sets, type B, Age-Gender and Patient-Distance dataset
show that the networks are unable to learn the conditions C1 and C6. This was
suggested in the original experimen
This study therefore rea rms the conclusions from previous work, while simultaneously introducing a model-agnostic method for assessing rationale discovery capabilities of machine learned black box models, using dedicated test datasets designed with expert knowledge of the domain. In future research, we aim to further detail and extend our method such that by employing it, the soundness of the rationale becomes tangible, and its quality can be asserted. Ultimately, based on this evaluation, the training data of the black-box systems can be altered to improve their rationale. Further expanding upon this design method will bring us closer to AI that is both explainable and responsible.