Deep learning has achieved state-of-the-art results in various application domains ranging from image recognition to language translation and game playing. However, it is now generally accepted that deep learning alone has not been able to satisfy the requirement of fairness and, ultimately, trust in Artiifcial Intelligence (AI). In this paper, we propose an interactive neural-symbolic approach for fairness in AI based on the Logic Tensor Network (LTN) framework. We show that the extraction of symbolic knowledge from LTN-based deep networks combined with fairness constraints ofer a general method for instilling fairness into deep networks via continual learning. Explainable AI approaches which otherwise could identify but not fix fairness issues are shown to be enriched with an ability to improve fairness results. Experimental results on three real-world data sets used to predict income, credit risk and recidivism in financial applications show that our approach can satisfy fairness metrics while maintaining state-of-the-art classification performance.
of XAI approaches with varied results, measured in diferent ways, towards achieving a better
understanding of the behaviour of black box AI and ML systems, often aimed at
discovering undesired model properties such as unfair treatment based on protected attributes. While
some approaches attempt to extract global decision-making information from a complex model
(known as the teacher model) by learning a simpler (student) model - starting with the
seminal TREPAN approach [
The contributions of this paper are:
(1) We introduce a method that allows one to act on information extracted by any XAI
approach in order to prevent the learning model from learning unwanted behaviour or bias
discovered by the XAI approach. We demonstrate how our method can leverage an existing XAI
method, known as SHAP [
(2) We implement and outline the use of LTN for continual learning and iterative querying by caching the learned representations and by using network querying in first-order logic to check for knowledge learned by the deep neural network.
(3) We apply the proposed method and tool to the field of quantitative fairness in finance.
Experimental results reported in this paper show comparable or improved accuracy across
three data sets while achieving fairness based on two fairness metrics, including a 7.1% average
increase in accuracy in comparison with a state-of-the-art neural network-based approach [
In Section 2, we position the paper in the context of the related work. In Section 3, we introduce the interactive continual-learning LTN method. In Section 4, we present and discuss the experimental results achieving fairness. In Section 5, we conclude the paper and discuss directions for future work.
Explainability: The goals of achieving comprehensible Machine Learning systems are diverse
[
Post-training methods seek to achieve explanation by approximating the behaviour of complex
black-box systems. One of the most prominent methods at present, based on this idea, is called
LIME, which stands for Local Interpretable Model-Agnostic Explanation [
A method that has gained traction recently in finance is the Shapley value approach. Initially
proposed by [
While the above XAI methods make a noticeable contribution to the obstacle of explainability, none of them address the problem of how one should act upon the extracted information. Consequently, we do not see the LTN approach as a method to be compared directly with the above XAI approaches but to complement them. By applying the neural-symbolic cycle multiple times, partial symbolic descriptions of the knowledge encoded in the deep network will be checked and, through a human-in-the-loop approach, incorporated into the cycle as a constraint on the learning process. This will enable an interactive integration of a desired behaviour, notably fairness constraints, by checking and incorporating knowledge at each cycle, instead of (global or local) XAI serving only to produce a one-of description of a static system. Fairness: One of the main goals of the recent advancements in explainability encompasses considerations of the fairness of automated classification systems. Although the discovery of such unwanted behaviour is essential and useful, in this paper, we can address specific undesired properties, discovered or specified symbolically, and alter the learned model towards a fairer description.
There have been a few methods addressing fair representation or classification: [
The above methods are related because they introduce constraints either on the data or the model during learning. The LTN-based approach used here introduces constraints as a regularisation which therefore may apply to any model or data set. Also, in LTN, additional fairness axioms can be specified during training time by the user, which may be unrelated to the existing fairness axioms. Finally, at test time, the protected variables defined by such axioms are not used, so that a final customer of the ML system will not be asked for sensitive information on gender, race, etc.
The framework used in this paper is that of Logic Tensor Networks [
(∀, ∃).
Logic Tensor Networks [
The semantics for the connectives is defined according to fuzzy logic semantics:
conjuncℝ as defined in
tions are approximated by t-norms (e.g.
(,
)
), disjunctions by t-conorms (e.g.
negation by fuzzy negation (e.g. 1− ) and implication by fuzzy implications (e.g.
The semantics of the quantifiers is defined by aggregation functions. For instance, in the
sen(1−, )
(, )
),
).
tence ∃ ( ( ) ∧ ( )), ∃ can be implemented using max and ∧. Krieken et al. [
The objects denoted by LTN constants and variables can be learned from data. LTN functions and predicates are also learnable. Thus, the grounding of symbols depend on a set of parameters . With a choice of a multilayer perceptron to model each logical predicate, the parametrization 2p-mean( 1, … , ) = ( =1 1 ∑ ) 1 . used in this paper is: (1)( ) = (1) ( (1) + (1)
) (2)( ) = (2) ( (1)( ) (2) + (2)
)
( )( ) = ( (2)( ) (3) + (3)
)
where each ( ) is an activation function, e.g. ReLU, ( ) is a weight matrix, and ( ) a bias
vector; denotes the sigmoid activation function which ensures that predicate is mapped
from ℝ to a truth-value in [
Since the grounding of a formula ( ) denotes the degree of truth of , one natural training signal is the degree of truth of the formulas in the knowledge-base . The objective function is therefore to maximize the satisfiability of all formulas in : ∗ = arg max Sat ( ())
∈ which is subject to an aggregation of all formulas, e.g. the above-mentioned p-mean.
Notice that in the above formulation we have substituted the Neural Tensor Network used
in [
LTN inference using first-order logic clauses is not a post-hoc explanation in the traditional sense. In this paper, we argue that inference should form an integral part of an iterative process allowing for incremental explanation through distillation of knowledge which is guided by data. We achieve this by computing the value of a grounding ( ), given a trained network (set of parameters ), for a user-defined query .
Specifically, we save and reinstate the learned parameters stored in the LTN implementation. This is done by storing the parameters resulting from ∗ = arg max ∈ Sat (). This also means that changes made to the knowledge-base followed by further training will not reinitialise parameters, but will instead start from saved ∗. Having this functionality allows us to continually query and guide the learning process according to added knowledge , an approach akin to that of continual learning.
A query is any logical formula expressed in first-order-logic. Queries are evaluated by
calculating the grounding of any formula whose predicates are already grounded in the multilayer
perceptron, or even by defining a predicate in terms of existing predicates. For example, the
logical formula ∀ ∶ ( ( ) → ( )) can be evaluated by applying the values of , obtained
from the data set, to the trained perceptron, obtaining the values of output neurons and
in [
Algorithm 1 illustrates the steps we take to continuously refine with a human-in-theloop. The queries derive from questions a user might have about the model’s response: how does the model behave for a specific group? How does the model behave for particular edgecases? These questions can be translated relatively easily into FOL-queries. Simultaneously, an XAI method further informs the user about possible undesired model behaviour which may not be as apparent as the above common questions. This can be accomplished by a variety of XAI methods which may give insight into the functionality of a black box model. In Figure 2, XAI method SHAP reports a discrepancy in how the variable reported income is used by the ML system for men and women. This can be changed by adding knowledge to and retraining, as will be illustrated in the experiments.
Quantitative fairness metrics seek to introduce mathematical precision to the definition of
fairness in ML. Nevertheless, fairness is rooted in ethical principles and context-dependent human
value judgements. This functional dependence on value judgements is perhaps manifested in
the existence of mutually incompatible definitions of fairness [
Definition 3.1. Demographic Parity (DP): A classifier ℎ satisfies demographic parity under a distribution on ( , , ) if its predictions ℎ( ) are independent of the protected attribute . That is, ∀ ∈ :
[ℎ( ) = ∣ = ] = [ℎ( ) = ] Since ∈ {0, 1}, we can say that:
∀ ∶ [ℎ( ) ∣ = ] = [ℎ( )] The metric itself is typically reported as the diference between the above expected values which should converge to zero for a fair classifier.
Definition 3.2. Disparate Impact (DI): Given ( , , disparate impact if: ) as specified above, a classifier ℎ has (ℎ( ) > 0 ∣ = 0)
≤
(ℎ( ) > 0 ∣ = 1)
Adopting the "80%-rule" from industry [
Algorithm 1: LTN-active learning cycle
Input: Data-set, Knowledge (in the form of FOL) Output: Model satisfiability measured as overall sat-level for each predicate in do
Initialize (P) for epoch < num-epochs do max sat ( )
while Revision do for each FOL-query do
Calculate ( ) if ( )< then
Add to Apply XAI-method for each predicate do
Inquire (P) if has undesired property ( ) then
Revise () to if ≠ ∅ then ← ∪ ∗ = arg max ∈ () // each can be a multilayer perceptron or output neuron // optimize to achieve max satisfiability of
// user-defined Boolean
// query the network to obtain the truth-value of
// in [
// we use Shapley values // query predicate-specific groundings // user-determined desiderata // method-dependent revision // re-train the network
The following experiments illustrate how LTN can be used to obtain insight into a neural
network model and interactively address an undesired behaviour by adding new knowledge to the
background knowledge as illustrated in Figure 2. Background knowledge is used to provide a
meaningful semantics to the explanations, facilitating human-machine interaction, while being
injected into the neural network to achieve a desired property [
Experiment 1. Fairness using objective features: This experiment draws a parallel with
a current state-of-the-art method in the area of explainability. We demonstrate how traditional
XAI methods are able to benefit from a neural-symbolic approaches. Most importantly, we
demonstrate how the LTN method can remove any undesired disparities in a model-agnostic
approach when having access to objective features as proposed in [
We use the same example as the authors of the popular SHAP library connecting XAI and
fairness [
We compare the demographic disparity between the gender groups by calculating their Shapley
values. We use such values as a popular way of gaining insight into model behaviour, although
other explainability methods could have been used, and show that one can intervene in the
model by adding knowledge for further training of the LTN to reduce disparities.
Since the SHAP method uses the same units as the original model output, we can decompose
the model output using SHAP and calculate each feature’s parity diference using their
respective Shapley value. Then, by adding clauses to LTN which seek to enforce equality as a soft
constraint, the neural network will be trained to reduce the diference (axioms 3-7 below).
Reapplying SHAP would then give a measure of the success of the approach on parity diference.
Axioms 3-7 are created based on the idea of treating similar people similarly from [
First, we split the data into two subsets for the protected ( ) and unprotected ( ) group, re4The code for the experiments can be found here: http://github.com/benediktwagner/LTN_fairness. 5The authors further advocate making such metric public to allow for transparency. They propose the use of a normative approach to fairness as the absolute guarantee of fairness.
Experiment 1 is summarised below.
Predicate: for the positive class (i.e. credit default) spectively, and create five subsets within each group, denoted quantile-based discretisation of customer quality. 6. The five axioms (3 to 7) then state, according to the discretisation, that if a member ( ) of set defaults on credit, i.e. ℎ( ) = 1, then a member ( ) of set
should also default, ℎ( ) = 1, and vice-versa. Given the diferent groups, one may wish to specify that equality in prediction is required, e.g. for the bottom 20% of the protected group w.r.t. the unprotected group according to a fairness measure. In our approach, the use of the generalised p-mean lends itself very well to this task by allowing for diferent forms of aggregation for each equality sub-group (referred to as customer quality 1 to 5 below). As a result, the user can specify in the system how strictly each fairness axiom is expected to be satisfied (using the p-mean parameter and the satisfiability threshold , c.f. Algorithm 1). and , 1 ≤ ≤ 5, using Training data: , a set of individuals who credit default; , a set of individuals who do not credit default; 1, ..., 5 ⊂ { ∪ }, a set of female individuals with customer quality 1 to 5; 1, ..., 5 ⊂ { ∪ }, a set of male individuals with the same customer quality. Axioms: ∀ ∈ ∶ ∀ ∈ ∶ ∀ ∈ 1, ∈ 1 ∶ ∀ ∈ 2, ∈ 2 ∶ ∀ ∈ 3, ∈ 3 ∶ ∀ ∈ 4, ∈ 4 ∶ ∀ ∈ 5, ∈ 5 ∶ ( ) ¬ ( ) ( ) ↔ ( ) ( ) ↔ ( ) ( ) ↔ ( ) ( ) ↔ ( ) ( ) ↔ ( ) (1) (2) (3) (4) (5) (6) (7) Model: ℎ( ) (denoting ( )) is the multilayer perceptron described in Section 3.3. We initially train the multilayer perceptron on the data alone (without axioms 3 to 7) and observe the disparities shown in the SHAP XAI chart shown in 3 (left). The network learned undesired disparities among gender groups as a result of under-reporting errors in the data (equivalent to using only axioms 1 and 2 with an LTN trained for 1000 epochs). We subsequently add axioms 3 to 7 to the knowledge-base and re-train the LTN with these axioms. As shown in the SHAP XAI chart of Figure 3 (right), this decreases disparity considerably, having reduced the Disparity Impact from 0.64 to less than 0.001. This illustrates the ability of LTN to account for fairness after having observed disparities in common XAI methods by adding appropriate fairness axioms for further training. Despite its usefulness as proof-of-concept, we acknowledge that the ideal notion of similarity among sub-groups can be impracticable. Next, we continue our investigation with additional real-world data and derive a notion of similarity automatically.
6This choice should be attribute-independent and application specific, e.g. based on reported income from very low, low and medium, to high and very high, diferent groups may require diferent interventions, i.e. diferent axioms according to policy and the situation in the real-world
Experiment 2. Fairness and Direct Comparisons: We compare results on three
publiclyavailable data sets used in the evaluation of fairness, obtained from the UCI machine learning
repository: the Adult data set for predicting income, German for credit risk, and COMPASS for
recidivism. We follow the experimental setup used in [
As before, we first train the network without fairness constraints, while before we relied on an objective notion of similarity that made it possible to split the individuals into sub-groups, now we use a continual learning approach (when such objective notion is not present). The trained network is queried to return the truth-value of the predicate used for the classification task ( ( )) for the entire training set . The output helps determine, as a proxy for similarity, the fairness constraints. As done in the previous experiment, a quantile-based discretisation is carried out, but this time according to the result of querying the network, after splitting the data into two subsets for each class according to the protected variable. Therefore, we obtain equally-sized groups for each protected and unprotected variable. Again, five sub-groups are used and the axioms from Experiment 1 apply.
Querying axioms 3 to 7 reveals a low sat level at first as an indication of an unfair model
( < 0.5). This is confirmed by measuring the fairness metrics with DI ≤ 0.4 and DP ≥ 0.03
across all data sets 7. The results are shown in Figure 4 which also includes the results of the
approach to account for fairness in naive Bayes classifiers [
7this is also revealed in SHAP value disparity. We measure the fairness here using pre-defined metrics as fairness is a known issue in these benchmarking datasets and therefore does not require an XAI method for detection
As illustrated in Figure 4, we are able to outperform other state-of-the-art methods and achieve
a lower variability across all data sets, and pass the DI and DP fairness thresholds proposed by
[
Combining XAI methods with neural-symbolic approaches allows us to not only learn about the undesired behaviour of a model but also intervene to address discrepancies which is ultimately the goal of Explainability. We have proposed an interactive model-agnostic method and algorithm for fairness and have shown how one can remove demographic disparities from trained neural networks by using a continual learning LTN-based framework. While experiment 1 demonstrated the efectiveness on addressing undesired gender-based disparities on simulated data, we have investigated such efectiveness on real-world data in experiment 2 and compared to other methods.
In the future, we plan to investigate the suitability of diferent XAI methods. In particular, knowledge extraction methods that extract rules directly by querying from general to specific knowledge, following the FOL language of LTN, could prove to be highly useful in practice. Furthermore, we aim to adapt this approach to varying notions of fairness as the proposed method itself is adaptable and there are lively discussions about the appropriateness of varying definitions.
We are thankful to Samy Badreddine, Michael Spranger and Luciano Serafini for their comments and useful discussions.