As predictive models become increasingly integrated into various domains, ensuring their fairness and transparency is of paramount importance [1]. Opaque predictive models in machine learning (ML), often referred to as black-box models, pose challenges in understanding the underlying mechanisms by which they make predictions. Consequently, biases and discrimination can inadvertently permeate these models, leading to unfair or prejudiced outcomes [2]. To address this critical issue, the present paper investigates the application of symbolic knowledge-extraction (SKE) techniques in uncovering biases and discrimination within opaque predictive models.
SKE offers a promising avenue to extract interpretable logic rules from black-box models, enabling a deeper understanding of decision-making [3,4]. By distilling complex model behaviours into human-readable rules, SKE facilitates the identification of specific conditions under which biases may arise. This approach proves particularly valuable when examining whether protected features play a role in decision-making since the presence of protected information in the preconditions of extracted rules can provide direct evidence of bias. The same considerations may also extend to sensitive features, e.g., those that are not protected themselves but are related to other features identified as protected (e.g., name or height, which allow ML models to infer race and/or gender of individuals; [5,6,7]). We point out that identifying correlations between protected/sensitive features and other input variables is not within the scope of SKE techniques, nor is the recognition of protected/sensitive attributes in the rule preconditions 1 . The classification of input features into "unfairness-enablers" and "potentially-fairness-neutral" should be performed by human users as an independent task.
The main objective of this paper is to demonstrate the effectiveness of SKE in identifying unfairness and discrimination within opaque predictions. To achieve this, we employ a wellknown classification data set aimed at predicting loan grants. We conduct various simulations to illustrate how SKE can be exploited to extract logic rules and evaluate their fairness implications.
Through these examples, we aim to shed light on the potential of SKE as a practical tool for highlighting biases and promoting fairness in predictive modelling.
By revealing biases and discrimination present in opaque predictive models, this research contributes to the broader discourse on fairness, accountability, and transparency in algorithmic decision-making. Understanding and rectifying biases in these models are crucial steps towards building equitable systems that mitigate the perpetuation of societal inequalities. The insights gained from this study serve as a foundation for developing strategies to enhance fairness in predictive models and promote the responsible deployment of artificial intelligence (AI) in critical domains.
In the following sections, we will discuss the methodology employed for SKE, present the results of our experiments, and discuss the implications and potential future directions of this research. By critically examining the power of SKE in identifying biases, we hope to provide practical insights and actionable recommendations for researchers, practitioners, and policymakers working towards fair and transparent predictive models.
Several studies have explored different approaches and methodologies to address bias in AI.
One line of research focuses on rule-based techniques for bias detection and explanation [8,9]. These studies aim to extract interpretable rules from black-box models and analyse them for potential biases. For instance, in [8] the authors have proposed algorithms mining association rules or decision trees to identify discriminatory patterns in the rule sets generated by predictive models. These approaches often leverage fairness criteria or sensitive attribute definitions to guide the rule extraction process.
Another area of related work involves the use of fairness-aware machine learning techniques [10,11]. These approaches aim to incorporate fairness considerations during the model training phase, ensuring that the resulting predictions are less likely to be biased. Fairness-aware algorithms often employ mathematical optimisation techniques to balance predictive accuracy and fairness objectives, taking into account various fairness definitions such as demographic parity [12], equalised odds [13], or individual fairness [14].
Furthermore, researchers have explored post-hoc methods to detect and mitigate biases in predictive models [15,16]. These methods involve analysing the outcomes of model predictions on different subgroups defined by sensitive attributes, such as race, gender, or age. By quantifying and comparing the disparities in prediction outcomes across subgroups, these techniques can help identify and address discriminatory behaviour in models.
SKE techniques, including rule extraction and logic rule analysis, have been used in various domains to interpret and understand black-box models [17,18,19,20]. However, thus far, their specific application for bias and discrimination identification in opaque predictions has not gained much attention. The proposed research aims to contribute to this body of work by demonstrating the effectiveness of SKE in uncovering biases and discrimination and providing insights into its practical application for fairness assessment in predictive models.
Through a comprehensive review of existing related work, this paper will situate SKE methods within the broader context of bias detection and fairness assessment in predictive modelling. It will build upon and extend the current knowledge by showcasing the unique capabilities of SKE techniques in addressing biases and discrimination in opaque predictive models, thus contributing to the growing literature on fair and transparent algorithmic decision-making.
SKE is a methodology aiming to extract interpretable and logic rules from complex black-box models, enabling a deeper understanding of their decision-making processes. There are two main approaches within SKE: pedagogical and decompositional [21].
In the pedagogical approach, the focus is on extracting human-readable rules providing meaningful explanations of the model's behaviour. These rules are often represented in if-then format, making them easily understandable by both humans and machines. The pedagogical approach prioritises generality, allowing stakeholders to gain insights into the decision criteria employed by any predictive model, even though the explanations may lose some of the underlying model's complexity and performance.
On the other hand, the decompositional approach aims to decompose the black-box model into simpler, more interpretable sub-models or components that are typically easier to understand and analyse individually. The inner black-box structure is carefully analysed and the resulting explanations may be more adherent to the underlying model behaviour. However, these techniques are strictly tailored to narrow categories of predictors, thus lacking flexibility and generality.
Since both approaches generate intuitive explanations that can be easily communicated and understood by a broader audience, this work prioritises bias evaluations independent of the underlying predictive model. Therefore, we exploit pedagogical approaches as the main tools for our experiments.
In the following, we provide a summary of some state-of-the-art pedagogical SKE techniques -namely, GridEx, CART and CReEPy -offering insights into the specific techniques employed in the experimentation section.
We leverage the implementations available within the PSyKE Python package2 [22,23]. This library encompasses all the aforementioned SKE implementations, allowing for their seamless comparison and evaluation [24]. The PSyKE platform offers a unified interface, enabling the application, assessment, and comparison of various SKE techniques. Moreover, it is fully compatible with other widely-used Python packages [25], such as Scikit-Learn [26], and provides additional extensions for SKE [27] and functionalities for feature engineering, data manipulation and visualisation, Semantic Web compatibility [28], and assessment of knowledge quality [29,30].
GridEx [31] is a pedagogical SKE algorithm originally designed for regression tasks and based on hypercubic partitioning of the input feature space. The partitioning is recursive, symmetric and performed top-down to obtain human-interpretable rules describing as many disjoint, hypercubic input space subregions. Thanks to the generalisation presented in [32,33], it is possible to apply GridEx to both classification and regression tasks if they are encoded via data sets having only continuous input features.
GridEx requires the following set of hyper-parameters to be defined by users:
recursion depth defining the maximum number of recursions to perform during the knowledge extraction;
splitting strategy to partition the input space. It may be fixed, if each input dimension is split into a fixed number of partitions, or adaptive if the number of splits depends on the relevance of the features; number of splits defining how many slices have to be performed along each input dimension;
error threshold used to decide on which regions the recursive step of the algorithm has to be performed. In particular, only regions with a predictive error greater than the user-defined threshold are recursively split.
This set of parameters may be automatically tuned with the PEDRO procedure [34].
The CART algorithm [35] is based on the induction of a classification or regression binary decision tree. It may be directly applied to a data set to build a human-interpretable predictor (if the induced tree is not deep) or it may be adopted as an SKE technique to produce humaninterpretable rules mimicking the behaviour of an opaque ML model. Human-interpretable rules are obtained by reading the complete paths from the tree root to each distinct leaf, given that internal nodes represent constraints on input variables and leaves contain output predictions.
The most important parameters to consider for CART are: maximum depth defining the maximum allowed depth for the decision tree; maximum number of leaves defining the maximum allowed number of tree leaves.
These two parameters are intertwined and both the predictive accuracy and the humanreadability extent of the tree critically depend on them. In particular, deep trees usually exhibit higher predictive performance but smaller human-readability extent than shallow ones. The same holds for trees with a large number of leaves compared to trees with fewer leaves.
The CReEPy algorithm [36,37] is a pedagogical SKE technique applicable to opaque classifiers and regressors. It relies on underlying explainable clustering procedures aimed at identifying hypercubic human-interpretable regions within the input feature space [38,39]. At the end of the knowledge extraction, each hypercubic region is translated into a Prolog rule describing the boundaries of the region and the corresponding output prediction. Suitable explainable clusterings adopted by CReEPy are CREAM [40] and ExACT [41]. They both perform hierarchical clustering according to different recursive strategies and require the following parameters, possibly tuned with the OrCHiD automated procedure [40]: recursion depth defining the maximum number of performed recursions; maximum number of Gaussian components defining the maximum number of components to use in the Gaussian mixture model clustering performed within ExACT and CREAM;
error threshold used to pre-emptively stop the recursive clustering when clusters exhibit a predictive error smaller than the threshold.
To execute CReEPy users have to provide the parameters required by the underlying instance of ExACT or CREAM as well as an optional feature relevance threshold used to drop from the output Prolog rules all the antecedents involving input features with relevance below the threshold.
We selected the Loan data set3 as a case study to carry out experiments and verify if SKE techniques are effective tools to identify discriminative predictions provided by opaque models. The data set is composed of 11 input features representing relevant variables to decide if a loan should be granted or not. The final decision is the binary output feature. The data set is completed by an additional feature representing a unique identification code for each loan. The data set counts 614 instances. Only 480 have no missing values. The names, types, and values of the features are reported in Table 1. In conducting the experiments presented in this study, instances in the data set that contained missing values were excluded, and nominal attributes were converted into discrete numeric features.
To evaluate the fairness of the data sets and opaque predictors, we employed the disparate impact index [42]. This metric measures the extent of differential treatment between two distinct groups, specifically by quantifying the proportion of individuals from each group who receive positive outcomes. The disparate impact index serves as a quantitative measure of the disparate treatment experienced by individuals from different classes.
The calculation of the disparate impact index involves grouping the instances in a data set ๐ฎ into two subgroups: a privileged (or base) group ๐ฎ ๐ and an unprivileged (or protected) group ๐ฎ ๐ , typically affected by fairness concerns. Formally, For each group, the ratio of positive outcomes to the total number of individuals is computed. Subsequently, the disparate impact index, denoted as ๐ท๐ผ, is defined as follows: where ๐พ(๐ฅ ๐ ) represents the output of instance ๐ฅ ๐ and โ is the positive output.
In our experimental setup, we specifically focus on the scenario of gender discrimination (๐ = ๐๐๐๐๐๐). Consequently, we designate female individuals as the unprivileged group (โ = ๐ ๐๐๐๐๐) and male individuals as the privileged group (โ = ๐๐๐๐). This choice allows us to investigate and analyse potential biases and disparities that may affect females within the context of the studied predictive models. In our case study ๐พ is a function denoting the approval or denial of a loan (therefore, ๐๐๐๐(๐ฅ) = ๐ฆ๐๐ corresponds to a positive outcome). The ๐ท๐ผ is accordingly defined as follows:
As reported in the first row of Table 2, 394 out of 480 instances describe loans demanded by male applicants. The remaining 86 instances correspond to female applicants. Even if the gender attribute is not balanced, it is possible to observe that loans are fairly granted to female and male applicants. Indeed, 278 out of 394 male applicants receive the loan, as well as 54 out of 86 female applicants. This corresponds to the 71% and 63% of male and female applicants, respectively. By applying Equation (2) it is possible to find a disparate impact score of 0.89, corresponding to a quite fair situation. We recall here that ๐ท๐ผ = 1 denotes a perfectly fair situation. Lower score values are associated with unfair conditions. A score of 0.8 is usually considered the threshold to divide fairness (๐ท๐ผ > 0.8) from unfairness (๐ท๐ผ < 0.8). As a consequence of all these observations, we consider the Loan data set fair from the gender standpoint.
The distribution of the output feature of the data set with respect to the gender attribute is visually presented in Figure 1a. The x-axis represents the credit history input feature, which is considered the most significant for classification purposes. Gender is reported in the y-axis. The
A random forest (RF) classifier has been trained upon the Loan data set. The data set has been randomly split into training (85%) and test (15%) sets. The RF predictor was composed of 50 base decision trees having a maximum depth of 5 and achieved a classification accuracy equal to 0.79. The decision boundaries of the RF classifier are reported in Figure 1e as a bidimensional projection on the credit history and gender input features. The RF can be considered a fair predictor since its disparate impact score is equal to 0.99 (cf. first row of the bottom part of Table 2). It is worth mentioning that fairness is not directly associated with classification accuracy. In this particular case, despite the RF classifier's predictive performance not being excellent, it is noteworthy that it demonstrates a high level of fairness from a gender perspective. Fairness, in this context, refers to the absence of bias or discrimination based on gender, regardless of the classifier's overall accuracy in making predictions.
The goal of our experiments is to demonstrate if SKE techniques can be used to detect unfair opaque predictors. To this purpose, we use the CART, GridEx and CReEPy algorithms to perform knowledge extraction on the trained RF classifier. Extractors have been parametrised as summarised in Table 3. The number of extracted rules, as a proxy of the human-readability extent of the models, and the fidelity measured for each extractor with respect to the RF predictions, expressed as classification accuracy, have been reported in Table 4. All extractors can achieve a fidelity of 0.99 with 2 rules.
The decision boundaries obtained via CART, GridEx and CReEPy are reported in Figures 1i, 1m (e) RF (original, fair ๐ท๐ผ).
(f) RF (28% perturbed, fair ๐ท๐ผ score).
(g) RF (56% perturbed, unfair ๐ท๐ผ score).
(h) RF (83% perturbed, unfair ๐ท๐ผ score).
(i) CART (original).
(j) CART (28% perturbed).
(k) CART (56% perturbed).
(l) CART (83% perturbed).
(m) GridEx (original).
(n) GridEx (28% perturbed).
(o) GridEx (56% perturbed).
(p) GridEx (83% perturbed).
(q) CReEPy (original).
(r) CReEPy (28% perturbed).
(s) CReEPy (56% perturbed).
(t) CReEPy (83% perturbed). Listing 3: Rules extracted with CReEPy for the Loan data set (original).
loan(Gender, Married, ..., CreditHistory, PropertyArea, NO) :-CreditHistory in [0.00, 0.00]. loan(Gender, Married, ..., CreditHistory, PropertyArea, YES).
The three SKE algorithms reveal that the predictions made by the RF model are solely influenced by the credit history input feature. Irrespective of the applicants' gender, loans are granted to individuals with a positive credit history (credit history = 1), while they are denied to those with a negative credit history (credit history = 0). The SKE techniques confirm the RF's fair behaviour concerning the applicants' gender, as the predictions are solely driven by the credit history attribute and are independent on gender.
Listing 4: Rules extracted with CReEPy for the Loan data set (28% perturbed). loan(Gender, Married, ..., CreditHistory, PropertyArea, YES) :-CreditHistory in [1.00, 1.00]. loan(Gender, Married, ..., CreditHistory, PropertyArea, NO).
To inject bias, we perturbed the output feature of the Loan data set, which was originally fair with respect to The perturbation involved changing the loan status from 'Yes' to 'No' for a variable number of female applicants. Specifically, we conducted three different perturbations, modifying the positive loan outcome for 15, 30, and 45 female applicants. These numbers correspond to 28%, 56%, and 83% respectively, of the total female applicants who originally had a positive loan outcome in the unaltered data set.
The output feature distribution for the biased data sets is reported in Figures 1b to 1d and in the top part of Table 2. The corresponding disparate impact measurements are reported in the rightmost column of the same table. As expected, the score values decrease by increasing the introduced bias, down to 0.15 for the most perturbed data set. Each data set has been used to train an RF classifier with 50 base predictors having maximum depth equal to 5 and a measured predictive accuracy on the test set varying between 0.75 and 0.79 (cf. bottom part of Table 2).
The RF classifier trained upon the Loan data set with a perturbation involving 28% of the positive female applicants has ๐ท๐ผ = 0.79, even though the biased data set has a lower score (๐ท๐ผ = 0.64). This difference is due to the predictive error of the RF. There are no noticeable differences in the decision boundaries of this RF compared to the one trained on the original Loan data set (cf. Figures 1e and 1f). Also CART, GridEx and CReEPy applied to the RF provide outputs similar to those obtained for the unbiased case study (see Figures 1j, 1n and 1r). The only difference is the Prolog theory obtained via CReEPy, having the same semantics as the unbiased counterpart, but different clauses. The theory is listed in Listing 4. Also in this case the human-interpretable rules extracted via SKE techniques do not identify discriminative predictions based on gender for the RF classifier and this is in agreement with the corresponding disparate impact scores.
A different situation is evident if we modify the data set in order to refuse the loan to the 56% of female applicants that conversely should have received it. In this case, the disparate impact score drops to 0.40 for the data set and to 0.07 for the corresponding trained RF. These values highlight strong unfairness, especially for the RF predictions. The corresponding decision boundaries are reported in Figure 1g. It is clearly visible that the loan is granted only to male applicants having a positive credit history.
Decision boundaries obtained via CART, GridEx and CReEPy and the corresponding extracted rules expressed as Prolog theories are reported in Figures 1k, 1o
In this scenario, the credit history of the applicant remains the primary feature considered during the prediction phase of the RF model. For instance, the initial split in the input feature space performed by CART focuses on this attribute. However, the gender feature also plays a role in predicting the outcomes for a subset of instances, specifically those with a good credit history. As a result, the SKE techniques demonstrate their effectiveness in identifying unfair predictors by revealing the influence of the gender attribute on outcomes within specific credit history subgroups.
Finally, we report here the results obtained for the Loan data set with a perturbation involving 83% of the female applicants receiving positive outcomes. The disparate impact scores for this data set and the corresponding trained RF are equal to 0.15 and 0.01, respectively. The scores highlight severe unfairness. Decision boundaries identified by the RF, CART, GridEx and CReEPy are reported in Figures 1h, 1l, 1p and 1t, respectively. Prolog rules provided by the SKE techniques are reported in Listings 8 to 10.
The extracted rules clearly emphasise the significant reliance of the RF predictions on the gender feature. Despite the decision boundaries being the same as in the previous case study with a 56% perturbation, in this instance gender is employed as the primary feature for decisionmaking, followed by credit history as the secondary feature. Essentially, loans are primarily granted or denied based on gender, with credit history playing a secondary role. The SKE techniques effectively identify and reveal this unfair behaviour, presenting it to human users in the form of interpretable logic rules.
This paper provides preliminary insights into the value of leveraging SKE techniques for studying biases in AI predictors. The findings demonstrate the potential of SKE techniques, particularly in analysing the relationships between decision outcomes and sensitive input attributes. This work highlights the importance of considering the correlation between decisions and sensitive attributes, such as gender, and how SKE can effectively identify and highlight these dependencies.
Looking ahead, future research will focus on further testing and refining the proposed approach. This will involve exploring the application of SKE techniques with proxy variables, investigating intersectional discrimination, and employing counterfactual techniques. Additionally, the study will delve into the evaluation of different fairness metrics to gain a more comprehensive understanding of bias and discrimination within predictive models.
Merging the field of AI fairness with explainable AI seems to be a promising approach. By doing so, we can develop robust methodologies to mitigate biases and promote fairness in AI systems. The ongoing exploration of SKE techniques holds great promise in fostering a more equitable and unbiased landscape for AI decision-making.
This work has been supported by the EU ICT-48 2020 project TAILOR (No. 952215) and the European Union's Horizon Europe AEQUITAS research and innovation programme under grant number 101070363.