This work explores the eficacy of symbolic knowledge-extraction (SKE) techniques in identifying biases and unfairness within opaque predictive models. Logic rules extracted from black-box predictors make it possible to verify if decisions are influenced by protected or sensitive features. In particular, the identification of biased or unfair decisions can be achieved through the evaluation of if-then rules, detecting the inclusion of protected and/or sensitive information in the rules' precondition. The efectiveness of SKE in this regard is demonstrated here by conducting various simulations on a well-known data set for loan grant prediction. Our findings highlight the potential of SKE as a valuable tool to reveal biases and discrimination in opaque predictions, ultimately contributing to the pursuit of fair and transparent decision-making systems.
As predictive models become increasingly integrated into various domains, ensuring their
fairness and transparency is of paramount importance [
SKE ofers a promising avenue to extract interpretable logic rules from black-box models,
enabling a deeper understanding of decision-making [
The main objective of this paper is to demonstrate the efectiveness of SKE in identifying unfairness and discrimination within opaque predictions. To achieve this, we employ a wellknown classification data set aimed at predicting loan grants. We conduct various simulations to illustrate how SKE can be exploited to extract logic rules and evaluate their fairness implications. Through these examples, we aim to shed light on the potential of SKE as a practical tool for highlighting biases and promoting fairness in predictive modelling.
By revealing biases and discrimination present in opaque predictive models, this research contributes to the broader discourse on fairness, accountability, and transparency in algorithmic decision-making. Understanding and rectifying biases in these models are crucial steps towards building equitable systems that mitigate the perpetuation of societal inequalities. The insights gained from this study serve as a foundation for developing strategies to enhance fairness in predictive models and promote the responsible deployment of artificial intelligence (AI) in critical domains.
In the following sections, we will discuss the methodology employed for SKE, present the results of our experiments, and discuss the implications and potential future directions of this research. By critically examining the power of SKE in identifying biases, we hope to provide practical insights and actionable recommendations for researchers, practitioners, and policymakers working towards fair and transparent predictive models.
Several studies have explored diferent approaches and methodologies to address bias in AI.
One line of research focuses on rule-based techniques for bias detection and explanation
[
Another area of related work involves the use of fairness-aware machine learning techniques
[
Furthermore, researchers have explored post-hoc methods to detect and mitigate biases in predictive models [15, 16]. These methods involve analysing the outcomes of model predictions on diferent subgroups defined by sensitive attributes, such as race, gender, or age. By quantifying and comparing the disparities in prediction outcomes across subgroups, these techniques can help identify and address discriminatory behaviour in models.
SKE techniques, including rule extraction and logic rule analysis, have been used in various domains to interpret and understand black-box models [17, 18, 19, 20]. However, thus far, their specific application for bias and discrimination identification in opaque predictions has not gained much attention. The proposed research aims to contribute to this body of work by demonstrating the efectiveness of SKE in uncovering biases and discrimination and providing insights into its practical application for fairness assessment in predictive models.
Through a comprehensive review of existing related work, this paper will situate SKE methods within the broader context of bias detection and fairness assessment in predictive modelling. It will build upon and extend the current knowledge by showcasing the unique capabilities of SKE techniques in addressing biases and discrimination in opaque predictive models, thus contributing to the growing literature on fair and transparent algorithmic decision-making.
SKE is a methodology aiming to extract interpretable and logic rules from complex black-box models, enabling a deeper understanding of their decision-making processes. There are two main approaches within SKE: pedagogical and decompositional [21].
In the pedagogical approach, the focus is on extracting human-readable rules providing meaningful explanations of the model’s behaviour. These rules are often represented in if-then format, making them easily understandable by both humans and machines. The pedagogical approach prioritises generality, allowing stakeholders to gain insights into the decision criteria employed by any predictive model, even though the explanations may lose some of the underlying model’s complexity and performance.
On the other hand, the decompositional approach aims to decompose the black-box model into simpler, more interpretable sub-models or components that are typically easier to understand and analyse individually. The inner black-box structure is carefully analysed and the resulting explanations may be more adherent to the underlying model behaviour. However, these techniques are strictly tailored to narrow categories of predictors, thus lacking flexibility and generality.
Since both approaches generate intuitive explanations that can be easily communicated and understood by a broader audience, this work prioritises bias evaluations independent of the underlying predictive model. Therefore, we exploit pedagogical approaches as the main tools for our experiments.
In the following, we provide a summary of some state-of-the-art pedagogical SKE techniques – namely, GridEx, CART and CReEPy – ofering insights into the specific techniques employed in the experimentation section.
We leverage the implementations available within the PSyKE Python package2 [22, 23]. This
library encompasses all the aforementioned SKE implementations, allowing for their seamless
comparison and evaluation [24]. The PSyKE platform ofers a unified interface, enabling
the application, assessment, and comparison of various SKE techniques. Moreover, it is fully
compatible with other widely-used Python packages [
GridEx [
GridEx requires the following set of hyper-parameters to be defined by users: recursion depth defining the maximum number of recursions to perform during the knowledge extraction; splitting strategy to partition the input space. It may be fixed, if each input dimension is split into a fixed number of partitions, or adaptive if the number of splits depends on the relevance of the features; number of splits defining how many slices have to be performed along each input dimension; error threshold used to decide on which regions the recursive step of the algorithm has to be performed. In particular, only regions with a predictive error greater than the user-defined threshold are recursively split.
This set of parameters may be automatically tuned with the PEDRO procedure [
The most important parameters to consider for CART are: maximum depth defining the maximum allowed depth for the decision tree;
maximum number of leaves defining the maximum allowed number of tree leaves. These two parameters are intertwined and both the predictive accuracy and the humanreadability extent of the tree critically depend on them. In particular, deep trees usually exhibit higher predictive performance but smaller human-readability extent than shallow ones. The same holds for trees with a large number of leaves compared to trees with fewer leaves.
The CReEPy algorithm [
Suitable explainable clusterings adopted by CReEPy are CREAM [
To execute CReEPy users have to provide the parameters required by the underlying instance of ExACT or CREAM as well as an optional feature relevance threshold used to drop from the output Prolog rules all the antecedents involving input features with relevance below the threshold.
We selected the Loan data set3 as a case study to carry out experiments and verify if SKE techniques are efective tools to identify discriminative predictions provided by opaque models. The data set is composed of 11 input features representing relevant variables to decide if a loan should be granted or not. The final decision is the binary output feature. The data set is completed by an additional feature representing a unique identification code for each loan. The data set counts 614 instances. Only 480 have no missing values. The names, types, and values of the features are reported in Table 1.
In conducting the experiments presented in this study, instances in the data set that contained missing values were excluded, and nominal attributes were converted into discrete numeric features.
To evaluate the fairness of the data sets and opaque predictors, we employed the disparate
impact index [
The calculation of the disparate impact index involves grouping the instances in a data set into two subgroups: a privileged (or base) group and an unprivileged (or protected) group , typically afected by fairness concerns. Formally, = {︁ : = (1 , 2 , . . . , ) , }︁ where 1 , 2 , . . . , are the features of instance and = { : ∈ ∧ = ⊕} , = { : ∈ ∧ = ⊖} , by assuming that the sensitive feature have values in {⊕ , ⊖} , with = ⊕ representing the membership to the privileged group.
For each group, the ratio of positive outcomes to the total number of individuals is computed. Subsequently, the disparate impact index, denoted as , is defined as follows: ⃒⃒ {︀ : ∈ ∧ () = ⊙ ︀} ⃒⃒
| | ⃒⃒ {︀ : ∈ ∧ () = ⊙ ︀} ⃒⃒
| | where () represents the output of instance and ⊙ is the positive output.
In our experimental setup, we specifically focus on the scenario of gender discrimination ( = ). Consequently, we designate female individuals as the unprivileged group (⊖ = ) and male individuals as the privileged group (⊕ = ). This choice allows us to investigate and analyse potential biases and disparities that may afect females within the context of the studied predictive models. In our case study is a function denoting the approval or denial of a loan (therefore, () = corresponds to a positive outcome). The is accordingly defined as follows: ⃒⃒⃒ {︁ : ∈ ∧ }︁⃒ = ∧ () = ⃒⃒ ⃒⃒⃒ {︁ : ∈ ∧
}︁⃒ = ⃒⃒ ⃒⃒⃒ {︁ : ∈ ∧ }︁⃒ = ∧ () = ⃒⃒ ⃒⃒⃒ {︁ : ∈ ∧ ⃒
= }︁⃒⃒ .
(2)
As reported in the first row of Table 2, 394 out of 480 instances describe loans demanded by male applicants. The remaining 86 instances correspond to female applicants. Even if the gender attribute is not balanced, it is possible to observe that loans are fairly granted to female and male applicants. Indeed, 278 out of 394 male applicants receive the loan, as well as 54 out of 86 female applicants. This corresponds to the 71% and 63% of male and female applicants, respectively. By applying Equation (2) it is possible to find a disparate impact score of 0.89, corresponding to a quite fair situation. We recall here that = 1 denotes a perfectly fair situation. Lower score values are associated with unfair conditions. A score of 0.8 is usually considered the threshold to divide fairness ( > 0.8) from unfairness ( < 0.8). As a consequence of all these observations, we consider the Loan data set fair from the gender standpoint.
The distribution of the output feature of the data set with respect to the gender attribute is visually presented in Figure 1a. The x-axis represents the credit history input feature, which is considered the most significant for classification purposes. Gender is reported in the y-axis. The size of the circles corresponds to the number of instances in each subregion of the input feature space. Orange circles indicate granted loans, whereas green circles indicate denied loans.
A random forest (RF) classifier has been trained upon the Loan data set. The data set has been randomly split into training (85%) and test (15%) sets. The RF predictor was composed of 50 base decision trees having a maximum depth of 5 and achieved a classification accuracy equal to 0.79. The decision boundaries of the RF classifier are reported in Figure 1e as a bidimensional projection on the credit history and gender input features.
The RF can be considered a fair predictor since its disparate impact score is equal to 0.99 (cf. first row of the bottom part of Table 2). It is worth mentioning that fairness is not directly associated with classification accuracy. In this particular case, despite the RF classifier’s predictive performance not being excellent, it is noteworthy that it demonstrates a high level of fairness from a gender perspective. Fairness, in this context, refers to the absence of bias or discrimination based on gender, regardless of the classifier’s overall accuracy in making predictions.
The goal of our experiments is to demonstrate if SKE techniques can be used to detect unfair opaque predictors. To this purpose, we use the CART, GridEx and CReEPy algorithms to perform knowledge extraction on the trained RF classifier. Extractors have been parametrised as summarised in Table 3. The number of extracted rules, as a proxy of the human-readability extent of the models, and the fidelity measured for each extractor with respect to the RF predictions, expressed as classification accuracy, have been reported in Table 4. All extractors can achieve a fidelity of 0.99 with 2 rules.
The decision boundaries obtained via CART, GridEx and CReEPy are reported in Figures 1i, 1m and 1q, respectively. The corresponding Prolog rules are shown in Listings 1 to 3, respectively. (a) Loan data set (original, fair score).
(b) Loan data set (28% perturbed, unfair score).
(c) Loan data set (56% perturbed, unfair score).
(d) Loan data set (83% perturbed, unfair score). (e) RF (original, fair ).
(f) RF (28% perturbed, fair score).
(g) RF (56% perturbed, unfair score).
(h) RF (83% perturbed, unfair score). (i) CART (original).
(j) CART turbed).
(28% per(k) CART (56% turbed).
per(l) CART turbed).
(83% per(m) GridEx (original).
(n) GridEx (28% perturbed).
(o) GridEx (56% perturbed).
(p) GridEx (83% perturbed). (q) CReEPy (original).
(r) CReEPy (28% perturbed).
(s) CReEPy (56% perturbed).
(t) CReEPy (83% perturbed). Listing 1: Rules extracted with CART for the Loan data set (original and 28% perturbed data set). loan(Gender, Married, ..., CreditHistory, PropertyArea, NO) :
CreditHistory < 0.5. loan(Gender, Married, ..., CreditHistory, PropertyArea, YES). Listing 2: Rules extracted with GridEx for the Loan data set (original and 28% perturbed data set). loan(Gender, Married, ..., CreditHistory, PropertyArea, NO) :
CreditHistory in [0.00, 0.33]. loan(Gender, Married, ..., CreditHistory, PropertyArea, YES) :
CreditHistory in [0.67, 1.00].
Listing 3: Rules extracted with CReEPy for the Loan data set (original). loan(Gender, Married, ..., CreditHistory, PropertyArea, NO) :
CreditHistory in [0.00, 0.00]. loan(Gender, Married, ..., CreditHistory, PropertyArea, YES).
The three SKE algorithms reveal that the predictions made by the RF model are solely influenced by the credit history input feature. Irrespective of the applicants’ gender, loans are granted to individuals with a positive credit history (credit history = 1), while they are denied to those with a negative credit history (credit history = 0). The SKE techniques confirm the RF’s fair behaviour concerning the applicants’ gender, as the predictions are solely driven by the credit history attribute and are independent on gender.
Listing 4: Rules extracted with CReEPy for the Loan data set (28% perturbed).
To inject bias, we perturbed the output feature of the Loan data set, which was originally fair with respect to gender. The perturbation involved changing the loan status from ‘Yes’ to ‘No’ for a variable number of female applicants. Specifically, we conducted three diferent perturbations, modifying the positive loan outcome for 15, 30, and 45 female applicants. These numbers correspond to 28%, 56%, and 83% respectively, of the total female applicants who originally had a positive loan outcome in the unaltered data set.
The output feature distribution for the biased data sets is reported in Figures 1b to 1d and in the top part of Table 2. The corresponding disparate impact measurements are reported in the rightmost column of the same table. As expected, the score values decrease by increasing the introduced bias, down to 0.15 for the most perturbed data set. Each data set has been used to train an RF classifier with 50 base predictors having maximum depth equal to 5 and a measured predictive accuracy on the test set varying between 0.75 and 0.79 (cf. bottom part of Table 2). 4.3.1. 28% Perturbed Data Set The RF classifier trained upon the Loan data set with a perturbation involving 28% of the positive female applicants has = 0.79, even though the biased data set has a lower score ( = 0.64). This diference is due to the predictive error of the RF. There are no noticeable diferences in the decision boundaries of this RF compared to the one trained on the original Loan data set (cf. Figures 1e and 1f). Also CART, GridEx and CReEPy applied to the RF provide outputs similar to those obtained for the unbiased case study (see Figures 1j, 1n and 1r). The only diference is the Prolog theory obtained via CReEPy, having the same semantics as the unbiased counterpart, but diferent clauses. The theory is listed in Listing 4.
Also in this case the human-interpretable rules extracted via SKE techniques do not identify discriminative predictions based on gender for the RF classifier and this is in agreement with the corresponding disparate impact scores. 4.3.2. 56% Perturbed Data Set A diferent situation is evident if we modify the data set in order to refuse the loan to the 56% of female applicants that conversely should have received it. In this case, the disparate impact score drops to 0.40 for the data set and to 0.07 for the corresponding trained RF. These values highlight strong unfairness, especially for the RF predictions. The corresponding decision boundaries are reported in Figure 1g. It is clearly visible that the loan is granted only to male applicants having a positive credit history.
Decision boundaries obtained via CART, GridEx and CReEPy and the corresponding extracted rules expressed as Prolog theories are reported in Figures 1k, 1o and 1s and Listings 5 to 7, Listing 5: Rules extracted with CART for the Loan data set (56% perturbed). Listing 6: Rules extracted with GridEx for the Loan data set (56% perturbed).
Listing 7: Rules extracted with CReEPy for the Loan data set (56% perturbed).
Listing 8: Rules extracted with CART for the Loan data set (83% perturbed). respectively.
In this scenario, the credit history of the applicant remains the primary feature considered during the prediction phase of the RF model. For instance, the initial split in the input feature space performed by CART focuses on this attribute. However, the gender feature also plays a role in predicting the outcomes for a subset of instances, specifically those with a good credit history. As a result, the SKE techniques demonstrate their efectiveness in identifying unfair predictors by revealing the influence of the gender attribute on outcomes within specific credit history subgroups. 4.3.3. 83% Perturbed Data Set Finally, we report here the results obtained for the Loan data set with a perturbation involving 83% of the female applicants receiving positive outcomes. The disparate impact scores for this data set and the corresponding trained RF are equal to 0.15 and 0.01, respectively. The scores highlight severe unfairness. Decision boundaries identified by the RF, CART, GridEx Listing 9: Rules extracted with GridEx for the Loan data set (83% perturbed).
Listing 10: Rules extracted with CReEPy for the Loan data set (83% perturbed). and CReEPy are reported in Figures 1h, 1l, 1p and 1t, respectively. Prolog rules provided by the SKE techniques are reported in Listings 8 to 10.
The extracted rules clearly emphasise the significant reliance of the RF predictions on the gender feature. Despite the decision boundaries being the same as in the previous case study with a 56% perturbation, in this instance gender is employed as the primary feature for decisionmaking, followed by credit history as the secondary feature. Essentially, loans are primarily granted or denied based on gender, with credit history playing a secondary role. The SKE techniques efectively identify and reveal this unfair behaviour, presenting it to human users in the form of interpretable logic rules.
This paper provides preliminary insights into the value of leveraging SKE techniques for studying biases in AI predictors. The findings demonstrate the potential of SKE techniques, particularly in analysing the relationships between decision outcomes and sensitive input attributes. This work highlights the importance of considering the correlation between decisions and sensitive attributes, such as gender, and how SKE can efectively identify and highlight these dependencies.
Looking ahead, future research will focus on further testing and refining the proposed approach. This will involve exploring the application of SKE techniques with proxy variables, investigating intersectional discrimination, and employing counterfactual techniques. Additionally, the study will delve into the evaluation of diferent fairness metrics to gain a more comprehensive understanding of bias and discrimination within predictive models.
Merging the field of AI fairness with explainable AI seems to be a promising approach. By doing so, we can develop robust methodologies to mitigate biases and promote fairness in AI systems. The ongoing exploration of SKE techniques holds great promise in fostering a more equitable and unbiased landscape for AI decision-making.
This work has been supported by the EU ICT-48 2020 project TAILOR (No. 952215) and the European Union’s Horizon Europe AEQUITAS research and innovation programme under grant number 101070363. classifiers: A comprehensive survey, CoRR abs/2207.07068 (2022). URL: https://doi.org/10. 48550/arXiv.2207.07068. doi:10.48550/ARXIV.2207.07068. arXiv:2207.07068. [13] M. Hardt, E. Price, N. Srebro, Equality of opportunity in supervised learning, in: D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, R. Garnett (Eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5–10, 2016, Barcelona, Spain, 2016, pp. 3315–3323. URL: https://proceedings. neurips.cc/paper/2016/hash/9d2682367c3935defcb1f9e247a97c0d-Abstract.html. [14] J. Chakraborty, K. Peng, T. Menzies, Making fair ML software using trustworthy explanation, in: 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020, Melbourne, Australia, September 21–25, 2020, IEEE, 2020, pp. 1229–1233. URL: https://doi.org/10.1145/3324884.3418932. doi:10.1145/3324884.3418932. [15] D. Madras, T. Pitassi, R. Zemel, Predict responsibly: improving fairness and accuracy by learning to defer, Advances in Neural Information Processing Systems 31 (2018). [16] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, R. Zemel, Fairness through awareness, in: Proceedings of the 3rd innovations in theoretical computer science conference, 2012, pp. 214–226. [17] G. Bologna, C. Pellegrini, Three medical examples in neural network rule extraction,
Physica Medica 13 (1997) 183–187. URL: https://archive-ouverte.unige.ch/unige:121360. [18] L. Franco, J. L. Subirats, I. Molina, E. Alba, J. M. Jerez, Early breast cancer prognosis prediction and rule extraction using a new constructive neural network algorithm, in: Computational and Ambient Intelligence (IWANN 2007), volume 4507 of LNCS, Springer, 2007, pp. 1004–1011. doi:0.1007/978-3-540-73007-1_121. [19] R. Setiono, B. Baesens, C. Mues, Rule extraction from minimal neural networks for credit card screening, International Journal of Neural Systems 21 (2011) 265–276. doi:10.1142/ S0129065711002821. [20] F. Sabbatini, C. Grimani, R. Calegari, Bridging machine learning and diagnostics of the esa lisa space mission with equation discovery via explainable artificial intelligence, Advances in Space Research 74 (2024) 505–517. URL: https://www.sciencedirect.com/science/article/ pii/S0273117724003880. doi:https://doi.org/10.1016/j.asr.2024.04.041. [21] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, D. Pedreschi, A survey of methods for explaining black box models, ACM Computing Surveys 51 (2018) 1–42. doi:10.1145/3236009. [22] F. Sabbatini, G. Ciatto, R. Calegari, A. Omicini, Symbolic knowledge extraction from opaque ML predictors in PSyKE: Platform design & experiments, Intelligenza Artificiale 16 (2022) 27–48. URL: https://doi.org/10.3233/IA-210120. doi:10.3233/IA-210120. [23] R. Calegari, F. Sabbatini, The PSyKE technology for trustworthy artificial intelligence 13796 (2023) 3–16. URL: https://doi.org/10.1007/978-3-031-27181-6_1. doi:10.1007/ 978-3-031-27181-6_1, xXI International Conference of the Italian Association for Artiifcial Intelligence, AIxIA 2022, Udine, Italy, November 28 – December 2, 2022, Proceedings. [24] F. Sabbatini, C. Sirocchi, R. Calegari, Symbolic knowledge comparison: Metrics and methodologies for multi-agent systems, in: M. Alderighi, M. Baldoni, C. Baroglio, R. Micalizio, S. Tedeschi (Eds.), Proceedings of the 25th Workshop “From Objects to Agents”, Bard (Aosta), Italy, July 8–10, 2024, volume 3735 of CEUR Workshop Proceedings, CEUR-WS.org, 2024, pp. 202–216. URL: https://ceur-ws.org/Vol-3735/paper_17.pdf.