The importance of an equal treatment of individuals by AI models drastically grows due to the demands of modern society. The potential discrimination or favoritism of specific groups of individuals is one of the common perspectives for the evaluation of model behavior. However, most of the available fairness tools require human intervention in the selection of subgroups of interest and therefore expert knowledge. In this paper we propose a new tool, the ASDF-Dashboard, which automates the process of subgroup fairness assessment. It automates the subgroup detection by applying a method based on unsupervised clustering algorithms and pattern extraction to ease the usage also for non-expert users.
The research, that was conducted in the past decades, has enabled a level of AI featuring complex systems of hundreds of possible applications and an ever growing interest in their further development. There has been a great efort in improving the developed techniques and algorithms with the goal of optimizing their performance. The more powerful and faster technologies available today facilitate the expressiveness and performance of such AI systems making them omnipresent and essential in the modern world. Machine learning methods already outperform humans in certain tasks and AI supported decision making is no longer a rarity in sensitive fields like finance or medicine. However, the consideration of the societal impact of such potentially life-changing decisions has become an increasingly important objective and, thus, it needs to be evaluated in a transparent and critical way. More precisely, AI systems must not only be designed and optimized for performance, accuracy and quality but also reckon with aspects like transparency, explainability or fairness.
Fair machine learning models have to provide an equal treatment to diferent individuals regardless of their sensitive characteristics, e.g., their genders, races or ethnics. It is crucial to ensure that no individual experiences discrimination or favoritism by the model choices as a consequence of their membership in a certain population. Nevertheless, it is challenging to test the behavior of a model for fairness against subgroups when considering the intersections of (sensitive) characteristics as this causes an exponentially large number of potentially discriminated or favored subgroups to test. Furthermore, it is not obvious, in general, which characteristics of a dataset or which intersections induce subgroups sufering discrimination by the model, and it usually is infeasible to test each possible intersectional subgroup for a fair treatment. Hence, an automated suggestion of subgroups for the fairness testing is desirable.
In this work we propose the ASDF-Dashboard tool for the automated subgroup fairness
analysis of binary classification models. It implements the previously contributed
methodology of automatic subgroup detection using an unsupervised clustering and a subsequent
entropy-based pattern extraction [
There exist quite some machine learning tools to support AI developers, data scientists and also end users to realized and understand a model’s behavior when presented the data. Such tools enable the enhancement of the model in the development process, deep analyses of AI systems under various criteria and features, and transparency to end users, that can get an idea on the processing of their data out of it. In particular, the latter point is increasingly interesting as our modern society demands for more transparency in AI and an equal treatment of individuals regardless of their gender or ethnics, for example. This directly leads to the concept of AI fairness, which can be tested and visualized using diverse tools. A common drawback of such supporting tools is that they are not designed for non-expert user lacking deeper knowledge and, thus, cannot perform the required interaction with the tool appropriately.
The Boxer [2] tool provides the functionality to analyze and compare models for their behavior on the same task in an interactive fashion. It is able to identify intersectional bias in the predictions of the models for subgroups of interest as specified by the tool user. This functionality is also ofered by the Fairkit-learn [ 3] toolkit in a similar way to monitor the performance and fairness of potentially discriminating models. Models for graph mining tasks can be investigated with a tool called FairRankVis [4], which allows to explore visualizations of the model fairness wrt. individuals and subgroups. Another approach is provided by the What-if tool [5] that performs a subgroup fairness analysis and automatically optimizes the classification threshold of the considered model based on the results of the fairness analysis. Morina et al. [6] developed a framework that delivers multiple intersectional fairness metrics and estimators. However, none of the previously mentioned tools or frameworks is able to perform a subgroup fairness analysis of a given model automatically as they all require human intervention when it comes to detecting the intersectional bias. In each of these tools, the user has to specify the subgroups of interest manually before a subgroup fairness metric is applied. Our approach, in contrast, facilitates the subgroup fairness analysis by an automated detection of subgroups for the assessment of the classifier’s fairness.
The FairVis tool [7] suggests subgroups to the user, that were detected automatically by clustering the data with the k-means clustering algorithm and extracting patterns of instance prototypes. The prototypes describe the makeup of the clusters and the corresponding patterns are obtained from the dominant features matching most of the subgroup members. This means that the aggregation into a cluster made most of the individuals of this subgroup having a uniform value for the dominant attribute, which is then extracted as pattern to match data in the whole dataset. Our ASDF-Dashboard extends the approach behind FairVis by ofering also diferent clustering algorithms for the initial subgroup detection and refining the pattern extraction by a more intuitive method to quantify the uniformity of a certain feature. Instead of ranking the features by their cluster feature entropy, we apply a configurable, global threshold to identify dominant features independent of the feature domains.
Another approach to automatically detect subgroups uses frequent-pattern mining on the dataset. The Divexplorer [8] tool searches possible patterns to evaluate diferences in the model’s behavior between subgroups and the whole population in the dataset. The search space of possible patterns is explored exhaustively while considering only patterns with a specific degree of support and dropping less supported patterns. The model fairness regarding the subgroups is then evaluated as the diference in the probability for prediction using FPR or FNR. Similarly, the DENOUNCER [9] tool generates possible patterns by traversing the pattern graph and searches for the most general patterns which have support above a given threshold and define subgroups where the model performs poorly (low accuracy). As the space of patterns grows exponentially with the number of features and highly depends on the complexity of the domains of the features, the detection of subgroups and the assessment of the model fairness wrt. to the detected subgroups can be very time consuming. Hence, the support thresholds need to be defined very carefully to prune the search space appropriately while also generating patterns inducing meaningful subgroups.
The ASDF-Dashboard automatically assesses the subgroup fairness of a binary classifier on a given dataset. To this end, the system detects subgroups in the data by computing a clustering of the data. The found clusters are then treated as subgroups themselves while alternatively general patterns are derived from the clusters. The obtained patterns also induce subgroups for an evaluation of the classifier’s fairness. This procedure facilitates the assessment as no set of protected attributes has to be predefined and the intersections of multiple protected attributes are covered implicitly.
Formally, we denote a dataset as = {1, . . . , } of instances over a set of attributes
= {1, . . . , } with the possible values ∈ ( ) for ∈ . Given a dataset and
a subset of protected attributes {1, . . . , } ⊆ , we define a pattern = (1, . . . , ) ∈
(1) × · · · × () over such that an instance = (1, . . . , ) satisfies if its
attribute values match the pattern values ( = for ∈ {1, ..., }). Then, partitions into a
protected subgroup = { ∈ | ⊨ } and an unprotected subgroup ¯ = { ∈ | ⊭
} = ∖ [
Many subgroup fairness metrics quantify the model fairness by using the values derived
from confusion matrices [10, 11] such as the positive predictive value (PPV) or the true positive
rate (TPR). Barocas et al. [12] further categorize subgroup fairness metrics by the three criteria
“independence”, “separation” and “suficiency” which relate to most of the proposed fairness
definitions. Regarding independence, a fair classifier satisfies non-discrimination if the classification
is statistically independent from the membership in the protected or unprotected subgroup. The
rate of acceptance (or denial), i.e., P(^ = 1 | ∈ ) (or ^ = 0), is then equal between the two
subgroups. Separation extends this category by also considering a potential correlation between
the subgroup membership and the ground-truth class such that the protected and unprotected
subgroup should experience equal TPRs and FPRs. Finally, suficiency requires an independence
of the probability for the ground-truth class given a positive or negative prediction. This results
in the same positive/negative predictive values for the protected and unprotected subgroup.
The ASDF-Dashboard computes three diferent subgroup fairness metrics for a broader analysis
and investigation of the classification model, namely, statistical parity, equal opportunity and
equalized odds. In the following, the formulas of these criteria are given in the context of our
notion of patterns and the induced protected and unprotected subgroups as introduced in [
Statistical parity (Eq. 1) is satisfied if the protected subgroup has the same chance for the prediction of a positive outcome (^ = 1) as the unprotected subgroup ¯ [11]: P(^ = 1 | ∈ ) = P(^ = 1 | ∈ ¯ ) (1)
This definition requires that a fair classifier predicts the favorable label ( ^ = 1) with a probability independently of the protected attribute values. The same is also implied for the unfavorable label (^ = 0) due to the complementary probability. However, if instances fall into multiple of the protected groups, statistical parity tends to magnify the bias of the classifier against them [13].
Equal opportunity (Eq. 2) judges a classifier based on the probability of giving instances of the favorable class ( = 1) a correct prediction, i.e., is assigned the favorable class label by classifier ^ . Formally, it is fulfilled if
P(^ = 1 | = 1, ∈ ) = P(^ = 1 | = 1, ∈ ¯ ). (2) Assuming equal opportunity, the TPRs for instances regardless of their subgroup membership have to coincide. From Equation 2 also follows that the probability of a false prediction of the unfavorable class given actually is a member of the favorable class has to be equal between the subgroups (FNR) as P(^ = 0 | = 1, ∈ ) = 1 − P(^ = 1 | = 1, ∈ ).
The equalized odds subgroup fairness metric extends the equal opportunity definition by additionally forcing the equality of the subgroup’s FPRs:
P(^ = 1 | = 0, ∈ ) = P(^ = 1 | = 0, ∈ ¯ ) (3)
Thus, equalized odds is satisfied if the probabilities of correct positive predictions (Eq. 2) and incorrect positive predictions (Eq. 3) are the same for the protected and unprotected subgroup.
The previously defined fairness criteria can be used to derive metrics that quantify the subgroup fairness of the binary classifier instead of enforcing the strict equality only. Hence, the model can then considered fair if the probabilities for an equal treatment are similar and unfair if they are not close. The ASDF-Dashboard relaxes the three fairness criteria as shown in Table 1. For and the probability for an instance of the protected subgroup is subtracted from the probability for an instance of the unprotected subgroup ¯ . The equalized odds metric is computed as the average of the equal opportunity metric and the diference between the probability for an incorrect positive prediction by the classifier on the unprotected and protected subgroup. These relaxations of the three fairness criteria are implemented as fairness metrics in the “AI Fairness 360” toolkit [14]. Alternatively, ratios of the subgroup probabilities can be computed, e.g., as applied in the -diferential fairness definitions [ 6, 15] of statistical parity, equal opportunity or equalized odds. Statistical parity ( ) = P(^ = 1 | ∈ ¯ ) − P(^ = 1 | ∈ ) Eq. opportunity Equalized odds ( ) = P(^ = 1 | = 1, ∈ ¯ ) − P(^ = 1 | = 1, ∈ )
1 ( ) = 2 [︀ P(^ = 1 | = 0, ∈ ¯ ) − P(^ = 1 | = 0, ∈ ) +
P(^ = 1 | = 1, ∈ ¯ ) − P(^ = 1 | = 1, ∈ )]︀
Whenever one of these fairness metrics given a pattern over dataset is close to zero, it means that the classifier produces fair results on individuals from the subgroup as they are treated similar to the rest of the population. Fairness metric values less than zero indicate a favoritism of the individuals from over the rest of the population due to a higher probability for a positive prediction. If the fairness metrics, in contrast, yield a value greater than zero, the classifier discriminates against individuals from the protected subgroup according to the underlying fairness definition.
The unsupervised task of detecting meaningful groups in a dataset can be performed by computing a clustering = {1, . . . , } that divides into such groups of similar instances. The groups are so-called clusters and a pair of instances 1 and 2, that belong to the same cluster, shares some similarity. The cluster structure and degree of similarity between the individuals in the same group depend on the clustering type, distance measure and parameter selection. Our ASDF-Dashboard computes such a clustering either in an automated fashion or controlled by the parameters the user specified. After the clusters are found, we employ out notion of the previously defined patterns and the induced protected and unprotected subgroups to assess the classifiers fairness.
Based on the clustering , a pattern can be extracted that partitions the dataset into
protected and unprotected subgroups according to the clusters. To this end, a clustering-based
pattern [
=1
As an alternative method we refined the clustering-based subgroup detection by deriving
more sophisticated patterns from the clusters [
∑︁
, = − log2 |( )| · ∈()
,, · log2 ︂( ,, ︂)
(5)
where is the size of and ,, denotes the number of instances ∈ that have
value for attribute . The closer , is to zero, the more instances with the same value for
feature are contained in the cluster and a single value for is found at all instances
if the , = 0. If , is close to 1, this indicates more variation in the feature values across
the instances in the cluster . The set of dominant features of a cluster is determined as
= { ∈ | , ≤ } for some threshold 0 ≤ ≤ 1. An entropy-based pattern
= (1, . . . , ) ∈ (1) × · · · ×
()
is then obtained for each cluster ∈ by extracting the most frequent values of each of
the dominant features ∈ . These patterns map all instances ∈ to the protected
subgroup that exactly match the most frequent values of the dominant features of cluster
. However, if all candidate features exceed the threshold , i.e., = ∅, no pattern can
be extracted. In contrast to the clustering-based patterns, the protected subgroup does not
exclusively contain individuals from but also other individuals from other clusters that match
the dominant attributes’ values. Here, the normalization of the feature entropy ensures that
an appropriate global threshold can be set ignoring difering sizes of the active domains of the
attributes throughout the dataset [
Consider a clustering-based pattern and an entropy-based pattern extracted from the same cluster ∈ of dataset . The two patterns induce the diferent protected subgroups and , respectively. Generally, there might be instances ∈ with ⊭ such that ∈ but ∈/ . On the contrary side, there might be also individuals ∈ from other clusters , ̸= that satisfy ⊨ and, thus, are member of the protected subgroup but not of . However, the protected subgroups of two entropy-based patterns and might share some individuals or even coincide due to the same dominant features and most frequent values in both and . This is not possible for the protected subgroups of two clustering-based patterns and as we assume a hard partitional clustering with disjoint clusters, i.e., ∩ = ∅. Two entropy-based patterns, in contrast, might be identical ( = ) which causes an induction of the same subgroup = .
Our ASDF-Dashboard1 implements the subgroup fairness analysis based on the proposed methodology of clustering- and entropy-based subgroups. It is hosted as a publicly accessible web application that supports an automatic, user-friendly subgroup fairness analysis and provides a broad visualization of the subgroup fairness results. Registered users can upload their datasets, that contain also the ground-truth labels as well as the labels obtained as the predictions of their binary classifier, to the system. The ASDF-Dashboard further provides a tabular view of each of the uploaded dataset that can be used to interactively browse the uploaded data using sorting and column filters to get a better insight into the structure of the data. Figure 1 shows an exemplary table for the COMPAS [16] dataset, which is provided in the FairVis [7] repository incl. the predicted labels but without the clustering labels. Each of the table rows corresponds to one defendant whose recidivism for a period of 2 years was predicted by a binary classifier.
To perform the subgroup fairness analysis, at least a dataset , the positive (favorable) class label (0 or 1) and the entropy threshold (0 ≤ ≤ 1) for the pattern extraction have to specified in the control tile, which is depicted at the top of Figure 2). Then, the ASDF-Dashboard can already compute the subgroup fairness of the given classfier automatically. Optionally, the user can also specify the categorical columns by selection, which need to be one-hot encoded for the computation of the clustering as the distances measures most commonly require vectors of numeric data as input. If no categorical attributes are selected, the system automatically detects them to apply the one-hot encoding. Furthermore, the fully numeric features of the selected dataset are then also scaled using the min-max normalization before computing the clustering. However, clustering a mixture of numeric and categorical attributes is very sensitive to the choice of algorithm and distance metric as there is no consensus on the optimal technique [17]. Therefore, we decided for the usual processing involving encoding and scaling. In addition to the automatic subgroup fairness calculation, the classifier’s fairness can also be evaluated manually by choosing a clustering algorithm and specifying its parameters. In case of the automatic fairness assessment, the SLINK clustering algorithm is applied to dataset = {1, . . . , } 1https://github.com/jeschaef/ASDF-Dashboard with the desired number of clusters 2 ≤ ≤ ⌊ √︀ 2 ⌋, which we estimate before by using the x-means clustering algorithm.
Figure 2 shows the subgroup fairness analysis on the COMPAS dataset with entropy threshold = 0.65, favorable class label 0 (not recidivism in two years) and the three categorical columns “c_charge_degree” (felony or misdemeanor), “race” (african-american, caucasian, asian, hispanic or other) and “sex” (female or male) using the SLINK clustering algorithm (agglomerative clustering with single linkage) with = 30. The average absolute values of statistical parity, equal opportunity, equalized odds, accuracy and diference between the subgroup and global accuracy (accuracy error) over both the clustering- (red) and entropy-based subgroups (blue) are shown by the radar chart in the left tile of the dashboard (Figure 2). The average absolute values give a good insight over violations of any of the fairness definitions by discrimination or favoritism throughout all the detected subgroups. The tile next to it displays the sizes of the clusters and entropy-based subgroups by bars. Here, it can be clearly seen that the both types of subgroups do not coincide in general as the sizes difer. Especially, the cluster 14 is much smaller than the entropy-based subgroup 14 , for instance, as 14 = (“Felony”, “AfricanAmerican”, “Male”) is a common attribute pattern in the dataset matching 1836 individuals.
To get an overview over the subgroups, the extracted entropy-based patterns are displayed in a table (Figure 3). Each row in the table corresponds to one entropy-based pattern extracted from cluster ∈ . The columns of the table represent all the features ∈ of the dataset that were found to be dominant in at least one of the clusters ∈ , i.e., , ≤ 0.65 for some ∈ {1, ..., }.The remaining features are not relevant for the entropybased patterns and, thus, not listed in the table. The column “id” identifies cluster and the corresponding entropy-based pattern . For example, the entropy-based subgroup 8 in Figure 3 is defined by 8 = (“Felony”, “Caucasian”, “Female”, − 1) for the set of dominant attributes 8 = {“c_charge_degree”, “race”,“sex”,“days_b_screening_arest”}. Each table row can also be expanded to show an embedded table listing the individual fairness metrics.
These fairness metrics can also be investigated for each cluster and entropy-based subgroup in the chart displayed in the left tile of Figure 4 individually. The individual fairness metrics are visualized on click on one of the subtables in the pattern table or on one of the cluster-/entropybased subgroup size bars. Here, the selected cluster and entropy-based subgroup are 0 and 0 , respectively. The bars reveal that they share a similar accuracy score of ≈ 65%. However, the tested classifier slightly discriminates the protected instances ∈ 0 as compared to the unprotected instances according to the three subgroup fairness metrics whereas it treats the protected instances wrt. the clustering-based subgroup 0 equally to the unprotected individuals. The five most discriminated or favored subgroups by subgroup fairness metric can be quickly found by the ranking chart in the right tile (Figure 4). Sorting the individual subgroup fairness values in ascending order yields the top five favored subgroups and the descending sorting order yields the most discriminated ones. Next to the three subgroup fairness metrics also the cluster or entropy-based subgroup accuracy values can be ranked in the same manner.
In our experiments [
As our previous experiments [
The visualizations of the fairness analysis results support the comprehension of the classification model’s behavior when presented individuals of diferent subgroups in the data. The ASDF-Dashboard presents various charts with the fairness metric values for a broad coverage of diverse aspects. The users can investigate the characteristics of the found subgroups, i.e., the sizes of the clustering- and entropy-based subgroups and the extracted patterns for each cluster, as well as the subgroup fairness metrics as measured for each subgroup individually. The rankings of the clusters or entropy-based subgroups allow for a direct access to the most discriminated or favored subgroups assuming a certain subgroup fairness metric. Additionally, the global fairness values are displayed to the user as an overall judgement of the classifier’s fairness. However, our tool is limited to fairness assessment for the task of binary classification and can not be applied to multi-class settings which require diferent subgroup fairness definitions and metrics. Another limitation is that datasets and models are not uploaded separately for modular compositions of dataset and model but just the dataset containing the predictions.
In this work we presented the ASDF-Dashboard for carrying out a subgroup fairness analysis of a binary classifier. Our tool is able to detect meaningful subgroups being treated unfairly by the classification model as measured by three common subgroup fairness metrics. The detection of the discriminated or favored subgroup uses an unsupervised clustering and an entropy-based pattern approach to automatically identify subgroups of similar instances with as little user interaction as possible. After the subgroup fairness assessment, users can explore the visualizations of the analysis results in various ways including global and local fairness measurements. In future research one could revise and further improve the subgroup detection methods by testing more clustering algorithms and datasets to get more insights into the performance and robustness of the proposed methods in various scenarios. Another future direction could be a qualitative comparison between the clustering-based and other approaches like frequent pattern mining approaches. In particular, an investigation on the properties of the extracted patterns could yield valuable information. Furthermore, it might also be beneficial to derive a cluster validation index based on some subgroup fairness criterion that allows to select the best out of multiple clustering models for the subgroup detection.