The increasing amount of applications of Artificial Intelligence (AI) has led researchers to study the social impact of these technologies and evaluate their fairness. Unfortunately, current fairness metrics are hard to apply in multi-class multidemographic classification problems, such as Facial Expression Recognition (FER). We propose a new set of metrics to approach these problems. Of the three metrics proposed, two focus on the representational and stereotypical bias of the dataset, and the third one on the residual bias of the trained model. These metrics combined can potentially be used to study and compare diverse bias mitigation methods. We demonstrate the usefulness of the metrics by applying them to a FER problem based on the popular Affectnet dataset. Like many other datasets for FER, Affectnet is a large Internet-sourced dataset with 291,651 labeled images. Obtaining images from the Internet raises some concerns over the fairness of any system trained on this data and its ability to generalize properly to diverse populations. We first analyze the dataset and some variants, finding substantial racial bias and gender stereotypes. We then extract several subsets with different demographic properties and train a model on each one, observing the amount of residual bias in the different setups. We also provide a second analysis on a different dataset, FER+.
[5], which is already a standard for model pretraining, means that any transference of bias from the dataset to the trained models could impact very large populations in 1. Introduction unpredictable ways.
This work explores a new metric-based methodology When algorithms and automated systems interact with for the analysis of bias in machine learning problems, fousers, they can often cause harm in many unintentional cusing on the measure of bias transfer between the dataset ways. This effect is multiplied when the system is a ma- and the trained model. Despite the definition of multiple chine learning system trained to imitate human behavior, fairness metrics [6], to the best of our knowledge there is which is inherently conditioned by prejudice and cogni- no bias metric supporting multi-class classification probtive biases. In machine learning, the biases gathered in the lems studied for multiple potentially protected groups, training information can leak to the trained models, which capable of isolating dataset bias from model bias. In parare otherwise expected to be fair. When these systems are ticular, our methodology is oriented to the bias transfer deployed to the real world they have been shown to exhibit from the dataset to the model, where most mitigation sysgender, racial and other demographic biases [1, 2, 3]. tems can be implemented (given a fixed dataset with its
The current state-of-the-art systems employ very large own inherent bias). datasets, with the most renowned example being Imagenet, Three metrics are proposed in this work. The first two with over 14 million images. Further analysis on this metrics measure representational bias in the dataset. One dataset has shown critical biases and problematic data of them is dedicated to quantifying the representational [2, 4]. The amount of models trained on this dataset imbalance, where some demographic groups are over or under-represented in the dataset. The other one is a novel The IJCAI-ECAI-22 Workshop on Artificial Intelligence Safety usage of the Normalized Mutual Information (NMI) and *(ACIoSrarfeestpyo2n0d2i2n)g, aJuutlhyo2r4.–25, 2022, Vienna, Austria Normalized Pointwise Mutual Information (NPMI) met$ iris.dominguez@unavarra.es (I. Dominguez-Catena); rics [7]. We propose using these metrics for stereotype daniel.paternain@unavarra.es (D. Paternain); measurement, a type of bias where demographic group mikel.galar@unavarra.es (M. Galar) representations differ among classes. These two metrics 0000-0002-6099-8701 (I. Dominguez-Catena); serve as a baseline for measuring the bias present in the (0M00.0G-0a0la0r2)-5845-887X (D. Paternain); 0000-0003-2865-6549 input data. Over that baseline, we can then employ a third © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License proposed metric to measure the bias present in the trained CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutiRon 4W.0Iontrekrnsathioonapl(CPCrBoYc4e.0e)d.ings (CEUR-WS.org) model. This metric measures the variation of recall per Copyright © 2022 for this paper by its authors. Use permitted under class among multiple simultaneous demographic groups, Creative Commons License Attribution 4.0 International (CC BY giving a single output value quantifying the amount of 4.0). bias with respect to the different demographic groups. source dataset and the trained model. The measurement The three metrics combined enable the study of the bias of bias in the source dataset aggregates historical and reptransfer from the dataset to the trained model. resentational biases, and is common to any model and
In particular, we apply these metrics to analyze the deployment that employs the same dataset. The measurebias transfer in a Facial Expression Recognition (FER) ment of model bias includes learning and aggregation problem. The objective of FER systems is to identify the biases. Jointly measuring both biases will give us an estifacial expression of people in either video or images, in mation of the amount of bias that leaks from the dataset an attempt to detect the underlying emotion. The nature to the model. This can help guide our efforts to the most of the data (human faces) in these problems make them critical parts of the system, easing the study of mitigation prone to diverse biases and misrepresentations. Human strategies. The remaining forms of bias are not directly biases in this context have already been studied [8]. applicable to our analysis.
Although the application of these metrics requires demographic information of the subjects in the dataset, the 2.1.1. Bias metrics usage of existing demographic models makes recovering some of this information possible for unlabeled datasets. There are multiple fairness and bias metrics defined In the context of FER problems we employ a pretrained through the literature [6]. They are commonly defined model, Fairface [9] to obtain some demographic descrip- for binary classification problems and two populations, tors, namely apparent race and gender, from the face im- a general population and a protected group, with one or ages. Although these demographic predictions are only several protected attributes indicating the membership to an approximation of the real attributes, in the absence of these populations. more accurate data they can be used to perform a general However, in many applications there are multiple demoanalysis. graphic groups that could require protection at the same
This work is aimed at helping reduce the racial, ethnic time, and more than two target classes, with none of them and gender inequalities that can arise in the development being clearly advantageous. To the best of our knowledge, and deployment of AI systems. Specifically, the proposed there is no bias metric covering this casuistry that is also study of demographic bias transfer can be used to both able to isolate true model bias from validation dataset detect demographic bias in machine learning systems and imbalance. Therefore, we focus on multi-class systems, assess the impact of mitigation methods, guiding the ef- studying bias across multiple demographic groups. Our forts in the implementation of fairer systems in general. bias definition considers any differential treatment suf
Code to replicate the reported results is available at fered by any of the demographic groups in any of the
GitHub1. An additional Appendix to this work with the problem classes. To design our metrics, we take the
folresults for a second dataset, FER+ [
subgroups independently, and more specifically, in the
where is the protected attribute (1 for the privileged The concept of algorithmic fairness is usually built around group and ̸= 1 otherwise) and ˆ the class prediction (1 the absence of bias or harm. In particular, several inde- for the positive class and ̸= 1 otherwise). pendent bias taxonomies have been proposed, focused on Disparate Impact is arguably the most common fairness different definitions and aspects of bias. Taxonomies like metric. However, it requires a defined positive outcome that of [11] consider up to 7 sources of bias in a machine and privileged and protected groups, making it unfit for learning pipeline, split in two subgroups: our application.
Overall accuracy equality [13] requires a similar accuracy across groups: | ( = ˆ| = 1) − ( = ˆ| ̸= 1)| ≤ ,
(2) where is the protected attribute (1 for the privileged group and ̸= 1 otherwise), ˆ the predicted class, and the real class.
As it focuses on the accuracy, the Overall accuracy equality metric is applicable in both binary and multiclass problems. It also treats all target classes equally, without requiring one of them to be defined as positive or advantageous. Unfortunately, it still requires defined privileged and protected groups, making it suitable only for the analysis of individual demographic groups.
Mutual Information [14] measures the statistical dependence between two attributes: 3. Proposal
(ˆ, ) (ˆ) () ≤ ,
(3) ∑︁ ∑︁ (ˆ, ) log ^∈^ ∈
The most basic level of demographic analysis that can be performed in a dataset is that of representational bias, where there is unequal representation of different demographic groups in the overall dataset. A clear predominance of a demographic group in the dataset can hint at a potential bias in favor of that group, generating a differentiated and potentially harmful behavior of the resulting model. Previous bias metrics (presented in Section 2.1.1) only measure the bias for the final model, so they cannot detect this specific bias. As a way to measure the representational bias of the dataset, we propose using a metric based on the standard deviation of the normalized demographic distribution. This Normalized Standard Deviation (NSD) adjusts the standard deviation of a normalized vector: where denotes the protected attribute (from a set ) and ˆ denotes the prediction (from a set ˆ ).
Mutual Information is one of the few fairness metrics that can be directly applied to multi-class classification problems, even when considering several potentially protected groups. Despite this, the metric only measures the dependence between the protected attributes and either the predicted or real class. In problems where the validation partition is not balanced (the real class and protected sauttrreibduotevseratrheedterapienneddemnto),dtehleprMeduitcutaiolnInbfeocrommaetisounnmabelae- NSD() = √− 1 √︂ ∑︀=1( − ¯) 2 , (4) to disentangle that imbalance from the potential model bias. where is the number of elements of the vector and
Nonetheless, for our analysis we propose the employ- stands for the arithmetic mean. ment of two variants of the Mutual Information metric to The NSD calculated over a normalized demographic measure the bias not in the final model, but in the source distribution is bounded in the interval [0, 1], where 0 is no dataset, where they only consider the real class and the bias (uniform distribution) and 1 is total bias (the entire protected attributes. These variants are the Normalized population belongs to a single group). Mutual Information (NMI) and the Normalized Pointwise Mutual Information (NPMI) proposed by [7] in the context of collocation extraction. They are both bounded and 3.2. Stereotypical Bias Metric easier to interpret than the classical Mutual Information, A second level of analysis of the dataset not covered by and in the case of the NPMI, it can also detect specific previous metrics is that of the presence of stereotypes, stereotypes on top of the general dataset bias. The mathe- understood as a variation in the demographic profile of matical definitions are given in Section 3.2. each target class in the dataset. In the context of FER, this can result in a different prior for an emotion label 2.2. Facial Expression Recognition depending on the perceived demographic group of the sample.
The problem of automatic FER is commonly used as a For most FER datasets, and in general for most
multiproxy to the more general emotion recognition. Although class multi-demographic datasets, the analysis and
quanmany works raise questions about the universality of facial tification of stereotypical bias is complex due to the
douexpressions in conveying emotions across cultures [8, 15], ble class and demographic imbalance usually accepted in
and despite developments in other emotion measurement these datasets. To decouple this secondary bias from the
modalities [16], FER is still one of the most widely used main representational bias, we employ the NPMI metric
methods. The applications of these systems are multiple, proposed by [7].
ranging from robotics [17] to assistive technology [
Most works deal with either a continuous emotion cod- (NPMI) measures the statistical dependence between two ification, such as the Pleasure-Arousal-Dominance model attributes for a specific pair of values: [19] or a discrete codification, such as the six/seven basic emotions proposed by [20]. This work focuses on ln ((),()) , the second approach, the most used in modern machine NPMI(, ) = − ln (, ) (5) learning. where denotes the protected attribute and denotes the class. NPMI values lay in the range [− 1, 1], with 1 being total correlation (overrepresentation), 0 being no ∑︀
∈ ∑︀
∈ correlation, and − 1 being inverse correlation (underrepre- classes , giving us an Overall Disparity OD for the sentation). model, defined as:
For an aggregated value representing the summary of the NPMI biases, we employ the NMI metric, also pro- OD = 1 ∑︁ ID() . (9) posed by [7]. || ∈
The Normalized Mutual Information (NMI) measures the statistical dependence between two attributes:
an upper bound of 1 (maximum bias). The same bounds apply to the individual ID scores.
NMI(, ) = − (6) where denotes all the demographic groups studied and the set of classes of the problem. The NMI value lies in the range [0, 1], with 0 being no bias and 1 being total bias.
∑︀∈ (, ) ln ((),()) , ∑︀∈ (, ) ln (, )
420,299 images of facial expressions. 291, 651 of these 3.3. Model Bias Metric images are classified in basic emotions, namely neutral, happy, sad, surprise, fear, disgust, angry and contempt.
Based on the metrics presented in Section 2.1.1, we re- The dataset is divided into two partitions, one for training quire a new metric to measure the model bias in prob- and one for validation, with 287, 651 and 4, 000 images lems like FER, where we operate over multiple demo- respectively. graphic groups and multiple classes, with an unbalanced The dataset was created from Internet image searches, test dataset. For the calculation of the model bias metric, based on queries related to both the emotional labels conwe expect the model to perform with a similar recall for sidered and gender, age, and ethnicity descriptors. The each demographic group and for each target class to be search query strings were also translated to 6 different considered fair. The Recall R(, ) for a class and a languages. demographic group is defined as The final images have 425 by 425 pixels of resolution.
An additional Appendix to this work with the results
R(, ) = (ˆ = |, ) . (7) for a second dataset, FER+ [
It is important to note that each of the classes in the problem can have different inherent difficulties. Therefore, we first calculate the intraclass disparity for each class by aggregating the recalls of that class for all demographic groups. Later, we aggregate those intraclass disparities into a final dataset metric. The Intraclass Disparity (ID) for each class , aggregated for demographic groups, is defined as:
lar Internet image searches, with relatively low data curation. Outside FER problems, many datasets have already undergone heavy bias analysis [4], but the lack of meta⎛ ⎞ information and demographic labels on subjects for in
ID() = − 1 1 ∑∈︁ ⎝1 − m′∈aRx(R(,,)′) ⎠ , (8) ytdhseeisspwiitnieldtthh(ieIsTdcWioven)rtFseiExtytR. cdAoanftsfaeisdceettnrsaehttiaoissnhsnidonutdreairnnegdeixttshceegpebtniiaoesrnaataninoadnl-,, with the convention that ID() = 0 if max R(, ′) = 0. demographics labels are not available.
′∈ In recent years, the development of new datasets for
This metric uses the maximum recall R(, ) of any demographic annotation, such as Fairface [9], has enabled demographic group () for the class () as the baseline the demographic relabeling of existing datasets. It is imto obtain a relative value between 0 (same performance portant to note that this kind of annotation is highly subas the maximum recall group) and 1 (recall 0 relative to jective and imperfect, and any demographic label obtained the maximum recall group) for each demographic group. constitutes only a proxy measure for the real demographic These values are then aggregated, obtaining the final met- characteristics of the subjects, which is already a subric. This measure considers all the groups, maximizing jective and complex concept. For example, the gender the ID metric to 1 when all groups except the one with classification of Fairface is binary (Male, Female), which the highest recall have an accuracy of 0 (situation of max- already constitutes a bias against nonbinary people [1]. imum privilege or bias). The race classification only uses the seven most common
Finally, we can study the class disparities by themselves or aggregate them with a simple mean over the set of 2https://github.com/irisdominguez/Dataset-Bias-Metrics descriptors (White, Latino/Hispanic, East Asian, Black, Middle Eastern, Indian, and Southeast Asian). Furthermore, both categories are treated as single-label classifications. Despite these issues, the Fairface model gives us an approximation of the demographic profile of the dataset, enough for a general bias analysis in the absence of more accurate data.
The experiments are developed in PyTorch 1.8.0 and Fastai 2.3.1. The hardware used is a machine equipped with a GeForce RTX 2080 Ti GPU, 128 GB of RAM, an Intel® Xeon® Silver 4214 CPU, and running CentOS Linux 7.7.
All the models are trained under the same conditions and hyperparameters, namely, a maximum learning rate of 1− 2 with a 1cycle policy (as described in [24] and implemented in Fastai) for 100 iterations. This parameter was decided using the lr_finder tool in Fastai. The batch size is set to 256, the maximum allowed by the hardware setup. For each dataset, we train the model 10 times and average the results over them.
We have also applied the basic data augmentation provided by Fastai through the aug_transforms method, including left-right flipping, warping, rotation, zoom, brightness, and contrast alterations.
4.4.1. Dataset bias To analyze the dataset, first we process it using the Fairface model to obtain a demographic estimation for each image. We then proceed to calculate the demographic representation profile of the dataset, and compute the NSD bias metric presented in Section 3.1. After that, we can also use the NPMI metric to highlight any stereotypical bias inherent in the distribution of labels for each demographic group, as explained in Section 3.2.
The demographic information added to the dataset through this relabeling process also enables the creation of derived datasets that can be used to simulate different bias situations. In particular, we generate: • Two balanced subsets for the racial and gender demographics. These datasets have the same representation of each demographic group considered for each of the target labels. This balancing removes both representational and stereotypical biases. • Two artificially biased gender subsets , that contain examples of only one gender.
4.4.2. Model bias
net validation partition, and the OD metric described in Section 3.3 is used as a measure of bias.
In addition to the subsets proposed in the previous experiment, a series of stratified subsets is generated to evaluate the influence of the size of the dataset on the residual bias of the model. Each subset will contain a ifxed percentage of the original train partition, maintaining the same demographic distribution. Hence, they will preserve the dataset biases of Affectnet.
In summary, we want to study the behavior of the model bias metric when increasing the number of training examples and when training on balanced subsets. If balancing the dataset is an efficient mitigation strategy, we expect a lower metric with little to no impact on general accuracy, for the same training data size.
Figure 1 shows the apparent race distribution of the dataset in Affectnet. We can observe a strong imbalance in favor of the white race, which comprises 64.4% of the training data. The apparent gender distribution (not shown) is much more balanced, with 49.7% of the training data classified as male and the rest as female. In the case of gender, imbalances arise in the analysis per label, summarized in Figure 2, reaching an imbalance 72% − 28% in the case of the angry class.
These intuitive indicators of bias are also reflected in the NSD and NMI metrics results presented in Table 1, where both the original dataset and some variations are analyzed. Representational bias is measured with the NSD metric and stereotypical bias with the NMI. Note that the NSD and NMI metrics do not share the same scales and are not comparable to each other. The number in parentheses denotes the number of demographic groups considered, 2 for gender and 7 for race. These results show a zero representational and stereotypical bias for the artificially balanced datasets in their respective categories, as expected. Note that the gender biased datasets are excluded from the gender bias calculations, as they only include examples of one gender. For the original dataset, we can observe a relatively low representational bias in the gender category (NSD = 0.0057) compared to the race overrepresentation of people recognized as male), and the category (NSD = 0.5902). On the contrary, the stereotyp- opposite for the happy class, with an overrepresentation ical bias is higher in the gender category (NMI = 0.0089) of people recognized as female and underrepresentation compared to the race category (NMI = 0.0021). This of people recognized as male. The results for the gender agrees with the imbalance perceived in Figure 2. category are consistent with the angry-men-happy-women social bias already known in the literature [25].
W
LH
ME
B
EA
I
SA rangy ttecpnom sad tlreaun itssgud irrssepu frea ahppy TaFnihgdeuarreecsc4uu.lrtascryegaarredisnhgo wthne itnraTinaebdlem2o,daenldbigarsampheitcraicllyOiDn Regarding racial bias, across the stratified subsets Figure 2: Apparent gender distribution of Affectnet for we can observe consistently high model bias values, each label. clearly decreasing from the highest bias in the smallest dataset (0.422 ± 0.043) to the lowest in the largest dataset (0.268 ± 0.023). The race balanced dataset, when Representational Stereotypical compared to an imbalanced one of similar size (Stratibias (NSD) bias (NMI) ifed / 0.13), does not appear to improve the bias metric Dataset Race (7) Gender (2) Race (7) Gender (2) (0.297 ± 0.016 vs. 0.284 ± 0.017), while decreasing acOriginal 0.5902 0.0057 0.0021 0.0089 curacy (45.7 ± 0.3 vs. 48.4 ± 0.5). In this case, increasing Balanced RGaecneder 00..05090209 00..00010509 00..00000107 00..00000901 the dataset size (Stratified / 0.22 and higher) improves the Gender biased FM 00..65172092 −− 00..00002200 −− aacrocuunradcy0.w29ithaonudt dimecpraecatsiinnggthfoerrlaacrigaelrbisaizsems)e.trNico(tsetathbalet Table 1 the accuracy is calculated over the whole test partition of Representational (NSD) and stereotypical (NMI) bias Affectnet, which is not balanced in terms of labels and the metrics for the original dataset and the considered sub- demographic groups studied. sets. In the gender category, we observe comparatively lower bias values overall, from 0.264 ± 0.053 to 0.133 ± 0.015,
The stereotypical bias detected through the NMI can be consistent with the lower gender bias of the original further analyzed with NPMI. Figure 3 shows the NPMI dataset (NSD = 0.0057, NMI = 0.0089). Balancing matrices for both the apparent gender and race analysis. In the gender does not significantly improve the accuracy rethe race category, the most prominent value indicates an sults for a similar sized dataset (Stratified / 0.36, accuracy underrepresentation (− 0.13) of the East Asian group in 51.8 ± 0.4 vs. 51.1 ± 0.6), but in this case the gender bias the angry class. In the gender category two stereotypical value improves significantly ( 0.091 ± 0.014, lower than biases stand out: for the angry class, an underrepresenta- all other models). Additionally, although the artificially tion of people recognized as female (and a corresponding gender biased datasets have a similar accuracy to the balanced one (50.7 ± 0.5 and 50.2 ± 0.4 vs. 51.8 ± 0.4), successful mitigation approach in this context (because their bias metric is substantially higher (0.185 ± 0.017 of quality differences between the images associated with and 0.242 ± 0.018 vs. 0.091 ± 0.014). each race, for example).
Regarding the gender bias, the metrics reveal a com
Bias paratively lower representational bias, but a much more Train data Size Accuracy Race OD Gender OD pronounced stereotypical bias. In this case, balancing the Original 12%% 25,,863798 3393..25 ±± 00..99 00..346222 ±± 00..002473 00..221644 ±± 00..001533 dataset seems to substantially improve the trained model 53%% 148,,159157 4441..45 ±± 00..84 00..331457 ±± 00..003139 00..119826 ±± 00..002323 bias, mitigating the bias transfer. This suggests that the 8% 22,712 46.4 ± 0.5 0.288 ± 0.017 0.174 ± 0.029 stereotypical bias detected in the dataset has a large im2123%% 6326,,495087 4489..46 ±± 00..56 00..228942 ±± 00..002248 00..114646 ±± 00..001286 pact on the trained model, but the balancing of the dataset 6306%% 117002,,324004 5531..61 ±± 00..56 00..227899 ±± 00..003104 00..114597 ±± 00..002158 corrects it properly.
100% 283,901 55.8 ± 0.2 0.268 ± 0.023 0.133 ± 0.015 Additionally, we observe a strong tendency to reduce Balanced Race 32,452 45.7 ± 0.3 0.297 ± 0.016 0.177 ± 0.017 the bias scores as the size of the training dataset increases, Gender biased GMender 111177,,779900 5510..87 ±± 00..45 00..227737 ±± 00..002165 00..019815 ±± 00..001147 even when the datasets have the same representational and
F 117,790 50.2 ± 0.4 0.315 ± 0.022 0.242 ± 0.018 stereotypical biases.
Table 2 Although the metrics have unveiled both gender and Bias metric summary for the model when trained on racial bias in the source dataset, these bias transference redataset variations. sults suggest that dataset gender bias has a greater impact in the final model. Thus, dataset gender bias seems more susceptible to mitigation measures in the early stages of the AI life-cycle, whereas racial bias may require different mitigation measures in later stages. Further studies would cay0.5 be required to evaluate the impact of different mitigation ru techniques in this case, but are out of the scope of this ccA0.4
paper. s a i lab0.4 i c a lr ed0.3 o M isa0.3 b r e end0.2 g l e od0.1 M
Stratified subsets Race balanced Gender balanced Gender biased (M)
Gender biased (F) 0 50,000 100,000 150,000 200,000 250,000
Training size
Regarding the apparent race analysis of the system, the proposed metrics reveal an important representational imbalance of the dataset coupled with some stereotypical bias. Despite this, when comparing a stratified subset to a dataset of the same size but balanced by race, we observe no improvement in the trained model bias. This suggests that either the source of the model bias is not the dataset bias (inherent differences between racial expressions, for example) or that directly balancing the dataset is not a
The metrics presented have been shown to be useful, relfecting some of the biases present in both real and manipulated datasets through easily interpretable values. The analysis of these metrics allows the study of the bias transfer from dataset to trained model, which can be useful for understanding the bias in different stages of a machine learning pipeline, and consequently in the study of mitigation strategies.
In our case study, we have revealed the heavy racial representational bias of a popular FER dataset, Affectnet, and the presence of stereotypical gender biases. The experiments also show how the resulting model seems almost invariant to the removal of the racial bias, while being severely impacted by any gender bias, either induced or corrected, in the source dataset. In the Appendix to this document3 the results for a second dataset, FER+, are provided, showing similar tendencies to the ones found for Affectnet. These results, while specific to this model and training setup, expose the complexity of the bias analysis and its impact in real world problems.
As future work lines, the same analysis could be performed for more datasets, models, and training setups. Further work in the development of new demographic datasets and models could also improve the accuracy and
ship from the Research Service of the Universidad Publica de Navarra, the Spanish MICIN (PID2019108392GB-I00 and PID2020-118014RB-I00 / AEI / 10.13039/501100011033), and the Government of Navarre (0011-1411-2020-000079 - Emotional Films). detail of this bias analysis and extend it to other problems. Different mitigation techniques, especially in the dataset preprocessing stage, could differ wildly in their impact. Finally, the metrics still require further analysis of their properties and potential application to other contexts. For example, although our proposed OD reflects both representational and stereotypical biases, having metrics capable of decoupling them could enable a more in-depth bias analysis.
Furthermore, new application areas could be researched in other multi-class multi-demographic AI systems, such as age, gender and race recognition, AI-based medical diagnosis and sign language gesture recognition, to name a few.