This paper is the first to explore an automatic way to detect bias in deep convolutional neural networks by simply looking at their weights, without the model inference for a specific input. Furthermore, it is also a step towards understanding neural networks and how they work. We analyze how bias is encoded in the weights of deep networks through a toy example using the Colored MNIST database and we also provide a realistic case study in gender detection from face images using state-of-the-art methods and experimental resources. To do so, we generated two databases with 36K and 48K biased models each. In the MNIST models we were able to detect whether they presented strong or low bias with more than 99% accuracy, and we were also able to classify between four levels of bias with more than 70% accuracy. For the face models, we achieved 83% accuracy in distinguishing between models biased towards Asian, Black, or Caucasian ethnicity.
Artificial intelligence is generating more and more
expectations. But is it really living up to those expectations? Its use
is being reviewed in all areas, from natural language
processing for virtual assistants, to computer vision for citizen
monitoring systems or medical follow-up
On the other hand, thanks to adequate public outreach and debate, more and more investigations are emerging that un
ො = 0.99
ො = 0.87
ො = 0.99
ො = 0.99
Labels image 3 ={Male, Black}
cover some erratic and biased behaviors of these artificial
intelligence systems. These errors and biases are calling into
question the safety of AI systems, both because of privacy
issues
One way to build trust into AI systems is to relate their
inner workings to human-interpretable concepts
In this regard, bias detection is a major challenge to
ensure trust in machine learning and its applications
To the best of our knowledge there are no prior works that
have attempted to detect the bias of a network by modeling it
from learned weights. Existing literature in bias analysis
focuses on the performance (outcome)
There is significant work on understanding neural networks learning processes, which has been useful for diagnosing CNN representations to gain a deep understanding of the biased features encoded in a CNN. In general, they map an abstract concept (e.g. a predicted class) into a domain that the human can make sense of, e.g. images or text; or they collect features of the interpretable domain that have contributed for a given example to produce a decision.
When an attribute often appears alongside other specific visual features in training images, the CNN may use these features to represent the attribute. Thus, features that appear together, but are not semantically related to the target attribute, are considered biased representations. Zhang, Wang, and Zhu (2018) presented a method to discover such potentially biased representations of a CNN.
1We used the terms parameters and weights indistinctly to refer the learned filters of a Neural Network.
Nagpal et al. (2019) used Class Activation Maps (CAMs (Zhou et al. 2016)) to obtain the most discriminative regions of interest for input face images in deep face recognition models, and observed that activation maps vary significantly across races.
Also, some investigations show how psychology-inspired
approaches can help elucidate bias in DNNs. Examples
include Ritter et al. (2017);
Research on bias analysis focuses largely on detecting
causal connections between attributes in the input data and
outcomes of the learned models
Our work goes beyond proposals that seek to model bias
through the observation of the model outcome in response to
particular inputs.
The hypothesis behind our proposed Inference-Free Bias Detection (IFBiD) is that bias is encoded in the parameters of a learned model and it can be detected. IFBiD is an interesting and noteworthy effort to contribute to tackling the bias problem. The inference of IFBiD is performed directly over the weights of a learned model, and therefore does not require a causal benchmark based on input/output analysis.
In this work we adopt the formulation proposed by
Definition 1 (Data). D is a dataset (collection of multiple samples from different classes) used for training and/or evaluating a model M. Samples in D can be classified according to some criterion d. The set Ddc D represents all the samples corresponding to class c of criterion d.
Definition 2 (Learned Model). The learned model M is trained according to input data I D, a Target function T (e.g., digit classification or gender detection), and a learning strategy that maximizes a goodness criterion G on that task (e.g., typically a performance function) based on the output O of the model and the Target function T for the input data I.
Definition 3 (Biased Model). A learned model M is biased with respect to a specific class c of criterion d if the goodness G on task T when considering the full set of data c D is significantly different to the goodness G(Dd) on the subset of data corresponding to class c of the criterion d.
Typically, as in our case, the model M is a Neural Network ( ), parameterized by , and the goodness-of-fit criterion consists in minimizing an objective function (e.g. the cross-entropy loss function).
The training process of Neural Networks is usually not deterministic and the resulting parameters depend on several elements: training data, learning architecture (e.g., number of layers, number of neurons per layer, etc.), training hyperparameters (e.g., loss function, number of epochs, batch size, learning rate, etc.), initialization parameters, and optimization algorithm.
The existing literature on bias analysis is mainly focused
on the inputs I
The aim of the bias detection model is to find patterns in
associated with biased outcomes. We designed the bias detector as a Neural Network ( ) represented by its parameters .
In our approach (detailed in next sections), we train the bias detector using a dataset of biased and unbiased models.
The models ( j ) for task T , are biased by training them with biased subsets of the database S1; :::; Sn D. To build a training set for the detector ( j ), we train a number of
i i=1;:::;n models ( j ) with each subset Si, forming f j gj=1;:::;B (see Fig. 2), where n is the number of biased subsets and B is the number of models trained with each biased subset.
ij denotes the jth instance of a learned model trained with biased subset i.
Because of the non-deterministic nature of the training
process of the network ( j ), the same training subset i is
likely to give rise to different (i.e., ij 6= ik). The reason
for this is that since the solution space is very large, the
solution (which is iteratively approximated) typically arrives
at a local minimum that depends on the initialization, the
particular training configuration, and the order of the data
It is important to underline that the approach requires the target DNNs (i.e., Input in Fig. 2) to have exactly the same architecture as the DNNs used to train the detector. This does not detract from the fact that the task is still challenging, since, as we have explained before, the filters learned by a convolutional network never appear in the same place and are never identical due to the randomness of data presentation and weight initialization (which is, incidentally, the reason why we have also used convolutions in the detector architecture).
Learned Model (⋅ȁ Ω)
Bias Detection Model (Ω ȁ Θ) Weights Layer 1 Weights Layer Ω1 Ω
Convolutional Block
Convolutional Block Weights Layer Ω
E.g., r×5×5×3
Convolutional Block Conv2D filter1(1,1) Θ1,1,1
MaxPooling2D 1 = [ 11, 21, … , 1] = [ 1 , 2 , … , ] filter1(2,1) Θ1,2,1
Conv1D
1-conv module variant. The architecture depends on the number of layers k of the model
to be audited. The depth of the module filters depends on the depth of the input weights. Module variant 1 1-conv consists of the subsequent layers: 1 1 convolution followed by d d MaxPooling, then again a one-dimensional convolution with kernel size of 1 followed by a MaxPooling with pool size equal to the number of input filters. Do not confuse the suffixes in this figure (k,i indicates layer) with those in Fig. 2 (j indicates model number).
We evaluated many different learning architectures for IFBiD. The design of the possible architectures has not only taken into account the number and types of layers, but has also depended on the selection of the parameters of the model ( j
) used as input for the detector
The detector architecture consists of a module to process the weights/filters of each layer, and then a dense layer that concatenates all the outputs of each module. The bias detector architecture consists of multiple modules to process the weights/filters of each layer (thus one module for each layer), and then a fully connected layer that concatenates all the outputs of each module. Fig. 3 shows the general architecture designed for a specific module variant (see below). The components of a module are the same for all layers (convolution, maxpooling, etc.), as well as their order; the only thing that changes are their parameters, which depend on the ( j ). size of the input weights.
We have developed different approaches, in which the general architecture remains stable, and what changes are the modules. The module variants we analyzed were the following (where d
d is the dimension of the input filter weights, c is the number of input channels, and r is the number of input filters): • MLP: Flatten ! Dense(r) and there is always 0:1 dropout afterwards (we have seen that it works best among the values: 0.0, 0.1, 0.2 and 0.3).
for further research.2 We have created two databases for experimenting in automatic bias detection: DigitWdb and GenderWdb. The databases contain the weights of the models in our experiments for the tasks of digit and gender classification. The databases include 84K models trained with different types of biases (each model has an associated label identifying the bias). These databases are publicly available ( ; j ) used 2https://github.com/BiDAlab/IFBiD/
together a database that contains the weights of 48K digit
classification networks ( ; j ). For this we have used the
colored MNIST database
To synthesize the color bias, ten different colors were selected and assigned to each digit category as its mean color. Then, for each training image, a color was randomly sampled from the normal distribution of the corresponding mean color, and the digit was colorized. The level of bias of each replica depends on the value of the variance used in the normal distribution: the lower the more bias.
The architecture is the same for all models: a CNN ( j ) with three convolutional layers with relu activation, each followed by a maxpool, and two fully connected layers at the end (with 128 and 10 neurons, a relu and a softmax activation function respectively), with a dropout layer of 0.3 between the two. Each of the trained models results in a total of 50K parameters.
All model parameters have been initialized randomly
with Glorot uniform
Each Colored MNIST biased subset has 60K training digits, so the 30K for train and 30K for test are independent. This means that the DigitWdb models assigned to test have learned with different data than the DigitWdb models assigned to train.
Properties. All models have the same architecture and similar class performance. In this case study, the bias is determined by the color jitter variance of the digit images.
Table 1 shows the average digit classification accuracy of all models trained with the four subsets of different bias levels, from very low bias (subset S1 with variance of 0:05) to very high bias (subset S4 with variance of 0:02). The table shows performance on the Colored MNIST’s test set, that is, a set of randomly colored numbers, and therefore not biased.
Although not shown, all models exceeded 99% accuracy during training (i.e. in their respective training subsets Si). This means that our models learn as far as the training set allows. But then, in the (unbiased) test set (i.e., with all digits colored randomly) the number of correct classifications drops considerably. The reason is that the color (present in the training set in a biased way) has been learned as a differentiating element when classifying digits. Thus, the network was not only learning to associate a number to a shape, but also to a color. This is why, subsequently, when finding a digit with a random color, it has much more difficulty in classifying it correctly.
Also, Table 1 shows a clear difference in the performance of the models as a function of the level of bias of the dataset with which it has been trained. This difference is the basis for the experiments carried out in this work.
Case Study B: Gender Classifier (GenderWdb). We have gathered together a database that contains the weights of 36K gender classification networks ( j ). We are aware there are more gender categories other than male and female. Since establishing ground-truth genetic sex is not possible, we use gender as a proxy for sex. We use it as a simplified application of a real-life based problem.
To train the gender classification models that constitute
this database, we used DiveFace
As in the DigitWdb database, we have separated the data for training into two independent sets of the same size. (This serves to ensure the future independence of the bias detector training and testing.)
We have trained 36K gender classification models ( j ), divided into 30K for training and 6K for testing. The architecture is the same for all models: a CNN with six convolutional layers with relu activation, each followed by a maxpool, and two fully connected layers at the end (with 128 and two neurons, a relu and a softmax activation function respectively). The result is a model with a total of 100K parameters.
All model parameters have been initialized randomly
with Glorot uniform
Properties. All models have the same architecture and similar class performance. Table 2 shows the average accuracy in gender classification of all models, separated by bias. Bias has been introduced through the subset Si with which the model has been trained. In this case study the bias is determined by the ethnicity of the face images. What becomes clear from looking at the table is the strong bias in the performance of the models in each of the groups. Note that ethnicity attributes include the color of the skin, but also more complex anthropomorphic face features.
all we have attempted a binary classification problem: detect strong bias against minimal or no bias.
In this first case we have used models with very high bias and very low bias. Fig. 4 shows the accuracy of bias detection in digit classification models ( j ) for the different architectures given the number of samples the detector ( j ) was trained with. It can be seen that the convolutional architectures show a saturation of classification performance and that it does not take many samples to get great performance. In fact, with the best architectures, 100 training samples are sufficient to achieve a performance of around 90%. These initial results suggest that bias is encoded in the weights of the learned models ( ) and it can be detected.
A second experiment has been trying to detect the level of bias of a model ( ), or in other words, to classify the models according to their level of bias. This is a more complex problem and has required us to test more architectures for the detector ( ).
Fig. 5 shows the classification accuracy of the 4 bias levels (cf. initial subsection within Experiments describing the Datasets for Case Study A) for the digit classification models ( j ) and the different architectures given the number of samples with which the detector ( j ) was trained. We see that distinguishing the level of bias in digit classification models is more complicated than simply stating bias-no bias, and that in this case the maximum success rate we achieve in the classification is 70% (note that random chance is 25% for this task). Another important thing to note is the tendency (of good architectures) to keep improving as the training set is increased, they do not seem to be reaching their performance limit.
After seeing positive results, we made the leap to a more complex problem (i.e., more covariates): detecting ethnic bias in models trained for gender recognition.
Fig. 6 shows the bias classification accuracy of the biased gender recognition models ( ) for the different architectures and the number of samples used for training.
The curves show that after a certain number of training
samples, the accuracy is no longer increasing in the model
with more parameters (that containing Conv3D: 1 1
1max). However, it can be seen that when trained with
little data, it performs similarly to the rest. The hypothesis
that best seems to explain this behavior is that, since there
are so many parameters , the solution space is so large
that the choice of a better architectural configuration occurs
automatically, leaving unnecessary parameters unchanged,
as if they were not present
The gender classifier ( ) has more layers than the digit classifier ( ), twice as many. So the bias detection network ( ) for these models has more parameters, and thus the performance is different. The best performance is obtained with the same architecture that also obtains the best performance in the digit models, the architecture with two convolutions: 1 1-conv; reaching 83% detection accuracy. The improvement in the MLP, that has the most parameters, seems to be growing steadily. The rest of the architectures seems that from 15k training samples onwards is when doubling the samples does not increase its performance so much. But it would be necessary to keep doubling the number to check if the trend holds.
We have dealt with many more architectures that are not worth describing here: using two dense layers at the end, adding a dense layer after each convolution, replacing convolutions with dense layers, playing with dropout, etc.; all resulting in worse performance than the learning architectures reported here.
Table 3 shows the comparison with a recent state-of-the-art
bias detection method
In order to apply InsideBias, we used 60 images from DiveFace, with 20 of each ethnicity and the same number of men and women. Separately, as input to the SVM, we used the parameters of the models put together as a vector of length 97K.
Our method shows considerable superiority: it has a good hit performance on models with all biases, whereas the other methods only detect well a single type of bias in the models.
We presented a novel approach called IFBiD (Inference-Free
Bias Detection) to analyze biases in neural networks: by
auditing the models through their weights. Our experiments
demonstrate the existence of identifiable patterns associated
with bias in the weights of a trained Neural Network
We used each database to train bias detectors following the proposed IFBiD principles. We have evaluated a number of architectures and have found that in both cases it is possible to achieve a good performance in bias detection. In the digit classification models we were able to detect whether they presented strong or low bias with more than 99% accuracy, and we were also able to classify between four levels of bias with more than 70% accuracy. For the face models, we achieved 83% accuracy in distinguishing between models biased towards Asian, Black, or Caucasian ethnicity. In both cases the experiments are open-ended in the absence of increasing both databases. This has been evident in the plots with the experiments carried out for different sizes of the training set.
We evaluated our approach by varying the nature of the data (i.e., digits and face images), type of architecture (i.e., number of layers, units), and optimization strategy (e.g., loss function). For future work, the generalization capabilities of our proposed bias detection approach should be studied in more depth. The training process of the biased models used for training IFBiD can be affected by hidden confounders that need to be considered.
This work has been supported by projects: TRESPASSETN (MSCA-ITN-2019-860813), PRIMA (MSCAITN-2019-860315), BIBECA (RTI2018-101248-B-I00 MINECO/FEDER), and BBforTAI (PID2021-127641OBI00 MICINN/FEDER). I. Serna is supported by a FPI fellowship from UAM.
Yosinski, J.; Clune, J.; Nguyen, A.; Fuchs, T.; and Lipson, H. 2015. Understanding Neural Networks Through Deep Visualization. In Intenational Conference on Machine Learning (ICML) Deep Learning Workshop. Lille, France.
Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; and Torralba, A. 2016. Learning Deep Features for Discriminative Localization. In Conference on Computer Vision and Pattern Recognition (CVPR), 2921–2929. Las Vegas, Nevada, USA: IEEE.