Transformer-based algorithms such as BERT are typically trained with large corpora of documents, extracted directly from the Internet. As reported by several studies, these data can contain biases, stereotypes and other properties which are transferred also to the machine learning models, potentially leading them to a discriminatory behaviour which should be identified and corrected. A very intuitive technique for bias identification in NLP models is the visualization of word embeddings. Exploiting the concept of that a short distance between two word vectors means a semantic similarity between these two words; for instance, a closeness between the terms nurse and woman could be an indicator of gender bias in the model. These techniques however were designed for static word embedding algorithms such as Word2Vec. Instead, BERT does not guarantee the same relation between semantic similarity and short distance, making the visualization techniques more dificult to apply. In this work, we propose a weakly supervised approach, which only requires a list of gendered words that can be easily found in online lexical resources, for visualizing the gender bias present in the English base model of BERT. Our approach is based on a Linear Support Vector Classifier and Principal Component Analysis (PCA) and obtains better results with respect to standard PCA.
With the afirmation of Artificial Intelligence in everyday user experience, many AI systems
exhibited a behaviour that can be legally defined, when acted by humans, discriminatory.
In particular, for Natural Language Processing (NLP) and Machine Learning techniques, the
problem of algorithmic discrimination is mainly caused by prejudiced data involved in the
learning process and it produces an uneven outcome for demographic minorities, i.e. subgroups
of people difering by gender, race, religion, sexual orientation, disability, etc [
In NLP systems, an intuitive way to assess bias is visualization. As can be seen in the simple example showed in Figure 1, more stereotypical masculine jobs such engineer or mechanic occupy a specific region of the two-dimensional space. Instead, stereotypical feminine jobs such electrician(1%) mechanic(1%)
engineer(2%) as nurse or hairdresser are grouped together in another region.
In the last few years, the state of the art for many NLP tasks has been profoundly changed by
Transformer-based algorithms [
Considering static word embeddings, there are several studies for measuring and visualizing
the gender bias [
In our opinion, this is mainly due to two factors. First, while Word2Vec vectors have a length
between 100 and 300 [
In this work, we focus on the problem of visualizing gender prejudice in human occupations,
analysing the word representation produced by BERT. We propose a new weakly supervised
method which requires a simple and minimal dataset for obtaining a graphical representation
of biased word embeddings by reducing the BERT vector space to a two-dimensional plane
showing the gender distortion. We use a linear Support Vector Classifier [ 19], trained on a
simple list of gendered English words (e.g. woman, man, sister, brother, etc.), to decide which
features are more involved in the gender definition. This training process does not require
any time-consuming data collecting or labelling tasks, as these words can be easily found in
many online lexical resources. Secondly, we apply the Principal Component Analysis (PCA)
[20] in order to further reduce the word representation to a two-dimensional space which can
be visualized. With respect to standard PCA, we show that our approach produces a better
visualization, providing an intuitive understanding of gender bias in human occupations, as
captured by the classical Masked Language Model for bias identification [
In the last few years, several scientific papers [ 21, 22] have pointed out the presence of bias or discriminatory behavior in machine learning algorithms and in Natural Language Processing systems. The term bias is often used in reference to an algorithm to indicate a systematic distortion of outputs that produces unfair results, such as favoring or discriminating against certain groups of people [23]. For an algorithm, bias is something unwelcome, the absence of which is necessary to satisfy the fairness property. A statistical, and commonly adopted, definition of bias is presented in [ 24] and it can be summarized as the distance between two conditional probabilities ( |) and ( |) that a word denoting a stereotype will appear in a sentence, given words and characterizing two distinct categories and of subjects. In order to address this issue, the same work outlined a standard three-steps approach: (1) definition, (2) identification and (3) bias correction. The focus of the current work is visualization, therefore a part of the step 2.
Based on this and other similar definitions, several techniques for identifying the presence of
bias in models have been developed over the years with variable efectiveness. For example, in
2017 Caliskan et al. [
Another interesting work is the study by Bolukbasi et al. [
However, with the advent of more complex deep learning architectures, such as Transformer
[
Thus, for BERT a typical approach to bias identification is not based on visualization. Instead,
R
BERT
trains
relevant features
trains
PCA
^
to the features selection; the most relevant features related to the gender information are identified
by the LVSC model trained over the dataset of gendered words. In the second phase, the PCA model
is trained over and applied to , obtaining the final resulting two-dimensional vector ^ ⊆ R2.
it focuses on evaluating the behaviour of the so-called Masked Language Modeling (MLM) task
[
Our objective is to visualize the distribution of BERT-encoded words, i.e. to reduce high dimensional vectors to a 2D space, in order to highlight the gender spectrum. More specifically, we use the base version of BERT for the English language1 which has an embedding space of length 768. We do this in two steps: we first select the most relevant features for gender, and secondly we reduce them while preserving their variance. This procedure, called Weakly Supervised Visualization (WSV) is showed in Figure 2.
We apply this method to the 1678 single-word job titles appearing in the JNeidel dataset2. Given a word describing an occupation, we exploit BERT for calculating an embedding representation ¯. However, given that this representation depends also on the overall context of the sentence into which appears, we use a basic template which will form the context of every word in the dataset. This template is composed by three tokens: the special token [cls], which is used by BERT to calculate an entire representation of the sentence, and [sep], which marks the end of the sequence. Although BERT will produce a vector for each token in this simple sequence, we extract only the one representing , which is composed by 768 real numbers.
In order to measure potential bias of the embedded representation of job titles, we need also to encode gendered words, i.e. relationships, titles, pronouns and other expressions clearly indicating the gender of the subject such as mother, brother, grandpa, girlfriend, he, him, she, her, sir, madam, queen, king, etc. This list consists in 102 terms coming from the WinoBias dataset [27], 51 for each considered gender. In order to obtain a vector representation of these words, we apply the same technique used for the job titles. Once the gendered words (denoted as ) and the occupation words (denoted as ) are encoded, we begin the dimensionality reduction.
The first phase exploits a Support Vector Classifier [ 19] with a linear kernel (LSVC). The model is trained over the dataset labeled with a binary classification value representing the word gender3. The LSVC model then learns a separator hyperplane in R768: w¯ − = 0. From the vector w of weights, we extract the higher absolute values, deducing the most relevant dimensions of the vector for the gender identification. After this procedure, the remaining 768 − features are cut of from the vectors.
This features selection procedure is applied both to and to , obtaining respectively ⊆ R and ⊆ R, i.e. two compressed representations which would contain most of the gender-related information. Please note that, in order to train the LSVC, we need only a small list of about 100 gendered words which can be easily found online, as in the WinoBias dataset or in lexical databases such as WordNet. No actual labelling process or data collecting is required, therefore we claim that our approach is weakly supervised.
The second phase consists in the Principal Component Analysis (PCA) [20] of . PCA defines the optimal linear transformation : R ↦→ R2 such that the maximum percentage of variance in the starting samples is preserved. Thus, applying the same to , we get ^ ⊆ R2 summarising the gender bias in the evaluation set . Given that we chose the most important dimensions of the vector related to the gender prediction, the variance preserved by the PCA in should capture whether there is a gender distortion in the job titles.
The WSV implementation requires to set a single hyperparameter , namely, the vectors dimension extracted in the first phase: : R76→8−− ↦− LSVC
R→−↦− PCA R2 As we describe in Section 4, the best choice of are from middle values, while very low (such as = 2) or high ( > 300) values do not obtain satisfying results.
In general, we can see our approach as a linear transformation from a high dimensional vector space to a two-dimensional plane. This requires just a single training of a simple classifier. In our case, we have trained a LSVC under the hypothesis (later confirmed by the results, as we show in Section 4) that the gendered words are linearly separable. However, in our opinion this is not necessary, and diferent kernels or other models (such as XGBoost or Feed-Forward Neural networks) can be used alongside with a technique for extracting the most important features (e.g. SHAP [28]) and then applying the PCA. In all these cases, after than the classifier has learned which dimensions are mostly related to the gender, this information can be exploited 3For now, we improperly simplify the social perception of gender by considering only the male and female classes. [mask] works as a [job]. [mask] worked as a [job]. [mask] was a [job]. [mask] will soon be a [job]. [mask] has a job as [job]. [mask] is a [job].
[mask] should be [job] soon. [mask] has studied for years to become a [job].
One day [mask] will be a [job].
From tomorrow, [mask]’s going to work as a [job] [mask] is studying to be a [job].
[mask] has always wanted to become a [job]. for any set of words. Moreover, this whole process could be easily generalized to diferent types of bias, as long as we provide a labeled set of training samples. As future work, we will perform a more in-depth study considering diferent classifiers, sets of words and types of bias.
In this section we present the results achieved by the proposed model, comparing it to the bare use of the Principal Component Analysis (PCA). Our Linear Support Vector Classifier has been trained using the default hyperparameters in the implementation provided by the Scikit-learn4 library [29] and with the hinge squared loss function.
Considering the 1678 jobs in the JNeidel dataset, first we calculated the gender distortion with the standard Masked Language Modeling (MLM) technique: we chose 12 templates regarding the career domain (Table 1) and we measure for each job the diference between the probabilities of “[mask]” being “she” or “he”. With this procedure, we obtain a score between − 1, that indicates an only man profession, and +1, that indicates an only female profession. From now on, we will refer to this score as the MLM score. The MLM score is incorporated in our visualization as follows. In Figure 3, each point represents a diferent occupation, obtained by only applying PCA (on top) and by our method (on the bottom) selecting = 50. The colour of each point indicates the MLM score: jobs with a MLM score very close to − 1 are represented as a blue point, while those ones very close to +1 are represented as a magenta point.
As it can be seen in the first chart in Figure 3, the PCA results have no spatial relations with the bias detected by the MLM score. In fact, the pink points are sparse across all the space, without any noticeable logic. On the other hand, samples in the second chart are placed according to the stereotypical gender with most of the pink and magenta points on the left region of the space, highlighting the gender spectrum of prejudiced jobs. While for PCA reducing vectors in 2D is not enough to intuitively show a bias, being only suficient to show the distortion of the most extreme samples, our approach visualizes a more evident trend. For instance, jobs like nurse or hostess are strongly related to the female gender and occupies the left region of the two-dimensional space, while typically masculine jobs like priest or infantryman are in the opposite region. 4https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html
chairman infanptroypumoltlaricynemwaonman midwife
PCA hostess nuprrsieest bankman
busgirl hostess nurse midwife
busgirl policewoman
WSV for n = 50 chairman
priest bankman poultryman infantryman
G e n d e r p e r c e i v e d w i t h M L M G e n d e r p e r c e i v e d w i t h M L M
In order to provide a more quantitative indication of the ability of PCA and WSV (selecting diferent values of ) to show the bias, we have calculated the Pearson Correlation Coeficient, in absolute value, among the two components extracted by both methods and the MLM score. We present the results in Table 2. Results in terms of correlation for PCA are very low (0.09 for the first component, 0.05 for the second one), demonstrating that this method is not able to represent the gender bias in a two-dimensional space. However, simply extracting the two most relevant features ( = 2) without applying the PCA algorithm does not provide satisfying results. Instead, combining the selection of a relatively small number of features with the PCA provide the best results. In fact, the correlation between the first component (plotted Value of 1st comp. 2nd comp. as the horizontal axis in Figure 3) and the MLM score with = 50 is 0.42, confirming the intuition provided by the plot. In general, the choice of the hyperparameter , namely the number of elements extracted from the BERT embeddings is a fundamental vector for evaluating the eficacy of WSV. In fact, a value too low could cut of important gender information, but a value too high would make the selection useless and admit a lot of unrelated information. Experimental tests (like the ones reported in Table 2) showed us that values between 20 and 100 are usually good options, however these results may depend also on the dataset considered.
In Figure 4, we performed the same experiment but considering the WinoGender dataset [
Considering the WinoGender dataset, we have evaluated how PCA and WSV are correlated
with the real world statistics for gender employment rate. While the most correlated component
of standard PCA has a Pearson Coeficient of 0.24, our approach reaches 0.56. Given that, for
the same dataset, the MLM score has a correlation with the gender employment rate of 0.59, this
result is particularly important. In fact, we are able to produce a visual plot which has a very
similar correlation to the one obtained by the standard technique for bias identification [
We presented a new weakly supervised graphical approach to identify the gender bias in a BERT model. The results indicate that our method gives better representations of prejudiced gender than the standard PCA approach, ofering the possibility to easily grasp distortion. When used on real world data, analysing the relation between the extracted bias and the gender employment rate, it provides comparable results with respect to the commonly used Masked Language Model method. An import characteristic of our work is that we propose an algorithm which only needs a list of common gendered words for training, without any expensive data collecting or labelling processes.
As future work, the first thing to do will be to apply WSV to other types of bias, such as ethnicity or religion. This can be done by identifying a set of words related to the bias in question to quickly train our proposed weakly supervised system. However, the number of training words could vary depending on the kind of bias and on the dificulty of the identification process. In adapting our approach to other fields and datasets, another fundamental aspect which has to be considered is the tuning of the hyperparameter , which could require several trials. Another possible development will be to apply the technique to models which analyse the Italian language. Since this language, like Spanish or French, has a rich morphology for nouns and adjectives and often has terms directly specifying the gender (such as studente or studentessa, which identify respectively a male and a female student), we may need to modify our bias detection strategy. Finally, we will try to apply and to adapt our method to newer and more complex NLP models, such as GPT. embeddings, in: G. Kondrak, T. Watanabe (Eds.), Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 - December 1, 2017, Volume 2: Short Papers, Asian Federation of Natural Language Processing, 2017, pp. 31–36. [16] K. Ethayarajh, How contextual are contextualized word representations? comparing the geometry of bert, elmo, and GPT-2 embeddings, in: K. Inui, J. Jiang, V. Ng, X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLPIJCNLP 2019, Hong Kong, China, November 3-7, 2019, Association for Computational Linguistics, 2019, pp. 55–65. [17] S. Rajaee, M. T. Pilehvar, An isotropy analysis in the multilingual BERT embedding space, in: S. Muresan, P. Nakov, A. Villavicencio (Eds.), Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, Association for Computational Linguistics, 2022, pp. 1309–1316. [18] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, in: K. Inui, J. Jiang, V. Ng, X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, Association for Computational Linguistics, 2019, pp. 3980–3990. [19] C. Cortes, V. Vapnik, Support-vector networks, Mach. Learn. 20 (1995) 273–297. [20] K. P. F.R.S., Liii. on lines and planes of closest fit to systems of points in space, The London,
Edinburgh, and Dublin Philosophical Magazine and Journal of Science 2 (1901) 559–572. [21] L. Sweeney, Discrimination in online ad delivery, Commun. ACM 56 (2013) 44–54. [22] I. Zliobaite, A survey on measuring indirect discrimination in machine learning, CoRR abs/1511.00148 (2015). [23] D. Rozado, Wide range screening of algorithmic bias in word embedding models using large sentiment lexicons reveals underreported bias types, PLOS ONE 15 (2020) 1–26. [24] I. Garrido-Muñoz, A. Montejo-Ráez, F. Martínez-Santiago, L. A. Ureña-López, A survey on bias in deep nlp, Applied Sciences 11 (2021). doi:10.3390/app11073184. [25] R. H. Maudslay, H. Gonen, R. Cotterell, S. Teufel, It’s all in the name: Mitigating gender bias with name-based counterfactual data substitution, in: K. Inui, J. Jiang, V. Ng, X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLPIJCNLP 2019, Hong Kong, China, November 3-7, 2019, Association for Computational Linguistics, 2019, pp. 5266–5274. [26] C. May, A. Wang, S. Bordia, S. R. Bowman, R. Rudinger, On measuring social biases in sentence encoders, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Association for Computational Linguistics, 2019, pp. 622–628. [27] J. Zhao, T. Wang, M. Yatskar, V. Ordonez, K.-W. Chang, Gender bias in coreference resolution: Evaluation and debiasing methods, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), Association for Computational Linguistics, New Orleans, Louisiana, 2018, pp. 15–20. [28] S. M. Lundberg, S. Lee, A unified approach to interpreting model predictions, in: I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, 2017, pp. 4765–4774. URL: http://papers.nips.cc/paper/ 7062-a-unified-approach-to-interpreting-model-predictions. [29] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830.