Commercial applications increasingly build on large language models (LLMs). Given the inherent biases of LLMs, advancements in fairness research are urgent. Prior methods for mitigating biases in LLMs only address fairness in either language generation tasks or downstream tasks. Additionally, they often incur substantial computational costs by training from scratch. We propose a novel debiasing method that employs adversarial learning during model pre-training. Without hyperparameter optimization our comparably computationally eficient method demonstrates increased fairness on a natural language generation task while maintaining performance. In addition, we show that our fairness gains transfer to a downstream task, at a performance cost. We explore a fairness approach which holds a significant potential for redefining the landscape of fairness of LLMs: By learning a single debiased model which can be applied to a variety of tasks, this approach eliminates the need for additional or task-specific debiasing steps. Hence, it facilitates the development of fair commercial applications and constitutes a step towards the broader goal of fairness in societies at large.
State-of-the-art models in natural language processing (NLP) referred to as large language
models (LLMs) commonly have a transformer architecture and perform well on a variety of tasks
such as generating coherent and semantically meaningful text or providing a good translation of
text from one language to another. In order to become the powerful models they are, these LLMs
are trained on large text corpora taken from internet platforms. Researchers have found that the
training data used for this purpose often contain biases against identity groups, ranging from
sexist or racist attitudes to prejudice against religious beliefs, professions, or political ideology
[
Fairness in NLG has escaped close examination until recently, due to the inherent dificulty
of asserting unfairness in text. However, lately more research dedicated to NLG fairness was
published, including a benchmark dataset and evaluation metrics by Dhamala et al. [
The possibility to transfer fairness to diferent tasks directly, however, would allow employing a single debiased LLM for a variety of tasks without further steps or further consideration of bias. A single debiased model hosted on a centralized hub would enable developers to efortlessly build fair applications. With task-specific debiasing methods which cannot transfer fairness to other tasks on the other hand, developers will have to apply debiasing methods themselves a step that most likely not all developers have the time, skill, and literacy for. Only fairness transfers enable the application of LLMs to diferent tasks with fair results, thus, utilizing their true power in a just way.
In this paper we propose an adversarial debiasing method in order to mitigate bias in transformer-based LLMs. In our experiments on gender bias mitigation using a BERT architecture, we show two main advantages of the proposed method: First, we present an efective and relatively computationally eficient debiasing method for pre-trained transformers on the language generation task autocompletion. We report the two extrinsic fairness metrics regard and sentiment fairness and further demonstrate that our debiased models maintain their performance as measured by perplexity on this task. Second, we show that fairness improvements from our debiasing method transfer to the downstream task text classification without further adjustments at a performance cost. We measure text classification fairness with the TPR Gap. For our experiments we consider a gender bias as it is the most researched bias. We demonstrate the use of our approach on the most researched LLM, BERT.
Previous research in fairness has come up with debiasing methods which can mitigate bias
efectively either for language modeling and NLG or downstream tasks. In accordance with
Steed et al. [
NLG Fairness Counterfactual Data Augmentation (CDA) is a method commonly employed
for achieving upstream fairness which evolves around adding complementary counterfactual
sentences to the original training sentences to address bias [
More advanced variations of the CDA approach have also been developed, such as Gupta et al.’s [14] debiasing method for GPT-2 and Lauscher et al.’s [15] utilization of adapters for debiasing. These two CDA-based techniques show promise in mitigating biases upstream without requiring the model to be trained from scratch thereby significantly reducing the computational complexity for debiasing. However, Gupta et al. [14] report that their fairness gains are not transferable. Lauscher et al.[15] show that their achieved fairness gain for language modeling is transferable to downstream tasks when using an additional task-specific adapter, albeit with a decrease in performance. However, the necessity for a task-specific adapter on downstream tasks is a disadvantage, because developers need to be aware of bias and perform this extra step.
Schick et al. [16] apply a self-debiasing technique to GPT-2 for NLG tasks, instructing the model to avoid generating biased text. Their results indicate increased fairness with minimal impact on model performance and particularly little computational costs, but self-debiasing cannot be transferred to downstream tasks.
Text Classification Fairness There is extensive research on approaches specifically tailored to downstream fairness, the most important of which we outline here. For downstream tasks, a trade-of between fairness and performance is generally assumed. We outline the most important text classification debiasing methods here and elucidate most of their shortcomings only in the last paragraph.
One example for downstream debiasing is called INLP [17] - it constitutes an established approach for text classification. INLP tackles debiasing by iteratively projecting the last representation layer to the null-space of the protected attribute. INLP efectively removes biases, but in comparison to more recently proposed methods lacks performance. Many other fairness approaches developed only for downstream tasks are based on pre-processing the training data. That is, they focus on removing bias before model training by modifying the distribution of the input data. Brunet et al. [18] proposed an early pre-processing approach that focuses on excluding strongly biased texts in order to improve fairness. Han et al. [19] propose the BTEO debiasing method, which employs resampling strategies at the dataset level and reweighting at the instance level to balance the dataset with respect to biased features. Han et al. report BTEO achieves a particularly good fairness-performance trade-of for text classification in comparison to most other debiasing methods.
Another debiasing approach is founded on adversarial learning on a text classification task
(in contrast to our use of adversarial learning on a language modeling task). Han et al. [
Because all of these methods are applied before, during or after fine-tuning the last layers of a LLM for downstream tasks, as opposed to training an LLM from scratch, the computational costs of these methods is relatively low. However, only the adversarial learning approach of Jin et al. [20] allows for transfers within the downstream domain, and none of the downstream-specific debiasing method’s fairness improvements can be transferred to language modeling or NLG.
We propose a novel debiasing approach which debiases LLMs upstream. In contrast to other adversarial learning approaches, we use adversarial learning on the task that the LLM is pretrained with. Our method builds on already pre-trained LLMs and continues their pre-training, rather than training from scratch - thereby reducing the computational efort.
During the debiasing process on the continued pre-training task, the discriminator is used for regularization and is dropped after. The debiased LLMs can then be applied to language modeling or downstream tasks in the same way a standard pre-trained LLM would be used. Because our adversarial debiasing aims to reduce or remove the representation of the high-level feature gender, corresponding biases are also less likely to be relearned downstream which sets our approach apart from other methods which improve upstream fairness.
Our formal assumption is that we have a dataset = {( , , )}=1 , where ∈ represents a d-dimensional vector representing the input text, ∈ denotes the pre-training task label (which is often a word or a sequence of words), and ∈ represents the protected attribute associated with (for instance, the gender of the person the text is about).
Our proposed method builds on previous adversarial learning approaches by Li et al. [22]
and Han et al. [
Our adversarial learning addresses an optimization problem with two key objectives: to form representations which are useful for a variety of tasks and to come up with bias-free representations. The optimization problem is formally denoted by the following equation taken from Li et al.[22] with the cross entropy loss denoted by : min max ( y, ŷ) − ( , a, â).
(1)
To maximise the adversary’s loss, it is backpropagated with gradient reversal to the LLM. Using gradient reversal instead of gradient descent stirs the LLM away from optima in which the protected variable gender is represented. One concrete example of our debiasing method would be to use it on a pre-trained BERT model, for which the pre-training task is masked language modeling (MLM). In this case, the LLM encodes ℎ from an input sentence . The LLM then predicts the masked word from ℎ , and the discriminator predicts the gender the sentence is about.
We select the two tasks autocompletion and text classification to experimentally evaluate our debiasing approach. Many researchers contend that a model’s inherent bias, its upstream, can be best captured by quantifying the NLG bias [23, 24, 25, 26, 27]. We additionally test our models on the task text classification to examine the fairness transfer downstream and because it is a well-researched task under the fairness aspect.
We use non-overlapping subsets of the BIOS dataset towards achieving two of our goals, debiasing and testing model performances on the text classification task. For the evaluation of our pre-trained model in regard to autocompletion we choose the BOLD dataset. We address potential fallacies of using the same data distribution for two steps in our pipeline in Section 4.1.3.
BIOS The dataset BIOS is named after its contents - it is a collection of about 396,000 biographies scraped with the scripts by Ravfogel et al. [17].1 Gender labels are included in the dataset, alongside profession labels, making it a suitable dataset for various applications such as adversarial debiasing and fairness evaluation in regard to gender. The dataset includes profession labels on top of the gender labels, which is why it is widely recognized and frequently used in fairness research in NLP, specifically as a benchmark dataset for fairness in the downstream task text classification.
We first split the dataset into train (65%), test (25%), and validation data (10%) as suggested by Han et al. [19] for comparability. Of each of these data subsets, we split of 30% which we use for our debiasing. Before debiasing, we pre-process the biography texts for the LLM’s pre-training task. Because we experiment with BERT, we only use the first sentence from the biographies and mask random words for the MLM pre-training.
We later use the remaining 70% of each subset of the BIOS dataset for the text classification task of predicting professions from the biographies in BIOS. We confined our test set for the text classification task to a representative sample maximum of 10,000 instances. BOLD For the NLG task we chose the large-scale dataset BOLD which is an autocompletion task. Complete with corresponding BOLD evaluation metrics, the BOLD dataset is an emerging benchmark for NLG fairness, providing insights into five diferent bias dimensions with gender being among them. It contains approximately 24,000 instances, each representing the beginning of a sentence about an individual with a specified gender label. We autocomplete the sentence prompts about women and men respectively, and compare the generated texts with BOLD metrics.
The term intrinsic metrics refers to measures based on embedding similarity, whereas extrinsic metrics measure fairness on a particular task. Extrinsic metrics are considered better fairness measures because intrinsic metrics do not reliably predict task fairness [28, 29]. Due to this ifnding from recent fairness research in NLP the selected metrics presented here are exclusively extrinsic fairness measures.
Autocompletion Metrics In our investigation, we employ the perplexity measure, a commonly used metric in NLG, as a performance metric for autocompletion. It quantifies the degree of uncertainty or perplexity associated with predicting the next word or sequence of words given a context. Perplexity, thus, reflects the fluency and coherence of the generated language, with lower perplexity values indicating a higher performance. We additionally report the BLEU metric, which is based on n-gram similarity and can reveal content drifts in generated language when a ground-truth text is available.
For the selection and implementation of automated NLG fairness metrics, we follow the
approach of Dhamala et al. [
The sentiment metric is a measure of the language polarity in a sentence e.g. about women.
For an automated sentiment score, we avail ourselves of the Valence Aware Dictionary and
Sentiment Reasoner (VADER) model trained [30] without changing the default settings.2 We
follow Dhamala et al. [
Additionally, we measure the regard metric, as in the sentiment expressed towards a person
or group in a sentence, which can difer from the overall sentiment expressed in the sentence.
For instance, consider the following racist statement: ”I am so happy and relieved, these black
folks are gone.”. The overall sentiment is positive, while the regard expressed towards black
people is negative. Analogously, to the regard evaluation of Dhamala et al. [
We then conduct a two-tailed A & B test for the significance of the disparities for both
the sentiment and regard metrics positively and negatively classified instances. The A & B
significance test [ 31] asserts how unlikely it is that two distinct rates (e.g. in sentences with
negative sentiment) for two corresponding groups are actually from the same distribution.
Text Classification Metrics In accordance with the studies conducted by Han et al. [
We aggregate at both the group and class levels to determine the RMS TPR GAP. Regarding the group level, we calculate the absolute diference in TPR for each class between each group and the overall TPR GAP denoted by Equation 2. On the class level, we further aggregate the RMS values to obtain the RMS TPR GAP, expressed in Equation 3.
= , = ∑ | ∈ 1 ∑ ( √ | | ∈ , −
|
,
2
)
(2)
(3)
4.1.3. Models
We carry out our experiments with the pre-trained BERT ”bert-base-cased” from the huggingface
hub. We debias two models with our proposed approach: ADV 1 and ADV 2. For debiasing, we
use an instance equipped with an NVIDIA T4 GPU and 64 vCPUs, which takes about 18 hours
to debias our models with the described setup. For the debiasing, we froze 10 layers out of the
13 transformer layers. Based on the findings of Han et al. [
Model ADV 1
ADV 2 CONTROL STANDARD WIKIPEDIA
Treatment adversarial debiasing adversarial debiasing continued pre-training 1 3
Hyperparameters Adv. Layers
Data Size
Epochs 2 2 regardless of adversarial learning. We include a CONTROL model for which we continue the pretraining of the STANDARD model as with the ADV models, however without our adversarial debiasing. By comparing our adversarially debiased LLMs with the CONTROL we can estimate the impact of adversarial learning and avoid mistakenly attributing the efects of continued pre-training to our adversarial debiasing.
For the autocompletion task, we follow follow Dhamala et al. [
After a trial run of fine-tuning a STANDARD BERT model for
ten epochs, we found the accuracy gains to be marginal after the first five epochs, leaving us to
train our models for five epochs only which is more than Devlin et al. [
However, for some of the models debiased with our approach, only the three last layers were debiased. We therefore freeze more than ten layers for these models during fine-tuning to preserve the debiasing, even though the performance of our models is lower as a result of this. We ran a handful experiments with the CONTROL and STANDARD model as well with more than ten layers frozen which confirms freezing ten layers generally does not lead to an improved fairness-performance trade-of, except for our method. 4The code is available on https://github.com/nyu-dl/bert-gen.
Our adversarial debiasing method processes a maximum of 7.1 million words during continued
pre-training and passes only two times over the dataset. In contrast, debiasing methods which
depend on training from scratch require a minimum of 40 passes over a 3.3 billion-word corpus
[
Autocompletion Results The results of the autocompletion fairness metrics for the texts generated by the four models and the WIKIPEDIA baseline metrics are visualized in Figure 2. In the categories of negative sentiment and positive regard, our models display a fairness gain compared to the non-debiased baselines. Additionally, for all four fairness aspects, one of our models achieve the best (lowest) value.
The STANDARD model’s perplexity scores at least 4.11 units higher than all our models in perplexity, which proves that our method increases performances on NLG tasks, while simultaneously debiasing the LLM. The CONTROL model’s results show the increase in performance is due to the continued pre-training. However, our debiased models, pre-trained with adversarial learning, have only slightly higher perplexity scores than the CONTROL model, suggesting that debiasing does not or only marginally decrease performance when controlling for longer pre-training. Additionally, many other debiasing methods cannot improve performance when compared to the STANDARD pre-trained model. As for the BLEU score, we again ascertain a positive efect of continued pre-training by itself, however, the best BLEU scores are achieved by our approach.
Text Classification Results In accordance with the recent literature on the topic, we consider the TPR GAP to be the most important indicator for fairness and therefore visualized a transformed version of the TPR Gap as a fairness measure in the mentioned figure. 5
The optimal balance between fairness and performance is observed in the BTEO debiasing method when employing a downsampling strategy. The increased fairness of our models in comparison to the STANDARD and CONTROL models substantiates the efectiveness of our adversarial debiasing method in attaining transferable fairness improvements. Furthermore, our proposed methods exhibit a more favorable fairness-performance trade-of in contrast to some of the alternative approaches, thus establishing their competitiveness on this downstream task.
We introduced a novel, relatively computationally cheap approach for debiasing pre-trained LLMs through adversarial learning on the pre-training task. Our approach increases fairness upstream and we showed that this fairness gain transfers to a downstream task without additional steps as well which constitutes a significant improvement over previous debiasing approaches.
As we did not carry out a search for optimal hyperparameters, we believe that our results do not reflect the true potential of this approach. We anticipate that our approach can improve fairness levels further for both tasks. For the NLG task, autocompletion, we assume fairness can be increased without decreasing performance and for the downstream task we expect an improved fairness-performance trade-of with altered debiasing hyperparameters. One essential hyperparameter and a promising avenue for future research on our proposed approach and downstream fairness would be to explore freezing fewer layers of LLMs during debiasing, so that when only freezing e.g. ten layers during fine-tuning for the downstream tasks the debiasing is harder to undo.
Additionally, incorporating a diverse adversarial debiasing method could further benefit our
approach which involves learning multiple adversarial networks, each encouraged to learn
orthogonal representations. A study by Han et al. [
Beyond the potential enhancements of our proposed approach, our work opens up promising
directions for future research. While gender bias is the most researched type of bias in NLP,
5Previous literature, such as the visualization by Han et al.[
Processing, 2019. URL: http://arxiv.org/abs/1807.11714, arXiv:1807.11714 [cs]. [13] H. Liu, W. Wang, Y. Wang, H. Liu, Z. Liu, J. Tang, Mitigating Gender Bias for Neural Dialogue Generation with Adversarial Learning, 2020. URL: http://arxiv.org/abs/2009.13028. doi:10.48550/arXiv.2009.13028, arXiv:2009.13028 [cs]. [14] U. Gupta, J. Dhamala, V. Kumar, A. Verma, Y. Pruksachatkun, S. Krishna, R. Gupta, K.-W.
Chang, G. V. Steeg, A. Galstyan, Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal, 2022. URL: http://arxiv.org/abs/2203.12574, arXiv:2203.12574 [cs]. [15] A. Lauscher, T. Lüken, G. Glavaš, Sustainable Modular Debiasing of Language Models, 2021. URL: http://arxiv.org/abs/2109.03646, arXiv:2109.03646 [cs]. [16] T. Schick, S. Udupa, H. Schütze, Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP, 2021. URL: http://arxiv.org/abs/2103.00453. doi:10.48550/ arXiv.2103.00453, arXiv:2103.00453 [cs]. [17] S. Ravfogel, Y. Elazar, H. Gonen, M. Twiton, Y. Goldberg, Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection, arXiv:2004.07667 [cs] (2020). URL: http: //arxiv.org/abs/2004.07667, arXiv: 2004.07667. [18] M.-E. Brunet, C. Alkalay-Houlihan, A. Anderson, R. Zemel, Understanding the Origins of Bias in Word Embeddings, in: Proceedings of the 36th International Conference on Machine Learning, PMLR, 2019, pp. 803–811. URL: https://proceedings.mlr.press/v97/ brunet19a.html, iSSN: 2640-3498. [19] X. Han, T. Baldwin, T. Cohn, Balancing out Bias: Achieving Fairness Through Balanced Training, in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2022, pp. 11335–11350. URL: https://aclanthology.org/2022.emnlp-main.779. [20] X. Jin, F. Barbieri, B. Kennedy, A. M. Davani, L. Neves, X. Ren, On Transferability of Bias Mitigation Efects in Language Model Fine-Tuning, 2021. URL: http://arxiv.org/abs/2010. 12864. doi:10.48550/arXiv.2010.12864, arXiv:2010.12864 [cs, stat]. [21] P. Delobelle, P. Temple, G. Perrouin, B. Frénay, P. Heymans, B. Berendt, Ethical Adversaries: Towards Mitigating Unfairness with Adversarial Machine Learning, SIGKDD Explor. Newsl. 23 (2021) 32–41. URL: https://doi.org/10.1145/3468507.3468513. doi:10.1145/ 3468507.3468513, number: 1. [22] Y. Li, T. Baldwin, T. Cohn, Towards Robust and Privacy-preserving Text Representations, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Melbourne, Australia, 2018, pp. 25–30. URL: https://aclanthology.org/P18-2005. doi:10.18653/v1/P18- 2005. [23] E. Sheng, K.-W. Chang, P. Natarajan, N. Peng, Societal Biases in Language Generation: Progress and Challenges, 2021. URL: http://arxiv.org/abs/2105.04054, issue: arXiv:2105.04054 arXiv:2105.04054 [cs]. [24] S. Bordia, S. R. Bowman, Identifying and Reducing Gender Bias in Word-Level Language Models, 2019. URL: http://arxiv.org/abs/1904.03035. doi:10.48550/arXiv.1904.03035, arXiv:1904.03035 [cs]. [25] E. Sheng, J. Arnold, Z. Yu, K.-W. Chang, N. Peng, Revealing Persona Biases in Dialogue
Systems, 2021. URL: http://arxiv.org/abs/2104.08728, arXiv:2104.08728 [cs]. [26] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, Language Models are Few-Shot Learners, ArXiv (2020). URL: https://www. semanticscholar.org/paper/Language-Models-are-Few-Shot-Learners-Brown-Mann/ 6b85b63579a916f705a8e10a49bd8d849d91b1fc. [27] I. Solaiman, M. Brundage, J. Clark, A. Askell, A. Herbert-Voss, J. Wu, A. Radford, G. Krueger, J. W. Kim, S. Kreps, M. McCain, A. Newhouse, J. Blazakis, K. McGufie, J. Wang, Release Strategies and the Social Impacts of Language Models, 2019. URL: http://arxiv.org/abs/ 1908.09203. doi:10.48550/arXiv.1908.09203, arXiv:1908.09203 [cs]. [28] P. Czarnowska, Y. Vyas, K. Shah, Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics, Transactions of the Association for Computational Linguistics 9 (2021) 1249–1267. URL: https://doi.org/10.1162/tacl_a_00425. doi:10.1162/tacl_a_00425. [29] S. Goldfarb-Tarrant, R. Marchant, R. M. Sanchez, M. Pandya, A. Lopez, Intrinsic Bias Metrics Do Not Correlate with Application Bias, 2021. URL: http://arxiv.org/abs/2012.15859, arXiv:2012.15859 [cs]. [30] C. Hutto, E. Gilbert, Vader: A parsimonious rule-based model for sentiment analysis of social media text, in: Proceedings of the international AAAI conference on web and social media, volume 8, 2014, pp. 216–225. [31] D. C. Montgomery, G. C. Runger, Applied statistics and probability for engineers, John wiley & sons, 2010. [32] D. P. Kingma, J. Ba, Adam: A Method for Stochastic Optimization, 2017. URL: http://arxiv.
org/abs/1412.6980. doi:10.48550/arXiv.1412.6980, arXiv:1412.6980 [cs]. [33] A. Merchant, E. Rahimtoroghi, E. Pavlick, I. Tenney, What happens to bert embeddings during fine-tuning?, arXiv preprint arXiv:2004.14448 (2020). [34] J. Lalor, Y. Yang, K. Smith, N. Forsgren, A. Abbasi, Benchmarking Intersectional Biases in NLP, in: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Seattle, United States, 2022, pp. 3598–3609. URL: https: //aclanthology.org/2022.naacl-main.263. doi:10.18653/v1/2022.naacl-main.263.