The emergence of large language models (LLMs) has revolutionized the field of natural language processing, facilitating remarkable progress across various domains. However, the inherent opaqueness of these models, functioning as black boxes, presents significant challenges. The lack of transparency obstructs our comprehension of their internal mechanisms and decision-making processes, raising concerns about their reliability and fairness. Various forms of biases have already been identified within these models. It is crucial to identify the location and encoding of these biases within LLMs to enable necessary modifications that ensure their safe and equitable application free of social biases in all kind of areas. Given the extensive deployment of LLMs in real-world applications, their impact on individuals' lives is magnified. Thus, the subsequent phase of this thesis will focus on effectively mitigating biases in deep learning models.
The advent of GPT-3 [1] has sparked a massive adoption of this model, with predictions of its profound impact on the labor market, as outlined by [2]. This remarkable influence stems from the diverse range of capabilities that these models possess, including question answering, text generation, translation, summarization, information retrieval, act as a conversational agent, programming assistance, educational support, story telling and more.
However, despite the tremendous utility of LLMs, they also pose an emerging challenge: their tendency to operate as black boxes. While they exhibit impressive performance, their internal mechanisms and decision-making processes often remain opaque, making them difficult to comprehend and explain. This lack of transparency gives rise to concerns regarding their trustworthiness, fairness, and the potential biases that may be embedded within their models.
The concept of a black box refers to a system or model where the inputs and outputs are known, but the inner mechanisms and algorithms that generate those outputs remain concealed or poorly understood. LLMs, with their complex neural networks and millions, or even billions, of parameters, are intricate black boxes that often surpass human comprehension. This opaqueness hampers our ability to fully grasp the decision-making processes of these models, making it difficult to tackle biases, recognize potential vulnerabilities, and guarantee ethical and responsible utilization. Consequently, there is a pressing need to enhance transparency and develop techniques that shed light on the inner workings of LLMs.
In recent years, artificial intelligence has made significant advances, and a substantial portion of this progress can be attributed to neural network models. These models, trained on extensive datasets, have showcased remarkable capabilities in capturing various aspects of reality. However, while their ability to capture reality with precision is commendable, it can also have negative implications. One such concern arises from their propensity to inadvertently perpetuate and replicate undesirable stereotypes.
These models are already being used in multiple production systems such as medical systems [3], legal systems [4], hiring [5], content moderation [6], CRM [7], marketing [8], virtual assistants, harmful content detection [9], chatbots, etc.
These systems are used in products despite having proven to be unsafe. It is well known that sometimes these black boxes cause unintended harm. One example is the police COMPAS system, which assigned an unreal recidivism value to both white and black people. For white individuals, the assigned value was lower than the actual value, while for black individuals, it was higher than the actual value [10]. Another example is the medical system called Optum [11], which systematically allocated fewer resources for the treatment of black patients compared to white patients with the same level of need.
This realization raises concerns about the fairness and potential harm that may arise from the application of non-explainable models in certain situations. For instance, Amazon discontinued the use of a recruitment tool [12] when it was discovered to be biased against women. These examples highlight the presence of biases not only in language models but also in systems employing computer vision [13], audio processing [14], and linguistic corpora [15], [16]. It is crucial to address these biases as they can perpetuate inequality and have real-world consequences. Understanding and mitigating biases in such systems is a pressing concern.
In the case of GPT-3 [1] (or its frontends like Chat-GPT or Bing GPT) or Google's alternative, Bard [17], studying these models is not feasible because they are provided as services through APIs or web interfaces. However, there have been releases of models with similar numbers of parameters and capabilities as the aforementioned ones. For instance, models like Llama [18], Vicuna [19], Bloom [20], OPT [21], XGLM [22], and the recent Falcon [23] do provide access to the trained models weights. This access enables us to review, correct, or mitigate any biases present in them.
This will be the next step of the thesis. Next, we provide a brief overview of evaluation techniques, followed by a collection of the most relevant techniques for bias mitigation. In previous works, such as the one mentioned in , a broader summary of the state-of-the-art in studying bias in language models can be found.
This will be the next step of the thesis. In the following section, we provide a brief overview of evaluation techniques, followed by a collection of the most relevant techniques for bias mitigation. In previous works, a broader summary of the state-of-the-art in studying bias in language models can be found [24].
When we talk about bias in language models, we can approach it as a representational problem [25]. This refers to the bias that certain demographic groups face in terms of misrepresentation, including negative associations or even their absence in the data and consequently in the model. On the other hand, we can approach it as an allocation problem, which refers to issues of opportunities or resource distribution for individuals belonging to specific demographic groups.
There is extensive work when it comes to evaluating language models for bias, starting with the work of Bolukbasi et al. [26] on simple word embeddings. Later studies approached the bias issue from the perspective of coreference resolution, such as [27] with GloVe embeddings. Bias is also examined by measuring the association between concepts and protected attributes. Caliskan et al. [28] created the Word Embedding Association Test (WEAT) for this purpose. This test was extended by Dev et al. [29] and Manzini et al. [30]. Also it was extended by Lauscher et al. [31] by adding more protected attributes and applying it to languages other than English . It was later on adapted to more complex models like BERT, under the name SEAT, by May et al. [32] and Tan and Celis [33].
There are other approaches for more complex models like BERT or GPT-2.Vig [34] introduced visualization tools to understand where these models capture unwanted biases by examining their attention. Additionally, adaptations of WEAT, such as SEAT, have been developed. SEAT tests the protected attribute against a sentence instead of a word, specifically designed for contextual models like BERT. This work was further extended to consider the full context instead of just the sentence level. The latest evaluation method is applied to models like GPT-2, BERT, ELMo, among others.
More complex models also make serious errors. A compendium of errors discovered in ChatGPT is presented in the work of Borji [35]. The paper explains that this model is unable to successfully complete tasks that require spatial, temporal, or physical reasoning unless it has been specifically trained for those tasks.
The main approaches to address bias in language models consist of the following: fine-tuning the model [36], data augmentation to balance categories and avoid distortions towards one category [27], protecting the attribute during model training to prevent bias capture [37], or correcting the vector space of the model as presented in the works of Manzini et al. [30], Zhou et al. [38], Dev and Phillips [37]. Among these techniques, fine tuning and model editing are considered the most realistic, especially in the case of large-scale models, since retraining a model from scratch would be very costly in terms of time, hardware resources, money and the effort required to perform the pre-processing and tuning of training data.
One of the most promising techniques for model editing involves identifying how the model encodes certain knowledge and then making edits accordingly. The proposal of Meng et al. [39] focuses on editing factual knowledge and serves as a foundation for further adaptation. This technique first identifies the model's influential parts that contribute the most weight in choosing the last token by using causal mediation analysis. From there, the model's weights are edited to guide it towards the desired token. For example, if the model answers Obama to the question "What is the surname of the U.S. president?", the weights can be located and corrected to select the desired token Biden since this would be the updated and accurate answer. Similarly, this method can be generalized to make broader corrections. In fact, in a subsequent work [40] they adapt this method to perform mass corrections across the model weights. Then they evaluate whether the model only edits knowledge for the specific context given in the prompt or if it can generalize by asking questions about the same fact using different questions or contexts.
These techniques hold great potential in tackling bias, enhancing the accuracy, and bolstering the reliability of language models. By facilitating targeted edits that align with desired outcomes, these approaches enable the mitigation of unwanted biases in the models' responses. As a result, they contribute to an improved understanding of fairness and ensure more reliable and unbiased outputs from language models.
Every day, these enormous models are increasingly integrated into various products and production systems. However, this integration comes with its own set of challenges. From an economic standpoint, utilizing a biased system can lead to significant disadvantages, as it may not function effectively for all users. On the other hand, the impact of these models on people's lives cannot be overlooked. There are specific contexts, such as systems for resource distribution, employment, or bank credit, where it is crucial to avoid using models that may contain any form of bias. Therefore, it is imperative to thoroughly study bias in data models and understand its underlying causes. This knowledge will enable us to either avoid deploying biased models altogether or develop strategies to mitigate harmful biases when they arise.
Furthermore, when a language model is identified as not performing adequately in a production system, such as a commercial product, companies face important decisions. Given the immense size and cost associated with training these models, some proposed solutions may be difficult to justify from an economic perspective. For instance, training the model from scratch with revised, filtered, or corrected training data would entail significant expenses. Another option, albeit costly, could involve discontinuing the use of the model, as a poorly performing model is unsuitable for deployment in production systems. This proposition gains some relevance considering the potential non-compliance of such models with new European AI regulations [41]. Alternatively, more practical approaches could involve retraining the model or leveraging state-of-the-art bias mitigation techniques to address the identified issues.
The choice of approach will depend on various factors, including the severity of the bias, the feasibility of retraining or mitigating the model, and the legal and ethical obligations that must be met. Regardless of the chosen course of action, it is essential to proactively address and rectify bias issues to ensure responsible and fair deployment of language models in real-world applications. By doing so, we can foster inclusivity, promote equitable outcomes, and uphold the principles of fairness and ethical AI.
The following hypothesis is assumed: Given a language model based on deep learning, it will be possible to discern whether it contains biases, and characterize, measure, and mitigate them.
The following objectives are established:
• At this phase of the thesis, our primary focus is on the last point.
As we move forward with the use of large language models, our next step will involve adapting and evaluating the previous work [42] in the context of LLMs. Specifically, the previous study shed light on how models tend to perceive women based on their physical appearance, while men are assessed primarily based on their behavior. This pattern was observed across the majority of the models investigated.
To proceed, we will replicate the aforementioned experiment using large language models (LLMs) and analyze to what extent increasing the model size affects bias, whether it exacerbates or reduces it. Once this evaluation is completed, our focus will shift towards bias mitigation strategies.
To mitigate bias, we will construct a corpus of prompts that elicit biased responses from the models. This corpus will serve as a foundation for our work in two main areas. First, we will develop methods to detect and identify biased terms produced by the model in its responses. Second, we will explore the previously discussed fact editing techniques to edit the behavior of the model for the detected biases in order to reduce or eliminate them. This will require adapting the causal mediation analysis mechanism to our problem, since editing a specific fact is not the same as making an edit that causes a trade-off between different classes of a protected attribute. After the editing process, we will evaluate the performance of the model with the same set of prompts to check the effectiveness of the mitigation method.
By undertaking these steps, we aim to gain insights into the behavior of large language models regarding bias and work towards developing effective strategies for bias mitigation.