Language, the cornerstone of human interaction, encapsulates our thoughts, knowledge, experiences, and creative endeavours. It reflects our societies' historical and cultural complexities, perpetuating values, structures, and, inevitably, stereotypes [1,2]. Stereotypes, first dissected in the Social Sciences by Lippman in 1922, delineate the "typical image" or representation conjured when referring to specific groups or situations [3]. Despite their role in simplifying our understanding of the world, stereotypes often carry negative connotations, particularly in the realm of gender stereotypes, where gender identities are considered vastly different, echoing assumptions of biologically determined roles that dictate societal positions based on physical attributes or emotional capacities [4].
Feminist theory critically examines gender stereotypes, exploring how social norms and expectations shape cis-normative perceptions of femininity and masculinity [4]. In particular, A. Oakley contributed to exploring gender as a social construct, challenging the reduction of gender to mere biological determinants and asserting its foundation in social, economic, and cultural constructs [5].
Given the deeply rooted nature of gender stereotypes in societal and cultural contexts, it is essential to examine how these biases are transferred into and expressed within digital technologies, particularly AI. While AI systems hold immense potential to revolutionize various aspects of our lives, they are not immune to embedding discrimination in subtle yet pervasive ways. Historically dominated by masculine perspectives, the technological field prompts concerns about inclusivity and the possibility that AI may perpetuate existing gender biases [6,7].
For instance, in 2021, it was revealed that API image labelling services generated sexist labels [8]. In a dataset where all individuals had visible hair and wore professional attire, the Google Cloud Vision image labelling algorithm consistently paid more attention to women's hairstyles and fashion than men's, even though both had the same occupation as members of Congress. These algorithms labelled men as "officials, " "entrepreneurs, " and "military officers, " while women were often associated with labels like "hairstyle" or "beauty" [8]. Additionally, Microsoft's NSFW service detected female subjects as adult content at a higher rate [9].
Such disparities underscore the urgency of our exploration into AI and gender bias, aiming to uncover whether these digital advancements are catering exclusively to male-dominated narratives or forging a path toward inclusivity.
Humans can introduce biases that become embedded in AI systems by determining which datasets, variables, and rules the algorithms learn from to make predictions. In this context, a critical approach towards the data collection process is sought to be adopted by our work, fully aware of how this can be influenced by human perception and acknowledged that according to a 2019 report by the European Union Agency for Fundamental Rights [10], data quality is an important risk factor for bias in AI.
As demonstrated in the aforementioned cases, the datasets used in these systems and their filtering processes highlight the interaction between words, images, and stereotypical connotations, which can negatively influence AI. To understand these mechanisms, this work aims to identify potential human biases in gender classification systems.
To lead this exploration, we adopt an approach aligned with the field of "Artificial Cognition" [11]. This area of research is based on the hypothesis that just as cognitive psychology has been used to understand the human mind as a 'black box', its theories can serve as a starting point for understanding the mechanisms of AI systems in their opacity. Consequently, this field suggests that by applying cognitive psychology theories to AI, we can gain insights into how human biases might be reflected in AI's decision-making processes. Specifically, we explore the "averageness theory" [12,13,14,15] within cognitive psychology, examining how perceptions of attractiveness could influence AI's gender classification accuracy. Through a dataset generated by the AI model Stable Diffusion 2.1, theoretical insight and empirical analysis are blended to scrutinise the manifestations of human gender bias in AI classification systems.
To contextualise our empirical investigation, we examine how the AI Act and GDPR address cognitive gender bias in classification systems. Moreover, this examination underscores the importance of regulation in addressing the legal challenges posed by generative AI-based data augmentation, particularly those related to gender bias and data protection principles such as collection limitation, purpose specification, use limitation, data minimisation, transparency, data quality, access and correction, retention limitation, automated decision-making, and profiling [16].
This investigation, rooted in a deep understanding of gender dynamics based on the feminist legal theory [17] and averageness theory, seeks to shed light on how unconscious assumptions and biases can shape future technologies, highlighting the need for careful ethical and legal consideration in the design and implementation of AI.
The work is structured as follows: in Section 1, an introduction of the work is presented; in Section 2, state-of-the-art studies about bias in gender classification systems and text-to-image generation methods are presented; in Section 3 the core idea of the investigation is presented; in Section 4 the experiment set up is described; in Section 5, the results of the experiments are included; in Section 6 provides a legal analysis of the issue; and in Section 7, the work is concluded.
In transitioning to a detailed analysis of gender bias within AI systems, particularly in image gender classification, it's evident that these biases extend and amplify existing societal imbalances. One of the critical works in the debate on gender bias in AI systems is "Gender Shade" [18], which revealed a trend of gender classification systems, such as those offered by APIs like Microsoft or IBM, showing more significant inaccuracies in classifications related to women and, in particular, women of colour. Furthermore, it emerged that such inaccuracies tend to increase with darker skin tones, highlighting a problematic combination of gender and racial biases. However, understanding the causes of these biases is a complex task. AI systems often operate as "black boxes, " and the data used to train them are opaque, making it difficult to identify underlying behaviour.
Another consideration is that different gender classification services use different training data and infrastructure requirements. This suggests that each service may employ unique training data and have specific infrastructure requirements that influence gender classification [19]. This diversity among services raises questions about the consistency and reliability of gender classifications produced by AI systems. One of the main theories explaining gender bias in AI systems suggests that the discriminated demographic group may be underrepresented within the training data sets used for model training, as supported by the authors [18], showing that the most common data sets used in gender classification lacked diversity in terms of skin colour. However, this theory has been debunked by studies showing that balancing the training dataset did not eliminate bias [20]. In the search for further causes, an important discovery is that skin type does not seem to be the determining factor in the accuracy of gender classification. The problem seems more complex and is related to the persistence of stereotypes within this classification. This could make the issue even more intersectional, as it could harm multiple social classes. For example, [21] highlighted that makeup and eye features are significant predictive factors for classifying a face as female, raising concerns about the perpetuation of gender stereotypes. Another study [22] has suggested that besides makeup, features such as hairstyle, facial structure, and clothing could be more relevant than skin type in determining gender, justifying that women were more prone to false non-match rate (FNMR) than men in a face recognition context. [23] reported that gender differences in FNMR are not universal. Gender social conventions related to hairstyle and makeup, by definition, can vary significantly among social groups, so it seems likely that they manifest in various ways. Social conventions related to hairstyle and makeup also change with a person's age, and therefore, they play a role in understanding how facial recognition accuracy varies across age groups.
Furthermore, an interesting study [19] attempted to analyse these algorithms by considering the transgender community and trying to classify transwomen and transmen. One of the most exciting results of this research is that transgender men had the lowest true positives in gender classification, suggesting that there is a more cis-normative male representation for the male category, which is less subject to variety and diversity in the training data. Outside the world of research and computer science, the public has also begun to wonder about the causes of misclassifications or certain system decisions. In 2021, an algorithmic artist and transgender individual, Ada Ada Ada [24], attempted to test how algorithms perceive gender. By using her own transgender body, the artist found several methods for tricking gender recognition technology into seeing a gender different from the initial judgment. For example, she discovered how varying her emotional expression, head tilt, hair and beard, eyes, and nose could lead to being classified as a specific gender. This result comes after a series of machine tests and post-analysis of the results obtained. This project once again demonstrates how these classifications are based on social stereotypes.
Regarding text-to-image (TTI) systems, academic inquiry has underscored a marked prevalence of demographic biases, especially those pertaining to gender and race. These biases are evidenced through stereotypical depictions across assorted domains, such as vocations and personal traits, underscoring an inclination towards the over-representation of attributes linked with whiteness and masculinity [25,26]. Examinations have elucidated that a principal factor contributing to these biases is the instructional content utilised to train models, typically sourced from the internet, which mirrors the stereotypes and prejudices extant within society [25]. Despite recognising their inherent biases, employing models like CLIP [27] to steer the generative process in apparatuses such as Stable Diffusion further exacerbates the issue, as it amplifies the perpetuation of biases [28]. Furthermore, the exploration of representations engendered by TTI models has divulged that biases and stereotypes are not confined to the portrayal of individuals but also extend to objects, clothes, and even national identities [29], reflecting a wide spectrum of demographic biases. Whilst endeavours have been undertaken to ameliorate these biases, for instance, through the analysis of models' latent spaces to render the generated images more representative, the efficacy of such measures remains in question [30,31].
This work is inspired by the theories of cognitive psychology concerning gender perception and embarks upon a translational exploration from human to machine.
Gender discrimination within human cognitive processes has been extensively probed by cognitive psychology, with initial examinations focusing on how distinct facial features between males and females influence gender perception. Studies such as [32,33] have illuminated that facial dimensions, nose shape, prominence of jaws and eyebrows, and the structure of cheekbones contribute to gender perception, yet it is acknowledged that no single trait definitively dictates it. Indeed, it is highlighted that the complexity of gender perception surpasses mere physiognomy, showcasing how generalisations can readily evolve into stereotypes. This nuance in defining gender underscores the necessity of critically evaluating stereotypical visual representations of men and women-a phenomenon that has been extensively documented across various media [34,35,36], yet remains underexplored in the digital domain [37].
In this context, a pivotal role is played by the averageness theory, suggesting that faces deemed attractive, due to their prototypical nature, are more easily classified by gender [12,13,14,15]. This theory is supported by findings that facial attractiveness facilitates classification in adults. However, this correlation is observed to vary across genders, with attractive female faces generally perceived as highly feminine, unlike their male counterparts [38,39,40]. These dynamics introduce the concept of "face space", where faces are categorised in a multidimensional space based on the variance of facial traits and their encoding [41], with attractive and prototypically feminine faces positioned at the centre of this space.
Building on these foundations, our study explores how cognitive biases manifest within AI systems, specifically focusing on gender classification mechanisms. We delve into the potential propagation of gender stereotypes and investigate the role of attractiveness in classification accuracy. Recent research has largely focused on identifying biases related to gender, ethnicity, makeup usage, and skin colour. However, our approach aims to innovate by emphasising the impact of attractiveness-a composite attribute influenced by multiple facial features-which plays a crucial role in the perception and classification of gender.
Our analysis of bias is approached from two dimensions:
• First Level of Representation [42]: We question whether classification systems consistently achieve the same level of accuracy across all analysed groups (attractive/unattractive, women/men). • Conditional Demographic Parity [43]: We consider scenarios where biases may arise if the system systematically produces only a subset of possible labels, even if the algorithm's output is correct. For example, if men and women in a sample are dressed similarly, an unbiased algorithm would be expected to return the "clothing" label with equal frequency for each gender.
To summarise, our primary research question is: RQ1: How does the averageness theory influence the performance of AI algorithms in gender classification?
Several secondary questions support this:
• RQ1.a: Is there a difference in classification accuracy between attractive and unattractive faces within gender groups in AI algorithms? • RQ1.b: Do the performances of AI algorithms in gender classification maintain uniformity across different demographic groups, particularly when considering attractiveness? • RQ1.c: Do gender classification algorithms exhibit gender stereotypes, and in what forms do these stereotypes manifest?
When addressing the subject of attraction, a frequent critique is raised concerning its inherently subjective nature. However, it is noteworthy that datasets have been developed to tackle this aspect, exemplified by the HotOrNot dataset [44,45,46], the SCUT-FBP dataset [47], and the CelebA dataset [48]. In particular, The HotOrNot dataset was created by collecting user ratings of attractiveness from the HotOrNot website, a site launched in the 2000s where users rated pictures of individuals on a scale from 1 to 10, generating a large dataset of images paired with attractiveness scores. For example, a dataset version was formally presented in [46], where researchers used it to improve image annotation in attractiveness task scenarios. The SCUT-FBP dataset was developed to provide a more systematic set of facial images for beauty prediction tasks. Like the previous dataset, SCUT-FBP contains facial images with beauty scores annotated by multiple human raters. This dataset aimed to create a more controlled environment for studying facial attractiveness by ensuring in-scene and face expression conditions. Lastly, the CelebA dataset is a large-scale face attributes dataset with celebrity images, each annotated with 40 attribute labels, including one related to attractiveness. Unlike the previous two datasets, CelebA was designed to facilitate general research in face analysis (detection, attribute prediction, and other fields like beauty prediction). Despite these efforts, datasets in this domain exhibit significant variability, especially a lack of representation across various ethnicities or groups of people (e.g., SCUT-FBP is limited to 500 samples and focuses primarily on Asian and Caucasian faces, or CelebA contains only celebrity individuals). Moreover, HotOrNot is based on data collected from a website where people voluntarily uploaded photos of themselves to be rated; there was no control over the data type collected. This resulted in a lack of consistent distribution across ethnicity and gender, leading to an uncontrolled environment.
Consequently, for this study, a compact synthetic dataset has been compiled in light of these limitations, generated using Stable Diffusion [49,50] as shown in Figure 1. This approach's choice is deeply rooted in Stable Diffusion's comprehensive training across diverse image and label datasets. This training equips the model with a grasp of the subjective nuances of human attractiveness, allowing it to reflect the varied interpretations of facial attractiveness found in datasets. Moreover, the rationale for using a synthetic dataset lies in the ability to control and vary the generated images' attributes systematically. This approach addresses the limitations of existing datasets by ensuring a balanced representation of different ethnicities and providing a consistent framework for studying the subjective aspects of attractiveness. Additionally, synthetic data is gaining traction in data augmentation [51? ], making it intriguing to explore the type of representations these datasets can provide for this study. Finally, this method allows the investigation of the two dimensions of bias presented in section 3. In particular, these dimensions are analysed by testing various gender classification models, observing the variation in accuracy (First Level of Representation), and qualitatively analysing the results generated by Stable Diffusion for the requested prompt (Conditional Demographic Parity).
As mentioned, to create a balanced and diverse dataset, Stable Diffusion was tasked with generating images based on the following prompts: 'frontal photograph of an attractive/unattractive ethnicity man/woman'. Within these prompts, the descriptor ethnicity was systematically alternated with "White", "Black", and "Asian", ensuring a diverse representation of ethnicities within the resulting dataset.
Following this approach, the Stable Diffusion API was utilised to generate the dataset; specifically, version 2.1 of the stability/stable-diffusion model [50], accessible through the demo provided by Hugging Face, was employed. The default guidance scale value of 9 was kept. This process resulted in 200 images per class, resulting in 2400 images. The images were subsequently cropped to focus on the face, using the Multi-Task Cascade CNN (MTCNN) [52] face detection algorithm to standardise the size of the face portions in all images. This process made lose some of the sample generated since the face was not detect during the process. The final dataset contains 2324 crop images distributed according to specific proportions, as shown in Table 1.
Regarding the accuracy analysis, the models considered are those offered by Amazon Rekognition by Amazon Web Services (AWS) [53], the DeepFace library [54], and the InsightFace one [55]. Our selection included a commercial API, Amazon, and two projects widely recognised on GitHub: DeepFace (with 9.6k stars) and InsightFace (with 20.9k stars). Among these, DeepFace is the only one that provides data on the accuracy of its gender recognition model, achieving an accuracy of 97.44%.
To evaluate the model performances we take into account the following metrics:
• PPV (Positive Predictive Value): Positive Predictive Value, or PPV, is a metric used to analyse diagnostic test results or predictive models. In this formula, "TP" stands for true positives, cases where the test or model correctly predicted membership in a category. In contrast, "FN" stands for false negatives, where the test or model incorrectly predicted that an item does not belong to a class when it does. PPV measures the proportion of correct positive predictions relative to the total positive predictions and is expressed as a percentage.
• ER (Error Rate): Error Rate is another metric used to evaluate the performance of tests or prediction models. In this formula, "FP" stands for false positives, i.e., cases where the test or model erroneously predicted membership in a category when it does not, and "FN" still represents false negatives. Error Rate measures the proportion of incorrect predictions relative to the total predictions and is expressed as a percentage. A lower error rate indicates a higher accuracy of the test or model. The metrics of interest spanned gender, attractiveness, and ethnicity, as shown in the tab. 2, 3 where A stands for attractive, "U" for unattractive, "M" for men and "W" for women. An analysis of gender classification revealed a noticeable difference in model performance, particularly when examining the gradient of attractiveness. Whether deemed attractive or not, the model's accuracy showed little variation for male subjects. However, when classifying female subjects, the models displayed a change in accuracy between attractive and unattractive groups. InsightFace's Positive Predictive Value (PPV) decreased from 85.61% for attractive women to 62.74% for those considered unattractive. The disparity was even more pronounced for DeepFace, which saw a PPV drop from 67.5% for attractive women to 21.22% for unattractive women. Amazon Rekognition generally emerged as the most robust model; nonetheless, a slight performance difference was observed: from 100% PPV for attractive women to 88.96% for unattractive women. Regarding ethnicity, the most disadvantaged group for InsightFace and DeepFace was unattractive black women, with an error rate of 44.44% for InsightFace and 85.86% for DeepFace. Conversely, for Amazon Rekognition, black subjects were less disadvantaged, while unattractive Asian women showed a more significant performance degradation.
While InsightFace and DeepFace showed variable performance between attractive and unattractive females, Amazon Rekognition consistently maintained high accuracy across gender and attractiveness attributes. The calculated error rate disparities across Amazon Rekognition, InsightFace, and DeepFace models underscore distinct biases concerning gender and perceived attractiveness, as documented in the figure 2. InsightFace exhibited a minimal gap for men, suggesting uniform performance across levels of attractiveness. However, a substantial disparity was observed for women, with unattractive women experiencing significantly higher error rates. DeepFace revealed the starkest contrast in error rates among women, with unattractive women facing dramatically higher error rates, highlighting a potential bias towards attractiveness in female gender recognition. Deepface presents a higher difference between the error rate gaps between the male and female samples, 45.80.
These findings indicate a persistent trend where women, especially those categorised as unattractive, are at a disadvantage due to higher error rates.
To understand the kind of physiognomy generated by the Stable Diffusion model in response to the prompt, a qualitative analysis of the dataset was conducted by examining images of individuals, leading to several observations. Initially, average faces were created by overlaying images of various groups of attractive and nonattractive individuals, both women and men. This analysis observed expressive differences between averagely attractive and non-attractive faces. The average attractive face invariably appears smiling or calm, whereas the average non-attractive face tends to exhibit a more severe expression. Furthermore, attractive average faces seem younger than their non-attractive counterparts (fig. 3). From this initial analysis of averages, with scrutiny applied to each image in turn, three interesting observations have emerged:
• Differences in makeup: There is a significant incidence of makeup on the average attractive and non-attractive face of women. Attractive women show signs of makeup with a more pronounced application, especially on the lips. On the other hand, non-attractive women have lighter or even absent makeup, especially Asian women. An exception is Black women, who are rarely generated without makeup regardless of attractiveness. • Similarities among Black subjects:It is interesting to note that there are no significant differences between the average attractive and non-attractive faces for Black men and women, except for a slightly more smiling expression in attractive Black subjects. Moreover, Black subjects are never generated without a beard, as shown by the average faces, which are attractive and non-attractive in the presence of the beard. • Age Gap: Among women of all ethnicities, youth appears to be a distinctive trait of attractiveness.
White women deemed attractive tend to show youthful visual characteristics, with rare exceptions of white hair, suggesting a strong link between youthfulness and the perception of beauty. Conversely, non-attractive women of the same ethnicity appear older, with a greater frequency of white hair, suggesting that ageing may negatively impact their aesthetic perception. A similar trend is observed among Asian women, where youthful features prevail among those considered attractive, while their non-attractive counterparts show more marked signs of ageing. However, Black women seem to follow the trend but with a less pronounced age gap. Regarding men, the link between age and attractiveness manifests less markedly than in women but remains significant. Attractive white men can vary in age, displaying both youthful characteristics and signs of ageing, such as white hair or wrinkles, indicating a broader range of attractive traits. Non-attractive men, however, tend to be characterised by a seemingly more advanced age. This pattern of age-related variation is also reflected among Asian men. As with women, the trend is respected for Black men but with a smaller gap.
After conducting the primary analysis, it was decided to validate the observations concerning makeup differences and the age gap using attribute classification systems. Amazon Rekognition was utilized for the age attribute, as the API also provided age detection. Among all the evaluated models, it exhibited the fewest errors in gender classification. A trained lightened moon Mxnet model was employed for facial attributes [56], especially the pre-trained on the CelebA dataset's attribute labels [57].
• Age analysis: Amazon Rekognition allows the detection of an age class. Since this range is variable, to facilitate the analysis, we decided to adopt a set of fixed ages ranging from 0 to "≥ 70", proceeding by decades as reported in the tab. • Attribute analysis: Some earlier observations can be validated by examining the datasets through attribute detection. Initially, the age gap is validated by the observation that for both women and men, the percentage of the "Young" attribute decreases when moving from attractive to unattractive subjects, supporting the notion that unattractive individuals are typically older. Furthermore, the difference in makeup application is also evident. Among attractive women, "Wearing Lipstick" and "Heavy Makeup" are observed as among the top four detected attributes, with respective percentages of 91.50% and 71.74%. Conversely, for unattractive women, makeup does not present as a significantly detected attribute, with only a "Wearing Lipstick" percentage of 16.15% being observed (tab 5).
Some controversial images were generated during the generation of images using Stable Diffusion. Despite the explicit prompt instructing the model to display a face, often the generated images represented bodies or body parts instead. One noteworthy observation is that body parts such as lips and prominently emphasized breasts were often generated for attractive women 4.B. Another remarkable case was the image generated in response to the prompt for unattractive Black women, as shown in figure 4.A. The image primarily depicts a partially nude, censored in intimate areas, pregnant abdomen. Additionally, there is a broader case of curvy faces among the non-attractive groups (figure 4.C). For men, this is also evident in the attribute detection results showing the attribute "chubby" among the top 10 detected. This last observation resonates with the stigmatization of fat bodies, which are often seen as unproductive and inefficient in Western culture [58]. This connection again underscores societal biases' role in shaping AI outputs, where attributes such as 'chubby' linked to 'unattractive' become markers of negative judgment.
The analysis of the averageness theory revealed that specific physical attributes associated with attractiveness, such as facial features or perceived age, inadvertently influence gender classification within AI systems. Following the Gender Shades study [18] hypothesis, it is possible that this result is linked to the type of descriptive features of the class most present in the training data.
The EU Non-discrimination legislation is crucial for safeguarding a high level of (e)quality in AI development and implementation settings. The obligation to respect the principle of non-discrimination is enshrined in EU primary law, in Article 2 of the Treaty on European Union (TEU) 1 , Article 10 of the Treaty on the Functioning of the European Union (TFEU) 2 (requiring the Union to combat discrimination on several grounds) and Articles 20 and 21 of the EU Charter of Fundamental Rights (equality before the law and non-discrimination based on a non-exhaustive list of grounds) 3 . All prohibited grounds of discrimination, as listed in the Charter, are relevant regarding using algorithms.
In the AI context, algorithmic discrimination has become one of the critical points in the discussion about the consequences of an intensively datafied world [59]. The quality of the datasets used to train machine learning algorithms is of prime importance to the performance of AI systems, as "an algorithm is only as good as the data it works with" [60]. When data is gathered, it may contain socially constructed biases, inaccuracies or errors that must be tackled before any training based on this dataset [61]. One of the reasons explaining the existence of bias in datasets is the "choice of subjects to the omission of certain characteristics or variables that properly capture the phenomenon we want to predict, to changes over time, place or situation, to the way training data is selected" [62]. AI algorithms trained on poor-quality information -both from the quantitative and qualitative point of view-can negatively affect the outputs or decisions of these mechanisms, leading to "incorrect model predictions" [63]. It is worth recalling that a dataset "is always a reflection of the society from which the information has been obtained. If the society contains discriminatory elements and structures, these are also in the training data set" [64].
More than that, datasets used for training AI systems "may suffer from the inclusion of inadvertent historical bias, incompleteness and bad governance models". The perpetuation of such biases could lead to inadvertent (in)direct prejudice and discrimination against certain groups or people, potentially exacerbating prejudice and marginalisation [60]. For instance, AI training datasets may exclude or under/misrepresent people from different geographical areas, neglecting or misconstruing their interests and needs. This exclusion can potentially exacerbate current inequalities and further marginalise these communities. [65].
The AI Act mandates specific requirements and restrictions regarding the use of data for the development, training and testing of AI systems, encompassing factors like the quality, relevance, accuracy, representativeness and diversity of the data (Recitals 14a, 28a, 38, 43,44, 45 AI Act etc.), as well as the respect for the rights and interests of the data subjects and the data providers (Articles 9, 10, 54 and 55 AI Act). AI training and testing can be made either by datasets of real data or synthetic data (fake data)-like in our case. Synthetic data is artificial data generated from original data (real data) and a pre-trained model to reproduce the characteristics and structure of the original data (Recital 111 AI Act).
Whenever real data is used, Articles 9 (1), 6 (1)(a), (b), and (f) GDPR are applied. Yet, the AI Act lacks explicit guidance on the proper procedures and legal basis for processing such data, particularly concerning consent acquisition and providing information and transparency, which are enshrined in the GDPR. GDPR provides the proper legal framework by imposing different or additional conditions and constraints on the use of data for AI purposes. However, in practical terms, GDPR might be hardly invoked by the data subjects. The absence of this clarity raises questions about how the data subjects could ask to restrict processing (Article 18 GDPR) and to delete and erase data (Article 17 GDPR).
At this point, we should highlight that the initial requirement outlined in Article 10 (5) (a) AI Act states that data processing under this article is permissible only when its goal, specifically bias detection and correction, "cannot effectively be achieved through processing synthetic or anonymised data." This means synthetic or anonymised data should first be used to identify and correct bias. In contrast, real data can be used only when the requirement of synthetic or anonymised data is exhausted. Furthermore, when biases are based on sensitive data-like ethnicity data-the AI Act requires using anonymised 1 Treaty of the European Union, 1992, https://www.cvce.eu/content/publication/2002/4/9/ 2c2f2b85-14bb-4488-9ded-13f3cd04de05/publishable_en.pdf. 2 Treaty on the Functioning of the European Union, 1958, https://eur-lex.europa.eu/resource.html?uri=cellar: 2bf140bf-a3f8-4ab2-b506-fd71826e6da6.0023.02/DOC_1&format=PDF. 3 European Charter of Fundamental Rights,2012, https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:12012P/ TXT. data to process sensitive data as a primary bias detection and correction tool [66]. AI-friendly data anonymisation tools align with the Recital 45 AI Act, which states, "Practices that are prohibited by Union law, including data protection law, non-discrimination law, consumer protection law, and competition law, should not be affected by this Regulation". It is worth mention [66]ing that the inclusion of synthetic data in the AI Act was underlined by the European Commission's Joint Research Centre [67], supporting rebalancing mis/under-represented groups of people in ethnicity, gender, etc.
Another important aspect of gender classifiers, from a legal point of view, is to which extent they pose a risk to human rights (Articles 3 (2) and 27 AI Act). The AI Act adopts a risk-based approach (Article 9 AI Act) to oversee AI systems, categorising them into prohibited, high-risk and low-risk categories. However, the criteria and thresholds for determining the risk level of an AI system are not consistently clear. This lack of clarity may result in the exclusion of specific AI systems that may pose significant risks to data protection rights, such as those that process sensitive personal data or involve large-scale processing of personal data. For this purpose, a gender-based risk assessment in our study case is needed.
To begin with, AI systems that profile individuals based on automated processing of personal data to assess various aspects of a person's life, such as work performance, economic situation, health, preferences, interests, reliability, behaviour, location, or movement, are always considered high-risk AI systems. For instance, one area in which AI profiling is used is border management control by law enforcement agencies [68,69].
The classification rules for high-risk AI systems are enshrined in Article 6 (2) AI Act and Annex III AI Act. Remote biometric identification, biometric categorisation and emotion recognition systems are considered high-risk AI systems. Yet, the two requirements outlined in Article 6 (1) AI Act should be fulfilled to consider the above-mentioned AI systems as high-risk AI systems. More particularly, the AI system should intended to be used as a safety component of a product, or the AI system is itself a product, covered by the Union harmonisation legislation listed in Annex I. Additionally, the product whose safety component under point (a) is the AI system, or the AI system itself as a product, is required to undergo a third-party conformity assessment, with a view to the placing on the market or the putting into service of that product according to the Union harmonisation legislation listed in Annex I.
Needless to say, these requirements are not fulfilled in our study case, which was an experiment for academic purposes (Article 2 (6) AI Act). Yet, for the above reasons, if this gender classifier was placed in the market, it would be considered a high-risk AI system. Moreover, solely for the legal analysis of our study case, we must stress that the data used is biometric derived from facial images. Biometric data is personal data resulting from specific technical processing relating to a natural person's physical, physiological or behavioural characteristics, which allow or confirm the unique identification of that person. Biometric data, like data revealing racial or ethnic origin or someone's sexual behaviour or sexual orientation, are considered a special category of data (Articles 3 (35) AI Act and 9 (1) GDPR). However, gender data is not. This difference is important, as GDPR does not offer extra protection for gender data processing, which signals that gender can be stored and processed without further constraints.
The legal analysis concludes with three key points. Firstly, it underscores the importance of enhancing data curation practices to improve the reliability and predictability of gender classifiers. This measure aims to mitigate the risk of generating biased outcomes that could result in arbitrary discrimination, unfair decisions, denial of services, or inappropriate interference with individuals' fundamental rights or freedoms. Data collection, curation and selection are essential components of AI risk management [70] and a fundamental aspect of the AI data governance framework. This framework aims to establish robust procedures ensuring high-quality data availability, labelling, and use.
Secondly, the feminist theory of the social construction of gender offers insights into patriarchal power dynamics and the privileges of data (cis-normative) male domination. Through this lens, we can interrogate how to advance gender equality and prevent non-discrimination through data quality and curation. Given that "attractiveness" and beauty ideals differ across cultural and geographic boundaries, we endorse a feminist intersectional approach [71] that acknowledges and accommodates these diversities. This approach aims to ensure that datasets used for AI training are not predominantly focused on white, middle-class, young, and heterosexual women.
Lastly, advocating for an interdisciplinary approach to developing AI applications becomes imperative in light of this background. This entails integrating the averageness theory from the field of psychology and analysing the legal safeguards to address gender bias in AI. The AI Act emphasises the necessity of such collaboration, as highlighted in Recital 142, to ensure that AI advancements yield socially advantageous results and address socio-economic disparities. This involves fostering cooperation among AI developers, experts in inequality and non-discrimination, academics, and other relevant stakeholders.
This exploration of the complexities of gender perception and classification within artificial intelligence highlights the intricate interplay between technical capabilities and socio-psychological insights. Through the lens of the "averageness theory" and the creation of a synthetic dataset, the study illuminates the intrinsic biases associated with traditionally attractive features. Demonstrating how attractiveness, understood as youthfulness, the use of make-up, or particular facial expressions, algorithmically influences representations, particularly those of women.
A qualitative and quantitative analysis of the generated dataset reveals that men are also subject to gender stereotypes, with solid conformity to cis-normative stereotypes (for instance, having a beard or short hair). Consequently, the issue does not concern only the binary categorisation of gender but also how, within this dichotomy, stereotypes of femininity and masculinity, often tied to stereotypical beauty standards, are perpetuated and reinforced by AI algorithms. Such distortions can have unintended and harmful consequences for individuals and groups, underscoring the urgent need for ethical and responsible AI development.
In this context, the introduction of legislative discussions, particularly the efforts of the European Union exemplified by the AI Act, emphasises the necessity of integrating ethical, legal, and technical considerations to ensure the development of artificial intelligence technologies that respect human rights and equality. Indeed, from a legal standpoint, ensuring the regulation of diverse and representative datasets is fundamental [72]. However, adequate data curation requires a solid understanding of the various biases and discriminations that need attention and correction [73,74]. For this reason, this work promotes the necessity of an interdisciplinary approach in the development phases of AI technologies. Beyond cognitive psychology, we highlight how studies such as the feminist theory of the social construction of gender offer insights into the dynamics of patriarchal power and the privileges of cis-normative male dominance in the data collection process.
Only through an integration of technological innovation, social analysis, and solid regulatory oversight can we achieve an AI that is not only advanced but also fair and inclusive. Artificial intelligence technologies that truly serve society require commitment from all social actors since implementing fairness measures within AI algorithms is indispensable despite their unavoidable non-neutrality [74]. As we lay the groundwork for future explorations, this study advocates for a multidisciplinary strategy to identify and mitigate AI biases. The intertwining of technological innovation with robust regulatory oversight presents a promising avenue towards developing AI that is not only advanced but also aligned with the principles of fairness and inclusivity, thereby ensuring that AI serves the diverse needs of the global community.