This work examines the representation of protected attributes across tabular datasets used in algorithmic fairness research. Drawing from international human rights and anti-discrimination laws, we compile a set of protected attributes and investigate both their availability and usage in the literature. Our analysis reveals a significant underrepresentation of certain attributes in datasets that is exacerbated by a strong focus on race and sex in dataset usage. We identify a geographical bias towards the Global North, particularly North America, potentially limiting the applicability of fairness detection and mitigation strategies in less-represented regions. The study exposes critical blindspots in fairness research, highlighting the need for a more inclusive and representative approach to data collection and usage in the field. We propose a shift away from a narrow focus on a small number of datasets and advocate for initiatives aimed at sourcing more diverse and representative data.
Algorithmic fairness has become a significant area of research in recent years, with a growing body of work aimed at addressing bias and discrimination in machine learning systems. Identifying and mitigating harmful practices against vulnerable individuals and groups in prediction algorithms lies at the core of this field and to study these issues adequate and nuanced data sources are needed.
In this work, we examine datasets and how they are used within the fairness literature. We present an overview of attributes which are protected by anti-discrimination legislation across multiple continents and study their availability in datasets and usage in fairness research. We identify issues regarding the diversity of protected attributes represented in datasets and their geographic representativeness, highlighting how populations are neglected in the literature.
For this work, we collected and manually annotated usage of tabular datasets in fair classification
tasks. We built on top of a comprehensive survey of fairness datasets by Fabris et al. [
To define protected attributes, we draw from domain-specific legislation and human rights
law. We define as protected all attributes which are explicitly mentioned as prohibited drivers of
discrimination and inequality. For example, Article 21 of the Charter of Fundamental Rights of
the European Union states “Any discrimination based on any ground such as sex, race, colour,
ethnic or social origin, genetic features, language, religion or belief, political or any other
opinion, membership of a national minority, property, birth, disability, age or sexual orientation
shall be prohibited” [
We try to adress the Global North and especially U.S. focus in AI ethics and fairness research
[
Drawing from this literature, we provide a shallow categorization of protected attributes, identifying seven main categories (Table 1). It is worth noting that this is not a complete categorization of all protected attributes around the globe and across sectors. Rather, our categorization aims to guide an inclusive discussion of algorithmic fairness research through the lens of protected attributes.
The geographical provenance of datasets used in the examined literature is clearly skewed.
Among datasets from a single continent, 21 come from North America, 5 from Europe, 2 from
Asia, and 2 from South America, confirming a Global North and especially North American
dominance in AI ethics research [
We further notice a highly uneven distribution of both the availability and usage of protected attributes. The left bar chart in Figure 1 depicts protected attributes available in fairness datasets and the right chart their usage in the examined literature. There is a particular focus in both availability (n=17) and usage (n=167) on race as a protected attribute. On the other hand, attributes about religion, belief and opinion are entirely missing on both sides. Information on disability and health conditions is also infrequently available ( = 3) and never used in the surveyed literature. Socioeconomic status descriptors are more commonly available yet often neglected. This threatens the applicability of research findings across contexts, as information on race for example is hardly available in EU data [16].1
We unveil blindspots in fairness research, demonstrating a neglect of vulnerable subpopulations in the literature. We will further present additional results from our data collection at the conference, indicating other troubling practices in the field, such as a lack of suficient reporting 1There was also a small number of protected attributes used in the literature but not referenced in legislation, such as employment status, alcohol consumption, neighborhood, body-mass index, and profession.
n=14 n=17 n=8 n=8 n=8 n=8 n=7 of dataset usage impacting reproducibility and potentially harmful practices in the processing of protected attributes leading to a neglect of minorities.
While valid reasons exist against the collection of protected data [17], motivating e.g. the line of work on fairness under unawareness [14, 18], we believe they are not suficient to explain the observed lack in usage of particular attributes. We observe a clear trend towards certain protected attributes being more readily available in datasets which, however, is amplified by a strong tendency of papers to (1) repeatedly focus on the same small number of datasets and (2) especially rely on race and sex as protected attributes. It is worth noting that this trend extends to fairness research more broadly, including qualitative studies. These practices also have a tendency to self-reinforce, increasing the likelihood of future research to conform. Recent articles published at fairness conferences, such as FAccT (the ACM Conference on Fairness, Accountability, and Transparency) and AIES (the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society), for example, mention race and gender by an order of magnitude more frequently than religion, disability, socioeconomic status, and sexual orientation [19].
We argue for a move towards a research roadmap to tackle these issues within the complex
social, legal and technical landscape they reside in (as advocated, for example, in Guo et al.
[20]). In particular, we propose a move away from focusing exclusively on a small number
of datasets[