In recent years, artificial intelligence (AI) systems have been increasingly utilized in the labor market, with many employers relying on them in the context of human resources (HR) management. However, this increasing use has been found to have potential implications for perpetuating bias and discrimination. The BIAS project kicked of in November 2022 and is expected to develop an innovative technology (hereinafter: the Debiaser) to identify and mitigate biases in the recruitment process. For this purpose, an essential step is to gain a nuanced understanding of what constitutes AI bias and fairness in the labor market, based on cross-disciplinary and participatory approaches. What follows is a preliminary overview of the design and expected implementation of the project, as well as how our project aims to contribute to the existing literature on law, AI, bias, and fairness.
(A. Puttick); (M. Kurpicz-Briki)
Addressing a lack of consensus about the interpretation of AI bias and fairness in the labor
market, we begin our research with a systematic literature review. In brief, we draw on the
traditional distinction of individual fairness versus group fairness [
In validating and contextualizing the research outputs of the literature review, we also perform some fieldwork. To be precise, we have interviewed 70 HR managers and AI developers, who are located in Estonia, Iceland, Italy, the Netherlands, Norway, Switzerland, and Turkey, to ascertain their views on the use of AI applications in the labor market and its impact on fairness. Briefly, it appears that most respondents have a positive attitude towards the deployment of AI applications for recruitment and selection purposes, while voicing concern about the management of staf. Although lacking a common understanding of fairness, a number of requirements are listed for AI-driven human resources practices to be considered fair, especially in terms of human oversight and transparency. Also, respondents call for the adoption of mitigation measures to address diversity biases, with special emphasis on gender bias. Besides AI-driven solutions similar to the Debiaser, they cite the examples of diversity quotas, DI oficers, specific guidelines, and training on the matter. At the same time, we have drafted a Qualtrics survey of workers’ attitudes towards AI, the aim of which is to understand workers’ experiences and opinions of AI applications used in an HR management context and how they might lead to bias and discrimination. Although the survey has just been launched and will remain open until July, it has been reported that the majority of respondents have a fairly negative attitude towards the use of AI applications in the labor market. In any case, for AI-driven HR practices to be considered fair, they believe in the provision of equal opportunities despite the personal characteristics of the individual as well as the transparency and explicability of the technology making a specific decision.
Based on the literature review and the fieldwork, the next step of the BIAS project will be the design and piloting of the Debiaser. In doing so, we contribute to an existing body of research suggesting possible remedies for AI bias in both social and computer science [18]. The Debiaser is conceptualized as a tool for bias detection and mitigation in the context of a common use case for text-based AI recruitment tools—the automatic pre-selection of job applicants for first-round interviews. The tool will be built upon the principles of trustworthy AI [20] and incorporate the rich knowledge base assembled by consortium partners in the framework of the BIAS project. A major contribution of our work will be the extension and adaptation of existing methods to EU regional languages and local prejudices, with a focus on specialized methods targeting fair recruitment practices.
The scientific research we conduct explores weak points in existing AI recruitment tools. The first concerns of-the-shelf word embeddings such as Word2Vec [ 21] and pre-trained large language models (PLMs) such as BERT [22] or GPT-3 [23]. Highly capable text processing models require extensive resources in terms of time, computing power and data, and thus many AI applications, including recruitment tools, are built using such word embeddings or PLMs. These machine-learning-based methods often reproduce prejudices existing in their training corpora, with a notorious example being Amazon’s own (now scrapped) job candidate selection tool, which was demonstrated to systematically discriminate against women [24]. We aim to utilize and further develop methods for detecting, measuring and mitigating bias in word embeddings and PLMs, such as those introduced by Caliskan et al. [25], Guo and Caliskan [26] and Ahn and Oh [27]. This builds upon previous work which extends the methods of Caliskan et al. [25] to French, German, Italian and Swedish word embeddings [28, 29].
Most current AI applications make use of black-box neural network models whose decisionmaking process is very dificult to distill into human-understandable explanations. This inhibits trust in such systems and complicates the task of auditing algorithmic decisions. To this end, we aim to utilize state-of-the-art methods in explainable natural language processing (XNLP) [30], aiming to provide clear explanations for all stages of bias detection and mitigation deployed within the Debiaser. The XNLP methods we will integrate can be sorted into two basic approaches toward improving the interpretability of algorithmic decisions: Feature-selection methods [31, 32, 33] aim to identify which words or phrases in the input data played an important role in a given model’s output decision. The second category of techniques generate natural language explanations which should explain the model’s decision in comprehensible, plain-language terms [34, 35]. In addition, our research will target the detection of text that is susceptible to unfair discrimination and explore the potential of suggested rewrites as a technique for avoiding such discrimination, using reinforcement learning-based techniques in the same vein as Sharma et al. [36]. Both the suggested rewrites and natural language explanations generated by the Debiaser can be further improved using human feedback to further train the underlying models with the goal of generating output corresponding to user preferences [37, 38].
The growth of the role of AI in HR management is to continue, but its vast potential comes with various challenges. AI applications are far from being free from bias and discrimination, and AI developers and HR executives should be prepared to grapple with dificult questions and ensure that this technology is implemented in a responsible manner. For this purpose, the BIAS project aims at identifying and mitigating diversity biases in the labor market, by combining cross-disciplinary fieldwork and designing new technological solutions.
This work is part of the Europe Horizon project BIAS funded by the European Commission, and has received funding from the Swiss State Secretariat for Education, Research and Innovation (SERI). [16] M. MacCarthy, Standards of fairness for disparate impact assessment of big data algorithms,
Cumb. L. Rev. 48 (2017) 67. [17] B. Friedman, H. Nissenbaum, Bias in computer systems, ACM Transactions on Information
Systems (TOIS) 14 (1996) 330–347. [18] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, A. Galstyan, A survey on bias and fairness in machine learning, ACM Computing Surveys (CSUR) 54 (2021) 1–35. [19] A. D. Selbst, D. Boyd, S. A. Friedler, S. Venkatasubramanian, J. Vertesi, Fairness and abstraction in sociotechnical systems, in: Proceedings of the conference on fairness, accountability, and transparency, 2019, pp. 59–68. [20] A. Hleg, Ethics guidelines for trustworthy ai, B-1049 Brussels (2019). [21] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality, Advances in neural information processing systems 26 (2013). [22] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [23] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [24] J. Dastin, Amazon scraps secret ai recruiting tool that showed bias against women, in:
Ethics of data and analytics, Auerbach Publications, 2018, pp. 296–299. [25] A. Caliskan, J. J. Bryson, A. Narayanan, Semantics derived automatically from language corpora contain human-like biases, Science 356 (2017) 183–186. [26] W. Guo, A. Caliskan, Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases, in: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 2021, pp. 122–133. [27] J. Ahn, A. Oh, Mitigating language-dependent ethnic bias in bert, arXiv preprint arXiv:2109.05704 (2021). [28] M. Kurpicz-Briki, Cultural diferences in bias? origin and gender bias in pre-trained german and french word embeddings (2020). [29] M. Kurpicz-Briki, T. Leoni, A world full of stereotypes? further investigation on origin and gender bias in multi-lingual word embeddings, Frontiers in big Data 4 (2021) 625290. [30] M. Danilevsky, K. Qian, R. Aharonov, Y. Katsis, B. Kawas, P. Sen, A survey of the state of explainable ai for natural language processing, arXiv preprint arXiv:2010.00711 (2020). [31] M. T. Ribeiro, S. Singh, C. Guestrin, ” why should i trust you?” explaining the predictions of any classifier, in: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016, pp. 1135–1144. [32] S. Rathi, Generating counterfactual and contrastive explanations using shap, arXiv preprint arXiv:1906.09293 (2019). [33] J. Li, X. Chen, E. Hovy, D. Jurafsky, Visualizing and understanding neural models in nlp, arXiv preprint arXiv:1506.01066 (2015). [34] O.-M. Camburu, T. Rocktäschel, T. Lukasiewicz, P. Blunsom, e-snli: Natural language inference with natural language explanations, Advances in Neural Information Processing Systems 31 (2018). [35] S. Narang, C. Rafel, K. Lee, A. Roberts, N. Fiedel, K. Malkan, Wt5?! training text-to-text models to explain their predictions, arXiv preprint arXiv:2004.14546 (2020). [36] A. Sharma, I. W. Lin, A. S. Miner, D. C. Atkins, T. Althof, Towards facilitating empathic conversations in online mental health support: A reinforcement learning approach, in: Proceedings of the Web Conference 2021, 2021, pp. 194–205. [37] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, D. Amodei, Deep reinforcement learning from human preferences, Advances in neural information processing systems 30 (2017). [38] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, G. Irving, Fine-tuning language models from human preferences, arXiv preprint arXiv:1909.08593 (2019).