Candidate ranking systems (CRSs) for vacancies can pose a significant risk in terms of ethical considerations if they are prone to gender bias or even have legal implications if discriminatory behavior is found. In the case of content-based CRSs, which identify suited candidates for a given job opening based on their resumes and the job advert, gender bias in these texts can also lead to discriminatory behavior of the CRS algorithm. We propose an algorithm to automatically identify gendered words in the job advertisement responsible for gender bias in the rankings. The algorithm determines the words with gendered connotations in the rank distribution for a given job advertisement using content-based job-candidate matching based on the actual biography of a candidate and a counterfactual version in which explicit gender-mentioning terms are swapped between male and female. To this end, we employ the neural network explainability method of integrated gradients to compute CRS's association of the job advertisement words with the gender of candidates, which we call the bias score of words. At the core of our CRS is a cross-encoder architecture. To showcase and validate our approach, we conduct a study investigating the gendered words identified by the proposed algorithm in job advertisements from a private dataset and biographies from the BIOS dataset. We analyze the gendered words along multiple job categories and diferent linguistic categories. Finally, we statistically and qualitatively compare them with standardized lists manually created by social psychologists to contrast the gender associations CRSs make with human associations.
Candidate selection for jobs has become a very dificult
task for human recruiters to complete due to the vast
amount of applicants for a job advertisement. This has
led to the usage of candidate ranking systems (CRSs). As
man recruiters [
CRSs commonly leverage content like resumes and nEvelop-O (M. Schedl) Human Resources, in conjunction with the 17th ACM Conference on advertisements that cause CRSs to rank candidates differently only because of their gender. Our contribution is two-fold: 2.
and compare them with a list of words from pre- candidate pair as input and outputs the relevance score
This work can help us understand the distinction be- bag-of-words based BM25 model [
• We present an algorithm to identify the words in job advertisements causing gender bias in the candidate ranking. • We analyze the words identified by our algorithm,
vious studies curated by social psychologists. vertisement. Both perspectives are essential as the CRS’s perspective will help us reduce algorithmic gender discrimination while the human perspective will help us create a job advertisement desirable to both male and female job seekers. Furthermore, the comparison between the two views may help us better understand the distinction or similarity in the potentially discriminatory nature of a job advertisement for CRSs and humans. As we will see, seemingly gender-neutral job advertisements can inadvertently lead to discriminatory practices by CRSs.
Additionally, the use of biased language in job ads may not necessarily result in discriminatory behavior by CRSs.
It is essential to remain mindful of these potential issues in order to cultivate a more inclusive and equitable hiring process.
The structure of the remaining paper is as follows: In Section 2, we introduce the CRS used in this work and explain our algorithm to identify words in job advertisements causing gender bias in the rankings of the CRS. Subsequently, in Section 3, we describe the dataset used in the experiments. Section 4 describes the setup of the experiments, and their results are presented and discussed in Section 5. Finally, Section 6 summarizes our work and gives directions for future research endeavors.
We use an LLM-based cross-encoder [
() Given: Job ad , and recommended candidate list Given: Trained ranking model . (< , >) is used for ranking , ∈ Given: to create gender counterfactual candidate.
is candidate with opposite gender
Job Ad : = { 1, ..., } , consist many tokens
: ( Bias score : = { 1, ..., } Temporary bias score : = { 1, ..., } , < , >, ) ← | ( , < , >, ) − ( , < , () >, )| ← 0∀ ∈ = {} for each ∈
do ← 0∀ ∈ for each ∈ do end for ← ( ) for each ∈ do if < 1 then
← 0 end if end for ← ⊕ ∗ 1 log(()∈+1) end for for each ∈ do if > then
. ( end if end for return )
We approach the first contribution by identifying the words in a job advertisement responsible for gender bias in the ranking of candidates. For this purpose, we create for each candidate, an artificial gender-counterfactual candidate. We do this by replacing gendered words, pronouns, and names in the candidate’s textual materials Job title distribution in train, test, and validation set. Job Titles #Train
#Validation
the integrated gradient explainability method [
Integrated gradient of a word in job ad with candidate and ranking model is where is the number integral approximation steps and < , >
is the original input to the model , < ′, ′ > is the same size masked input (i.e., for the model it is a blank job advertisement and candidate content of the same length as original input)
Algorithm 1 is used to identify words in job ad that contribute to the diference in the relevance score of a candidate and its gender counterfactual ()
by a trained CRS model . Integrated gradient
is used to identify the contribution of words towards the diference, and we further use
to normalize . We scale the normalized based on the rank of candidate in the ranking by . Finally, we use a bias threshold to cut of the less critical red-words. Counterfactual didate’s nouns and pronouns with that of the opposite gender. For simplicity, in this work, we restricted the transformation to a binary behavior.
We created the dataset for the experiment using
biographies from BIOS dataset [
The size of the dataset created by matching is 2775 job posts and 322,337 biographies covering 24 diferent jobs.
Of these 24 jobs, 10 have a job advertisement frequency of less than 5 and are removed. Further, a subset of the BIOS dataset is created such that the subset is balanced according to the job and gender of candidates. The size of the balanced subset of biographies is 1400, where each profession has 50 male and 50 female biographies. Thereafter, we split the job advertisement into train, test, and validation sets with stratification of job titles by 70:20:10 split, respectively. So, finally, we are left with 14 diferent job titles, 1400 biographies with 50 males and 50 females of each job title, and 2085 job advertisements with job distributions shown in Table 1 The biographies have been pre-processed by replacing real names with ”bob” for males and ”alice” for females.
Additionally, counterfactual biographies have been generated by replacing gender-specific words with those of the opposite gender. The male-coded words used are female-coded words are “alice,” “mrs,” “hers,” “she,” “her,” and “herself.”
Experiments are conducted using a BERT-based cross
encoder, i.e., CRS, over our collection of job
advertisements and biographies of candidates (Section 3). CRS
is trained for four epochs using the sigmoid variant of
binary cross entropy loss3 on our collection. We report
ranking performance in terms of nDCG and bias in terms
of true positive rate parity (TPRP) [
BCEWithLogitsLoss.html, access: July 2023 software, senior, engineer, development, team, engineering, experience, design, code, java software, engineer, team, development, experience, technology, engineering, data, code, engineers dental, dentist, practice, associate, nhs, care, patients, clinical, private, patient legal, para, team, firm, law, litigation, client, property, role, commercial nurse, nursing, nurses, residents, home, training, registered, clinical, shifts, team school, pupils, teaching, teachers, children, teacher, students, staf, schools, curriculum architect, projects, design, architectural, practice, residential, team, working, architects, experience accountant, accounting, accounts, management, tax, finance, audit, reporting, business, experience painter, decor, painters, painting, looking, shift, working, refurbishment, email lead, depend, support, logic, principle, depend, understand, active, child, superior
∈ {, } and recommendation list for
a job advertisement is defined as
On the test set for candidate ranking, the CRS achieves a
score of 0.82 nDCG@10. While the ideal nDCG score is 1,
( ) = | ( ∈ | = , ( , ) = 1) this performance is still considered decent [
(2) The average of TPRP over all job advertisements is 0.326,
where ( , ) = 1 implies that a candidate sampled and according to Equation. 2, the ideal value of TPRP is 0.
from the candidate set is suitable for job advertisement The top red-words according to the bias score of tokens
. Furthermore, = 0 implies that equal opportu- ( in Algorithm 1) for each job are shown in Table 2 after
nity [
We used Algorithm 1 to create three lists of red words with less than three letters. The majority of words here
with diferent bias thresholds: 0.05, 0.02, and 0.002. It’s are relevant for identifying the job, unlike the terms in the
worth noting that the likelihood of randomly selecting a expert-list examples which are not related to any specific
token from CRS’s input is about 1/512 or 0.002. For com- job but are rather generic. This behavior is expected as
parison, We use the word list by social psychologist [
We observe later that with the increase in the number The common words between the expert-list and redof job advertisements, the curated red-words exhibit less words are few, which aligns with the observation from variance. On average, job advertisements contain about 250-400 other. The number of words given by red-words for words. Table 4 displays the average number of red-words = 0.05 and the expert-list are close, although, the comper job for each profession. With the relaxation of bias mon words between them are very low. We compare threshold value , the number of red-words increases, and also, the number of common words with the expert-list their parts of speech distribution in Figure 2 to better understand their composition. The expert-list does not increases. Here, with the increase in the number of job contain any digits, stopwords, and unknown parts, readvertisements, the red-words for more relaxed values moving these from red-words will make its distribution (a) Senior Software Engineer
(b) Nurse similar to the expert-list for jobs with more than 15 job expert-list and might not appear in a more refined redadvertisements. The prevailing part of speech in both list. But anything conclusive cannot be deduced from the red-words and the expert-list is observed to be the these common words and require further investigation. noun, followed by the verb, adjective, adjective satellite, Sample job advertisements are shown in Figure 3. Here, and adverb. Although the two lists have diferent words red-words are highlighted with red color, and expert-list (common words are few), the parts of speech distribu- words are highlighted with green color. As can be seen, tions are very similar after removing digits, stopwords, the two methods highlight diferent types of words based and unknown parts. The existence of digits, stopwords, on their association with either the job or the candidate. and unknown parts can only be justified due to their rela- In the example of the “Senior Software Engineer” shown tionship with more meaningful words and needs further in Figure 3a, the words in red are mainly related to the job investigation. The small overlap between the expert-list of a software engineer, while the word ”understanding” and red-words observed in the Table 2 can also be seen in from the expert-list is a description of candidate and is not the Table 3. Table 3 presents the most frequent adjective associated with the job of a software engineer. Similarly, red-words for each job title and overall most frequent in the example of “Nurse” (Figure 3b), the red-words expert-list adjectives. Here, the top 2 expert list adjec- “care” and “rehabilitation” aligns with the job of the nurse. tives, “responsible” and “competitive”, also appear in the The expert-list in this example describes the employer red-words of “software engineer”, “dentist”, and “nurse”. and the candidate because the same terms can be used to Some other words common between both lists are “sup- describe both candidate and the employer. Here, the term port”, “commit”, “child”, and “principle”. These words “ambitious” can be used for both the employer (ambitious do not show any specific diference from other words in care house) and the candidate (ambitious nurse). Both examples (see Figure 3) and Table 4 confirm that genderbiased wording of a job advertisement is quite distinct from the words causing bias in CRSs. Hence, to come a bit closer to our goal of a bias-free recruitment process, we have to give attention to both red-words and expert-list.
We present an algorithm to create a list of words from job advertisements, which CRS associate with the gender of candidates. In contrast to a well-established list generated by social psychologists, this list addresses CRSs’ gender bias instead of the perceived gender bias of experts. So, a gender bias-free wording of a job advertisement is diferent for an expert and a CRS leveraging LLMs, and this distinction should be kept in mind while debiasing the candidate selection process. Although expert-list and red-words contain diferent words, their composition is similar in terms of parts of speech distribution.
As for future work, we plan to investigate more thoroughly possible similarities and diferences between the two lists of words. Further, we want to understand how both lists afect the debiasing of the candidate selection process. Also, we plan to improve our algorithm after understanding the reasons for the existence of digits, stopwords, and unknown parts of speech in red-words.
This research is funded by the Austrian Science Fund (FWF): DFH-23 and P33526; and by the State of Upper Austria and the Federal Ministry of Education, Science, and Research, through grants LIT-2020-9-SEE-113 and LIT-2021-YOU-215.