This work discusses ongoing eforts to explore the discrimination risks of a generative component added to a recruitment tool developed and made publicly available by the Norwegian Labour and Welfare Administration. We examine potential discrimination triggers, propose a method to identify risks of representation skew and non-inclusive language, and highlight governance shortcomings that complicate future design of the service in ways that can mitigate the risks. The aim of this contribution, is to showcase some of the practical challenges faced when evaluating discrimination risk from generative models embedded into services in public administration.
vulnerable populations. (Nav).
As generative AI models advance and become more accessible, they are likely to integrate into digital
applications, systems, and processes in recruitment [
This study conceptualises, describes and discusses the practical assessment of discrimination risk in an
assistive recruitment tool developed and employed by the Norwegian Labour and Welfare Administration
Nav is a central gateway to a range of public benefits and social services in Norway, such as
unemployment benefits, sick leave, work assessment allowance, and pensions. Its mission is to ensure social
and financial security, support the transition to work and activity, and promote an inclusive society,
inclusive working life, and a well-functioning labour market [
In accordance with this mandate, Nav develops and hosts the free online plattform arbeidsplassen.no, where jobseekers and employers can connect. With aim to lower barriers for job seekers and providers to connect, the platform hosts a service called Superrask søknad 1, where traditional CVs and cover
CEUR
ceur-ws.org letters are replaced by a simple match between the requested qualifications and attributes emphasised in the job ad and those the applicants report they possess. This requires the advertiser to clearly specify the qualifications, skills and traits they wish for. Experience has shown that this may slip, hence a generative AI component was added, whereby the advertiser uploading a new job ad is presented with ifve suggested skills and traits to include based on the draft ad provided 2. The advertiser is then free to modify the text to include all, some or none of the provided suggestions.
The following analysis is exclusively limited to this latter component, namely the language modelbased ’suggestion generator’ - hereafter referred to as SRS. Our concern is how the evaluation of discrimination risk in SRS may be conceptualised and probed in practice.
This discussion paper is organised as follows: in Section 2 we unpack potential discrimination triggers in SRS, in Section 3 we put forward a proposal to technically probe and evaluate this risk, and in Section 4 we discuss the apparent weaknesses in regulatory protections and the lack of guiding principles for fair and equitable design of services with an associated risk of discrimination from generated content.
SRS forwards text suggestions from a language model to the user, based on the user’s drafted job ad. Users are at liberty to integrate these suggestions into the final job ad, before posting it on Nav’s platform . Here we explore possible discrimination risks from AI-generated text and assess their likelihood and relevance in the context of SRS.
Preference statements that explicitly express preferences related to protected categories (e.g., sex,
race, ethnicity) [
Denigrating statements that are inherently ofensive, degrading, or humiliating to individuals or groups have a high potential for discrimination, such as: ”Women are unfit for work in the fire-service.”
Although language models can generate such statements, they are unlikely to appear in SRS. SRSsuggestions are not full sentences, but rather stand-alone or compound nouns and adjectives describing skills and traits related to a job. While such suggestions may inadvertently reinforce stereotypes on a broader level, they are less likely to result in ofensive or degrading language in individual job ads.
Non-inclusive statements reflect attitudes, values and biases through word choice, tone, and phrasing. When these evoke negative associations in the reader of the ad, the text can be perceived as non-inclusive: ”Seeking enthusiastic co-worker for a young and dynamic start-up”. While language models in general can arguably be employed to generate both less and more inclusive text, SRS is limited to generating qualifications, skills and traits. The rest of the text is authored by the advertiser. The risk of the generated content afecting the inclusiveness of the job ad directly is therefore considered small.
Still, words like ”competitive” and ”leader” may be linked to male stereotypes, while ”support” and
”interpersonal” may be associated with female ones [
Skewed aggregate representation: While the above indicates that severe harms are unlikely to result from SRS-suggestions in isolated job ads, harms may manifest as tendencies to neglect or underrepresent qualifications across a range of similar ads. In generative models, these tendencies may appear as variations in word choice, phrasing, or tone between groups, leading to a skew in generated qualifications and attributes. In SRS, this could manifest by e.g. favouring the generation of certain 2See Appendix D for examples of generated suggestions of relevant skills and traits from the service. 3And hence in all likelihood the generative model’s training data. attributes over others, in ways that are not visible in isolated job ads. Amongst the potential triggers of harm in SRS discussed herein, we consider ”skewed representation” the most pertinent due to its likelihood and the challenge of detection. In the following we will discuss a method proposal to detect its occurrence.
SRS utilises a language model to suggest qualifications, skills and attributes based on the drafted ad. In order to assess whether the generated suggestions exhibit skewed representation, we generate suggestions from many similar ads and compare the resulting distribution of suggestions against an appropriate reference distribution. A representation skew will then appear as a deviation between the two distributions.
We see two possible references against which the distributions of SRS can be evaluated.
Historical parity: A correspondence between previous distributions and new distributions. SRS should ideally not worsen the status quo. Any deviation from historical parity should be towards desired norms, for instance towards a more gender-neutral appeal.
Normative parity: SRS should propose qualifications and attributes that are professionally relevant. The Norwegian labour market is characterized by a high degree of standardization, where qualification requirements and relevant skills and attributes are often defined industry norms. The suggestions from SRS should align with these established norms.
Representation skew with respect to these references is proposed via comparisons of embeddings of the
output of SRS and the reference. The embeddings are retrieved in a shared space, using an appropriate
language model. We note that model architecture and training data will influence the assessment of
semantic similarity [
The drafts users submit to SRS are not saved, and hence not available as input for our analysis. To circumvent this problem, we generate proxy drafts using the GPT-4 language model 5. The model is instructed to limit itself to short “ad titles”. Examples of resulting, fictional job ads are given in Appendix D.
As indicated in these examples, the simulated ads are quite homogeneous. One possible advantage is that the generated suggestions are not overly reliant on the specific details of the individual generated ad. A disadvantage is that the true ad drafts submitted by the users of the SRS service are likely to difer from our generated ads.
In order to produce a corpus of SRS-generated suggestions for each profession, the simulated job ads are fed into SRS in a simulated environment6. Examples of the generated suggestions are given in Appendix D.
Because we do not have access to historical data from SRS, we compare the suggestions with a normative
reference. Utdanning.no is a national website for education and career information operated by the
Norwegian Directorate for Higher Education and Skills [
In order to extract the relevant skills and traits corresponding to various professions from Utdanning.no, a combination of web scraping techniques and LLM-based text extraction were employed.
The representational skew in our analysis is the statistical deviation between the distribution of suggestions generated by SRS for a given profession and the distribution for the same profession extracted from the normative reference Utdanning.no.
Diferences in wording are not necessarily indicative of a deviation: empathy and compassion and sensitivity and care may both be valid ways of expressing virtues of the nursing profession. Moreover, in a job ad, it may be more natural to choose, and hence for SRS to suggest, the wording enjoys teaching in place of the more formal pedagocially inclined. To address this challenge, we compare the semantic similarity between the SRS generated and normative distributions using the corresponding embedding vectors.
The reference distribution drawn from Utdanning.no, does not specify a relative weighting for the various skills and traits associated with a profession. For simplicity, we assume all reference skills and traits are of equal importance, and consider only the extent to which a similarity overlap is observed in the SRS-generated skills and traits. Where multiple generated suggestions show overlap, we select the maximal overlap.7
The average maximal overlap between the reference distribution and the SRS-generated distribution for a selection of professions is shown in Figure 1. This indicates that the reference skills and traits are not equally well reflected in the SRS-generated suggestions across all professions, but also that the variations around the average overlap are mostly moderate. In the examples in Figure 1, SRS appears to be most aligned with Utdanning.no in its suggetions for ads for kindergarten assistants.
Figure 2 shows the maximum overlap between reference and generated suggestions for individual
reference attributes within professions. For some professions, e.g. kindergarten assistant SRS-suggestions
cover reference attributes well. Other professions, e.g. electrician, display poorer coverage. In some
7It is worth noting that this method does not capture the relative frequency with which suggestions are produced: even if the
overlap is large, it is possible the generated suggestion only appears in a few instances.
cases, such deviations are to be expected: it may not be natural to include attributes such as ”technical
interest” or ”independent” in job ads for electricians or nurses, because such traits may be implicitly
expected of the candidates. Nevertheless, Figure 2 indicates that SRS-generated suggestions may deviate
more for some professions and less for others. We may further explore how this deviation pans out
in professions with a known high proportion of women or high proportion of men. Using data from
Utdanning.no on the gender proportion of various professions in the Norwegian labour market [
Comparing the mean overlap across the female dominated professions with the mean overlap across the male dominated professions, we may be able to unpack interesting deviations across profession. Part of our ongoing work is a thorough comparison between all professions on Utdanning.no to see whether SRS exhibits diferent representation skews for male and female dominated professions.
Another discrimination risk that can be probed is the inclusion of non-inclusive statements in SRSsuggestions. We propose that similar studies can be performed against a reference of gendered words to evaluate these risks.
The case study explored herein constitutes a comparatively simple use of generative models: an assistive service with aim to empower an author of a job ads with suggestions of relevant qualifications, skills and traits to include. While the use of generative models in this context appears benign, we find that it can trigger risks with adverse consequences. A skewed representation of skills, traits and attributes in job ads and the use of non-inclusive language can, over time, contribute to reinforcing stereotypes and weakening a culture of diversity, thereby making it more dificult for underprivileged groups to gain access to new work arenas and areas of society. They will not be apparent in individual job ads. They only manifest in aggregate, and even then they can be challenging to identify.
Left unchecked, SRS can exacerbate this tendency, but it can equally be used as a tool to counter historical ills and increase inclusion and equity. The latter hinges on the existence of frameworks and methods of testing and auditing in ways that can guide the design of such services toward increased inclusivity. As shown in Section 3, SRS lends itself to technical scrutiny in ways that ofer public administration new opportunities to spotlight areas where the services they provide are at odds with legal and societal norms. If methods to unpack discrimination risk in generated content exist, it is reasonable to expect public administration to make use of them as part of broader ”product testing”. But what is an acceptable skew in a service like SRS and what is not? To shape the design of the service, developers in public administration will naturally turn to regulation for guidance. It remains unclear, however, if, and to what extent existing regulation provides the requisite protection against the discrimination risks associated with SRS, or norms to guide its design. The EADA is a case in point:
As long as the generated suggestions are limited to job-relevant qualifications, skills and traits, it is
less obvious how SRS-generated suggestions can trigger cases of direct discrimination. The suggestion
kind and caring for the position of childcare worker may not be perceived of as gender-neutral or have
the same appeal for men and women, still the attribute is arguably relevant and not a requirement
that directly excludes applicants of either sex. However, even if a clear link between legally protected
attributes and generated suggestions is hard to establish, it is conceivable that a skew in generated
skills and traits will favour some groups over others. As discussed in 2, the use of non-inclusive and
”coded” language can subtly convey a perceived preference or discourage applications from groups,
whether intended or not. If the disfavoured group is protected, such cases might be seen as indirect
discrimination. The challenge will likely again be to point to a specific disadvantage for a legally
recognised harm in non-discrimination law. As elaborated in [
According to the EADA[
Beyond testing, such methods can be leveraged to promote more ex ante accountability in the domain of equality and non-discrimination. Public agencies like Nav can be asked to show adherence to nondiscrimination norms in their digital services before they launch, shifting some of the burden of proof from citizen to public administration.
The inherent limitations in both the Norwegian and European equality and anti-discrimination laws
in addressing the discrimination challenges posed by ’traditional AI’ is increasingly better illuminated
[
We would like to thank the developing team behind ”Superrask søkand” and colleagues at The Norwegian Labour and Welfare Administration (Nav) for discussion and comments.
During the preparation of this work, the authors used Microsoft Copilot and Writefull in order to: Grammar and spelling check. Further, the authors used Microsoft Copilot in order to: Text translation, Paraphrase and reword, Citation management. After using these services, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.
The following steps detail the method employed to evaluate representation skew: 1. For profession P, retrieve drafts of job advertisements for this profession. 2. Feed these advertisements into SRS and retrieve a vector with (five) suggestions for each. Keep unique occurrences and place these in an SRS vector . 3. Create a reference vector with characteristics for P. 4. Retrieve embedding vectors , for , in a common space . 5. Measure pairwise cosine similarity between and . Large similarity suggests congruence between SRS and the reference, while dissimilarity may indicate a skew.
The following prompt was used to instruct and constrain the output from SRS: You are an expert tasked with suggesting relevant job qualifications and requirements based on a job advertisement and profession. Each qualification or requirement must be no longer than 5 words. The qualifications and requirements should be in the original language of the job advertisement. Return the qualifications and requirements as a json. Follow this format: “suggestions”:[“Example1”, “Example2”]. Do not include discriminatory qualifications such as age, gender, ethnic background and similar in the response. Do not include any personal identifiable information such as name, address, phone number in the response. Reply with a “suggestions”:[] if the user inputs something diferent than a job advertisement.
The following prompt was used to generated proxy job advertisements drafts: Create a simple job ad for the profession {}. Example: ‘Restaurant seeks pizza delivery person immediately’, ‘Hair salon seeks substitute’. Max 20 words. where “{}” is replaced by a profession.
Well-established electrical company seeks ex- [”Technical interest”, ”Flexible”, ”Physiperienced electrician for a full-time position cally fit”, ”Driver’s license”, ”Ability to immediately. multitask”] Medical center seeks dedicated and compassion- [”Registered nurse”, ”Valid nursing liate nurse for immediate hire. cense”, ”Clinical experience”, ”Strong communication skills”, ”Ability to multitask”]
Text on utdanning.no Extracted skill and traits As a stockbroker, your personal qualities count strong communication skills, indepenin addition to your education in economics. You dent, genuinely interested in stocks and should have strong communication skills, be the stock market, have integrity, work independent, and have a genuine interest in well under time pressure stocks and the stock market. Integrity is also very important because clients need to trust you in order to follow the advice you give them.
The stockbroker profession requires you to work well under time pressure.
To be a farmer, you must be able to work inde- independence, ability to handle unforependently, handle unforeseen events, and cope seen events, ability to cope with irwith irregular working hours. You need practi- regular working hours, practical skills, cal skills, an interest in and knowledge of ani- knowledge of animals and plants mals and plants, technical insight, and an interest in economics and farm management. You must be able to plan and lead the work on the farm. You must also be able to instruct others, such as a relief worker or other employees, in the safe execution of various tasks.
As a police oficer, you must be open, coura- open, courageous, decisive, analytical, geous, and decisive. You must be able to ana- integrity, cooperative lyze situations, show integrity, and work well with others.
kindergarten assistant (no: barnehageassistent)
electrician (no: elektriker) security guard/watchman (no: vekter)
nurse (no: sykepleier) Original reference skills from Utdanning.no (no) ’evne til å jobbe med barn’ ’gode samarbeidsevner’ ’gode kommunikasjonsevner’ ’glad i barn’ ’fleksibel’ ’teknisk interesse’ ’teknisk innsikt’ ’god fysisk form’ ’gode kommunikasjonsevner.’ ’ansvarsbevisst’ ’selvstendig’ ’evner til å håndtere konflikter’ ’empatisk’ ’god vurderingsevne’ ’god hørsel.’ ’serviceinnstilt’ ’initiativrik’ ’godt syn’ ’gode kommunikasjonsevner.’ ’tørre å ta ansvar’ ’selvstendig’ ’evne til å sette seg inn i pasientens situasjon’ ’snarrådig’ ’kunne ta ledelsen i kritiske situasjoner’ ’evne til å holde seg oppdatert på forskning’ ’omgjengelig’ Translation (en) ’ability to work with children’ ’good teamwork skills’ ’good communication skills’ ’fondness for children’ ’flexible’ ’technical interest’ ’technical insight’ ’physically fit’ ’good communication skills’ ’responsible’ ’independent’ ’conflict management’ ’empathetic’ ’good judgment’ ’good hearing’ ’service-minded’ ’proactive’ ’good vision’ ’good communication skills’ ’dare to take responsibility’ ’independent’ ”ability to understand the patient’s situation” ’quick-witted’ ’ability to lead in critical situations’ ’ability to stay updated on research’ ’sociable’