An Exploration of the Information Seeking Behavior of Recruiters

Abstract

We explore the information seeking behaviour of the recruiters while they search to find candidate job seekers that match the open positions posted by the organizations. We perform a set of contextual inquiries with recruiters at one of Scandinavia's largest job portals and recruitment agencies by using its job database system. We aim to better understand short-term (matching) and long-term (recruitment) information seeking behaviour of the recruiters and their interaction with the search engine based on Solr. Based on the conducted contextual inquiries, we list a set of design implications to be used for better matching systems that can assist the recruiters to find more relevant candidates.

Keywords

• Job recommendation
• recruitment
• HR
• professional search
• information seeking

CCS CONCEPTS

• Information systems → Recommender systems; Personalization.

INTRODUCTION

Being able to draw on the relevant knowledge, skills, and abilities of their employees is essential for any organization to thrive. Recruiting the right employees is the first step in this process and a key to success. Recruitment consists of identifying relevant candidates for an open position and shortlisting them for the position by assessing their education level, their knowledge, skills, abilities, their work experience and their interests [ 3 ]. Historically, the recruitment process has taken a long time to complete with a large amount of paperwork involved, but this has been changing in recent years thanks to online recruitment gaining in popularity [ 24 ]. Job portals and corporate websites allow for easier collection of candidate CVs in electronic form [ 21 ] and providing recruiters access to them through search engines can help streamline the diferent stages of the recruitment process, illustrating the potential of information technology to support human resource management [ 2, 9 ].

However, despite these technological advances, finding relevant candidates for a position remains a manual process with a heavy post-processing burden on recruiters in terms of assessing their qualifications [ 21 ]. Both domain and search experience can have a big influence on the quality of the search results returned [ 35 ] and thereby on the duration and efectiveness of the recruitment process. Furthermore, a problem with most current approaches is that their designs are not based on solid user research of the professional practices of recruitment professionals. Russell-Rose and Chamberlain [ 29 ] argues that this has resulted in recruiters’ needs being poorly supported by developers of recruitment systems: “The problem does not lie with the algorithms; it lies with the assumptions made by developers that do not understand how head-hunters think.” [29, p. 46]. A better understanding of the information seeking process of recruiters using online tools such as job databases and search engines could aid in the design of more efective and eficient systems that use AI techniques to augment the work of human recruiters [ 2, 9 ]. This would enable an organization to not only streamline recruitment, but also identify and alleviate biases, preconceptions and time restraints in their recruitment process. A logical next step could then be to using AI techniques to support recruiters by automating part of the process of identifying and matching relevant candidates with an open position.

In this paper, we present the results of a set of contextual inquiries conducted with recruiters at one of Scandinavia’s largest job portals and recruitment agencies (henceforth referred to as Jobindex). These inquiries were aimed at better understanding their information seeking process with the end goal of designing better matching systems to help them identify a larger number of more relevant candidates. The main contributions of this paper are:

We perform a set of contextual inquiries conducted with recruiters at Jobindex. These inquiries focuses on three main tasks of the recruiters: (i) searching & filtering candidates, (ii) shortlisting candidates, (iii) contacting candidates. We use the results of the contextual inquiries to better understand recruiters’ short-term (matching) and long-term (recruitment) information seeking process. Then, based on the analysis of the results, we list a set of design implications that can be used for better matching systems that can help recruiters through searching and filtering, shortlisting and contacting relevant candidates.

In Section 2, we review relevant research on information seeking behaviour of recruiters. Section 3 presents followed methodology to conduct the contextual inquiries with recruiters. Then, Section 4 describes the results and analysis of the contextual inquiries. Finally, Section 5 concludes the paper with a discussion. Recruitment is the process of identifying relevant candidates for an open position and shortlisting them for the position by assessing their qualifications. According to the model of the recruitment process by Breaugh [ 3 ], these qualifications include (1) their education level, (2) their knowledge, skills, abilities, (3) their work experience, and (4) their interests. In addition, diversity considerations and cultural fit can play a role in the recruitment process [ 3 ]. Depending on the intended scope of the author(s), the recruitment process is modeled as a sequence of stages with diferent models having been proposed over the years [ 3, 22, 33 ]. One of the more straightforward models is the one proposed by Nikolaou [ 23 ], who identifies four main stages of the recruitment and selection process: (1) attraction of candidates (e.g., through job portals), (2) screening, (3) selection, and (4) on-boarding. Our focus in this study is on the first three stages from the perspective of the recruiter, which correspond to our focus on the identifying and shortlisting relevant candidates and contacting them about the job posting. 2.2

AI in recruitment

The popularity of online job portals and corporate websites have allowed for the easier collection of increasing numbers of candidate CVs in electronic form [ 2, 15, 21 ], yet the assessment of these candidates still places a heavy post-processing burden on recruiters [ 21 ]. Many consider AI technology to have the potential to reduce this manual burden considerably at diferent stages in the recruitment process, thereby saving recruiters time and reducing costs [ 15, 21 ].

One such application of AI is the automatic extraction of relevant information from CVs, such as contact information, previous education and job experience, as well as the candidates’ knowledge, skills and abilities [ 32 ]. Skill extraction in particular has received a great deal of attention in recent years [ 1, 10, 14 ], as skills are seen as an essential part of candidate assessment. In addition, having a better understanding of strengths and weaknesses in the skillset of newly-hired candidates could enable better career management.

Another application of AI technology is the automatic prediction of personality traits based on the text supplied by the candidates in the form of their application letters and CVs [ 6, 15, 20 ]— assessing the personality of potential candidates is a common yet time-consuming task in the recruitment process [ 3 ].

One of the areas where AI technology can be argued to have the greatest potential [ 15 ] is automating the process of matching jobs to job seekers. From the perspective of the latter, this is the task of recommending relevant jobs to a job seeker. The majority of the related work has focused on this scenario and much of the work has gone into using semantic networks or embeddings to alleviate the vocabulary problem between how HR departments and job seekers describe what they are looking for. Typically, a combination of techniques from information retrieval and recommender systems are used that are trained on training data supplied by human recruiters, which represent their past actions and assessments. The goal of the AI algorithm is then to learning a proper scoring function which represents the preferences of the individual job seeker [ 6, 7, 11, 12, 21 ].

From the perspective of recruiters, this is the task of candidate ranking: given an open position, rank the available candidates by how relevant they are for the job, thereby allowing recruiters to spend more time on attracting, screening and selection the most promising candidates. Providing this candidate ranking often falls to the search engines implemented in the job portal or job database that contains all the candidates’ CVs. Section 2.3 goes into more detail about how recruiters use search engines to identify relevant candidates.

Finally, another oft-mentioned advantage of applying AI technology to the recruitment process is the potential for eliminating explicit and implicit biases and beliefs held by human recruiters. Karaboga and Vardarlier [ 15 ], for instance, argue that AI-augmented systems could be programmed to avoid biases and thereby enable more inclusive hiring practices. However, most AI technology applied to matchmaking between jobs and candidates relies on large amounts of training data supplied by human recruiters, which represent their past actions and assessments. Without mitigating actions to remove the human biases present in this data, any algorithms would typically learn these biases along with the rest of the data [ 25, 26, 31 ].

Overall, the current consensus on AI in recruitment appears to be that while it has great potential to speed up the process, there remains an essential human component to recruitment [ 15, 34 ]. 2.3

Information Seeking behavior of Recruiters

Despite an increase in AI-based approaches to skill extraction, candidate ranking and job recommendation, most current approaches are not based on solid user research of the professional practices of recruitment professionals [ 29 ]. To the best of our knowledge, only one study has examined the information seeking behavior of recruitment professionals.

Russell-Rose and Chamberlain [ 29 ] presented the results of a survey of 64 recruitment professionals regarding their professional information seeking behavior and their needs regarding tools to support their recruitment practices. They found that recruitment professionals use some of the most complex queries of any professional community with a wide range of search operators [ 28, 29 ]. Around 49% always or often re-used their previous queries. RussellRose and Chamberlain [ 29 ] found that recruiters’ search behaviour is characterized by “satisficing strategies”, where the goal is to identify a suficient number of qualified candidates in the shortest possible time. Recruiters expressed their ideal number of results to be around a median of 33 candidates, yet reported inspecting fewer results ( = 30), suggesting that recall is less important than producing a candidate shortlist of predefined length. Their search process was strongly interactive with multiple iterations of query formulation followed by candidate selection and evaluation. On average, it took the recruiters around three hours to complete a search task with a median number of 5 queries. The median time to assess a single candidate was 5 minutes.

In terms of data sources used, recruiters used many diferent sources, including job boards, social networks, commercial and proprietary internal database as well as the open Web. In the work presented in this paper, our recruiters have internal access to Jobindex’s CV database, which is the largest in its home country, so this variety is unlikely to be replicated in our study.

Domain knowledge—such as in-depth knowledge about the industry sector and expected renumeration for diferent positions— plays an important role when assessing the relevance of candidates [ 29 ]. When evaluating search results and deciding which candidates to add to their shortlist, the recruiters surveyed by Russell-Rose and Chamberlain [ 29 ] were most likely to use location and prior job experience to make this decision. In addition, the industry sector, current career level as well as availability and desired salary were all used to decide which candidates to add to their shortlists. These choices mirror many of the factors previously found to influence expert selection as reported by Woudstra and van den Hoof [ 37 ]. While Russell-Rose and Chamberlain [ 29 ] do not mention the typical size of a shortlist, for 88% of recruiters the shortlisting process was concluded when they found a specific result or when they could not find any new relevant results anymore.

To the best of our knowledge, there is no related work on how recruiters contact their shortlisted candidates and to what degree they personalize their messages to these candidates. 3

METHODOLOGY

The goal of the research project underlying the work presented in this paper is to use AI technology to augment the core activities of Jobindex, one of Scandinavia’s largest job portals. Job seekers can post their CV on Jobindex’s job portal for free and, for a small fee, organizations with open positions can post their job openings to the job portal as well as get access to Jobindex’s CV search engine.

Organizations can also, for a higher fee, enlist the help of Jobindex’s professional recruiting department to actively recruit relevant candidates for their open position. Jobindex ofers diferent variants of this service: a light matching service where recruiters spend at most 60 minutes to find relevant candidates, and a regular recruiting service, where recruiters spend around 3-4 hours on candidate search. The latter estimate for recruitment tasks is in line with the ifndings by Russell-Rose and Chamberlain [ 29 ]. Jobindex also ofers complete recruitment trajectories including candidate interviews, but these stages of the recruitment process were not a part of this study.

In both the matching and recruitment scenarios, Jobindex’s recruiters analyze each new job posting and use the internal CV search engine to identify relevant candidates, similar to the recruiters surveyed by Russell-Rose and Chamberlain [ 29 ]. They then shortlist these candidates and contact them using a semi-personalized message. One of the core goals of our underlying research project is to support recruiters in these parts of their job by developing better matching algorithms that model the recruiters’ internalized relevance criteria and algorithms for the generation of personalized messages.

To avoid developing algorithms that do not adequately support the recruiters in their daily activities—a problem highlighted by Russell-Rose and Chamberlain [ 29 ]—we studied the information seeking behavior and relevance criteria of five diferent recruiters employed at Jobindex. We used contextual inquiry (CI), a qualitative method for understanding and gathering information about how people perform certain tasks in context [ 27 ]. It allows for the observation of the participants in their own environment or context while performing their tasks while, as researchers can learn from them by asking for explanation and clarification. CI is based on three principles [ 27 ]: (1) data must be gathered in the participant’s own context or environment; (2) the inquiry is a partnership between the participant and the researcher to explore issues and behavior together, as opposed to a traditional interview; and (3) data collection is based on an exploratory approach with a pre-defined focus instead of a pre-determined agenda or set of questions. We chose CI over more traditional methods such as surveys, interviews or observations, because it is better at uncovering tacit knowledge according to Holtzblatt and Jones [ 13 ] as surveys and interviews can sufer from recall bias. By asking participants to demonstrate how they perform their daily tasks without directing them using specific questions, we are more likely to uncover their natural behavior and the relevance criteria they have internalized over the years. CI is also better able to adequately capture the context in which a participant performs their tasks [ 4, 36 ].

Raven and Flanders [ 27 ] distinguish between three diferent types of contextual inquires: (1) work-based interviews, where the inquiry takes place during the activity; (2) post-observation inquiries, where the observation is recorded and participant is interviewed afterwards; and (3) artefact walk-through, where artefacts produced by the work activities are the topic of inquiry in case they take place sporadically or over a period of time. We performed workbased inquiries, where the recruiter was studied while performing their day-to-day recruitment and matching activities. Section 3.3 contains more detail about our inquiry design and procedure. 3.1

Participants

We recruited five participants for our contextual inquiries, all employed at Jobindex. To ensure that we had a sample that was as representative of the recruiters at Jobindex as possible, we aimed for variety in terms of recruitment experience, whether their primary focus was on matching or recruiting, and the industry sector(s) they specialized in. Table 1 presents an overview of the five recruiters that participated in our contextual inquiries. It shows their work experience and preferred industry sector as well as their ‘primary task’, which shows whether the recruiter’s primary task is recruitment or matching. In practice, however, all recruiters spend a small share of their time of short-term matching activities too. Our contextual inquiries were conducted and recorded using Zoom. Our participants used Jobindex’s CV search engine, which contained around 138,000 CVs when we conducted our inquiries. Figure 1 shows the English-language interface of the CV search engine used by Jobindex’s recruiters.

The search engine is powered by Solr and allows for search using four diferent search bars (shown in area 1 in Figure 1). In addition to keyword search and location-based search, it also support searching through two diferent types of job title fields: one containing all the desired jobs that job seekers include in their CV and one that contains automatically assigned job categories from a controlled vocabulary of occupations. Search terms are automatically highlighted in the search results (shown in area 4) to aid the recruiter in evaluating the information more quickly. In addition to the search bars, recruiters can also narrow their search results by applying diferent filters in the filter panes in areas 2 and 3. Filters are equivalent to what is known is faceted search in the IR literature [ 30 ]. Available filters include education level, work and management experience, salary expectations, language skills, and type of contract (e.g., full-time, part-time, student job).

Recruiters can click on the title of each CV in the result list to inspect that candidate’s CV in more detail. Promising candidates can then be shortlisted by check-mark selection. After a shortlist of suficient size and quality has been collected, the recruiter can then write a custom message for the selected candidates. This message is sent as an email from the Jobindex’s job portal to the candidates, who then have the option of either contacting the recruiters directly or responding to their inquiry through the job portal. 3.3

Design and Procedure 3.3.1 Introduction. After obtaining informed consent from each recruiter for participating in the study and allowing us to record the session with them, we first asked them to introduce themselves, including their recruiting experience, preferred industry sector and whether they primarily spent their time on matching or recruitment—Table 1 contains the answers to these questions.

Next, we asked our participants to give us a live demonstration where they show us how they use the CV search engine to identify, short-list and contact relevant candidates. Four participants demonstrated their process for two diferent job ads; the fifth participant went through a single example. We encouraged participants to think aloud during the process and where relevant we asked follow-up questions to understand how and why they perform their recruitment and/or matching tasks. The next three subsections provide an overview of what our foci were in the contextual inquiry. 3.3.2 Searching & filtering candidates. The first stage of the recruitment process focused on searching for relevant candidates and using the filtering functionality to improve the quality of the search results. Participants first analyzed the job posting in question for relevant information and recruitment criteria. When selecting a new job posting to process, the CV search engine uses the location1 and the automatically assigned job category to generate an initial results list. Participants can inspect this list and change the content in the diferent search bars and their process for this was one of our foci, as well as the use of the diferent filter panes. Here, we were particularly interested in how they decide which search terms and filters to use, how long this search process took, how may query reformulations they went through, how their inspection of the search results changed their perception of their search terms and search criteria, and whether this was diferent for short-term matching vs. long-term recruiting. 3.3.3 Shortlisting candidates. During the search process, participants would start adding relevant candidates to their shortlist using the check marks next to each CV result. Here, we were interested in what information about jobs and CVs was used by the participants to determine whether a candidate was relevant and, again, whether and how this difered between matching and recruitment. 3.3.4 Contacting candidates. After creating a shortlist of relevant candidates, participants would review this list one more time, after which they start formulating a custom message to send out to these candidates. The CV search engine contains a number of personal 1The location is taken from the original job posting and if the company provides an address, this is converted into geographic coordinates. Otherwise, a larger area such as a zip code or a municipality is used. A radius around this location is then used to match potential candidates. and company-wide templates for constructing such messages. Here, our interest was in whether and to what degree these messages are personalized before they are sent out, which aspects of the job or CVs are highlighted in these messages, and whether participants use personal templates or company-wide templates. 3.4

Analysis

All five recordings were assigned one of the two authors as the main responsible for paraphrasing the content of the recordings. Each clean-copy transcript was organized into the four sections listed in Section 3.3 and further content analysis was performed to identify the use of relevant features and assessment criteria to aid in the development of better matching algorithms. After the initial transcription, the other author checked the transcripts to add any missing details. 4

RESULTS & ANALYSIS

In this section, we present the results of our contextual inquiries with the five Jobindex’ recruiters, from the initial search and filtering stage to shortlisting and contacting relevant candidates. In the rest of this section we will refer to our participation using their IDs, i.e., P1 to P5. 4.1

Searching & Filtering Candidates 4.1.1 Initial analysis of the job posting. Every participant started the recruitment process by analyzing the job posting in question, starting with the description of the organization if they are unfamiliar with it. All participants then go through the job posting to identify the job title as well as the knowledge, skills and abilities the ideal candidate is required to have. Participants P2 and P3 use their recruiting experience to further distinguish between essential and useful requirements. Examples include identifying the most relevant software experience for a position (P2, P3, P5) or prioritizing the diference between experience with accounts payable and accounts receivable for a financial accounting position (P3). In general, domain knowledge was seen as very valuable at all stages, from search to shortlisting, similar to the findings of Russell-Rose and Chamberlain [ 29 ]. 4.1.2 Searching. As mentioned in Section 3.3, for each new job posting, the CV search engine automatically takes the location and the automatically assigned job category to generate an initial list of matching CVs. All participants inspect the top of this initial list to get a feel for the type and the number of candidates that the search engine returns. Several recruiters expressed a desire for shortlisting at least 20 relevant candidates, given the historic response rates of around 10% to the custom messages. This means that much of the search behavior is influenced by how many initial search results are returned. Depending on the position, between 50-150 search results were seen as ideal to be able to reach the desired number of 20 relevant candidates—considerably lower than the 33 reported by Russell-Rose and Chamberlain [ 29 ]. Too few results and the participants exhibited divergent search behavior, e.g., by removing pre-filled locations or job categories, or using wildcard operators to increase recall. Too many results and the participants adjusted their queries to converge on a smaller set of search results by adding more specific keywords or job titles. The duration of the overall search and filtering process is therefore dependent on the number of search results returned and how efective their convergent or divergent search tactics are. We asked all participants about how many query formulations they typically go through, but most participants were not able to provide us with a reliable estimate, except for P3. His matches typically consist of four query reformulations and his recruitment jobs around 15. This would be consistent with the ifndings of Russell-Rose and Chamberlain [ 29 ]. An analysis of the search logs of the CV search engine would be able to shed more light on this question in the future. In general, apart from the duration and number of queries, there were very few diferences in the search behavior for matching vs. recruiting.

Location. The auto-generated location in the location search bar is usually left in place by our participants and rarely altered. Experience has shown that candidates are unlikely to respond if they are contacted about a position that is outside of their preferred location. Jobindex’s recruiters are also encouraged to leave it in place, as it is more comprehensive than most manual location searches tend to be.

Keyword search. The most commonly used search bar is the keyword search bar. All participants used the essential knowledge, skills and abilities identified by the participants during the analysis of the job posting as a source of search terms. Participants P2 and P4 also use the search results themselves as a source of inspiration for efective search terms as a way of addressing the vocabulary problem [ 8 ]. The terms that job seekers use to describe themselves and their knowledge, skills and abilities does not always match the way an organization described the requirements for a job, resulting in fewer matches. This suggests that better semantic matching between job postings and CVs using, for instance, embeddings could alleviate some of these problems.

In terms of search tactics and operators, participants tend to go from basic ‘quick-and-dirty’ search to using more advanced search operators. For instance, P3 stated that he typically starts by using the most unique or discriminatory skill-related terms in an initial ‘quick-and-dirty’ search to gauge the dificulty of this particular recruitment assignment. Depending on the size of the results set, he would then engage in convergent or divergent behavior. Similar to the findings of Russell-Rose and Chamberlain [ 29 ], all our participants also used a wide range of advanced search operators, such as Boolean operators, grouping search terms using parentheses, phrase search, flexible matching using the wildcard operator, and addition/subtraction operators to force the inclusion/exclusion of certain terms, like software skills (P2, P3, P5). P1 stated the she believed that possessing good search skills was more important for recruitment trainees than having in-depth knowledge of the industry sector, as job postings often contain a lot of useful hints about relevant requirements.

Job titles. Searching for job titles was done by some but not all of the participants. P3 would sometimes search only using job titles instead of search terms, because of the auto-completion functionality ofered in the job title search bar. This would inspire him to add job titles he would not have thought of on his own, something echoed by P5. They both agreed that job titles can be as important as matching on knowledge, skills and abilities. Both P3 and P4 cautioned that matches based on job titles should be taken with a grain of salt, since the job titles extracted from the CVs are both historical as well as aspirational—which job titles the candidates would like to have in the future. This means that matching candidates may not always possess the required qualifications.

Job categories. Job categories were not as popular with the participants as the other search bars. Due to their automatic assignment, some participants were mistrustful of the quality of the categories and would therefore remove them. Categories were typically one of the first things to be removed to increase the size of the result set by P3, P4 and P5. In some cases, these recruiters would add categories to reduce the size of the result set. 4.1.3 Filtering. In addition to the four search bars, recruiters can converge or diverge on a desired result set by using the diferent iflters in the filter panes. Available filters include education level, work and management experience, salary expectations, language skills, and others. Filters were commonly used by all participants to narrow down the result sets for their searches. We found no meaningful distinctions in filter use between matching and recruitment tasks.

The salary filter was seen as one of the most useful filters by some (P2, P3 and P4), but not all of the participants. While some recruiters only set a maximum salary level, others insisted on setting both minimum and maximum salary level expectations to uncover the relevant candidates. For part-time positions, salary expectations are harder to interpret so participants usually did not use this filter for those types of jobs. Setting the right salary levels was a matter of experience, according to our participants, although Jobindex has extensive statistics on salary levels for diferent positions and industry sectors. In case a participant was unaware of the expected salary for a job position, they would contact the organization itself. However, this was only done for recruitment cases; for the shorterterm matching cases, this was usually estimated based on the job posting text.

Participant P5, who did not use the salary filter often, instead use management experience—and to a lesser extent work experience— to reach the same group of people as a salary filter could reach. However, the other participants will typically only use the management experience filter if it is explicitly mentioned in the job posting, similar to work experience (P3). In addition to filtering out people without enough work or management experience, P3 occasionally also uses the experience filters to remove people that are overqualified for a position by filtering out all CVs for more than three years of experience (unless the job ad in question is for a more senior role).

Our participants also shared two interesting observations. P3, whose expertise is in the accounting industry, had observed that accountants tend to be more loyal to their employer than in other industry sectors. This means that the required level of work or managerial experience can vary also by industry sector, something an algorithm for automatic matching should take into account. Another, more critical observation by P4 is that unless explicitly required by an organization, experience is dificult to evaluate through the use of the filter—without inspecting a CV in detail and check a candidate’s previous employment history, it is dificult to know whether all seven years of experience in position or industry sector constitute relevant experience for the organization in question.

Another filter that is commonly used is the company filter. After adding their CV to Jobindex’s CV database, job seekers are asked to select organizations that they admire or would be interested in working for. By selecting the company filter, only CVs whose creator is ‘following’ the organization in question will be returned. Several recruiters (P2, P5) noted that they thought it was better to start searching for candidates among the followers of a company, as they are more likely to apply for the position. However, in case the results set is too small, this filter is one of the first ones to be removed.

As for the other available filters, such as language or education level, they are only used occasionally when the specific job posting calls for it (P4). The filters on employment type and notice period were mostly restricted to matching or recruiting for jobs for jobs that are not full-time positions, such as part-time positions, freelancing or student jobs (P4). 4.2

Shortlisting candidates

As mentioned before, most recruiters aim for a shortlist of around 20 relevant candidates before they move on to the contacting phase. The most important relevance criteria when assessing the relevance of a candidate for the position are whether they match the required knowledge, skills and abilities stated in the job posting. Search term highlighting—shown in yellow in Figure 1—make it more eficient for the participants to determine whether and where the skills they added as search terms occur in the search results (P4).

Past work experience was also important in relation to the required knowledge, skills and abilities: participants would contrast the most recent positions a candidate has held to the required skills to better be able to determine whether the candidate really possesses those skills. Again, this is in line with the findings by Breaugh [ 3 ] and Russell-Rose and Chamberlain [ 29 ]. P3 stated that he would usually consider the 1-3 most recent jobs or at most the last five years of work experience to prevent recommending candidates whose skills had atrophied too much. P3 added that he would examine past work experience in more detail for recruitment cases than for matching.

Another important relevance criterion is location as it has a strong influence on whether candidates will be interested in applying for a job. Here, participants typically relied on the autogenerated location in the location search bar. This is similar to its importance in the study by Russell-Rose and Chamberlain [ 29 ].

When asked about the importance of education level as a relevance criterion, the majority of participants did not find it particularly important, because relevant knowledge, skills, and abilities can also be acquired on the job. Only if a required education level was explicitly mentioned in the job posting would they include it as part of their assessment.

Finally, many participants indicated that in their assessment they would predominantly check the structured Jobindex version of the CVs, which is due to job seekers being asked to enter the information in their CV into diferent text fields. When necessary, participants would check unstructured personal CVs that are included as PDFs, although this was not common due to this taking more time to assess. For instance, P5 stated that she will only examined unstructured attached CVs if it proves dificult to reach a shortlist of 20 candidates. 4.3

Contacting candidates

After shortlisting a suficient number of candidates, recruiters move on to the last phase of matching and recruitment where they write a message to each candidate persuading them to apply for the position. These messages are sent out as emails by Jobindex’s job portal and typically include a line about the job they are being recommended and why, which recruiter is doing the recommending and how they can get in touch with both the recruiter and the organization. Each message always includes an embedded description of the job posting at the end of the email. Finally, messages also include a link which candidates can use to provide easy feedback on the recommendation.

In principle, each candidate on the shortlist can be contacted separately with a uniquely personalized message. However, to save time all of the participants indicated that they use templates to help formulate the messages. Over the years from participants (such as P2) have formulated their own templates for diferent occasions, while others use the default templates available to all Jobindex recruiters. While these templates can and are typically stored in the system, making it easy to select them, participant P2 stored his own template in a Word document.

Nevertheless, all participants believe that proper personalization of these messages—where each candidate gets a uniquely worded message highlighting what would make them such a good candidate for the position based on their own skills and past work experience—could have a positive efect on the response rate. The reason for not personalizing messages and sending the same version to all shortlisted candidates is lack of time, which suggests that the automatic generation of personalized, persuasive messages could be valuable feature to add to Jobindex’s systems. At the moment, one of the participants opined that the increased response rate was not worth the extra efort of personalizing each message. Personalization was seen as most valuable for junior recruiters and matchers according to senior participant P2, due to their lack of experience with crafting persuasive messages for a variety of positions.

When asked about which elements of the message are personalized for the short list, the participants difered in their approach. Senior recruiter P1 barely used the job posting text at all to personalize the messages whereas others, depending on the position, took the time to explain why that position would be relevant for the candidates. Often, such explanations will include relevant work experience that matches the new position, while others, depending on the position, may stress the responsibility or the the salary level that comes with the new position (P3, P5). Some participants, such as P3, vary the writing style and length of their messages depending on the socio-economic status—as an example he contrast CFO positions versus carpenters, where the candidates for the CFO position are more likely to be willing to read more information about the job than carpenters would be for an entry-level carpentry job.

Several participants stressed that it was important to make a message feel personal, even if it is not. Messages that are perceived to be auto-generated by a bot were seen as problematic for response rates and much less persuasive (P2, P3). Another important persuasive element to include is to avoid patronizing candidates and instead ofer tips and suggestions for seeking the position in question.

All participants indicated they were willing try to try out a system that could help generate persuasive, personalized messages, either by generating them automatically or by suggesting the most relevant elements to include for the recruiter, such as relevant skills or work experience. The most important condition for this was shared by all recruiters: that such a system plays a supporting role and that they remain in control of the final decisions, a sentiment echoed by Karaboga and Vardarlier [ 15 ] and Tomassen [ 34 ]. 5

DISCUSSION & CONCLUSIONS

In this paper, we have presented the results of a contextual inquiry study with recruitment professionals to better understand how they perform their matching and recruitment duties, how they interact with the diferent features of Jobindex’s CV search engine, and what their professional information seeking behavior looks like. Our aim with this study is to better understand their professional information needs and how we can design better systems that supports them in matching and recommending relevant candidates for open job positions.

We find that the search and filtering process in which recruiters use the CV search engine to identify relevant candidates is a complex and interactive process. Drawing on their domain knowledge, recruiters identify the most relevant knowledge, skills, abilities required in the job posting and match these to the skills, past work experience and job titles of the thousands of candidates in the CV index. Factors like salary, experience, and location play an important role in the relevance assessment of candidates. Many of these ifndings align with the earlier work by Russell-Rose and Chamberlain [ 29 ], who conducted a survey of the information seeking practices of recruitment professionals. When contacting the shortlisted candidates, recruiters have to balance a trade-of between cost and efectiveness with regard to the degree of personalization. Personalized messages are more likely to get a positive response, but due to the ‘satisficing’ nature of online recruitment, there is not enough time to customize beyond the shortlist itself. Nevertheless, provided they remain in control, recruiters are generally positive about the possibilities of personalization support.

Our work has several design implications for an improved version of the current recruitment workflow and infrastructure. With regard to the search for relevant candidates, the vocabulary problem originally identified by Furnas et al . [ 8 ] also plagues recruiters as job seekers will use diferent terms to describe the same concepts as organizations do in their job ads. Semantic matching technology, such as the use of word or document embeddings could be a fruitful way of addressing this problem [ 16, 17 ]. Given their importance in the relevance assessment process, properly representing and embedding knowledge, skills and abilities [ 1, 10, 14 ] as well as job titles [ 5 ] and location [ 18, 19 ] would seem to be a priority for any state-of-the-art job matching system. The role of domain knowledge was evident in the use of filters by the recruiters: for diferent industry sectors and positions, they were able to leverage that knowledge and experience to select the appropriate salary, location, work experience and management experience filters. Leveraging Jobindex’s sizable historic data about past job ads, CVs and recruitment searches, it should be possible to identify such values automatically for diferent contexts, thereby improving the quality of the results list and saving recruiters time.

Finally, our contextual inquiry study has also revealed several opportunities for supporting recruiters in contacting shortlisted candidates by identifying the most important overlapping elements between a job posting and a shortlisted CV and using them to personalize the message templates already in use at Jobindex. In this personalization process, and in fact throughout all stages of the recruitment and matching process, the intention is not to replace the recruiters, but to assist them intelligently. In other words, they will always have the final say by being in the loop.

ACKNOWLEDGEMENTS

This research was supported by the Innovation Fund Denmark, grant no. 0175-000005B. We thank Qiuchi Li for his contributions during the contextual inquiries. We also thank the Jobindex recruiters who participated in our contextual inquiries.

References

[1] Mathieu Bastian , Matthew Hayes, William Vaughan , Sam Shah, Peter Skomoroch , Hyungjin Kim, Sal Uryasev, and Christopher Lloyd . 2014 . Linkedin Skills: Largescale Topic Extraction and Inference . In RecSys '14: Proceedings of the 8th ACM Conference on Recommender systems. 1-8.

[2] Tanya Bondarouk and Chris Brewster . 2016 . Conceptualising the Future of HRM and Technology Research . The International Journal of Human Resource Management 27 , 21 ( 2016 ), 2652 - 2671 .

[3] James A. Breaugh . 2008 . Employee Recruitment: Current Knowledge and Important Areas for Future Research . Human Resource Management Review 18 , 3 ( 2008 ), 103 - 118 .

[4] Duen Horng Chau , Brad Myers, and Andrew Faulring . 2008 . What to Do when Search Fails: Finding Information by Association . In Proceedings of CHI '08. ACM , New York, NY, USA, 999 - 1008 .

[5] Ahmed Elsafty , Martin Riedl , and Chris Biemann . 2018 . Document-based recommender system for job postings using dense representations . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Volume 3 (Industry Papers). 216 - 224 .

[6] Evanthia Faliagka , Athanasios Tsakalidis, and Giannis Tzimas . 2012 . An Integrated e-Recruitment System for Automated Personality Mining and Applicant Ranking . Internet research ( 2012 ).

[7] Snorre S. Frid-Nielsen . 2019 . Find my Next Job: Labor Market Recommendations using Administrative Big Data . In RecSys '19: Proceedings of the 13th ACM Conference on Recommender Systems. ACM , 408 - 412 .

[8] George W. Furnas , Thomas K. Landauer, Louis M. Gomez , and Susan T. Dumais . 1987 . The Vocabulary Problem in Human-System Communication . Communications of the ACM 30 , 11 ( 1987 ), 964 - 971 .

[9] Eleanna Galanaki , Alessandra Lazazzara, and Emma Parry . 2019 . A Cross-National Analysis of e-HRM Configurations: Integrating the Information Technology and HRM Perspectives . In Organizing for Digital Innovation . Springer, 261 - 276 .

[10] Akshay Gugnani and Hemant Misra . 2020 . Implicit Skills Extraction Using Document Embedding and Its Use in Job Recommendation . In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 34 . 13286 - 13293 .

[11] Shiqiang Guo , Folami Alamudun, and Tracy Hammond . 2016 . RésuMatcher: A Personalized Résumé-Job Matching System . Expert Systems with Applications 60 (Oct 2016 ), 169 - 182 . https://doi.org/10.1016/j.eswa. 2016 . 04 .013

[12] Francisco Gutiérrez , Sven Charleer, Robin De Croon, Nyi Nyi Htun, Gerd Goetschalckx, and Katrien Verbert . 2019 . Explaining and Exploring Job Recommendations: A User-driven Approach for Interacting with Knowledge-based Job Recommender systems . In RecSys '19: Proceedings of the 13th ACM Conference on Recommender Systems. ACM , 60 - 68 . https://doi.org/10.1145/3298689.3347001

[13] Karen Holtzblatt and Sandra Jones . 1993 . Contextual Inquiry: A Participatory Technique for System Design . Lawrence Erlbaum Assoc., Hillsdale, NJ, 177 - 210 .

[14] Faizan Javed , Phuong Hoang, Thomas Mahoney, and Matt McNair . 2017 . Largescale occupational skills normalization for online recruitment . In Twenty-ninth IAAI conference.

[15] Ugur Karaboga and Pelin Vardarlier . 2020 . Examining the Use of Artificial Intelligence in Recruitment Processes . Bussecon Review of Social Sciences (2687- 2285) 2 , 44 (Dec 2020 ), 1 - 17 . https://doi.org/10.36096/brss.v2i4. 234

[16] Emanuel Lacic , Dominik Kowald, Markus Reiter-Haas, Valentin Slawicek , and Elisabeth Lex . 2017 . Beyond accuracy optimization: On the value of item embeddings for student job recommendations . arXiv preprint arXiv:1711.07762 ( 2017 ).

[17] Emanuel Lacic , Markus Reiter-Haas, Tomislav Duricic, Valentin Slawicek, and Elisabeth Lex . 2019 . Should we embed? A study on the online performance of utilizing embeddings for real-time job recommendations . In Proceedings of the 13th ACM Conference on Recommender Systems . 496 - 500 .

[18] Liangyue Li , How Jing , Hanghang Tong, Jaewon Yang, Qi He , and Bee-Chung Chen . 2017 . Nemo: Next career move prediction with contextual embedding . In Proceedings of the 26th International Conference on World Wide Web Companion . 505 - 513 .

[19] Mengshu Liu , Jingya Wang, Kareem Abdelfatah , and Mohammed Korayem . 2019 . Tripartite vector representations for better job recommendation . arXiv preprint arXiv: 1907 . 12379 ( 2019 ).

[20] François Mairesse , Marilyn A. Walker , Matthias R. Mehl , and Roger K. Moore . 2007 . Using Linguistic Cues for the Automatic Recognition of Personality in Conversation and Text . Journal of Artificial Intelligence Research 30 ( 2007 ), 457 - 500 .

[21] Paolo Montuschi , Valentina Gatteschi, Fabrizio Lamberti, Andrea Sanna, and Claudio Demartini . 2014 . Job Recruitment and Job Seeking Processes: How Technology Can Help . IT Professional 16, 5 (Sep 2014 ), 41 - 49 . https://doi.org/10. 1109/MITP. 2013 .62

[22] Jefrey R Mueller and Bernadette Baum . 2011 . The Definitive Guide to Hiring Right . Journal of Applied Business and Economics 12 , 3 ( 2011 ), 140 - 153 .

[23] Ioannis Nikolaou . 2021 . What is the Role of Technology in Recruitment and Selection? The Spanish Journal of Psychology 24 ( 2021 ). https://doi.org/10.1017/ SJP. 2021 .6

[24] Deirdre O'Donovan . 2019 . HRM in the Organization: An Overview . Springer, 75 - 110 . https://doi.org/10.1007/978-3- 030 -13229- 3 _ 4

[25] Andi Peng , Besmira Nushi, Emre Kıcıman, Kori Inkpen, Siddharth Suri, and Ece Kamar . 2019 . What you see is what you get? The Impact of Representation Criteria on Human Bias in Hiring . In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , Vol. 7 . 125 - 134 .

[26] Manish Raghavan , Solon Barocas, Jon Kleinberg, and Karen Levy . 2020 . Mitigating bias in algorithmic hiring: Evaluating claims and practices . In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency . 469 - 481 .

[27] Mary Elizabeth Raven and Alicia Flanders . 1996 . Using Contextual Inquiry to Learn about your Audiences . ACM SIGDOC Asterisk Journal of Computer Documentation 20 , 1 ( 1996 ), 1 - 13 .

[28] Tony Russell-Rose and Jon Chamberlain . 2016 . Real-World Expertise Retrieval: The Information Seeking Behaviour of Recruitment Professionals . In Advances in Information Retrieval (Lecture Notes in Computer Science) , Nicola Ferro, Fabio Crestani, Marie-Francine Moens , Josiane Mothe, Fabrizio Silvestri, Giorgio Maria Di Nunzio, Claudia Hauf, and Gianmaria Silvello (Eds.). Springer International Publishing, 669 - 674 . https://doi.org/10.1007/978-3- 319 -30671-1_ 51

[29] Tony Russell-Rose and Jon Chamberlain . 2016 . Searching for Talent: The Information Retrieval Challenges of Recruitment Professionals . Business Information Review 33 , 1 (Mar 2016 ), 40 - 48 . https://doi.org/10.1177/0266382116631849

[30] Tony Russell-Rose and Tyler Tate . 2012 . Designing the Search Experience: The Information Architecture of Discovery . Newnes.

[31] Javier Sánchez-Monedero , Lina Dencik , and Lilian Edwards . 2020 . What does it mean to 'solve' the Problem of Discrimination in Hiring? Social, Technical and Legal Perspectives from the UK on Automated Hiring Systems . In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency . 458 - 468 .

[32] Yeresime Suresh , A Manusha Reddy , et al. 2021 . A Contextual Model for Information Extraction in Resume Analytics Using NLP's Spacy . In Inventive Computation and Information Technologies . Springer, 395 - 404 .

[33] T.P. Thebe and Gerrit Van der Waldt . 2014 . A Recruitment and Selection Process Model: The Case of the Department of Justice and Constitutional Development . ( 2014 ).

[34] Mark E. Tomassen . 2016 . Exploring the Black Box of Machine Learning in Human Resource Management: An HR Perspective on the Consequences for HR Professionals . Master's thesis . University of Twente.

[35] Ryen W. White , Susan T. Dumais, and Jaime Teevan . 2009 . Characterizing the Influence of Domain Expertise on Web Search Behavior . In WSDM '09: Proceedings of the Second ACM International Conference on Web Search and Data Mining. ACM , 132 - 141 . https://doi.org/10.1145/1498759.1498819

[36] Steve Whittaker , Ofer Bergman, and Paul Clough . 2010 . Easy on that Trigger, Dad: A Study of Long-Term Family Photo Retrieval . Pers. Ubiquitous Comput . 14 , 1 ( 2010 ), 31 - 43 .

[37] Lilian Woudstra and Bart van den Hoof. 2008 . Inside the Source Selection Process: Selection Criteria for Human Information Sources . Information Processing & Management 44 , 3 ( 2008 ), 1267 - 1278 .