The Web has created a global marketplace for e-Commerce as well as for talent. Online employment marketplaces provide an e ective channel to facilitate the matching between job seekers and hirers. This paper presents an initial exploration of user behavior in job and talent search using query and click logs from a popular employment marketplace. The observations suggest that the understanding of users' search behavior in this scenario is still at its infancy and that some of the assumptions made in general web search may not hold true. The open challenges identi ed so far are presented.
•Information systems →Query log analysis; Job search; Talent search; Employment marketplace
The Web has created a global marketplace for e-Commerce and also employment. Job and talent search are two complementary sides of the employment marketplace, with both intended to pair people with opportunities. Job search – the process of an individual monitoring for opportunities, or seeking fresh employment in roles for which they have the skills and experience and for which they will be paid a suitable level of compensation – is a traditional marketplace, dominated for many decades by newspaper classi ed advertising. Talent search – in which a company or employer seeks candidates who might be suitable for a position within their business that is (or might shortly be) available – is a more recent addition to this ecosystem.
Both job search and talent search have become increasingly
o ered as e ective online services: unemployed persons who look
for work online are re-employed about 25% faster than comparable
workers who do not search online [
As search activities, the job and talent search processes have di erent aims to standard web or enterprise information search. Users of job and talent search services tend to have search needs with at least some speci c parameters (usually including the type of job, even if expressed via an uncontrolled vocabulary, and often including locational constraints), but may not have a speci c document or even employer in mind. Indeed, for a job searcher there is an element of “feeling lucky” every time they search, even if they enter the same query as they did last week. At the same time, the psychology of job seekers and hirers also contributes to a somewhat more recall-oriented searching process than is usually ascribed to web search. The desire to not miss out on a dream job or a talented candidate may mean that the users are more engaged with the search process, and invest more time in perusing results listings.
However, job seekers – unlike patent or legal searchers – are
unlikely to wish to examine all results that satisfy a measure of
relevance. When faced with hundreds of matches for one query, they
are instead likely to add re nements, and also adjust their internal
calibration as to what they are seeking. A person who searches
for “barista in Melbourne” and is shown hundreds of matching
position vacancies might well immediately re-query with an added
“salary range” lter, or specify a more precise geographical location.
This type of search activity is not dissimilar to certain domains of
vertical search such as automobile or real estate sales [
Talent search – when a company or organization is searching
across resumes and personal descriptions in order to identify
candidates that might be interested in applying for vacant or forthcoming
positions – is similar to the task of nding an expert [
In this paper, we present preliminary work comparing users’ search behavior for job search, talent search and more traditional web-search. Our purpose is to better understand whether the underlying assumptions we have with regard to user models, ranking factors and success metrics in web search can (or should) hold true for job and talent search. Longer-term, our aim is to understand what properties of user behaviors, target documents (job ads or user pro les) and their summary descriptions lead to users clicking through to the documents and on to job applications or recruitment requests. Ideally, this would include an accounting of the di erent reasons a user may have for posing a job or talent search query. For example, an unemployed person might be actively job hunting, whereas someone currently employed might be researching market salaries in order to negotiate within their existing position. Similarly, a user of a talent search service might be primarily seeking to understand how competitive the marketplace is at present and trying to decide whether it is even worth commencing a recruitment campaign for a proposed new role.
The overall goal of such analysis is to improve services to users via improved matching of positions on o er, improved pools of potential candidates being generated, and higher levels of employer and employee satisfaction. With that objective as our goal, the next section examines characteristics of user behavior when performing job and talent search, and compares these to characteristics of web search behaviors. We use job and talent search click and query logs from SEEK Ltd., one of the world’s leading job seeking and talent search companies, with over 30 million user visits per month in Australia and New Zealand alone. Our results show some fundamental di erences compared to standard web search behavior. The following section then outlines some speci c research challenges in relation to understanding the intents and goals of job and talent seekers as they search, and mechanisms to improve search performance and experience in these two important contexts. 2
The datasets used to compute the statistics and the main results of our log analysis are described in this section. Note in particular that these datasets are distinct for job and talent search, and for web search. They were generated by di erent search systems in di erent time periods, for di erent populations of users, and with the results presented via di erent interfaces and pagination. In particular, there are 20 results per page for the job and talent search applications we examine, and 10 results per page in the web search interface.
Logs for Job and Talent Search. SEEK Ltd. (“SEEK” thereafter) is a diverse group of companies, comprising online employment, educational, commercial and volunteer businesses which span across Australia, New Zealand, South East Asia, China, Brazil, Mexico, Africa and the Indian subcontinent. SEEK’s online marketplaces are exposed to approximately 4.1 billion people and more than 30% global GDP.2 The click and query logs used in this paper are proprietary data from the Australia and New Zealand employment business of SEEK.3
The domestic SEEK employment business facilitates candidates to nd employment opportunities, and helps hirers to nd candidates for advertised roles. Hirers currently pay to have their ads posted on SEEK; then, on the job search side, anyone can access these jobs via the search interface at no cost. Candidates also have the option of registering to create pro les which are used to streamline the job application process. In addition to keywords, a number of search facets are made available to the candidates, including a job classi cation taxonomy, a location taxonomy, work types, and salary ranges. Each job ad is represented by a title, a short description summarized via bullet points, and some meta-data about the job (including posting date, job location, and classi cation of the ad). Relevance and posting date (“newest rst”) are the two features by which results can be sorted. Candidates click on the job titles in the results page if they wish to read the full content of the job ads or to apply for them.
SEEK also o ers hirers the ability to pro-actively search for candidates via their pro les; this product is known as Talent Search. Hirers are provided with the option to lter the search results using a location taxonomy, the work type and salary information from the pro les, companies where a candidate has previously worked, an industry taxonomy, and the candidate’s right to work (visa and citizenship) information. On the search result pages, hirers can contact the candidates by sending them messages or inviting them to apply for jobs. These are known as connection methods.
The datasets used in this paper are in two parts. The rst tranche covers overall queries and the clicks by candidates searching job ads in response to search result pages over a 4-month period between January 2016 to April 2016. This set contains about 140M searches by job seekers. The second contains about 1.2M searches by hirers and the connections that were performed over the same period of time.
Logs for Web Search. The frequency of click positions for web
search was computed using the click logs provided by Yandex [
Web Search 10- 1 y c n e u q e r F10- 3 d e z il a m r o N
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 10 50 60 20
30 Rank Position 40 from the Relevance Prediction Challenge.4 The public dataset for Russian-language web search contains 340,796,067 records with 30,717,251 unique queries, retrieving 10 URLs each. We used the training set, which consists of 5,191 assessed queries corresponding to 30,741,907 records.
Depth of Clicks. In this section we examine the click frequency of search results in Search Engine Result Pages (SERPs) for talent, job and web search. For each rank position, the frequency of clicks is computed by aggregating all the queries in the dataset, that is, number of clicks divided by the total number of impressions. For each query, SERPs including at least one clicked result are considered.
Figure 1 shows the click probability for talent, job and web search. The overall pattern is as one would expect, with a considerable emphasis on the rst few positions of the rst page of results, followed by a steady decline. On the other hand, what is notable in both curves is that the distribution is relatively smooth over page boundaries. Both hirers and job seekers are more likely to click the last document in each SERP page than the one before it, but the usual steep drop-o at page boundaries – a disinclination to load the next page at all – is not present. Another interesting observation is that hirers carrying out talent search activities tend to be more persistent and explore further down subsequent pages than do job seekers in the job search context.
Figure 1 also provides the corresponding frequency of clicks for each rank position across the Yandex web search dataset. Positions
101
103
Query Rank
105
early in the ranking are again the ones most likely to be clicked.
This tendency is common to all types of searches, job, talent, and
web, and is known in literature as position bias [
Finally, it can be seen that the slope of the frequency graph is steeper for web search than for job and talent search. This may be due to presentation bias, since job results are paginated with 20 documents per page while web result pages contain 10 documents; or, as hypothesized in Section 1, it may be an innate di erence in user behavior. We plan to investigate this further as we continue with this project.
Query Popularity. We next analyze the distribution of queries in the three di erent datasets. For this analysis, a random sample of one million queries was extracted from each one; Figure 2 shows the frequency (normalized by total) of distinct queries for each subset using, log-log scales.
The gure shows that the top frequent queries in job search occur substantially more often than in talent and web search. The frequency in job search queries starts decreasing faster than in web and talent search. The rightmost values plotted for the three curves relative to the horizontal axis indicate the di erent number of unique queries across the three dataset samples. The total number of distinct queries is substantially lower for talent and job search than for web search (note that the gure is in log-scale). This 100 r e sU75 r e p s e i r e u50 Q t c n it s i D .m25 u N ● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
suggests that the long tail e ect is less pronounced in the specialized employment domain.
In fact, the total number of unique queries in the samples from
the job and talent search logs correspond to the 15% and 20% of the
number of distinct queries in the same-sized web search sample,
respectively. That is, the queries submitted by job seekers or hirers
are substantially less diverse than the queries submitted by users
of a web search engine. This trend is similar to that observed by
Jansen et al. [
Query Diversity Within Users. We now explore the diversity of queries submitted by individual users. For all the job seekers and hirers with more than one query in the samples, we computed the number of distinct queries they had issued. Figure 3 shows the distribution of unique queries per user for job and talent search.
The box-plots show that hirers submit a higher number of
different queries than job seekers. In fact, job seekers tend to repeat
the same queries, whereas in talent search 50% of hirers submit
more than 10 unique queries. Intuitively, the large variety of roles
that recruiters and employees from HR departments have to hire
in uences the more varied queries that they use. A job seeker, on
the other hand, generally assume a small number of roles over one’s
working life. This in turn dictates the range of words that are used
for job search. On the web search front, one may expect a higher
variability of queries submitted by users – the same user may have
several informational, navigational or transactional needs in a same
day. Moreover, it is known that users struggle to remember web
search queries even after a relatively short amount of time [
The observations above show that job and talent search have di erent characteristics from standard information search tasks. Hence, existing click models and evaluation frameworks designed primarily for web search may not translate to this domain. For both job and talent search, the quality of a result tends to be signi cantly more nuanced and important, hence: (1) searchers (job seekers or hirers) tend to lter results more carefully; (2) searchers care about the “freshness” of results (that is, results not previously seen) for a given query; (3) certain aspects of a query have di erent weights (for example, location). Overall, searchers tend to spend more time on examining a set of search results, and will pose more queries for a given need.
Speci c questions to be investigated for this class of search problems include:
Are click models used for web search [
Click models [
Finally, a searcher may have di erent goals for posing the same query: for instance, in a job search portal, a user may be a genuine job seeker or may just be monitoring the market to research salary ranges for certain positions (this may also be done by a talent seeker). An evaluation framework should consider such di erences in order to properly account for user search intent and thereby e ectively measure the success of the search.
Addressing these challenges in the context of job and talent search could potentially inform investigations into other complex search tasks such as searching for cars or properties in the automotive and real estate domains. As in the employment domain, such search tasks are part of a decision making process that involves high costs (e.g., buying a house). Therefore, the users are more likely to invest in the search process. 4
Job and talent search have become increasingly o ered as e ective online services. However, little work has been done to understand users’ search behavior in these verticals. Our initial exploration suggests that the assumptions we have with regard to user models, ranking factors and success metrics in web search may not hold true for job and talent search. Ongoing and further work – which will include deeper analyses of substantially larger click and query logs from one of the most popular online employment marketplaces – will shed some light on the problem of modeling users’ behavior and satisfaction for job and talent search.
This research was partially supported by Australian Research Council Project LP150100252 and SEEK Ltd.