=Paper=
{{Paper
|id=Vol-2311/paper_2
|storemode=property
|title=Understanding User Behavior in Job and Talent Search: An Initial Investigation
|pdfUrl=https://ceur-ws.org/Vol-2311/paper_2.pdf
|volume=Vol-2311
|authors=Damiano Spina,Maria Maistro,Yongli Ren,Sargol Sadeghi,Wilson Wong,Timothy Baldwin,Lawrence Cavedon,Alistair Moffat,Mark Sanderson,Falk Scholer,Justin Zobel
|dblpUrl=https://dblp.org/rec/conf/sigir/SpinaMRSWBCMSSZ17
}}
==Understanding User Behavior in Job and Talent Search: An Initial Investigation==
https://ceur-ws.org/Vol-2311/paper_2.pdf
Understanding User Behavior in Job and Talent Search:
An Initial Investigation
Damiano Spina Maria Maistro Yongli Ren
RMIT University Univerity of Padua RMIT University
Melbourne, Australia Padua, Italy Melbourne, Australia
damiano.spina@rmit.edu.au maistro@dei.unipd.it yongli.ren@rmit.edu.au
Sargol Sadeghi Wilson Wong Timothy Baldwin
SEEK Ltd. SEEK Ltd. The University of Melbourne
Melbourne, Australia Melbourne, Australia Melbourne, Australia
ssadeghi@seek.com.au wwong@seek.com.au tb@ldwin.net
Lawrence Cavedon Alistair Moffat Mark Sanderson
RMIT University The University of Melbourne RMIT University
Melbourne, Australia Melbourne, Australia Melbourne, Australia
lawrence.cavedon@rmit.edu.au ammoffat@unimelb.edu.au mark.sanderson@rmit.edu.au
Falk Scholer Justin Zobel
RMIT University The University of Melbourne
Melbourne, Australia Melbourne, Australia
falk.scholer@rmit.edu.au jzobel@unimelb.edu.au
ABSTRACT paid a suitable level of compensation – is a traditional marketplace,
The Web has created a global marketplace for e-Commerce as well dominated for many decades by newspaper classified advertising.
as for talent. Online employment marketplaces provide an effective Talent search – in which a company or employer seeks candidates
channel to facilitate the matching between job seekers and hirers. who might be suitable for a position within their business that is
This paper presents an initial exploration of user behavior in job and (or might shortly be) available – is a more recent addition to this
talent search using query and click logs from a popular employment ecosystem.
marketplace. The observations suggest that the understanding of Both job search and talent search have become increasingly
users’ search behavior in this scenario is still at its infancy and that offered as effective online services: unemployed persons who look
some of the assumptions made in general web search may not hold for work online are re-employed about 25% faster than comparable
true. The open challenges identified so far are presented. workers who do not search online [7]. In other research, it has
been shown that employees who found their roles online tend to
CCS CONCEPTS stay on for longer. More specifically, “exit rates are lowered by at
least 28% when the internet is used as a job search tool” [10]. The
•Information systems →Query log analysis;
scale and reach of these services, and their benefits in terms of both
personal and corporate productivity, make online job and talent
KEYWORDS
search enormously valuable: the global online job/talent search
Job search; Talent search; Employment marketplace market has been recently estimated at $20-30 billion annually.1
As search activities, the job and talent search processes have
1 INTRODUCTION different aims to standard web or enterprise information search.
Users of job and talent search services tend to have search needs
The Web has created a global marketplace for e-Commerce and also
with at least some specific parameters (usually including the type
employment. Job and talent search are two complementary sides
of job, even if expressed via an uncontrolled vocabulary, and of-
of the employment marketplace, with both intended to pair people
ten including locational constraints), but may not have a specific
with opportunities. Job search – the process of an individual moni-
document or even employer in mind. Indeed, for a job searcher
toring for opportunities, or seeking fresh employment in roles for
there is an element of “feeling lucky” every time they search, even
which they have the skills and experience and for which they will be
if they enter the same query as they did last week. At the same
Copyright © 2017 by the paper’s authors. Copying permitted for private and academic time, the psychology of job seekers and hirers also contributes to a
purposes. somewhat more recall-oriented searching process than is usually
In: J. Degenhardt, S. Kallumadi, M. de Rijke, L. Si, A. Trotman, Y. Xu (eds.):
Proceedings of the SIGIR 2017 eCom workshop, August 2017, Tokyo, Japan, published
ascribed to web search. The desire to not miss out on a dream job
at http://ceur-ws.org or a talented candidate may mean that the users are more engaged
1 http://www.hhmc.com.au/2015/07/examining-the-job-board-market/
D. Spina, M. Maistro, Y. Ren, S. Sadeghi, W. Wong, T. Baldwin
SIGIR eCom 2017, August 2017, Tokyo, Japan L. Cavedon, A. Moffat, M. Sanderson, F. Scholer and J. Zobel
with the search process, and invest more time in perusing results 2 JOB AND TALENT SEARCH
listings. The datasets used to compute the statistics and the main results
However, job seekers – unlike patent or legal searchers – are of our log analysis are described in this section. Note in particular
unlikely to wish to examine all results that satisfy a measure of rel- that these datasets are distinct for job and talent search, and for
evance. When faced with hundreds of matches for one query, they web search. They were generated by different search systems in
are instead likely to add refinements, and also adjust their internal different time periods, for different populations of users, and with
calibration as to what they are seeking. A person who searches the results presented via different interfaces and pagination. In
for “barista in Melbourne” and is shown hundreds of matching particular, there are 20 results per page for the job and talent search
position vacancies might well immediately re-query with an added applications we examine, and 10 results per page in the web search
“salary range” filter, or specify a more precise geographical location. interface.
This type of search activity is not dissimilar to certain domains of
vertical search such as automobile or real estate sales [14], in which Logs for Job and Talent Search. SEEK Ltd. (“SEEK” thereafter) is
users are similarly conscious of the high/long-term impact of the a diverse group of companies, comprising online employment, edu-
decision that is being considered, and may iterate dozens or even cational, commercial and volunteer businesses which span across
hundreds of times before taking a further step, such as applying for Australia, New Zealand, South East Asia, China, Brazil, Mexico,
a position, or seeking more information. Africa and the Indian subcontinent. SEEK’s online marketplaces
Talent search – when a company or organization is searching are exposed to approximately 4.1 billion people and more than
across resumes and personal descriptions in order to identify candi- 30% global GDP.2 The click and query logs used in this paper are
dates that might be interested in applying for vacant or forthcoming proprietary data from the Australia and New Zealand employment
positions – is similar to the task of finding an expert [1, 2], although business of SEEK.3
arguably in a richer environment, since certain factors are likely The domestic SEEK employment business facilitates candidates
to be more critical (for example, specific experience, or geographic to find employment opportunities, and helps hirers to find candi-
location). dates for advertised roles. Hirers currently pay to have their ads
In this paper, we present preliminary work comparing users’ posted on SEEK; then, on the job search side, anyone can access
search behavior for job search, talent search and more traditional these jobs via the search interface at no cost. Candidates also have
web-search. Our purpose is to better understand whether the un- the option of registering to create profiles which are used to stream-
derlying assumptions we have with regard to user models, ranking line the job application process. In addition to keywords, a number
factors and success metrics in web search can (or should) hold true of search facets are made available to the candidates, including a
for job and talent search. Longer-term, our aim is to understand job classification taxonomy, a location taxonomy, work types, and
what properties of user behaviors, target documents (job ads or salary ranges. Each job ad is represented by a title, a short descrip-
user profiles) and their summary descriptions lead to users clicking tion summarized via bullet points, and some meta-data about the
through to the documents and on to job applications or recruitment job (including posting date, job location, and classification of the
requests. Ideally, this would include an accounting of the different ad). Relevance and posting date (“newest first”) are the two features
reasons a user may have for posing a job or talent search query. by which results can be sorted. Candidates click on the job titles in
For example, an unemployed person might be actively job hunting, the results page if they wish to read the full content of the job ads
whereas someone currently employed might be researching market or to apply for them.
salaries in order to negotiate within their existing position. Simi- SEEK also offers hirers the ability to pro-actively search for
larly, a user of a talent search service might be primarily seeking to candidates via their profiles; this product is known as Talent Search.
understand how competitive the marketplace is at present and try- Hirers are provided with the option to filter the search results using
ing to decide whether it is even worth commencing a recruitment a location taxonomy, the work type and salary information from
campaign for a proposed new role. the profiles, companies where a candidate has previously worked,
The overall goal of such analysis is to improve services to users an industry taxonomy, and the candidate’s right to work (visa and
via improved matching of positions on offer, improved pools of citizenship) information. On the search result pages, hirers can
potential candidates being generated, and higher levels of employer contact the candidates by sending them messages or inviting them
and employee satisfaction. With that objective as our goal, the next to apply for jobs. These are known as connection methods.
section examines characteristics of user behavior when performing The datasets used in this paper are in two parts. The first tranche
job and talent search, and compares these to characteristics of web covers overall queries and the clicks by candidates searching job ads
search behaviors. We use job and talent search click and query in response to search result pages over a 4-month period between
logs from SEEK Ltd., one of the world’s leading job seeking and January 2016 to April 2016. This set contains about 140M searches
talent search companies, with over 30 million user visits per month by job seekers. The second contains about 1.2M searches by hirers
in Australia and New Zealand alone. Our results show some fun- and the connections that were performed over the same period of
damental differences compared to standard web search behavior. time.
The following section then outlines some specific research chal-
Logs for Web Search. The frequency of click positions for web
lenges in relation to understanding the intents and goals of job and
search was computed using the click logs provided by Yandex [11]
talent seekers as they search, and mechanisms to improve search
performance and experience in these two important contexts. 2 http://www.seek.com.au/investor/about-us
3 http://www.seek.com.au
Understanding User Behavior in Job and Talent Search:
An Initial Investigation SIGIR eCom 2017, August 2017, Tokyo, Japan
1.0 ●
0.9
Job Search
Job Search 10−1
●
Talent Search Talent Search
0.8 Web Search
Normalized Click Frequency
Web Search
Normalized Frequency
0.7
●
0.6
10−3
0.5
●
0.4 ●
●
0.3 ●
●
●
10−5
●
0.2 ●
●
●
●
●
●●
●●●●
0.1 ●●
●●●
●●●●
●●●●●
●●●●●●●
●●●●●●●●
●●●●●●●●●●●●
0.0
0 10 20 30 40 50 60 101 103 105
Rank Position Query Rank
Figure 1: Normalized frequency of click positions in talent search Figure 2: Rank (log) and normalized frequency (log) of queries in
and job search for clicks on the first three pages (20 results per job, talent and web search.
page), and on the first result of the fourth page, and on the first
page (10 results per page) for web search.
early in the ranking are again the ones most likely to be clicked.
This tendency is common to all types of searches, job, talent, and
from the Relevance Prediction Challenge.4 The public dataset for web, and is known in literature as position bias [5, 15]. In the web
Russian-language web search contains 340,796,067 records with search dataset, the frequency of clicks for the last rank position on
30,717,251 unique queries, retrieving 10 URLs each. We used the the page (the 10th position) is similar to the frequencies associated
training set, which consists of 5,191 assessed queries corresponding with the 8th and 9th positions, without the up-tick in activity at
to 30,741,907 records. the end of the ranking. Web search users also tend to click less at
the bottom of the ranking than do the job search and talent search
Depth of Clicks. In this section we examine the click frequency users. This difference may be intrinsic, or may be a consequence of
of search results in Search Engine Result Pages (SERPs) for talent, the user interface in some way.
job and web search. For each rank position, the frequency of clicks Finally, it can be seen that the slope of the frequency graph is
is computed by aggregating all the queries in the dataset, that steeper for web search than for job and talent search. This may be
is, number of clicks divided by the total number of impressions. due to presentation bias, since job results are paginated with 20
For each query, SERPs including at least one clicked result are documents per page while web result pages contain 10 documents;
considered. or, as hypothesized in Section 1, it may be an innate difference in
Figure 1 shows the click probability for talent, job and web search. user behavior. We plan to investigate this further as we continue
The overall pattern is as one would expect, with a considerable with this project.
emphasis on the first few positions of the first page of results,
followed by a steady decline. On the other hand, what is notable Query Popularity. We next analyze the distribution of queries in
in both curves is that the distribution is relatively smooth over the three different datasets. For this analysis, a random sample of
page boundaries. Both hirers and job seekers are more likely to one million queries was extracted from each one; Figure 2 shows
click the last document in each SERP page than the one before it, the frequency (normalized by total) of distinct queries for each
but the usual steep drop-off at page boundaries – a disinclination subset using, log-log scales.
to load the next page at all – is not present. Another interesting The figure shows that the top frequent queries in job search
observation is that hirers carrying out talent search activities tend occur substantially more often than in talent and web search. The
to be more persistent and explore further down subsequent pages frequency in job search queries starts decreasing faster than in web
than do job seekers in the job search context. and talent search. The rightmost values plotted for the three curves
Figure 1 also provides the corresponding frequency of clicks for relative to the horizontal axis indicate the different number of
each rank position across the Yandex web search dataset. Positions unique queries across the three dataset samples. The total number
of distinct queries is substantially lower for talent and job search
4 http://imat-relpred.yandex.ru/en/ than for web search (note that the figure is in log-scale). This
D. Spina, M. Maistro, Y. Ren, S. Sadeghi, W. Wong, T. Baldwin
SIGIR eCom 2017, August 2017, Tokyo, Japan L. Cavedon, A. Moffat, M. Sanderson, F. Scholer and J. Zobel
3 FUTURE CHALLENGES
100 ●
●
The observations above show that job and talent search have differ-
ent characteristics from standard information search tasks. Hence,
● ●
●
●
●
●
●
existing click models and evaluation frameworks designed primar-
●
● ●
●
●
●
●
ily for web search may not translate to this domain. For both job
●
●
●
Num. Distinct Queries per User
●
●
●
and talent search, the quality of a result tends to be significantly
●
● ●
●
●
75 ●
●
more nuanced and important, hence: (1) searchers (job seekers or
●
●
●
●
●
●
hirers) tend to filter results more carefully; (2) searchers care about
●
●
●
●
● ●
●
the “freshness” of results (that is, results not previously seen) for a
●
●
● ●
●
●
●
●
●
given query; (3) certain aspects of a query have different weights
50 ●
(for example, location). Overall, searchers tend to spend more time
●
●
●
●
●
on examining a set of search results, and will pose more queries for
a given need.
●
●
●
Specific questions to be investigated for this class of search prob-
●
●
●
●
●
●
25
lems include:
●
●
●
●
●
●
●
●
●
• Are click models used for web search [3] applicable to job
●
●
●
●
●
●
search? Are the biases observed in web search for clicks
●
●
●
●
0 also occurring in job search?
• How can user behaviour and user satisfaction be modelled
Job Search Talent Search
from analyzing interactions in job search logs? [9, 13]
• How should job search be evaluated? Can job seeking
evaluation models [6, 8] inform the evaluation of job search
Figure 3: Query diversity within users for job and talent search. engines?
Click models [3] are valuable to describe behavior of web users
suggests that the long tail effect is less pronounced in the specialized by defining a set of rules in order to simulate user interactions with
employment domain. the search system. These models have a wide range of application
In fact, the total number of unique queries in the samples from in Information Retrieval, including predicting user clicks for A/B
the job and talent search logs correspond to the 15% and 20% of the testing experiments, inferring document relevance and defining
number of distinct queries in the same-sized web search sample, click based evaluation metrics.
respectively. That is, the queries submitted by job seekers or hirers Finally, a searcher may have different goals for posing the same
are substantially less diverse than the queries submitted by users query: for instance, in a job search portal, a user may be a genuine
of a web search engine. This trend is similar to that observed by job seeker or may just be monitoring the market to research salary
Jansen et al. [4], who compared job-related queries to the entire ranges for certain positions (this may also be done by a talent
set of queries submitted to a commercial web search engine. They seeker). An evaluation framework should consider such differences
found that around 60% of the job-related queries used the 100 most in order to properly account for user search intent and thereby
frequently occurring terms, whereas in the entire set the proportion effectively measure the success of the search.
is markedly lower, at around 20%. Addressing these challenges in the context of job and talent
search could potentially inform investigations into other complex
Query Diversity Within Users. We now explore the diversity of search tasks such as searching for cars or properties in the automo-
queries submitted by individual users. For all the job seekers and tive and real estate domains. As in the employment domain, such
hirers with more than one query in the samples, we computed the search tasks are part of a decision making process that involves
number of distinct queries they had issued. Figure 3 shows the high costs (e.g., buying a house). Therefore, the users are more
distribution of unique queries per user for job and talent search. likely to invest in the search process.
The box-plots show that hirers submit a higher number of dif-
ferent queries than job seekers. In fact, job seekers tend to repeat
the same queries, whereas in talent search 50% of hirers submit 4 CONCLUSION
more than 10 unique queries. Intuitively, the large variety of roles Job and talent search have become increasingly offered as effective
that recruiters and employees from HR departments have to hire online services. However, little work has been done to understand
influences the more varied queries that they use. A job seeker, on users’ search behavior in these verticals. Our initial exploration
the other hand, generally assume a small number of roles over one’s suggests that the assumptions we have with regard to user models,
working life. This in turn dictates the range of words that are used ranking factors and success metrics in web search may not hold
for job search. On the web search front, one may expect a higher true for job and talent search. Ongoing and further work – which
variability of queries submitted by users – the same user may have will include deeper analyses of substantially larger click and query
several informational, navigational or transactional needs in a same logs from one of the most popular online employment marketplaces
day. Moreover, it is known that users struggle to remember web – will shed some light on the problem of modeling users’ behavior
search queries even after a relatively short amount of time [12]. and satisfaction for job and talent search.
Understanding User Behavior in Job and Talent Search:
An Initial Investigation SIGIR eCom 2017, August 2017, Tokyo, Japan
ACKNOWLEDGMENTS [7] Peter Kuhn and Hani Mansour. 2014. Is Internet job search still ineffective? The
Economic Journal 124, 581 (2014), 1213–1233.
This research was partially supported by Australian Research Coun- [8] Maarten Lindeboom, Jan Van Ours, and Gusta Renes. 1994. Matching employers
cil Project LP150100252 and SEEK Ltd. and workers: An empirical analysis on the effectiveness of search. Oxford
Economic Papers 46, 1 (1994), 45–67.
[9] Daan Odijk, Ryen W. White, Ahmed Hassan Awadallah, and Susan T. Dumais.
REFERENCES 2015. Struggling and success in web search. In Proc. 24th ACM Int. Conf. on
[1] Krisztian Balog, Leif Azzopardi, and Maarten de Rijke. 2006. Formal models for Information and Knowledge Management. ACM, 1551–1560.
expert finding in enterprise corpora. In Proc. 29th Ann. Int. ACM SIGIR Conf. on [10] Anila Prakash. 2015. Three Essays on Labor Market Outcomes. Technical Report.
Research & Development in Information Retrieval. ACM, 43–50. The University of Arizona. http://arizona.openrepository.com/arizona/handle/
[2] Krisztian Balog, Yi Fang, Maarten de Rijke, Pavel Serdyukov, and Luo Si. 2012. 10150/560807.
Expertise retrieval. Foundations and Trends in Information Retrieval 6, 2–3 (2012), [11] Pavel Serdyukov, Nick Craswell, and Georges Dupret. 2012. WSCD 2012: Work-
127–256. shop on web search click data 2012. In Proc. 5th ACM Int. Conf. on Web Search
[3] Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. 2015. Click models for and Data Mining. ACM, 771–772.
web search. Synthesis Lectures on Information Concepts, Retrieval, and Services 7, [12] Jaime Teevan, Eytan Adar, Rosie Jones, and Michael A. S. Potts. 2007. Information
3 (2015), 1–115. re-retrieval: Repeat queries in Yahoo’s logs. In Proc. 30th Ann. Int. ACM SIGIR
[4] Bernard J. Jansen, Karen J. Jansen, and Amanda Spink. 2005. Using the web Conf. on Research & Development in Information Retrieval. ACM, 151–158.
to look for work: Implications for online job seeking and recruiting. Internet [13] Ryen W. White. 2016. Interactions with Search Systems. Cambridge University
Research 15, 1 (2005), 49–66. Press.
[5] Thorsten Joachims, Laura A. Granka, Bing Pan, Helene Hembrooke, Filip Radlin- [14] Xiaofang Yuan, Ji-Hyun Lee, Sun-Joong Kim, and Yoon-Hyun Kim. 2013. Toward
ski, and Geri Gay. 2007. Evaluating the accuracy of implicit feedback from clicks a user-oriented recommendation system for real estate websites. Information
and query reformulations in web search. ACM Trans. Information Systems 25, 2 Systems 38, 2 (2013), 231–243.
(2007), 7. [15] Yisong Yue, Rajan Patel, and Hein Roehrig. 2010. Beyond position bias: Examin-
[6] Marianna Kudlyak, Damba Lkhagvasuren, and Roman Sysuyev. 2012. System- ing result attractiveness as a source of presentation bias in clickthrough data. In
atic job search: New evidence from individual job application data. Technical Proc. 19th Int. Conf. on World Wide Web. ACM, 1011–1018.
Report 12-03R. FRB Richmond Working Papers. https://www.richmondfed.org/
publications/research/working_papers/2012/wp_12-03r.