Query-level evaluation metrics such as nDCG that originate from field of Information Retrieval (IR) have seen widespread adoption in the Recommender Systems (RS) community for comparing the quality of diferent ranked lists of recommendations with diferent levels of relevance to the user. However, the traditional (ofline) RS evaluation paradigm is typically restricted to evaluating a single results list. In contrast, IR researchers have also developed evaluation metrics over the past decade for the session-based evaluation of more complex search tasks. Here, the sessions consist of multiple queries and multi-round search interactions, and the metrics evaluate the quality of the session as a whole.
Sessions for Talent Search
Arguably the most common application scenario for a recommender system is presenting
the user with a single list of recommended items that are personalized to their tastes and
interests, after which they engage with some (or none) of these items, thereby providing
Perspectives on the Evaluation of Recommender Systems Workshop (PERSPECTIVES 2022), September
https://mesutkaya.github.io/ (M. Kaya); https://http://toinebogers.com/ (T. Bogers)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
feedback on the quality of the recommendations. As a result, this scenario is also the one
most commonly simulated in ofline evaluation with many standardized evaluation metrics
available, such as RMSE, hit rate, and nDCG [
Another, less-researched scenario is that of human augmentation where, instead of
a primary role, the recommender system can play a supporting role to aid the user in
fulfilling their information need [
An example of such a scenario is talent search (or job matchmaking), where recruiters
attempt to identify relevant candidates for an open job posting and shortlist them by
assessing their qualifications, such as their knowledge, skills, abilities, work experience,
education level, and interests [
In recent years, various approaches have been proposed that use AI techniques to assist
recruiters in this matchmaking process, for instance by automatically extracting relevant
skills from the job postings [
At first glance, however, assessing the efectiveness of a RS in such a scenario may
seem like comparing apples to oranges. How can we fairly and realistically compare the
quality of an individual list of recommended items to the quality of an entire manual
search session (RQ1)? And how can we measure the contribution that the RS is making
to the entire search session (RQ2)? In this paper, we introduce a possible approach to
comparing and evaluating recommendation and search that attempts to answer these
research questions by adopting and adjusting session-level evaluation metric(s) from the
ifeld of IR [
While online A/B testing experiments are generally recognized as providing the most
realistic evaluation conditions for a recommender system, it is common to first run ofline
evaluation experiments to narrow down the space of promising candidate algorithms.
Due to its lack of real user interaction, ofline testing allows for rapid prototyping and
comparison of a wide range of candidate algorithms at much lower cost [
single result list, denoted as ()
vector ⃗ and a discount vector ⃗ [
Information seeking often takes the form of a multi-stage process with several rounds
of query formulation and interaction with search results [
As single-list evaluation metrics such as nDCG are aimed at measuring the efectiveness
of a single query or list of recommendations, several session-level evaluation metrics have
been proposed over the years [
=1 =1 () = ∑
∑ , ( ) ⋅ , where is a search session in which the searcher submits queries with gain , ( ) and discount factor ,
for the -th result returned for the -th query of respectively.
One instantiation of Eq. 1 is the Session Discounted Cumulated Gain (or sDCG), an
extension of nDCG to entire search sessions as proposed by Järvelin et al. [
1 )(1 + )
This discount factor has two diferent discount components: for the rank discount also part of nDCG, and for the query discount, which penalizes results interacted with near the end of the session more. These two components can be controlled by changing their logarithm base values, where larger values of and can be used to model users that spend more time, submit more queries and assess more search results. For a session consisting of a single query, sDCG is equal to DCG. The main diference between the diferent instantiations of
Eq. 1 is in the use of diferent discount factors, such as the
sRBP metric proposed by Lipani et al. [
There has also been related work on session-based recommendation and evaluation
in the RS community. In session-based RS, the goal is to generate recommendations
for an ongoing session based on a time-ordered sequence of interactions, organized in
sessions [
As we argued in Section 1, RS can also be used in conjunction with other information access methods, such as search engines in the case of talent search. Dzyabura and Tuzhilin [17] studied the problem of how best to combine search and recommendation results as they compared the performance of pure recommendation and pure search separately to that of an interleaved combination of search and recommendation results lists. They evaluated these three conditions on their recall of relevant items and found that the combination outperformed the individual results lists. However, none of the individual components represented entire interaction sessions, which is contrast with our focus.
Another approach that considered the intersection of search and recommendation was the narrative-driven recommendation scenario presented by Bogers and Koolen [18]. They defined narrative-driven recommendation as a recommendation scenario where the recommendation process is driven by both a log of the user’s past interactions as well as a narrative description of their current interest(s). This has many similarities with the case of talent search being supported by a RS: past search behavior and past interactions between job seekers and job postings constitute the interaction data and the job posting can be seen as a representation of the current information need of the recruiter. The authors illustrated their RS scenario with book recommendation, but they only evaluated single lists using nDCG and never considered search sessions.
To the best of our knowledge there are no other examples of direct comparisons between search and recommendation or evaluations of the contribution of RS in a supporting role. In the next section, we introduce our use case and argue for the value of session-based evaluation metrics such as sDCG for evaluating such an human-augmentation scenario.
To contextualize our advocacy of session-based evaluation metrics from IR to help with the ofline evaluation of a recommender system’s contributions in a human-augmentation scenario, we will first introduce a concrete use case for talent search at Jobindex1, a major job portal and recruitment agency in Scandinavia. We start by explaining their recruitment process and the augmentation scenario in more detail, followed by notational definitions and a description of our dataset. We then present our results to aid in answering our research questions.
Figure 1a is a visualization of the recruitment process at Jobindex. For each open job posting, recruiters extract the most important job requirements and then search Jobindex’s CV database for relevant candidate. When a recruiter starts a new job by selecting an open job posting , the search engine uses the location and the industry of the job posting as the initial query ( 0), in efect performing what Allan et al. [19] refers to as zero-shot retrieval. The results for this initial query tends to be imprecise but with (a) Original recruiter search session (b) Recommendation-augmented search session high recall, to give recruiters an indication of the dificulty of the task . Recruiters can inspect the results list 0 to see whether it contained relevant candidates (e.g., candidate CV2+).
After this initial results list, the recruiters start their search process with 1 by formulating a search query and (de)selecting filters. After issuing a new query, recruiters are shown another results list corresponding to the query in question. After each query, they may identify more relevant candidates (e.g., CV6o and CV7- for 1 and CV9+ for 2). It is possible (and common) that the same CV is returned diferent queries (e.g., CV2+). Recruiters are required to identify at least 20 relevant candidates based on their historic response rates, so they will keep assessing results lists, shortlisting candidates and reformulating their queries until they reach this number at query . Next, recruiters send the shortlisted candidates a message, to which they can respond positively (green in Figure 1a), negatively (red), or not at all (grey). White CVs in Figure 1a were not seen as relevant candidates by the recruiter.
This talent search process at Jobindex is a clear example of professional search and falls somewhere between pure lookup search and pure exploratory search. Typically, recruiters do not know in advance who they are looking for and will reformulate their queries and adjust their strategy based on their intermediate findings like in exploratory search. However, the recruiters have a clear goal in mind, are usually experts on the domain of their goal, and know how to achieve their goals—all uncommon for exploratory search [20].
Figure 1b shows how RS can assist recruiters for this search session. The ideal outcome for the recommendation algorithm would be to return a ranked list that (1) contains all the CVs that the recruiter would find after issuing multiple queries in the course of a search session; (2) ranks the most relevant candidates near the top (i.e., shortlisted candidates that also responded positively in the example, such as CV2+ and CV9+)); and (3) ranks relevant candidates higher if the recruiter spent more efort finding them. This way, the RS can arguably assist the recruiter in such a way that they will issue fewer number of queries, spend less time and efort.
In principle, exposing the recruiters to this list of CV recommendations need not be restricted to only the start of the search process. The automatic recommendations could be combined or interweaved with their manual search results at each search step either automatically or by giving the recruiters full control over the combination process. This augmented cognition scenarios shares several similarities with interactive recommendation [21], although they are far from identical.
Estimating the recruiters’ efort to find relevant candidates is a dificult task. According
to Vakkari [
For a job posting , a recruiter performs the talent search task and issues queries during the course of a session = { 1, 2, … , }. For each submitted query ∈ , a ranked result list = { 1, 2, … , } that contains CVs is returned where 1 is the first shown CV and is the last shown CV respectively. The session can be represented as = { 1, 2, … , }. For , a recruiter selects relevant candidate CVs from ⋃=1 and sends them a contact message about , where the set of relevant candidates for session can be referred as = {⟨ 1, rsp1, 1⟩ , … , ⟨ , rsp , ⟩}, where rsp is the response type (positive, negative or no response) given by contacted CV , and is the first time CV has been shown to the recruiter during the course of the , i.e., after the -th query for instance.
As argued in the previous section, augmenting recruiter’s search process using a RS ifts best in the initial 0 results list through zero-shot retrieval or zero-query search [19], thereby anticipating the recruiter’s needs and predicting which candidate CVs are most relevant for the job posting in question. We can compute and denote the recommendation list generated for a job posting as = { 1, 2, … , }, which contains a ranked set of CVs sorted by relevance as computed by some recommender algorithm. In our use case, for all job postings available in the dataset provided by Jobindex, we compute recommendation list by using a content-based recommender that uses job-title based embeddings of job postings and CVs to recommend relevant candidates for each job [23]. It is important to note that the focus of this paper is not on developing a new recommendation algorithm or claiming that our example algorithm is the best choice. Instead, we use this simplistic baseline algorithm to illustrate the evaluation approached. While this is a non-personalized run where the job ad is used as a query, one could use interaction data between job seekers and CVs to integrate item-to-item recommendations and even personalize the results per recruiters based on their past selection of filters. Diferent recommendation algorithms that have been proposed recently can be used instead of the content-based recommender we are using to recommend relevant candidates for jobs [24, 25].
We use search log data provided by Jobindex consisting of 7,425 unique search tasks performed by Jobindex’s recruiters in the period of February 12, 2022 to April 20, 2022. Each of these search tasks corresponds to a single job posting which is handled by a single recruiter. For each of the completed tasks, Jobindex logs the following data that we can use as part of our use case evaluation: Retrieved CVs For each submitted query in a session, a list of shown CV IDs based on the submitted query that includes the rank of the CV in that query’s result list.
All CV IDs are anoynmized and no personal information is available in the dataset. Response data For each completed session, response data on the set of contacted candidates is available, which includes the contacted candidate’s CV ID and the type of their response to the contact message (positive, negative, none).
3.5.1. Comparing search sessions to recommendation lists For a given session , we could consider the CVs that have been assessed as relevant by the recruiters, as test set and use evaluation metrics like nDCG to estimate and compare the efectiveness of the diferent recommendation lists. However, single-list evaluation metrics such as nDCG are not suitable for comparing single lists to the efectiveness of entire search sessions, since they will miss the searcher’s reformulation efort.We therefore propose using sDCG to measure the efectiveness of the search session and also recommendation list by using the example from Figure 1 and the data provided by Jobindex.
First, for each session , we assign relevance weights based on the response types of the contacted CVs, . There are a few constraints for assigning these relevance weights in the Jobindex use case. First, we argue that all of the contacted candidates have been shortlisted and contacted by the recruiter should be seen as relevant, even if they respond negatively. Considering the job a good feedback but not wanting to switch jobs at the moment is also covered by the negative response, so these negative responses could represent a relevant candidate. In addition, the relevance weights for the diferent responses—positive, negative and no response (= ‘nr’)—should follow > > . While these relevance weights are contextually dependent, we set them to = 10, = 2 and = 1 for our use case to allow us to optimize more for the positive responses2.
To show how the sDCG metrics are calculated, we include Table 1, which corresponds to the toy example shown in Figure 1a. It shows the vectors representing each query result list containing the gain (G), discounted gain (sDG), discounted cumulated gain (sDCG) and normalized sDCG (nsDCG) values for the initial results list 0, the three queries issued by the recruiter 1– 3. We also include an artificial top-5 recommendation list from Figure 1b which takes the place of the initial results list in Figure 1a. Because sDCG defaults to nDCG for single-query sessions, we can use the same metric to compare the single list of recommendations to the full search session and compute the G, sDG, sDCG and nsDCG values for it, allowing us to compare to seemingly disparate information access artefacts.
Note that some CVs may be returned multiple times for diferent queries in a session.
In order to compute the gain vectors, we must decide whether to consider the gain only
the first time a CV is returned or every time it is returned by the system. In our dataset,
we do not have the information about at which point ( -th query) the recruiter assessed
a candidate CV as relevant, so we include a CVs gain value every time it is shown to the
recruiter. For computing the nsDCG values, we need to construct the ideal gain vector
for each query. In the ideal situation, a recruiter would only have to issue a single query
to find all the relevant candidates to be contacted in the result list returned for the
. When calculating the ideal gain vector for the entire search session, we follow Järvelin
et al. [
To produce a summary statistic similar to nDCG, we calculate the average sDCG and nsDCG values at diferent ranks (RQ1). This allows us to compare at which rank of query the sDCG for the search session surpasses the sDCG of the recommendation algorithm. In other words, how much of the recruiter’s work can the recommender system do before the recruiter’s experience takes over. We also plot the gain vectors containing the sDCG values to visually show this intersection point, as shown in Figures 2a and 2b.
Figures 2a and 2b show the sDCG and nsDCG values averaged over the 7,425 unique search sessions in our Jobindex dataset. For each session we computed the sDCG and nsDCG values for top-10 queries and their top-20 results. In addition, the vertical lines represent the average sDCG and nsDCG scores for our baseline embeddings-based 2We note that, we have data about whether candidates responded to a job suggestion positively or not, we do not have any information about whether they actually applied for the position or whether they were hired. This means, if available applying or being hired information can be assigned larger relevance weights. recommendation algorithm. The latter scores are represented as a horizontal line to show at what session-query position the recruiter’s experience finally beats the recommender system. The further to the right this intersection point is, the closer the recommender system is compared to the entire search session. If the recommendation list corresponded to the ideal result list, then the horizontal line would always be above the curve representing the search session.
Figures 2a and 2b also have horizontal lines showing the sDCG and nsDCG values of recommendation lists computed by a recommendation algorithm. For Figures 2a and 2b, we can look at the point (marked with red in the plots) that searcher surpasses the recommendation list efectiveness during the course of multiple-query search session. This can be done by looking at the intersection point of the horizontal lines (sDCG of recommendation algorithm) and line plots (sDCG values for the search session). For instance, if we look at Figure 2b, we can see that nsDCG value of the multiple-query search session surpasses that of recommendation algorithm right at the beginning of the 3rd query. Ideally, the better the recommendation list generated is, the further to the right of the plot the intersection point is. This shows that by using sDCG we can efectively compare a recommender system supplying the initial zero-query search results to the recruiter’s entire session in an ofline setting.
We can also use sDCG to measure the actual contribution that the RS is making to the recruiters’ search process (RQ2) once it is integrated into Jobindex’s systems (as visualized in Figure 1b). In such an online evaluation scenario, we can set up an A/B test where we test their original search process to the recommendation-augmented process. Once the contacted candidates have had a chance to respond to the contact messages sent by the recruiters, we can again compute sDCG for both variants and compare them. For instance, in the toy example in Figure 1b and Table 1, the sDCG@5 score for the recommendation list would be 7.62, which would allow one to estimate the potential contribution of the recommendation list to the entire search session.
In this paper, we investigated how session-level evaluation metrics can be used to evaluate and compare both recommender systems and search engines. Through our use case of Jobindex, we show that it is possible to compare the efectiveness of recommendations from a single ranked list to that from a whole search session with multiple queries issued. Although we used talent search as a use case in this paper, our proposed evaluation scenario can be used for any professional, complex information access tasks where RS can be used to augment the cognitive abilities of an information professional, such as patent search, search for legal documents. We believe that this is an under-investigated problem setting that deserves more attention.
We note that this is a preliminary study to explore how to compare both recommender systems and search engines by using session-level evaluation metrics. There are some limitations of our current work. In this paper, we do not make a principled analysis of the robustness and the discriminative power of the adapted session-level evaluation metrics for the ofline evaluation of recommender systems.
Searcher efort can be incorporated into session-based evaluation and can be used to compare the performance of separate recommendation lists generated by diferent recommendation algorithms against each other using/adapting metrics like sDCG. For instance, for the toy example shown in Figure 1a, as we argued before , for CV9+ the efort that the recommendations can save for the recruiter is more than CV2+, then we should incorporate searcher’s efort factor when comparing diferent recommendation lists. This factor for instance will give higher gain in this example to CV 9+ than CV2+.
One way to do this is to incorporate a dynamic value adjustment for a more realistic
evaluation and comparison of diferent recommendation lists. Consider the gain factor in
Eq. 1, () , which is the relevant judgment for the results. For completed search sessions,
we know the set of returned search results that had been assessed by the recruiters and
we also know the response type given by the contacted job seekers. We can multiply
the relevance weights by a factor similar to recency efect proposed by Zhang et al. [
We leave this and the discussion of how to design better function(s) for dynamically adjusting the gain values based on, for instance, a historic analysis of the time and efort spent by searchers, to future work.
We are also planning to get feedback of job recruiters on the usefulness of the recommendations during their search sessions, and compare the results of the user study with the results of the proposed evaluation methodology.
This research was supported by the Innovation Fund Denmark, grant no. 0175-000005B.
Information Science 42 (2016) 7–18. [17] D. Dzyabura, A. Tuzhilin, Not by search alone: How recommendations complement search results, in: Proceedings of the 7th ACM Conference on Recommender Systems, RecSys ’13, Association for Computing Machinery, New York, NY, USA, 2013, p. 371374. URL: https://doi.org/10.1145/2507157.2507231. doi:1 0 . 1 1 4 5 / 2 5 0 7 1 5 7 . 2 5 0 7 2 3 1 . [18] T. Bogers, M. Koolen, Defining and Supporting Narrative-driven Recommendation, in: RecSys ’07: Proceedings of the 11th ACM Conference on Recommender Systems, 2017, pp. 238–242. [19] J. Allan, B. Croft, A. Mofat, M. Sanderson, Frontiers, challenges, and opportunities for information retrieval: Report from swirl 2012 the second strategic workshop on information retrieval in lorne, in: Acm sigir forum, volume 46, ACM New York, NY, USA, 2012, pp. 2–32. [20] R. W. White, R. A. Roth, Exploratory Search: Beyond the Query-Response Paradigm, Synthesis Lectures on Information Concepts, Retrieval, and Services 1 (2009) 1–98. [21] C. He, D. Parra, K. Verbert, Interactive Recommender Systems: A Survey of the State of the Art and Future Research Challenges and Opportunities, Expert Systems with Applications 56 (2016) 9–27. [22] M. Kaya, T. Bogers, Under Review (Anon.). [23] M. Kaya, T. Bogers, Efectiveness of job title-based embeddings on résumé-to-job-ad recommendation, in: Proceedings of the RecSys in HR 2021 workshop, 2021, pp. 35–41. [24] E. Lacic, M. Reiter-Haas, T. Duricic, V. Slawicek, E. Lex, Should we embed? a study on the online performance of utilizing embeddings for real-time job recommendations, in: Proceedings of the 13th ACM Conference on Recommender Systems, 2019, pp. 496–500. [25] D. Lavi, Learning to match job candidates using multilingual bi-encoder bert, in: Fifteenth ACM Conference on Recommender Systems, 2021, pp. 565–566.