Recruiting changed drastically with the emergence of professional social networks that bring together many people and companies. It is a chance as it helps to increase the adequacy between a position and a candidate. However, it creates new challenges. First, the many possible combinations make it hard to find the perfect match. Second, it brings together talents with many diferent skills and backgrounds that can be hard to understand for a recruiter. In particular, in computer science, technologies tend to change quickly and can be obscure for non-technical employees. Therefore, using automatic tools is crucial to guide the recruiting process. More specifically, a recommender system that matches candidates with open positions can improve the overall satisfaction of all the agents in our system. Yet, job-matching data sufers from the cold start problem: Once a person gets a position, they are very unlikely to obtain a new one soon. Thus, traditional techniques based on collaborative filtering are very limited, and we must rely on the unique characteristics of each candidate. In this paper, we propose a new recommender system based on a recruiting heterogeneous graph. This graph brings together information about a job posting and the personal knowledge graph of the candidates. We tested our model on a new real-world dataset, and we showed that it outperforms state-of-the-art methods.
dominantly use online applications to find a job: 77% use company websites, 58% use job posting sites, and 47% use a professional social network. These numbers keep increasing, and COVID-19 enforced the trend [2].
Dealing with the high number of candidates and job postings is a challenge for all the actors in the recruiting ecosystem. The applicants cannot browse through millions of open positions and need to be recommended the most relevant ones for their skills and experience. Likewise, recruiters cannot look at all possible candidates on a professional social network. Besides, it might be hard for them to fully understand a technical market such as IT (Information Technology), where skills are constantly changing. They need to find quickly the best talents who MySQL Has skill Lives in
Paris
Database Subskill
Subskill
Python Has skill
Requires skill Matched
PostgreSQL Requires skill
Data Scientist Wants salary
Proposes salary
Work in Berlin training data: Once a candidate gets a job, they are un- and explain why they are too limiting or unsuitable in our likely to get another one soon and on the same platform case. Then, we look at existing recommender systems, (on average, a person holds 12.5 jobs in their lifetime [4]). particularly those that could work in a similar scenario. Conversely, we expect an individual to have seen many series and movies, even on a single website. Therefore, 2.1. Datasets we cannot use traditional collaborative filtering techniques and must rely on additional features. A crucial component of a recommender system is the
One could only limit themselves to applications rather dataset used to train it. We want to compare them
rethan successful recruitments. This approach has the ad- garding our problem, which is job recommendation with
vantage of creating more training data but also produces cold start and semantic information. We include in our
a lot of noise for the recruiters. Indeed, candidates usually study the following datasets:
apply to many positions, especially online. According to
TalentWorks [
A further distinction in the recruiting domain is the strong presence of relevant features. When one creates an account on a streaming website, they do not fill in much personal information. As the number of training data is enough, straightforward collaborative filtering algorithms can produce good results. It is diferent for recruiting as a job or a candidate usually comes with information that helps both parties find the perfect match.
For example, we often find the city, skills, experience, or education of a person.
These diferences make the problem of job recommendation worth studying, especially in the case of cold start recommendations with semantic information. However, few datasets allow testing this configuration, so few proposed systems solve this problem.
This paper studies this specific problem and makes the following contributions: • MovieLens (ML) 100k, 1M, 10M [6] is the most common dataset used for benchmarking in recommender systems. It can be found at multiple scales depending on the use. It comprises data about users (age, sex, occupation), movies (title, release date, genres), and the users’ ratings that may be associated with a comment. It is essential to notice that the users’ information is unavailable for MovieLens 10M. • Gowala [7] is a location-based social networking website where users share their locations by checking in. It contains pairs of user and location identifiers associated with a timestamp and GPS coordinates. • Yelp [8] is a website allowing users to review businesses and other locations. The dataset contains reviews and classic information about the business, like the name, address, categories, or opening hours. • CareerBuilder’s job recommendation challenge [9] is a dataset published for an online data science hackathon on Kaggle by CareerBuilder.
This dataset is the one that shares the most similarity with our dataset. • Our newly created dataset, JobTrackingHistory
(JTH). Further details are provided in Section 4.1.
We compare these datasets according to several metrics. The standard ones are the size, the number of users, and the number of available features. Then, we introduce additional metrics to describe the cold start problem better. First, we have the density, which is the number of user-item associations divided by the total number of possible recommendations. This number highly correlates to the number of items and users. Therefore, we also include the average number of user and item associations. Finally, we report the number of users and items without an association and whether or not the dataset has a temporal dimension. The results are presented in Table 1.
From this table, we can see that our dataset stands out on two points. First, we have many users and items that • The creation of a dataset for job recommendation
with strong semantic information. • The building of a recruiting graph connecting the
candidates with jobs. • The implementation of a recommendation system
based on the created recruiting graph.
art methods in Section 2. Then, we will formally define our problem in Section 3. Section 4 presents our solution, and we give implementation details in Section 5. Finally, we compare our approach with the state-of-the-art in Section 6.
CareerBuilder Recruitment 6.1 GB 321,231 365,649 12 0.0014% 4.99 4.38 0 0 Yes
JTH
Recruitment
2.7 GB
26,078
4,026
37
0.04%
2.58
16.69
45,938
771
Yes
do not belong to any association. It creates a pool of graph-based recommender systems [17, 18, 19, 20]. Most
potential candidates and jobs that we could exploit. Sec- of them are based on Graph Neural Networks (GNN).
ond, the number of relations a user engages in is below Authors in [21] construct a knowledge graph for movie
all other datasets, which stresses the cold start problem. recommendation using DBpedia [22] and connect it to
Note here that we do not make a diference between a items only. Then, using Node2Vec [23] they compute
candidate application, a recruiter selection, and a suc- embeddings for users and items and plug them into a
cessful recruitment (see Section 4.1). Compared to the classifier. Here, the representation of the entities is not
CareerBuilder dataset, our dataset is much smaller. How- directly linked to the recommendation. KTUP [24] is
ever, it contains better features and a finer granularity inspired by TransH [25]. It contains two components: A
for the association between a candidate and a job. This knowledge graph encoder (TransH) and a user preference
fundamental diference will allow us to tackle the cold generator. This last part needs to learn embeddings for
start problem. all users and items solely from user-item interactions,
which is impossible in our case due to the sparsity of
2.2. Existing Architectures data. KGAT is [
The most popular systems are based on collaborative
filtering [
Another category of algorithms is based on matrix variations of conventional collaborative filtering [ 30, 31].
factorization [
We also find other kinds of recommender systems in the job market. For example, [37] recommends useful skills to learn. They construct of graph of skills and apply topic-modelling techniques to make the recommendation depend on the context. However, their approach only considers job postings (no candidates), and the graph they use is a simple cooccurrence graph. Still, they investigate useful features for job postings. [38] focuses on skill recommendation from the candidate’s point-of-view. They construct a skill ontology based on information mined on several websites. The job postings are not considerate.
In our work, we use an existing ontology (ESCO) that we complete with an external knowledge base (Wikidata).
Our dataset was constructed from a real-life IT recruitment system containing three main types of entities. The ifrst one is the candidates who are looking for a job. The second one is job positions that need to be filled. The last one is recruiters, whose primary role is to match candidates with positions. Once this matching is done, the company behind the job posting can choose to interview the candidates and potentially recruit them. Therefore, we do not have a binary association between users and items but rather a gradual score. In comparison with the CareerBuilder dataset, we have a less noisy dataset. In their case, an interaction is simply when a candidate applies for a job, whereas in our case, at least a professional recruiter validated the match.
To help the recruiter in this process, semantic information surrounds the candidates and the job postings. For the candidate, we have the resume from which skills are extracted, the current position, the current salary, the experience (in years), the asked salary, the area of expertise, the source (the system finds candidates through various websites), the desired type of contract, and the location.
For the job postings, we have the company that emitted it, the type of contract, the area of expertise required, a description, the required skills, the proposed salary, the date of emission, and the location. Most features are manually entered into the system by expert recruiters. The data also comes with the timestamp of significant events (creation of a posting, date of an interview, creation of a profile).
We call our final dataset Job Tracking History (JTH).
Statistics about it can be found in Table 1.
It comprises eleven kinds of nodes: the candidates, job postings, skills, salaries, years of experience, recruiters, companies, areas of expertise, employment types, localization, and candidate source. The relations are the ones described in Section 4.1. For candidates, job postings,
a tuple = (, , ) where is the set of all the nodes (or entities), is the set of all relationships, and is the set of edges (0, , 1) where 0 ∈ , 1 ∈ , and ∈ . Besides, each node is associated with a type that is a subset of .
In our case, the nodes will be composed of types like candidates, job postings, skills, or salaries. The relations will encode semantic information like “has skill”, “wants salary”, “requires skill”, or “was selected for job”. Figure 1 shows an example of a heterogeneous graph. Graph-Based Recommendation Given a heterogeneous graph = (, , ), a type (the users or candidates), a type (the items or jobs), and a relation ∈ , we want to predict for all ∈ and all ∈ if (, , ) ∈ .
In our case, we are mainly interested in relationships between candidates and job postings that indicate a positive recruitment. Of course, when we make the prediction, we need to be careful not to use the original edge if it exists.
In this paper, we tackle the case of heterogeneous graph-based recommendation for job recommendation. and skills, we also add a feature vector corresponding to the embeddings of the resume, job description, and skill name obtained with a sentence encoder built on top of MiniLM [39]. For some nodes representing continuous values (salaries, years of experience, location), we created nodes representing value ranges.
Then, we enhanced the semantic information for skills using two external knowledge bases: ESCO [40] (European Skills, Competences, Qualifications, and Occupations) and Wikidata [41]. ESCO is a European taxonomy of skills, competencies, and occupations. We only use the skill hierarchy from this taxonomy, i.e., how the skills are classified. As ESCO is limited regarding IT-specific skills (libraries, platforms), we augmented it using information extracted from Wikidata. In this new taxonomy, a skill can be associated with several labels. We use them to merge previous skill names (e.g., Javascript and JS).
Figure 1 shows an example of our recruiting graph.
a candidate and a job posting exists, even if we have a ifner granularity (matched, interviewed, selected). Each node in our graph is associated with an embedding. For nodes with features, the embedding combines the graph embedding and the feature vector through a linear layer.
Given a candidate and a job posting , we start by sampling a subgraph centered on and . To do so, we follow [42] by sampling interactively a certain number of neighbors given previously known nodes. This step is crucial as running a GNN on the entire graph would be impossible in a reasonable time.
Then, we pass our subgraph into a multilayer graph convolutional network (GCN) following a similar architecture as [42]. Initially, this architecture only worked for homogeneous graphs, so we used the transformation presented in [43]. The idea of this transformation is to duplicate the original network for each relation and then recombine the obtained representations. After the GNN layers, we receive a vector representation for and . We take the dot product of the two to make the prediction. We used the standard cross-entropy loss to evaluate the performance of the classifier. The architecture is presented in Figure 2. We call our final model RecruiterGCN.
at 2.2GHz and 14 cores and an NVIDIA Tesla V100 GPU with 32 GB of RAM. We stop the training phase following an early stop mechanism on the loss function. In total, for a given set of hyperparameters, the computation time was around five hours.
For the hyperparameters, we vary the number of layers, the number of neighbors used during sampling, the learning rate, and the negative sampling rate. We found the best results with six layers, a learning rate of 0.0001, a weight decay of 0.001, a two-stage sample with first twenty neighbors and then ten neighbors, and the creation of one random negative sample for each positive sample.
Our ifnal code is available on GitHub github.com/EricPoulet/RecSysInHR2023.
We include a large variety of algorithms in our baselines. First, we have traditional collaborative filtering approaches. UserCF [11] is a user-based collaborative ifltering based on the interaction matrix. Item2Vec [ 13] is an item-based collaborative filtering algorithm that learns embeddings for each item. ALS [16] is a matrix factorization algorithm. YouTubeRanking [12] is an approach that emphasizes external features more strongly. We manually create a feature vector that, for a given user and item, is composed of the number of common skills, if they have the same expertise area, the kind of contract, the source of the candidate, the years of experience, the salary, the distance to go to the job, and the cosine similarity between the resume and the job description. AutoInt [46] is another system that also leverages user-item features.
For the graph-based approaches, we used LightGCN [47], which takes the interaction graph (only candidates and jobs) as input and is based on a GCN. GraphSage [48] is a more advanced variation that takes similar input but puts more emphasis on sampling.
recommender systems: the mean precision at (P@K), the mean recall at (R@K), the mean average precision at (MAP@K), and the normalized discounted cumulative gain at (NDCG@K).
We used Python and the PyTorch Geometric library [44]. 6.1. Comparison With Baselines For the implementation of the baselines, we used LibRecommender [45]. We split our dataset into train, valida- Table 2 compares our approach with the baselines. We tion, and test sets following the proportion 0.8/0.1/0.1. can see that RecruiterGCN beats all baselines for the preSAMPLING
GCN
X P@10 0.0175 0.0111 0.0097 0.002 0.0108 0.001 0.0005 0.0048
R@10 0.159 0.0917 0.0801 0.0157 0.0926 0.0075 0.0041 0.0401
MAP@10 0.0505 0.0534 0.0343 0.0045 0.0463 0.0032 0.001 0.0253
Algorithm RecruiterGCN UserCF [11] Item2Vec [13] GraphSage [48] LightGCN [47] AutoInt [46] YouTubeRanking [12]
ALS [16] cision at K and the recall at K, but not for MAP@K and NDCG@K. These two last metrics emphasized the order of the recommendations, showing that our model might not be able to rank the top recommendations correctly.
Still, if the order is unimportant (a recruiter can easily post-process the ten suggestions), our algorithm brings extra value compared to the competitors. In particular, it improves over the other graph-based baselines. Regarding the model, it remains close to them in structure, which shows the importance of semantic information.
The simple user-based collaborative filtering algorithm works surprisingly well when we look at the baselines. Table 3 This is not intuitive as the cold start problem should Ablation Study - Ranked Results by P@10 disadvantage it. However, looking more into the data, we notice that recruiters like to cluster similar candidates and match them to identical job postings. Therefore, it biases with minimal human actions. RecruiterGCN automatithe results as knowing the group of a user makes the cally extracts relevant features and integrates them into predictions easier. This suggests we should use a more the final results. Besides, it is more flexible as adding advanced sampling mechanism and train/validation/test features does not require more feature engineering. split based on the date and by ignoring some types of nodes. 6.2. Ablation Study
The results are disappointing when we look at featurebased baselines (AutoInt and YoutubeRanking). We had We performed an ablation study to evaluate the impact of to manually translate the semantic information into a each semantic information in our graph. For each type of continuous vector for them. Therefore, the features engi- node (except candidates and jobs), we remove it from the neering can be long and fastidious. On the contrary, se- recruiting graph and rerun the experiments on it. The mantic information is naturally represented with a graph study results are presented in Table 3.
As expected, the graph without semantic information First, we saw that splitting the dataset into train, valida(Job + Candidate) gets poor results. However, the entire tion, and test sets is an essential step for preventing biases graph (All) does not get the best results. There can be introduced by the recruiters. The temporal aspect has a three main explanations for that fact. First, we might huge impact, although it is less studied in the literature. have noise in the evaluation that can cause slight varia- Likewise, the sampling strategies must be improved to tions. Second, we remove potential noise in the graph by include time and prevent too much noise in the network. eliminating some nodes. Therefore, the network might Continuous values can be tricky to use. We must find a be able to reach a better optimum. Third, because of the way to encode distances or cardinal orders between nodes sampling strategy, we might add additional noise if we to keep the standard graph representation. Otherwise, have too many semantic nodes. Here, we call “noise” the we should adjust the message-passing algorithm used presence of too much information or irrelevant informa- in the GCN to have a special treatment for these nodes. tion in the graph for the final recommendation. In particular, we need a way to include the temporal
The best setup is obtained with no skill in the graph, dimension in the graph as it is crucial for recruitment: which is a bit surprising. In fact, we still have general We cannot suggest an old job posting to a new candidate skills with the category nodes that are areas of expertise as it is very likely it is already filled. manually entered by recruiters. When we remove the Next, we ignored the granularity of the interactions skills, we remove some noise, and the network can better between a candidate and a job, whereas it could help to focus on other nodes. We can see that by eliminating reward top recommendations that lead to a new job rather areas of expertise (All - Category), we deteriorate the than a simple match. In the future, we must include this results deeply. We also notice decreased performance granularity during training and testing. Therefore, we when we remove the skill hierarchy (All - Concept). This must adapt the loss functions and the evaluation metrics. goes in a similar direction as the area of expertise: What Finally, as our system is supposed to assist one of the is essential are general skills (e.g., databases) rather than actors of our system (candidate, company, or recruiter), specific skills (e.g., MySQL). we need to work on how to present the recommendation.
We notice that continuous values are hard to exploit for Notably, we must be able to explain the recommendations the network: Removing salaries (All - Salary) improves using the semantic information in the graph. the performance, and removing the location (All - Zip) does not have much impact. However, these two factors should be crucial. As we used a graph structure, we had 7. Conclusion to discretize the values into ranges. We lost ordinal and comparative information in the process, making the end In this paper, we studied the problem of job recommennodes useless. dation. We leveraged a new human-annotated dataset
Some nodes do not seem to have much impact. Re- containing semantic information. We showed this informoving the company (All - Company) does not change mation can be translated into a heterogeneous graph the results. This is expected, and no prior reason exists without much manual feature engineering traditionfor a company to recruit a random candidate. Likewise, ally used in other systems. Then, we applied a graph the candidate’s origin does not change the results (All - neural network to produce relevant recommendations. Origin). It shows the quality of the candidates does not We showed that our system, RecruiterGCN, beats the depend on the website where the recruiter found them. state-of-the-art methods to isolate the top recommen
Three other kinds of nodes have a negative influence dations but still lacks more precise ranking capabilities. on the results. First, the type of contract (All - Contract), Our ablation study revealed that there are still points which is logical as both the company and the candidate of improvement, particularly regarding the inclusion are generally stringent on this point. Second, the re- of time in our model. Our final code is available on cruiter who found the candidate (All - Recruiter). This GitHub github.com/EricPoulet/RecSysInHR2023. shows that some recruiters might be better at finding good candidates. Our recommender system could help Acknowledgments junior recruiters to improve their skills. Finally, years of experience are crucial (All - Experience). This is also understandable, as a candidate with more experience is likelier to get a position.
[
1016/j.knosys.2020.106194. [37] T. Xu, H. Zhu, C. Zhu, P. Li, H. Xiong, Measuring [28] F. Abel, A. Benczúr, D. Kohlsdorf, M. Larson, the popularity of job skills in recruitment market: R. Pálovics, Recsys challenge 2016: Job recom- A multi-criteria approach, in: Proceedings of the mendations, in: Proceedings of the 10th ACM Thirty-Second AAAI Conference on Artificial InConference on Recommender Systems, RecSys ’16, telligence and Thirtieth Innovative Applications of Association for Computing Machinery, New York, Artificial Intelligence Conference and Eighth AAAI NY, USA, 2016, p. 425–426. URL: https://doi.org/ Symposium on Educational Advances in Artifi10.1145/2959100.2959207. doi:10.1145/2959100. cial Intelligence, AAAI’18/IAAI’18/EAAI’18, AAAI 2959207. Press, 2018. [29] P. K. Roy, S. S. Chowdhary, R. Bhatia, A [38] A. Gugnani, V. K. R. Kasireddy, K. Ponnalagu, Genmachine learning approach for automation erating unified candidate skill graph for career of resume recommendation system, Proce- path recommendation, in: 2018 IEEE International dia Computer Science 167 (2020) 2318–2327. Conference on Data Mining Workshops (ICDMW), URL: https://www.sciencedirect.com/science/ IEEE, 2018, pp. 328–333. article/pii/S187705092030750X. doi:https: [39] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, M. Zhou, //doi.org/10.1016/j.procs.2020.03.284, Minilm: Deep self-attention distillation for taskinternational Conference on Computational agnostic compression of pre-trained transformers, Intelligence and Data Science. Advances in Neural Information Processing Sys[30] S. Yang, M. Korayem, K. AlJadda, T. Grainger, tems 33 (2020) 5776–5788.
S. Natarajan, Combining content-based and col- [40] ESCO, Esco: European skills, competences, qualilaborative filtering for job recommendation system: ifcations and occupations, esco.ec.europa.eu, 2023. A cost-sensitive statistical relational learning ap- Accessed: 2023-08-01. proach, Knowledge-Based Systems 136 (2017) 37– [41] D. Vrandečić, M. Krötzsch, Wikidata: a free col45. URL: https://www.sciencedirect.com/science/ laborative knowledgebase, Communications of the article/pii/S095070511730374X. doi:https://doi. ACM 57 (2014) 78–85.
org/10.1016/j.knosys.2017.08.017. [42] W. Hamilton, Z. Ying, J. Leskovec, Inductive rep[31] R. Mishra, S. Rathi, Eficient and scalable job rec- resentation learning on large graphs, Advances in ommender system using collaborative filtering, in: neural information processing systems 30 (2017). ICDSMLA 2019: Proceedings of the 1st Interna- [43] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. Van tional Conference on Data Science, Machine Learn- Den Berg, I. Titov, M. Welling, Modeling relational ing and Applications, Springer, 2020, pp. 842–856. data with graph convolutional networks, in: The Se[32] E. Lacic, M. Reiter-Haas, D. Kowald, mantic Web: 15th International Conference, ESWC M. Reddy Dareddy, J. Cho, E. Lex, Using au- 2018, Heraklion, Crete, Greece, June 3–7, 2018, Protoencoders for session-based job recommendations, ceedings 15, Springer, 2018, pp. 593–607. User Modeling and User-Adapted Interaction 30 [44] M. Fey, J. E. Lenssen, Fast graph representation (2020) 617–658. learning with pytorch geometric, arXiv preprint [33] J. Zhao, J. Wang, M. Sigdel, B. Zhang, P. Hoang, arXiv:1903.02428 (2019).
M. Liu, M. Korayem, Embedding-based recom- [45] LibRecommender, Librecommender, librecommender system for job to candidate matching on mender.readthedocs.io, 2023. Accessed: 2023-08-01. scale, arXiv preprint arXiv:2107.00221 (2021). [46] W. Song, C. Shi, Z. Xiao, Z. Duan, Y. Xu, M. Zhang, [34] J. Yuan, W. Shalaby, M. Korayem, D. Lin, K. AlJadda, J. Tang, Autoint: Automatic feature interaction learning via self-attentive neural networks, in: Proceedings of the 28th ACM international conference on information and knowledge management, 2019, pp. 1161–1170. [47] X. He, K. Deng, X. Wang, Y. Li, Y. Zhang,
M. Wang, Lightgcn: Simplifying and powering graph convolution network for recommendation, 2020. arXiv:2002.02126. [48] W. L. Hamilton, R. Ying, J. Leskovec, Inductive representation learning on large graphs, 2018. arXiv:1706.02216.