In this paper we focus on constructing useful embeddings of textual information in vacancies and resumes, which we aim to incorporate as features into job to job seeker matching models alongside other features. We explain our task where noisy data from parsed resumes, heterogeneous nature of the diferent sources of data, and crosslinguality and multilinguality present domain-specific challenges. We address these challenges by fine-tuning a Siamese SentenceBERT (SBERT) model, which we call conSultantBERT, using a large-scale, real-world, and high quality dataset of over 270,000 resume-vacancy pairs labeled by our stafing consultants. We show how our fine-tuned model significantly outperforms unsupervised and supervised baselines that rely on TF-IDF-weighted feature vectors and BERT embeddings. In addition, we find our model successfully matches cross-lingual and multilingual textual content.
• Computing methodologies → Ranking; • Information systems → Content analysis and feature selection; Similarity measures; Language models. job matching, BERT, fine-tuning
Randstad is the global leader in the HR services industry. We support people and organizations in realizing their true potential by combining the power of today’s technology with our passion for people. In 2020, we helped more than two million job seekers find a meaningful job with our 236,100 clients. Randstad is active in 38 markets around the world and has top-three positions in almost half of these. In 2020, Randstad had on average 34,680 corporate employees and generated revenue of € 20.7 billion.
Each day, at Randstad, we employ industry-scale recommender systems to recommend thousands of job seekers to our clients, and the other way around; vacancies to job seekers. Our job recommender system is based on a heterogeneous collection of input data: curriculum vitaes (resumes) of job seekers, vacancy texts (job descriptions), and structured data (e.g., the location of a job seeker or vacancy). The goal of our system is to recommend the best job seekers to each open vacancy. In this paper we explore methods for constructing useful embeddings of textual information in vacancies and resumes.
The main requirements of the model are (i) to be able to operate on multiple languages at the same time, and (ii) to be used to eficiently compare a vacancy with a large dataset of available resumes.
Our end-goal is to incorporate these embeddings, or features derived from them, in a larger recommender system that combines a heterogeneous feature set, spanning, e.g., categorical features and real-valued features. 1.1
Several challenges arise when matching jobs to job seekers through textual resume and vacancy data.
First, the data we work with is inherently noisy. On the one hand, resumes are user-generated data, usually (but not always) in PDF format. It goes without saying that parsing those files to plain text can be a challenge in itself and therefore out of scope for this paper. On the other hand, vacancies are usually structured formatted text.
Second, the nature of the data difers. Most NLP research in text
similarity is based on the assumption that two pieces of information
are the same but written diferently [
Third, as a multinational corporate that operates all across the globe, developing separate models for each market and language does not scale. Therefore, a desired property of our system is multilinguality; a system that will support as many languages as possible. In addition, as it is common to have English resumes in non-English countries (e.g., in the Dutch market around 10% of resumes are in English), cross-linguality is another desired property, e.g., being able to match English resumes to Dutch vacancies.
This paper is structured as follows. First, in Section 2 we
summarize related work on the use of neural embeddings of textual
information in the recruimtent/HR domain. Next, in Section 3.1 we
describe how we leverage our internal history of job seeker
placements to create a labeled resume-vacancy pairs dataset. Then, in
Section 3.2 we describe how we fine-tune a multilingual BERT with
bi encoder structure [
Neural embeddings are widely used for content-based retrieval,
and embedding models became an essential component in the
modern recommender system pipelines [
Bian et al. propose to construct two sub-models with a co-teaching
mechanism to combine predictions of those models. The first
submodel encodes relational information of resume and vacancy, and
the second sub-model, which is related to our work, encodes textual
information in resume and vacancy. Documents are processed per
sentence, with every sentence being encoded using the CLS token
of the BERT model. The Hierarchical Transformer is applied on top
of sentence embeddings to get the document embeddings. The final
match prediction is obtained by applying a fully-connected layer
with sigmoid activation on concatenated embeddings of resume and
vacancy. Bhatia et al. propose to fine-tune BERT on the sequence
pair classification task to predict whether two job experiences
belong to one person or not. The proposed method does not require
a dataset of labeled resume-vacancy pairs. The fine-tuned model
is used to embed both the job description of the vacancy and the
current job experiences of the job seeker. Zhao et al. process words
in resumes and vacancies using word2vec embeddings and
domainspecific vocabulary. Word embeddings are fitted into a stack of
convolutional blocks of diferent kernel sizes, on top of which Zhao
et al. apply attention to get the context vector and project that into
the embedding space using the FC layer. The model is trained using
the binary cross-entropy loss on the task of predicting a match
between job seeker and vacancy. Ramanath et al. use supervised
and unsupervised embeddings for their ranking model to
recommend candidates to recruiter queries. The unsupervised method
does not use the unstructured textual data but relies on the data
stored in Linkedin Economic Graph [
Zhu et al. utilize skip-gram embeddings of various dimensionalities (64 for resume and 256 for a vacancy) to encode words, which are then passed through two convolutional layers. A pooling operation (max pooling for resume and mean pooling for vacancy) is applied to the output of convolutional layers to get the embeddings of resume and vacancy. The model is optimized using the cosine similarity loss. Qin et al. divide job postings into sections of ability requirements, and resumes into sections of experiences. The words are encoded using pre-trained embeddings and processed with biLSTM. To get the embeddings of job postings and resumes, they propose a two-step hierarchical pipeline. Every section is encoded using the attention mechanism, and finally to get embeddings of the job postings and resumes they run bi-LSTM on top of section encodings and aggregate bi-LSTM outputs using the attention mechanism. Additionally, during processing information in resumes Qin et al. propose to add encodings of job postings to emphasize skills in a resume relevant for a specific job.
We can not directly compare our work with other approaches, because of the diferent datasets used. For example, Bhatia et al . and Qin et al. assume well-structured resumes, which is not the case in our situation. Ramanath et al. builds an embedding model for recruiters’ queries which are shorter than vacancies and lack the context provided in the vacancy. Bhatia et al. propose solution when limited amount of data is available. We on the other hand work in a setting of abundant data of heterogeneous nature, but at the same time resumes lack consistent structure, while vacancies are given in a more structured way. Additional issues that are not considered by most of the reviewed works are (i) cross-linguality, so that we aim at predicting English resumes to Dutch vacancies if there is a potential match, and (ii) multilinguality, where we aim to serve a single model for multiple languages. We address both issues with our approach. Next, Qin et al. and Zhu et al. observe that an embedding model may benefit from constructing parallel pipelines to process resume and vacancy. Our approach relies on a shared embedding model of resumes and vacancies.
While there are many organizations capable to train of the shelf transformers, not many of them have the availability of an abundance of high-quality labeled data. As a global market leader, we are situated in a unique position. We have both rich histories of high-quality interactions between consultants, candidates, and vacancies, in addition to having the content to represent those candidates and vacancies. 3
Here we describe our method, more specifically, in Section 3.1 we describe how we acquire our labeled dataset of resume/vacancypairs. Next, in Section 3.2 we describe our multilingual SBERT with bi-encoder and cosine similarity score as output layer. 3.1
We have rich history of interaction between consultants (recruiters)
and job seekers (candidates). We define a positive signal any point
of contact between a job seeker and consultant (e.g., a phone call,
interview, job ofer, etc.). Negative signals are defined by job seekers
who submit their profile, but get rejected by a consultant without
any interaction (i.e., consultant looks at the job seeker’s profile,
and rejects). In addition, since we have unbalanced dataset, for
each vacancy we add random negative samples, which we draw
randomly from our job seeker pool. This is done in spirit with
other works, which also complement historical data with random
negative pairs [
Our dataset consists of 274,407 resume-vacancy pairs, out of which 126,679 are positive samples, 109,724 are negative samples
Num ber of candidat es per vacancy 4000 3000 2000 1000 0 0 20 40 60 80 100 as defined by actual recruiters, and 38,004 are randomly drawn negative samples. We have 156,256 of unique resumes and 23,080 unique vacancy texts, which implies that one vacancy can be paired with multiple resumes. Figure 1 shows the histogram of the number of resume-vacancy samples per vacancy. We see that for the majority of vacancies, we have a small number of paired resumes with approximately 10.5% of our vacancies being paired with a single resume, and approximately 30% of our vacancies being paired with at most three resumes.
resumes are user-generated PDF documents which we parse with Apache Tika.1 Overall, these parsed resumes can be considered quite noisy input to our model; there is a wide variation in format and structure of resumes, where common challenges include the ordering of diferent textual blocks, the diversity of their content (e.g., spanning any type of information from personalia, education, work experience, to hobbies), and parsing of tables and bullet points. At the same time, vacancies are usually well structured and standardized documents. They consist of on average 2,100 tokens (while resumes are on average longer and comprised of 2,500 tokens), and are roughly structured according to the following sections: job title, job description, job requirements, including required skills, job benefits, including compensation and other aspects, and company description, which usually includes information about the industry of the job ofered in the vacancy. 3.2
Our method, dubbed conSultantBERT (as it is fine-tuned using
labels provided by our consultants), utilizes the multilingual BERT [
The original formulation of the Sentence-BERT (SBERT) model takes a pair of sentences as input, with words independently encoded in every sentence using BERT, which are aggregated by pooling to get the sentence embeddings. After that, it either optimizes the regression loss, which is MSE loss between the cosine similarity score and true similarity label, or optimizes the classification loss, namely the cross-entropy loss.
Whereas SBERT is aimed at computing pairwise semantic similarity, we show that it can be applied efectively for our task of matching two heterogeneous types of data. Our fine-tuning pipeline is illustrated at the Figure 2. So we pass resume and vacancy pairs to the siamese SBERT and experimented with classification and regression objectives. 3.2.1 Document representation. Most transformer models, including SBERT, aim to model sentences as input, while we are interested in modeling documents.
We experimented with several methods for document representation by using the embeddings of the pre-trained BERT model. First, we attempted to split our input documents into sentences to encode each sentence; we tried diferent sentence representations, first we used the pooled layer output from BERT to represent each sentence, which we then average to represent the document. Here, we experimented with both simply averaging sentences, and weighted averaging (by sentence length).
Next, we tried to take the mean of the last 4 layers of the <CLS> token, which is a special token placed at the start of each sentence, and considered a suitable representation for the sentence (according to Devlin et al.). With these sentence representations, too, we represented the underlying document through averaging, both weighted and simple averaging.
Our final approach however, was more simple. We ended up treating the first 512 tokens of each input document as input for the SBERT model, ignoring sentence boundaries. To avoid trimming too much content, we pre-processed and cleaned our input documents by, e.g., removing non-textual content that resulted from parsing errors. 3.2.2 Fine-tuning method. We fine-tune our model on our 80% training data split for 5 epochs, using a batch size of 4 pairs, and mean pooling, which was found during hyperparameter tuning on our validation set as optimal parameters. To fine-tune the pretrained BERT model, we need to trim the content of every resume and vacancy to 512 tokens, as the base model limits the maximum number of tokens to 512. 4
In this section, we describe the final datasets we use for training, validation, and testing in Section 4.1, the baselines and why we employ them in Section 4.2, our proposed fine-tuned conSultantBERT approach in Section 4.3, and finally, in Section 4.4 we explain our evaluation metrics and statistical testing methodology. 4.1
As described in Section 3.1, our dataset consists of 274,407 resumevacancy pairs. We split this dataset into 80% train (219,525 samples), 10% validation and 10% test (27,441 samples each).
We use our training set to (i) fine-tune the embedding models and train the supervised random forest classifiers, the validation set for the hyperparameter search of the SBERT hyperparams described in Section 3.2.2, and report on performance on our test set. 4.2
As this paper revolves around constructing useful feature representations (i.e., embeddings) of the textual information in vacancy and resumes, we compare several approaches of generating these embeddings. 4.2.1 Unsupervised. Our first baselines rely on unsupervised feature representations. More specifically, we represent both our vacancies and resumes as either (i) TF-IDF weighted vectors (TFIDF), or (ii) pre-trained BERT embeddings (BERT).
We then compute cosine similarities between pairs of resumes
and vacancies, and consider the cosine similarity as the predicted
“matching score.” Our TF-IDF vectors have 768 dimensions, which
is equal to the dimensionality of BERT embeddings.3 We fitted our
TF-IDF weights on the training set, comprising both vacancy and
resume data. As described in Section 3.2, we rely on BERT models
pre-trained on Wikipedia from the HuggingFace library [
These unsupervised baselines help us to assess the extent to
which the vocabulary gap is problematic, i.e., if vacancies and
resumes use completely diferent words, both bag of words-based
3We experimented with increasing the dimensions of TF-IDF vectors, but it did not
gave any substantial increase in performance (increasing dimensions 3-fold improved
ROC-AUC by +0.7%).
approaches such as TF-IDF weighting and embedding
representations of these diferent words will likely show low similarity.
Formulated diferently, if word overlap or proximity between
vacancies and resumes is meaningful, these baselines would be able
to perform decently.
4.2.2 Supervised. Next, we present our two supervised baselines,
where we employ a random forest classifier that is trained on top
of the feature representation described above (TFIDF+RF, BERT+RF).
These supervised baselines are trained on our 80% train split, using
the default parameters given by the scikit-learn library [
We add these supervised methods and compare them to the previously described unsupervised baselines to further establish the extent of the aforementioned vocabulary gap, and the extent in which the heterogeneous nature of the two types of documents mentioned in Section 1.1 plays a role. That is to say, if there is a direct mapping that can be learned from words in one source of data (vacancy or resume), to the other source, supervised baselines should be able to pick this up and outperform the unsupervised baselines. 4The capitalized S in conSultant is a reference to SBERT. #
Model 1. BERT+Cosine 2. TFIDF+Cosine 3. BERT+RF 4. TFIDF+RF 5. conSultantBERTClassifier+Cosine 0.7474 6. conSultantBERTClassifier+RF 0.8366∗ 7. conSultantBERTRegressor+Cosine 8. conSultantBERTRegressor+RF
Our intuition for using the classification objective is our task and dataset; which consists out of binary class labels, making the classiifcation objective the most obvious choice. At the same time, our work revolves around searching for meaningful embeddings, not necessarily solving the end-task in the best possible way, for which, we hypothesize, the model may benefit from the more fine-grained information that is present in the cosine similarity optimization metric.
Similarly to our baselines, we consider (i) our fine-tuned model’s direct output layer (i.e., the cosine similarity output layer) as “matching score,” between a candidate and a vacancy (conSultantBERTRegressor+Cosine, conSultantBERTClassifier+Cosine), in addition to the predictions made by a supervised random forest classiifer which we train on the embedding layer’s output (see Figure 2), yielding the following supervised models: conSultantBERTRegressor+RF and conSultantBERTClassifier+RF. We explore the latter since we aim to incorporate our model in a production system alongside other models and features. 4.4
In order to compare our diferent methods and baselines, we turn to standard evaluation metrics for classification problems, namely we consider ROC-AUC as our main metric, as it is insensitive to thresholding and scale invariant. In addition, we turn to macroaveraged (since our dataset is pretty balanced) precision, recall, and F1 scores for a deeper understanding of the specific behaviors of the diferent methods.
Finally, we perform independent student’s -tests for statistical significance testing, and set the -level at 0.01. 5
See Table 1 for the results of our baselines and fine-tuned model. We first turn to our main evaluation metric ROC-AUC below, and next to the precision and recall scores in Section 5.2. 5.1
Overall performance 5.1.1 Baselines. As expected, using BERT embeddings or TF-IDF vectors in an unsupervised manner, i.e., by computing cosine similarity as a matching score, does not perform well. This confirms our observations about the challenging nature of our domain; word overlap or semantic proximity/similarity does not seem a suficiently strong signal for identifying which resumes match a given vacancy.
At row 3 and 4 we show the methods where a supervised
classifier is trained using the input representations described above
(TFIDF+RF and BERT+RF). Here, we see that with a supervised
classifier, both TF-IDF-weighted vectors as feature representation and
pre-trained BERT embeddings vastly improve over the
unsupervised baselines, with a +31.0% improvement for BERT, and a +25.2%
improvement for the TF-IDF-weighted vectors. This suggests to
some extent, a mapping can be learned between the two types of
documents, as we hypothesized in Section 4.2.
5.1.2 conSultantBERT. Next, we turn to our consultantBERT runs
in rows 5 through 8. First, we consider models trained with the
classification objective, in rows 5 and 6. We note that fine-tuning
BERT brings huge gains over the results the unsupervised
pretrained embeddings (BERT+Cosine), which is in line with previous
ifndings [
As our primary goal is to get high quality embeddings for the down-stream application of generating useful feature representations for recommendation, alongside of which we can train whatever model we want with additional features types such as categorical, binary, or real-valued features, we are more interested in the former unsupervised, rather than the latter approach with random forest classifier.
Finally, we turn to our fine-tuned conSultantBERT with the regression objective (conSultantBERTRegressor), to study the direct cosine similarity optimization objective. Looking at the last two rows, we find that conSultantBERT with regression objective performs similarly to conSultantBERT trained with the classification objective in supervised approach with a random forest classifier on top. In fact, the small diference turns out not to be statistically significant with a -value of 0.2.
What stands out most, is that the approach with cosine similarity outperforms all other runs, including significantly outperforming the conSultantBERTClassifier+RF approach. We explain this difference by the fact the architecture with the classifier objective having a learnable layer (the Softmax classifier in Figure 2). We drop this layer after the fine-tuning phase to yield our embeddings, losing information in the process. On the other hand, the architecture with the regression objective’s had a non-learnable last layer, namely simply cosine similarity, which means all the necessary information has to propagate to the embedding layer. 5.2
When we zoom into the more fine-grained precision and recall scores, we observe the following: First, our unsupervised baselines in row 1 and 2 show how TFIDF-based cosine similarity yields substantially higher precision compared to cosine similarity with pre-trained BERT embeddings, keeping similar recall.
Next, these same baselines with a supervised random forest classifier on top (row 3 and 4) show that in both cases the classifiers seem to perform somewhat similarly, irrespective of whether it is trained with TFIDF-weighted feature vectors or pre-trained BERT embeddings. With only a slight increase across precision and recall (around +2.5%) for the TFIDF-based method, we conclude the feature space in itself does not provide substantially diferent signals for separating the classes.
Then, comparing our fine-tuned models, rows 5 and 6 shows the SBERT model fine-tuned with the classification objective yields marginally higher precision compared to the random forest classiifer trained on the SBERT’s output layer. Row 7 and 8 show similar stable improvements across the board compared to the methods in rows 1–4.
Overall, we conclude that all of the supervised approaches (row 3 through 8) show roughly similar behavior with precision and recall balanced and substantial improvements over the unsupervised baselines. 6
After analyzing the results in the previous section, in this section we take a closer look at the diferent methods, by studying the distributions of prediction scores across the two labels (positive and negative matches) in Section 6.1, and we zoom into another desired property of our embedding space: multilinguality (in Section 6.2) 6.1
In Figure 3 we plot the distributions of predicted scores per run, split out for positive and negative labels. The predicted scores correspond to cosine similarities in the case of TFIDF, BERT, conSultantBERT*, and probabilities of the random forest classifier in the case of TFIDF+RF, BERT+RF, and conSultantBERT*+RF.
A few things stand out, first, it is clear that fine-tuning the BERT embeddings in our conSultantBERT models yield a clearer separation in the predictions per class. The left-most plots (TFIDF, both with Cosine similarities) show largely overlapping distributions. The second column of plots (denoted BERT) shows largely similar
Ik ben de afgelopen 10 jaar leraar geweest 0 0 0 0 r e k r o w c iit s g o l d e t n e lt a a fr o g n i k o o l e r a e W 0 0 0 0 tr o a c u d e d e t n e lt a a fr o g n i k o o l e r a e W 0 0 0 0 r e k r e w e d e m k e iit s g o l e d r e e t n e lt a e g n e e r a a n k e o z p o n ij z e W 0 0 0 0 t n e c o d e d r e e t n e lt a e g n e e r a a n k e o z p o n ij z e W patterns; with as main diference the unsupervised BERT having a bias towards higher scores compared to the unsupervised TFIDF.
At the same time, while the distributions of scores for each label from the classifier with TF-IDF weighted vectors or pre-trained BERT embeddings do seem rather close, the model does succeed in separating the classes more often than not (as can be seen by the score improvements over the non-supervised baselines in Table 1).
Observing the latter, though, makes clear that unsupervised similarity scores using default embeddings or TF-IDF weights does not allow a strong signal to separate both classes, which can be witnessed by the largely overlapping score distributions.
We observed the diference in performance between our regressionoptimized conSultantBERT with the classification-optimized one, in Section 5.1. We now turn to comparing the prediction distributions of both models. We note how the classification-optimized conSultantBERT (top row, third plot from the left) seems to yield less separable cosine similarity scores compared to the regression-optimized one (top right plot). Compared to the three other conSultantBERT approaches, the former stands out. The random forest classifier trained on top of the embeddings, though, efectively learns again to separate between the classes, suggesting the embeddings keep distinguishing information.
As a multinational corporate that operates all across the globe, developing a model per language is not scalable in our case. Therefore, a desired property of our system is multilinguality; a system that will support as many languages as possible.
In some of the countries we operate, there is a high percentage of
job seekers that are not native to that country. For example, many
of the job descriptions in the Netherlands are in Dutch, however
around 10% of the resumes are in English. Due to that fact another
expected property from our solution is to be cross-lingual [
Classic text models, like TF-IDF and Word2vec, capture information within one language, but hardly connect between languages. Simply put, even if trained on multiple languages each language will have its own cluster in space. So “logistics” in English and “logistiek” in Dutch are embedded in a completely diferent point in space, even though the meaning is the same.
Furthermore, we know that the language of resume correlates
with nationality and therefore can be a proxy discriminator. Due to
the impact of these systems and the risks of unintended algorithmic
bias and discrimination, HR is marked as a high risk domain in
the recently published EC Artificial Intelligence Act [
In Figure 4 we see few examples for sentences that a candidate might write “I worked ...” and few examples for vacancy “We are looking for ...”. In order to demonstrate cross lingual and multilingual the same examples are written in both English and Dutch.
On the left hand side of Figure 4 (TFIDF) we can see that there is no match at all. This is due to the vocabulary gap, so by definition TFIDF can not match between “warehouse” and “logistic worker”.
To solve the vocabulary gap we introduced BERT (center of Figure 4), which we see does find similarity between candidate and vacancy sentences. However, it also hardly separates between the positive and negative pairs. Moreover, we can see a slight clustering around languages, so Dutch sentences have comparitively higher similarity to Dutch sentences, and likewise English sentences are more similar to each other.
On the right hand side of Figure 4 (conSultantBERTRegressor) we observe that the vocabulary gap is bridged, cross-lingual sentences are paired properly (e.g “I worked in a warehouse for 10 years” and “We zijn op zoek naar een getalenteerde logistiek medewerker” have high score), and finally, both Dutch to Dutch and English to English sentences are properly scored, too, thus achieving our desired property of multilinguality. 7
In this work we experimented with various ways to construct embeddings of resume and vacancy texts.
We propose to fine-tune the BERT model using the Siamese SBERT framework on our large real-world dataset with high quality labels for resume-vacancy matches derived from our consultants’ decisions. We show our model beat our unsupervised and supervised baselines based on TF-IDF features and pre-trained BERT embeddings. Furthermore, we show it can be applied for multilingual (e.g., English-to-English alongside Dutch-to-Dutch) and crosslingual matching (e.g., English-to-Dutch and vice versa). Finally, we show that using a regression objective to optimize for cosine similarity yields more useful embeddings in our scenario, where we aim to apply the learned embeddings as feature representation in a broader job recommender system.
Special thanks to the rest of the SmartMatch team: Adam, Evelien, Najeeb, Sandra, Wilco, Wojciech, and Zeki. And Sepideh for her helpful comments.