The transformative power of artificial intelligence is revolutionizing the talent acquisition domain. Automatic job recommendation systems are emerging as a key component of this transformation. This study presents a new multilingual job recommendation solution that leverages combined unsupervised and contrastive learning to efectively model the semantic similarity between job titles across 11 languages. Our approach pre-trains a multilingual encoder using unsupervised learning on co-occurrence information of skills and job titles, followed by fine-tuning via contrastive learning on a dataset of similar and dissimilar job pairs based on the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy. This sequential learning strategy significantly enhances representation quality. Our novel multilingual job title encoder achieves strong ranking results across all languages, with 4.3% improvement in mean Average Precision (mAP) for English compared to previous state-of-the-art monolingual solutions. The proposed method also ofers very good cross-lingual capabilities, enabling the ranking of jobs in diferent languages with improved alignment and uniformity properties in the representation space.
increased competitive pressure to find and retain the best
suitable skilled candidates [1]. The introduction of
Information and Communication Technologies (ICTs) in
Human Resources (HR) processes in recent years, and
especially with more recent advancements in artificial
intelligence (AI) has helped in alleviating this pressure
in time-sensitive recruitment processes, especially those
that have relied heavily on human specialists [2]. The
integration of AI and machine learning has been
transforming the recruitment process by allowing companies
to rapidly process and analyze large amounts of candidate
and supporting data-driven decisions to optimize and
refine recruitment strategies [
Workshop Proce dings htp:/ceur-ws.org ISN1613-073
CEUR
Workshop Proceedings (CEUR-WS.org) manager gerente directour engineer
Cosine Similarity
Loss Function Train Job Title Encoder (Multilingual) Stage 1
Trained weights
Fine-tune Job Title Encoder (Multilingual)
Stage 2
Loss Function
Normalized Temperature-scaled Cross Entropy
Job Title Encoder
Text Ranking software engineer application programmer animal osteopath textile inspector to build a multilingual Job Title Encoder for intelligent job recommendation. The process starts from a pre-trained encoder, and we propose two steps: 1) Unsupervised training: we define target representations ( 1) for every job based on jobs into these representations. 2) Contrastive training: we use synonyms and relations from ESCO ( 2) to fine-tune the encoder with contrastive learning techniques. tion systems are increasingly relying on this task to develop matching algorithms that accurately pair job vacancies with candidates based on their skills and experience [5, 6, 7]. Novel STS approaches leverage neural networks to automatically generate vector representations that efectively capture the semantic nuances of text elements, thus significantly enhancing matching performance compared to traditional methods [8].
CEUR
ceur-ws.org
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License broadly divided into two approaches: supervised methlabeled data. Examples of unsupervised procedures ap- ters [17, 18]. However, these approaches sufer from data plied to the HR domain were proposed by Zbib et al. [6] sparsity and the cold start problem, especially when makand Lavi et al. [9]. They do not rely on human anno- ing recommendations in the case where a user has few tations, and instead, they use noisy data automatically interactions with other items [19]. collected from resumes and job listings. Conversely, su- Recent advancements in this field leverage techniques pervised methods rely on large amounts of labelled data from machine learning, natural language processing to train, which can be sometimes hard to obtain in terms (NLP), and AI. The incorporation of content-based emof quality and quantity [10]. Contrastive learning has bedding strategies in recommendation systems has led shown very strong results for STS as a technique that to the development of eficient and reliable item retrieval helps improve the learned representations and increase methods, ofering significant benefits in terms of generthe training data eficiency [ 11, 12, 13]. alization, scalability, and cold start capabilities [20].
Our paper builds upon these advances and presents a multi- and cross-lingual intelligent job recommenda- 2.1. Content-based Job Recommender tion solution that relies on STS to measure the similarity between job titles across 11 languages (English, Ger- Systems man, French, Spanish, Italian, Dutch, Portuguese, Polish, In the existing literature, numerous studies on Japanese, Chinese, and Korean). We introduce a novel embedding-based recommender systems are based two-stage methodology for training an encoder focused on Deep Learning techniques. For instance, Zhao et on job title semantic similarity for text ranking (see Fig- al. [20] proposed a Convolutional Neural Network ure 1). (CNN) with an attention layer as a key component in
In the first stage we pre-train a multilingual encoder their recommendation system to encode job titles, skills, following an unsupervised approach using noisy data and other context-aware elements into a dense-vector automatically collected from job postings and resumes. representation for ranking candidates with job vacancies. In the second stage we fine-tune the encoder via con- In [7], authors propose a method to train a job title trastive learning on a dataset with pairs of relevant and encoder in which they fine-tune a pre-trained language non-relevant jobs based on the European Skills, Compe- model with an auxiliary classification loss, using job-skill tences, Qualifications and Occupations taxonomy (ESCO) co-occurrence statistics extracted from vacancies. Zbib [14, 15]. The ESCO occupations taxonomy is a hierar- et al. [6] first build dense-vector representations for jobs chically structured standard classification that organizes based on their distribution of skills (also extracted from jobs into groups and concepts based on their similarity vacancies), which are expected to encode the semantics in terms of skill level and required specialization. For of the job. Then, they train the encoder to map job title instance, ESCOXLM-R [16] work has demonstrated the surface forms to such representations. At inference time, efectiveness of using the ESCO taxonomy for adapting a the encoder only sees the surface form of (potentially multilingual language model to the job market domain. new) jobs, and cosine similarity is used to measure the
We study the efectiveness of the proposed methodol- semantic distance between encoded occupations. These ogy by training various encoders with diferent backbone approaches are based on the premise that skills are esarchitectures (Transformer-based vs. CNN-based) and sential to understand the meaning of a job, and therefore, original pre-training objectives (semantic-similarity vs. semantically similar job titles should exhibit similar skill masked language modeling). We find that trained en- profiles. coders outperform state-of-the-art benchmarks on the Similarly, [21] develops a novel approach to train an job title ranking task. Moreover, the proposed solution encoder derived from BERT [22] to link skills with occushows very good multi- and cross-lingual capabilities, pations by leveraging co-occurrence information via a achieving ranking performance results for each language contrastive learning method. that are remarkably close to English, the language with the best results. Notably, the encoders exhibit strong 2.2. Contrastive Learning methods for alignment and uniformity properties in cross-lingual evaluations. Semantic Textual Similarity
2. Related Methods powerful technique for learning high-quality sentence embeddings. This technique was initially popularized in Job recommendation systems are a vital component of computer vision [23]. It focuses on learning representamodern employment platforms that help match job seek- tions by distinguishing positive pairs (similar data points) ers with relevant opportunities. Traditional systems against negative pairs (dissimilar data points). Models are based on matrix factorization and collaborative fil- are trained to bring the representations of positive pairs caltoivseerptaoigrsetfhaertrhwerhialpeaprutsihninthgetehmerbeepdrdeisnegntsaptaiocnes[2o4f,n2e5g]-. NJooibssy+(SEkNil)ls PDVo-Dc2BVOeWc acmecnaognuianngteaeenrrt
Contrastive learning techniques can help alleviate the amnainsocteroinpySTpSrotabslekms[a2n6,d1i1m, 1p3ro].vTehreeparneisseontrtoatpiyonprpoebrlfeomr- T(Jr1ao0nbslTlaainttilgoe.sn)s rmeMpurleatispleinanugtaxutiailolianjroyfbosr dmgeirneaergnceiatnnogteueeerrr refers to the issue where the vector representations do not uniformly occupy the embedding space. Instead, they Figure 2: Creation of Multilingual Dataset from Noisy Skills might cluster in a narrow cone of the space, leading to an (ing1)pfroorceUsnssfuopre rv1iseisdstirmaiinlainrgtoofthJoeboTnietledeEpniccotedderi.nT[h6e].builduneven distribution that diminishes the model’s ability to distinguish between diferent data points [ 27, 28].
The adaptation of contrastive learning to NLP has led to the development of several solutions in the context of the multilingual encoder obtained in the previous step. STS for measuring how closely the meaning of two sen- In this case, we use pairs of similar and dissimilar jobs tences align. Works in this area often rely on pre-trained derived from the ESCO taxonomy. language models, such as BERT [22] or RoBERTa [29], as Below we describe how the data used in the diferent the backbone for generating initial sentence embeddings stages of this approach was prepared. Then, we describe [30]. As in SimCSE [11] or ConSERT [31], contrastive the procedure for training the neural network in each learning is then applied to fine-tune these embeddings, stage to build the Multilingual Job Title Encoder for improving their ability to capture semantic similarity, in automatic job title ranking.
STS tasks.
The design of the loss function is crucial in contrastive 3.1. Data Preparation learning, and has undergone a significant evolution from Siamese pair loss [32], triplet loss [33], to alternatives with multiple negatives such as N-pair loss function [34], or NT-XENT (Normalized Temperature-scaled Cross Entropy) [23], which can lead to improved representation learning [11].
In this work, we leverage the relationship between jobs and skills to build auxiliary representations that later guide the training of the multi- and cross-lingual job title encoder. Then, we fine-tune the encoder using a contrastive learning loss with multiple negatives to improve matching and text ranking between candidates and job vacancies.
We construct 1, a multilingual dataset of job positions by linking the surface form of job occupations to a dense-vector representation. These representations are based on the distribution of noisy skills related to each job title collected from a collection of job postings and the work experience sections of anonymized resumes. In this dataset we include data from 11 diferent languages: English (en), German (de), Spanish (es), French (fr), Italian (it), Japanese (ja), Korean (ko), Dutch (nl), Portuguese (pt), Polish (pl), and Chinese (zh). Bear in mind that we 3. Our Approach do not require any manual annotation for building this dataset (unsupervised).
In this work, we propose a novel approach for training The other dataset is 2, which we compile as a cola semantic textual similarity solution for intelligent job lection of multilingual pairs of similar and dissimilar recommendation. Then, we describe how we build an job titles, based on relations extracted from the ESCO encoder model for multilingual job title ranking that taxonomy. This dataset includes the same languages as operates across 11 languages. the ones described before, except for the three Asian
Our main contribution is the two-step training pro- languages (Japanese, Korean, and Chinese) since they cedure depicted in Figure 1. In Step 1, similarly to the are not included in ESCO. Our experiments show that process presented in [6], we use doc2vec [35] and the the contrastive learning step benefits these languages skills distribution for every job title to create auxiliary through cross-lingual transfer despite not being included dense-vector representations. These are in turn used to in the training data of that stage. train an encoder that maps the surface form of each job title to its reference doc2vec representation. Since auxil- 3.1.1. Job Title Similarity from Noisy Skills Dataset iary representations encode language-agnostic semantics, we expose the encoder to job surface forms in various The 1 dataset described above is used to learn the selanguages, all mapped to the same point in the representa- mantic relations between job occupations based on the tion space, resulting in a multi- and cross-lingual encoder. co-occurrence distribution of skills for each job title. The Then, in Step 2, we use contrastive learning to fine-tune knowledge provided by the skills is transferred to an ebnetcwodeeern, ethneasbeli njogbuosctcoucpaaptitounres tbhaesesdemoanntthiceisrims uilrafraictye 0 - 1 - 2 - 6 7 - 8 - 9 forms. SGurobu-mpajor 61 - 62 63
The process of creating this dataset consists of two Minor Group 621 - 622 steps. In the first one, we start from a collection of job titles and associated noisy skills for each occupation. We Unit Group 6221 - 6222 - 6223 6224 consider these skills as noisy since they are retrieved in Occupation 6223.1 + 6223.2 + agnlisahujtoobmpaotsetdinwgsayanbdyasnimonpylmesizteridnrgesmuamtcehs.inTghefrdoamtaEsent-, NOaccrruopwation 6223.2.1 + sourced from proprietary real data, covers a broad range Figure 3: ESCO groups for job occupations. For creating the of industries, including manufacturing, healthcare, tech, positive pairs, we do all possible combinations of synonyms ifnance, or banking, among others. Next, we follow a for each occupation concept. Also, we retrieve positive pairs similar procedure to that presented in Zbib et al. [6], us- from parent occupations and from other occupations from ing PV-DBOW Doc2vec to create a dataset with pairs the same Unit Group. Negatives are retrieved from concepts of job titles and fixed-length vectors (512 dimensions). outside of the selected for the positives ones. These vectors are the auxiliary representations that will be used as synthetic targets in the first training stage. We refer to the resulting dataset as 1 . minor, sub-major and major groups, based on their sim
As shown in Figure 2, the next step is to expand the ilarity in terms of the skill specialization and skill level pairs of job titles and fixed-length vectors to the other required for the job. ESCO therefore includes occupalanguages. To do this, we translate a subset of the English tions located at level 5 and lower of this classification job titles (starting with around 2.3 Million) to the target structure. All occupation concepts contain one preferred languages using Machine Translation1. Afterwards, we label and a number of alternative labels. include in the previous dataset new pairs with the surface We use these hierarchically structured groups of oclexical form of the resulting translations, but with the cupation concepts to extract a collection of multilingual same vector representation as the corresponding source (positive and negatives) job title pairs (see Figure 3). English. We create the positive pairs by including all possible
The same job title in diferent languages will have the combinations without repetition of the preferred and alsame auxiliary representation (Eg. manager (en), gerente ternative labels of each occupation concept. In addition, (es), and directeur (fr) will share the same doc2vec vector). to ensure lexical diversity among similar jobs, we generBy this, we aim to foster the multi- and cross-lingual ate a small subset of positive pairs by retrieving labels capabilities of the proposed solution for text retrieval in from parent occupations (30%) only if the concept is a Human Resource applications based on job title semantic narrow occupation. Finally, we retrieve more positive similarity. We refer to this multilingual dataset as 1. pairs from other occupations of the same Unit Group The final dataset has 14.5 Million English job titles and (10%2). an average of 2.3 Million job occupations per each of the Regarding the negative pairs, we retrieve the samples other languages. for each occupation concept from labels of a diferent Major Group (50%), from another Sub-major Group (25%), 3.1.2. Similarity Pairs from ESCO Dataset from another Minor Group (15%), and from another Unit Secondly, we prepare the 2 dataset used for further Group (10%). ifne-tuning the encoder using multilingual pairs (English, The process described above is performed for each German, French, Spanish, Italian, Dutch, Portuguese, and language independently, and once more to create crossPolish) of similar and dissimilar job titles, aiming to ex- lingual pairs selecting job occupations from any language. tract relations between job occupations that can help The final dataset includes around 136 K of diferent job improve the matching performance for automatic job occupations, detailed into around 1 M diferent positive recommendations. pairs (anchor + positive elements) and about 128 negative
We build this collection of pairs using the ESCO Tax- samples per unique job title.
onomy (version 1.2.0). The ESCO occupations taxonomy
is built on ISCO-08 [
accountant tecnico del eluci technicien lumière técnicade restauração abgeordnete directorade hotel gerentede alojamiento treedoctor kierownikds operacyjnych huishoudhulp clerical assistant 3.2.1. Unsupervised training from Noisy Skills
representations the batch size.
First, we train the multilingual encoder to align the output representation for each job occupation with the doc2vec auxiliary representation generated in 1
. With this aim, we train the encoder receiving as inputs pairs of job title ( ) and fixed-length auxiliary embedding ( ) from 1. In Equation 1 we show the cosine similarity function used for training the model based on the representation generated from distribution of noisy skills related to every job, where ℎ refers to the output of the model after processing the surface text form , and to
1 =1 ℒ = − ∑ (ℎ , )
(1) 3.2.2. Contrastive Learning from Similarity Pairs
coder following a contrastive method using a NT-Xent loss function without in-batch negatives (see Equation 3).
sample, which helps the model to better discriminate between similar and dissimilar examples, and provides more stability during training (see Figure 4).
In the following ( , that are fed as input to the +, ,1.. ) represents the tuples − model during training. refers to the anchor element, + to the positive sample, − and ,1.. to each negative element. On the other hand, refers to the output embedding of the model for a particular input, resulting in ( , +, ,1.. ). stands for the − number of negatives per anchor element, is equal to the batch size, and is the temperature value used in the function. To compare vectors, we use the cosine similarity ( ) function.
(, ) = (,)/ ℒ = − ∑ log ( 1 =1 ( , +)
( , +) +∑=1 ( , ,−) ) (2) (3)
to push away the representation of the negative pairs with respect to the anchor and positive element. During training, the tuples of inputs are built dynamically from the collection of positive and negative pairs of 2.
In this Section, we describe the implementation details and report on results obtained from experiments using the proposed methodology for training a multilingual encoder to develop an intelligent job occupation recommendation solution. First, we introduce the Multilingual Language Models selected to study the proposed approach, these architectures are studied in terms of their eficiency (performance on job title ranking vs. model size).
for training the model with the introduced solution to assess the benefits that each stage provides. Finally, we analyze the performance of the trained encoder for crosslingual job title ranking.
To evaluate the matching performance of the proposed method, we use the English job titles similarity dataset3 from Zbib et al. [6]. This set includes more than 2,500 job occupations distributed between query and corpus document elements. Each pair of query-document is labeled for binary relevance. We replicate this dataset for the other languages by using Human Translation (de, fr, ja, and zh) or Machine Translation (for the remaining languages). These datasets are publicly available at GitHub4.
sion (mAP) of the output of the ranked lists. We use the
trec_eval software library to compute this metric 5.
Architecture approach using the selected architectures. We start with
Pm.UMSuEl-tCi.NMNP[N3e7t][
We observe that the best results for European languages (de, es, fr, it, nl, pl, pt) are obtained with the 4.1. Text Ranking with Multilingual ifne-tuned Transformer-based architectures. Also, the Encoder XLM-RoBERTa architecture obtains the best results on the Asian languages (ja, ko, zh). We select this architecWe perform an analysis of state-of-the-art Multilingual ture as the best candidate for building the multilingual Language Models by testing them on text ranking for job intelligent job recommendation system. title recommendation. In particular, we selected three We also find that the eficient mUSE-CNN encoder candidate architectures based on their size in terms of pa- delivers competitive results with significantly lower comrameters and their zero-shot performance on the ranking putational costs, making it an optimal choice for deploytask. ment in resource-limited environments. Refer to Ap
It is worth noting that in this paper, our aim is not pendix A for details on the configuration for training the solely to identify the base model with the best results, but selected encoder architectures. also to study the efectiveness of models in terms of their size and computational cost, with the goal of providing 4.2. Ablation Analysis the practitioner with a practical guide for accuracy vs. eficiency trade-ofs. In the ablation analysis we examine the contribution of
With that in mind, we select two Transformer-based every stage of the proposed approach to the final ranking
architectures: XLM-RoBERTa [
On the other hand, we choose the Multilingual Uni- architecture directly on the second training stage (see
versal Sentence Encoder model CNN-based (mUSE-CNN ) §3.2.2) using only contrastive learning on similar and
[
Table 1 shows the results obtained with our proposed Also, note the diference for ranking performance in Korean for the Paraphrase Multilingual MPNet architecture. This is because this architecture without fine-tuning ofers very poor results for this language compared to the other alternatives (with mAP values below 0.05). mUSE-CNN P. Multi. MPNet XLM-RoBERTa mUSE-CNN P. Multi. MPNet XLM-RoBERTa mUSE-CNN P. Multi. MPNet XLM-RoBERTa
Fine-Tuning Unsup. Contr.
1 7 1 7 1 7 7 2 7 2 7 2 1 2 1 2 1 2
Finally, the combination of both stages sequen- Architecture Fine-tuned tially leads to the best mAP values. For the mUSE-CNN, mUSE-CNN 7 we observe an average improvement of 4.29% with re- PX.LMMu-Rltoi.BMERPTNaet 77 spect to results from Stage 1. Also, improvements of 9.01% mUSE-CNN 1 + 2 and 7.18% are observed for the Paraphrase Multilingual P. Multi. MPNet 1 + 2 MPNet v2 and XLM-RoBERTa architectures respectively. XLM-RoBERTa 1 + 2 We remark that, after this stage, Transformer-based ar- Table 3 chitectures surpass the eficient mUSE-CNN for job title Cross-lingual Evaluation. Each column represents a test set ranking for up to 1.3 points on average. indicating the language of the queries and corpus elements
To evaluate the statistical significance of the improve- respectively. For example, in the en-de set, the queries are in
ments achieved by the two-stage approach compared to English and corpus documents in German.
applying only the first-stage method, we employed the
Wilcoxon signed-rank test [
It is relevant to note that the contrastive learning stage of our solution. We significantly improve the ranking brings gains for ranking in European languages (which performance for German and Chinese with respect to are included in the dataset used for training - 2). How- English compared to the architectures without any fineever, all Asian languages also benefit from this training tuning. For example, for the encoder based on XLMprocedure, as the cross-lingual capabilities of these archi- RoBERTa after fine-tuning, we improve mAP when testtectures (see §4.3) can contribute in improving ranking ing using Chinese queries and English corpus samples on languages not seen during this training stage. from only 0.0414 for up to 0.7226. Our cross-lingual
In short, training the models for semantic-similarity analysis reveals improved performance on non-English based on the skills of each job occupation leads to a rea- languages compared to monolingual results in Table 2. sonable solution for multilingual job title ranking. In This can be attributed to two key factors: the better repreaddition, fine-tuning this alternative using relations of sentation quality for English job titles and the encoder’s job titles from the ESCO Taxonomy furthers adjust the ability to align representations of job occupations across embedding space to improve the ranking performance languages. of the final solution on each language. This shows that In Figure 5, we investigate the representation quality both stages of our proposed method are beneficial. of encoders trained on cross-lingual occupation pairs, consisting of pairs of the same job title in two difer4.3. Cross-lingual Evaluation ent languages. For simplicity, we focus on a subset of languages and on two of the presented encoders. To Finally, we conducted a cross-lingual evaluation of the evaluate this, we employ the alignment (ℒ ) and proposed method on the selected architectures. For this uniformity (ℒ ) metrics introduced by Wang et purpose, we created several datasets where the language al. [26]. The alignment metric evaluates how close the of the query elements difered from that of the corpus. features from positive pairs (i.e., same occupation in difIn particular, we studied the ranking performance when ferent languages) are positioned in the feature space. On English was fixed as one of the languages and assessed the other hand, the uniformity metric measures how
evenly the normalized features are distributed over the unit hypersphere, allowing for the analysis of features density. Refer to Wang et al. [26] for a detailed definition In this paper we presented a methodology for developof these metrics. ing a solution for intelligent multilingual job occupation
A good encoder is one with very good alignment (fea- recommendation, resulting in a job title encoder model
tures of the same occupation in diferent languages are for text ranking based on semantic similarity across 11
very close), and good uniformity (otherwise, the learned languages.
representations would occupy a narrow cone in the vec- This method optimizes the neural network
architector space, which severely limits their expressiveness [
Figure 5-left shows that the XLM-RoBERTa without any similarity based on the distribution of skills linked to ifne tuning (pre-trained on masked language modeling) each job. This stage provides large insights to the trained ofers very good alignment. However, the uniformity is model about the semantic similarity of the diferent mulvery poor, which means that the vector space is quite tilingual occupations, resulting in a good solution for job compressed and limited when encoding job occupations. title ranking.
However, after our fine-tuning, the uniformity is signif- Secondly, we take advantage of the ESCO taxonomy icantly improved from 0 to around -3.4 at the expense and its hierarchically structured classification of jobs of slightly worse alignment, which results in an encoder based on their similarity in terms of skill level and skill that is very competitive at recommending cross-lingual specialization, to extract similar and dissimilar pairs of jobs, ofering mAP values for up to 0.72 points. job occupations. This information together with con
Figure 5-right shows the same analysis for the mUSE- trastive learning techniques helps to further optimize the CNN encoder. In this case, the model without fine-tuning embedding representation obtained with the models, and ofers reasonable uniformity with low alignment. Both can improve the average mAP for ranking for up to 9% properties are significantly improved after our training with respect to only carrying out the first training stage. procedure, resulting in an encoder that is better at rank- To the best of our knowledge, the contrastive learning cross-lingual occupations. Note, too, that when com- ing stage introduced in this study contributes to surpass paring with the XLM-RoBERTa trained model, better job title ranking performance in English by 4.3% over alignment values lead to better mAP values. state-of-the-art monolingual approaches. Finally, we observe that the presented solution exhibits remarkable cross-lingual capabilities, enabling the ranking of jobs in diferent languages, ofering good alignment and uniformity properties.
Our work presents limitations that open promising job titles similarity from noisy skill labels, 2022. directions for future research. In particular, our solution arXiv:2207.00494. does not leverage contextual information, which might [7] J. Decorte, J. V. Hautte, T. Demeester, C. Develder, be essential for a comprehensive understanding of job Jobbert: Understanding job titles through skills, requirements. As a result, it may struggle to provide CoRR abs/2109.09605 (2021). arXiv:2109.09605. more tailored job recommendations, particularly for job [8] D. Chandrasekaran, V. Mago, Evolution of semantic seekers with diverse backgrounds and experiences. Fur- similarity - A survey, ACM Comput. Surv. 54 (2022) thermore, the lack of contextual information may lead to 41:1–41:37. a narrow focus on job titles, which may not accurately [9] D. Lavi, V. Medentsiy, D. Graus, consultantbert: reflect the complexities of modern job markets. Includ- Fine-tuned siamese sentence-bert for matching jobs ing information such as skills, job descriptions, career and job seekers, in: M. Kaya, T. Bogers, D. Graus, path, or education could significantly enhance the solu- K. Verbert, F. Gutiérrez (Eds.), Proceedings of the tion’s ability to provide personalized and relevant job Workshop on Recommender Systems for Human recommendations. Resources (RecSys in HR), volume 2967 of CEUR
In future work, we aim to address these limitations by Workshop Proceedings, CEUR-WS.org, 2021. focusing on a key direction. We will study the potential [10] P. Neculoiu, M. Versteegh, M. Rotaru, Learning benefits of incorporating skills and job descriptions into text similarity with siamese recurrent networks, in: the encoder, developing novel techniques to represent P. Blunsom, K. Cho, S. B. Cohen, E. Grefenstette, and combine these contextual features, with the focus in K. M. Hermann, L. Rimell, J. Weston, S. W. Yih (Eds.), improving the overall quality of the job recommendation Proceedings of the 1st Workshop on Representation system. Learning for NLP, Association for Computational Linguistics, 2016, pp. 148–157. [11] T. Gao, X. Yao, D. Chen, Simcse: Simple contrastive References learning of sentence embeddings, in: M. Moens, X. Huang, L. Specia, S. W. Yih (Eds.), Proceedings [1] S. Böhm, O. Linnyk, J. Kohl, T. Weber, I. Teetz, of the 2021 Conference on Empirical Methods in K. Bandurka, M. Kersting, Analysing gender bias in Natural Language Processing, EMNLP, Association IT job postings: A pre-study based on samples from for Computational Linguistics, 2021, pp. 6894–6910. the german job market, in: S. Laumer, J. L. Quesen- [12] H. Qiu, W. Ding, P. Chen, Contrastive learning of berry, D. Joseph, C. Maier, D. Beimborn, S. C. Sri- sentence representations, in: S. Bandyopadhyay, vastava (Eds.), SIGMIS-CPR 2020, Computers and S. L. Devi, P. Bhattacharyya (Eds.), Proceedings of People Research Conference, ACM, 2020, pp. 72–80. the 18th International Conference on Natural Lan[2] Z. Tasheva, V. Karpovich, Transformation of re- guage Processing ICON, NLP Association of India cruitment process through implementation of ai (NLPAI), 2021, pp. 277–283. solutions, Journal of Management and Economics [13] L. Xu, H. Xie, Z. Li, F. L. Wang, W. Wang, Q. Li, (2024). Contrastive learning models for sentence represen[3] P. V. Yadav, U. S. Kollimath, T. V. Chavan, D. T. tations, ACM Trans. Intell. Syst. Technol. 14 (2023) Pisal, S. A. Giramkar, S. M. Swamy, Impact of arti- 67:1–67:34. ifcial intelligence (ai) in talent acquisition process: [14] M. le Vrang, A. Papantoniou, E. Pauwels, P. Fannes, A study with reference to it industry, 2023 Inter- D. Vandensteen, J. D. Smedt, ESCO: boosting job national Conference on Intelligent and Innovative matching in europe with semantic interoperability, Technologies in Computing, Electrical and Electron- Computer 47 (2014) 57–64.
ics (IITCEE) (2023) 885–889. [15] E. Commission, ESCO (European Skills,
Com[
ECAI, IEEE, 2024, pp. 1–5. [16] M. Zhang, R. van der Goot, B. Plank, ESCOXLM-R: [5] J. Yao, Y. Xu, J. Gao, A study of reciprocal job Multilingual taxonomy-driven pre-training for the recommendation for college graduates integrat- job market domain, in: A. Rogers, J. Boyd-Graber, ing semantic keyword matching and social net- N. Okazaki (Eds.), Proceedings of the 61st Annual working, Applied Sciences (2023). URL: https: Meeting of the Association for Computational Lin//api.semanticscholar.org/CorpusID:265236145. guistics, Association for Computational Linguistics, [6] R. Zbib, L. L. Alvarez, F. Retyk, R. Poves, J. Aizpuru, Toronto, Canada, 2023, pp. 11871–11890.
H. Fabregat, V. Simkus, E. G. Casademont, Learning [17] B. M. Sarwar, G. Karypis, J. A. Konstan, J. Riedl, Item-based collaborative filtering recommendation mity on the hypersphere, in: Proceedings of the algorithms, in: V. Y. Shen, N. Saito, M. R. Lyu, 37th International Conference on Machine LearnM. E. Zurko (Eds.), Proceedings of the Tenth Inter- ing, ICML, volume 119 of Proceedings of Machine national World Wide Web Conference, WWW 10, Learning Research, PMLR, 2020, pp. 9929–9939. Hong Kong, China, May 1-5, 2001, ACM, 2001, pp. [27] J. Gao, D. He, X. Tan, T. Qin, L. Wang, T. Liu, Rep285–295. resentation degeneration problem in training nat[18] R. Patel, S. K. Vishwakarma, An eficient approach ural language generation models, in: 7th Interfor job recommendation system based on collabora- national Conference on Learning Representations, tive filtering, in: M. Tuba, S. Akashe, A. Joshi (Eds.), ICLR, 2019.
ICT Systems and Sustainability, Springer Singapore, [28] B. Li, H. Zhou, J. He, M. Wang, Y. Yang, L. Li, On the Singapore, 2020, pp. 169–176. sentence embeddings from pre-trained language [19] I. Obadic, G. Madjarov, I. Dimitrovski, D. Gjorgje- models, in: B. Webber, T. Cohn, Y. He, Y. Liu (Eds.), vikj, Addressing item-cold start problem in recom- Proceedings of the 2020 Conference on Empirical mendation systems using model based approach Methods in Natural Language Processing, EMNLP, and deep learning, in: D. Trajanov, V. Bakeva (Eds.), Association for Computational Linguistics, 2020, pp. ICT Innovations 2017 - Data-Driven Innovation. 9119–9130. 9th International Conference, volume 778 of Com- [29] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, munications in Computer and Information Science, O. Levy, M. Lewis, L. Zettlemoyer, V. StoySpringer, 2017, pp. 176–185. anov, Roberta: A robustly optimized BERT pre[20] J. Zhao, J. Wang, M. Sigdel, B. Zhang, P. Hoang, training approach, CoRR abs/1907.11692 (2019).
M. Liu, M. Korayem, Embedding-based rec- arXiv:1907.11692. ommender system for job to candidate match- [30] X. Sun, Y. Meng, X. Ao, F. Wu, T. Zhang, J. Li, C. Fan, ing on scale, CoRR abs/2107.00221 (2021). Sentence similarity based on contexts, Trans. AsarXiv:2107.00221. soc. Comput. Linguistics 10 (2022) 573–588. doi:10. [21] M. Bocharova, E. Malakhov, V. Mezhuyev, Vacan- 1162/TACL\_A\_00477. cysbert: the approach for representation of titles [31] Y. Yan, R. Li, S. Wang, F. Zhang, W. Wu, W. Xu, Conand skills for semantic similarity search in the re- sert: A contrastive framework for self-supervised cruitment domain, CoRR abs/2307.16638 (2023). sentence representation transfer, in: C. Zong, F. Xia, arXiv:2307.16638. W. Li, R. Navigli (Eds.), Proceedings of the 59th An[22] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: nual Meeting of the Association for Computational pre-training of deep bidirectional transformers for Linguistics and the 11th International Joint Conferlanguage understanding, in: J. Burstein, C. Doran, ence on Natural Language Processing, Association T. Solorio (Eds.), Proceedings of the 2019 Confer- for Computational Linguistics, 2021, pp. 5065–5075. ence of the North American Chapter of the Associ- [32] S. Chopra, R. Hadsell, Y. LeCun, Learning a similaration for Computational Linguistics: Human Lan- ity metric discriminatively, with application to face guage Technologies, NAACL-HLT), Association for verification, in: Computer Society Conference on Computational Linguistics, 2019, pp. 4171–4186. Computer Vision and Pattern Recognition (CVPR), [23] T. Chen, S. Kornblith, M. Norouzi, G. E. Hinton, A IEEE Computer Society, 2005, pp. 539–546. URL: simple framework for contrastive learning of visual https://doi.org/10.1109/CVPR.2005.202. representations, in: Proceedings of the 37th Inter- [33] F. Schrof, D. Kalenichenko, J. Philbin, Facenet: A national Conference on Machine Learning, ICML, unified embedding for face recognition and clusvolume 119 of Proceedings of Machine Learning Re- tering, in: IEEE Conference on Computer Vision search, PMLR, 2020, pp. 1597–1607. and Pattern Recognition, CVPR, IEEE Computer [24] R. Hadsell, S. Chopra, Y. LeCun, Dimensionality Society, 2015, pp. 815–823.
reduction by learning an invariant mapping, in: [34] K. Sohn, Improved deep metric learning with IEEE Computer Society Conference on Computer multi-class n-pair loss objective, in: D. D. Lee, Vision and Pattern Recognition CVPR, IEEE Com- M. Sugiyama, U. von Luxburg, I. Guyon, R. Garnett puter Society, 2006, pp. 1735–1742. (Eds.), Advances in Neural Information Processing [25] M. Ye, X. Zhang, P. C. Yuen, S. Chang, Unsupervised Systems 29: Annual Conference on Neural Inforembedding learning via invariant and spreading mation Processing Systems, 2016, pp. 1849–1857. instance feature, in: IEEE Conference on Computer [35] Q. V. Le, T. Mikolov, Distributed representations of Vision and Pattern Recognition, CVPR, Computer sentences and documents, in: Proceedings of the Vision Foundation / IEEE, 2019, pp. 6210–6219. 31th International Conference on Machine Learn[26] T. Wang, P. Isola, Understanding contrastive repre- ing, ICML, volume 32 of JMLR Workshop and Consentation learning through alignment and unifor- ference Proceedings, JMLR.org, 2014, pp. 1188–1196.
lected encoders used for doing the analysis of the proposed methodology for multilingual job recommendation. Table 4
training configuration for each architecture.
• mUSE-CNN: Multilingual Universal Sentence
Encoder CNN [
Transformer-based model pre-trained on 100
languages using masked language modeling. We use
the model implemented in Tensorflow available
at Kaggle8. This model has 278 M parameters.
• Paraphrase Multilingual MPNet [
500 K Training configuration on Stage 1 with unsupervised method
RoBERTa, and it is distilled to mimic the vector- for each encoder with the contrastive learning method dense representation of English sentences on +50 languages. This architecture is implemented in (Stage 2). Due to its reduced size in terms of parameters, for mUSE-CNN encoder, we use a bigger batch size and PyTorch and is publicly available at Hugging- larger number of negatives per anchor element compared to Transformer-based architectures. As for Transformerbased solutions, due to the limitation in terms of the
size of XLM-RoBERTa encoder, consequently, we increase the number of training steps. Lastly, regarding the temperature ( ), we do a grid search with the following values
vs. Transformer-based encoders, due to the diferent nature of their backbone architecture. Also, training for
modeling, and the other on sentence similarity. These configurations are the result of preliminary experiments in which we compared the impact of the diferent hyperparameters for every encoder model.
A common element shared by all variants is the use
of the AdamW [