In the last years, job recommender systems have become popular since they successfully reduce information overload by generating personalized job suggestions. Although in the literature exists a variety of techniques and strategies used as part of job recommender systems, most of them fail to recommending job vacancies that t properly to the job seekers pro les. Thus, the contributions of this work are threefold, we: i) made publicly available a new dataset formed by a set of job seekers pro les and a set of job vacancies collected from di erent job search engine sites; ii) put forward the proposal of a framework for job recommendation based on professional skills of job seekers; and iii) carried out an evaluation to quantify empirically the recommendation abilities of two state-of-the-art methods, considering di erent con gurations, within the proposed framework. We thus present a general panorama of job recommendation task aiming to facilitate research and real-world application design regarding this important issue.
Nowadays, job search is a task commonly done on the Internet using job search engine sites like LinkedIn1, Indeed2, and others. Commonly, a job seeker has two ways to search a job using these sites: 1) doing a query based on keywords related to the job vacancy that he/she is looking for, or 2) creating and/or updating a professional pro le containing data related to his/her education, professional experience, professional skills and other, and receive personalized job recommendations based on this data. Sites providing support to the former case are more popular and have a simpler structure; however, their recommendations are less accurate than those of the sites using pro le data.
Personalized job recommendation sites implemented a variety of types of recommender systems, such as content-based ltering, collaborative ltering, knowledge-based and hybrid approaches [AlO12]. Moreover, most Copyright c 2018 for the individual papers by the paper's authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. of these job recommender systems perform their suggestions based on the full pro le of job seekers as well as by considering other data sources such as social networking activities, web search history, etc. Despite the fact that many data sources can be useful to improve the job recommendation, previous studies showed that the best person-job t is possible when the personal skills of a job seeker match with the requirements of a job o er [Den15].
Based on the person-job t premise, we propose a framework for job recommendation based on professional skills of job seekers. We automatically extracted the skills from the job seeker pro les using a variety of text processing techniques. Therefore, we perform the job recommendation using TF-IDF and four di erent con gurations of Word2vec over a dataset of job seeker pro les and job vacancies collected by us. Our experimental results show the performances of the evaluated methods and con gurations and can be used as a guide to choose the most suitable method and con guration for job recommendation.
The remainder of this paper is organized as follows. In Section 2, we brie y describe the natural language processing methods we are used in our experimental setup. In Section 3 we present our proposal, including a new dataset collected by us and the framework for job recommendation. In Section 4, we show our experimental results. Finally, in Section 5, we o er conclusions and directions for future work. 2
In this section, we brie y describe two methods used in our experiments: Term Frequency-Inverse Document Frequency (TF-IDF) and Word2vec. Moreover, for Word2Vec we also present two models commonly used over it: Continuous Bag-of-Words (CBOW) and Skip-gram. 2.1
TF-IDF assigns weights to the words as a statistical measure used to evaluate the relevance of a word in document of a corpus [Sal88]. This relevance is proportional to the number of times a word appears in the document and inversely proportional to the frequency of the word in the corpus.
This method has been successful in topic identi cation over large text datasets, but its performance decrease when applied over small ones as those commonly found in job descriptions. However, TF-IDF has been applied to deal with recommendation obtaining interesting results [Dia13] [Dia14]. 2.2
Word2vec is a general predictive model for learning vector representations of words [Mik13b]. These vector representations, also called word embeddings, capture distributional semantics and co-occurrence statistics [Mik13a]. There are two Word2vec models we can use to obtain word embeddings: CBOW and Skip-gram. Continuous Bag-of-Words model (CBOW). This model predicts a target word based on the n words before and n words after the target word [Mik13b]. For example, in the following sentence:
Lorem ipsum dolor sit amet CBOW will predict the word dolor taking as inputs n = 2 words before and after it, i.e. Lorem, ipsum, sit and amet. These words are called the context of the target word and their quantity is a parameter of the model.
Skip-gram. Rather than predicting a word based on its context, Skip-gram aims to predict the context based on one word [Mik13a]. For instance, based on our previous example, skip-gram will try to predict the words Lorem, ipsum, sit and amet having only the word dolor as input. 3
In this section, we describe our framework for job recommendation. We narrow down the scope and focus on recommendation of job vacancies for Information Technology (IT) professionals acting in the Brazilian market. The proposed framework, depicted in Fig.1, is composed by three stages: data collection, data preparation and recommendation. We automatically collected a set of job vacancies/o ers from the Brazilian recruitment site called Catho3 and a set of Brazilian professional pro les from the well-known LinkedIn. We make available these datasets in a public repository4 with personal data anonymised. It is important to note that we collected more data from similar sites but, due to the validation issues, we only managed to work with these two sources in our framework.
To perform job o ers scraping, we created a list of keywords from the IT industry and used them as search terms. For each keyword, we search all the related job o ers using Catho's search engine and save the retrieved results in our database; thus, the content's quality is highly related to the quality of the Catho's search engine. Additionally, the scraper is set up to avoid duplicate job o ers, thus all the job o ers are unique. On the other hand, to perform professional pro les scrapping, we created a list of areas of professional practice from the IT industry and, from that, we search among the professional contacts of rst, second and third degree of our research group using Linkedin's search engine and save the retrieved results in our database; thus, all the professional pro les also are unique.
We use text mining approaches to process both pro les and job o ers data. Therefore, we selected the work experience, education and competencies/skills from the pro les and, the description and title from the job o ers. Finally, we concatenate these elds into a new one and discard the original elds, thus we end up with a document-like representation for each job o er and professional pro le. 3.2
Although we retrieved data from job search sites using only IT keywords, there were still some job o ers that do not correspond to this eld, then, the rst step in this phase is ltering out job o ers that do not belong to the IT eld. To achieve this, we use a dictionary of weighted IT terms to match each job o er in its document-like format. This way, we calculate the weighted sum of the appearances of each word of the job o er in the dictionary and divided it by the appearances of the rest of words in the document (job o er). Finally, we get a score with a value from 0 to 1, where a higher value indicates that the o er contains many relevant words on IT and it is very likely that corresponds to this eld. Subsequently, we select only those job o ers with a value of this score greater than 0.5. This setback only happens with the job o ers since pro les were collected only into a IT professionals network.
Once job o ers and pro les are ltered, the second step is text preprocessing. In this task, we perform stop words removal, tokenization and lemmatization for the Portuguese language.
The third step, feature representation, aims to represent these documents (job o ers and pro les) as vector space models. For this purpose, we adopted two approaches: word embeddings and TF-IDF. The latter technique does not require so much e ort to be implemented unlike the former. From the variety of word embedding representations we selected Word2Vec, which has di erent variants. We explore the two model architectures CBOW and Skip-Gram, and also the use of n-grams (bigrams and trigrams) in order to nd the variation that best t our problem. This way, we tested 5 di erent representations, TF-IDF, Word2Vec using CBOW, Word2Vec using Skip-Gram, Word2Vec using CBOW with n-grams and Word2Vec using Skip-Gram with n-grams. For the Word2vec models, a vector space size of 200 was selected after some initial experimentation.
For both word embedding and TF-IDF representation, we only used the corpus composed by the job o ers. Although we lose some data, it was necessary since we realized that job seeker pro les added some noise because of the existence of professionals with a very di erent background and skill set looking for a job on IT, which could foster spurious relations among skills. Finally, we transform both job o ers and pro les into these 5 new representations and then proceed to use them in the recommendation phase. In Table 1, we can see the description of the corpora used for our word embeddings. 3 https://www.catho.com.br 4 http://visibilia.net.br/text2story-job-recomendation/ In this last phase, given a certain pro le with a proper representation, we select a group of the nearest job o ers based on the distance to that pro le (job matching). In the case of TF-IDF representation, we use the cosine distance while for word embeddings, we use the relatively new Word Mover's Distance (WMD) [Kus15]. Once retrieved the top "k" job o ers for the pro le, we sort them in descending order based on the inverse of this distance (ranking). 4
In this section, we present extensive empirical experiments focused on evaluating the quality of job recommendations. For these experiments, we take the case of recommending a set of job o ers given a speci c professional pro le.
Our data set is composed by 50 professional pro les from LinkedIn and 3877 job o ers from Catho. Both pro les and job o ers correspond to Brazilian professionals and companies from the IT eld. Due to the extensive of the IT eld, professional pro les can also di er a little bit among them. Table 2 shows the distribution of sub elds within our sample of 50 professional pro les which re ects the greater number of developers and BI consultants.
First, we use our framework to generate 10 job o er recommendations for 50 di erent pro les. Thus, for each evaluated technique, we obtained a total of 500 recommendations. Second, a group of 5 Resource Human professionals evaluated manually these recommendations and allocate a score ranging from 1 to 10. The more accurate or suitable the recommendation, the greater the score. In order to make the results more understandable, we standardize these scores dividing them by the maximum score. Third, once these scores are obtained, we averaged them and also calculated Precision and Minimum E ectiveness (ME).
Precision for a single pro le by dividing the number of relevant documents (recommendations with a score greater than 0.5) by all the retrieved documents (total of recommendations); then, we average this precision over all the pro les. On the other hand, the Minimum E ectiveness (ME) allocates a score of 1 if at least one out of the 10 recommendations for a pro le has a score greater or equal to 0.5, otherwise it allocates 0. Thus, we average this value to have an estimator of the global e ectiveness of the system of 10 job recommendations per pro le. In Table 3, we show the result of applying these metrics over our dataset for the 5 di erent evaluated techniques.
Here, we can observe that Word2Vec with Skip-Gram obtains a slightly better average score than TF-IDF, which has the second best average. On the other hand, Word2Vec with Skip-Gram clearly gets a better average precision over all the other techniques by a good margin and it is the best option according to the three metrics. The Word2Vec variant using Skip-Gram with n-grams ranked second. Furthermore, we also observe that not all the pro les were given a good recommendation as the maximum value of the average minimum e ectiveness is 0.96 (48 out of 50 pro les). This last metric is highly dependent on the quality of the ltering process and the variety of job o ers since there can be a shortage of o ers for some speci c pro les. Finally, we can see that the two versions of Word2Vec using n-grams perform better than the Word2Vec with CBOW according to all metrics used. Plus, these n-gram versions have a slightly better average precision than TF-IDF, but a lower average score. 5
In this paper, we proposed a framework for job recommendation task. This framework facilitates the understanding of job recommendation process as well as it allows the use of a variety of text processing and recommendation methods according to the preferences of the job recommender system designer. Moreover, we also contribute making publicly available a new dataset containing job seekers pro les and job vacancies.
Future directions of our work will focus on performing a more exhaustive evaluation considering a greater amount of methods and data as well as a comprehensive evaluation of the impact of each professional skill of a job seeker on the received job recommendation.
This work was partially supported by the S~ao Paulo Research Foundation (FAPESP) grants: 2016/08183-5, 2017/14995-5, 2017/15070-5, 2017/15247-2 and 2017/17312-6.