Over the past decade, a growing number of employers has been using the Web for advertising job opportunities through Web job vacancies, that usually specify a job position, along with a set of skills a candidate should possess. Reasoning with these Web job advertisements can e ectively support the decision marking processes of several labour market stakeholders, including public organisations, educational and employment agencies, and analysts as well. Here, Labour Market Intelligence refers to the design and de nition of automated methodologies and tools for supporting real-time labour market monitoring at a very ne-grained level. This, in turn, represents a competitive advantage to labour market stakeholders with respect to classical survey-based analyses, as they are quite expensive and may require up to one year before being available. In this paper we discuss how Web job vacancies have been collected from selected websites, processed, and classi ed over a standard taxonomy through machine learning algorithms, extracting the most relevant skills from raw texts. Then, we show how our approach has been applied to some real-life studies and we discuss the bene ts provided to end users.
as they are \valued for fostering job-speci c and transversal skills, facilitating the transition into employment and maintaining and updating the skills of the workforce according to sectorial, regional and local needs".1 In 2014, the Cedefop EU Agency2 launched a call-for-tender3 aimed at collecting Web job vacancies from ve EU countries and extracting the requested skills from the data. The rationale behind the project is to turn data extracted from Web job vacancies into knowledge (and thus value) for policies design and evaluation through a factbased decision making. In 2016, the EU launched the ESSnet Big Data project, involving 22 EU member states with the aim of \integrating big data in the regular production of o cial statistics, through pilots exploring the potential of selected big data sources and building concrete applications".
The rationale behind all these initiatives is that reasoning over Web job vacancies represents an added value for both public and private labour market operators to deeply understand the Labour Market dynamics, occupations, skills, and trends: (i) by reducing the time-to-market with respect to classical surveybased analyses (o cial Labour Market surveys results actually require up to one year before being available); (ii) by overcoming the linguistic boundaries through the use of standard classi cation systems rather than proprietary ones; (iii) by representing the resulting knowledge over several dimensions (e.g., territory, sectors, contracts, etc) at di erent level of granularity and (iv) by evaluating and comparing international labour markets to support fact-based decision making.
Paper's Goal. This paper would summarise some results of our research activities and outcomes in Web Labour Market Intelligence in two distinct research projects: WollyBI 4 and Cedefop3, by focusing on (i) job vacancy classi cation and skill extraction tasks, and (ii) the support to decision making activities that our approach provided to the involved stakeholders. 2
LMI is anemerging cross-disciplinary eld of studies that is attracting research interests in both industrial and academic communities, as we summarise below.
Scienti c Literature. Since the early 90s, text classi cation (TC) has been
an active research topic. It has been de ned as \the activity of labelling
natural language texts with thematic categories from a prede ned set" [
Industries. This problem is also relevant for business purposes, and this motivates the growing of several commercial products providing job seekers and companies with skill-matching tools. Concerning rms, they strongly need to automatize Human Resource (HR) department activities; as a consequence, a growing amount of commercial skill-matching products has been developed in the last years, for instance BurningGlass, Workday, Pluralsight, EmployInsight, and TextKernel. To date, the only commercial solution that uses standard taxonomies as thesauri is Janzz: a Web based platform to match labour demand and supply in both public and private sectors. It also provides APIs access to its knowledge base, but it is not aimed at classifying job vacancies. Worth of mentioning is Google Job Search API, a pay-as-you-go service announced in 2016 for classyfying job vacancies through the Google Machine Learning service over O*NET, that is the US standard occupation taxonomy. Though this commercial service is still a closed alpha, it is quite promising and also sheds the light on the needs for reasoning with Web job vacancies using a common taxonomy.
All these approaches are quite relevant and e ective, and they also make evidence of the importance of the Web for labour market information. Nonetheless, they di er from our approach in two aspects. First, we aim to classify job vacancies according to a target classi cation system for building a (language independent) knowledge base for analyses purposes, rather than matching resumes on job vacancies. Second, our approach aims to build a knowledge-graph for supporting the fact-based decision making activities for LMI. 3
Background on Web Labour Market Information. A Web job o er extracted can be seen as a document mainly composed of a pair of texts: a title and a (full job) description. The title summarises the working position o ered by the employer, while the description usually provides the position details, including all the required relevant skills, according to the employer preferences.
One of the most important classi cation system designed for this purposes is ISCO: The International Standard Classi cation of Occupations is a four-level classi cation that represents a standardised way for organising the labour market occupations. ESCO is the multilingual classi cation system of European Skills, Competences, Quali cations and Occupations, it is the European standard for supporting the whole labour market intelligence over 24 EU languages. Basically, the ESCO data model includes the whole ISCO structure, and extends it through (i) a further level of ne-grained occupation descriptions and (ii) a taxonomy of skills, and competences.
Web job vacancy classi cation through Text Classi cation. Text
categorization aims at assigning a Boolean value to each pair (dj ; ci) 2 D C where D
is a set of documents and C a set of prede ned categories. A true value assigned
to (dj ; ci) indicates document dj to be set under the category ci, while a false
value indicates dj cannot be assigned under ci. In our scenario, we consider a set
of job vacancies J as a collection of documents each of which has to be assigned
to one (and only one) ISCO occupation code. We can model this problem as a
text classi cation problem, relying on the de nition of [
Building up the Machine Learning Model. We build a machine learning model
for classifying multilingual Web job vacancies exploiting a single-label classi er
using both titles and descriptions. Indeed, titles often does not contain enough
information for performing a correct classi cation as we shown in [
Focusing on the Italian Labour Market Information, a set of 57,740 vacancies previously classi ed by domain experts belonging to ENRLMM5 was used. 5 The European Network on Regional Labour Market Monitoring The set of already classi ed vacancies are split into train, validation, and test sets. Then, a grid-search was performed over each classi er parameter space to identify the values maximizing the classi cation e ectiveness (using training and validation sets). Tab. 1 shows the scores computed over the test set.
Skills Identi cation. The text pieces stating the required skills usually concentrate on a small portion of job vacancy descriptions. Thus, these relevant text pieces are extracted through a look-up search of sentinel expressions selected by domain experts involved within the projects. The domain expert kept adding new sentinel expressions until the number of n-grams identi ed in the next step grows (i.e., the set converged to a stable set). Then, the n-gram Document Frequency (DF ), i.e., the number of vacancies where the n-gram is found, is computed considering as a scope both (1) the whole dataset and (2) the vacancy subsets homogeneous w.r.t. the ISCO occupation code.
The n-gram produced by the previous step underwent a string similarity comparison with respect to ESCO skill concept labels. The pair <skill candidate n-gram, ESCO Skill label> matching the following criteria were suggested for domain experts approval. The string similarity was computed as the mean value among the following well-known string metrics: Levenshtein distance, Jaccard similarity, and the S rensen-Dice indexes6. The pairs having a similarity lower than 70% are dropped while the others are proposed to the domain experts for evaluation. Each <candidate n-gram, ESCO Skill label> above the threshold has been reviewed by a domain expert to decided whether to consider an n-gram as: (1) a skill described in ESCO; (2) a skill not enlisted in ESCO (i.e., a novel skill); (3) or to reject the proposed n-gram as a skill concept. The outcome of this process can be seen as a dictionary of n-gram related skills and their corresponding ESCO skill concept (when available). Finally, the n-gram dictionary produced by the previous steps and the mappings among the ESCO skill concepts have been used to look for skills among the downloaded vacancies.
LM Knowledge as a Graph. The resulting knowledge on LM is then modelled as a graph. In Fig. 1 we report a selection of the graph-db data model according to the Neo4j property graph structure. The model is basically composed of two main node labels, occupations and skills. The former are the ISCO occupation codes whilst the latter are the union of both ESCO skills and the skills recognised as novel in the skill extraction phase, as described above. Then, two distinct directed relationships are allowed between skills and occupations to model that a skill s belongs to a given occupation o. The :BELONG relation would represent an occupation o requiring an ESCO skill s with a relevance of w in the ESCO taxonomy. This relationship measures the importance of the skill for a given occupation according to a set of labour market experts. Di erently, the :BELONG DATA relation models that w job vacancies have asked for skill s in the Web job vacancy text.
Such a knowledge graph allowed us to perform a several path-traversal analyses, such as Skill2Job, to identify the most promising occupations that one could 6 Though the latter is not a proper distance metric, it has been selected as we do not ask string metrics to satisfy the triangle inequality be interested into given a set of skills, the Gap Analysis to identify the most important skills that one should acquire given a set of skills that a candidate already holds. Due to the space restrictions, here we only give the idea of how the graph-structure can be used to identify occupation groups on the basis of the skills they have in common, distinguishing between groups according to the ESCO and real data. To this end, we employed a local-clustering-coe cient metric to identify all the occupations that share at least k% skills in common (i.e., having a clustering coe cient equals to 1). We employed a weigthed Jaccard metric to compute the similarity between occupations on the basis of skills in common. Fig. 2 shows the ESCO skills category asked for a group of occupations related to mathematicians and statisticians. The outer circle refers to ESCO skills whose relevance has been computed using the :BELONGS relation, whilst the inner circle refers to ESCO skills with a relevance computed using the :BELONG DATA relation (i.e., Web job vacancies). As one might note, computing skills account for 6% according to the ESCO experts, whilst this value grows up to 66% in the real data that mainly speci es skills such as SQL, Relational Databases, Python and Data Warehouse. Conversely, the Business & Administration sector seems to be overstimated by ESCO taxonomy, that indicates up to 56 B&A related skills whilst only few on them are actually repeatedly asked by companies. This analysis would allow one to measure the gap between (1) an LMI system built through an expert-driven approach as in the ESCO case, where labour market experts indicates a list of skills that are relevant in an occupation pro le, and (2) a data-driven system, where skills are recognised as important on the basis of the real labour market expectations. 4
The WollyBI Experience. WollyBI is a SaaS tool for collecting and classifying Web job vacancies on the ISCO/ESCO standard taxonomies, and extracting the most requested skills from job descriptions. It has been designed to provide ve distinct entry points to the user on the basis of the analysis purposes, that are, Geographical Area, Skill, Firm, Occupation, and free OLAP queries. Competition Analysis for Strategic Decision Making. WollyBI supported a recruitment agency to identify and measure the market share with respect to its competitors, that included the most relevant recruitment Agencies in Italy, namely: GiGroup, Adecco, ManPower, RandStad, ObiettivoLavoro, and Umana. In Fig. 4 we report the market share distribution over the top-10 ISCO occupations, by analysing the Italian Web Job Vacancies since February 2013 to April 2015. Clearly, due to undisclosure agreements, the agency labels reported in Fig. 4 have been anonymized. We analysed about 850K Web job vacancies posted by these agencies. This competition analysis has been validated as helpful to the customer (Agency B) as it allowed to (i) measure the position in the market and the gap with respect to their competitors; (ii) to drive the identi cation of strategic decisions to improve its market share and, in turn, to identify the corresponding strategies that allow achieving the desired goals. Just to give a few examples, our analysis revealed that Agency B is leader in recruiting "shop sales assistants" whilst its market share ranges between 9% and 15% in the remaining professions. Thanks to this results, Agency B has been made in state to design its strategic intervention through fact-based decision-making. 10000 isce 8000 cboaan jvb 6000 feW 4000 eo r ubm 2000 N 0
FBAECD
The Cedefop Experience. In 2014, the experience of WollyBI put the basis
of a prototype system we realised within a call-for-tender for the Cedefop EU
Agency aimed at collecting Web job vacancies from ve EU countries and
extracting the requested skills from the data. The rationale behind the project is
to turn data extracted from Web job vacancies into knowledge (and thus value)
for policies design and evaluation through fact-based decision making. The
architecture of the system basically relies on WollyBI, that is now running on the
Cedefop data centre since June 2016, gathering and classifying job vacancies
from 5 EU countries, namely: United Kingdom, Ireland, Czech Republic, Italy
and Germany. To date, the system collected 7+ million job vacancies over the
5 EU countries, and it accounts among the research projects that a selection of
Italian universities addressed in the context of big data [
Concluding Remarks and Research Directions. In this paper we described our approach to Web Labour Market Intelligence, focusing on the realisation of a machine learning model for classifying job vacancies and showing the bene ts of a graph-based representation of the knowledge base. Our research goes toward two directions. From an application point of view, we have been commited by Cedefop for extending the prototype to the whole EU community to all 28 EU Countries, building the system for the EU Web Labour Market Monitoring7. From a methodological perspective, reasoning with Web job vacancies raises some interesting research issues, such as the automatic synthesis of the labour market knowledge through word embeddings, the identi cation of AI heuristic-search algorithms for path-traversal over big knowledge-graph, as well as the design of novel AI techniques for data cleasing in a big data scenario.