Understanding labour market dynamics requires accurately identifying the skills required for and possessed by the workforce. Automation techniques are increasingly being developed to support this efort. However, automatically extracting skills from job postings is challenging due to the vast number of existing skills. The ESCO (European Skills, Competences, Qualifications and Occupations) framework provides a useful reference, listing over 13,000 individual skills. However, skills extraction remains dificult and accurately matching job posts to the ESCO taxonomy is an open problem. In this work, we propose an end-to-end zero-shot system for skills extraction from job descriptions based on large language models (LLMs). We generate synthetic training data for the entirety of ESCO skills and train a classifier to extract skill mentions from job posts. We also employ a similarity retriever to generate skill candidates which are then re-ranked using a second LLM. Using synthetic data achieves an RP@10 score 10 points higher than previous distant supervision approaches. Adding GPT-4 re-ranking improves RP@10 by over 22 points over previous methods. We also show that Framing the task as mock programming when prompting the LLM can lead to better performance than natural language prompts, especially with weaker LLMs. We demonstrate the potential of integrating large language models at both ends of skills matching pipelines. Our approach requires no human annotations and achieve extremely promising results on skills extraction against ESCO.
able roadblocks. Most notably, many eforts are limited The Job Market is often described as constantly evolving, in scale by the lack of available data for both training as technological developments and societal changes re- and evaluation. Recent eforts have begun to try and sult in large changes in its makeup, as has been frequently alleviate this efort, by publicly releasing manually annostudied and demonstrated [1, 2]. In recent years, the rapid tated datasets. These approaches, while promising, sufer digitisation of society has led to entirely new categories from the complexity of the task, as the ESCO taxonomy of skills becoming requirements for many jobs[3]. As contains 13 890 individual skills. As a result, the task is demands for skills evolve, there is an increasing need often re-framed or simplified, for example by convertto better understand the skills required by jobs. This ing the task into a span-extraction task [7, 8], leaving has supported the continuous development of skill tax- direct matching to the taxonomy for future work. Some onomies such as the European Union’s European Skills, recent work has explored this Extreme Multi-Label ClasCompetences, Qualifications and Occupations, or ESCO sification (XLMC) task with an end-to-end approach to [4], framework, developed to improve understanding and matching and extraction with non-ESCO taxonomies, eficiency of the wider EU job market. with encouraging results [6, 9], although sometimes re
While useful, such frameworks require skill extrac- lying on simplifying the taxonomy by using only highertion (SE) approaches to understand skills present in job level labels [10]. postings and enable automation at scale. Skills Extrac- In fact, the task of Skills Extraction against a taxtion has recently been the subject of an increased amount onomy could be framed as a two-tasks process: 1 an of interest[5], which has been further compounded by extraction step, focused on recognising the potential research showing that a large proportion of required mentions of skills, or groups of skills, from the content skills are implicitly expressed rather than explicitly stated of job postings, on which strong progress has been made within postings [6]. [7, 11, 6], and 2 a matching step, akin to extreme multiRecSys in HR’23: The 3rd Workshop on Recommender Systems for label classification, focused on linking these mentions Human Resources, in conjunction with the 17th ACM Conference on with fine-grained taxonomies, which remains a dificult Recommender Systems, September 18–22, 2023, Singapore, Singapore. problem. * Corresponding author: Benjamin Clavié The sheer number of existing skills makes it very difi$ ben.clavie@brightnetwork.co.uk (B. Clavié); cult to obtain suficient training data for comprehensive gui0ll0a0u0m-0e0.s0o2u-8li3e4@6-b8r7ig0h3t(nBe.twCloarvki.éc)o.uk (G. Soulié) coverage. As such, various techniques, such as using the © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License ESCO API as a form of distant supervision [12] and genCPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) erating training data through such distant supervision strate later, LLMs are strong zero-shot rerankers, it’s techniques [13], have been explored. impossible for them to work as standalone classifiers for
Meanwhile, the rapid development and improvement all ESCO skills, as the 13890 skills listed in ESCO do not of generative large language models (LLMs) [14] and es- fit within the context window of most Language Models. pecially instruction-tuned LLMs [15, 16] , models further Even if this were possible, the increased compute and trained to specifically follow natural language instruc- processing time required by current Transformer-based tions, has resulted in the use of generative LLMs on a models (used by all LLMs) would make this approach large array of applications, often yielding competitive unsuitable in many cases, despite promising advanceor even state-of-the-art results across many tasks, as ments in more computationally eficient approaches to highlighted by the release and strong performance of Transformer models [21].
OpenAI’s GPT-4 [17]. Notably, LLMs have shown their We thus choose to leverage large generative models ability to improve performance on text retrieval tasks for the skills matching task through the generation of through the generation of synthetic training data based synthetic training data. We believe that this synthetic on a handful of real examples [18]. They have also shown training could allow us to perform the skills matching considerable problem-solving ability, often anthropomor- task without having to reframe the task or rely on useful phised as reasoning, which appears to be even stronger but limited distant supervision techniques[13]. when the task is framed as a programming task [19]. For each of the 13890 skills contained in ESCO, we
Due to the nature of the training data used by Large prompt GPT-3.5 1 to generate forty example sentences Language Models, which contains very large volumes that could be used in a job posting in order to refer to of content scraped from across the internet [20], we hy- the skill. We specifically request that the sentences be pothesise that the knowledge representation embedded phrased in a variety of ways, and be of various lengths within the models makes them particularly suitable for (from just a few words directly referring to the skill to broader job understanding tasks, as many job postings, a few sentences mentioning it implicitly). We provide as well as skills descriptions, were present in this data. As slightly diferent instructions based on the "skill type", for such, we will explore the creation of a zero-shot skills instance requesting explicit mentions in more examples matching pipeline through the use of large language when generating data for a skill contained in the tech models, focusing in this case on the use of GPT-3.5 and ESCO skill listing, as programming languages tend to GPT-4. be clearly mentioned in job ads. We also use additional information or skill descriptions present in the ESCO data
Contributions In this paper, we show that: 1 Large to enrich our prompt and help the model disambiguate Language Models can reliably generate zero-shot train- between potentially ambiguous terms. ing data that improves performance in the skill matching Once the training data is generated, it is not thoroughly task. We then show that using this data to power both manually reviewed. We performed programmatic checks, similarity-based retrieval approaches and linear classifier showing that a full forty examples were generated for models trained on the data can generate good lists of more than 97% of skills, with a few of them having fewer potential skills found within given text extracts, outper- examples due to model context size limits or failure to forming the previous approaches. 2 The Skill Match- properly follow instructions, which were not addressed, ing task can be framed as a two-step problem, with the as the entirety of skills but one 2 had more than 30 genifrst stage consisting of generating a list of potential skill erated examples. We sampled a random 100 skills for matches and the second stage focusing on re-ranking manual review to ensure that the generated data met these potential matches. We show that LLMs can be used our criteria and that the prompts contained no obvious as zero-shot rerankers for this second step of the extrac- mistakes. tion pipeline with very strong performance without the The prompts used for data generation are provided in need for annotated data. 3 Framing the skills matching appendix A. task as a mock programming problem provides a further performance boost for both large language models tested.
More notably, it improves the performance of the less capable model by over 10 percentage points by enhancing its ability to follow instructions. 1Our limited experiments showed that all recent LLMs, including the open-source Flan-UL2 [22], could potentially perform this task without a strong negative impact on performance. Thoroughly evaluating diferent LLMs for this data generation step is beyond the scope of this work, but would likely be a valuable future area of research, as we noticed considerable style diferences between diferent models, leading us to believe a combination of diferent ones could generate a more diverse training data set. 2The skill "semen insertion", related to veterinary work, could not have training data generated for it by the GPT model family, as its name triggers OpenAI’s content filters.
Skills Matching pipeline. While, as we will demon
Following data generation, our next step is the identification of potential skills contained within a given text span. We generate these potential skills through two main approaches: a linear classifier-based approach , using linear regression classifiers on frozen embeddings, and textual similarity approaches, where we rely on 4. Zero-Shot Potential Skills the cosine distance between embeddings to determine Ranking whether a skill is potentially present or not.
For both of these approaches, we use the e5-large- Once the list of potential skills found in a given span v2 text embedding model [23], the current state-of-the- is generated, we prompt an LLM to extract and rank art embedder for similarity-based information retrieval the ten most likely skills in order of suitability. This is whose generated embeddings have also been shown to the reranking step, a key component of information reach strong performance for few-shot text classifica- retrieval pipelines, which is increasingly performed by tions, making it particularly suitable for both of our ap- fine-tuned language models [ 26]. In this work, we explore proaches. how a zero-shot approach leveraging LLMs’ "learned knowledge" performs on our task. the synthetic example sentences. If two sentences with the same label are part of the 40 most similar sentences, we add the label to the list of candidates. This is in effect similar to a simplified form of k-nearest neighbour classification [25].
Our first set of candidates is generated by the use of simple logistic regression classifiers. We train one binary We report results for both GPT 3.5, an instruction-tuned classifier per label, in a one-versus-all classification ap- [16], more powerful version of GPT-3 [27], more specifproach. We use no real-world data for these classifiers, icallygpt-3.5-turbo-0301 and GPT 4 [17] (gpt-4-0314), instead training only on the synthetic data generated as which produced the most promising results in our exdescribed in section 3.For any given class, we treat all ploratory work. Most of our prompt engineering work example sentences generated for the label as positive ex- was performed for GPT 4, and re-used as-is for GPT 3.5. amples and sample twice as many examples from other While we have not conducted extensive experiments labels to use as negative examples. using them, our exploratory work has shown that open
Following previous work on skill classification [ 13], source LLMs, of which Flan-UL2 [22] was the best perwe use partial hard negative sampling [24] to ensure our forming at the time of this work, failed to produce reliable models are better at distinguishing between very similar outputs, frequently "hallucinating" skills, in a way similar labels. To do so, we make it so 10% of the negative exam- to GPT 3.5. However, in the case of GPT3 3.5, this was ples are hard negatives, sampled from labels associated entirely mitigated by the Python approached described with the example sentences having the highest cosine below. similarity with the positive examples. At the time of conducting our experiments, Falcon
At inference time, we consider every positive classifi- [28] had not yet been released and the authors did not cation from the classifiers as a candidate label to be used have access to LLaMa [29]. for re-ranking.
on cosine similarity between embeddings. We generate candidate through two distinct approaches: label similarity and sentence similarity.
between a target extract and the full list of existing labels. We do not set a threshold for this step, rather, we treat the 40 most similar labels as candidates to be provided to the re-ranker.
use the cosine distance between the current extract and
scribe the task or reranking in the prompt. We use a chat-formatted prompt, through OpenAI’s ChatML [31].
We give the model a broad description of its role as its initial prompt, followed by the detailed instructions for the task, and a mocked message from the model acknowledging and summarising the instructions.
We then provide the model with a list of potential skills, generated by the previously described methods.
We experiment with both providing the model with information about the score it received as a potential skill, through either classification class probability or textual similarity, depending on the potential skill’s source. We found this had no impact on performance, and therefore 5. Experimental Setup do not provide this information to the model in our final evaluation to reduce the number of token used in our prompts, thus reducing the required compute. We As ESCO-based skills matching is an Extreme Multi-Label pass all skills identified by the classifier approach as Classification (XMLC) task, we choose to frame it simiwell as up to 60 skills identified by the similarity-based larly to an information retrieval task and use IR-inspired approach. methods, for which we provide a high-level overview
We request an ordered list of the ten most likely skill of our architecture in Figure 1. Skills matching against matches in our prompt. In all cases, we provide the model the ESCO taxonomy, due to the very large number and with the ability to use the NO_LABEL skill to reach 10 granularity of skill labels, justifies this framing: while we skills it identifies fewer or no matches. want to assign as many labels present in our test set as
All the prompts used for the reranking task are pro- possible, it is also highly likely that many potentially relvided in Appendix A. evant labels are not attached to our test examples, either because of oversight or because of subjective judgement 4.2.1. Mock Python Programming Variant in situations where multiple similar labels applied. As such, our aim is to maximise our retrieval of test labels, without harsh penalties for additional labels assigned by the model.
Recent work has shown that Large Language Models can often perform better on "reasoning" tasks when they are approached as programming exercises [19]. Additionally, anecdotal evidence often states that it is easier to control the output of large language models when requiring programming-like outputs, supposedly due to programming languages’ more structured nature. While investing the full extent of these claims is beyond the scope of this work, we experiment with modifying our re-ranking prompt to include explicit instructions to answer exclusively in Python, in the form of a function returning an ordered list of ranked skills, and outputting the justification for inclusion as a comment. No other modifications to the instructions are made.
We choose Python over other programming languages as it is often a good proportion of the programming data commonly used to train and evaluate LLMs [32, 33] and requires very little adjustments to the existing re-ranking step presented in the previous section, which is itself implemented in Python.
5.1.1. Data
Decorte et al. [13]. Their work built upon the SkillSpan dataset[7], a publicly available dataset focused on the detection of text spans containing the mention of either skills or knowledge, which are two sub-categories of skills as broadly defined within the ESCO framework. Using the extracted spans, Decorte et al. manually assigned ESCO skills to the extracted spans in order to create a dataset of spans annotated with the matching ESCO skill(s). To the best of our knowledge, this represents the best efort at creating an evaluation dataset using the full extent of ESCO’s fine-grained approach rather than approximations or groupings. We use the validation set provided to tweak our prompts and evaluate our models on the test set.
The data contains two distinct subsets, TECH, which focuses on data extracted from jobs within the tech sector, and HOUSE, containing more generalist jobs. We report Decorte et al. [13] (best approach) Classifiers (no rerank) Similarity (no rerank) GPT3.5 Re-ranking +Classifier +Similarity +Both +Both + Python GPT4 Re-ranking +Classifier +Similarity +Both +Both + Python results for each set separately, following the literature. Tech one. These results are encouraging, as they re
Both subsets contain a list of skills that were identified quire no real-world training data and are extremely fast within a given span by human annotators. at inference-time, requiring only simple computations. Their RP@k scales particularly well with higher k val5.1.2. Metrics ues, highlighting their ability to propose a number of correct labels but not ranking them optimally.
As the authors introducing the dataset note [13], it is GPT-4 reranking results in considerable improvements highly unlikely for human annotation to be fully exhaus- over all non-reranked methods, strongly outperformtive given the wealth of label, as such, the aim of this ing all other methods in all approaches and the besttask is to extract as many labels as possible, without pe- performing variant reaching an RP@10 of 61.02 on the nalising the model for "near misses", which could still be House dataset and 68.94 on the 68.94, a respective imappropriate labels. We thus follow their lead and report provement of 22.33 and 29.75 percentage points over the macro-averaged R-Precision@k (RP@k), which is the previous best approach and 14.55 and 10.28 over particularly well-suited to evaluated extreme multilabel our best non-reranked methods. We notice that in all classification tasks such as this one [ 34, 13] as well as the cases, the performance obtained by combining potential Mean Reciprocal Rank (MRR) of the highest ranked skills generated by both the classifier and the similarity correct label as a further indication of ranking quality. approaches is noticeably stronger than when using only one method of generating candidates. However, when 6. Results and Discussions using a single method of generating potential candidates, we notice that the similarity-based approach tends to outThe results of our experiments are presented in in Table 1. performs the classifier-based approach on both datasets, We report the performance of the full pipeline, with both especially on the House dataset.
GPT 3.5 and GPT 4 re-ranking, as well as the previous The performance of GPT-3.5 re-ranking is more mixed. state-of-the-art performance obtained by Decorte et al. With natural language prompting, its performance is an in the paper introducing the dataset [13]. We also report overall downgrade over the non-reranked approaches. the results of both our Classifier and Similarity ap- Unlike GPT-4, we also notice that combining both methproaches without the re-ranking step, both to showcase ods of potential skill generation does not systematically the performance obtained via the use of LLM-generated improve performance, especially on the Tech dataset training data and to serve as a baseline for the re-ranking where using only similarity-based entries resulted in approaches. overall stronger results. When using only the classifiers
We notice that, on their own, both of these no- based candidates, we notice that the GPT-3.5 ranking reranking approaches achieve competitive performance actually decreases performance. One of the noticeable against previous methods, with the similarity approach reasons for this weaker performance is GPT-3.5’s seemmarginally outperforming the classifier one on the ingly weaker ability to follow guidelines: despite our House dataset but performing noticeably worse on the experiments in modifying the model prompt, it would frequently "hallucinate" skills whose wording was di- behind APIs and their weights are not publicly available. rectly inspired from the target span, and ranking them While they perform well, future work should explore higher than the skills provided. the applicability of open-source LLMs, such as Falcon
For both GPT variants, we notice strong performance [28], as well as look for more eficient approaches. Adwith the Python prompt variation, where we explic- ditionally, we intend to explore if using a more diverse itly request that the model output is a Python function set of generative models, trained on diferent datasets, returning the ranked list of skills. In the case of GPT- could improve our synthetic training data generation by 4, the Python variant significantly outperforms natural generating more semantically varied examples. language prompting on the Tech dataset, but performs Domain-Specific Models Our approach focuses on slightly worse on House for all metrics but RP@10. For the use of general domain model, with no further training GPT-3.5, however, Python prompting nearly entirely to adapt them to the language used within job postings eliminates the problem of hallucinating skills, and greatly specifically. While we believe that this kind of informaimproves the performance across all metrics on both tion is present within the training corpora of the large datasets. This appears to suggest that framing the prob- language models we use, we believe that better targeted lem as a programming problem, which are frequently models could facilitate the development of more eficient used to train LLMs, helps ground reasoning and improve approaches as mentioned above. Notably, models such performance in re-ranking tasks in a way natural lan- as JobBERT [35] and ESCOXLM-R [5] have shown the guage prompt engineering cannot, although more exper- potential of domain-specific fine-tuning on existing tasks. iments are needed to confirm this. Meanwhile, the LLM literature highlights how consider
Overall, the use of LLM-generated training data out- ably smaller language models, with an order of magniperforms the state-of-the-art distant supervision ap- tude fewer parameters than GPT-3, can reach competitive proaches, and that zero-shot LLM re-ranking further performance through fine-tuning on small but very high increases performance, considerably outperforming all quality datasets [36, 37]. previous approaches. Potential Skills Generation Our experiments indicate that varying the number of potential skills given to GPT-4 does not have a major impact, if any, on its 7. Limitations and Future Work ranking performance. However, we did not extensively experiment with diferent ways of generating the potenWhile our work shows very strong potential for LLMs in tial skills list, and the impact that prompt modifications, both generating training data and improving inference- such as diferent ordering or indicating the source or time predictions for skills matching, we believe that there probability given to a label by the initial classifier would are three key limitations to our work that should be ex- have. Additionally, we conducted only very moderate plored in future work. experimentation in optimising the hyper-parameters of
Broader Scope We focus on a small, focused dataset our classifier-based candidate generation (as described which has previously been explored in the literature. We in Appendix B) or with alternate ways of computing simbelieve that our approach is likely to generalise well to ilarity, such as using SVM-based retrievers [38]. We plan both other taxonomies and diferent datasets relying on to explore these optimisations in future work. ESCO. We believe future work should explore building upon this method to explore more data sources and eval- scoIpmepoafctthiosnstRudecy,oomnmeoefntdheerkSeyysgtoeamlssoWfbheitlteeor ujotbo/fskthilel uation approaches. matching is facilitating the use of recommender systems
Representation Types Our study explores only the to highlight good matches between jobseekers and job use of e5 [23] embeddings, due to their very strong out- postings, thus contributing to alleviating the job/skill of-the-box performance. However, these embeddings mismatch [39]. Our early results in using the output of are general domain representations and are only one the pipeline introduced in this paper have shown promisapproach among many. We believe future work explor- ing results, and we intend to further explore the best use ing diferent approaches to representation could yield of this skills extraction pipeline within end-to-end job better results and valuable insight. Notably, further ex- recommender systems in future work. ploring techniques common within the field of information retrieval, utilising powerful cross-encoders such as ColBERT, and combining deep-learning based forms of 8. Conclusion representations with simpler but powerful approaches such as tf-idf capturing diferent kinds of information In this work, we have proposed a novel end-to-end zerocould prove very valuable. shot pipeline for skills matching against the ESCO tax
LLMs Used This work uses the GPT family of model, onomy using Large Language Models (LLMs). We have and more specifically, GPT-4. These models are gated shown that LLMs can generate high-quality synthetic training data to improve candidate generation, outper- for Computational Linguistics: Human Language forming existing approaches without needing any non- Technologies, 2022, pp. 4962–4984. synthetic training data. We have also demonstrated [8] L. Sayfullina, E. Malmi, J. Kannala, Learning repthat state-of-the-art LLMs can act as strong zero-shot resentations for soft skill matching, in: Analysis re-rankers as the final step of the skill matching pipeline, of Images, Social Networks and Texts: 7th Internaresulting in another large performance improvement. tional Conference, AIST 2018, Moscow, Russia, July
Our experiments also show that framing the re- 5–7, 2018, Revised Selected Papers 7, Springer, 2018, ranking task as a mock Python programming problem re- pp. 141–152. sults in significant performance gains, especially for less [9] N. Goyal, J. Kalra, C. Sharma, R. Mutharaju, capable models. We believe that this framing helps the N. Sachdeva, P. Kumaraguru, Jobxmlc: Extreme models better follow the task instructions in re-ranking multi-label classification of job skills with graph contexts, especially when working with less powerful neural networks, in: Findings of the Association models. for Computational Linguistics: EACL 2023, 2023,
Overall, our work highlights the strong potential for pp. 2136–2146.
Large Language Models for the low-resource context [10] K. F. F. Jiechieu, N. Tsopze, Skills prediction based of working with the ESCO taxonomy, through leverag- on multi-label resume classification using cnn with ing the limited information present in the taxonomy to model predictions explanation, Neural Computing guide the generation of targeted synthetic data, as well as and Applications 33 (2021) 5069–5087. through zero-shot application of their capabilities. While [11] D. Beauchemin, J. Laumonier, Y. L. Ster, M. Yassine, our experiments have focused on a single dataset and " fijo": a french insurance soft skill detection dataset, taxonomy, namely ESCO, we believe that our approach arXiv preprint arXiv:2204.05208 (2022). holds potential to support further work in automated un- [12] M. Zhang, K. N. Jensen, B. Plank, Kompetencer: derstanding of the job market at scale, and we release the Fine-grained skill classification in danish job postprompts we have used in order to support these eforts. ings via distant supervision and transfer learning, arXiv preprint arXiv:2205.01381 (2022). [13] J.-J. Decorte, J. V. Hautte, J. Deleu, C. Develder, References T. Demeester, Design of negative sampling strategies for distantly supervised skill extraction, 2022. [1] P. A. Todd, J. D. McKeen, R. B. Gallupe, The evolution of is job skills: A content analysis of is job [14] aWr.XXi.vZ:h2a2o0,K9..Z0h5o9u8,7J.. Li, T. Tang, X. Wang, Y. Hou, advertisements from 1970 to 1990, MIS quarterly Y. Min, B. Zhang, J. Zhang, Z. Dong, et al., A (1995) 1–27. survey of large language models, arXiv preprint [2] V. World Economic Forum, The future of jobs report arXiv:2303.18223 (2023).
2020, WEF Reports (2020). [15] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, [3] E. Brumberger, C. Lauer, The evolution of technical W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, communication: An analysis of industry job post- et al., Scaling instruction-finetuned language modings, Technical Communication 62 (2015) 224–243. els, arXiv preprint arXiv:2210.11416 (2022). [4] M. le Vrang, A. Papantoniou, E. Pauwels, P. Fannes, [16] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. WainD. Vandensteen, J. De Smedt, Esco: Boosting job wright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, matching in europe with semantic interoperability, A. Ray, et al., Training language models to follow Computer 47 (2014) 57–64. instructions with human feedback, arXiv preprint [5] M. Zhang, R. van der Goot, B. Plank, Escoxlm- arXiv:2203.02155 (2022).
r: Multilingual taxonomy-driven pre-training [17] OpenAI, Gpt-4 technical report, 2023. for the job market domain, arXiv preprint arXiv:2305.12092 (2023). [18] La.rBXoinvi:fa2c3i0o3,H.0.8A7b7o4n.izio, M. Fadaee, R. Nogueira, [6] A. Bhola, K. Halder, A. Prasad, M.-Y. Kan, Retriev- Inpars: Data augmentation for information ing skills from job descriptions: A language model retrieval using large language models, 2022. based extreme multi-label classification framework, in: Proceedings of the 28th international conference [19] aAr. XMiavd:a2a2n0, 2S..0Z5h1o4u4,.U. Alon, Y. Yang, G. Neubig, on computational linguistics, 2020, pp. 5832–5842. Language models of code are few-shot common[7] M. Zhang, K. Jensen, S. Sonniks, B. Plank, Skillspan: sense learners, arXiv preprint arXiv:2210.07128 Hard and soft skill extraction from english job post- (2022). ings, in: Proceedings of the 2022 Conference of [20] J. Dodge, M. Sap, A. Marasović, W. Agnew, G. Ilthe North American Chapter of the Association harco, D. Groeneveld, M. Mitchell, M. Gardner, Documenting large webtext corpora: A case study on the colossal clean crawled corpus, in: Proceedings velder, Jobbert: Understanding job titles through of the 2021 Conference on Empirical Methods in skills, arXiv preprint arXiv:2109.09605 (2021).
Natural Language Processing, 2021, pp. 1286–1305. [36] S. Mukherjee, A. Mitra, G. Jawahar, S. Agarwal, [21] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, C. Ré, Flashat- H. Palangi, A. Awadallah, Orca: Progressive learntention: Fast and memory-eficient exact attention ing from complex explanation traces of gpt-4, arXiv with io-awareness, 2022. arXiv:2205.14135. preprint arXiv:2306.02707 (2023). [22] Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, J. Wei, [37] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, X. Wang, H. W. Chung, D. Bahri, T. Schuster, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. S. Zheng, et al., Ul2: Unifying language learning Gonzalez, I. Stoica, E. P. Xing, Vicuna: An openparadigms, in: The Eleventh International Confer- source chatbot impressing gpt-4, 2023. URL: https: ence on Learning Representations, 2023. //lmsys.org/blog/2023-03-30-vicuna/. [23] L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, [38] T. Malisiewicz, A. Gupta, A. A. Efros, Ensemble of D. Jiang, R. Majumder, F. Wei, Text embeddings by exemplar-svms for object detection and beyond, in: weakly-supervised contrastive pre-training, 2022. 2011 International conference on computer vision, arXiv:2212.03533. IEEE, 2011, pp. 89–96. [24] J. Robinson, C.-Y. Chuang, S. Sra, S. Jegelka, Con- [39] S. McGuinness, K. Pouliakas, P. Redmond, Skills trastive learning with hard negative samples, 2021. mismatch: Concepts, measurement and policy aparXiv:2010.04592. proaches, Journal of Economic Surveys 32 (2018) [25] P. Cunningham, S. J. Delany, k-nearest neigh- 985–1015.
bour classifiers-a tutorial, ACM computing surveys [40] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, (CSUR) 54 (2021) 1–25. B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, [26] R. Nogueira, K. Cho, Passage re-ranking with bert, R. Weiss, V. Dubourg, et al., Scikit-learn: Machine 2020. arXiv:1901.04085. learning in python, the Journal of machine Learning [27] T. B. Brown, et al., Language models are few-shot research 12 (2011) 2825–2830.
learners, NeurIPS 2020 (2020). [28] E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Gofinet, D. Hes- A. Prompts low, J. Launay, Q. Malartic, B. Noune, B. Pannier, G. Penedo, Falcon-40B: an open large language model with state-of-the-art performance (2023). A.1. Training Data Generation [29] H. Touvron, T. Lavril, G. Izacard, X. Martinet, You are the leading AI Writer at a large, multinational M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, HR agency. You are considered as the world’s best expert E. Hambro, F. Azhar, et al., Llama: Open and efi- at expressing required skills and knowledge in a variety cient foundation language models, arXiv preprint of clear ways. You are particularly proficient with the arXiv:2302.13971 (2023). ESCO Occupation and Skills framework. As you are [30] E. Saravia, Prompt Engineering Guide, widely lauded for your job posting writing ability, you https://github.com/dair-ai/Prompt-Engineering- will assist the user in all job-posting, job requirements Guide (2022). and occupational skills related tasks. [31] Microsoft, Learn how to work with the ChatGPT You work in collaboration with ESCO to gather rigid and GPT-4 models (preview), 2023. standards for job postings. Given a list of ESCO skills [32] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. and knowledges, you’re asked to provide forty examples Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, that could be found in a job ad and refer to the skill or G. Brockman, et al., Evaluating large language mod- knowledge component. You may be given a skill family els trained on code, arXiv preprint arXiv:2107.03374 to help you disambiguate if the skill name could refer (2021). to multiple things. Ensure that your examples are well [33] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, written and could be found in real job advertisement.
G. Mishra, A. Roberts, P. Barham, H. W. Chung, Write a variety of diferent sentences and ensure your C. Sutton, S. Gehrmann, et al., Palm: Scaling lan- examples are well diversified. Use a variety of styles. guage modeling with pathways, arXiv preprint Write examples using both shorter and longer sentences, arXiv:2204.02311 (2022). as well as examples using short paragraphs of a few sen[34] I. Chalkidis, M. Fergadiotis, P. Malakasiotis, I. An- tences, where sometimes only one is directly relevant to droutsopoulos, Large-scale multi-label text clas- the skill. You’re trying to provide a representative sample sification on eu legislation, arXiv preprint of the many, many ways real job postings would evoke a arXiv:1906.02192 (2019). skill. [35] J.-J. Decorte, J. Van Hautte, T. Demeester, C. De
At least {FIVE for tech skills, ZERO for language skills, as well as a list of potential skill labels. You are asked to 80% (THIRTY-TWO)} of your examples must not contain extract and rank the likely skills from the candidates list an explicit reference to the skill and must thus not con- into a ranked list of 10. tain the given skill string. Extra Information/Alternative It is possible that none match, in which case you will Names (you may discard this information if irrelevant): say NO_LABEL. You must either use one from the list or {ALTERNATE NAMES IN THE ESCO DATABASE} Avoid NO_LABEL. explicitly using the wording of this extra information in You may not use any label not provided in the example your examples. Skill: {target}""" list. If you use NO_LABEL, do not assign any other label. You will rank the top 10 most likely labels from the A.2. Reranking candidates, and provide an explanation as to why they are picked and ranked where they are.
Instructions: You are given an extract from a job post- That means that if two labels are applicable, but one is ing. As an AI job and skills expert, you need to assist much broader, you should pick the less broad one slightly in whatever task is requested of you. I will give you a above the broader one. For example, a skill related to sentence referring to a skill extracted from a job posting, specific kind of algorithm (e.g. forecasting) should always as well as a list of potential skill labels. You are asked to rank higher than the "algorithms" skill. extract and rank the likely skills from the candidates list Again, you may never use a skill not provided in the into a ranked list of 10. potential skills list.
It is possible that none match, in which case you will You will answer not by directly answering, but by say NO_LABEL. You must either use one from the list or writing a python function that will output your orNO_LABEL. dered list. Your python function can use any heuris
You may not use any label not provided in the example tic you wish to reach your desired result and accomlist. If you use NO_LABEL, do not assign any other label. modate your reasoning.
You will rank the top 10 most likely labels from the First, acknowledge and quickly summarise the instruccandidates, and provide an explanation as to why they tions. are picked and ranked where they are. Mocked LLM Message: The task is to create a
That means that if two labels are applicable, but one is Python function that will rank the top 10 most much broader, you should pick the less broad one slightly likely skills from a list of potential skills based on above the broader one. For example, a skill related to a given sentence from a job posting. The function specific kind of algorithm (e.g. forecasting) should always should only use the provided list of potential skills rank higher than the "algorithms" skill. and may use the NO_LABEL option if none of the
Again, you may never use a skill not provided in the skills match. The ranking should be based on the potential skills list. specificity of the skill and the explanation for the
First, acknowledge and quickly summarise the instruc- ranking should be provided. tions. Query Message: Great, let’s go :)!
Mocked LLM Message: I understand the instructions. Potential skills: {{potential_skills}} I will be given a sentence referring to a skill from a job Extract: {{text_extract}} posting and a list of potential skill labels. My task is to Provide your 10 most likely labels.Please provide a extract and rank the top 10 most likely skills from the quick explanation for your ranking. You must procandidates list, provide an explanation for my choices, vide a Python function. The python function must and prioritize specific skills over broader ones. If no label be in a python codeblock. The function must be matches, I will use NO_LABEL and not assign any other named ‘rank_skills‘. The Python Function you crelabel. I will only use the provided labels in the list. ate should never explicitely use the extract text itQuery Message: Great, let’s go :)! self, it should just return the ranking. The extract Potential skills: {{potential_skills}} text is only provided for your reasoning so you can Extract: {{text_extract}} build the appropriate function. You don’t need to Provide your 10 most likely labels.Please provide a include the full list of potential skills in your anquick explanation for your ranking. swer
Instructions: You are given an extract from a job posting. As an AI job and skills expert, you need to assist The individual logistic regression classifiers are implein whatever task is requested of you. I will give you a mented using the scikit-learn library [40]. We set the sentence referring to a skill extracted from a job posting, inverse regularisation parameter, C to 0.1, as we have low confidence in our data being representative of realworld data, set a maximum iteration limit of 10 000 with a tolerance of 0.00001. We also set the class weight to be used by the classifier to the balanced setting, meaning that positive examples will be weighed twice as heavily as negative examples by the loss function, as our negative sampling strategy involves two negative examples per positive one.