The increased digitization of the labour market has given researchers, educators, and companies the means to analyze and better understand the labour market. However, labour market resources, although available in high volumes, tend to be unstructured, and as such, research towards methodologies for the identification, linking, and extraction of entities becomes more and more important. Against the backdrop of this quest for better labour market representations, resource constraints and the unavailability of large-scale annotated data cause a reliance on human domain experts. We demonstrate the efectiveness of prompt-based tuning of pre-trained language models (PLM) in labour market specific applications. Our results indicate that cost-eficient methods such as PTR and instruction tuning without exemplars can significantly increase the performance of PLMs on downstream labour market applications without introducing additional model layers, manual annotations, and data augmentation.
dations [
Many of these approaches rely on supervised learning,
The increasing availability of raw labour market infor- where a commonly identified limitation in literature is
mation allows businesses, educational facilities and job the availability of multilingual, labour market and
taskseekers to gain a clear and more complete understanding specific datasets. In addition, due to the dynamic nature
of the labour market [
On the one hand, the majority of available data is of an onomies); updating and maintaining these knowledge unstructured nature, with a lack of large-scale annotated structures is typically done by human domain experts, datasets that could be used in training, and/or fine tun- making them time- and resource-intensive, and meaning ing of models for downstream applications. On the other that whenever such a structure is updated, datasets for hand, there is much efort in creating structured repre- supervised learning may become obsolete. sentations of labour market data, through taxonomies In this paper, we propose a novel method that relies and ontologies such as ESCO,1 ISCO,2 or O*NET.3 on pretrained language models (PLMs), prompt tuning
Leveraging structured ontologies to enrich and inter- with rules (PTR) [10], and the structured multilingual
pret unstructured labour market data has considerable ESCO taxonomy, to eficiently and cheaply generate large
research attention, through skill and occupation recog- amounts of labeled data for learning a variety of
downnition, classification, and linking [
In this paper, we aim to address the following research questions:
ing learned behavior to labour market specific
applications? the masked language modelling (MLM) pre-training
ob2. Does instruction, and sub-prompt finetuning a jective, the authors also introduce the so-called ESCO
PLM on a mixture of task-specific (i.e. general and Relation Prediction (ERP) task to internalize knowledge
labour market specific) datasets increase the per- of non-hierarchical relations within ESCO [
In this paper, we demonstrate domain-specific prompt- Language-Knowledge Graph Pretraining (DRAGON) probased tuning and its efect on the performance of skill posed by Yasunaga et al. [21]. Results from the paper extraction, occupation classification, link prediction, and indicate that DRAGON outperforms existing LM and entity linking tasks. We propose leveraging instruction LM+KG models on diverse downstream tasks in particutuning without exemplars (i.e. no examples at infer- lar on complex reasoning about language and knowledge. ence time) and sub-prompts for a more cost-eficient Despite the success of fine-tuning PLMs, there is a big solution for downstream labour market applications gap between the MLM objective and fine-tuning objec[11, 12, 13, 10]. We provide manually constructed tem- tives for downstream applications. Prompt-based learnplates that encode the knowledge embedded in the ESCO ing has been a widely explored method that uses temoccupation and skill taxonomies. We benchmark difer- plates to transform the input into classification probent configurations of finetuning the PLMs, to demon- lems, and as such, closes the gap between task-specific strate the efectiveness of e.g., adding instructions or and MLM objectives [22]. [23] propose KnowPrompt, a sub-prompts. method for task-oriented prompt template construction where they use special markers to highlight entity men2. Related Work tions in the template. [24] also proposed a template-based NER model using BART. The model enumerates all posRecent successes of PLMs such as GPT [14], BERT [15], sible text spans and considers the generation probability RoBERTA [16] and T5 [17] have demonstrated the use- of each type within manually crafted templates [22, 24]. fullness and adaptability of the transformer architecture. Since the manual creation of templates is labourAlthough these PLMs can capture rich knowledge from intensive, methods for the automated generation of massive corpora, a fine-tuning process with extra task- prompts and labels are well-researched. In principle, a specific data is still required to transfer their knowledge prompt consists of a template and label words. As such, for downstream tasks. Besides fine-tuning language mod- Schick and Schütze [13] first searches the label word els for specific tasks, recent studies have explored better space for the manually created templates. Next, gradientoptimization and regularization techniques to improve guided search automatically generates both templates ifne-tuning. and label words. Compared to human-picked prompts,
Several works try to integrate ontological and/ or taxo- most auto-generated prompts cannot achieve comparable nomical knowledge into task-specific models, to improve performance [10]. the performance of downstream applications. Take the Prior literature has shown that increasing the numwork by [18], who introduced KnowBert, a methodology ber of tasks in finetuning improves the generalization to that explicitly models entity spans in the input text. They unseen tasks [11]. Experiments from Chung et al. [11] further use an entity linker to retrieve relevant embed- show that "instruction finetuning" scales well with the dings of the entity from a knowledge base to enhance number of tasks and the size of the model. Wei et al. their representations. Another approach would be the [25] further suggests that instruction-tuned models rework by [19], the so-called TransE model, that focused pri- spond better to continuous outputs from prompt tuning. marily on representing hierarchical relationships. Similar Prompt tuning on FLAN even achieves more than 10% to our work, TransE models multi-relational data from improvement over a non-instruction-tuned equivalent knowledge bases (i.e. triplestores) to improve perfor- model [25]. mance for link prediction [19].
[20] took a diferent approach, proposing ERNIE, a 3. Methodology
method that consists of two stacked modules, namely the
T-Encoder responsible for capturing lexical and
syntactic information, and the K-Encoder responsible for aug- 3.1. Preliminaries
menting this lexical and syntactical information with ex- 3.1.1. ESCO
tra token-oriented knowledge from the underlying layer
[20]. Lastly, we have ESCOXLM-R that employ further ESCO (European Skills, Competences, Qualifications and
pre-training on the ESCO taxonomy [
is an optional skill for is an essential skill for ....
skill .... occupation MLM Head
MLM Head
MLM Head the [MASK] ensure correct metal temperature [MASK] the [MASK] electron beam welder
Craft and related trades workers is-a
Metal, machinery and related trades workers butt welder is-a
welder atomic welder is-a electron beam welder alternativeLabel isOptionalSkill ensure correct metal
temperature isEssentialSkill wear appropriate protective gear spot metal imperfections
framework allows us to directly apply the same model, objective, training procedure, and decoding process to of skills, competences and occupations. In total, ESCO every task we consider [17]. In addition, we turn to describes 3,008 occupations and 13,980 knowledge, skill, an instruction-tuned variant of the T5 model: FLANand competences in 28 diferent languages. T5. Instruction-based finetuning has shown to improve
ESCO has both hierarchical and non-hierarchical re- zeroshot performance on unseen tasks [25, 11]. In this
lationships: hierarchical relationships, or hypernymies, paper we aim to study whether this property also applies
are relations of the form x is-a y [
PTR extends this prompt tuning approach with prior knowledge encoding, i.e., leveraging logic rules to encode prior knowledge about tasks and classes into prompt tuning, and eficient prompt design, through composing multiple sub-prompts and combining into prompts [10].
Q: Answer the following with yes/no Does "Assign and manage staff tasks in areas such as scoring, arranging, copying music and vocal coaching." describe "manage musical staff"?
the ESCO taxonomy to construct and populate subprompts, as proposed by Han et al. [10]. A: [MASK]
Consider a (sub-)prompt template for entity type classiifcation, as; "[CLS] the [MASK] [ENTITY]." Which can be instantiated for the skill "ensure correct metal temperature," as: "[CLS] the [MASK] ensure correct metal temperature," and for the occupation "electron beam welder," as: "[CLS] the [MASK] electron Figure 2: Visual representation of method: Instruction tuning beam welder." for the QA examples. The instruction is prepended to the
Finally, we can combine the above instantiations question, instructing the PLM how to proceed in answering of the same sub-prompt into a final prompt, that the given question. spans entity type and relation classification, as such: "[CLS] the [MASK]1 ensure correct metal temperature [MASK]2 the [MASK]3 electron compute [11]. Furthermore, Chung et al. [11] demonbeam welder". strate that smaller models that are instruction tuned can
PTR relies on so-called "verbalizers" that map class outperform larger models without it. labels to label words. In our example, the class labels Figure 2 demonstrates how we leverage the ESCO tax{"skill", "occupation"} for entity classification are mapped onomy to construct instruction tuning templates for the to (the same) label words {"skill", "occupation"} in place of QA examples. [MASK]1 and [MASK]3, and the class label {"isEssentialSkill", "isOptionalSkill"} in place of [MASK]2 are mapped to the corresponding label words {"is an essential skill 4. Experimental Setup for", "is an optional skill for"} in the case of relation classification. The aim of this paper is to leverage prompt-based and i.e., []1 and []3 aim to assign an en- instruction-based finetuning, to cost-eficiently optimize tity class through predicting a label word from X, and PLMs performance on four downstream labour market []2 aims to classify the type of relation between tasks. As described in the previous section, we propose the two through label words Y. four diferent tasks for evaluation, namely: entity classiifcation (EC), relation classification (RC), entity linking (EL), and question answering (QA).
form certain tasks, by responding to instructions in natural language [25]. For two of our three datasets (i.e., the QA and EL), we manually constructed templates that result in natural language instructions that describe the task for that dataset to the PLM.
While scaling language model sizes seems to be a reliable predictor for improved model performance, it comes at the price of high compute. Therefore, development of compute-eficient techniques that improve performance at the cost of a relatively small amount of computational resources is important. Instruction-based finetuning improves performance of PLMs on evaluation benchmarks by up to 9.4%, requiring only 0.2% of the pre-training
labour market-specific downstream tasks through benchmark datasets we generate through populating handcrafted templates, with instances from the ESCO taxonomy.
We generate three datasets of prompts, that address four diferent tasks; i) entity classification (EC) and ii) relation classification (RC) as illustrated above (combined in a single set of prompts), in addition to iii) entity linking (EL), and iv) question answering (QA).
Construction of a self-supervised dataset comprises three diferent components; i) a subset of ESCO relations, ii) a template to map the triples associated to the relations # total # skills # occupations # essential # optional # altlabels # pos # neg
EC + RC 123,752 13,890 3,008 64,877 58,875
123,752 0 to (sub-)prompts, and iii) verbalizers that map class labels to label words. 4.1.1. Entity Classification + Relation
Classification
tion (EC + RC) dataset, we leverage the isEssentialFor and the isOptionalFor relations as found in ESCO. For both entity classification and relation classification, we largely follow the work by Han et al. [10], i.e., we extract all triples that have as subject a skill entity, the isEssentialFor or the isOptionalFor as predicate, and finally as object an occupation entity.
Formally, our triples look as follows: < , , >, (1) where ∈ { , }. The entity and relation classification template () is formalized as:
1 = ℎ [ ] [] 2 = ℎ [ ] [] where 1 [ ] 2 relation classifications we have not included the possibility of "no relation" and/ or "no entity", for the simple reason of self-supervision. While we fully believe these negative examples to be useful for better learning how to recognize entities and the relation connecting entities, they would require manual annotation, and as such fall beyond the scope of this research. 4.1.2. Entity Linking
relation in ESCO, i.e., our task is to map an entity surface form or entity mention (alternative label), to the canonical entity name (i.e., label).
We can formalize the entity linking task as the following triplestore: < , , >, (4) where ∈ the set of skill and occupation entities, and ∈ the set of skill and occupation mentions (i.e., alternative labels for the ESCO skill and occupation labels). Lastly, ∈ , meaning that the predicate can be either signalling that the mention is an alternative label or not an alternative for the given ESCO entity.
Given an entity, and a mention , we are interested in finding out what type of relation there exists between and . As such, we formalize the entity linking as a masked language problem via = [ ] .
In Figure 1, the blue boxes represent two examples of alternative labels for the occupation electron beam welder.
We formalize the template for our second set of prompts as:
e [MASK] m = → , (5) Lastly, we formulate two diferent verbalizers 1 and 2 = {is an synonym for, it not a synonym for} such that: 1 = 1 → 1, 2 = 2 → 2, where 1={’Occupation’, ’Skill’} and the accompanying label words 1 = {’occupation’, ’skill’}. Similarly, 2 = {’isOptionalFor’, ’isEssentialFor’}, and the label words 2 = {’is optional for’, ’is essential for’}.
Note that in our case, the verbalizers are one-to-one mappings, whereas in the PTR methodology, many-toone mappings are also supported. For the entity and
(2) stores, we also randomly sample negative examples by (3) randomly shufling the objects and subject of the positive triplestores and changing the predicate label to noAlternativeLabel. 4.1.3. Question Answering
as defined by Chung et al. [11] and Wei et al. [25]. Instructional templates prepend an instruction to the prompt. = {alternativeLabel, noAlternativeLabel}
"Answer the following with yes/no", instructing the PLM how to answer the question that follows.
The question and answering dataset is constructed with the descriptions of the entities in ESCO. As such, 4.2.3. Experiment 3: Multitask Learning we can construct a dataset as: from the original training set and development set, and evaluate the models on the original test set.
We propose using = {64, 128, 256} sets. < , {description} > (6)
We do so by fine-tuning the FLAN-T5 on all combinations of tasks from a single to the full set, i.e., we train FLAN-T5 on the RC+EC and consecutively on the EL and QA tasks. We then test the performance of the resulting model on all three data sets to identify whether, e.g., prompt tuning on EL can help performance on the QA dataset.
example in Figure 2. Where we first prepend the instruction "Q: Answer the following with yes/no" to the body "Does [description] describe [entity label]", to finish it of with "A: [MASK]".
The verbalizer then maps the {′′,′ ′} to the label words {′′,′ ′}.
We randomly sample correct examples, in addition to generating negative examples by randomly sampling a skill or occupation entity, and pairing this with a ran- 4.3. Implementation Details domly sampled description from the set of available descriptions, and tagging the label for the answer to be "no". Our model implementation relies on the HuggingFace, This results in a balanced dataset, with a fifty-fifty split PyTorch and OpenPrompt frameworks (albeit with some of positive and negative examples. customizations), proposed by Wolf et al. [27], Paszke et al. [28] and Ding et al. [29] respectively. 4.2. Experiments For the zero-shot approach of the first experiment, we turn to T5 and FLAN-T5, for which we use the imIn order to answer our research questions, we propose plementation by the original authors [17, 25, 11]. More the following experiments. specifically, we use the 3 billion parameter checkpoints as found on huggingface under the names; ’t5-3b’, and
To answer our second research question we adjust First, to better understand the labour market-specific the number of examples used for training the models by tasks that we propose, we first test of-the-shelve PLMs comparing diferent values for parameter (i.e., number in a zero-shot setting, using our own generated prompt of samples). We optimize our PTR and Instruction-based datasets for inference. ifnetuning models using AdamW, with a learning rate of
In addition, to test the hypothesis that FLAN-T5’s mul- respectively 3 − 5 and 2 − 5. Furthermore, we reset titask learning enables a better ability of learning addi- the weight decay on the normalization layers and bias. tional (domain-specific) tasks, in our first experiment we We fine-tune all models using batch size 32, and train directly compare of-the-shelve T5 and FLAN-T5 models, the PTR models for 10 epochs, whereas, we train the on each of our three datasets. instruction based finetuning models for only 5. The best model checkpoint is selected. 4.2.2. Experiment 2: K-shot Learning
between the of-the-shelve PLMs, we study the impact of few-shot learning to steer the best performing PLM from experiment 1 towards the domain-specific data and tasks, where we perform an ablation study on the number of examples () we use for few-shot learning.
This is motivated by a.o., Han et al. [10], who report comparable or even better results in the few-shot scenario than e.g., methods that inject special symbols to index the positions of entities and methods that inject both type information and special symbols. The authors sample training instances and validation instances per class
In order to systematically evaluate few-shot performance, we randomly pick K samples from the total dataset, and use the remaining data to sample evaluation sets. This sampling is done 9 times, each iteration we sample 512 random examples from the remaining data after the train/ test split. We report F1 scores averaged over 9 runs in addition to standard deviations (± ). We argue that sampling multiple splits gives a more robust measure of the actual performance.
Since the single EC+RC dataset contains two separate tasks, it is important to avoid contamination between the Model train and test sets. Therefore, after the initial division, we check all individual skill and occupation entities from the train set, and remove all relations in the test set that contain any of those entities. For the QA and EL training data the risk of contamination is mitigated through the train/test split (i.e., after the normal split unique entries belong either to the train or test set).
into how sample eficient the model is in learning the behavior. Larger models are in-general more sample eficient and as such require less examples to learn a particular behavior [30].
tiple tasks at once. Results of our ablation experiments are shown in Table 4, where we vary with models that are trained on all combinations of diferent train sets of prompts, which we evaluate on each of the three test set of prompts.
Here, we note that first, in some cases adding prompts for additional tasks increases performance for the original tasks, consider, e.g., the case for (testing on) EL, where adding QA prompts yields an F1-score of 97.61 (row 4, Table 4), and adding EC+RC prompts even gets performance up to 98.48 (row 5, Table 4), whereas the model tuned with EL prompts only, scores 95.17 F1 (row 3, Table 4).
However, this does not apply for QA nor EC+RC, where only tuning with respectively QA and EC+RC prompts yields the highest score, nor for the case of training on all additional prompts—these runs (bottom row in Table 4) do not outperform the best performing models tuned on one or two sets of prompts.
Overall, this indicates that multitask learning can contribute in some cases to increased performance.
0-shot learning, i.e., applied of the shelve for inference on our generated prompts.
First, we see that FLAN-T5 substantially outperforms the non instruction-based finetuned counterpart T5 on the QA (83.44 vs. 33.75 respectively) and EL tasks (57.38 5.3.1. Unseen task performance vs. 33.89 respectively), but slightly underperforms on the EC+RC tasks, at 44.54 for FLAN-T5 and 48.07 for the T5 Supporting these observations is the pattern around permodel. formance on unseen tasks, i.e., models tuned on (a) task(s)
A potential explanation for this might be the fact that that do not include the test task used for evaluation. ConFLAN-T5 is trained on a variety of entity classification sider, e.g., EL; models that have not seen any EL prompts tasks that do not involve skill and occupation entities (i.e., in their tuning stage, perform substantially worse with the primary focus is on person and organisation entities). 58.96 for EC+RC and QA, 57.40 for EC+RC, and 60.31 for As such, the learned patterns may interfere with the QA, versus between 95.17 and 98.48 for models that have PLMs ability to recognize skills and occupations. seen EL prompts.
Similar patterns are seen with the other tasks, where 5.2. Experiment 2: K-shot learning for EC+RC models that have not seen any EC+RC prompts perform between 45.05–47.68, and around 50.55 In Table 3 we show the performance diferences at difer- and 51.60 for models that have. For QA, we see that ent levels of in the fewshot learning scenario. models without QA prompts in tuning score between
First, we note how the performance of FLAN-T5 + 68.22–87.98, and models that have range from 93.24 to PTR substantially outperforms both T5 and FLAN-T5 94.23. from Table 2 with F1 scores between 50.42 and 51.60 However, increasing the number of tasks in tuning across diefrent values of , compared to 48.07 and 44.54 does increase performance for unseen tasks in two out of respectively for the zero-shot T5 and FLAN-T5. three cases: when testing on the EC+RC prompts, a model
Next, we see that diferent values of are optimal that combines QA and EL prompts in tuning scores 47.68, for diferent tasks; with maximum scores at = 128 and outperforms QA-only (45.93) and EL-only (45.04) for EC+RC and QA at 51.60 and 94.23 respectively, and a models. Similarly, for QA, combining EC+RC and EL maximum score of 98.06 for = 256 for EL. prompts yields an F1-score of 87.98, whereas EC+RConly yields 78.36, and EL-only a mere 68.22 F1.
Model ↓/ → FLAN-T5 + PTR FLAN-T5 + Instruction tuning
Train ↓ / Test → EC+RC QA EL sub-prompt finetuning a PLM on a mixture of task-specific EC+RC 51.60± .47 78.36± .86 57.40± .16 datasets could increase the performance on labour market QA 45.93± .26 94.23± .24 60.31± .12 specific benchmarks? , we performed experiment 2, where EL 45.04± .26 68.22± .70 95.17± .39 we varied our instruction samples for training our EC+RC, QA 51.34± .23 93.24± .21 58.96± .52 best-performing PLM: FLAN-T5. Results demonstrated EC+RC, EL 51.21± .45 87.98± .31 98.48± .29 that PTR-based finetuning with 128 examples leveraged QA, EL 47.68± .23 93.69± .24 97.61± .14 the best performance. Overall, this yielded an 7.06% perall 50.55± .70 94.10± .27 98.19± .32 formance increase over the zero-shot performance of FLAN-T5. Additionally, further scaling of the number of Table 4 examples, to 256, yielded only a 6.24% increase, suggestF1 scores of our previously best performing model: FLAN-T5 ing no further performance increases for further scaling with 128-shot learning, on the diferent combinations of tasks of the number of examples. Our results seem to indicate we propose. that using PTR with labour market specific examples yields improvements above and beyond the 1836 tasks
FLAN-T5 was tuned on.
Finally, models that are tuned on all tasks do not out- Lastly, we investigated the efects of transfer learning perform models tuned on two tasks in two out of three across labour market specific tasks. sets (only for QA does the full model perform better than Here, our results suggest that first, learning more tasks models trained on two tasks). does yield increased performance on new, unseen tasks. At the same time, the best-performing models often were 6. Discussion those that were trained on the evaluation task exclusively (for EC+RC and QA). Overall, unsurprisingly, directly Our paper explored three diferent questions. First, are learning the task at hand yields the best performing mod"out-of-the-box" PLMs capable of generalizing learned be- els, but the fact that multiple tasks improve performance havior to labour market specific applications? In order to for unseen tasks does suggest that the domain-specific answer this question, we created three self-supervised knowledge that the PLMs receive in the tuning stage, benchmarks from the ESCO taxonomy. do help solving the unseen task at hand. Prompt tun
To answer this question, we performed zero-shot com- ing on the QA and EL (i.e., instruction based finetuning) paring between T5 and the instruction-tuned FLAN-T5, examples lead to a 3.14% improvement on the EC + RC that has seen 1,836 additional tasks in prompts. Results task. Similarly, prompt tuning on the EC + RC and QA showed that FLAN-T5 substantially outperforms T5 on examples yielded a 1.58% increase in performance on the two labour market-specific tasks, with a 49.7% increase in EL task, with an overall 4.54% increase over the zero-shot F1 score for QA, and 23.5% for EL. However in the EC+RC scenario. A possible explanation, "the ability to recogtask where T5 outperforms FLAN-T5 by 3.53%. These nize whether an entity is an occupation or skill help ifndings confirm that overall, the instruction-tuned FLAN discriminate whether two entities are not synonymous". PLM benefits from having seen multiple tasks. The result However, training on all tasks did not seem to increase for the EC+RC task can be explained by "misleading" the overall performance on any of the tasks. We believe patterns learned from the more general finetuning on this is potentially caused by overlaps in learned behavior named entity recognition (i.e., recognition of "Persons" from these diferent labour market specific tasks and the and "Organizations", etc.). However, further investiga- 1836 tasks FLAN-T5 is already pre-trained on. tions and ablation studies on general task tuning and its exact influence on the performance is needed for a more 6.1. Implications definite answer.
On the second question, whether instruction and/or
Finetuning PLMs is often an efective transfer mechanism in NLP. However, an entire new model is often required for every task. Our results indicate that cost-eficient methods such as PTR and instruction-based finetuning can significantly increase the performance of PLMs on downstream labour market applications without introducing any additional model layers, manual annotations, and data augmentation.
Furthermore, our results suggest that while training on general tasks can increase the overall performance on labour market specific applications, providing the general models with labour market specific examples increases performance above and beyond the general finetuning.
should be considered. First, we only used one-to-one verbalizers between our classes and label words. Meaning that every class label is mapped to one respective label word. This would be a fruitful area for future research, e.g., occupation can also be rewritten as job, or work. Adding these alternatives to the label words will probably yield improved performance over the current one-to-one verbalizers.
Second, for the purpose of this initial exploration we focused primarily on binary classification tasks. As such, we did not incorporate the possibility for a non-existing relation in the PTR finetuning.
Third, while the underlying methods support multiple languages, we chose to conduct our experiments on English. In part because the descriptions used in the QA dataset are not complete for all 28 languages for which ESCO is available. A future study could assess the performance of PTR and instruction based finetuning without examples in other languages.
Lastly, this study primarily focused on the actual isEssentialFor and isOptionalFor relations as they were present in the ESCO taxonomy. As such, we did not implement the reversed and or negative relations, even though this was suggested to further increase performance.
In this study, we demonstrated that FLAN-T5 substantially outperforms T5 on the QA and EL tasks with respectively 49.7% and 23.5% F1 scores. However, on the remaining EC+RC task, T5 outperformed FLAN-T5 by 3.53%. Overall it seems that PLMs benefit from instruction based finetuning even on labour market specific benchmarks. However, if the task at hand is very diferent from the task at hand, it can potentially hurt performance, as demonstrated with the EC+RC tasks.
Furthermore, our results seem to indicate that using PTR with labour market specific examples yields improvements above and beyond the 1,836 tasks FLAN-T5 was tuned on. Unsurprisingly, directly learning the task at hand leads to the best performing models. But, results also show prompt tuning on other labour market specific tasks can improve performance on unseen tasks. For example, prompt tuning on EC+RC and QA improved the performance on the EL task with 1.58%, and prompt tuning on QA and EL improved the performance on the EC+RC task by 3.14%.
There are several limitations to the current study; i) we solely used one-to-one verbalizer, ii) we focused primarily on binary classification tasks, iii) we only focused on English, and lastly we only used relations actually present in the ESCO taxonomy, meaning that we did not implement the reversed relations. Future studies could address the limitations of this study by using incrementing the amount of used label words, adding negative and reversed relations, and using ESCO to construct parallel datasets for all available languages.