industry. This paper investigates the feasibility of employing basic prompting systems for domain-specific language models. The study focuses on bureaucratic language and uses the recently introduced BureauBERTo model for experimentation. The experiments reveal that while further pre-trained models exhibit reduced robustness concerning general knowledge, they display greater adaptability in modeling domain-specific tasks, even under a zero-shot paradigm. This demonstrates the potential of leveraging simple prompting systems in specialized contexts, providing valuable insights both for research and curate in the fill mask task5][,1 where the model had to predict both random and in-domain masked words, we
CEUR ceur-ws.org
wide variety of downstream tas1k, s2,[3, inter alia].
Pre-trained Language Models (PLMs) have had a signwifi-anted to further inspect the domain lexical knowledge cant impact on Natural Language Processing (NLP), aancdquired by this model during the domain adaptation. the pre-train and fine-tune paradigm has become the prWe-e aimed at leveraging this knowledge to implement dominant approach for applying efective models ontawo classification tasks in the PA domain, modeled as prompt-based classification. Thus, we challenged the efective with Large Language Models (LLMs4)].[How
However, one of the main concerns when workinmgodel to predict both the topics of PA texts, and the type with PLMs is the paucity of annotated data, especialloyffgoerneric and PA-related named entities occurring in specific domains, required to fine-tune the additional classe-ntences extracted from administrative documents. sification layer on top of these models for downstreamWe conducted two prompting experiments for each tasks, such as classification. Recently, prompt-based tutna-sk. We first adopted the Italian name of the classificaing has started to afirm as a promising way to perfortmion classes as label words, then we associated in-domain similar tasks, significantly reducing the need for antneor-ms to each class. We also compared BureauBERTo tated data. This approach has been proven to be vwerityh an Italian generic PLM, UmBERTo (Sect3i)o. n
Our findings show that in a zero-shot classification exploiting the prompt-based tuning technique. ever, it is often the case that LLMs are not availablesfcoenrario when the label words of each class are shallowly low-resource languages, and that their performancerderlaast-ed to the content of the text or to the entity type fed tically decreases when they are challenged on spetcoificthe model in the prompt template, both the generic domains. Hence, we decided to test a domain-specificand the domain-specialized models perform poorly in the model, BureauBERTo5[], a LM further pre-trained ocnlassification task. However, when the classes are repreItalian bureaucratic texts (e.g., administrative acts,sbeanntke-d by multiple word labels semantically related to the ing and insurance documents), in a zero-shot scenatreixot/entity to be classified, the PLMs improve their performance by a wide margin. This gaining is particularly evi
Since BureauBERTo has shown to be particularlydaecn-t in the domain-adapted model BureauBERTo, which 0009-0006-6846-5826 (S. Auriemma); 0009-0002-7844-3963 CEUR htp:/ceur-ws.org ISN1613-073 © 2023 Copyright for this paper by its authors. Use permitted under Creative
CEUR
Workshop ProceedingsC(EUR-WS.org) 0000-0003-3426-6643 (A. Bondielli)0;000-0001-5790-4308 (A. Lenci); compared to when the same task is accomplished via outperformed UmBERTo in both prompt-document classification and prompt-entity typing tasks, suggesting that the domain linguistic knowledge acquired by this model during the additional pre-training phase could be particularly useful in a prompt-based tuning scenario where the model is much more reliant on its word knowledge,
1See AppendixB for the plot of the model results in the fill-mask task
specific terms, to investigate how domain-related word labels afect the model’s performance in diferent classification tasks.
PLMs have proven to be efective in NLP tasks re3-. Models lated to specific domains, whether they were trained from scratch6,[ 7], or further pre-trained on domaFinor our experiments, we decided to compare the perdata 8[, 9, 10] with a Masked Language Modeling (MLMf).ormance of two PLMs, namely UmBERTo and BuMore recently, the MLM training objective has been lerveearu-BERTo. UmBERTo2 is a RoBERTa-based language aged to solve various NLP tasks reformulated as amsordtel trained on the Italian section of the OSCAR corof cloze task, allowing the PLM to directly solve itpwuist,3ht-hat has been shown to perform well on adminout any or with very few labelled examples. Oneisotfrative dat2a3][ compared to other generic PLMs of the first works in this direction was proposed1b1y],[ the same size (110M parameters). BureauBER4T[o5] is a who performed zero-shot learning using pre-trained LdMomsain-adapted model obtained by further pre-training without fine-tuning on a dataset of training exampUlemsB.ERTo on Italian PA, banking, and insurance docuWithin similar conditions, but using the larger GPmTe-n3,ts. [4] achieved near state-of-the-art results for some SuperGLUE [12] tasks. [13] showed that competitive perfor
4. Experimental settings mance with those of GPT-3 can be achieved with much smaller models like the 220M parameters ALBERT, by performing some gradient-based fine-tuning of the mod4el.1. Prompting using the labeled examples on a cloze task. Since thPerno,mpt-based classification requires a specific temprompt-based learning has gained attention as a simppllaete to reformulate the original classification task as a way to perform, among other tasks, zero-shot classicficlao- ze-task, where the text to be classified is fed to the tion 1[4]. However, it’s essential to note that the permfoodre-l followed by a prompt sentence, such as “This mance of prompt-based learning techniques scales w<ittehxt> is about [MASK]”. In this way, the model has model size [15]. Consequently, general purpose Largteo predict the probability that a certain word is filled Language Models (LLMs) with billions of parameteinrsthe “[MASK]” token. The mapping from the label are typically used in prompt-learning experiments, evceanndidate word to a specific class is gained through the for specialized domains such as the legal 1o6n]e. I[n verbalizer [13], which represents the original class names contrast, for the biomedical and clinical dom1a7]ins,a[s a set of label words, greatly influencing the model pershowed that smaller specialized models like BioB8E]RT f[ormance in the tas2k4[]. Hence, we decided to conduct and Clinical BERT18[] outperform GPT-2 and T5 in aour prompt-based classification experiments in two setfew-shot prompt based classification of medical texttisn.gs, using a standard and a cusvteormbalizer to better The authors hypothesize that the advantage of the BuEnRdTe-rstand the correlation between the lexical knowlbased models is possibly due both to their domain adeadpg-e of PLMs and the use of domain-related terms as the tation and to their bidirectional MLM training objescettivoef,word labels in the promveprtbalizer. which is more similar to the prompt template formaTthe first verbalizer, i.e., thbease-verbalizer simply uses than those of auto-regressive and sequence-to-sequetnhceeItalian name of the classification classes as label words models like GPT-2 and T5.1[9] reported a similar finding (e.g., Ambiente - “Environment” is the label word for the even for the much larger GPT-3 over BioBERT. Nevertchlea-ssAmbiente - “Environment”), while the second less, these approaches are constrained by the model invpeurtbalizer is maanual verbalizer that we constructed by size, which limits the length of the conditioning inadpudting some synonyms of the class name and some recontext and can significantly afect performan1c9e].[ lated PA terms as label words for each class, to better
Although prompt-based classification with specializdeedpict the document classes and the entity types (in this models has been explored for the medical and clincicaasle the label words for the cAlamsbsiente - “Environdomains, to the best of our knowledge, this is the firmsetnt” are: ambiente - “environment”, natura - “nature”, work that focuses on applying prompts to the Italianterardit-orio - “territory”, flora - “flora”,etc. ). ministrative language and in a zero-shot classification scenario. Additionally, a notable challenge in promptbased approaches lies in their sensitivity to variations in prompt templates and verbalize2r0s, 2[1, 22]. We conducted experiments using diferent verbalizers, i.e., a 2https://github.com/musixmatchresearch/umberto generic verbalizer and a custom verbalizer using domain3-https://oscar-corpus.com 4https://huggingface.co/colinglab/BureauBERTo 4.2. Datesets classification task as a masked language modeling problem: <text>.Questo documento parla di <mask>.7 We evaluate the models in two tasks on two diferentThus, PLMs are challenged to infer the topic of the datasets. For thperompt document classification , document by predicting the most appropriate label word we used a subset of the ATTO corpu2s3][, which is a to represent the masked token in the prompt, following collection of administrative documents annotatedtwheitdhocument text. Since the ATTO corpus contains labels denoting topics. We filtered this dataset keeping
only short documents of a maximum of 600 tokens, by only those instances (2,811) that were annotated wsitethtaing the tokenizer’s truncation at 5128t,owkeenwsere single topic label. able to feed the models the entire document in almost all
For theprompt entity typing task, we used the PA-cases. Like with the prompt entity typing, we perform the corpus of 2[5], a collection of 460 PA-documents with
prompt-based classification twice. In the first experiment, token-level annotations of Named Entities denoting botuhsed thebase verbalizer, where each class is linked to we general entities, such as persons, locations, organizatoinoenso,r few label words that correspond to the names of and domain-specific entities, like legislative norms, acts,
the classes in the original annotation of the ATTO corpus. and PA-related organizations. For the second experiment, we use tmhaenual verbalizer, which contains, in addition to the label wordsboafsethe 4.3. Evaluation metrics verbalizer, a collection of domain terms manually selected We evaluated the performance of the models with caosmP-A representative topic labels for each class. The complete list of the label words used in both verbalizers mon classification metrics. is shown in Table1.9 4.4. Prompt entity typing
We modeled the NER task introduced2b5y][as an entity typing task. Entity typing can be considered a subtTaasbkle2 shows the results of prompting applied to the of NER and focuses on entity classification. In otheenrtity typing task. words, systems assign a label to an already extractIendthe first experiment, where a single class label is entity. This task is often formulated to challenge systeumsesd (see Sec.4.4), UmBERTo almost doubled the results at retrieving sub-categories organized in a hierarcohbitcaailned by BureauBERTo for F1 Micro (0.404 vs. 0.263) structure (e.g., an entity corresponding to a personamnady Macro Average (0.335 vs. 0.201). Surprisingly, for be specified as director, major, lawyer, etc.) As 2in5][, a domain entity likAeCT, BureauBERTo missed all the we asked models to identify only coarse-grained entietnietsi:ties, whereas UmBERTo obtained a low but higher generic ones, such as personPsE(R), locationsL(OC), and score (0.140). For thLeAW entity, UmBERTo overpasses organizationsO(RG); and related to the administratBivuereauBERTo, as well. We may suppose that this is due to domain: law referenceLsA(W), administrative actAsC(T), the fact that UmBERTo was trained on Common-Crawl, and PA organizationOsP(A). which also contains legal and administrative texts in its
We prompted the models by giving as input a sentenIctealian section. Very high results are obtained by Umand an entity occurring in it, asking to predict the eBnEtRitTyo forPER entities, reaching 0.827 in our zero-shot type in place of a masking token. The resulting tescme-nario. On the contrary, both models obtain very low plate is:<text>. In questa frase, <entity> è un results foLrOC, OPA, andORG. These two latter classes esempio di <mask>.5 are very similar to each other: ORG refers to organi
As anticipated, we verbalized the entities in two wzaaytsi.ons in general, comprising firms and associations, In the first experiment, we provided an Italian translatwiohnereasOPA can be considered as a subclassOoRfG, of the entity or a single word representing the entityacnldarsse.fers to organizations within the Public AdministraIn the second experiment, we expanded most of the labteioln, such as municipal departments. Such overlapping words by including synonyms and other terms relatedmatyo impact on classification. the various class6es. For what concerns the second experiment, we added to the prompt also highly distinctive words for each class. 4.5. Prompt document classification In this case, we notice a better ability of BureauBERTo to recognize domain-specific entities suchAaCsT, LAW, For the recognition of the topics in PA documents, we designed the following template to model the document 7In English:<text>. This document is about <mask>.
8512 is the maximum number of tokens that these Transformers 5In English:<text>. In this sentence, <entity> is an models can receive as input. example of <mask>. 9See AppendixA for the English translation of the label words 6Both verbalizers for entity typing are in AppAen.dix for document classification.
Basic Labels +In-domain Lexicon ambiente, natura, territorio, flora, fauna, animali, clima, inquinamento, rifiuti, igiene, caccia, pesca, verde, ecologia, agricoltura, acque avvocatura, avvocati, giustizia, legale, ricorso, giudici, Tribunale, Corte, Appello,
Assise, notifica, atti, Albo, Pretorio, protocollo Bandi-Contratti
bandi, contratti, bando, contratto, gara, appalto, assunzione, liquidazione commercio, economia, attività, economica, beni, commerciare, vendite, acquisti, commercianti, confesercenti cultura, turismo, sport, culturale, turisti, musei, arte, cinema, vacanze, spettacolo, scuola, manifestazioni demografico, popolazione, abitanti, residenti, censimento, anagrafe, residenza, domicilio, cittadinanza, leva edilizia, costruzioni, cantiere, ristrutturazione, planimetrie, residenziale personale, risorse, umane, assunzioni, lavoro, part-time servizi, informazioni, informativi finanza, euro, finanziario, contabilità, contabile, copertura, rimborsi, pagamenti, versamenti, bilancio, spese, sanzioni, multe, tributi, retribuzioni, emolumenti sociale, leva, militare, disabili, protezione, civile, invalidi Pubblica-Istruzione
istruzione, istituto, scolatisco, scuola, insegnante, formazione, educazione urbanistica
urbanistica, trasporti, trasporto, trafico, circolazione, veicoli, viabilità, viaria and OPA. However, despite the general improvemenatdded for thPeER entity class, i.eg.eneralità - “particuin recognizing such classes, we notice that it perfolarrms”s andnominativo - “name”. worse than UmBERTo for traditional entities. This eTxh- e results in Tabl3eshow that the performance of periment based on the comparison of general-purpoUsmeBERTo increases not only for tPhEeR entities but language models and domain-adapted ones has yieldtehdat the ablation improves the F1-score oOfRtGhcelass compelling insights. Generally, both types of modaeslswell. Whereas UmBERTo reaches the highest perdemonstrate enhanced performance when enriched wfiotrhmances for overall F1 Micro Avg, the deletion of indomain-specific terms within their prompts. However, idtomain lexicon from the verbalizer seems to penalize is evident that the domain-adapted model outperfoBrumrseauBERTo in the recognitionPoEfR entities. Folthe general-purpose model, exhibiting an improvemelnotwing the trend observed in UmBERTo, the ablation of more than twofold (0.516 vs 0.368 for Macro Averaimgpeacts the model’s ability to properly recognize the F1 score). This significant boost in performance sugo-ther classes. Despite this, the adapted model still obgests that the domain-adapted model is likely to be mtoariened higher results on the in-domain entity classes: attuned and proficient in leveraging domain-specific teAr-CT, LAW, andOPA further solidifying the advantages of minology. domain-adapted models in specialized contexts. Finally,
Nevertheless, it is important to acknowledge tithaistworth noting that we observed a high variability domain-specific terms may wield less influence over of results according to diferent prompts and verbalizer generic entities such PaEsR. With the in-domain lex-configurations, as shown in the ablation study. In fact, icon added to the verbalizer, UmBERTo fails to recogdneizleting the in-domain lexicon related to one of the entity anyPER entity. By looking at the confusion matrixcfloarsses afected the performance achieved by the models UmBERTo, we observed that the model identifies almosotn all the others, due to wrong classifications (e.g., peoall the people’s names aOsRG entities. Thus, we carriedple names confused with location addresses or company out an ablation study by deleting the in-domain tenrammses). Therefore, future investigations into prompt tuning are necessary and can lead to further interetsytpiensgoccurring in administrative sentences. insights. We compared the results obtained in these two tasks
Regarding the prompt document classification experbiy- the PA-specialized model BureauBERTo with those ments, whose results are summarized in ta4,bwlee ob- of the domain-agnostic model UmBERTo. Our findings served a similar trend. When only one or few wosrhdow that by enriching with domain terms the set of labels are used to represent a topic class, both the gewnoeridclabels encoded in the promveprtbalizer both modand the domain-specialized models obtained a ratherelloswdemonstrated enhanced performances. Moreover, accuracy (0.22 vs. 0.09) and Macro Average F1 scoreBsureauBERTo exhibited an improvement over UmBERTo (0.16 vs. 0.06). In this case, UmBERTo outperformed Buo-f +0.06 Weighted Average F1 score in the document clasreauBERTo in almost all classes, with the exceptiosnificoaftion (0.51 vs. 0.57) and of more than twofold in the Cultura, Turismo e Sport - ‘Culture, tourism, and entity typing task (0.516 vs. 0.368 for Macro Average F1 sports’, Demografico - ‘Demographics’, andPerson- score), meaning that the domain adapted model is more ale - ‘Personnel’. Looking into the details of the scorpersoficient in leveraging domain-specific terminology. obtained by UmBERTo in its most recognizable classesThese results underscore the importance of tailoring ( Pubblica istruzione - ‘Public Education’, Edilizia language models to specific domains to unlock their full - ‘Constructions’ andUrbanistica - ‘Urban plan- potential and address the nuanced challenges posed by ning’), we speculate that the single-word labels usedivtoerse subject matters. However, it is also worth mendefine these classes provided a suficient cue to enable thteioning that we noticed a high variability in the task model to appropriately recognize these topics. This irseisnults according to diferent prompting and diferent line with the fact that the UmBERTo pre-training colrapbuesl words. In particular, when the label words adopted included texts extracted from Italian municipalitiest’owdebepict a certain topic class are, within the domain conpages, which often refer to such topics. text, semantically related to the label words of another
On the other hand, in the second experiment, whecrlaess, the models’ classification output seems to be biased we manually added to the promvpetrbalizer a set of in favor of one of the two classes. salient PA-related terms to depict the document topIincsconclusion, our study underscores the critical need at a finer-grained level, we observed a significant imf-or a thorough exploration of prompt engineering, parprovement in the overall performance of both modteiclsu.larly in the context of the entity typing task. This The benefits of a custom-made set of domain-relateimdperative arises not only from the potential to augment terms are particularly evident for the specialized mtohdeeplredictive capabilities of models, but also from the BureuBERTo, which reached a better accuracy (0.60nvese.d to consolidate the knowledge related to general en0.54) and Weighted Average F1-score (0.57 vs. 0.51) thatnity classes. Notably, the Public Administration (PA) doUmBERTo. It appears that the model adapted to themdaoi-n exhibits distinctive characteristics, both in terms of main may possess heightened sensitivity, enabling itrteoferencing entity names within documents and employefectively capitalize on the contextual cues ofered binyg domain-specific terminology. Notably, the identified domain-specific terms. However, by performing a classp-atterns within the PA domain deviate from the broader, wise comparison between the two experimental settinggens,eral-purpose Italian style, indicating the necessity for we observed that for some classes that shared a commtaoinlored, domain-specific prompt experimentation. domain lexicon, such aPsubblica Istruzione - ‘Public This investigative efort shed the linguistic intricacies Education’ andCultura, Turismo e Sport - ‘Culture, that exert an impact on Transformer model performance. Tourism, and Sports’, orServizi finanziari - ‘Finan- Our findings, as revealed in the ablation study on entity cial services’ andBandi e Contratti - ‘Tenders and linking, emphasize the pivotal importance of delving into Contracts’ the models’ classification could have beenthe interplay among diferent entity classes present in influenced in favor of one of the two classes, due to thdeaitrasets. A nuanced analysis of how these classes interact topic descriptor lexical overlap. These findings confirmand potentially overlap is indispensable for honing the the necessity of further inquiry into the efect of lexmiocdaell’s ability to distinguish between them in a domainspecificity on prompt-based classifications, especially fosrpecific context. domain-adapted models. To conclude, this leads us to surmise as a future direction for our work a further inspection of how domainadapted PLMs encode in their embedding the semantics 6. Conclusion and future work of domain-related terms and how this information relates to their performance in prompt-based tasks.
In this paper, we propose a zero-shot prompt tuning classification approach for solving two tasks related to the Italian PA domain: the classification of documents according to their topic and the recognition of the entity
P R F1 P R F1 P R F1 P R F1 P R F1 P R F1
0.455
0.618
0.524
Associates, Inc., 2019. URLh: ttps://proceedings.ne[24] T. Gao, A. Fisch, D. Chen, Making pre-trained
lanu
URL: https://aclantholo
shot learning paradigm for clinical natural language processing., in: AMIA... Annual Symposium proceedings. AMIA Symposium, volume 2022, 2022, pp.
972–981. [18] E. Alsentzer, J. R. Murphy, W. Boag, W.-H. Weng,
D. Jin, T. Naumann, M. McDermott, Publicly available clinical bert embeddings, arXiv preprint arXiv:1904.03323 (2019). [19] M. Moradi, K. Blagec, F. Haberl, M. Samwald, Gpt-3 models are poor few-shot learners in the biomedical domain, arXiv preprint arXiv:2109.02555 (2021). [20] E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang,
L. Wang, W. Chen, et al., Lora: Low-rank adaptation of large language models, in: International
Conference on Learnin
Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 8086–8098. [22] D. Trautmann, A. Petrova, F. Schilder, Legal prompt engineering for multilingual legal judgement prediction, arXiv preprint arXiv:2212.02199 (2022). [23] S. Auriemma, M. Miliani, A. Bondielli, L. C. Passaro, A. Lenci, Evaluating pre-trained transformers on italian administrative texts, in: Proceedings of 1st Workshop AIxPA (co-located with AIxIA 2022), 2022. persona (person), generalità (particulars), nominativo (name) luogo (place), località (locality) organizzazione (organization), azienda (firm), società (corporation), associazione (association), compagnia (company) legge (law), norma (rule), decreto (decree), legislativo (legislative) atto (act), delibera (resolution), determina (decision), deliberazione (deliberation), regolamento (regulation) uficio (ofice) environment, nature, land, flora, fauna, animals, climate, pollution, waste, hygiene, hunting, fishing, green, ecology, agriculture, water advocacy, attorneys, justice, legal, appeal, judges, courthouse, court, appello, assise, notification, acts, albo, pretorio, protocol tenders, contracts, notice, contract, tender, hiring, liquidation trade, economy, business, economic, goods, trade, sales, purchases, merchants, confesercenti culture, tourism, sports, cultural, tourists, museums, art, cinema, vacations, entertainment, school, events demographics, population, inhabitants, residents, census, registry, residence, domicile, citizenship, conscription building, construction, yard, renovation, planimetry, residential personnel, resources, human, hiring, work, part-time education, institute, school, teacher, training, education services, information, informative finance, euro, financial, accounting, accountant, coverage, refunds, payments, disbursements, budget, expenses, penalties, fines, taxes, wages, emoluments welfare, conscription, military, disabled, protection, civilian, disability urban planning, transportation, transports, trafic, circulation, vehicles, roadway