BureauBERTo: adapting UmBERTo to the Italian bureaucratic language

Abstract

In this work, we introduce BureauBERTo, the first transformer-based language model adapted to the Italian Public Administration (PA) and technical-bureaucratic domains. We further pre-trained the general-purpose Italian model UmBERTo on a corpus of PA, banking, and insurance documents, and we expanded UmBERTo's vocabulary with domain-specific terms. We show that BureauBERTo benefitted from the adaptation by comparing it with UmBERTo in both an intrinsic and extrinsic evaluation. The intrinsic evaluation has been conducted through specific fill-mask experiments. The extrinsic one has been faced with a named entity recognition task on one of the sub-domains in BureauBERTo.

1. Introduction

The use of artificial intelligence (AI) in the context of Public Administration (PA) serves a dual purpose: increasing the eficiency of public entities by expediting data management processes and ensuring greater transparency, al

Since their first appearance in 2017 [ 1], transformer

based models have been leveraged in many ways to create models adapted to specific domains and efective in ilarly, in the context of Italian PA, it could be advantageous to tailor a pre-trained model to such domain, as Italian administrative lingo diferentiates semantically ministrative lexicon is, indeed, characterized by extensive use of technicisms (e.g., ravvedimento operoso, imponibile, capitolato), some of which are directly derived from the legislative language, Latinisms (e.g., una tantum; pro capite), archaisms (e.g., testè, quantunque), neologisms and Anglicisms (e.g., governance, front-ofice ). Texts are also rich of abbreviations, acronyms, legislative references, and formulaic or stereotypical expressions, such Ital-IA 2023: 3rd National Conference on Artificial Intelligence, orga0009-0006-6846-5826 (S. Auriemma); 0009-0002-7844-3963

PA domain. Domain adaptive pre-training (DAPT) has

proven to be an efective technique to exploit of-the-shelf pre-trained models and obtain substantial gains in their performance on domain data simply by further training the model with domain texts [ 9 ]. Extending [ 8 ], we chose to additionally pre-train the general purpose model UmBERTo1 on administrative, banking, and insurance corpora,2 creating BureauBERTo, the first transformer

1https://huggingface.co/Musixmatch/

umberto-commoncrawl-cased-v1 and syntactically from standard Italian. The Italian ad- suite of NLP tools to automatically extract information (e.g., esternalizzare [to entrust a task to an external body]), to identify the best-performing model to adapt to the

2. Related Work

based model adapted to the Italian bureaucratic language diferent sub-sector of the PA. For the Construction sector, (Section 3). Before training, we expanded the model vo- [ 23 ] created ArchiBERTo, a multi-label sentence classifier cabulary with about 8k new domain terms selected as to individuate the sentences corresponding to the criteria the most frequent ones in the new corpus. and quality objectives required by the public appointing

In this work, we address the following questions: party in the Design Guidance Document (Documento di (i) What is the overlap among the vocabularies Indirizzo alla Progettazione, DIP). of our target technical-bureaucratic domains? We Despite the growing deployment of transformer-based estimated the overlap of the terms added to the Bu- models in the PA sector, a specific model for this doreauBERTo vocabulary occurring in texts related to the main is still missing. We, therefore, decided to create administrative, insurance, and banking domains (Sec- BureauBERTo, the first transformer model trained to untion 3.3). (ii) To what extent the vocabulary expan- derstand Italian bureaucratic language. sion is beneficial for the domain-adaptation of BureauBERTo? Did further pre-training afect the semantic representation of words? We compared Um- 3. BureauBERTo

BERTo and BureauBERTo accuracy in the fill-mask task

to assess the contribution of further pretraining (Section 4.1) (iii) What are the advantages of employing a domain-specific vs. a generic model in a downstream task? We evaluated BureauBERTo and Um

BERTo performances on Named Entity Recognition (NER) in the administrative domain (Section 4.2).

We initialized our model starting from UmBERTo, which

is the best generic model for handling administrative data [ 8 ]. UmBERTo is a cased Italian monolingual model based on RoBERTa [ 11 ]. It is trained using a SentencePiece tokenizer and Whole Word Masking on a large subset of the OSCAR corpus of approximately 70 GB of text.

We additionally trained UmBERTo with a MLM objective

(randomly masking 15% of the tokens), on a composite corpus containing PA, banking, and insurance documents.

Further details on the training corpus and procedure are

given in the next sections.

Pre-trained transformer-based models such as BERT [ 10 ] and its variants [ 11, 12, 13, 14 ] have achieved state-ofthe-art performances in several NLP downstream tasks, 3.1. The Bureau Corpus many of which included in generic benchmark datasets such as GLUE [ 15 ] or SQUAD [ 16 ]. However, generic We constructed the pre-training corpus, henceforth the models tend to under-perform specialized ones when Bureau Corpus, by selecting administrative, banking and applied to domain-related texts and tasks. This has insurance documents. The corpus consists mostly of led to the development of models trained on domain- administrative acts of several Italian municipalities (65% specific data, like the medical-scientific [ 2, 4 ] or legal of the whole corpus) collected from a Solr database as ifelds [ 5, 17, 18, 19 ]. Following [ 5 ], who proposed the a part of the project SEMPLICE.3 For the insurance and ifrst legal domain-specific BERT further pre-trained on banking domains, documents were collected within the English legal documents, [ 17 ] created ITALIAN-LEGAL project ABI2LE by domain experts, who provided us with BERT by additionally pre-training the Italian BERT ver- a collection of non-life insurance product information sion on civil law corpora. Their domain-adapted model sheets and banking public communications, circulars, achieved better results on NER for the legal domain and and provisions. the classification of sentences belonging to diferent sec- All documents were pre-processed by first removing tions of civil judgments. Another model fine-tuned on line breaks, typical of PA and insurance texts layout. We the Italian legal domain is LamBERTa [ 20 ], trained for then split documents into sentences using our customized retrieving the most pertinent civil code article to a given version of the Italian spaCy tokenizer. We added a list of legal query. Italian legal texts share with administrative exceptions to the tokenization rules of the spaCy model ones some linguistic features typical of the Italian bureau- containing acronyms and conventional abbreviations of cratic language that contribute to making the language the legal domain, released by [ 17 ] common to the PA of the Italian Public Administration rather complex and domain, and other abbreviations that we gathered from artificial [ 21 ]. To improve the accessibility to public infor- bank and insurance texts. Sentences containing OCR ermation in PA documents, [ 22 ] adapted a Neural Pairwise rors, special characters, excessive punctuation, or written Ranking Model based on BERT architecture to assess in foreign languages4 were filtered out. In addition, we the readability level of sentences extracted from Italian removed the whole document when sentences containing administrative texts.

Another peculiar aspect of the PA domain is that it

comprises several sub-domains, each corresponding to a

3SEMantic instruments for PubLIc administrators and CitizEns:

www.semplicepa.it

4https://github.com/saffsd/langid.py

3.2. Domain-adaptive pre-training

Vocabulary expansion To allow the model to better capture the domain lexicons, we expanded the vocabulary of BureauBERTo with new domain-specific tokens. We extracted from the Bureau Corpus 8,305 representative words by applying the TF-IDF to the whole corpus. These terms were added to the original 32,000 tokens UmBERTo vocabulary, thus resulting in a domain-specific tokenizer with 40,305 tokens and an expansion of the model size from 110M to 117M parameters. 0.3% recur solely in banking and insurance documents, respectively. However, even though most of the domain words derive from the PA language (92%), approximately half of them also occur (at least ten times) in insurance (50.7%) and banking (52.3%) documents (see Figure 1).6

Hence, the three domains share a rather significant portion of the lexicon, considering that this analysis does not take into account the common Italian vocabulary.

3.3. Lexical overlap among domains

Model input format Following the “full sentences” approach in [ 11 ], we constructed the input dataset by applying the BureauBERTo tokenizer to contiguous sen- We assessed the efectiveness of our domain adaptation tences from one or more documents, using the separating with an intrinsic evaluation measuring the model acspecial token after each sentence. Additionally, we shuf- curacy in predicting top- (where ∈ = {1, 3, 5, 10} ) lfed the documents to alternate texts pertaining to the candidates for random and in-domain masked words (Secthree sub-domains in the Bureau Corpus, and avoid ef- tion 4.1). We did not use Pseudo Log-Likelihood (PLL) fects akin to “catastrophic forgetting” [ 24 ]. as proposed in [ 26 ], because a meaningful comparison would require the use of the same tokenizer (i.e., the same Pre-training details The model was trained for 40 vocabulary) for both models [ 27 ] that would prevent the epochs, resulting in 17,400 steps with a batch size of ~8K5 adapted model from using the new terms added to its on a NVIDIA A100 GPU. We used a learning rate of 5e-5 vocabulary. along with an Adam optimizer (β1=0.9, β2 = 0.98) with We also performed an extrinsic evaluation by fineweight decay of 0.1 and a 0.06 warm up steps ratio. tuning the model on a PA-specialized NER task. In both cases, we compared the performance of BureauBERTo with that of UmBERTo (Section 4.2).

4. BureauBERTo Evaluation

To address the question of the lexical overlap among PA, 4.1. Fill-mask evaluation insurance, and banking in-domain words, we computed the percentage of the 8,305 tokens extracted via TF-IDF Datasets We evaluated BureauBERTo on the fill-mask from the Bureau Corpus, which belong to the three sub- task in each sub-domain in the Bureau Corpus. As for the domains. This analysis shows that 21.6% of the tokens PA one, we selected the ATTO corpus [ 8 ], a collection of are exclusive of the PA domain, while only 3.6% and 11,019 short administrative texts covering diferent PA

5Following [25], we used the “gradient accumulation” technique to

have a batch size not bound by the size of GPU memory.

6See Appendix A for the vocabulary overlap between all the sub

domain corpora used for adaptation and fill-mask evaluation. topics (e.g., Environment, Construction, Urban Planning, Table 2 Education, etc.). For the banking domain we selected a group of 1,262 documents from the dump received by domain experts within the project ABI2LE. These documents are similar to those included in the Bureau Corpus (e.g., public circulars, communications, etc). For the insurance domain, we tested the model on a sample of 319 information sheets concerning life insurance products. Experimental settings

In the first fill-masking evalu

ation, we used a pre-tokenizer7 that split the input into whole words according to white spaces and punctuation. PA - ATTO

We randomly masked one word per sentence, choosing

words composed of at least two alphabetic characters. We only scored sentences with more than five and at most 100 of such words. In the second evaluation, we masked only domain-specific words, chosen from three manually created lists of about 100-200 terms related to the three technical-bureaucratic domains. All sentences that contained at least one of these words were selected to be scored for this task. When a sentence included more than one of those domain-specific terms, the word to mask was chosen randomly.

Results and discussion masked word) and on the insurance dataset (+18,9%).8

Furthermore, the high performance in the insurance do

main, despite the fact that only about 12% of the training documents came from this domain, suggests that transfer learning took place from the PA domain, which covers the largest portion of the training corpus.

4.2. PA specialized NER

ORG Datasets

We fine-tuned BureauBERTo on the PA cor

pus in [ 28 ], which contains 460 documents from the Albo Pretorio Nazionale, annotated with standard NER entities (i.e., person, locations, and organizations), and in-domain classes: LAW (national legislation), ACT (PA acts), and (PA organizations, like city hall’s ofices). Following [ 28 ] and [ 8 ], we also evaluated the model on 25 documents from diferent municipalities to test the model behavior in dealing with diferent ways of indicating entities and diferent writing styles.

7https://huggingface.co/docs/tokenizers/api/pre-tokenizers

8The Appendix B includes a few samples of masked sentences for a

qualitative comparison of the predictions of the two models. UmBERTo (UmB.) and BureauBERTo (BB) results in the fillmask task. The percentages refer to how many times the masked word is predicted within the first candidates. On the left, the results when a random word (Random) is masked; on the right, when an in-domain term belonging to the vocabulary of both models (In-dom+in-voc.) is masked.

Experimental settings

We fine-tuned BureauBERTo

using the same PA corpus train, validation, and test split as [ 8 ], to make our results comparable. We, therefore, employed as baseline the results obtained on the same datasets by UmBERTo [ 8 ] and by INFORMed PA, a PAspecialized model implemented by [ 28 ] based on the

Stanford NER with a CRF as learning algorithm. Bu

reauBERTo was fine-tuned for 5 epochs with a learning rate of 2e-5 and a batch size of 4. Sentences were tokenized and then truncated at 512 tokens. The training was executed on a NVIDIA A100 GPU. class ORG Results and discussion

9The reported results in this section refer to F1 score.

INFORMed PA and BureauBERTo for LOC (+2.6%) and of BureauBERTo? Did further pre-training afect the sePER (+4.8%). This suggests that the domain adaptation mantic representation of words? BureauBERTo benefited did not provoke forgetting, since the adapted model is from the vocabulary extension since it performed betstill able to generalize in recognizing general-purpose ter than UmBERTo in the fill-mask task. Moreover, it entities. To conclude, we assessed the benefits of domain benefited from domain adaptation, which is evident by adaptation of UmBERTo only in the PA sub-domain. observing the higher performances in fill-masking for Nevertheless, we expect to observe an additional already-known terms; (iii) What are the advantages of improvement in its other sub-domains. Moreover, we employing a domain-specific vs. a generic model in a downexpect a further improvement in the results in more stream task? In a PA-specialized NER task, BureauBERTo complex tasks, possibly inspired by real-world scenarios, shows a gain in performance after the domain adaptation. where it is even more evident the advantage ofered by In future work, we plan to assess the benefits of dothe additional vocabulary entries. main adaptation in the other sub-domains and in other downstream tasks, specifically tailored to the examined technical-bureaucratic domains. To evaluate the perfor5. Conclusions and future work mance of BureauBERTo in real-world scenario, we aim at exploiting it in tasks required for implementing the In this paper we presented BureauBERTo, the first NLP tools provided by the ABI2LE project. Furthermore, transformer-based model adapted to the Italian bureau- we would like to test the model in tasks where generalcratic language. BureauBERTo was created by further purpose Italian transformer models were applied to bupre-training UmBERTo on documents belonging to the reaucratic texts, such as in readability [ 22 ] and in senPA, insurance, and banking domains. Coming back to tence classification [ 23 ], to compare the results achieved our initial questions, we showed that: (i) What is the before and after the domain adaptation performed in Buoverlap among the vocabularies of our target technical- reauBERTo. Finally, we plan to challenge our model to bureaucratic domains? the three domains share a signifi- solve tasks on a diferent, albeit close domain, such as cant portion of their lexicon; (ii) To what extent the vocab- the legal one. This will assess the transfer-learning capaulary expansion is beneficial for the domain-adaptation bilities of BureauBERTo to other bureaucratic domains.

Acknowledgments

This research has been funded by the Project “ABI2LE (Ability to Learning)”, funded by Regione Toscana (POR Fesr 2014-2020).

A. Vocabulary overlap

B. Examples of top-k candiates

We decided to report some examples of the results returned by UmBERTo and BureauBERTo in the fill-mask task. In particular, we wanted to observe more closely how the domain adaptation would afect the semantic knowledge related to in-domain words already present in UmBERTo vocabulary. We show five sentences where a word was masked and, for each of them, five candidates provided by both models. In Table 5 results obtained on the ATTO corpus, which belongs to the administrative domain, are reported. Table 6 shows the results achieved in the banking domain. Finally, Table 7 shows results regarding the insurance domain.

Examples of candidates returned by UmBERTo and BureauBERTo in the fill-mask task, where in-domain terms belonging to the vocabulary of both models where masked. The data (we chose the ATTO corpus) and the terms belong here to the Examples of candidates returned by UmBERTo and BureauBERTo in the fill-mask task, where in-domain terms belonging to the vocabulary of both models where masked. The data and the terms belong here to the banking domain.

BureauBERTo

UmBERTo Alle società di gestione e alle imprese d investimento extracomunitarie tali previsioni si applicano a condizione che… …Si ipotizzi che l’intermediario C (intermediario standardizzato) abbia erogato nel mese di agosto dell’Anno T-2 un mutuo per un importo di… 030) Sottogruppo: Banca d’Italia (cod.300); Si tratta di misure che impongono, ad esempio, restrizioni sulla durata massima dei finanziamenti o limiti al piano di operante. base B) Rivalutazione del capitale assicurato La Misura di Rivalutazione, se positiva, viene attribuita, al Capitale Assicurato, a partire dal 1°

Sentence Examples of candidates returned by UmBERTo and BureauBERTo in the fill-mask task, where in-domain terms belonging to the vocabulary of both models where masked. The data and the terms belong here to the insurance domain, concerning life

References

[1] A. Vaswani , N. Shazeer , N. Parmar , J. Uszkoreit , L. Jones , A. N. Gomez , Ł. Kaiser, I. Polosukhin , Attention is all you need , Advances in neural information processing systems ( 2017 ).

[2] J. Lee , W. Yoon , S. Kim , D. Kim , S. Kim , C. H. So , J. Kang , Biobert: a pre-trained biomedical language representation model for biomedical text mining , Bioinformatics ( 2020 ).

[3] E. Alsentzer , J. Murphy , W. Boag , W.-H. Weng , D. Jindi , T. Naumann , M. McDermott, Publicly available clinical bert embeddings , in: Proceedings of the 2nd Clinical NLP Workshop , 2019 .

[4] I. Beltagy , K. Lo , A. Cohan , Scibert: A pretrained language model for scientific text , in: Proceedings of the 2019 Conference on EMNLP-IJCNLP , 2019 .

[5] I. Chalkidis , M. Fergadiotis , P. Malakasiotis , N. Aletras , I. Androutsopoulos , Legal-bert: The muppets straight out of law school , in: Findings of the Association for Computational Linguistics: EMNLP 2020 , 2020 , pp. 2898 - 2904 .

[6] Y. Yang , M. C. S. Uy , A. Huang , Finbert: A pretrained language model for financial communications , arXiv preprint arXiv: 2006 . 08097 ( 2020 ).

[7] M. A. Cortelazzo , Il linguaggio amministrativo. principi e pratiche di modernizzazione , 2021 .

[8] S. Auriemma , M. Miliani , A. Bondielli , L. C. Passaro , A. Lenci , Evaluating pre-trained transformers on italian administrative texts ( 2022 ).

[9] S. Gururangan , A. Marasović , S. Swayamdipta , K. Lo , I. Beltagy , D. Downey , N. A. Smith, Don't stop pretraining: Adapt language models to domains and tasks , in: Proceedings of the 58th Annual Meeting of the ACL , 2020 .

[10] J. Devlin , M.- W. Chang , K. Lee , K. Toutanova , Bert: Pre-training of deep bidirectional transformers for language understanding , in: Proceedings of the 2019 Conference of the NACL) , 2019 .

[11] Y. Liu , M. Ott , N. Goyal , J. Du , M. Joshi , D. Chen , O. Levy , M. Lewis , L. Zettlemoyer , V. Stoyanov , Roberta: A robustly optimized bert pretraining approach , arXiv preprint arXiv: 1907 . 11692 ( 2019 ).

[12] V. Sanh , L. Debut , J. Chaumond , T. Wolf, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter , arXiv preprint arXiv: 1910 . 01108 ( 2019 ).

[13] Z. Yang , Z. Dai , Y. Yang , J. Carbonell, R. R. Salakhutdinov , Q. V. Le , Xlnet: Generalized autoregressive pretraining for language understanding , Advances in neural information processing systems ( 2019 ).

[14] M. Joshi , D. Chen , Y. Liu , D. S. Weld , L. Zettlemoyer , O. Levy , Spanbert: Improving pre-training by representing and predicting spans, Transactions of the Association for Computational Linguistics ( 2020 ).

[15] A. Wang , A. Singh , J. Michael , F. Hill , O. Levy , S. Bowman , Glue: A multi-task benchmark and analysis platform for natural language understanding , in: Proceedings of the 2018 EMNLP Workshop BlackboxNLP , 2018 .

[16] P. Rajpurkar , J. Zhang , K. Lopyrev , P. Liang , Squad: 100 ,000+ questions for machine comprehension of text , in: Proceedings of the 2016 Conference on EMNLP , 2016 .

[17] D. Licari , G. Comandè, Italian-legal-bert: A pretrained transformer language model for italian law ( 2022 ).

[18] S. Douka , H. Abdine , M. Vazirgiannis , R. El Hamdani , D. R. Amariles , Juribert: A masked-language model adaptation for french legal text , in: Proceedings of the NLLP Workshop 2021 , 2021 .

[19] A. Chriqui , I. Yahav , I. Bar-Siman- Tov , Legal hebert: A bert-based nlp model for hebrew legal, judicial and legislative texts, Judicial and Legislative Texts (June 27 , 2022 ) ( 2022 ).

[20] A. Tagarelli , A. Simeri , Unsupervised law article mining based on deep pre-trained language representation models with application to the italian civil code , Artificial Intelligence and Law ( 2021 ).

[21] S. Lubello , Il linguaggio burocratico, Carocci , 2014 .

[22] M. Miliani , S. Auriemma , F. Alva-Manchego , A. Lenci , Neural readability pairwise ranking for sentences in italian administrative language , in: Proceedings of the 2022 Conference of AACLIJCNLP , 2022 .

[23] M. Locatelli , L. C. Tagliabue , G. M. Di Giuda , et al., Archiberto: a hierarchization quality objectives nlp tool in the italian architecture, engineering and construction sector , in: Proceedings of 1st Workshop AIxPA (co-located with AIxIA 2022 ), 2022 .

[24] A. Robins , Catastrophic forgetting, rehearsal and pseudorehearsal , Connection Science ( 1995 ).

[25] L. Gao , Y. Zhang , J. Han, J . Callan, Scaling deep contrastive learning batch size under memory limited setup , in: Proceedings of the 6th the Workshop RepL4NLP , 2021 .

[26] J. Salazar , D. Liang , T. Q. Nguyen , K. Kirchhof , Masked language model scoring , in: Proceedings of the 58th Annual Meeting of ACL , 2020 .

[27] C. Huyen , Evaluation metrics for language modeling, The Gradient ( 2019 ).

[28] L. C. Passaro , A. Lenci , A. Gabbolini , Informed pa: A ner for the italian public administration domain , in: CLiC-it, 2017 .

'gestione' ( 24 .25%) 'capitali' ( 23 .99%) 'partecipazioni' ( 12 .69%) 'investimento' ( 8 .22%) 'capitale' ( 6 .63%) 'stipulato' ( 33 .94%) 'acceso' ( 15 .10%) 'sottoscritto' ( 11 .53%) 'concesso' ( 6 .70%) 'contratto' ( 6 .48%) 'Fratelli' ( 40 .45%) 'Banca' ( 32 .36%) 'Unità' ( 6 .16%) 'Consiglio' ( 2 .17%) 'Regno' ( 1 .30%) 'finanziamenti' ( 29 .26%) 'prestiti' ( 26 .56%) 'mutui' ( 20 .70%) 'contratti' ( 11 .46%) 'pagamenti' ( 2 .03%) 'investimento' ( 68 .31%) 'assicurazione' ( 21 .54%) 'gestione' ( 3 .35%) 'assicurazioni' ( 2 .04%) 'servizi' ( 1 .41%) 'stipulato' ( 47 .81%) 'erogato' ( 16 .55%) 'contratto' ( 8 .44%) 'sottoscritto' ( 6 .69%) 'concesso' ( 6 .55%) 'Banca' ( 99 .76%) 'Leggi' ( 0 .05%) 'Consiglio' ( 0 .03%) 'banca' ( 0 .02%) 'Banco' ( 0 .01%) 'prestiti' ( 77 .96%) 'mutui' ( 11 .02%) 'finanziamenti' ( 7 .70%) 'debiti' ( 1 .06%) 'titoli' ( 0 .84%) 'garanzia' ( 47 .76%) 'polizza' ( 25 .88%) 'copertura' ( 14 .48%) 'Convenzione' ( 1 .82%) 'convenzione' ( 1 .51%) 'importo' ( 28 .82%) 'quanto' ( 14 .14%) 'capitale' ( 14 .00%) 'premio' ( 11 .50%) 'prezzo' ( 6 .13%) 'mercato' ( 32 .84%) 'riferimento' ( 14 .85%) 'liquidazione' ( 5 .18%) 'rimborso' ( 3 .28%) 'rivalutazione' ( 2 .68%) 'capitale' ( 58 .71%) 'Capitale' ( 40 .65%) 'patrimonio' ( 0 .22%)