In recent years, Transformer-based models have been widely used in NLP for various downstream tasks and in diferent domains. However, a language model explicitly built for the Italian administrative language is still lacking. Therefore, in this paper, we decided to compare the performance of five diferent Transformer models, pre-trained on general purpose texts, on two main tasks in the Italian administrative domain: Name Entity Recognition and multi-label document classification on Public Administration (PA) documents. We evaluate the performance of each model on both tasks to identify the best model in this particular domain. We also discuss the efect of model size and pre-training data on the performances on domain data. Our evaluation identifies UmBERTo as the best-performing model, with an accuracy of 0.71, an F1 score of 0.89 for multi-label document classification, and an F1 score of 0.87 for NER-PA.
models, such as BERT [
Named Entity Recognition (NER) is the task of identifying and classifying entities mentioned
in unstructured text into predefined categories, e.g., proper names of people, places,
organizations, dates, etc. [
Similarly, the document classification task associates each document with a label related to
its content. It has extensive applications, including topic labeling, sentiment classification, and
spam detection [
Generally, Transformer-based models vary in size, number of parameters, pre-training objective, and pre-training data. Hence, it is a challenging decision to select one of these models for a specific downstream task. Therefore, we decided to assess their performance in the administrative domain to identify which model best suits each of these tasks and for this particular domain.
The main contributions of this work are: • a comparative evaluation of five Transformer-based models on two main downstream tasks, namely NER and multi-label document classification in the administrative domain; • the creation of a new Italian dataset for multi-label document classification in the administrative domain, which we called atto; • a comparison between Transformer-based models and a PA-specialized machine learning model on NER.
The rest of this paper is organized as follows. Section 2 provides a brief overview of the Transformer models we considered for comparison. Section 3 presents the experimental details. We describe the datasets, the parameter configuration used for fine-tuning the models on the NER-PA and multi-label document classification tasks, the baselines, and the metrics used for evaluation. Section 4 reports on the results and discusses them. Finally, Section 5 draws some conclusions and highlights future works.
In this work we focus on Transformer-based models pre-trained on the Italian language or available in a multilingual version, to identify which model represents the best choice for handling administrative data, even in the absence of adaptive domain pre-training. We compare the performance on NER-PA and PA multi-label document classification of 5 Transformerbased language models: BERT-base-Ita,1 UmBERTo,2 Multilingual BERT,3 XLM-RoBERTa,4 and GePpeTto.5 .
BERT [
UmBERTo is a RoBERTa-based Language Model trained on a subset of the OSCAR corpus7
1https://huggingface.co/dbmdz/bert-base-italian-uncased
2https://huggingface.co/Musixmatch/umberto-wikipedia-uncased-v1
3https://huggingface.co/bert-base-multilingual-uncased
4https://huggingface.co/xlm-roberta-base
5https://huggingface.co/LorenzoDeMattei/GePpeTto
6https://opus.nlpl.eu
7https://oscar-corpus.com
containing about 70 GB of plain text. Compared to BERT, RoBERTa was trained longer, over
more data, and with larger training batch size. In addition, the training process was carried
out with a dynamic masking of tokens in place of the Next Sentence Prediction task [
The multilingual version of BERT (mBERT) [
Another multilingual encoder is XLM-RoBERTa. This model was pre-trained on a significantly larger amount of data, 2.5 TB of clean Common Crawl data in 100 diferent languages. Like mBERT, XLM-RoBERTa pre-training task is solely monolingual MLM. It achieves state-of-the-art results on several multilingual benchmarks, including XNLI, MLQA, and NER, outperforming mBERT [24].
GePpeTto [25] is the first autoregressive language model for Italian. It is built using the GPT-2 architecture [26]. The latter is a scale-up of GPT [27], with 10x parameters and 10x training data. GPT-2 is a unidirectional generative Transformer model trained on next token prediction given all of the previous words within some text. Its Italian version, GePpeTto was trained on a collection of Wikipedia Text and the ItWac corpus [28], amounting to almost 13GB.
The Transformer-based models we discussed difer with regard to pre-training objective, size, and number of parameters. Moreover, they are also pre-trained on corpora of diferent sizes, as highlighted in Table 1.
We aimed to compare the performance of five Transformer-based models on two diferent tasks related to the administrative domain. To this end, we fine-tuned these five models on two main downstream tasks in the administrative domain: NER-PA and multi-label document classification. More specifically, we chose the base-cased version of all the models and finetuned each model on two datasets, one for each task. We describe the used datasets in Section 3.1. We compared their performance in terms of precision, recall, F1 score, and accuracy while analyzing the model pre-training and architecture efect on each task, as described in Section 3.5.
In order to fine-tune the models on the NER task for the administrative domain, we selected a corpus containing documents from the Italian Public Administration, i.e. the PA Corpus. The corpus is annotated with domain entity labels. As for the multi-label document classification task, we proposed a new dataset, named the Atto corpus, specifically for this purpose. Unfortunately, the raw annotated dataset used in this paper cannot be released due to sensitive information being present in the data. However, trained models (both for NER and document classification) are available via huggingface8. 3.1.1. PA Corpus.
It was first proposed in [
The corpus contains 460 documents taken from the Albo Pretorio Nazionale for a total of 724,623 tokens. The first 100 documents of the corpus were annotated using the aforementioned Name Entities (NE) by two annotators, including one domain expert. Then, a CRF model was trained using these documents and it was used to automatically annotate new documents. Finally, two other annotators manually revised the entire output. The distribution of the diferent NEs in the corpus, as well as examples, is listed in Table 2.
The ATTO (Administrative Texts labeled by TOpic) corpus is a dataset that includes administrative texts that are labeled with one or more topics (e.g., Environment, Construction, Urban Planning, Social, etc.) pertaining to PA. The dataset was built upon a domain ontology created in the context of the project SEMPLICE.9 The ontology contains approximately 2,700 administrative domain terms that domain experts divided into 13 classes, each corresponding to a diferent sub-sector of the PA (e.g. Environment, Construction, Urban Planning, Social, etc.). We collected up to about 1,000 documents for each of the 13 classes of topics. The collected data are a subset of a dataset including documents from several Italian municipalities annotated and indexed by topic. The complete list of considered topics, as well as their distribution, are shown in Table 3.
The corpus was built by querying the whole PA document collection by topic. As each document was associated with more than one topic in the original dataset, we only selected unique documents (i.e., a document retrieved through two or more diferent topics was added only once to the ATTO dataset). We further filtered the documents, to obtain high-quality data. Specifically, we removed documents containing OCR-level errors, which were identified by means of shallow matching rules. Finally, we removed documents with a length higher than 600 words. The final version of atto includes 11,019 documents.
In order to evaluate the performances of the chosen Transformer models on Named Entity
Recognition for PA (NER-PA), we fine-tuned each model on the PA corpus (see Sec. 3.1.1).
First, the data are pre-processed to unify cross-sentence entities, merging sentences with a
common entity. Then, we split the dataset into 70% training, 20% validation, and 10% test. In
addition to the test set, we evaluated the models on a diferent test set of 25 documents from 25
various municipalities following the original approach proposed in [
As for the multi-label document classification on the atto dataset (see Sec. 3.1.2), the training process was straightforward. We chose to perform 5-fold cross-validation, in order to ensure the reliability of the results in absence of a test set. We first performed a preliminary evaluation in order to find the best hyperparameters. By considering the per-epoch training and validation loss, we concluded that all the models could be trained for 10 epochs before starting to overfit. Thus, we fine-tuned our models for 10 epochs using a batch size of 16, 2e-5 as learning rate, and a maximum sequence length of 512. The final results for each model are obtained by averaging the performances on each training fold.
For what concerns NER, we used as baseline the results obtained on the same datasets by
Informed PA [
As for the baseline of the multi-label document classification task, we implemented a Bidirectional LSTM (bi-LSTM) model. We constructed the model with one bi-LSTM layer with 256 neurons and a single dense output layer with 13 neurons since we have 13 labels in the dataset. We chose the Sigmoid as the activation function and the binary-cross entropy as the 10https://huggingface.co/docs/Transformers/index loss function. We also pre-processed the documents to feed the bi-LSTM removing line breaks, typycal of administrative acts layout. We fed the neural network with 100-dimensional vector representations of the first 250 tokens of each document obtained using Word2Vec [ 30]. We performed 5-fold cross-validation on the ATTO corpus, as for all the other Transformer-based models, and we trained the bi-LSTM for ten epochs with a batch size of 128. The final results of the model are obtained by averaging the result of each training fold.
As the two considered tasks are fairly standard, we evaluated the performances on common metrics for classification. We used micro and macro averages for precision, recall, and F1-score for the NER-PA task. For the multi-label document classification, accuracy, precision, recall and micro average F1-score were considered. Note that the micro average F1-score on multi-label classification corresponds to the harmonic mean between precision and recall, while accuracy refers to the number of data points for which the predicted set of labels is identical to the true one over all the dataset.
Table 4 summarizes the results of the models on the multi-label document classification task. Diferences in performances are clearly minimal across all Transformer-based models, which significantly outperform the baseline with regards to F1-score. The best performing model was UmBERTo with an F1-score of 0.89 and an accuracy of 0.71. BERT-Ita and XLM-RoBERTa both achieved an accuracy of 0.69 and an F1-score of 0.88, followed closely by GePpeTto which achieved 0.67 accuracy and 0.87 F1-score. The worst performing model among Transformers was mBERT, whose accuracy and F1-score were 0.65 and 0.86, respectively. Ultimately, the bi-LSTM model achieved an accuracy of 0.55 and an F1-score of 0.39, both far below the average score obtained by Transformers.
These findings indicate that the most suited model for a multi-label classification task on administrative documents is UmBERTo. This is probably due to the fact that, with respect to the other monolingual Italian models (i.e., BERT-Ita and GePpeTto, UmBERTo), it has a bigger size and was pre-trained on more data. While, comparing Bert-Ita and GePpeTto, despite the models having approximately the same number of parameters and the same amount of data used for pre-training, it appears that BERT-Ita represents a more suitable architecture to perform a multi-label classification task than the generative model GePpeTto.
Regarding the multilingual models, our results confirm what has been reported in the literature for the general domain, i.e., the monolingual models perform better than their multilingual counterparts [31]. Indeed, mBERT was the model with the lowest scores in both accuracy and F1-score, and, although XLM-RoBERTa is almost three times as large as BERT-Ita in terms of parameters, their performance was identical.
As highlighted in Table 5, which shows the precision, recall, and F1-score obtained by the models for each topic class, the ranking of the models is quite consistent across all classes: the best-performing model, UmBERTo, obtained the highest F1-score in most cases, followed by Bert-Ita and XLM-Roberta, GePpeTto, mBERT, and lastly the bi-LSTM model. Only two classes constitute an exception: Trade and Business and Culture, tourism, and sport. In these cases, the best-performing models were GePpeTto and XLM-Roberta, although the scores are still lower when compared to the results of the other classes. That is probably related to the number of documents labeled with these classes in the dataset, as they have fewer instances than all others (see Table 3). Regarding the baseline, we can observe that the bi-LSTM performs poorly on almost the whole dataset. Only in the Financial Service class, the most represented in the corpus, the bi-LSTM achieves an F1-score of 0.88, comparable to that of the Transformer models.
Table 6 provides the results of each model for the NER-PA task in terms of precision, recall, and F1-score. These results refer to the scores obtained on the PA Corpus test set. Table 7 reports the results obtained by the models on the additional dataset of 25 administrative documents from 25 diferent municipalities.
As for the multi-label document classification, the best performing model was UmBERTo, with a micro average precision of 0.86, a recall of 0.89, and F1-score of 0.87. XLM-RoBERTa obtained the same recall score as UmBERTo, but with lower precision and F1-score, respectively of 0.83 and 0.86. BERT-Ita and mBERT achieved an equal precision of 0.83 and F1-score of 0.85. BERT-Ita performed better than mBERT in terms of recall, with 0.88 compared to 0.87. Again, the model that performed worse was GePpeTto with a micro average precision of 0.69, recall of 0.80, and F1-score of 0.74, deviating from the best model by about 0.13 F1-score points.
On the 25 document dataset, the best-performing Transformer models was XLM-RoBERTa, scoring 0.8, 0.88, and 0.84 on micro average precision, recall and F1-score, respectively. The second best performing model was UmBERTo, followed by BERT-Ita and mBERT. GePpeTto was the worst performing model with 0.63 precision, 0.78 recall, and an F1-score of 0.7.
This performance comparison highlights that for the token classification task, the amount of monolingual data used for pre-training appears to play a crucial role, together with the model’s size. We observed that the best performing model on the PA corpus was UmBERTo, which is pre-trained on a much larger sample of Italian texts with respect to all the other models. On the 25 document sample, which includes a higher variability in terms of how Named Entities are mentioned in the text, the larger model in terms of parameters, namely XLM-RoBERTa, has a slight advantage over UmBERTo. Moreover, results show that a rather small generative model, in comparison with current state-of-the-art ones such as GPT-3 [32], is the least suited for fine-tuning on a token classification task.
If we look at the detailed scores for each class on the 25 documents dataset, we can also observe that the classes for which the Transformer models struggle the most are ORG_PA P R F1 P R F1 P R F1 P R F1 P R F1 P R F1 P R F1 P R F1 P R F1 P R F1 P R F1 P R F1 mBERT XLM-RoB.
UmBERTo GePpeTto INFORMed
PA P R F1 P R F1 P R F1 P R F1 P R F1 and ACT_U (see Tab. 7 and 8). As for ORG_PA, this may be due to the fact that the names of the organizations are more closely related to the individual Public Administration entity (e.g., Settore LL.PP. - Public Work Sector; Uficio politiche abitative - Housing Policies Ofice). Given that this dataset includes documents from 25 diferent municipalities, it is understandable that even the best-performing models, namely UmBERTo and XLM-RoBERTa, may encounter dificulty handling such variability in the nomenclature of PA organizations. On the other hand, UmBERTo and XLM-RoBERTa seem to handle well the class ACT_U, which labeled the unparsable token, i.e., the reference to the PA act expressed as a unique string that can vary in terms of templates, such as n.12/2013; 67/96; 57/2008. On the contrary, BERT-Ita, mBERT, and GePpeTto performed poorly on this class.
We also computed the macro average of precision, recall, and F1-score for each model and
dataset to make our data comparable with the results obtained by INFORMed PA [
UmBERTo GePpeTto INFORMed
PA P R F1 P R F1 P R F1 P R F1 P R F1
Measure ACT LAW
LOC ORG ORG PER MicAvg MacAvg recall, and 0.12 points higher in F1-score. All the others models outperformed INFORMed PA of at least 0.11 points in precision, 0.6 in recall, and 0.9 in F1-score. Surprisingly, the results of INFORMed PA on the 25 documents dataset exceed those of the Transformer-based models in terms of both precision and F1 score. We speculate that the CRF model is better able in dealing with classes whose instance show a high degree of linguistic regularity. In particular, the model seems to benefit from linguistic and shallow features such as word shape, n-grams, PoS-tags, and the presence of complex terms. By performing a class-wise comparison between models, we can observe that Transformer-based ones are more prone to errors in extracting Organization classes (both ORG and ORG_PA), LAW and ACTS while remaining efective on the others, which are arguably less dependent on the domain. In this case, in fact, their prediction capabilities are closer to InformedPA. These results show that language models based on the Transformer encoder architecture can guarantee higher performance when adequately trained on domain data than traditional tools such as INFORMed PA. Conversely, when there is some variability between the data on which the models are evaluated and the data used during the fine-tuning, the performance of Transformers tends to sufer with respect to that of a CRF model.
This means that to guarantee high performance on domain-specific linguistic data, such as administrative texts, and in domain-specific tasks, such as the recognition of Public Administration entities, the domain adaptation of the model via additional pre-training on domain-specific data may prove to be necessary. In this way, the model should be better able to derive an adequate representation of domain-specific terms and cope with the high degree of variability mBERT P R F1 P R F1 P R F1 P R F1 in how entities and acts are mentioned in bureaucracy documents. Thus, one of our future goals will be to make a domain adaptation of the model that performed best in this comparison and to extend further the suite of tasks more closely related to the PA domain on which the model will be assessed.
In this paper, we aimed to confront the performance of various Transformer-based models on administrative texts. Our goal was to evaluate the performance of generic pre-trained models on administrative data and to identify the most suitable model for this type of task in this particular domain. To this end, we considered a multi-label document classification task and a Named Entity Recognition task on Public Administration texts.
Among the diferent kinds and sizes of Transformer models, UmBERTo was shown to be the best option to handle both text and token classification tasks. It achieved the highest accuracy of 0.71 and F1-score of 0.89 on the multi-label classification task and the highest micro average precision, recall, and F1-score on the NER-PA task of 0.86, 0.89, and 0.87, respectively.
While on multi-label document classification, no clear diferences emerged among
Transformer-based models, which outperformed the bi-LSMT model by a wide margin, we
have observed a much more significant variance in performances for the NER-PA task. This is
especially true if we consider encoder-only models such as UmBERTo and BERT and
encoderdecoder ones such as GePpeTto. Indeed, the latter performed worst in the NER-PA, both in
comparison with other Transformer-based models and with INFORMed PA [
Furthermore, we have observed that in the case of high variability in the data, document structures, and the way in which entities are expressed, the best performances are obtained with the domain-adapted CRF model INFORMed PA. These findings lead us to surmise that as a future direction, it may be crucial to adapt Transformer-based language models to this domain via additional pre-training steps in order to improve their performances.
In addition, we also aim at extending the kind of tasks and the number of datasets related to the Italian administrative domain in order to have more data available for the training and the evaluation of the models. We also plan to anonymize these data to share them.
This research has been funded by the Project “ABI2LE (Ability to Learning)”, funded by Regione Toscana (POR Fesr 2014-2020). The project partners are University of Pisa, CoLing Lab and the companies 01Semplice s.r.l. (coordinator), IPKOM s.r.l., and Insurance OnLine S.p.A. ceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, 2019, pp. 4996–5001. URL: https://aclanthology.org/P19-1493. doi:10.18653/v1/P19-1493. [24] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, arXiv preprint arXiv:1911.02116 (2019). [25] L. De Mattei, M. Cafagna, F. Dell’Orletta, M. Nissim, M. Guerini, Geppetto carves italian into a language model, arXiv preprint arXiv:2004.14253 (2020). [26] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., Language models are unsupervised multitask learners, OpenAI blog 1 (2019) 9. [27] A. Radford, T. Salimans, Gpt: Improving language understanding by generative pretraining. arxiv (2018). [28] M. Baroni, S. Bernardini, A. Ferraresi, E. Zanchetta, The wacky wide web: a collection of very large linguistically processed web-crawled corpora, Language resources and evaluation 43 (2009) 209–226. [29] E. F. Sang, F. De Meulder, Introduction to the conll-2003 shared task: Language-independent named entity recognition, arXiv preprint cs/0306050 (2003). [30] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality, Advances in neural information processing systems 26 (2013). [31] M. Polignano, P. Basile, M. De Gemmis, G. Semeraro, V. Basile, Alberto: Italian bert language understanding model for nlp challenging tasks based on tweets, in: 6th Italian Conference on Computational Linguistics, CLiC-it 2019, volume 2481, CEUR, 2019, pp. 1–6. [32] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, Language models are few-shot learners, in: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, volume 33, Curran Associates, Inc., 2020, pp. 1877–1901. URL: https://proceedings.neurips.cc/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.