In this work, we present a lightweight language model based on BERT (Blaze-IT) and a lightweight language model based on MiniLM (Flare-IT), both specifically designed for the Italian language. Starting from the multilingual cased DistilBERT and MiniLM models, we modified the embedding layers and then carried out a continued pre-training procedure on Italian Wikipedia data using whole word masking, resulting in two uncased models. Blaze-IT has 55M parameters and weighs 217MB, while Flare-IT has 17M parameters and only weighs 67MB. The models are tailored to analyze large volumes of natively digital text, such as wikis, web pages and news articles, written in correct and fluent Italian. We evaluate their performances on various downstream tasks and compare them to other models in the same class. We also discuss the limitations of our models and suggest possible directions for future work. Our results show that our models achieve competitive performances while being much smaller than other monolingual models, making them suitable for deployment in resource-constrained environments.
Natural Language Processing (NLP) has rapidly advanced
in recent years, with language models such as BERT [
While some previous work focused on creating a small uncased model for the Italian language exploiting knowledge distillation [3] (which produced an efective Italian DistilBERT model [4], with ∼ 40% less parameters than a classic BERT model), other research went on to reduce the size of the embedding layer to focus a multilingual model on a single language [5]. Quantization and pruning techniques are also widely used. [6] [7].
In this paper, we present two lightweight language
models, based on BERT [
Our models can efectively process natural language inputs and perform a wide range of NLP tasks such as topic modeling, named entity recognition and question answering, therefore highlighting the importance of developing lightweight language models that can operate efectively in resource-constrained settings, making NLP accessible to a wider range of use-cases.
The first proposed language model (Blaze-IT) is based on the multilingual cased DistilBERT model [3] (distilbertbase-multilingual-cased, 6 hidden layers and hidden size of 768, developed by the HuggingFace team as a distilled version of the original multilingual BERT). To focus this model on the Italian language, we first modified the embedding layer, following the approach presented in [5], which we extended to the deletion of cased tokens to turn the original cased model into an uncased version. This was achieved by turning the Italian language subset of the Wikipedia dataset to lowercase, tokenizing the texts with the WordPiece tokenizer of the DistilBERT model, and then computing document-level frequencies of tokens, setting a minimum threshold of 0.1% to determine which tokens to keep.
The same procedure was followed with the second model (Flare-IT) except that in this case the mMiniLMv2 model [8] (L6xH384 mMiniLMv2, 6 layers and hidden size of 384, developed by Microsoft as a distilled version of XLM-RoBERTa-Large) was used as a starting point.
However, the resulting models were still relying on Table 1 their original training, which exploited capitalized repre- Comparison across the major model indicators between Blazesentations of several words (like words at the beginning IT and BERT. of sentences, or proper names of people, places and other Blaze-IT BERT Δ% entities), so they were not properly trained to deal with the lowercase equivalents. Moreover, while many com- Vocab 13.832 32.102 -56,9% amsoinnlgylecatpoikteanli,zethdewirolrodwsewrcearseeperqevuiiovuaslelyntreisprliekseelnytetdo bbye SPiazreams 2514.71M50B.920 414150.M72B7.782 --5511,,21%% splitted in several subword tokens, being less common than the capitalized version (e.g. ”Microsoft” → ”micro Table 2 ##so ##ft”), which makes it harder for the models to deal Comparison across the major model indicators between Flarewith them properly, especially in token classification IT and BERT. tasks. Flare-IT BERT Δ%
To make the models more robust to the lowercase representations of words previously capitalized and compen- Vocab 14.610 32.102 -54,5% sate for the deletion of cased tokens, we exploited a con- SPiazreams 6176.M61B8.770 414150.M72B7.782 --8845,,90%% tinued pre-training procedure [9] [10]. More specifically, we further pre-trained the models on the Italian split of the Wikipedia dataset, using the whole word masking Table 3 technique [11]. By masking whole words at once, rather Comparison across the major model indicators between Blazethan individual tokens, this technique makes the Masked IT and DistilBERT.
Language Modeling (MLM) task harder for the models, Blaze-IT DistilBERT Δ% encouraging them to learn more efective representations and to capture a wider range of linguistic structures. Vocab 13.832 32.102 -56,9%
Overall, these modifications allowed us to adapt the SPiazreams 5241.71M50B.920 6287.32M00B.550 --2200,,65%% two pre-existing multilingual language models to the Italian language, and to turn them from case-sensitive to case-insensitive, significantly reducing the size of the Table 4 models while maintaining their ability to produce efec- Comparison across the major model indicators between Flaretive representations of Italian text. IT and DistilBERT.
Blaze-IT has 55M parameters, a vocabulary of 13.832 Flare-IT DistilBERT Δ% tokens, and a size of 217MB. Flare-IT has 17M parameters, a vocabulary of 14.610 tokens, and a size of 67MB. The PVoarcaambs 1146..661108.770 3628..120020.550 --5745,,56%% models can be fine-tuned for a wide range of downstream Size 67MB 273MB -75,5% NLP tasks, making them highly versatile and useful for practical applications (we fine-tuned them on Text Classiifcation, Part Of Speech Tagging, Named Entity Recognition, Semantic Textual Similarity and Extractive Question ian subset of the Wikipedia dataset, which contains apAnswering, reporting the results in the dedicated section proximately 3.7GB of text data (we used a 2020 dump of Results). A short comparison 1 between Blaze-IT, Flare- Wikipedia, already pre-processed by the HuggingFace IT, BERT 2 and DistilBERT 3 is summarized in Tables 1, team). Specifically, adapting from the continued pre2, 3 and 4. training setups in [9] and [10], the models were trained for 10,000 steps using the AdamW optimizer with a batch size of 512, obtained through 128 gradient accumulation 2.1. Training details steps and an instantaneous batch size of 4 on a NVIDIA The proposed Italian language models have been trained GeForce RTX 3060 GPU. We kept the sequence length using Masked Language Modeling (MLM) on the Ital- fixed to 512 and applied a linearly decaying learning rate starting from 5 · 10− 5.
Following the original pre-training strategy of BERT
[
2We used the bert-base-italian-xxl-uncased model on HuggingFace, released by the Bavarian State Library MDZ team, as a reference BERT model
3We used the BERTino model on HuggingFace, released by indigo.ai, as a reference DistilBERT model the Next Sentence Prediction task from the pre-training. Table 5
To ensure optimal performance during training, we Text Classification results also employed dynamic masking [12] between epochs Model and utilized the whole word masking technique to encourage the models to learn more efective representations of Italian lowercased text. The dynamic masking technique involves randomly masking tokens in the input sequence during training over diferent epochs, while the whole word masking technique involves masking entire words at once rather than just individual tokens. Together, these techniques help to prevent overfitting and improve the robustness of the models.
The resulting models have been fine-tuned and eval- Model uated on a range of benchmark datasets, demonstrat- BERT ing comparable performances with other models in their DistilBERT class. The limited size of the models, combined with Blaze-IT their performances, makes them highly valuable as- Flare-IT sets for large-scale data analysis, especially in resourceconstrained settings or in applications where computational eficiency is a priority, without excessively compromising on output quality.
Model
Recall Precision Recall Precision
epoTchhe aTnedxtthCelaPsosiSficTatmioondmelosdfeolrs 5haevpeocbhese,nwthrailienethdefoNrE1R, lPerveeclisainodn athnednFm1macertor-icasvheraavgeebdeoenvecrotmhpeuctleadssaetst.hFeotroSkTenS wSTitShaandcoEnQsAtanmtoldeealrsnhinavgerbaetee,nfixterdainated f1o0r− 35e.pFoocrhsT,eaxltl abneednEuQsAedt,hreesPpeeacrtsiovnel’ys. and Exact Match metrics have Classification, Part of Speech Tagging, Semantic Similarity and Extractive Question Answering, the metrics have 3.1. Throughput been computed on the default test set provided with the dataset, while for Named Entity Recognition the metrics have been computed with a 5-fold cross-validation.
For Text Classification on the NC dataset, the Accuracy metric has been used, in agreement with the original In order to test the improvements that can be achieved by exploiting the limited weight of Blaze-IT and Flare-IT, we conducted an experiment which simulates the typical conditions of a cloud instance. More specifically, we set up a Docker image with the relevant requirements for an inference task (we chose the Text Classification task on the NC dataset), and then launched a container with 8 CPU cores and a 8GB RAM memory budget. For each one of the models, we tried to achieve the maximum level of parallelization allowed by the RAM (i.e. the maximum • Text Classification: XGLUE NC, machine- Table 8
translated from English 4 [13] Extractive Question Answering results • Part of Speech Tagging: UD Italian ISDT
dataset 5 [14] • Named Entity dataset 6 [15]
Recognition: WikiNER • Semantic Similarity: MULTI STS-B dataset
[16] • Extractive Question Answering: SQuAD
IT dataset [17]
for the translation
5Italian corpus annotated according to the UD scheme, obtained by conversion from ISDT, released for the shared task at Evalita-2014 6The B-type and I-type categories have been collapsed together since the B-type categories have extremely limited support F1 97.80 97.53 97.37 95.45 F1 formances of the models on specific domains can be improved through further pre-training by incorporating additional training data, which may help to overcome some of the limitations we have mentioned.
many natural language processing applications, ranging
Table 10 from text classification and sentiment analysis to machine
Throughput measurements with a fixed memory budget translation and question answering. Recent advances in
Model N. jobs Samples / s Δ% transformer-based architectures, such as BERT [
GPT [
Indeed, large transformers are especially cumbersome number of parallel jobs that could be launched without to deal with when only CPUs are available as processing getting a SIGKILL signal from the operating system) units, since the execution speed is going to be heavily
We then proceeded to measure the throughput reached limited and the sheer size of the neural networks makes by the models, each one with its maximum parallelization, it hard to deploy multiple models in parallel without and the relative increase in throughput compared to a lfooding the RAM. The huge weight of these models classical BERT model. The results are in Table 10 can also become an obstacle in cloud computing, when server-less applications exploiting state-less functions are involved, since loading large models takes lots of 3.2. Limitations time, which goes against the idea of executing a function The proposed lightweight Italian language models have on-the-fly. been further pre-trained, in our work, on the Italian To address these issues, several approaches have been subset of the Wikipedia dataset, which consists of high- proposed to reduce the complexity and size of language quality, natively digital text written in a correct and fluent models without compromising their performance. One form. As a result, the model is particularly well-suited such approach is distillation [3], in which a larger, prefor analyzing large volumes of text from the web, such as trained model is used to train a smaller, "distilled" model wikis, web pages, news articles, and other similar sources. that can achieve comparable performance. However,
However, it is worth noting that the models may have knowledge distillation is computationally expensive, and limitations when it comes to analyzing chaotic text that while several of these compressed version have been recontains errors, slang expressions, or other types of noise. leased for English models, or for multilingual models, This is because such text is often less structured and less only a few studies have focused specifically on developconsistent than the text found in more formal digital ing lightweight language models for low-resource lansources, which can make it more dificult for the models guages, which may also lack the large, high-quality trainto accurately process and interpret. Additionally, the ing datasets that are available for more widely spoken models may struggle when analyzing domain-specific languages. text, such as medical, financial, or legal content, which Another approach is pruning, in which unimportant often contains specialized terminology and conventions connections (or even entire layers) are removed from that may not be present in more general digital text. the model to reduce its size [7] [18]. While pruning
Despite these limitations, the lightweight design and techniques are efective up to a certain point, they can robust performances of the proposed models make them afect the performances when relevant fractions of the extremely valuable for a wide range of natural language models are removed. processing applications. In particular, their eficiency A diferent technique is quantization [ 6], which exand agility make them well-suited for analyzing large ploits less accurate representations of floating points (e.g. volumes of digital text or processing inputs in real-time, 16 bits instead of 32) so that the resulting model is lighter which can be useful in a variety of contexts, includ- (even though this is only strictly true for half-precision, ing intelligent document processing, conversational sys- because working in mixed-precision with master weights tems and web content analysis. Furthermore, the per- will actually lead to two copies of the model weights being loaded, one in FP32 and one in FP16 [19]). for local environments and applications where large vol
Lastly, the modification of the embedding layer pro- umes of data have to be processed, especially if no hardposed in [5], which is the method we followed in our ware acceleration is available, since the execution of these work (and can be seen as a form of pruning where only light models can be easily parallelized on multiple CPUs. weights corresponding to unused tokens are removed), They are also ideal when computational resources or allows to focus a multilingual model on a single language memory are limited, such as on mobile devices or edgeby getting rid of the extra parameters in the embedding computing environments, or even in cloud-computing layer, therefore reducing its size (the parameters in the scenarios where server-less applications are involved, embedding layer are a considerable portion of the total since these models can be quickly loaded and used in parameters when the model’s vocabulary is large). This state-less functions. We hope that our work will help procedure implies a limited (or sometimes even negligi- to lower the entry barrier for natural language processble) loss in performance, since it only afects statistically ing tasks for researchers and practitioners working in rare tokens for the target language. low-resource settings.
When applied to the distilled versions of multilingual In future work, we plan to investigate methods for models, this technique can further reduce their size and, further compressing the size of transformer-based modas we showed in our work, if cased tokens are also re- els while maintaining performances, perhaps combining moved (with the inclusion of an additional pre-training the techniques showed in this work with model quanphase to compensate for their deletion, possibly exploit- tization. We also aim to expand the capabilities of our ing whole word masking) the procedure ultimately deliv- models, to handle informal and noisy text, and to develop ers extremely light language models, with the additional domain-specific versions of the models for specialized apbenefit of the new uncased representations. plications. Overall, we believe that this work represents
The introduction of an additional pre-training phase, a step towards democratizing access to natural language formally known as continued pre-training, has been processing tools and techniques, and we look forward to mainly explored as a method to adapt language mod- further developments in this area. els to new domains or tasks [10] ("domain-adaptive pre- You can find the models online on the HuggingFace training" and "task-adaptive pre-training"), yielding im- platform at https://huggingface.co/osiria/blaze-it and provements in downstream performances on Text Classi- https://huggingface.co/osiria/flare-it ifcation. Recent work on Spoken Language Understand- You can also try the models online (fine-tuned on ing [9] has brought this further by investigating the efec- named entity recognition) using the web apps at https: tiveness of the continued pre-training of English models //huggingface.co/spaces/osiria/blaze-it-demo and https: in cross-lingual settings, showing that the domain knowl- //huggingface.co/spaces/osiria/flare-it-demo. edge obtained on intermediate data is even transferable Blaze-IT is released under Apache-2.0 license and Flareto other languages. IT under MIT license.
In our work, we exploited continued pre-training to adapt our language models to uncased text, taking inspiration from the setups in [9] and [10]. 6. Acknowledgments
In this paper, we presented Blaze-IT, a lightweight language model based on BERT, and Flare-IT, a lightweight language model based on MiniLM, both specifically tailored for the Italian language. Our models are significantly smaller than other monolingual Italian models, the first one weighing 217MB and having 55M parameters, the second one weighing only 67MB and having 17M parameters. We achieved this by starting with the multilingual DistilBERT and MiniLM models, reducing the embedding layer and further pre-training them on Italian Wikipedia data using whole word masking. While the models are designed to excel on correctly written digital text, they may struggle with noisy, informal language or domain-specific jargon.
The limited size of our models makes them well-suited
for the unconditioned support, availability and guidance received. The authors would also like to thank Alessio Palmero Aprosio (FBK) for his valuable suggestions and feedback. E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, [14] C. Bosco, F. Dell’Orletta, S. Montemagni, M. SanC. Berner, S. McCandlish, A. Radford, I. Sutskever, guinetti, M. Simi, The evalita 2014 dependency D. Amodei, Language models are few-shot learners, parsing task, Proceedings of the First Italian Conin: Proceedings of the 34th International Confer- ference on Computational Linguistics CLiC-it 2014 ence on Neural Information Processing Systems, and of the Fourth International Workshop (2014). 2020. [15] J. Nothman, N. Ringland, W. Radford, T. Murphy, [3] V. Sanh, L. Debut, J. Chaumond, T. Wolf, Distilbert, J. R. Curran, Learning multilingual named entity a distilled version of bert: smaller, faster, cheaper recognition from wikipedia, Artificial Intelligence and lighter, ArXiv abs/1910.01108 (2019). 194 (2013). [4] M. Mufo, E. Bertino, Bertino: An italian distilbert [16] P. May, Machine translated multilingual sts benchmodel, ArXiv abs/2303.18121 (2023). mark dataset., 2021. URL: https://github.com/ [5] A. Abdaoui, C. Pradel, G. Sigel, Load what you PhilipMay/stsb-multi-mt.
need: Smaller versions of mutlilingual bert, in: [17] D. Croce, A. Zelenanska, R. Basili, Neural learning SUSTAINLP, 2020. for question answering in italian, in: International [6] A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Ma- Conference of the Italian Association for Artificial honey, K. Keutzer, A survey of quantization meth- Intelligence, 2018. ods for eficient neural network inference, ArXiv [18] H. Sajjad, F. Dalvi, N. Durrani, P. Nakov, On the abs/2103.13630 (2021). efect of dropping layers of pre-trained transformer [7] M. A. Gordon, K. Duh, N. Andrews, Compressing models, Computer Speech & Language (2023). bert: Studying the efects of weight pruning on [19] P. Micikevicius, S. Narang, J. Alben, G. Diamos, transfer learning, ArXiv (2020). E. Elsen, D. Garcia, B. Ginsburg, M. Houston, [8] W. Wang, H. Bao, S. Huang, L. Dong, F. Wei, O. Kuchaiev, G. Venkatesh, H. Wu, Mixed precision Minilmv2: Multi-head self-attention relation dis- training, in: International Conference on Learning tillation for compressing pretrained transformers, Representations, 2018.
in: Findings, 2020. [9] S. M. B. Louvan, Samuel; Casola, Investigating continued pretraining for zero-shot cross-lingual spoken language understanding, in: Proceedings of the Eighth Italian Conference on Computational
Linguistics Lingua/e Inglese, 2018. [10] S. Gururangan, A. Marasovi’c, S. Swayamdipta,
K. Lo, I. Beltagy, D. Downey, N. A. Smith, Don’t stop pretraining: Adapt language models to domains and tasks, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
Association for Computational Linguistics, 2020. [11] Y. Cui, W. Che, T. Liu, B. Qin, Z. Yang, S. Wang,
G. Hu, Pre-training with whole word masking for chinese bert, ArXiv abs/1906.08101 (2019). [12] L. Zhuang, L. Wayne, S. Ya, Z. Jun, A robustly optimized BERT pre-training approach with posttraining, in: Proceedings of the 20th Chinese National Conference on Computational Linguistics, Chinese Information Processing Society of China, 2021. [13] Y. Liang, N. Duan, Y. Gong, N. Wu, F. Guo, W. Qi,
M. Gong, L. Shou, D. Jiang, G. Cao, X. Fan, R. Zhang, R. Agrawal, E. Cui, S. Wei, T. Bharti, Y. Qiao, J.-H.
Chen, W. Wu, S. Liu, F. Yang, D. Campos, R. Majumder, M. Zhou, XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, 2020.