Named Entity Recognition (NER) plays a fundamental role in Natural Language Processing (NLP), often being a key component in information extraction pipelines. The task involves identifying and categorizing entities in a given text according to a predefined set of labels. While person, organization, and location are the most common, applications of NER in certain fields may require the identification of domain-specific entities.
Manually annotated data has always been critical for the training of NER systems [1]. Traditional methods tackle NER as a token classification problem, where models are specialized on a narrow domain and a pre-defined labels set [2]. While achieving strong performance for the data distribution they were trained on, they require extensive human annotations relative to the downstream task at hand. Additionally, they lack generalization capabilities when it comes to addressing out-of-distribution input domains and/or unseen labels [1,3,4].
On the contrary, Large Language Models (LLMs) have recently demonstrated strong zero-shot capabilities. Models like GPT-3 can tackle NER via In-Context Learning [5,6], with Instruction-Tuning further improving performance [7,8,9]. To this end, several models have been proposed to tackle zero-shot NER [10,4,3,11,12,13]. In particular, SLIMER [13] proved to be particularly effective on unseen named entity types, by leveraging definitions and guidelines to steer the model generation. However, little has been done for zero-shot NER in non-English data. More in general, as pointed out in [1], NER is understudied in languages like Italian, especially outside the traditional news domain and person, location, organization classes.
To this end, we propose in this paper an evaluation framework for Zero-Shot NER, and we apply it to the Italian language. In addition, we fine-tune a version of SLIMER for Italian, which we call SLIMER-IT 1 . In the experiments, we explore different LLM backbones and we assess the impact of Definition and Guidelines (D&G). When comparing SLIMER-IT with state-of-the-art approaches, either using models pre-trained on English or adapted for Italian, results demonstrate SLIMER-IT superiority in labelling unseen entity tags.
Several works tackle Zero-Shot NER on English, such as InstructUIE [10], UniNER [4], GoLLIE [3], GLiNER [11], GNER [12] and SLIMER [13]. Most of them are based on the instruction tuning of an LLM and mainly differ in the prompt and output format design. GLiNER distinguishes itself by being a smaller encoder-only model, combined with a span classifier head, that achieves competitive performance at a lower computational cost.
As highlighted in SLIMER [13], most approaches mainly focus on zero-shot NER in Out-Of-Distribution input domains (OOD), since they are typically fine-tuned on an extensive number of entity classes highly or completely overlapping between training and test sets. In view of this, we proposed a lighter instruction-tuning methodology for LLMs, training on data overlapping in lesser degree with the test sets, while steering the model annotation process with a definition and guidelines for the NE category to be annotated. From this, the name SLIMER: Show Less, Instruct More Entity Recognition.
Although the authors of GLiNER propose also a multilingual model and evaluate zero-shot generalizability across different languages, neither they nor any other work has addressed the task of Zero-Shot NER specifically for the Italian language. NER for Italian. While NER has been extensively studied on English, less has been done in other languages, particularly outside the traditional general-purpose domains and entity labels set [14]. Indeed, in Italian, most NER datasets focus on news and, more recently, social media contents [15,16,17]. Currently, there has been no research into zero-shot NER, only a few exploratory studies into multi-domain NER. This challenge was introduced in the NERMuD task (NER Multi-Domain) at EVALITA 2023 2 , in which one sub-task required to develop a single model capable of classifying the common entitiesperson, organization, location -from different types of text, including news, fiction and political speeches. ExtremITA team [18] addressed the challenge proposing the adoption of a single LLM capable of tackling all the different tasks at EVALITA 2023, among which NERMuD. All the tasks were converted into text-to-text problems and two LLMs (LLaMA and T5 based) were instruction-tuned on the union of all the available datasets for the challenge. In the case of zero-shot NER, a model should be able to extract entities from inputs belonging to the same domain it was trained on (in-domain) and across other domains not encountered before (out-of-domain). Moreover, it should also generalize well to novel entity classes (unseen named entities). In our zero-shot evaluation framework we aim to measure each level independently. Hence, we define an evaluation benchmark that includes a collection of NER datasets divided by degree of generalization. In the following we describe the required properties to fit in.
In-domain. This evaluation helps measure how well the model can generalize from its training data to similar, but not identical, data. The model is evaluated on the same input-domains and named entities as those in the training set. This data often consists in the test partitions associated with each training set used for fine-tuning the model.
OOD evaluation tests the model's ability to generalize to input texts from domains that it has not encountered during training. While the named entities have been seen during training, this type of evaluation is particularly challenging because different input domains often exhibit unique linguistic patterns and domain-specific terminology.
Unseen Named Entities. This evaluation tests the model's ability to identify and classify entities that has not encountered during its training phase. The tag set comprises fine-grained categories which are often specifically defined for the domain in which NER is deployed. Because of this, the input data may often be also Out-Of-Domain (OOD), making this evaluation include the previously mentioned OOD scenario as well.
To adapt SLIMER for Italian, we translate the instructiontuning prompt of [13], as shown in Figure 1. The prompt is designed to extract the occurrences of one entity type per call. While this has the drawback of requiring |NE| inference calls on each input text, it allows the model to better focus on a single NE type at a time.
As in [13], we query gpt-3.5-turbo-1106 via OpenAI's Chat-GPT APIs to automatically generate definition and guidelines for each needed entity tag. The definition for a NE is meant to be a short sentence describing the tag. The guidelines instead provide annotation instructions to align the model's labelling with the desired annotation scheme. Guidelines can be used to prevent the model from labelling certain edge cases or to provide examples of such NE. Such an informative prompt is extremely valuable when dealing with unfamiliar entity tags, and can also be used to distinguish between polysemous categories.
Finally, the model is requested to generate the named entities in a parsable JSON format containing the list of NEs extracted for the given tag.
Experiments aim to assess our approach in Italian. We study the impact of guidelines and the usage of different backbones. Then, we compare our approach against stateof-the-art alternatives.
We construct the zero-shot NER framework (described in Section 3) for Italian upon NerMuD shared task and Multinerd dataset. In particular, we use NerMuD to build in-domain and OOD evaluation sets, while Multinerd-IT is used to assess the behaviour in the unseen named entites scenario.
NERMuD [1] is a shared task organized at evalita-2023, built based on the Kessler Italian Namedentities Dataset (KIND) [19]. It contains annotations for the three classic NER tags: person, organization and location. Examples are organized in three distinct domains: news, literature and political discourses. Unlike NERMuD, we restrict fine-tuning to a single domain. In such a way, we can evaluate both in-domain and outof-domain capabilities of the model. In particular, we designate WikiNews (WN) sub-set for training and indomain evaluation, being the most generic domain, while Fiction (FIC) and Alcide De Gasperi (ADG) splits are kept for out-of-domain evaluation only.
To construct the unseen NEs evaluation set, we exploit Multinerd3 [20], a multilingual NER dataset made of 15 tags: person, organization, location, animal, biological entity, celestial body, disease, event, food, instrument, media, plant, mythological entity, time and vehicle. We keep the Italian examples only. Such a dataset constitutes a perfect choice to assess models' capabilities on unseen NEs. Indeed, data belongs to the same news domain of the NERMuD split chosen for fine-tuning, but it includes a broader label set. Since we want to measure performance on never-seen-before entities, we exclude entity types seen in training, i.e. person, organization and location. We also remove biological entity, being poorly underrepresented, with a support of just 4 instances.
We implemented several version of SLIMER-IT based on different backbone models. We consider similarly sized LLMs, all in the 7B parameters range. In particular, we selected five backbones: Camoscio4 [21], LLaMA-2-7bchat [22], Mistral-7B-Instruct [23], LLaMA-3-8B-Instruct, LLaMAntino-3-ANITA-8B-Inst-DPO-ITA5 [24].
LLaMA-2-7b-chat was originally used in SLIMER [13], and LLaMA-3-8B-Instruct is the newest, improved version of it. As LLaMA family, Mistral-7B-Instruct is a multilingual model mainly English-oriented, but it has demonstrated greater fluency on Italian. Camoscio and LLaMAntino-3-ANITA-8B-Inst-DPO-ITA, instead, are two LLMs specifically fine-tuned on Italian instructions.
We compare the SLIMER-IT approach, implemented with different backbones, against other state-of-the-art approaches for zero-shot NER. All the methods are trained and evaluated in the defined zero-shot NER framework for a fair comparison. We evaluate against: Token classification. Although certainly not being suited for zero-shot NER, due to its architectural inability to cope with unseen tags, we decided to evaluate the most known approach to NER as baseline. As in NERMuD [1], we use the training framework dhfbk/bert-ner 6 . We fine-tune two different base models, bert-base-cased, pretrained on English, and dbmdz/bert-base-italian-cased 7 , an Italian version.
It is the best performing approach on zero-shot NER in OOD English benchmark. In GNER [12], they propose a BIO-like generation, replicating in output the same input text, along with a token-by-token BIO label. Here, we consider LLaMAntino-3 as its backbone.
Comparing SLIMER-IT based on different backbones, with and without Definition and Guidelines (D&G) in the prompt. LLMs with † symbol were instruction-tuned on Italian. In parentheses the (±Δ𝐹 1) of performance given by the usage of D&G. GLiNER. Differently from all other methods, GLiNER is based on a smaller encoder-only model, combined with a span classifier head, able to achieve competitive performance on the OOD English benchmark at a lower computational cost. We fine-tune it both using its original deberta-v3-large English backbone and the Italian dbmdz/bert-base-italian-cased model.
extremITLLaMA. Already described in Section 2, it represents an interesting approach to compare against. Based on Camoscio LLM, we compare it with SLIMER-IT approach implemented with the same backbone.
We kept the same training configuration of SLIMER [13] on English, except that we trained on all available samples. Depending on the backbone, the instructiontuning prompt (see Figure 1) was adjusted accordingly to the structure of its template (e.g. [INST] or <|start_header_id|> formats). For all the competitors, we replicated their training setup using their scripts and suggested hyper-parameters. For the evaluation, we use the micro-F1 as computed in the UniNER 8 implementation.
We compare SLIMER-IT with a version devoid of definition and guidelines in the prompt. To demonstrate the robustness of the approach, we train several SLIMER-IT instances, based on different LLM backbones. In Table 1, we report the results, highlighting the absolute difference in performance between the model steered by
Comparison with existing off-the-shelf models for zero-shot NER on Italian. We omit in-domain evaluation to not disadvantage them against SLIMER-IT. D&Gs and the one not using them. Generally, definition and guidelines yield improvements in F1. In particular, the gap is contained when evaluating on in-domain data, whereas it becomes significant in OOD and even more substantial in unseen NEs. This is expected since D&G help the most in conditions unseen during training. Notably, LLaMA-3-based backbones benefit the most from definition and guidelines, with improvements beyond 23 absolute F1 points, surpassing all the other models by substantial margins in never-seen-before entity tags.
Comparing SLIMER-IT with state-of-the-art approaches trained in the same zero-shot setting, and adopting the same backbone when possible. *Note that extremITLLaMA was fine-tuned also on the FIC and ADG train sets for the NERMuD task, so these datasets are not actually OOD for this model. 2. We can observe no remarkable difference in indomain evaluation, where most recent models outperform older ones, as one might expect. Also globally, Camoscio and LLaMA-2-chat obtain lower scores than the rest of the backbones, with the only exception of FIC dataset, where LLaMA-3 based architecture underperform. However, LLaMAntino-3-ANITA reaches the best performance on 3 out of 4 datasets, with a strong gap especially in unseen named entities scenario, the most challenging one. Interestingly enough, thanks to their better understanding capabilities, backbones specialized on Italian are particularly effective in the unseen NEs scenario. This is the case of LLaMAntino-3-ANITA and even Camoscio, which demonstrates higher F1 than LLaMA-2.
Off-the-shelf Italian NER models. Although there has been no prior work defining a Zero-Shot NER evaluation framework for Italian, there exist fine-tune specialized state-of-the-art zero-shot NER models for Italian language. In particular, we consider: GLiNER-ML [11], a multilingual instance of GLiNER, Universal-NER-ITA 9and GLiNER-ITA-Large10 , both specialized on Italian. These models were trained on synthetic data covering a vast number of different entity classes (up to 97k). Thus, it is impossible to directly compare them in a pure zeroshot framework, since there are no entity tags actually never-seen-before during training. However, we still report their results against SLIMER-IT. Table 2 reports the results. Despite this advantage, SLIMER-IT outperforms all these models by large a margin.
Thanks to the definition of our zero-shot evaluation framework, we can compare different state-of-the-art approaches fairly. Results are outlined in Table 3. When evaluating in the same domain where the model was trained, encoder-only architectures obtain strong results despite being much smaller models. This result is not surprising, given the acknowledged performance of these architectures for supervised NER. More unexpected is their ability to generalize well to OOD inputs. Also GNER proves to be quite competitive achieving the best results in in-domain evaluation, and in OOD on FIC dataset. However, all these approaches dramatically fail on never-seen-before tags, in contrast to SLIMER-IT that achieves almost 55 F1 score points. Compared with LLM-based approaches like GNER and extremITLLaMA, this proves once again that without definition and guidelines LLMs struggle in tagging novel kind of entities.
In this paper, we proposed an evaluation framework for Zero-Shot NER that we applied to Italian. Thanks to such a framework, we can better investigate different zero-shot properties depending on the scenario (in-domain, OOD, unseen NEs). On top of that, we compared several stateof-the-art approaches, with particular focus on SLIMER, which, thanks to the usage of definition and guidelines, is well suited to deal with novel entity types. Indeed, SLIMER-IT, our fine-tuned model based on LLaMAntino-3, surpasses other state-of-the-art techniques by large margins. In the future, we plan to further extend the zeroshot NER benchmark, and implement an input caching mechanism for scalability to large label sets.
In Table 4
The work was partially funded by: • "ReSpiRA -REplicabilità, SPIegabilità e Ragionamento", a project financed by FAIR, Affiliated to spoke no. 2, falling within the PNRR MUR programme, Mission 4, Component 2, Investment 1.3, D.D. No. 341 of 03/15/2022, Project PE0000013, CUP B43D22000900004 11 ; • "MAESTRO -Mitigare le Allucinazioni dei Large Language Models: ESTRazione di informazioni Ottimizzate" a project funded by Provincia Autonoma di Trento with the Lp 6/99 Art. 5:ricerca e sviluppo, PAT/RFS067-05/06/2024-0428372, CUP: C79J23001170001 12 ; • "enRichMyData -Enabling Data Enrichment Pipelines for AI-driven Business Products and Services", an Horizon Europe (HE) project, grant agreement ID: 101070284 13 .