=Paper=
{{Paper
|id=Vol-3878/109_main_long
|storemode=property
|title=SLIMER-IT: Zero-Shot NER on Italian Language
|pdfUrl=https://ceur-ws.org/Vol-3878/109_main_long.pdf
|volume=Vol-3878
|authors=Andrew Zamai,Leonardo Rigutini,Marco Maggini,Andrea Zugarini
|dblpUrl=https://dblp.org/rec/conf/clic-it/ZamaiRMZ24
}}
==SLIMER-IT: Zero-Shot NER on Italian Language==
https://ceur-ws.org/Vol-3878/109_main_long.pdf
SLIMER-IT: Zero-Shot NER on Italian Language
Andrew Zamai1,2 , Leonardo Rigutini2 , Marco Maggini1 and Andrea Zugarini2,*
1
Università degli Studi di Siena, Italy
2
expert.ai, Siena, Italy
Abstract
Traditional approaches to Named Entity Recognition (NER) frame the task into a BIO sequence labeling problem. Although
these systems often excel in the downstream task at hand, they require extensive annotated data and struggle to generalize to
out-of-distribution input domains and unseen entity types. On the contrary, Large Language Models (LLMs) have demonstrated
strong zero-shot capabilities. While several works address Zero-Shot NER in English, little has been done in other languages.
In this paper, we define an evaluation framework for Zero-Shot NER, applying it to the Italian language. Furthermore, we
introduce SLIMER-IT, the Italian version of SLIMER, an instruction-tuning approach for zero-shot NER leveraging prompts
enriched with definition and guidelines. Comparisons with other state-of-the-art models, demonstrate the superiority of
SLIMER-IT on never-seen-before entity tags.
Keywords
Named Entity Recognition, Zero-Shot NER, Large Language Models, Instruction tuning
1. Introduction
Named Entity Recognition (NER) plays a fundamental
role in Natural Language Processing (NLP), often being a
key component in information extraction pipelines. The
task involves identifying and categorizing entities in a
given text according to a predefined set of labels. While
person, organization, and location are the most common,
applications of NER in certain fields may require the
identification of domain-specific entities.
Manually annotated data has always been critical for
the training of NER systems [1]. Traditional methods
tackle NER as a token classification problem, where mod-
els are specialized on a narrow domain and a pre-defined
labels set [2]. While achieving strong performance for
the data distribution they were trained on, they require
extensive human annotations relative to the downstream
task at hand. Additionally, they lack generalization capa-
bilities when it comes to addressing out-of-distribution
input domains and/or unseen labels [1, 3, 4].
On the contrary, Large Language Models (LLMs) Figure 1: SLIMER-IT instruction tuning prompt. Dedicated
have recently demonstrated strong zero-shot capabilities. entity definition and guidelines steer the model labelling.
Models like GPT-3 can tackle NER via In-Context Learn-
ing [5, 6], with Instruction-Tuning further improving per-
formance [7, 8, 9]. To this end, several models have been However, little has been done for zero-shot NER in
proposed to tackle zero-shot NER [10, 4, 3, 11, 12, 13]. In non-English data. More in general, as pointed out in [1],
particular, SLIMER [13] proved to be particularly effective NER is understudied in languages like Italian, especially
on unseen named entity types, by leveraging definitions outside the traditional news domain and person, location,
and guidelines to steer the model generation. organization classes.
To this end, we propose in this paper an evaluation
CLiC-it 2024: Tenth Italian Conference on Computational Linguistics, framework for Zero-Shot NER, and we apply it to the
Dec 04 — 06, 2024, Pisa, Italy
Italian language. In addition, we fine-tune a version of
*
Corresponding author.
$ andrew.zamai@unisi.it (A. Zamai); lrigutini@expert.ai SLIMER for Italian, which we call SLIMER-IT1 . In the
(L. Rigutini); marco.maggini@unisi.it (M. Maggini); experiments, we explore different LLM backbones and
azugarini@expert.ai (A. Zugarini)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License 1
Attribution 4.0 International (CC BY 4.0). https://github.com/andrewzamai/SLIMER_IT
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
we assess the impact of Definition and Guidelines (D&G). 3. Zero-Shot NER Framework
When comparing SLIMER-IT with state-of-the-art ap-
proaches, either using models pre-trained on English In traditional Machine-Learning theory, a model 𝑓 ,
or adapted for Italian, results demonstrate SLIMER-IT trained for a task (e.g. NER) represented by a dataset
superiority in labelling unseen entity tags. 𝒳 , 𝒴, is typically evaluated on an held-out test set sam-
pled from the same task and distribution of the training.
In zero-shot learning instead, a model is expected to go
2. Related Work beyond what experienced during training. There are
different levels of generalization indicating up to what
Several works tackle Zero-Shot NER on English, such as extent the model goes beyond what directly learnt.
InstructUIE [10], UniNER [4], GoLLIE [3], GLiNER [11], In the case of zero-shot NER, a model should be able
GNER [12] and SLIMER [13]. Most of them are based on to extract entities from inputs belonging to the same do-
the instruction tuning of an LLM and mainly differ in the main it was trained on (in-domain) and across other do-
prompt and output format design. GLiNER distinguishes mains not encountered before (out-of-domain). More-
itself by being a smaller encoder-only model, combined over, it should also generalize well to novel entity classes
with a span classifier head, that achieves competitive (unseen named entities). In our zero-shot evaluation
performance at a lower computational cost. framework we aim to measure each level independently.
As highlighted in SLIMER [13], most approaches Hence, we define an evaluation benchmark that includes
mainly focus on zero-shot NER in Out-Of-Distribution a collection of NER datasets divided by degree of gen-
input domains (OOD), since they are typically fine-tuned eralization. In the following we describe the required
on an extensive number of entity classes highly or com- properties to fit in.
pletely overlapping between training and test sets. In
view of this, we proposed a lighter instruction-tuning
In-domain. This evaluation helps measure how well
methodology for LLMs, training on data overlapping in
the model can generalize from its training data to similar,
lesser degree with the test sets, while steering the model
but not identical, data. The model is evaluated on the
annotation process with a definition and guidelines for
same input-domains and named entities as those in the
the NE category to be annotated. From this, the name
training set. This data often consists in the test partitions
SLIMER: Show Less, Instruct More Entity Recognition.
associated with each training set used for fine-tuning the
Although the authors of GLiNER propose also a multi-
model.
lingual model and evaluate zero-shot generalizability
across different languages, neither they nor any other
work has addressed the task of Zero-Shot NER specifi- Out-Of-Domain (OOD). OOD evaluation tests the
cally for the Italian language. model’s ability to generalize to input texts from domains
that it has not encountered during training. While the
named entities have been seen during training, this type
NER for Italian. While NER has been extensively stud-
of evaluation is particularly challenging because different
ied on English, less has been done in other languages,
input domains often exhibit unique linguistic patterns
particularly outside the traditional general-purpose do-
and domain-specific terminology.
mains and entity labels set [14]. Indeed, in Italian, most
NER datasets focus on news and, more recently, social me-
dia contents [15, 16, 17]. Currently, there has been no re- Unseen Named Entities. This evaluation tests the
search into zero-shot NER, only a few exploratory studies model’s ability to identify and classify entities that has
into multi-domain NER. This challenge was introduced not encountered during its training phase. The tag set
in the NERMuD task (NER Multi-Domain) at EVALITA comprises fine-grained categories which are often specif-
20232 , in which one sub-task required to develop a single ically defined for the domain in which NER is deployed.
model capable of classifying the common entities - person, Because of this, the input data may often be also Out-
organization, location - from different types of text, in- Of-Domain (OOD), making this evaluation include the
cluding news, fiction and political speeches. ExtremITA previously mentioned OOD scenario as well.
team [18] addressed the challenge proposing the adop-
tion of a single LLM capable of tackling all the different 4. SLIMER-IT
tasks at EVALITA 2023, among which NERMuD. All the
tasks were converted into text-to-text problems and two To adapt SLIMER for Italian, we translate the instruction-
LLMs (LLaMA and T5 based) were instruction-tuned on tuning prompt of [13], as shown in Figure 1. The prompt
the union of all the available datasets for the challenge. is designed to extract the occurrences of one entity type
per call. While this has the drawback of requiring |NE|
2
https://www.evalita.it/campaigns/evalita-2023/tasks/
inference calls on each input text, it allows the model to vehicle. We keep the Italian examples only. Such a dataset
better focus on a single NE type at a time. constitutes a perfect choice to assess models’ capabilities
As in [13], we query gpt-3.5-turbo-1106 via OpenAI’s on unseen NEs. Indeed, data belongs to the same news
Chat-GPT APIs to automatically generate definition and domain of the NERMuD split chosen for fine-tuning, but
guidelines for each needed entity tag. The definition for it includes a broader label set. Since we want to measure
a NE is meant to be a short sentence describing the tag. performance on never-seen-before entities, we exclude
The guidelines instead provide annotation instructions entity types seen in training, i.e. person, organization and
to align the model’s labelling with the desired annotation location. We also remove biological entity, being poorly
scheme. Guidelines can be used to prevent the model underrepresented, with a support of just 4 instances.
from labelling certain edge cases or to provide examples
of such NE. Such an informative prompt is extremely 5.2. Backbones
valuable when dealing with unfamiliar entity tags, and
can also be used to distinguish between polysemous cat- We implemented several version of SLIMER-IT based on
egories. different backbone models. We consider similarly sized
Finally, the model is requested to generate the named LLMs, all in the 7B parameters range. In particular, we
entities in a parsable JSON format containing the list of selected five backbones: Camoscio4 [21], LLaMA-2-7b-
NEs extracted for the given tag. chat [22], Mistral-7B-Instruct [23], LLaMA-3-8B-Instruct,
LLaMAntino-3-ANITA-8B-Inst-DPO-ITA5 [24].
LLaMA-2-7b-chat was originally used in SLIMER [13],
5. Experiments and LLaMA-3-8B-Instruct is the newest, improved ver-
sion of it. As LLaMA family, Mistral-7B-Instruct is a
Experiments aim to assess our approach in Italian. We multilingual model mainly English-oriented, but it has
study the impact of guidelines and the usage of different demonstrated greater fluency on Italian. Camoscio and
backbones. Then, we compare our approach against state- LLaMAntino-3-ANITA-8B-Inst-DPO-ITA, instead, are
of-the-art alternatives. two LLMs specifically fine-tuned on Italian instructions.
5.1. Datasets 5.3. Compared Models
We construct the zero-shot NER framework (described We compare the SLIMER-IT approach, implemented with
in Section 3) for Italian upon NerMuD shared task and different backbones, against other state-of-the-art ap-
Multinerd dataset. In particular, we use NerMuD to build proaches for zero-shot NER. All the methods are trained
in-domain and OOD evaluation sets, while Multinerd- and evaluated in the defined zero-shot NER framework
IT is used to assess the behaviour in the unseen named for a fair comparison. We evaluate against:
entites scenario.
Token classification. Although certainly not being
NERMuD. NERMuD [1] is a shared task organized at
suited for zero-shot NER, due to its architectural inability
evalita-2023, built based on the Kessler Italian Named-
to cope with unseen tags, we decided to evaluate the most
entities Dataset (KIND) [19]. It contains annotations
known approach to NER as baseline. As in NERMuD
for the three classic NER tags: person, organization and
[1], we use the training framework dhfbk/bert-ner 6 . We
location. Examples are organized in three distinct do-
fine-tune two different base models, bert-base-cased, pre-
mains: news, literature and political discourses. Unlike
trained on English, and dbmdz/bert-base-italian-cased 7 ,
NERMuD, we restrict fine-tuning to a single domain. In
an Italian version.
such a way, we can evaluate both in-domain and out-
of-domain capabilities of the model. In particular, we
designate WikiNews (WN) sub-set for training and in- GNER. It is the best performing approach on zero-shot
domain evaluation, being the most generic domain, while NER in OOD English benchmark. In GNER [12], they
Fiction (FIC) and Alcide De Gasperi (ADG) splits are kept propose a BIO-like generation, replicating in output the
for out-of-domain evaluation only. same input text, along with a token-by-token BIO label.
Here, we consider LLaMAntino-3 as its backbone.
Multinerd-IT. To construct the unseen NEs evalua-
tion set, we exploit Multinerd3 [20], a multilingual NER
dataset made of 15 tags: person, organization, location, an- 4
https://huggingface.co/teelinsan/camoscio-7b-llama
5
imal, biological entity, celestial body, disease, event, food, https://huggingface.co/swap-uniba/
instrument, media, plant, mythological entity, time and LLaMAntino-3-ANITA-8B-Inst-DPO-ITA
6
https://github.com/dhfbk/bert-ner
3 7
https://github.com/Babelscape/multinerd https://huggingface.co/dbmdz/bert-base-italian-cased
Table 1
Comparing SLIMER-IT based on different backbones, with and without Definition and Guidelines (D&G) in the prompt. LLMs
with † symbol were instruction-tuned on Italian. In parentheses the (±Δ𝐹 1) of performance given by the usage of D&G.
Backbone Params w/ D&G In-Domain OOD unseen NEs
WN FIC ADG MN
False 81.80 82.44 79.01 32.28
Camoscio † 7B
True 81.50 (-0.3) 85.08 (+2.64) 76.00 (-3.01) 38.68 (+6.4)
False 80.69 80.45 73.81 32.38
LLaMA-2-chat 7B
True 83.24 (+2.55) 88.81 (+8.36) 79.26 (+5.45) 35.16 (+2.78)
False 82.71 85.61 75.80 35.63
Mistral-Instruct 7B
True 85.55 (+2.84) 92.78 (+7.17) 80.56 (+4.76) 40.64 (+5.01)
False 85.93 82.85 80.00 27.62
LLaMA-3-Instruct 8B
True 85.38 (-0.55) 84.38 (+1.53) 78.29 (-1.71) 50.74 (+23.12)
False 84.12 77.06 74.35 30.90
LLaMAntino-3-ANITA † 8B
True 85.78 (+1.66) 82.52 (+5.46) 81.65 (+7.30) 54.65 (+23.75)
100
GLiNER. Differently from all other methods, GLiNER 90
Camoscio
LLaMA2
is based on a smaller encoder-only model, combined with 80 Mistral
LLaMA3
a span classifier head, able to achieve competitive per- 70 LLaMAntino3
formance on the OOD English benchmark at a lower 60
50
-F1
computational cost. We fine-tune it both using its orig-
40
inal deberta-v3-large English backbone and the Italian 30
dbmdz/bert-base-italian-cased model. 20
10
extremITLLaMA. Already described in Section 2, it WN (supervised) FIC (OOD) ADG (OOD) MN (unseen NEs)
represents an interesting approach to compare against.
Based on Camoscio LLM, we compare it with SLIMER-IT Figure 2: SLIMER-IT performance for different backbones.
approach implemented with the same backbone.
Table 2
5.4. Experimental setup Comparison with existing off-the-shelf models for zero-shot
NER on Italian. We omit in-domain evaluation to not disad-
We kept the same training configuration of SLIMER [13] vantage them against SLIMER-IT.
on English, except that we trained on all available
samples. Depending on the backbone, the instruction- Model OOD unseen NEs
tuning prompt (see Figure 1) was adjusted accord- FIC ADG MN
ingly to the structure of its template (e.g. [INST] or Universal-NER-ITA 32.4 43.2 12.8 (all seen)
<|start_header_id|> formats). For all the competitors, we GLiNER-ITA-Large 36.6 42.0 15.5 (all seen)
replicated their training setup using their scripts and sug- GLiNER-ML 46.5 49.4 17.4 (all seen)
gested hyper-parameters. For the evaluation, we use the
micro-F1 as computed in the UniNER8 implementation. SLIMER-IT 82.5 81.7 54.7
5.5. Results D&Gs and the one not using them. Generally, definition
Impact of Definition and Guidelines (D&G). We and guidelines yield improvements in F1. In particular,
compare SLIMER-IT with a version devoid of definition the gap is contained when evaluating on in-domain data,
and guidelines in the prompt. To demonstrate the ro- whereas it becomes significant in OOD and even more
bustness of the approach, we train several SLIMER-IT substantial in unseen NEs. This is expected since D&G
instances, based on different LLM backbones. In Table help the most in conditions unseen during training. No-
1, we report the results, highlighting the absolute dif- tably, LLaMA-3-based backbones benefit the most from
ference in performance between the model steered by definition and guidelines, with improvements beyond
23 absolute F1 points, surpassing all the other models
8
https://github.com/universal-ner by substantial margins in never-seen-before entity tags.
Table 3
Comparing SLIMER-IT with state-of-the-art approaches trained in the same zero-shot setting, and adopting the same backbone
when possible. *Note that extremITLLaMA was fine-tuned also on the FIC and ADG train sets for the NERMuD task, so these
datasets are not actually OOD for this model.
Approach Backbone Language Params In-Domain OOD unseen NEs
WN FIC ADG MN
Token classification BERT-base EN 0.11B 83.9 75.6 75.0 -
Token classification BERT-base IT 0.11B 89.8 87.0 82.3 -
GLiNER deberta-v3-large EN 0.44B 87.8 77.2 80.3 0.2
GLiNER BERT-base IT 0.11B 89.3 87.5 84.9 0.6
extremITLLaMA Camoscio IT 7B 89.1 90.3* 83.4* 0.2
SLIMER-IT Camoscio IT 7B 81.5 85.1 76.0 38.7
GNER LLaMAntino-3 IT 8B 90.3 88.9 82.5 1.2
SLIMER-IT LLaMAntino-3 IT 8B 85.8 82.5 81.7 54.7
Some qualitative examples are shown in Appendix A. State-of-the-art comparison. Thanks to the defini-
tion of our zero-shot evaluation framework, we can com-
Impact of Backbones. Regarding the choice of the pare different state-of-the-art approaches fairly. Results
SLIMER-IT backbone, we better illustrate results in Fig- are outlined in Table 3. When evaluating in the same
ure 2. We can observe no remarkable difference in in- domain where the model was trained, encoder-only archi-
domain evaluation, where most recent models outper- tectures obtain strong results despite being much smaller
form older ones, as one might expect. Also globally, models. This result is not surprising, given the acknowl-
Camoscio and LLaMA-2-chat obtain lower scores than edged performance of these architectures for supervised
the rest of the backbones, with the only exception of NER. More unexpected is their ability to generalize well
FIC dataset, where LLaMA-3 based architecture under- to OOD inputs. Also GNER proves to be quite competitive
perform. However, LLaMAntino-3-ANITA reaches the achieving the best results in in-domain evaluation, and
best performance on 3 out of 4 datasets, with a strong gap in OOD on FIC dataset. However, all these approaches
especially in unseen named entities scenario, the most dramatically fail on never-seen-before tags, in contrast
challenging one. Interestingly enough, thanks to their to SLIMER-IT that achieves almost 55 F1 score points.
better understanding capabilities, backbones specialized Compared with LLM-based approaches like GNER and
on Italian are particularly effective in the unseen NEs sce- extremITLLaMA, this proves once again that without
nario. This is the case of LLaMAntino-3-ANITA and even definition and guidelines LLMs struggle in tagging novel
Camoscio, which demonstrates higher F1 than LLaMA-2. kind of entities.
Off-the-shelf Italian NER models. Although there 6. Conclusions
has been no prior work defining a Zero-Shot NER eval-
uation framework for Italian, there exist fine-tune spe- In this paper, we proposed an evaluation framework for
cialized state-of-the-art zero-shot NER models for Italian Zero-Shot NER that we applied to Italian. Thanks to such
language. In particular, we consider: GLiNER-ML [11], a framework, we can better investigate different zero-shot
a multilingual instance of GLiNER, Universal-NER-ITA9 properties depending on the scenario (in-domain, OOD,
and GLiNER-ITA-Large10 , both specialized on Italian. unseen NEs). On top of that, we compared several state-
These models were trained on synthetic data covering a of-the-art approaches, with particular focus on SLIMER,
vast number of different entity classes (up to 97k). Thus, which, thanks to the usage of definition and guidelines,
it is impossible to directly compare them in a pure zero- is well suited to deal with novel entity types. Indeed,
shot framework, since there are no entity tags actually SLIMER-IT, our fine-tuned model based on LLaMAntino-
never-seen-before during training. However, we still re- 3, surpasses other state-of-the-art techniques by large
port their results against SLIMER-IT. Table 2 reports the margins. In the future, we plan to further extend the zero-
results. Despite this advantage, SLIMER-IT outperforms shot NER benchmark, and implement an input caching
all these models by large a margin. mechanism for scalability to large label sets.
9
https://huggingface.co/DeepMount00/universal_ner_ita
10
https://huggingface.co/DeepMount00/GLiNER_ITA_LARGE
Acknowledgments [9] Y. Wang, et al., Super-Natural Instructions: Gen-
eralization via declarative instructions on 1600+
The work was partially funded by: NLP tasks, in: Y. Goldberg, Z. Kozareva, Y. Zhang
(Eds.), Proceedings of the 2022 Conference on Em-
• “ReSpiRA - REplicabilità, SPIegabilità e Ragiona-
pirical Methods in Natural Language Processing,
mento”, a project financed by FAIR, Affiliated to
Association for Computational Linguistics, Abu
spoke no. 2, falling within the PNRR MUR pro-
Dhabi, United Arab Emirates, 2022, pp. 5085–5109.
gramme, Mission 4, Component 2, Investment 1.3,
URL: https://aclanthology.org/2022.emnlp-main.
D.D. No. 341 of 03/15/2022, Project PE0000013,
340. doi:10.18653/v1/2022.emnlp-main.340.
CUP B43D22000900004 11 ;
[10] X. Wang, W. Zhou, C. Zu, H. Xia, T. Chen, Y. Zhang,
• “MAESTRO - Mitigare le Allucinazioni dei Large R. Zheng, J. Ye, Q. Zhang, T. Gui, et al., Instructuie:
Language Models: ESTRazione di informazioni multi-task instruction tuning for unified informa-
Ottimizzate” a project funded by Provincia Au- tion extraction, arXiv preprint arXiv:2304.08085
tonoma di Trento with the Lp 6/99 Art. 5:ricerca e (2023).
sviluppo, PAT/RFS067-05/06/2024-0428372, CUP: [11] U. Zaratiana, N. Tomeh, P. Holat, T. Charnois,
C79J23001170001 12 ; Gliner: Generalist model for named entity
• “enRichMyData - Enabling Data Enrichment recognition using bidirectional transformer, 2023.
Pipelines for AI-driven Business Products and arXiv:2311.08526.
Services”, an Horizon Europe (HE) project, grant [12] Y. Ding, J. Li, P. Wang, Z. Tang, B. Yan, M. Zhang, Re-
agreement ID: 101070284 13 . thinking negative instances for generative named
entity recognition, 2024. arXiv:2402.16602.
References [13] A. Zamai, A. Zugarini, L. Rigutini, M. Ernandes,
M. Maggini, Show less, instruct more: Enrich-
[1] A. P. Aprosio, T. Paccosi, Nermud at evalita 2023: ing prompts with definitions and guidelines for
Overview of the named-entities recognition on zero-shot ner, 2024. URL: https://arxiv.org/abs/2407.
multi-domain documents task (short paper), in: In- 01272. arXiv:2407.01272.
ternational Workshop on Evaluation of Natural Lan- [14] M. Marrero, J. Urbano, S. Sánchez-Cuadrado,
guage and Speech Tools for Italian, 2023. URL: https: J. Morato, J. M. Gómez-Berbís, Named entity
//api.semanticscholar.org/CorpusID:261529782. recognition: Fallacies, challenges and opportunities,
[2] J. Li, A. Sun, J. Han, C. Li, A survey on deep learning Computer Standards & Interfaces 35 (2013) 482–
for named entity recognition, IEEE Transactions on 489. URL: https://www.sciencedirect.com/science/
Knowledge and Data Engineering 34 (2020) 50–70. article/pii/S0920548912001080. doi:https://doi.
[3] O. Sainz, et al., Gollie: Annotation guidelines org/10.1016/j.csi.2012.09.004.
improve zero-shot information-extraction, 2024. [15] B. Magnini, E. Pianta, C. Girardi, M. Negri, L. Ro-
arXiv:2310.03668. mano, M. Speranza, V. Bartalesi Lenzi, R. Sprugnoli,
[4] W. Zhou, S. Zhang, Y. Gu, M. Chen, H. Poon, Uni- I-CAB: the Italian content annotation bank, in:
versalner: Targeted distillation from large language N. Calzolari, K. Choukri, A. Gangemi, B. Maegaard,
models for open named entity recognition, arXiv J. Mariani, J. Odijk, D. Tapias (Eds.), Proceedings
preprint arXiv:2308.03279 (2023). of the Fifth International Conference on Language
[5] A. Radford, et al., Language models are unsuper- Resources and Evaluation (LREC’06), European Lan-
vised multitask learners, OpenAI blog 1 (2019) 9. guage Resources Association (ELRA), Genoa, Italy,
[6] T. Brown, et al., Language models are few-shot 2006. URL: http://www.lrec-conf.org/proceedings/
learners, Advances in neural information process- lrec2006/pdf/518_pdf.pdf.
ing systems 33 (2020) 1877–1901. [16] V. Bartalesi Lenzi, M. Speranza, R. Sprugnoli,
[7] J. Wei, et al., Finetuned language models are Named entity recognition on transcribed broadcast
zero-shot learners, in: International Conference news at evalita 2011, in: B. Magnini, F. Cutugno,
on Learning Representations, 2022. URL: https: M. Falcone, E. Pianta (Eds.), Evaluation of Natural
//openreview.net/forum?id=gEZrGCozdqR. Language and Speech Tools for Italian, Springer
[8] H. W. Chung, et al., Scaling instruction-finetuned Berlin Heidelberg, Berlin, Heidelberg, 2013, pp. 86–
language models, 2022. arXiv:2210.11416. 97.
[17] P. Basile, A. Caputo, A. Gentile, G. Rizzo, Overview
11
RESPIRA: https://www.opencup.gov.it/portale/web/opencup/ of the evalita 2016 named entity recognition and
home/progetto/-/cup/B43D22000900004 linking in italian tweets (neel-it) task, 2016.
12
MAESTRO: https://www.opencup.gov.it/portale/web/opencup/
home/progetto/-/cup/C79J23001170001 [18] C. D. Hromei, D. Croce, V. Basile, R. Basili, Extrem-
13
https://doi.org/10.3030/101070284 ita at EVALITA 2023: Multi-task sustainable scaling
to large language models at its extreme, in: M. Lai, A. SLIMER-IT on some NE tags
S. Menini, M. Polignano, V. Russo, R. Sprugnoli,
G. Venturi (Eds.), Proceedings of the Eighth Evalua- In Table 4 we compare SLIMER-IT (LLaMAntino-based)
tion Campaign of Natural Language Processing and with a version of it devoid of Definition and Guidelines
Speech Tools for Italian. Final Workshop (EVALITA (D&G), in order to get a better insight into the usefulness
2023), Parma, Italy, September 7th-8th, 2023, vol- of such components in zero-shot NER. We present results
ume 3473 of CEUR Workshop Proceedings, CEUR- for both unseen named entities (from Multinerd) and pre-
WS.org, 2023. URL: https://ceur-ws.org/Vol-3473/ viously seen tags person, location and organization, but in
paper13.pdf. out-of-domain inputs (ADG and FIC datasets). The D&G
[19] T. Paccosi, A. Palmero Aprosio, KIND: an Italian components improve performance by up to 37 points for
multi-domain dataset for named entity recognition, unseen named entities, serving as a source of additional
in: N. Calzolari, F. Béchet, P. Blache, K. Choukri, knowledge to the model and providing annotation direc-
C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Mae- tives about what should be labeled. Particularly for these
gaard, J. Mariani, H. Mazo, J. Odijk, S. Piperidis named entities, the D&G enhance precision by reducing
(Eds.), Proceedings of the Thirteenth Language the number of false positives the model would otherwise
Resources and Evaluation Conference, European generate. The performance gain provided by D&G for
Language Resources Association, Marseille, France, known tags within out-of-domain inputs is smaller, with
2022, pp. 501–507. URL: https://aclanthology.org/ improvements of up to 17 points on some named entity
2022.lrec-1.52. tags. In this context, the definitions and guidelines serve
[20] S. Tedeschi, R. Navigli, MultiNERD: A multi- more as a reasoning support than as a source of additional
lingual, multi-genre and fine-grained dataset for knowledge.
named entity recognition (and disambiguation),
in: Findings of the Association for Computa-
tional Linguistics: NAACL 2022, Association for
Computational Linguistics, Seattle, United States,
2022, pp. 801–812. URL: https://aclanthology.org/
2022.findings-naacl.60. doi:10.18653/v1/2022.
findings-naacl.60.
[21] A. Santilli, E. Rodolà, Camoscio: an italian
instruction-tuned llama, 2023. URL: https://arxiv.
org/abs/2307.16456. arXiv:2307.16456.
[22] H. Touvron, et al., Llama 2: Open foun-
dation and fine-tuned chat models, 2023.
arXiv:2307.09288.
[23] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bam-
ford, D. S. Chaplot, D. de las Casas, F. Bressand,
G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-
A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang,
T. Lacroix, W. E. Sayed, Mistral 7b, 2023. URL: https:
//arxiv.org/abs/2310.06825. arXiv:2310.06825.
[24] M. Polignano, P. Basile, G. Semeraro, Advanced
natural-based interaction for the italian language:
Llamantino-3-anita, 2024. URL: https://arxiv.org/
abs/2405.07101. arXiv:2405.07101.
Table 4
Some examples of definition and guidelines. Absolute F1 gains between SLIMER-IT and its version without definition and
guidelines are reported. In green we highlight examples on unseen named entities, in blue examples on known tags such
person, organization and location, but in Out-Of-Domain input distributions.
NE (dataset) Definition & Guidelines w/o D&G F1 w/ D&G F1 Δ F1
Corpo celeste (MN) Definizione: ’CORPO CELESTE’ si riferisce a oggetti astronomici 27.07 64.00 +36.93
come pianeti, stelle, satelliti, costellazioni, galassie, comete e as-
teroidi. Linee guida: Evita di etichettare come ’corpo celeste’
entità non direttamente collegate al campo dell’astronomia. Ad
esempio, ’Vergine’ potrebbe riferirsi anche a un segno astrologico,
quindi il contesto è importante. Assicurati di non includere nomi
di fenomeni non astronomici come ’alba’ o ’tramonto’. Potresti
incontrare ambiguità quando un termine è usato sia in campo
astronomico che in contesti non astronomici, ad esempio ’aurora’
che può riferirsi sia all’evento astronomico che al nome di persona.
Pianta (MN) Definizione: ’PIANTA’ si riferisce a organismi vegetali come alberi, 13.76 49.89 +36.13
arbusti, erbe e altre forme di vegetazione., Linee Guida: Quando
identifichi entità ’pianta’, assicurati di etichettare solo nomi di
specie vegetali specifiche, come ’Fagus sylvatica’, ’Suaeda vera’,
’Betula pendula’, evitando generici come ’alberi’ o ’arbusti’ se non
accompagnati da una specificazione della specie.
Media (MN) Definizione: ’MEDIA’ si riferisce a entità come nomi di giornali, 47.78 65.86 +18.08
riviste, libri, album musicali, film, programmi televisivi, spettacoli
teatrali e altre opere creative e di comunicazione., Linee Guida:
Assicurati di etichettare solo nomi specifici di opere creative e di
comunicazione, evitando generici come ’musica’ o ’libro’. Presta
attenzione alle ambiguità, ad esempio ’Apple’ potrebbe riferirsi
alla società tecnologica o ad un’opera d’arte. Escludi i nomi di
artisti, autori o registi, che dovrebbero essere etichettati come
’persona’, e nomi generici di strumenti musicali o generi letterari
che non rappresentano opere specifiche.
Luogo (FIC) Definizione: ’LUOGO’ denota nomi propri di luoghi geografici, 59.34 76.32 +16.98
comprendendo città, paesi, stati, regioni, continenti, punti di inter-
esse naturale, e indirizzi specifici., Linee Guida: Assicurati di non
confondere i nomi di luoghi con nomi di persone, organizzazioni o
altre entità. Ad esempio, ’Washington’, potrebbe riferirsi alla città
di Washington D.C. o al presidente George Washington, quindi
considera attentamente il contesto. Escludi nomi di periodi storici,
eventi o concetti astratti che non rappresentano luoghi fisici. Ad
esempio, ’nel Rinascimento’ è un periodo storico, non un luogo
geografico.
Organizzazione (ADG) Definizione: ’ORGANIZZAZIONE’ denota nomi propri di aziende, 55.56 71.85 +16.29
istituzioni, gruppi o altre entità organizzative. Questo tipo di
entità include sia entità private che pubbliche, come società, orga-
nizzazioni non profit, agenzie governative, università e altri gruppi
strutturati. Linee Guida: Annota solo nomi propri, evita di anno-
tare sostantivi comuni come ’azienda’ o ’istituzione’ a meno che
non facciano parte del nome specifico dell’organizzazione. Assicu-
rati di non annotare nomi di persone come organizzazioni, anche
se contengono termini che potrebbero sembrare riferimenti a en-
tità organizzative. Ad esempio, ’Johnson & Johnson’ è un’azienda,
mentre ’Johnson’ da solo potrebbe essere il cognome di una per-
sona.
Persona (FIC) Definizione: ’PERSONA’ denota nomi propri di individui umani. 79.72 83.33 +3.61
Questo tipo di entità comprende nomi di persone reali, famose
o meno, personaggi storici, e può includere anche personaggi
di finzione. Linee Guida: Fai attenzione a non includere titoli o
ruoli professionali senza nomi propri (es. ’il presidente’ non è una
’PERSONA’, ma ’il presidente Barack Obama’ sì).