ITA-SENSE - Evaluate LLMs’ ability for ITAlian word SENSE disambiguation: A CALAMITA Challenge Pierpaolo Basile1,*,† , Elio Musacchio2,† and Lucia Siciliani1,† 1 Dipartimento di Informatica, Università degli Studi di Bari Aldo Moro (ITALY) 2 Italian National PhD Program in Artificial Intelligence, University of Bari Aldo Moro, Bari (ITALY) Abstract The challenge is designed to assess LLMs’ abilities in understanding lexical semantics through Word Sense Disambiguation, providing valuable insights into their performance. The idea is to cast the classical Word Sense Disambiguation task in a generative problem following two directions. Our idea is to propose two tasks: (T1) Given a target word and a sentence in which the word occurs, the LLM must generate the correct meaning definition, (T2) Given a target word and a sentence in which the word occurs, the LLM should choose from a predefined set the correct meaning definition. For T1, we compare the generated definition with respect to the correct one taken from a sense inventory, while for T2, a classical accuracy metric is used. In T1, we adopt metrics that measures the quality of the generated definition such as RougeL and the BERTscore. For CALAMITA, we test LLMs using a zero-shot setting. Keywords Natural Language Processing, Word Sense Disambiguation, Large Language Models 1. Challenge: Introduction and on massive text datasets, show outstanding capabilities in understanding and generating human-like language. Motivation LLMs have demonstrated their ability to solve tasks in Word Sense Disambiguation (WSD) [1, 2] is a Natural zero-shot or few-shot settings, i.e. providing them a Language Processing task that aims to build a system ca- prompt without specific training data, though fine-tuning pable of disambiguating a word occurrence and assigning for specific tasks is also possible. Their success suggests it the correct sense from an inventory defined a priori, an inherent ability to grasp language semantics. like WordNet [3]. Nevertheless, numerous challenges and issues remain Being a long-standing task in the field of NLP, several related to LLMs and their actual performance. A key techniques have been employed to solve it, reflecting difficulty lies in determining to what extent LLMs are the evolution of advances in machine learning. We can capable of understanding the meaning of a given task mainly distinguish two main phases. Initially, rule-based rather than merely juxtaposing text coherently. For this systems dominated, followed by knowledge-based meth- reason, tasks like WSD can help to shed light on these ods when digital sense inventories became available. As issues, as they target specific aspects of natural language. digital corpora emerged, supervised approaches took ad- In particular, WSD requires a deep understanding of word vantage of manually annotated data. The vast corpora meanings in context. available on the web and large knowledge graphs further WSD is a task particularly intertwined with the lan- transformed supervised and knowledge-based methods. guage to be analyzed. In Italian, for example, many words The introduction of transformer-based [4] language have multiple meanings that can only be adequately un- models marked a new era within the field. These models derstood in context. This is particularly challenging with represent words in context using dense vectors, offering words with high degree of polysemy. Addressing these new opportunities for word meaning disambiguation. ambiguities in Italian makes WSD important for accu- Recently, Large Language Models (LLMs) have revolu- rately representing the richness of this language. In the tionized the research in computational linguistics. These past, several evaluation campaigns have been organized models, built on the transformer architecture and trained such as SensEval and SemEval. Regarding model performance, we expect LLMs to per- CLiC-it 2024: Tenth Italian Conference on Computational Linguistics, form reasonably well at disambiguating common mean- Dec 04 — 06, 2024, Pisa, Italy ings. However, these models may struggle with rare * Corresponding author. cases (e.g., idiomatic expressions and words belonging † These authors contributed equally. to particular domains). We expect fine-tuning on Italian $ pierpaolo.basile@uniba.it (P. Basile); elio.musacchio@uniba.it corpora to be essential in developing an LLM capable of (E. Musacchio); lucia.sicialiani@uniba.it (L. Siciliani)  0000-0002-0545-1105 (P. Basile); 0009-0006-9670-9998 addressing this task. The complexity that characterizes (E. Musacchio); 0000-0002-1438-280X (L. Siciliani) Italian morphology and polysemy can be a real challenge © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings for LLMs unless they are provided extensive language- Sentence “...nonostante l’espulsione di Split- ter, la squadra di Ivonic ha man- specific knowledge. For the above reasons, we designed a tenuto il ritmo, ha difeso bene...” specific benchmark for CALAMITA [5] to evaluate LLMs’ Possible 1) Rapporto tra due quantità ability in Italian Word Sense Disambiguation. choices nell’unità di tempo. 2) Ritmo di marcia o di corsa. 3) Il ritmo è una successione di 2. Challenge: Description accenti forti e deboli ed eventuali pause, intervallati nel dominio del Our benchmark aims to measure how an LLM can solve tempo da pochi decimi di secondo the WSD task for understanding if the model somehow a qualche secondo, che seguono, di stores knowledge about word meanings. The benchmark solito ma non obbligatoriamente, is composed of two tasks: uno o più modelli ciclici. 4) Alternanza di sillabe di tipi 1. Given a sentence and an occurrence of the tar- diversi. get word, the model is tested in generating the definition of the word; Expected out- 2 2. Given a sentence, the list of possible definitions put and an occurrence of the target word, the model is Table 2 evaluated in selecting the correct definition from Example of task 2. the predefined set of possible choices. Given the same sentence and the target word “squadra”, Tables 1 and 2 show the two tasks. Task 1 aims at mea- all senses in BabelNet have an Italian gloss. For this suring the LLM ability to generate a definition given a reason, we build two versions of the dataset: without word in a specific context, while Task 2 aims to test the translation in which we consider only the word occur- capability of selecting the correct definition from a set rences that have Italian glosses in BabelNet, and with of predefined possibilities. The Task 2 is more similar to translation in which English glosses1 are automatically how the WSD problem is classically formulated in litera- translated in Italian. For the translation, we use the 1.3B ture, while Task 1 is designed to evaluate the generation variant of the Meta NLLB-200 model2 . capabilities. 3.2. Data format Sentence “...nonostante l’espulsione di Split- ter, la squadra di Ivonic ha man- We will introduce some formal notations before delv- tenuto il ritmo, ha difeso bene...” ing into the description of the benchmark construction. Expected out- Ritmo di marcia o di corsa. Given a sentence 𝑆𝑘 and one of its word occurrences 𝑤𝑖 , put we define 𝐿𝑖 as the list of possible meanings of 𝑤𝑖 and Table 1 𝑚𝑗 ∈ 𝐿𝑖 , the meaning assigned to 𝑤𝑖 . Each meaning Example of task 1. has several glosses, we use 𝑚𝑗 ∈ 𝐿𝑖 to refer to it. We need a strategy for building prompts for the two tasks, starting from the Italian sense-annotated corpus and the corresponding sense inventory. 3. Data description Task 1 aims to assess the LLM’s ability to generate an accurate definition of a word in a specific sentence. 3.1. Origin of data We create the prompt reported in Table 3 for each sense annotated word occurrence. In the dataset, we also store To create our benchmark, we need an Italian sense- the correct definition 𝑚𝑗 in a field called output. annotated corpus, i.e., a collection of sentences in which During the construction of the dataset, we need to each word is tagged with its correct meaning taken from manage the cases in which a word 𝑤𝑖 occurs more a sense inventory. For this reason, we also require an than once in the sentence 𝑆𝑘 . In these cases, we Italian sense inventory that provides the set of possible change the prompt as follows: “Give a brief definition meanings for each word. of the 𝑥 occurrence of the word "𝑤𝑖 "...”, where 𝑋 = We use XL-WSD [6] as our sense-annotated corpus. {𝑓 𝑖𝑟𝑠𝑡, 𝑠𝑒𝑐𝑜𝑛𝑑, 𝑡ℎ𝑖𝑟𝑑, 𝑓 𝑜𝑢𝑟𝑡ℎ, 𝑓 𝑖𝑓 𝑡ℎ} and 𝑥 ∈ 𝑋. We This dataset serves as a cross-lingual evaluation bench- exclude cases where the word occurs more than six times, mark for the WSD task, featuring sense-annotated devel- and we translate the set 𝑋 according to each language. opment and test sets in 18 languages (including Italian) from six different linguistic families. The sense inven- 1 The English gloss is always available. tory adopted in XL-WSD is BabelNet [7]. However, not 2 https://huggingface.co/facebook/nllb-200-1.3B Prompt template (generation) words are not considered in the construction of both Give a brief definition of the word "𝑤𝑖 " in the sen- tasks4 . tence given as input. Generate only the definition. Input: "𝑆𝑘 " English prompt 3.3. Example of prompts used for zero Give a brief definition of the word "art" in the sen- or/and few shots tence given as input. Generate only the definition. Input: "The art of change-ringing is peculiar to the Our challenge allows only zero-shot. Table 5 reports the English, and, like most English peculiarities, unin- prompt used in Task 1. telligible to the rest of the world." Prompt template (generation) Table 3 Fornisci una breve definizione della parola "𝑤𝑖 " nella Prompt for the generation benchmark. frase data in input. Genera solo la definizione. Input: "𝑆𝑘 " Italian prompt The goal of Task 2 is to evaluate the LLM’s ability to Fornisci una breve definizione della parola "sforzo" select the correct sense from a set of predefined possi- nella frase data in input. Genera solo la definizione. bilities. In this case, we exploit the list of all possible Input: "Che sforzo fate per valutare i risultati del meanings 𝐿𝑖 . In particular, from 𝐿𝑖 , we remove all the vostro programme?" annotated meanings3 and obtain the set 𝐶𝑖 . Then, we Table 5 randomly add to 𝐶𝑖 one of the correct meanings; in this Prompt for the Italian generation task. way, 𝐶𝑖 contains only one correct sense. For each oc- currence of a sense-annotated word in the corpus, we Table 6 reports the prompt for Task 2. create the prompt in Table 4. Additionally, we store the identifier (i.e. the option’s number) corresponding to the Prompt template (multiple choice) correct answer in a field called output. Data la parola "𝑤𝑖 " nella frase in input, scegli il significato corretto tra i seguenti: 𝐶𝑖 . Genera solo Prompt template (multiple choice) il numero dell’opzione selezionata. Input: "𝑆𝑘 " Given the word "𝑤𝑖 " in the input sentence, choose Italian prompt the correct meaning from the following: 𝐶𝑖 . Gener- Data la parola "valutare" nella frase in input, scegli ate only the number of the selected option. il significato corretto tra i seguenti: English prompt 1) Esaminare o ascoltare (prove o un intero caso) per Given the word "art" in the input sentence, choose via giudiziaria. the correct meaning from the following: 2) Fare la stima commerciale di qlco. 1) Photographs or other visual representations in a 3) Assegnare un valore a. printed publication 4) Ritenere dopo valutazione. 2) A superior skill that you can learn by study and 5) Apprezzare, tenere in grande stima. practice and observation 6) Avere una certa opinione di qualcuno. 3) The products of human creativity; works of art Genera solo il numero dell’opzione selezionata. collectively Input: "Che sforzo fate per valutare i risultati del 4) The creation of beautiful or significant things. vostro programme?" Generate only the number of the selected option. Input: "The art of change-ringing is peculiar to the Table 6 English, and, like most English peculiarities, unin- Prompt for the Italian multiple choice task. telligible to the rest of the world." Table 4 3.4. Detailed data statistics Prompt for the multiple choice benchmark. Table 3.4 reports the number of instances for each task. We also manage the case where the word 𝑤𝑖 occurs We also report different statistics for the dataset without more than once by modifying the prompt as in Task 1. translation and the one with machine translation. Moreover, given that the model is asked to choose among different options in Task 2, we need to manage cases in which the size of 𝐶𝑖 is less than two. In these cases, we remove the occurrence from the dataset. Monosemic 4 For Task 1 based on definition generation, it is also possible to 3 In the sense-annotated corpus, a word occurrence can be annotated consider monosemic words. We exclude this hypothesis since we with more than one correct meaning. want to test LLMs in the case of polysemy. Task 1 Task 2 5. Limitations without translation 1,673 1,529 with translation 1,888 1,823 We cannot guarantee that texts presented in XL-WSD do Table 7 not occur in the training data of some LLMs. However, Dataset statistics. even if the model is exposed to textual data from XL- WSD, it does not necessarily mean that it was asked to solve the disambiguation task on such data. A fixed sense 4. Metrics inventory may not cover all Italian senses, neologisms, or emerging phrases. However, our benchmark considers The idea is to measure the correspondence between the only words (and their contexts) annotated according to generated definition and the correct one provided by the the sense inventory used in XL-WSD. This ensures that sense inventory in Task 1. For Task 2, we want to measure all instances in our benchmark have at least one correct the accuracy in selecting the correct definition from the sense in the sense inventory. set of possibilities. For the above reasons, we use three different metrics. For Task 1, we compute F1-RougeL and F1-BERTscore between the reference and generated gloss. 6. Ethical issues For Task 2, we measure the accuracy as the ratio between the correct answers and the number of instances in the No ethical issues are reported in our dataset. dataset. ROUGE (Recall-Oriented Understudy for Gisting Eval- 7. Data license and copyright uation) is a set of metrics that assess the quality of generated texts, particularly summaries, by comparing issues them with reference texts. Common variants include ROUGE-N, which measures the correspondence of n- Our data are based on the data license of the XL-WSD grams, ROUGE-L, which considers the longest com- from which our benchmark is derived. XL-WSD is dis- mon sub-sequences, and ROUGE-W, which considers the tributed under a non-commercial license5 . weight of correspondences. We select the ROUGE-L to measure the lexical correspondence between the gener- Acknowledgments ated definition and the correct one. BERTScore relies on pre-trained language models to assess the semantic simi- We acknowledge the support of the PNRR project FAIR - larity between the generated and reference definitions, Future AI Research (PE00000013), Spoke 6 - Symbiotic AI going beyond mere superficial word matching. (CUP H97G22000210007) under the NRRP MUR program If a unique score for Task 1 is necessary, we propose the funded by the NextGenerationEU. harmonic mean between RougeL and BERTscore, giving BERTscore double the weight of RougeL. The idea is to give semantic similarity more importance than word References matching. [1] N. Ide, J. Véronis, Introduction to the special issue 5 * 𝑅𝑜𝑢𝑔𝑒𝐿 * 𝐵𝐸𝑅𝑇 𝑠𝑐𝑜𝑟𝑒 on word sense disambiguation: the state of the art, (1) 4 * 𝑅𝑜𝑢𝑔𝑒𝐿 + 𝐵𝐸𝑅𝑇 𝑠𝑐𝑜𝑟𝑒 Computational linguistics 24 (1998) 1–40. [2] R. Navigli, Word sense disambiguation: A survey, We have already performed some evaluations involv- ACM computing surveys (CSUR) 41 (2009) 1–69. ing several LLMs with a medium number of parameters. [3] G. A. Miller, Wordnet: a lexical database for english, Results are reported in Table 4 and show that Llama3.1- Communications of the ACM 38 (1995) 39–41. 8B-Instruct provides the best performance in gloss gen- [4] A. Vaswani, Attention is all you need, Advances in eration (Task 1), while Gemma2-9B-Instruct achieves the Neural Information Processing Systems (2017). best accuracy. [5] G. Attanasio, P. Basile, F. Borazio, D. Croce, M. Fran- cis, J. Gili, E. Musacchio, M. Nissim, V. Patti, M. Ri- Task 1 Task 2 RougeL BERTscore Accuracy naldi, D. Scalena, CALAMITA: Challenge the Abili- Llama3.1 8B-Instruct .1363 .6985 .4604 ties of LAnguage Models in ITAlian, in: Proceedings Mistral 7B-Instruct .0747 .6532 .5324 of the 10th Italian Conference on Computational Gemma2 9B-Instruct .1221 .6986 .5840 Linguistics (CLiC-it 2024), Pisa, Italy, December 4 - December 6, 2024, CEUR Workshop Proceedings, Table 8 CEUR-WS.org, 2024. Results of several LLMs on our benchmark. 5 https://sapienzanlp.github.io/xl-wsd/license/ [6] T. Pasini, A. Raganato, R. Navigli, XL-WSD: An extra-large and cross-lingual evaluation framework for word sense disambiguation., in: Proc. of AAAI, 2021. [7] R. Navigli, S. P. Ponzetto, BabelNet: Building a very large multilingual semantic network, in: J. Hajič, S. Carberry, S. Clark, J. Nivre (Eds.), Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Association for Compu- tational Linguistics, Uppsala, Sweden, 2010, pp. 216– 225. URL: https://aclanthology.org/P10-1023.