The challenge is designed to assess LLMs' abilities in understanding lexical semantics through Word Sense Disambiguation, providing valuable insights into their performance. The idea is to cast the classical Word Sense Disambiguation task in a generative problem following two directions. Our idea is to propose two tasks: (T1) Given a target word and a sentence in which the word occurs, the LLM must generate the correct meaning definition, (T2) Given a target word and a sentence in which the word occurs, the LLM should choose from a predefined set the correct meaning definition. For T1, we compare the generated definition with respect to the correct one taken from a sense inventory, while for T2, a classical accuracy metric is used. In T1, we adopt metrics that measures the quality of the generated definition such as RougeL and the BERTscore. For CALAMITA, we test LLMs using a zero-shot setting.
for LLMs unless they are provided extensive
languagespecific knowledge. For the above reasons, we designed a
specific benchmark for CALAMITA [
Our benchmark aims to measure how an LLM can solve the WSD task for understanding if the model somehow stores knowledge about word meanings. The benchmark is composed of two tasks: 1. Given a sentence and an occurrence of the target word, the model is tested in generating the definition of the word; 2. Given a sentence, the list of possible definitions and an occurrence of the target word, the model is evaluated in selecting the correct definition from the predefined set of possible choices.
2
“...nonostante l’espulsione di Splitter, la squadra di Ivonic ha mantenuto il ritmo, ha difeso bene...” 1) Rapporto tra due quantità nell’unità di tempo. 2) Ritmo di marcia o di corsa. 3) Il ritmo è una successione di accenti forti e deboli ed eventuali pause, intervallati nel dominio del tempo da pochi decimi di secondo a qualche secondo, che seguono, di solito ma non obbligatoriamente, uno o più modelli ciclici. 4) Alternanza di sillabe di tipi diversi.
Given the same sentence and the target word “squadra”, Tables 1 and 2 show the two tasks. Task 1 aims at mea- all senses in BabelNet have an Italian gloss. For this suring the LLM ability to generate a definition given a reason, we build two versions of the dataset: without word in a specific context, while Task 2 aims to test the translation in which we consider only the word occurcapability of selecting the correct definition from a set rences that have Italian glosses in BabelNet, and with of predefined possibilities. The Task 2 is more similar to translation in which English glosses1 are automatically how the WSD problem is classically formulated in litera- translated in Italian. For the translation, we use the 1.3B ture, while Task 1 is designed to evaluate the generation variant of the Meta NLLB-200 model2. capabilities.
To create our benchmark, we need an Italian senseannotated corpus, i.e., a collection of sentences in which each word is tagged with its correct meaning taken from a sense inventory. For this reason, we also require an Italian sense inventory that provides the set of possible meanings for each word.
We use XL-WSD [6] as our sense-annotated corpus.
This dataset serves as a cross-lingual evaluation benchmark for the WSD task, featuring sense-annotated development and test sets in 18 languages (including Italian) from six diferent linguistic families. The sense inven- 1The English gloss is always available. tory adopted in XL-WSD is BabelNet [7]. However, not 2https://huggingface.co/facebook/nllb-200-1.3B
We will introduce some formal notations before delving into the description of the benchmark construction. Given a sentence and one of its word occurrences , we define as the list of possible meanings of and ∈ , the meaning assigned to . Each meaning has several glosses, we use ∈ to refer to it. We need a strategy for building prompts for the two tasks, starting from the Italian sense-annotated corpus and the corresponding sense inventory.
Task 1 aims to assess the LLM’s ability to generate an accurate definition of a word in a specific sentence. We create the prompt reported in Table 3 for each sense annotated word occurrence. In the dataset, we also store the correct definition in a field called output.
During the construction of the dataset, we need to manage the cases in which a word occurs more than once in the sentence . In these cases, we change the prompt as follows: “Give a brief definition of the occurrence of the word ""...”, where = { , , ℎ, ℎ, ℎ} and ∈ . We exclude cases where the word occurs more than six times, and we translate the set according to each language.
Give a brief definition of the word "" in the sentence given as input. Generate only the definition. Input: ""
Give a brief definition of the word "art" in the sentence given as input. Generate only the definition. Input: "The art of change-ringing is peculiar to the English, and, like most English peculiarities, unintelligible to the rest of the world."
The goal of Task 2 is to evaluate the LLM’s ability to select the correct sense from a set of predefined possibilities. In this case, we exploit the list of all possible meanings . In particular, from , we remove all the annotated meanings3 and obtain the set . Then, we randomly add to one of the correct meanings; in this way, contains only one correct sense. For each occurrence of a sense-annotated word in the corpus, we create the prompt in Table 4. Additionally, we store the identifier (i.e. the option’s number) corresponding to the correct answer in a field called output.
Given the word "" in the input sentence, choose the correct meaning from the following: . Generate only the number of the selected option.
Given the word "art" in the input sentence, choose the correct meaning from the following: 1) Photographs or other visual representations in a printed publication 2) A superior skill that you can learn by study and practice and observation 3) The products of human creativity; works of art collectively 4) The creation of beautiful or significant things.
Generate only the number of the selected option. Input: "The art of change-ringing is peculiar to the English, and, like most English peculiarities, unintelligible to the rest of the world." words are not considered in the construction of both tasks4.
Our challenge allows only zero-shot. Table 5 reports the prompt used in Task 1.
Fornisci una breve definizione della parola "" nella frase data in input. Genera solo la definizione. Input: ""
Fornisci una breve definizione della parola "sforzo" nella frase data in input. Genera solo la definizione. Input: "Che sforzo fate per valutare i risultati del vostro programme?"
3In the sense-annotated corpus, a word occurrence can be annotated with more than one correct meaning. 4For Task 1 based on definition generation, it is also possible to consider monosemic words. We exclude this hypothesis since we want to test LLMs in the case of polysemy. without translation with translation
The idea is to measure the correspondence between the generated definition and the correct one provided by the sense inventory in Task 1. For Task 2, we want to measure the accuracy in selecting the correct definition from the set of possibilities. For the above reasons, we use three diferent metrics. For Task 1, we compute F1-RougeL and F1-BERTscore between the reference and generated gloss. For Task 2, we measure the accuracy as the ratio between the correct answers and the number of instances in the dataset.
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of metrics that assess the quality of generated texts, particularly summaries, by comparing them with reference texts. Common variants include ROUGE-N, which measures the correspondence of ngrams, ROUGE-L, which considers the longest common sub-sequences, and ROUGE-W, which considers the weight of correspondences. We select the ROUGE-L to measure the lexical correspondence between the generated definition and the correct one. BERTScore relies on pre-trained language models to assess the semantic similarity between the generated and reference definitions, going beyond mere superficial word matching.
If a unique score for Task 1 is necessary, we propose the harmonic mean between RougeL and BERTscore, giving BERTscore double the weight of RougeL. The idea is to give semantic similarity more importance than word matching.
5 * * 4 * + (1)
We have already performed some evaluations involving several LLMs with a medium number of parameters. Results are reported in Table 4 and show that Llama3.18B-Instruct provides the best performance in gloss generation (Task 1), while Gemma2-9B-Instruct achieves the best accuracy.
Llama3.1 8B-Instruct Mistral 7B-Instruct
Gemma2 9B-Instruct
We cannot guarantee that texts presented in XL-WSD do not occur in the training data of some LLMs. However, even if the model is exposed to textual data from XLWSD, it does not necessarily mean that it was asked to solve the disambiguation task on such data. A fixed sense inventory may not cover all Italian senses, neologisms, or emerging phrases. However, our benchmark considers only words (and their contexts) annotated according to the sense inventory used in XL-WSD. This ensures that all instances in our benchmark have at least one correct sense in the sense inventory.
No ethical issues are reported in our dataset.
Our data are based on the data license of the XL-WSD from which our benchmark is derived. XL-WSD is distributed under a non-commercial license5.
We acknowledge the support of the PNRR project FAIR Future AI Research (PE00000013), Spoke 6 - Symbiotic AI (CUP H97G22000210007) under the NRRP MUR program funded by the NextGenerationEU. [6] T. Pasini, A. Raganato, R. Navigli, XL-WSD: An extra-large and cross-lingual evaluation framework for word sense disambiguation., in: Proc. of AAAI, 2021. [7] R. Navigli, S. P. Ponzetto, BabelNet: Building a very large multilingual semantic network, in: J. Hajič, S. Carberry, S. Clark, J. Nivre (Eds.), Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Uppsala, Sweden, 2010, pp. 216– 225. URL: https://aclanthology.org/P10-1023.