The ability to convey both specific information (about individuals or events) and generalisations (about categories) with the same lexical item is one of the key feature of natural languages. Consider the examples in 1:

1.

a) the lion escaped yesterday from the zoo. b) the lion is a predatory cat.

The noun phrase (NP) the lion can describe either a specific individual (1a) or the entire category of large African felines (1b), thus it expresses a variable degree of inclusiveness of the possible number of individuals to which the NP correctly applies in each sentence it occurs. This demonstrates how human language follows a principle of economy, enabling a one-to-many mapping between lexical labels and meanings.

The syntactic form of the NP (definite, indefinite, or plural) does not provide sufficient information to discriminate between the two meanings, and we need to enlarge our focus to take into account the whole context in which the NP occurs [1]. This phenomenon can be observed in all languages [2], affecting nearly all nouns that can be used in referring expressions. Indeed, natural languages do not have explicit markers for generic NPs [3]; the genericity/specificity of an NP is derived from the meaning of the entire sentence. In other words, we cannot interpret language one word at a time; we need to consider the whole sentence or utterance as context to disambiguate and decipher the meaning of each single word composing it, and thus to understand the message conveyed through language.

Generalizations about kinds and categories, as in 1b, are called generics and are fundamental to human cognition, because they allow us to conceptualize properties linked to categories, shaping how we perceive the world [4].

Moreover, distinguishing between generic and nongeneric meanings for abstract entities is less straightforward than for concrete ones, and for this reason evaluate the inclusiveness of an abstract noun or a NP is even more challenging. Indeed, inclusiveness is not an exclusive feature of concrete-only entities. Consider the examples in 1: The concept behind the word idea is always referring to an abstract entity, with slightly different grades of abstractness, but it shows a greater variation in terms of inclusiveness. The noun ideas in 2a includes only a restricted number of elements with respect to the universe of the ideas (namely, only colorless green ones), while the reference in 2b shows a higher level of inclusiveness, not distinguishing among them on the basis of their color.

The ability to distinguish, interpret and use correctly the variability that natural language offers along these two graduated semantic features, abstractness and inclusiveness, is of paramount importance if we want to make talking machines which not only simulate language, but can also reason about natural language and the knowledge of the world it depicts.

The CALAMITA special event [5] offers the possibility to challenge Large Language Models on their ability to understand the abstractness and inclusiveness of the words, and compare with humans their behaviour in judging Italian sentences. With this report we present the ABRICOT Task: ABstRactness and Inclusiveness in COntexT.

The ABRICOT Task aims to challenge Italian language models on their understanding of abstractness and inclusiveness, features that we, as humans, naturally express in everyday language. These features are not discrete binary dichotomies like abstract/concrete or inclusive/exclusive; instead, they shade on a continuous spectrum, with the two extremes at opposite ends. The collection of sentences in this Task shows the same NP in a variety of different contexts, so that its meaning can oscillate between the extremes of both the axis of abstractness and inclusiveness.

We ask the participant models to express a judgment on a 5 point Likert scale for both the features of inclusiveness and abstractness of the target noun or NP in each sentence.

This task have some similarities with the CONcreTEXT Task1 [6], which has been presented at the 2020 edition of EVALITA. 2 Both tasks focus on the abstractness/concreteness of target words in natural Italian sentences, asking judgments by means of Likert scales, but the ABRICOT Task goes beyond by including also the inclusiveness feature of the targets. Moreover, for the construction of this dataset we considered exclusively nouns or NPs as targets, and in order to limit to the minimum the impact of the variability deriving from different semantic role or syntactic function, all the sentences have been selected with the target noun as subject of the main verb.

We propose two separate tasks for this benchmark, Task 1: abstractness and Task 2: inclusiveness the two tasks are formally identical, we use the same metric and the same samples, however they measure two different scores, respectively abstractness_mean and inclusiveness_mean, the first meant to measure the abstractness of the word in context and the second its inclusiveness.

Since both these concepts are evident but fuzzy also for humans, we don´t expect language models to have a perfect understanding of them and we will limit our metrics to regression ones. Despite the tasks being very similar from a formal perspective, we show how models' performance on these two tasks varies and there is sensible difference between the results in the two tasks.

The 20 target NPs of the dataset for the ABRICOT Task are derived (and translated in Italian) from the set of target nouns in the Situation Entities Corpus (SitEnt [7]), a collection of English sentences in which specificity and genericity have been annotated with a binary labelling scheme (i.e., GENERIC vs. NON-GENERIC). Using those as seeds, representative Italian sentences have been manually harvested from OpenSubtitles3 and Wiki-How. 4 These are widely used sources, the first contains the openly available subtitles of an extensive collection of movies and TV series, while the second is a website gathering articles on how-to do a variety of different things.

More specifically, the sentences have been extracted from the Italian section of the multilingual The Human Instruction Dataset [8], a structured collection of Wiki-How instructions pages, and from the Italian sub-corpus of the OpenSubtitles2018 corpus [9].

Our protocol proposes to the annotators groups of sentences (from a minimum of 4 to a maximum of 8), all containing the same noun, each to be evaluated using a continuous slider, from which values ranging from 0 to 1 will then be extracted.

After the annotation, the reliability of our data has been computed using the Intraclass Correlation Coefficient (ICC(k)). Human ratings have been then averaged, and the resulting figures will be used as gold standard.

An example of the samples present in the dataset can be seen in Figure ?? where examples with the NPs margherita (lilly), ambizione (ambition) and benzina (gasoline) are reported. In particular, Figure ?? and ?? show two examples containing the same token but in different contexts and report the effect of the context on the abstractness and inclusiveness of the token.

The data is stored on OSF [10].5

The data is proposed in a tabular format, with 12 columns:

• ID: a unique identifier for the sample;

• target token: the focus of the dataset, to be assinged an abstraction score in context;

• target lemma: the lemma of the target token;

• text: the sentence where the token appears;

• begin: the index of the first character of the token in the sentence;

Abstractness Prompt: Assegna un valore di astrazione da 1 a 5 alla parola parola nel contesto della frase seguente: frase Descrizione dei valori: 1 -La parola è estremamente concreta (e.g. un cane specifico) 2 -La parola è lievemente concreta (e.g. un cane di una certa razza) 3 -La parola è neutra (e.g. un cane tra tanti) 4 -La prola è lievemente astratta (e.g. un cane è un animale da compagnia) 5 -La parola è estremamente astratta (e.g. il cane è un mammifero). • end: the index of the last character of the token in the sentence;

• domain: the source where the token come from;

• inclusiveness mean: the average inclusiveness score assigned by the annotators;

• inclusiveness std: the standard deviation of the inclusiveness scores;

• abstractness mean: the average abstractness score assigned by the annotators;

• abstractness std: the standard deviatio n of the abstractness scores;

We use different prompts for the two tasks, they are shown in Figure 2, we ask the model to directly output a score from 1 to 5 specific to the task, we then propose an explanation for each point from 1 to 5, explaining the (approximate) meaning of assigning that score together with a very high-level example and on top of the explanation, we use 3-shot evaluation, we found 0-shot to be difficult Mean and standard deviation of the abstractness and inclusiveness for each token across all different possible contexts. for this dataset as without some reference example, the scoring becomes too variable. With a 3-shot approach and the prompts we used, all models we test appear to be able to understand the task and performance improves with these prompts when compared to less specific ones.

The dataset contains 127 samples each sample focused on a token, the same token appears more than once in the dataset, on average 6.35 times, in different contexts.

While the dataset contains 127 samples (a limited amount), Figure 3 shows that both abstractness and inclusiveness are well spread across the dataset and there are samples for all values between 0 and 1. Interestingly, while the two concept under study are different, the two scores are similarly distributed across the dataset, but there is a higher number of samples with abstract-mistral 7b llama-3.1-8b llama-3.1-70b abstractness 0.22 0.30 0.53 inclusiveness 0.00 0.30 0.41

Pearson correlation between the model predicsions and the human annotations for abstractness and inclusiveness scores, measure for three different models, mistral 7b, llama-3.1-8b and llama-3.1-70b.

ness value around 0.8 while for inclusiveness the peak is around 0.1, showing a partial anti-pattern between the two scores, and the concept they are meant to distill.

To investigate the relevance of the context in the assessment of abstraction and inclusiveness, Table 1 shows the mean and standard deviation of the abstractness and inclusiveness of a token when varying context, for all the tokens in the dataset. The standard deviation is often between 0.2 and 0.4 for a score bound between 0 and 1, this shows significant sensitivity to context and highlights how, even if tokens are repeated, each sample is valuable on its own and provides different insights about the token.

We measure Pearson correlation between the abstractness and inclusiveness scores predicted by the model and the gold human annotation. More specifically, since it is challenging to have the models output a continuous value for the abstractness or inclusiveness of a token in context, we have them generate a discrete score from 1 to 5.

The evaluation is done following a likelihood based approach, after prompting the model to answer our question, we pick the highest likelihood token among 1, 2, 3, 4 and 5 and pick that as the model selection. After doing so for each sample, we compute the Pearson correlation between these values and a discretized version of the continuous scores (discretization does not affect the results) assigned by humans to the same samples.

Table 2 shows our evaluation of three powerful, Emglish-first language models, mistral 7b [11], llama-3.1-8b and llama-3.1-70b [12], note that we use the instruct version of all three models, and we omit it from the names.

These initial results show that the models are able to capture both abstractness and inclusiveness, with the exception of mistral 7b that fails at understanding inclusiveness (Pearson correlation is 0). At the same time, a powerful LLM like llama-3.1-70b is not able to capture the full complexity of the task, with a Pearson correlation that is as low as 0.53 for abstractness and 0.41 for inclusiveness. This shows that while not alien to the concept of abstractness and inclusiveness, the models are still far from fully understanding it.

Assessing abstractness seems to be easier for LLMs, since every model performs better in this task than in the inclusiveness one. This is interesting although hard to interpret. One possible explanation is that abstractness is a feature that is already made explicit by the choice of the stimuli. Those words do show a variation between different contexts of use, and this is one of the objectives of such challenges with contextual information, but we can also organize these nouns, out of context, discretely along the axis of variation between abstract (e.g. ambizioneambition) and concrete (e.g. benzina -petrol). On the contrary, inclusiveness cannot be resolved in any way without considering a proper context; a word form by itself does not convey any information about how much generic, thus inclusive, is the concept behind that lexical label. In light of this, we can hypothesize that when a model has to deal with abstractness/concreteness, it may not be able to rank two occurrences of the same word in slightly different contexts, but for sure it can judge as more concrete or more abstract all the occurrences of one target word with respect to those of another. But when it comes to inclusiveness, thus evaluate if one occurrence is more specific or generic than another, the model is probably struggling more.

Another possible interpretation of these unbalanced results between abstractness and inclusiveness may depend on the quantity of information about the two features: while on abstractness/concreteness there are many studies available online (on English and Italian, as well as on other languages), inclusiveness (and also genericity/specificity, which are the most used terms in literature to refer to this semantic feature) is an understudied topic. We can thus hypothesize that knowledge about abstractness is more formalised in training data, while inclusiveness is not.

Moreover, we confirm that also for this task larger models perform better, Llama 3.1-70b outperforms llama-3.1-8b by a large margin, and that training on more data provides stronger models also in this case, indeed, llama 3.1 outperforms mistral 7b also by a large margin.

Finally, we remark that we avoid testing models that have been tuned for Italian to let participants to the Challenge measure the performance improvements provided by Italian focused training.

We propose the ABRICOT benchmark, a dataset composed of 127 humanly annotated samples to measure the abstraction and concreteness of words. Each sample is annotated by 5 -7 raters who ranked them with a continuous score from 0 to 1 from most concrete to most abstract and a second one measured in the same way from least to most inclusive.

We propose two Tasks, measuring abstractness and inclusiveness and we test three powerful language models on our benchmark, mistral 7b, llama 3 8b and llama 3 70b, we show that when correlating their generations with the humans scores, the highest result on abstractness is 0.53 achieved by the largest llama 3 while on inclusiveness the correlation is bound by 0.41, showing that inclusiveness is harder to understand than abstractness.

We hope that the ABRICOT benchmark will foster the development of new language models in Italian as well as new benchmarks investigating phenomena with a theoretical linguistic foundation such as abstractness and inclusiveness.

The main limitation of the datasets is the low number of samples it contains, in particular since samples can repeat tokens and there are indeed only 20 unique ones. This can limit the validity of the models assessment, since the topics and vocabulary we cover is rather limited, although we have shown that in terms of both abstractness and inclusiveness, the dataset is well spread and provides a good coverage of both concepts.