intrinsically subjective [10], as points of view might differ depending on users' social background, beliefs, and demographics. Using a single aggregated label has thus been increasingly questioned [11,12,13], and preserving disaggregated data is preferred. On the other hand, recent work has shown that design choices and biases affect datasets and models and often result in models unexpectedly aligned with a given population segment more than with another [14]; in fact, aggregated data tend to reflect a minority of perspectives, under-representing others [15,4].

As a result, disaggregated datasets have become more popular, as listed in the Perspectivist Data Manifesto 1 and by Plank [2] 2 .

Researchers are incresingly reporting annotators' demographics and other metadata when describing the dataset, which was first advised as a good practice to avoid excluding, minimizing, and misrepresenting certain groups of users [16]. Recent work has also explored whether annotators' demographics and background -as described by available metadata -influence their annotation [5,17,18,19,4] and can help during the modeling of the phenomenon under study [20,8,21].

Despite the increasing interest in disaggregated and metadata-rich datasets, few such datasets for irony detection exist. Simpson et al. [22] released a corpus for humor detection in English, used as a benchmark in the first edition of the Learning With Disagreement (LeWiDi) shared task [23]. No annotators' metadata, however, are included. Frenda et al. [4] proposed a dataset for irony detection and investigated the influence of the annotators' demographics on their perception [6]. The dataset contains English texts only.

For this challenge at CALAMITA [24], we propose to use the Italian portion of MultiPICo (Multilingual Perspectivist Irony Corpus)3 [25]. Multipico is a multilingual corpus of short Post-Reply conversational pairs extracted from Twitter and Reddit and annotated as ironic or not ironic by crowdsourcing workers with different demographics and backgrounds. MultiPICo covers 9 languages (Arabic, English, Dutch, French, German, Hindi, Italian, Portuguese, and Spanish) and 25 language varieties 4 , ranging from high-to low-resourced ones. Moreover, a rich set of annotators' sociodemographic information (balanced gender, age, nationality, ethnicity, student, and employment status) is provided.

While no perspectivist task leveraging the dataset has been proposed so far, PERSEID is related to the Learning With Disagreement task held at SemEval 2021 [11] and 2023 [13]. In LeWiDi, participant systems were challenged to learn the distribution of labels, tested by cross entropy-based metrics. In contrast, PERSEID aims at stimulating the development of models of human perspectives, in order to explain the label distributions rather than just quantifying them.

The task of Perspectivist Irony Detection aims to measure models' capability to detect irony in a short verbal exchange for each annotator, conditioned on the knowledge of demographic information about them. To this purpose, we want to look at different model performances if it is informed by one demographic trait or a combination of two. In particular, we focus on the gender and age of the annotator, due to the balanced number of male and female annotators by design 3.2, and due to the fact that age was shown to be one of the most polarized dimensions in [25].

The input to the task does not consist only of a text, but rather of a tuple <perspective, post, reply>.

In this iteration of PERSEID, we considered several variables for the perspective attribute:

• None (Task 0): acting as a baseline, we want to investigate the models' outputs when no information about the annotator is provided. • Age (Task 1): the perspective is one of four values encoding the age group of the annotator.

• Gender (Task 2): the perspective is the binary self-identified gender of the annotator. • Age + Gender (Task 3): in this case, both attributes are provided as the perspective.

The post is a textual post, to which the target reply is a reply. The output of the prediction is a binary label indicating whether the reply is ironic (or non-ironic) for a human bearing the characteristic of the perspective to the text. The performance of the model is evaluated through a global f1 metric on the disaggregated annotations.

The challenge is zero-shot: no training, fine-tuning, or in-context learning is considered for this version of PERSEID and the whole dataset can be used for inference.

Note that since each annotator can be described by no traits (Task 0), one single trait (Task 1 and Task 2), and two traits (Task 3), we do not aim at optimal performance when considering personalized irony detection; instead, our goal is to understand whether models improve their performance when one or multiple traits is provided and to understand the impact of different configurations.

The data for the challenge are part of MultiPICo [25], a corpus of 18, 778 short conversations collected from Reddit (8, 956) and Twitter (9, 822) in 9 languages, and a total of 25 varieties.

Data were collected to reproduce the structure of short conversations.

For both Reddit and Twitter, the post is typically a message initiating a thread and the reply a direct reply to that message 5 .

Reddit data were retrieved using the Pushshift repository 6 from January 2020 to June 2021. For Italian, data were downloaded from the subreddit /r/Italy.

Pairs having at least one deleted or removed comment were filtered out, and the language of the messages was further validated using the Python library for language identification LangID7 .

Twitter data were collected via Twitter Stream API, using the geolocation service and excluding quotes and retweets. Then, the full conversation was retrieved, and tweets that directly replied to the starting ones were retained.

The data collection resulted in 18, 778 instances, together with their metadata, consisting of Post-Reply original IDs, subreddits, and geolocation information. For Italian, data account for 1000 post, reply pairs, equally sourced from Reddit and Twitter.

Annotators were asked to read a set of post and reply pairs and answer whether the text of the reply was ironic or not, given the context.

The human annotation of the collected data was performed on the crowdsourcing platform Prolific 8 , through a custom-built annotation interface designed to collect a diverse and balanced set of annotators. The interface mimicked a message conversation, having the post as context and asking whether the reply was Ironic or Not ironic.

For Italian, 24 native-speaker annotators were hired, who performed 4,790 annotations in total, resulting in a mean of 4,79 annotations per instance (see Table 1). 8 https://www.prolific.com/ Annotators were selected based on three criteria:

• Their completion rate had to be greater or equal to 99% • They had to be native speakers of the considered language (i.e., Italian, for the portion of data used in the challenges) • The set of annotators needed to be balanced across genders.

The quality of the annotation was further assured using attention check questions in the form of "Please answer X to this question". Annotators had 1% probability of receiving these special questions. Annotators who failed to respond correctly to at least 50% of these questions were excluded from the final corpus.

A rich set of metadata is also provided. These include the self-identified Gender (balanced by design), their nationality, their Age Group (

Sociodemographic information about annotators per language.

Italian), Student status (14 yes, 9 no, for Italian), Employment status (9 in full-time jobs, 7 unemployed, 5 working part-time, 1 not in paid work and 1 due to start, for Italian), as reported in Table 2.

The dataset is in tabular format, one row per annotation.

The data contain the text in the form of two fields (post and reply), the binary label, and a series of metadata about the post, reply, and annotator. Here is an example of instance from the Italian section of MultiPICo:

The challenge is zero-shot, and the prompt depends on three variables: perspective, post, and reply. Task 0 No perspective is provided, and the prompt directly starts with the instruction.

The perspective variable is a verbalization of the Generation, which is expressed as an integer in the dataset. It can be instantiated with the following values9 :

• "una persona giovane della generazione Z" if Generation == GenZ (Age < 26) • "una persona giovane della generazione Y" if Generation == GenY (26 ≤ Age < 42) • "una persona adulta della generazione X" if Generation == GenX (42 ≤ Age < 58) • "una persona adulta della generazione baby boomer" if Generation == Boomer (Age > 58)

Task 2 The perspective variable is a verbalization of the Gender variable, which is expressed as a string in English. It can be instantiated with one of two values:

• "una donna" if Gender == "Female" • "un uomo" if Gender == "Male" Task 3 The perspective variable is a verbalization of both the Age and Gender variables, e.g., "una giovane donna della generazione Z. "

Inspired by Mokhberian et al. [26], the Perspectivist Irony Detection task is evaluated by means of global F1, that is, the F1-score computed across all the individual annotations in the dataset against the predictions of the model.

Data The sociodemographic information about the annotators is partial, bound to what was available from the crowdsourcing platform, and following a discretization of human personal traits that could be perceived as forced (e.g., representing selfidentified gender as a single binary label). Furthermore, as shown by Orlikowski et al. [21], annotators' sociodemographics do not always align with the most relevant grouping of annotators according to the language phenomenon under study.

Annotators of the Italian portion of MultiPICO tend to be young (with no annotators from the baby boomer generation and only one from GenX). This aspect might influence the results.

Similarly to Sachdeva et al. [5], Sap et al. [19], Forbes et al. [27], we noticed the ethnicity of annotators is unbalanced, and all but one annotators are white for the considered data.

In the vast majority (∼90%) of cases, the conversation-starting messages and their direct replies were downloaded to capture the full conversational context. In a few cases, the downloaded reply was not direct but rather a secondlevel reply (a reply to a direct reply); thus, some conversational context might be missing.

We describe annotators by no sociodemographic traits (Task 0), one single demographic trait (Task 1 and Task 2), or two demographic traits (Task 3). We evaluate disaggregated annotations at inference time, having the annotators represented only by those traits. Annotators' sociodemographic information does not always align with the most relevant grouping of annotators according to the language phenomenon under study [21,28], and the limited amount of sociodemographic traits we provide is undoubtedly not enough to describe every single annotator. We are aware of this limitation. In fact, our main aim is to understand whether providing one or more annotator traits makes the model predictions more aligned with annotators having a given characteristic.

This work places itself in an increasing amount of work that calls to consider and include the subjectivity of the annotators in NLP applications, encouraging reflection on the different perspectives encoded in annotated datasets to minimize the amplification of biases. We hope this challenge will be a starting point for investigating and evaluating LLMs in Italian to make them suitable for final users. The dataset used in the challenge was built by adopting measures to protect the privacy of annotators, and the data handling protocols were designed to safeguard personal information (like anonymization of users' mentions). Although the attention during the collection of data was focused on ironic content spread online, we acknowledge that some of the material contains racist, sexist, stereotypical, violent, or generally disturbing content.

Annotators are balanced through their self-identified gender. However, we are aware that considering gender in a binary form is limited; moreover, a substantial unbalance for some dimensions, like the self-identified ethnicities, is present in the dataset. This pattern suggests the need to interact differently with annotators or social communities if we want a diversity of annotators and perspectives in terms of social background.

MultiPICo is distributed under the Creative Commons Attribution 4.0 (CC-BY-4.0) license.