Language games were adopted as testbeds for measuring NLP progress in recent years [2,3,4], with a particular focus on (cryptic) crossword solving English [5,6,7,8,9]. For the Italian language, initial efforts focused on crossword solving and generation [10,11] and clue-based word guessing [12,13,9]. Recently, Sarti et al. [1] introduced an extensive collection of text-adapted Italian rebus puzzles to evaluate large language models' (LLMs) knowledge and sequential reasoning abilities. Rebuses are complex puzzles combining visual elements and graphic signs to encode a hidden phrase. Italian can boast a rich and long-standing rebus tradition dating back to the 19th century [14], popularized by high-diffusion magazines such as La Settimana Enigmistica 1 . The structure of Italian rebuses has, with time, been formalized into beauty canons [15], and their peculiarities and design principles were analyzed by several authors [16,17,18].

In Italian rebuses, rebus solving begins by combining derived by combining graphemes with their underlying visual elements in a left-to-right fashion, composing a first pass (prima lettura) representing an intermediate solution of the puzzle. Then, first pass elements are re- This work proposes to adopt the EurekaRebus introduced by Sarti et al. [1] to extend their evaluation of LLMs' multi-step reasoning and linguistic/cultural awareness to the systems evaluated as part of the CALAMITA evaluation campaign [19]. We believe the task is particularly relevant since the crossword definitions that compose verbalized rebuses rely heavily on idiomatic expressions, wordplay, and cultural references specific to Italian. Hence, the results of this task could provide valuable insights into the linguistic and cultural competence of LLMs trained on the Italian language. Moreover, the task is especially appealing since it is framed in a templated reasoning format, enabling us to disentangle the various components required to successfully solve a verbalized rebus step-by-step. More specifically, several metrics will be employed to assess LLMs' factual recall, textual concatenation and re-segmentation capabilities and, finally, constraint satisfaction given the provided cues.

In light of the results reported by [1] for state-of-theart proprietary LLMs, we expect all tested open-source systems to perform very poorly, with final solution accuracies well below 30%. We also note that the highest reported overall performance in previous work 2 was found by the original authors to be primarily the product of memorization. We anticipate that this challenge will highlight significant limitations in LLMs' current factual recall and multi-step reasoning ability and act as a catalyst for future improvements in these areas.

The proposed challenge aims to evaluate the capabilities of existing LLMs in solving verbalized Italian rebuses via prompting at various granularity levels. More specifically, LLMs will be evaluated in a few-shot prompting setting with two fixed in-context learning examples pre-selected at random from the available pool of verbalized rebuses in EurekaRebus, in two settings:

• Regular, matching the example in table 1 and the original input format used by Sarti et al. [1]. • Hints, in which the number of characters for every hidden word is provided alongside definitions in the verbalized rebus to help the model in identifying the correct choice. This variant was not tested by Sarti et al. [1].

Refer to section 3.3 for the respective example formats. Models will be evaluated on their performance at each step required to successfully solve the verbalized rebus and their overall ability to produce correct final solutions. 2 Namely 58% Solution Exact Match for a LLaMA-3.1 8B model LoRAtuned on 80k EurekaRebus examples [20,21]

The dataset used for this challenge is an extended version of EurekaRebus [1], a collection of 222,089 unique Italian rebuses extracted from Eureka5 platform 3 , an open database of rebuses and other linguistic puzzles maintained by the Associazione Culturale "Biblioteca Enigmistica Italiana -G. Panini"4 . Among these, 83,157 were converted by the original authors in verbalized form by leveraging the crossword definitions from the ItaCW collection [10], including 125,202 definition-solution pairs. While Sarti et al. [1] evaluated the performances of prompted and tuned LLMs on rebuses up to June 17th, 2024, the current test set include 168 new unseen examples released on Eureka5 after that date.

We employ the same procedure of Sarti et al. [1] for verbalizing available rebuses. More specifically, only rebuses having all lowercased or camel-cased words among ItaCW solutions are selected, and every word is replaced by sampling one of the available crossword definitions for it at random. 5 Moreover, only regular rebuses containing at least two hidden words are selected, avoiding examples requiring a single definition-solving step and those with more complex templates (e.g., anarebuses using anagrams of hidden words for the solution).

Each example in the dataset consists of:

• The verbalized rebus (verbalized_rebus) containing letters from the original rebus and crossword-style definitions enclosed in square brackets. • The whitespace-separated solution words obtained after resegmenting the first pass according to the solution key, provided in a semicolon-separated string in order of occurrence (solution_words). • The solution of the verbalized rebus used as the final prediction target for the LLM (solution).

An example is provided in Listing 1.

Table 1 shows the 2-shot prompting template adopted for generating a templated solution with the tested LLMs.

The second in-context example used in the template, omitted for brevity, corresponds to the one shown in Listing 1.

The task description provided to the model was derived from a trial-and-error process starting from the original prompt by Sarti et al. [1]. Notably, compared to the original authors the task description provides more detailed descriptions of individual components of the rebus to provide a clearer overview of the task to the LLM. We opted for a 2-shot setting as opposed to the 5-shot prompting employed by Sarti et al. [1] to accommodate the limited context length of some of the tested LLMs, thus ensuring that the total length after model generation does not exceed 1024 tokens 6 . The two examples provided remain the same shown here to simplify evaluation and ensure consistent results.

Verbalized rebus solving steps Table 1 provide labels for the steps necessary to solve the verbalized rebus that are considered in this challenge task. The model receives a problem input including a verbalized rebus (possibly with length hints) and a solution key (chiave di lettura). The first step involves resolving crossword definitions in order (Definition resolution), exploiting only the model's parametric knowledge to accomplish 6 The LLaMA 3 tokenizer was used to perform this estimate the task. Then, the resolved words need to be infilled into the original rebus to compose the first pass, and re-segmented in the Solution segmentation step. Finally, the individual solution words are reassembled into a single solution string. While prompted models should obtain similar performances across all test subsets, the aformentioned division will enable further comparisons with previously trained systems.

The challenge employs a comprehensive set of metrics adapted from the original evaluation of [1]:

Il tuo compito è quello di identificare le parole nascoste e sostituirle agli indizi nel Rebus, producendo una prima lettura dalla quale poi si deriverà una frase risolutiva. La chiave di lettura è una sequenza di numeri che rappresentano la rispettive lunghezze delle parole che compongono la frase risolutiva. La tua risposta deve essere una frase risolutiva sensata e che rispetti le lunghezze definite nella chiave di lettura.

Chiave di lettura: 11 2 10 Procediamo alla risoluzione del rebus passo per passo: ... (same format as the first example) The Solution Match metric will be used as a primary metric of correctness, since it captures the model ability to fully solve the verbalized rebus. While no baseline evaluation was conducted for the new test set used in this challenge, we expect the performances of most capable open-source systems to align with those of 5shot prompted LLaMA-3 70B and Qwen-2 72B models reported by Sarti et al. [

Baseline results for LLaMA-3 70B and Qwen-2 72B for the original test set, adapted from Sarti et al. [1].

to complete the task primarily due to incorrect word guesses, with errors propagating across resolution steps and ultimately resulting in a final accuracy of 0%.

Several limitations should be considered when interpreting the results of this challenge:

Verbalization Simplification The use of verbalized rebuses, while necessary for text-based LLMs, simplifies the original visual puzzle. This does not fully capture the complexity of solving traditional rebuses, which rely on visual cues and cultural knowledge, making verbalized rebus solving a much simpler proxy to the multi-step reasoning required for regular rebuses.

The selected rebuses and crossword definitions rely heavily on Italian-specific linguistic and cultural background. Performance on this task may not generalize to other languages or puzzle types, and it might be unrealistic to expect general-purpose LLMs to possess the specific lexicon and knowledge used for rebus solving.

Prompt Sensitivity While the selected prompt template was observed to perform well for capable proprietary LLMs in preliminary tests, there are no guarantees that the instructions provided in the prompt are sufficient for smaller open-source models to perform verbalized rebus solving proficiently. Moreover, alternative prompt formulations could lead to potentially better results.

The challenge currently lacks a clear human performance baseline, which would be valuable for contextualizing model performance on verbalized rebus solving.

While this challenge focuses on a relatively benign task of puzzle-solving, there are some ethical considerations to keep in mind. First, the dataset captures a very narrow subset of Italian language and culture. Hence, evaluation findings should not be overgeneralized to Italian language competence as a whole or to other cultures. This dataset's rebuses and crossword definitions are derived from commercially available published sources. While efforts have been made to ensure this data's exclusive, fair usage for research purposes, there may be copyright considerations to address.

As reported by the original EurekaRebus dataset license, the data is redistributed for research purposes only with the explicit approval of the Associazione Culturale "Biblioteca Enigmistica Italiana -G. Panini" (here onwards referred to as the Association), and the rights to each entry in the EurekaRebus collection are the property of the respective copyright holders. The usage and redistribution of these data is allowed only for users providing appropriate attribution to the original copyright holders and the Association, and the creation of derivative works is permitted only for research purposes, using terms no less restrictive than the EurekaRebus license. Researchers are encouraged to contact the challenge organizers with any questions or concerns about data usage and licensing.

• The Associazione Culturale "Biblioteca Enigmistica Italiana -G. Panini" for making their rebus collection freely accessible on Eureka5. • The creators of the ItaCW dataset for enabling the creation of verbalized rebuses. • The puzzle creators whose work is represented in this dataset.

Gabriele Sarti and Arianna Bisazza acknowledge the support of the Dutch Research Council (NWO) for the project InDeep (NWA.1292.19.399). Arianna Bisazza is further supported by the NWO Talent Programme (VI.Vidi.221C.009). We hope this challenge will contribute to the diffusion of the art of Italian enigmistica among computational linguistics and artificial intelligence researchers.