This paper presents ECWCA (Educational CrossWord Clues Answering), a novel challenge designed to evaluate knowledge and reasoning capabilities of large language models through crossword clue-answering. The challenge consists of two tasks: a standard question-answering format where the LLM has to solve crossword clues, and a variation of it, where the model is receives hints about the word lengths of the answers, which is expected to help models with reasoning abilities. To construct the ECWCA dataset, synthetic clues were generated based on entities and facts extracted from Italian Wikipedia. Generated clues were then selected manually in order to ensure high-quality examples with factually correct and unambiguous clues.
CEUR ceur-ws.org A
Crossword puzzles are well-known linguistic games that
are usually used for entertainment, but they are also
applied in education as a tool to assess knowledge, reason- tage of that.
ing skills and linguistic abilities of students [
In this challenge instead, we make use of educational
crossword clues to build a benchmark to assess the LLM
clue-answering skills on popular entities and facts about
the world. We refer to it as ECWCA, standing for
Educational CrossWord Clues Answering. ECWCA is an
Italian benchmark presented at [
In this challenge, we evaluate the knowledge abilities of LLMs by testing them on crossword clue-answering tasks. We propose two slightly diferent tasks in the challenge. The first one, is essentially a Question Answering problem, where the question is a clue and we expect the to build a QA dataset of clue-answer pairs. This happens in two steps, first we generate a set of examples constituted by an answer and the generated clues (as in clue-instruct), then we manually select the most suited clue-answer pairs (see Section 3.2 for further details).
In order to construct the examples with clue-instruct,
(a)
(b)
(c)
(d)
we identified the most visited Italian Wikipedia 1 pages. information, thereby ensuring the integrity of the dataset.
To count visits, we considered a period between
September 10, 2023 and May 31, 2024 and gathered stats from Answerability. Annotators were instructed to
Wikimedia APIs2. We considered the page title as the choose a clue that could be answered without a high
answer. Titles with non-alphabetic characters, with less degree of ambiguity. The focus was on clues that
than two characters or more than 20 were excluded. On provided enough information to infer the correct answer
the remaining pages, we extracted their content. Difer- with confidence. Clues that left room for multiple
ently from clue-instruct, we did not dispose of the cate- interpretations or guesses were rejected. For example,
gory information, therefore we generated it by querying generic definitions, such as ’a large mammal’, does not
GPT-4o [
The clue-instruct method produces three diferent clues for each given answer and its context. To select only one clue we add a human selection step. Doing so, we avoid the presence of multiple occurrences for the same answer. Moreover, we guarantee high quality definitions and answers.
The example selection process was carried out by three native Italian speaking annotators. Examples were split in 18 chunks of 100 examples each, equally distributed among the annotators.
Each example was presented with the answer, the three generated clues and the Wikipedia page paragraph that was used to create the clues. Annotators were tasked with selecting the best one, if any, based on the following criteria: Truthfulness and Accuracy. It was imperative that the content of the selected clue was factually correct. Annotators cross-verified the accuracy of the clue from the provided Wikipedia page content to ensure that it did not contain misleading or false 1https://it.wikipedia.org/ 2wikimedia.org 3https://huggingface.co/datasets/azugarini/crossword-clues-QA In cases where more than one clue satisfied all the criteria, annotators were directed to select the clue that provided the most relevant information with most clarity and simplicity. When no clue matched the criteria, the whole example was discarded.
Each example includes the clue-answer pair, the word length hint, some additional metadata (such as the category and the page views) and the reference to the wikipedia page url, whose content was exploited to generate the clue. More precisely, there are the following columns: clue, answer, answer_len, url, content, views, category, length_hint, raw_entity. A few examples are showcased in Table 1, where for the sake of simplicity, we only report the clue-answer pair, the hint and the category of the example.
We defined two diferent prompts, one with and the other without indications about the words length of the answer. The two prompts are presented in Figure 4 and Figure 3, respectively.
Task without hints. We construct a 2-shot prompt (Figure 3) for the task. First, we instruct the model to act as an expert in solving crossword clues without any additional hints related to the structure of the answer (such as words length). The format is clear and concise, focusing on the core task: resolving the crossword definition and providing only the solution. Then, the two static demonstration examples are showcased to illustrate to the model how to approach the task. Finally, following the same layout, we present a new clue and expect the model to complete it with the answer.
Task with word length hints. This prompt (see Figure 4) is very similar to the first one, but introduces an hint indicating the words length of the expected answer. The hint is a constraint that reduces the number of valid answers, giving indications on both how many words there are and their lengths, therefore, ideally, it should aid the language model.
Sei un esperto di enigmistica. Devi risolvere definizioni di cruciverba.
Trova la risposta alla definizione. Ritorna solo la risposta, nient'altro.
Esempi: DEFINIZIONE: Protagonista di Titanic al fianco di Kate Winslet RISPOSTA: leonardo dicaprio DEFINIZIONE: capitale dell'Impero romano d'Occidente nel 313 d.C.
RISPOSTA: milano
To evaluate the performance on the tasks we rely on the following metrics: Edit Distance (ED), Exact Match (EM), and average F1 score on words (F1).
Edit Distance. Edit Distance (also known as LevenOverall we collected 1,171 clue-answer pairs belonging shtein Distance) measures the minimum number of to 16 diferent categories. The distribution of answers single-character edits (insertions, deletions, or substiamong categories is outlined in Figure 5. Most of the ex- tutions) required to change one sequence into another. amples belong to Entertainment topic, indeed the dataset In this context, ED measures how close the generated includes many actors, tv shows, movies and fictional
Sei un esperto di enigmistica. Devi risolvere definizioni di cruciverba.
Ti verrà data una definizione corredata da un suggerimento, una sequenza di numeri indicante di quanti caratteri è composta ciascuna parola della risposta.
Trova la risposta alla definizione.
Ritorna solo la risposta, nient'altro. response is to the ground truth answer. A lower ED indicates better performance, as it signifies that the predicted text is more similar to the target text. Preliminary Results. We establish baseline results on ECWCA, testing some of the models in the Llama family.
In particular, we consider Llama3 8B and Llama3.1 8B in both instructed and non-instructed versions, and the Llama3.1 70B-instruct, to observe how model size afects the results. Table 2 illustrates the performance of the LLMs on the two tasks (with and without word-length hints), both evaluated on the defined scores. We can observe that Llama3.1 8B consistently outperforms its predecessor across all the metrics, both with and without hints. The gap between smaller LLMs and Llama3.1 70Binstruct is remarkable, proving once again that larger LLMs preserve much more knowledge.
Word-length hints instead are generally not helping the models, actually harming the performance in noninstructed models. For example, the F1 score of Llama3.1 8B drops significantly, from 37.35 without hints to 27.51 with hints, and similarly, EM decreases from 34.16 to 25.72 as well. Instructed models instead are not afected by this, but the suggestions lead to a small increase in all the metrics. Only in Llama3.1 70B-instruct, we can observe some statistically significant improvement. This may suggest that constraints are beneficial only on models with stronger understanding capabilities.
In Figure 6, we show how the performance of Llama3.1 family models vary with respect to the number of page views. We group examples in intervals, then we compute the metrics on each of them. Edit distance shows no significant trends, whereas EM and F1 exhibit an increasing trend on more visited pages for 8B sized models, whereas the 70B model has a behaviour that seems uncorrelated with the number of views. This suggests that the larger number of weights in 70B model, stored a broader and deeper knowledge about world facts and entities, covering also less popular ones, whereas smaller LLMs did embody only the most popular factual knowledge seen during training.
Large Language Models have all been exposed to vast amount of data. The clues proposed in this dataset were created from Wikipedia pages that were definitely seen by the LLMs during training. Clues are also generally very adherent to the pages content, since they were created from it. Indeed, one of the goals of the benchmark is to assess their memorization capabilities on facts that were likely to be well known by them. However, the proposed dataset is new, hence it could not have been part of the training set of such LLMs.
Data is released under apache-2.0 license.