Educational crosswords ofer numerous benefits for students, including increased engagement, improved understanding, critical thinking, and memory retention. Creating high-quality educational crosswords can be challenging, but recent advances in natural language processing and machine learning have made it possible to use language models to generate nice wordplays. The exploitation of cutting-edge language models like GPT3-DaVinci, GPT3-Curie, GPT3-Babbage, GPT3-Ada, and BERT-uncased has led to the development of a comprehensive system for generating and verifying crossword clues. A large dataset of clue-answer pairs was compiled to fine-tune the models in a supervised manner to generate original and challenging clues from a given keyword. On the other hand, for generating crossword clues from a given text, Zero/Few-shot learning techniques were used to extract clues from the input text, adding variety and creativity to the puzzles. We employed the fine-tuned model to generate data and labeled the acceptability of clue-answer parts with human supervision. To ensure quality, we developed a classifier by fine-tuning existing language models on the labeled dataset. Conversely, to assess the quality of clues generated from the given text using zero/few-shot learning, we employed a zero-shot learning approach to check the quality of generated clues. The results of the evaluation have been very promising, demonstrating the efectiveness of the approach in creating high-standard educational crosswords that ofer students engaging and rewarding learning experiences.
1.2k 1.5k 2.0k 2.5k 3.1k 4.2k 2.6k 1.6k
2 # of entries
The art of crafting crossword puzzle clues has been a
puzzle in itself, prompting diverse strategies to tackle
the challenge. Traditional methods often lean on
wellestablished dictionaries, thesauri, or language analysis of
web-retrieved texts to define clues [
Taking another path, [
Venturing into the realm of Spanish language puzzles, [15] explored extracting definitions from news articles to craft crossword puzzles. They employed a two-stage process: first, identifying crucial words and phrases and extracting their meanings from a trustworthy online dictionary, followed by utilizing those definitions as clues to construct engaging crosswords.
In another linguistic context, [16] presented SEEKH, a software application employing natural language processing to extract keywords and craft crossword puzzles in a multitude of Indian languages. Combining statistical and linguistic tools, SEEKH adeptly pinpointed essential keywords, bringing to life a medley of crosswords across linguistic landscapes.
Despite extensive research eforts, efectively producing comprehensive and distinctive sets of clues and answers from linguistic corpora remains a formidable challenge, especially when dealing with the nuanced intricacies of the Italian language. To tackle these challenges head-on, we present an innovative methodology utilizing Language Models (LLMs) to craft sophisticated educational clues. Representing a pioneering endeavor, our approach successfully generates Italian educational crossword puzzles, addressing a void that previous methods have left unattended. By creating intellectually stimulating and original crossword puzzles, this novel technique enriches learners’ profound comprehension of the subjects through detailed and encompassing answers. ThereUnique entries Unique answers
To fine-tune the LLMs, we leveraged a comprehensive collection of Italian crossword clues and answers. The sources of the clues-answer pairs are both internet sites that release solutions for crossword clues as h t t p s : //www.dizy.com/ and https://www.cruciverba.it/ that we scraped through apposite scripts. And also pdf versions of famous Italian crossword papers like Settimana Enigmistica and Repubblica, that we suitably converted to clue-answer pairs. The various sources where than cleaned, merged and the duplicates were removed. We intend to release this dataset with the support of this paper. This dataset consists of 125,600 entries that correspond to unique clue-answer pairs. It included clues related to diferent domains, such as history, geography, literature, and pop culture. The dataset under investigation contains a diverse array of linguistic features, including grammatical structures, syntactic patterns, and lexical elements.
A recurring structural pattern in the dataset is the usage of the phrase “known for" or “used for" to define a particular place or object. For example, the definition of a certain location might be “a place known for its historical significance" or “an object used for a specific purpose." In both cases, the answer is a specific instance of the category described in the definition. Moreover, the dataset includes instances where the definition employs clever wordplay or exploits general category definitions to arrive at a specific answer. For example, “In the middle of the Lake" might elicit the response “AK", while “An exotic legume" could be answered with “SOY" by virtue of its membership in the broader category of legumes. In figure 1 you can further go into detail regarding the distribution of the data divided by the length of the answers. Shorter answers tend to have more clues associated while as the answer gets longer the number of clues diminishes in proportion. One of the primary goals of this study was to establish the groundwork for future research by making the processed dataset publicly accessible, with the aim of encouraging other scholars to contribute to this field." 1
The system extracts clue-answer pairs from provided nesses the power of zero-shot learning for an approach texts (path (a) of Figure 2), or generates clues based on to our task. We meticulously craft two prompts in both given keywords (path (b) of Figure 2). As input texts Italian and English, ensuring they are well-structured we use paragraphs selected from Wikipedia pages on with clear objectives and detailed steps to achieve them. educational topics like science, geography, economics. You can access it in the appendix under the section laUsing this type of text allows us to create direct clues beled Prompts 1 and 4. This thoughtful design empowers like definitions, appropriate for the educational usage. the Language Model (LLM) to precisely extract the most The system evaluates the quality of the generated clue- relevant keywords, capitalizing on its robust zero-shot answer pairs using various validators. Following the learning capabilities. By providing guidance through our generation process, users are granted the opportunity prompts, we optimize the model’s ability to understand to review all the produced clue-answer pairs and select and respond to the intricacies of the task at hand their preferred combinations. These selected pairs are Clue generation: We use a few-shot learning apthen utilized by the final component of the system to proach to create compelling crossword clues for each generate the crossword puzzle schema. identified keyword in the paragraph. By leveraging an
In this segment, we will delve into the system’s fun- example educational text, crossword keywords, and valid damental aspects, encompassing three essential compo- clue examples, we empower the Language Model (LLM) nents: the generation and validation of clue-answer pairs to craft meaningful clues. We presented the paragraph from provided text, the creation of clues based on given and extracted clues as prompts to the LLM, allowing it to keywords, the validation of the result, and lastly, the generate clues based on the provided text and keywords. generation of the crossword puzzle layout or schema. This technique ensures precise and contextually relevant clues. We crafted prompts in both Italian and English, 4.1. Path (a) similar to the previous section. Two distinct types of prompts were developed, and all of them are accessible In this section, we analyze the path (a) of Figure 2. We in the Appendix under Prompts 2 and 5. used a multi-step process to apply zero-shot and few-shot Validation: We improved the quality of generated learning techniques to text. First, we divided the text keywords and clues by implementing a multi-stage filterinto paragraphs and extracted precise keywords. Then, ing process. First, we filtered out long keywords (over 3 we created personalized clues inspired by the original words) as they were less suitable for crossword puzzle text using those keywords. To ensure high quality, we answers. Some generated clues inaccurately described thoroughly validated the generated clue-answer pairs. their corresponding keywords, and some were halluciOur primary tool was the GPT-3 DaVinci base model nations from the provided text. To address this, we used [17]. We’ll explore each step in detail in the following. zero-shot learning to identify and filter out unwanted 1The dataset is available at https://huggingface.co/datasets/Ka clues, resulting in a significant improvement in the final myar-zeinalipour/ITA_CW output. We created Italian and English prompts, akin to the previous section. Both prompt types can be found in the Appendix under Prompts 3 and 6. textually relevant crossword clues remained, thereby elevating the overall accuracy and usability of our system. 4.2. Path (b)
where users lack access to the original text and wish Our algorithm for creating educational crosswords takes to generate crossword clues solely from given answers, input such as answer lists, work area dimensions, and we devised an approach to cater to this scenario. Our stopping criteria. It starts by randomly placing a central strategy encompassed multiple stages, each contributing answer, then adds other answers nearby. The algorithm to the overall efectiveness of the solution. iteratively adds answers, sometimes removing recent
Initially, we focused on fine-tuning various language ones or restarting. The best solution is selected based on models specifically tailored for this unique task. Lever- a global score of the generated schemes. Each solution aging the data generated from these fine-tuned models, produced is evaluated using the following formula: we then proceeded to create diverse classifiers. These classifiers were carefully designed with the primary ob- Score = (FW + 0.5 · LL) · FR · LR jective of distinguishing high-quality clue-answer pairs from those that were deemed less suitable. where FW (Filled Words) is the number of words
Fine-tuned models: In the pursuit of generating added; LL ( Linked Letters) is the number of letters that crossword clues from given answers, we undertook vari- belong to two crossing words; FR (Filled Ratio) is the ous fine-tuning processes of language models, using data number of total letters divided by the minimum rectangle collected from Section 3. Our selection of models com- area used; and LR (Linked Letters Ratio) is the Linked prised GPT3-DaVinci (175B parameters) and GPT3-Curie Letters (LL) divided by the number of total letters. (13B parameters). The algorithm incorporates various stopping criteria,
GPT3-DaVinci, with its vast parameter count, demon- including the minimum number of answers added to the strated unmatched depth, enabling it to uncover intricate grid; reaching the threshold of minimum Filled Ratio; patterns and craft nuanced clues. On the other hand, the limit on the number of times the grid is rebuilt from GPT3-Curie, while slightly smaller, proved remarkable scratch, and the maximum time duration. The solution in grasping language subtleties, further enhancing the with the highest score is deemed the best. These stopping ifne-tuning process [17]. criteria play a crucial role in guiding the algorithm’s
In our fine-tuning process, we employ a distinctive ap- decision-making process, determining when to conclude proach by inputting the answer and tasking the model to the crossword construction. Through the establishment generate the corresponding crossword clue. This iterative of thresholds and limitations, we successfully ensure the method not only refines the model’s ability to compre- eficient and efective generation of crosswords. hend context but also hones its skill in crafting clues that Within the filling process, we have the option to desare both challenging and contextually fitting. By continu- ignate a list of "preferred answers." The algorithm places ally providing the answer as input during fine-tuning, we a higher priority on selecting answers from this list, inguide the model toward a nuanced understanding of how creasing the probability of their incorporation into the to construct clues that align seamlessly with the given grid. solution. This tailored training methodology further enhances the model’s proficiency in delivering accurate 5. Experiments and engaging crossword clues, solidifying its role as a versatile and efective tool in the clue-generation process. The experimental evaluation of the designed system is
Validation: We developed diferent strong classifiers presented in this section, focusing on the individual comusing fine-tuned language models to distinguish good ponents and their roles in the overall framework. The crossword clues from poorly crafted ones since not all system’s performance is thoroughly analyzed to assess generated clues fit the given answers perfectly. its efectiveness and eficiency, providing insights into
In pursuit of this goal, we fine-tuned several models, its strengths and weaknesses. each boasting unique capacities: GPT3-DaVinci (175B parameters), GPT3-Curie (13B parameters), GPT3-Babbage (1.3B parameters), GPT3-Ada (350M parameters) [17], 5.1. Experimental Evaluation: Path (a) and BERT-uncased-base (110M parameters) [18]. In our experiments, we observed variations in model out
By harnessing the collective power of these models, put quality when altering the language of the prompts. To each with varying parameter counts, we gained a compre- demonstrate this, we conducted two sets of experiments hensive perspective on their efectiveness in filtering and using two types of prompts: one in English and the other validating the generated clues. Through this approach, in Italian. Our system underwent a rigorous evaluation our goal was to ensure that only high-quality and con- process using 50 paragraphs sourced from Wikipedia to assess the performance of each component using Ital- of acceptable clues compared to Curie’s 34.9% ian and English. Human supervision was employed, and guidelines for evaluation can be found in Appendix 6. Table 2 The results of these evaluations are summarized in Table. Assessment outcomes of the clues generated from the provided
Initially, our focus was on keyword extraction, and we keyword. achieved promising results in our experiments. Specifically, employing the zero-shot learning approach, we obtained 79.73% and 75.60% accuracy in generating suit- Model % of acceptable clues able keywords for crossword clues using Italian and En- GPT3-DaVinci 60.1 glish prompts, respectively. Subsequently, we subjected GPT3-Curie 34.9 the clue-generation process to human evaluation and found that, with Italian and English prompts, 68.34% and 76.70% of the generated clues were considered acceptable, respectively. To ensure the validity of our results, To gain deeper insights into the quality of the genwe employed various approaches outlined in Section 4.1. erated clues, we meticulously assembled a collection of Through this validation, we were able to identify 56.76% acceptable and unacceptable clues. These were randomly and 69.72% of the unacceptable clue-answer pairs gener- sampled from the human-supervised label dataset, ofated using the Italian and English prompts, respectively. fering a diverse clue for each answer. Please consult These results clearly demonstrate the efectiveness of our Table 3 (refer to table 5 in the Appendix for translation). system in producing satisfactory crossword clues based This detailed analysis helps us evaluate the quality and on the evaluated text. suitability of the clues for creating engaging crossword
Figure 3 demonstrates the step-by-step process of gen- puzzles. erating crossword clue-answer pairs from input text. The We developed multiple classifiers that integrate diferimage shows the various stages, such as keyword extrac- ent language models to diferentiate between acceptable tion, clue creation, and pair validation, and illustrates and unacceptable clue-answer pairs. The result of the how our system converts input text into pertinent cross- analysis on the test set is shown in Table 4. We utilized word clues. The results with the Italian data revealed a dataset of 6,000 human evaluations from the previous that, when the prompt is in English, the performance of step to construct various classifiers. This is the data the model is better than when the prompt is in Italian. which we tried to evaluate GPT-3-Davinci and GPT-3Curie by human supervision. For training and evaluation, we employed 80% of this data for fine-tuning the clas5.2. Experimental Evaluation: Path (b) sifiers and reserving the remaining 20% for testing the classifiers. Within the dataset, 51% comprised acceptable clues, while the remaining 49% consisted of unacceptable clues.
The evaluation results reveal significant distinctions among the classifiers in their ability to diferentiate between acceptable and unacceptable clue-answer pairs.
Earning the top position, the GPT3-DaVinci model achieved an accuracy of 79.88%, solidifying its role as the most efective classifier in this task. Following closely, the GPT3-Curie base model attained a commendable 77.82% accuracy. The GPT3-Babbage model demonstrated respectable performance with 74.12% accuracy, while GPT3Ada and BERT-uncased achieved accuracies of 69.17% and 65.62%, respectively.
generating and validating clues from keywords. Building upon the insights presented in Section 4.2, we devised and fine-tuned two distinct models GPT3-DaVinci and GPT3-Curie with a specific focus on creating clues based on given keywords. For the training phase, we selected a subset of the dataset introduced in Section 3, encompassing 50000 unique clue-answer pairs.
Once the fine-tuning phase concluded, we generated 4,000 clues from each of the fine-tuned models and subjected them to human evaluation using the guidelines provided in Appendix 6. The outcomes of this evaluation are summarized in Table 2. Remarkably, GPT-3 DaVinci outperformed GPT-3 Curie, yielding an impressive 60.1%
Our schema generation algorithm creates educational crosswords with diverse layouts using a single batch of words. Below is an illustration, check the Figure 4 of a comprehensive Italian educational crossword about movies produced with our system. The clue-answer pairs are both extracted from a text (path (a), see Figure 3) and generated directly from a keyword (path (b), contrassigned with a ⋆ below).
the introduction of a substantial dataset for Italian clueanswer pairs, we developed an innovative system using Large Language Models to generate educational crossword puzzles from given texts or answers. Our approach combines human supervision and specific guidelines to ensure high-quality and relevant clues.
Our system includes a keyword extraction component (79.73% high-quality keywords) and a crossword clue generation component (76.6% relevant and acceptable 2 R E C E N S I O N E
O U N T A I O
9M O V I M E N T O Orizzontali 1 ⋆ Traduzione Simultanea 2 ⋆ Una valutazione critica 4 ⋆ Lo `e uno spassoso racconto 7 Insieme delle arti, tecniche e attivit`a industriali produttive di un film 8 Lingua originale da cui deriva la parola cinema 9 Uno dei termini estratti dall’antica lingua greca, utilizzato per descrivere il cinema Verticali 1 ⋆ Un film... come documento 3 Prodotto commerciale finale di un insieme di lavoro compredente arti, tecniche e attivit`a industriali 5 ⋆ Un premio assai ambito 6 ⋆ Entrano nelle casse del botteghino clues). A validation component filters out unacceptable pairs, achieving a 69.72% detection rate. We conducted an in-depth investigation of fine-tuned generators and classifiers to enhance the quality of clues. Among the models tested, GPT3-Davinci demonstrated exceptional performance in generating clues based on given keywords, producing a remarkable 60.1% of acceptable clues. Moreover, GPT3-Davinci proved to be the most proficient classifier, accurately distinguishing between good clueanswer pairs and unacceptable ones with an impressive 79.88% accuracy.
Our algorithm for generating educational crossword schemes is eficient and produces diverse layouts. This study aims to enhance student skills and promote interactive learning. Educators can integrate our system into their instruction for more efective teaching practices.
Future research involves developing advanced models for direct clue-answer pair generation and exploring specialized models for diferent clue types. Our vision is to revolutionize educational crossword generation and unlock new innovations in teaching practice.
[15] J. Esteche, R. Romero, L. Chiruzzo, A. Rosá, Au- Relevance and Cohesion: A top-notch crossword tomatic definition extraction and crossword gen- clue-answer pair thrives on a profound and meaningeration from spanish news text, CLEI Electronic ful connection between the clue and the answer. The Journal 20 (2017). clue should provide ample context or clever hints that [16] B. Arora, N. Kumar, Automatic keyword extraction smoothly lead solvers to the intended solution. Simuland crossword generation tool for indian languages: taneously, the answer must be directly tied to the clue, Seekh, in: 2019 IEEE Tenth International Confer- fitting flawlessly within the puzzle’s theme or topic. ence on Technology for Education (T4E), IEEE, 2019, Wordplay and Inventiveness: Elevate your crosspp. 272–273. word clues with ingenuity and wordplay that challenge [17] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Ka- and delight solvers. Seek clues that encourage lateral plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- thinking, incorporate witty twists, or conceal intriguing try, A. Askell, et al., Language models are few-shot meanings. A well-crafted clue-answer pair captures the learners, Advances in neural information process- solver’s imagination, transforming the puzzle into an ing systems 33 (2020) 1877–1901. exhilarating journey of discovery. [18] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, Clarity and Precision: Precision is key in creating M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the crossword clues. Ensure your clues are crystal clear and limits of transfer learning with a unified text-to- unambiguous, presenting solvers with a distinct and pretext transformer, The Journal of Machine Learning cise solution. Avoid any ambiguity that might lead to Research 21 (2020) 5485–5551. multiple interpretations or numerous possible answers.
The goal is to deliver a single correct solution that aligns perfectly with the clue’s intended meaning.
Grammar and Language: Pay meticulous attention Appendix to grammar, syntax, and linguistic conventions in both the clue and the answer. Maintain grammatical correctGuidelines for Validating Clue-Answer ness, coherence, and an appropriate level of complexity Pairs for a crossword puzzle.
General Knowledge and Fairness: Strike a balance between challenge and accessibility by grounding your clues in general knowledge or commonly known facts.
Avoid overly obscure or specialized references that could alienate solvers. A great clue-answer pair caters to a diverse range of puzzle enthusiasts, ofering a fair and engaging experience for all.
Through the adoption of this framework, a robust dataset can be generated, facilitating the development of a dependable classifier that discerns commendable crossword clue-answer pairs from incongruous or inappropriate ones. This transformative classifier holds the promise of revolutionizing crossword puzzle creation, assessment, and solving, ofering invaluable revelations into the craft of constructing captivating and mentally stimulating puzzles.
In the course of our study, we embraced an enthralling challenge: constructing a classifier capable of discerning between acceptable and non-acceptable crossword clueanswer pairs. Crossword puzzles have held a cherished place as a beloved pastime, demanding a harmonious fusion of linguistic prowess, creative acumen, and adherence to intricate puzzle construction rules to fashion top-tier clue-answer pairs. Our pursuit of creating an automatic evaluator for generated crossword clues and their corresponding answers holds tremendous potential.
This advancement promises to aid puzzle creators, enrich puzzle-solving experiences, and unlock profound insights into the subtle nuances of language and puzzle design.
Ultimately, this endeavor not only elevates the world of crossword puzzles but also kindles a deeper appreciation for their linguistic artistry and cognitive allure.
To create a powerful classifier for crossword clueanswer pairs, we must establish a strong and comprehensive guideline that clearly delineates the attributes of acceptable and non-acceptable pairs. This guideline will be the cornerstone for training our classifier, enabling it to discern the defining characteristics that set apart high-quality clues from irrelevant or inappropriate ones.
With strict adherence to this guideline, we can guarantee the accuracy of our classifier in assessing the quality of clue-answer pairs, ultimately leading to the creation of more captivating and enjoyable crossword puzzles.
Let us now explore the pivotal components of the guideline, essential for evaluating crossword clue-answer pairs:
Prompt 1: Italian, for keyword extraction prompt = f""" Obiettivo: Il tuo compito é estrarre delle parole chiave, descritte nel testo proposto. Le parole chiave estratte saranno utilizzate per creare brevi definizioni di cruciverba riguardanti il testo da cui sono estratte le parole chiave. Le definizioni saranno d'aiuto per trovare la soluzione corrispondente e completare il cruciverba.
Completa l'obiettivo attraverso i seguenti passaggi:
Clue-Answer pair Mythology: It is known by anyone who knows myths Electricity: One of the zodiac signs
Curiosity: The desire to know Collaboration: One reaches it with anyone Model DV DV Curie Curie
Acc.
Yes No Yes No 1- Estrai le parole chiave piú importanti del testo. 2- Controlla le parole chiave: controlla se le parole chiave sono descritte e definite nel testo o non sono descritte e definite nel testo. 3- Parole chiave finali : sulla base del passaggio precedente, rimuovi tutte le parole chiave che non sono definite nel testo.
Utilizza il seguente formato di output: Parole chiave: <Parole chiave finali>
Prompt 2: Italian, for clue generation prompt = f""" Genera brevi definizioni di cruciverba per ciascuna delle parole chiave fornite: {keywords} sulla base del seguente testo: {text}. 2- Check keywords: check if the Italian keywords are well Explained in the given Italian text or not. 3- Final keywords : Remove all the Italian keywords which are not well defined in the Italian text based on the last step.
Use the following output format:
Prompt 5: English, for clue generation
Generate short crossword definitions in Italian for each provided Italian keyword: {keywords} based on the following Italian text: {text}. Follow these steps to achieve the objective: 1- For each provided Italian keyword detect the part of the Italian text which contains the keyword information. 2- Generate short definitions in Italian: For all the Italian keywords generate short definitions in Italian based on the Italian text, and place the correspondent keyword after each generated definition. Make sure that the Italian keyword is not present in the correspondent definition. 3- Do not use quotation marks and apostrophes in the output.
Follow this example to complete the task: "Text: La scienza é un sistema di conoscenze ottenute attraverso unattivit di ricerca prevalentemente organizzata con procedimenti metodici e rigorosi, coniugando la sperimentazione con ragionamenti logici condotti a partire da un insieme di assiomi, tipici delle discipline formali. Uno dei primi esempi del loro utilizzo lo si puó trovare negli Elementi di Euclide, mentre il metodo sperimentale, tipico della scienza moderna, venne introdotto da Galileo Galilei, e prevede di controllare continuamente che le osservazioni sperimentali siano coerenti con le ipotesi e i ragionamenti svolti.
Keywords: conoscenze, ricerca, rigorosi, assiomi, ipotesi, Galileo Clues: Conoscenze: informazioni acquisite tramite ricerca organizzata con procedimenti metodici e rigorosi.
Ricerca: attivit organizzata prevalentemente con procedimenti metodici e rigorosi finalizzata allottenimento di conoscenze.
Rigorosi: esatti e precisi nello svolgimento delle azioni.
Assiomi: un insieme di verit accettate come base dei ragionamenti logici.
Ipotesi: assunte per comprendere le osservazioni sperimentali e testare le conoscenze Galileo : egli introdusse il metodo sperimentale nel processo di scienza moderna. " """
Prompt 6: English, to auto check Objective: Your objective is to check whether each given Italian Sentence content is present in the provided Italian text or not. Print "True " if it is present in the provided Italian text and "False" if it is not present in the provided Italian text. Text: ```{text}``` """