A riddle from the game show “Wheel of Fortune” consists of a hidden sentence that can be discovered starting from a simple clue and by iteratively guessing its letters. Although the game is very popular and intuitive, solving one of these riddles is not trivial. In fact, for interpreting the clue, identifying the most probable letters, and leveraging the game's mechanics efectively, a player requires linguistic abilities, world knowledge, and even some form of strategic thinking. The goal of this study is to verify whether Large Language Models (LLMs) possess the necessary abilities to solve Wheel of Fortune riddles. We propose a software framework called LLMike in which an algorithmic Game Master interacts with an LLM: prompting it, enforcing the game's rules, updating the hidden sentence based on the model's guesses, and evaluating their correctness. We study several models with diferent sizes, evaluating their performance, behavioural patterns, and common types of errors. Our dataset and code are available at https://github.com/ejdisgjinika/LLMike.
does not need particular reasoning capabilities, such as
for choosing which words to complete first: LLMs may
Assessing linguistic and reasoning abilities of Large Lan- start wherever they want and complete the puzzle with
guage Models (LLMs) is an open challenge [
Especially in the last few years, LLMs have proved to ad- With non-textual games, such as Connect-4 or
Ticdress many Natural Language Processing tasks (such as Tac-Toe [
A popular research line is based on games [
Category: Around the house approach combined with a local search to choose possible
word candidates and rank them for completing crossword
puzzles. This game covers diferent aspects, such as
common sense, general knowledge, and metalinguistic
patterns. Another work on crossword puzzles with human
evaluation has also been proposed in [
L L words. Most of the proposed systems are based on techL niques that leverage the similarity between the vector representations of words.
With the growing popularity of LLMs, rather than
creating ad-hoc models to play and complete games,
reCategory: Around the house searchers have begun using these games to benchmark
the general abilities of LLMs [21, 22]. Qiao et al. [20]
A V A S E F U L L introduce the concept of evaluating LLMs using
conversaO F F L O W E R S tional games, such as a round-based interaction between
a questioner and an answerer called Ask-Guess. One
of the main claims of this study is that conversational
Figure 1: Example of the gameplay of the Wheel of Fortune games can diferentiate the capabilities of diferent LLMs.
game. At the top, we show how the game starts, i.e., with a Manna et al. [
Another interesting work is [23], which focuses on edge is also needed to solve many of the clues based on role-playing abilities of LLMs combined with external places, movies, etc. Finally, choosing which consonants tools. Similarly, the authors of [19] evaluated the abilito say, whether to buy a vowel, or when to try to guess ties of several LLMs in a multi-agent scenario to solve the sentence also needs some basic strategic skills. a detective-style game. Although linguistic and world
In this paper, we create LLMike, an algorithmic frame- knowledge are needed, their evaluation focuses more on
work that allows LLMs to play Wheel of Fortune games. the strategies the agents use to play the game.
The name comes from the TV presenter of the first edi- More generally, the knowledge possessed by LLMs
tions of the Italian version of Wheel of Fortune, Mike has been the subject of many studies [24], focusing on
Bongiorno. LLMike prompts the LLMs with all the pro- world knowledge [
Games and puzzles are a recurrent testbed for assessing show that lets multiple contestants compete with each
the capabilities of deep learning systems, especially to other to win the game and earn money. The goal is to
corimplement complex reasoning abilities [
Game Master
SPIN or BUY VOWEL ?
LLM
SPIN + CONSONANT
1,2
LLM BUY VOWEL +
VOWEL 1,3,4
Game Master Add the reward to the budget Game Ma_ster Subtract 250$ from the budget PASS
Game Master Show to the player the hidden riddle and make the LLM choose
LLM GUESS or PASS 1
GUESS a guess. The game works in several rounds. In the begin- The Game Master gives the prompt, which contains the ning, it is shown the word puzzle (with no letters present, rules, the goals, and an example of the game, and asks as at the top of Figure 1) which can reveal a sentence, the LLM to select an action, starting a round. The LLM a name of a person, a place, etc. Each participant has a selects an action and its budget is updated. Next, the budget that starts at 0 $ and can gradually grow over the Game Master shows the new conditions of the game, i.e. rounds. Starting from the first participant, he/she can the hidden riddle partially revealed and the new budget. spin a wheel composed of several wedges, with diferent Finally, it asks the LLM to provide a guess or pass to the amounts of money associated with each wedge. Next, next round. the participant chooses a consonant: if the consonant is We redesign the game by adapting the rules to a singlerevealed in the hidden riddle (as in the middle of Figure participant scenario with a slightly diefrent round struc1), the participant earns the amount of cash indicated ture, as shown in Figure 2. First, we removed the speby the wedge times the number of occurrences of the cial wedges from the wheel (i.e., “Bankrupt” and “Lose a consonant chosen. Next, he/she can spin the wheel again turn”), because they depend only on luck, and this can and continue to play another round. If the consonant is lead to a non-systematic analysis of the LLM’s abilities. not present in the riddle, the participant passes the turn Therefore, our wheel has only cash wedges, all between to another player. As the rounds progress and the player 100 $ and 1.000 $. has enough money, he/she can buy a vowel for a fixed In our interaction schema, first the Game Master asks amount of the budget and has to indicate which vowel the LLM to spin the wheel or to buy a vowel for 250 $. he chooses. If the vowel is present in the riddle, it will After the choice made by the LLM, the riddle and the be revealed, but if it is not, the player passes the turn. budget are adjusted accordingly and subsequently comAt any time in his/her game, the player can guess the municated to the LLM. Then, the LLM has the option to riddle by giving their final solution. If the correct answer give a guess or to pass and start another round. Since is given, the player wins the budget he earned. However, we have only one LLM playing, a key diference is that if the answer is wrong, the player passes the turn. in our adaptation of the game, if the LLM gives a letter
In the original game show, some special wedges of that is not present in the riddle, it does not lose the turn the wheel are also present: “Bankrupt”, which resets the in favour of another player, but only its budget is set to player’s budget and passes the turn; and “Lose a turn”, 0 $. The goal we give to the LLM is to complete the game which makes the player skip his/her turn. and to maximize the amount of money earned by solving the riddles. These goals are in line with the goals a real 3.2. LLMike: Evaluating LLM’s Abilities at player playing the Wheel of Fortune would have.
We also formalize some rules specifically for the LLMs’
Wheel of Fortune interaction with the game, intending to control and better In the adaptation we created for evaluating LLMs’ abil- understand the ability of the models to follow instrucities at solving Wheel of Fortune riddles, we defined tions. This formalizations results in four rules: two main roles: the Game Master, which is a specifically coded algorithm (not based on artificial intelligence tools) • Rule 1: The LLM cannot choose to do an action that interacts with the LLM and evaluates its answers, that is not possible in a given situation; for inand the LLM, which acts as a player of the game. stance, the LLM can’t pass the turn when it is An overview of our adaptation is presented in Figure 2. required to spin the wheel or buy a vowel. • Rule 2: If the LLM spins the wheel, it has to
choose a consonant and not a vowel. • Rule 3: If the LLM buys a vowel, it has to choose
a vowel and not a consonant. • Rule 4: The LLM has to buy a vowel if and only
if it has enough money to do so.
lose the game.
In Figure 2 also shows a brief version of the prompt used during the games. The prompt contains a short description of the context, followed by the instructions for playing the game, the goals, and an example. The goals are expressed in simple sentences, and the examples represent a standard conversation between an LLM and the Game Master. The complete prompt is available in the GitHub repository.1
Please note that the riddle cannot be solved by simply choosing all the letters in it, one at a time. In fact, all riddles are composed of consonants and vowels. However, the player can choose only consonants, which leads him/her to always deal with an incomplete riddle. This leads to two major possible decisions: buying vowels or guessing the sentence, which cannot be easily implemented in simple baseline approaches.
taken from a publicly available dataset4 and repurposed. The riddles are of variable length and divided into 16 categories. The shortest sentence is made up of 2 words while the longest is made up of 9 words. In terms of the number of characters, the range is from 9 to 47 characters. The average lengths are 19.47 and 3.16 in terms of characters and words, respectively.
conducted, the models and data we used, how the performance was evaluated, and the results. Then, we present an analysis of the main errors made by the models and provide some intuition on their strategy.
the performance of LLMs in our Wheel of Fortune task.
First, we consider the number of games won (# Wins) and the average amount of money won by the LLM (Total Final Budget). Other metrics are more complex and are based on the game rules listed in Section 3.2. First, we Models and implementation details. We selected consider a group of metrics to evaluate the model be29 open-source models available through Ollama2, which haviour, such as the number of letters chosen by the LLM are available in Table 1. Ollama is a framework designed (# Letters), the percentage of the letters that were actually to facilitate the local execution of open-source LLMs. found in the riddle (% Correct Letters), and the percentThe models considered difer considerably in terms of age of completion of the riddle when the LLM gives the architecture, family, and number of parameters. right guess (% Riddle Completion). Next, we consider sev
Moreover, we select three commercial models: GPT- eral error-related metrics, to understand when the model 4.1, Mistral Large 2 and Gemini 2.0 Flash3. The exact size does not follow the rules (perhaps, by not selecting a of GPT-4.1 and Gemini 2.0 Flash has not been disclosed letter, or by trying to buy a vowel with an insuficient publicly. However, they are much bigger than any of the budget), when it just provides a wrong guess or when it open-source models we considered. Mistral Large 2 has reaches the maximum number of possible consonants. about 123 parameters.
For both open source and commercial models, the responses are generated using the default parameters.
2https://ollama.com/ 3Specifically, we use the "mistral-large-2411", "gemini-2.0-flash-001", and "gpt-4.1-2025-04-14" snapshots.
the Wheel of Fortune game. Of the more than 30 mod
wheel-of-fortune-answers els tested, only 9 managed to guess at least one solution: age 85.21 completion to solve a total of only 8 games. three commercial models and six open-source LLMs, four This may suggest a higher understanding and knowledge of which belong to the Gemma family. Except for Gemma possessed by Gemma 3 27, with respect to Phi 4. A sim2 9, all models have more than 10 parameters. Fur- ilar comparison can be made with Gemma 3 12, which thermore, all models with more than 25 parameters obtains only 5 wins with a riddle completion of 86.46. can guess at least one correct solution, with the exception In this case, the diference seems entirely dependent on of Aya Expanse and Command-R. the diferent number of parameters.
In Table 2, we show the results ordered by the number Significantly better results are obtained with commerof games won. The best open-source model, by far, is cial LLMs: GPT-4.1 gets 62 wins, Gemini 2.0 Flash 35, Gemma 3 27 with 20 wins in 80 games, followed by and Mistral Large 2 25. Nevertheless, these models have Gemma 2 27 and Phi 4 14 with 8 wins, and Gemma 3 similar performance with respect to the open-source mod12 with 5. Although they reached one and two victories, els in terms of number of letters (all between 10.53 and respectively, we did not include in Table 2 Gemma 2 13.23), percentage of correct letters (which does not ex9 and Cogito 32 due to the low significance of their ceed 68%), and percentage of riddle completion. This results with such a small sample. behaviour suggests that although these larger models
However, these victories can come from two diferent possess a similar ability in guessing the correct letters abilities. The first is that a model may guess as many and completing the masked riddle, they are much better letters as possible and progressively fill in the riddle, at providing the correct solution. until the guess becomes very simple. The second is that a Table 2 also reports the final budget earned by the model may not need to fill the riddle as much as possible, models. The best performing model is GPT-4.1, with more because it has enough knowledge to find the correct than 65 $. Notably, Gemma 3 27 obtains a higher solution of a more complicated riddle. Analysing the amount of money (20.6) with respect to Mistral Large ability of the model of choosing letters, the best open 2 (15.25), despite obtaining fewer wins (20 versus 25). source model is Gemma 2 27, with 68.7% of correct Since every time a model chooses a wrong consonant, letters. This ability is reflected in the number of letters the budget is set to 0, this is probably due to its higher required to provide a correct solution, which is 8.38, the percentage of correct letters (62.73 versus 54.97). lowest of all models. The other LLMs perform worse, ranging from 51.19 (Gemma 3 12) to 62.73 (Gemma 4.2. Typical Errors 3 27). All the other open-source models tend to select a higher number of letters, ranging from 11.00 to 16.8. In this section, we discuss the most common errors made Interestingly, the former has the tendency to select as by the models considered. Since, an important first result many letters as possible, filling the riddle up to 86.46%, of our experiments is that 23 LLMs over a total of 32 on average. were unable to give a single correct solution, we first
Analysing the guessing capabilities, Gemma 3 27 analyse their main flaws. obtains 20 victories not only by selecting letters, but also In Figure 3, we show six types of errors made by those by guessing from a quite low completion of the riddle LLMs considered and their frequency calculated for all (71.30), whereas the least performing models require a 80 games. The most common error (in blue) is definitely higher completion. Instead, Phi 4 14 requires an aver- Insuficient Budget (33.1%), in which an LLM tries to Overviews of the Error Made by the Best Performing Models In the following, we investigate the flaws Figure 3: Error frequency for the LLMs unable to guess a made by the best performing models, i.e. those reported single riddle. Each colour represents a diferent error category. in Table 2. Starting from GPT-4.1, the major cause its The frequency of each error, in the form of a percentage over losses is the Wrong Guess (55.56%): i.e. the model, at all the 80 games for each LLM, is reported inside each sector. a certain riddle completion, has enough “confidence” to try to guess the riddle but provides the wrong answer.
Despite GPT-4.1 being the best model at following the inbuy a vowel without the necessary money. The next structions, it still shows some limitations on letter chooserror, Action Not Allowed (N/A), is quite more complex. ing (11.11% of Vowel N/A and 5.56% of Consonant N/A) As we show in Figure 2, the model is forced to generate and managing the budget (11.11% of Insuficient Budget specific text such as [SPIN], [BUY VOWEL] or a sin- Error). Gemini 2.0 Flash shows a diferent behaviour in gle consonant at diferent times during the game. This terms of errors. In fact, it manifests lots of problems on text indicates the choice of executing a specific action in instruction adherence and budget management (respeca strict way and any other answer is considered as an tively 40% of Instruction Error and 33.3% on Insuficient Action N/A error. This error recurs 20.2% of the time. Budget Error). Interestingly, Mistral Large 2 is good at Similarly, Consonant N/A (19.4%) refers to those times following instructions, managing its budget and choosthat the model, after choosing to buy a vowel, selects a ing the letters in the right contexts. However, it provides consonant instead. Both Action N/A and Consonant N/A many wrong answers (Wrong Guess 87.27%). An indenote a lack of understanding of the game rules and of teresting fact is that Mistral Large 2 and Gemma 3 27 the prompt instructions provided by the Game Master. obtain a comparable number of wins (respectively 25 Wrong Guess (14.0%) happens when the model simply and 20 wins) even if they have a significantly diferent provides a wrong solution to the riddle. In our analysis, number of parameters (123 and 27 respectively). Alan important aspect of this type of error is that often the though Gemma 3 27 has a lower percentage of Wrong LLM does not respect the format of the riddle, selecting Guess (56.7%), its limitations in dealing with single letwords with the wrong number of letters. Moreover, some ters (Vowel N/A 20% and Consonant N/A 5%) and budmodels (such as Olmo 2 and Llama 3.2) can be considered get management (10%) deteriorates its performance. “overconfident”, choosing to guess the solution with a very limited amount of letters. As Vowel N/A (12.0%), 4.3. Hints on Strategy we refer to those times the model, instead of choosing a consonant, selects a vowel instead. As for Action N/A and Consonant N/A, this error depends on not understanding the game rules. Finally, the remaining 1.3% of the errors occur when the model exceeds the round limit imposed (20 rounds), continuously spinning the wheel or buying vowels without trying to guess the solution of the riddle.
the strategy followed by the best performing models.
We think that a total absence of strategy would result in picking random consonants. Instead, a smarter approach would be to select consonants which appear frequently in English words. To highlight this behaviour, we analyse the first letters chosen by the model. Results are available in Table 3, in which we report:
Consonant generated more frequently by the models. In fact, considering Gemma 3 27 these consonants are the 46.41% of all the letters chosen by the model. Similarly, for GPT-4.1 they are 43.67%. Although this difers from the standard frequency in the English language (into which these five Std. Freq. Gemma 3 GPT-4.1 consonants reach a total of 34.2%), we can say that both models know which are the most common consonants T 9.1 10.40 10.08 and exploit this information in their games, combining NS 66..37 190.4.490 190.6.599 both linguistic knowledge and basic strategy. Both modH 6.1 4.29 3.49 els have a very similar behaviour, with T, N, S and R being R 6.0 11.83 9.82 the preferred consonants (with a frequency around 10%), and H is considered less important, with a frequency Total 34.2 46.41 43.67 that does not exceed 4%. This is quite diferent from the statistics calculated for the English language, in which has a frequency of 6.1, quite similar to R (6.0), and • the number of diferent pairs of letters chosen by S (6.3). This is probably due to the fact that H is very the LLM at the start of the game (# Pairs); present in very common stop words such as the, which, • the number of diferent triplets of letters chosen this, which may not be particularly important to solve by the LLM at the start of the game (# Triplets); our riddles. More specifically, models tend to start with • the number of # Vowels the model decided to buy; the two most frequent consonants (T, N or S) and then buy a vowel (mostly E or A). This behaviour is constant for most of the 80 riddles of our dataset, regardless of the sentence length or other characteristics.
We can see that there are notable diferences among the models with respect to the number of distinct pairs and triples chosen at the start of diferent games. Phi 4 14 has the highest variability, selecting 35 diferent pairs and 61 diferent triples of letters across the 80 riddles in 5. Conclusions and Future Work our dataset. Instead, the best performing models (such as GPT-4.1, Gemini 2.0 Flash and Gemma 3 27) present In this paper, we proposed a novel textual game based a much lower variability, with respectively 9, 10 and 11 on the famous “Wheel of Fortune” game show with the diferent pairs and less than 30 diferent triples. This sug- aim of assessing linguistic and reasoning abilities. We gests that they start many riddles with a similar strategy. created a framework for allowing LLMs to play under
Analysing the number of vowels bought by our mod- strict rules and showed how the task was structured, the els, we can see some other relevant information. The data, and the metrics used for the evaluations. We analmodels with highest variability in terms of letters chosen ysed 29 open source models and 3 commercial models (Phi 4 14 and Gemma 3 12) also tend to buy more to evaluate a variety of models with diferent model’s vowels (respectively, 4.00 and 3.80 on average). Com- architecture and sizes. Only 9 LLMs out of 32 managed paring these results with those in Table 2, we can see to solve at least one riddle. The most problematic aspects that this strategy does not provide notable advantages: in are their little ability to follow the instructions, such as fact, they win only 8 and 5 games respectively. Instead, the constraint of choosing only consonants. The best the best performing models (the commercial models and performing open-source model is Gemma 3 27, with Gemma 3 27) tend to buy fewer vowels (only 2.30 for 20 wins out of 80 riddles, whereas the commercial model Gemma 3 27 and 2.63 for GPT-4.1) obtaining a defi- GPT-4.1 solves 65 riddles. Analysing their strategy, we nitely higher number of wins. Moreover, since buying see that the best performing models select the most frevowels requires subtracting 250 $ from the budget, this quent consonants in the English language, resulting in a decision can be considered good also for the declared progressively easier riddle. However, they can also guess goal of maximizing the earnings. the right solution with a completion of around 70%.
In Table 4 we compare the standard frequency of the As future work, we want to analyse performance of ifrst five consonants in the English language 5 (Std. Fre- Large Reasoning Models (LRM), such as Deepseek-R1, o3 quency) with the percentage of times that such conso- and o4-mini, and to expand the framework to let several nants are chosen by two LLMs: the best performing open models play with each other. Moreover, another intersource one, Gemma 3 27, and the best commercial one, esting direction would be to exploit Multimodal LLMs to GPT-4.1. We can see that the most frequent consonants create a visual version of the game. We would also like (which in English are T, N, S, H, and R) are definitely those to consider data in other languages. Finally, we would like to implement new games and analyse the behaviour of models in a more complex environment.
was enrolled in the Italian National Doctorate on Artificial Intelligence run by Sapienza University of Rome in collaboration with the University of Brescia.
This work has been partly funded by Regione Lombardia through the initiative "Programma degli interventi per la ripresa economica: sviluppo di nuovi accordi di collaborazione con le università per la ricerca, l’innovazione e il trasferimento tecnologico" - DGR n. XI/4445/2021.
Declaration on Generative AI During the preparation of this work, the author(s) used ChatGPT (OpenAI) and DeepL Write / DeepL Translate in order to: Improve writing style and Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.