We designed Kwiri, a query system, as a support tool for an AI Game Design Assistant. With our tool we allow users to query for game events in terms of what (was the event), when (did it happen), where (did it happen), and who (was involved). With such a tool, we explore the possibilities of applying a query system to provide game-general AI-based design assistance. Throughout this paper, we will discuss the motivation, the design of Kwiri, use cases, and a preliminary qualitative study. Our first results shows that Kwiri has the potential to help designers in game debugging tasks, and has been served as infrastructure to build another system which relies on querying for game events.
AI-assisted design systems promise to help humans
perform design tasks faster, better and/or more creatively.
Within the field of game design there have been several
prototype systems that showcase how artificial intelligence
can be used in a specific role to assist a human game
designer
Most existing prototypes focus on a single type of
AIassisted design for content (typically levels) in a single
game. In contrast, Cicero is a tool that aims to be a
general AI-assisted game design tool, that is, providing
multiple types of AI-based design assistance not just for a
single game, but for any game that can be represented in the
language used by the tool
On top of Cicero, we designed Kwiri. A query system motivated by the idea that a designer (or tester) of a game will often want to figure out when and where something happens, and this might not be evident when either playing a game or watching a replay. For example, imagine that a particular NPC (Non-Player Character) occasionally dies even when one of the player’s bullets do not hit it. To find out what’s going on, a designer would need to rewatch endless replays attentively. However, what if you could simply ask the game when and where an NPC died? Kwiri makes use of the fact that the games that Cicero work have formally defined game mechanics, and provides the ability to interrogate replays for particular combinations of events.
Kwiri contribution relies on its generality. It is a single query system, whose interface allows designers to ask questions about any genre of game they can design with the platform. It also makes use of gameplay simulations and does not depend on human users to generate in-data game to be analyzed.
This section discusses the primary references and inspirations related to the development of our system.
Tanagra
Similar to Tanagra, Ropossum
Sentient Sketchbook
Also in the field of level generation, we have the work
of Smith et al. (Smith et al. 2012) and Butler et al.
All of these works presented significant results in the realm of AI-assisted game design tools. However, they are attached to a single game or a game genre in the best case. They have a lack of generality because their techniques need to be reimplemented every time someone starts a new game project. Their main focus is not querying for game events, but the way they provide assistance based on AI, even limited to one game, is an inspiration for our system’s design.
Game visualization is a topic which is gaining more
attention every day
One of the games of the Dead Space franchise uses the
tool Data Cracker to collect, analyze and summarize data
about player performance in a visual way
The game Cure Runners has a visualization system used
to track the player and assist designers on level balance
tasks. This work is a case study in the integration of an
analytics tool into a commercial game
GPlay
As well as the AI-assisted tools, most of these works are directly related to a single game. The visualization package of Unity Analytics is a more general approach, but it does not have agents to play the games and generate gameplay simulations.
Kwiri takes influence from these systems and applies visualization as a way to enhance the answers to the designer’s question (What, When, Where, and Who).
Declarative game engines go beyond the common idea of
having databases only as a data storage method. The work
of White et al
Deutch et al. (Deutch et al. 2012) developed a framework, based on SQL, to perform data sourcing in games. It extends SQL commands to allow recursive rule invocations and probabilistic operations. It has a runtime monitor that watches the game execution and give notifications to the user of properties violations. These properties formally specify expected behaviors of the game.
A more traditional use of databases can be seen
on
Finally, Varvaressos et al.
The General Video Game AI framework (GVGAI) was
designed for serving as a testbed for general video game
playing research (Perez et al. 2015). A competition, based on the
framework, runs annually and allow competitors to submit
their agents. These are then judged based on how well they
can play a set of unseen games. The Video Game
Description Language
The use of a simple, and analytically tractable game description language, as well as a collection of AI agents, gives GVGAI important benefits related to game editors and engines such as Unity, Unreal or GameMaker. The AI agents allow us to perform automatic gameplay and game testing, what is still not possible in the cited engines because of the lack of uniformity in how their games are specified. They are, of course, versatile and powerful, but not flexible about the use of general AI agents in the way we need for collecting data and implement a general query system.
Cicero is a general AI-assisted game design tool based on
the GVGAI framework and the VGDL. As stated before,
existing tools are mostly limited to a single game and a
single mode of design assistance. As the AI-assisted game
design tool is still an emerging paradigm
As stated before, the query system is developed on top of Cicero, and it integrates SeekWhence and the visualization system. In this section, we will highlight some of the implementation details about these two systems in order to facilitate further discussion about how the query system works.
SeekWhence stores every frame of a gameplay session played by an agent. In order to store the frames, we capture every game state, at every game tick. The game state contains all the information of the set of game elements available, such as their positions in the level for example. The indexation by a game tick is what makes SeekWhence runs like a video player.
Every game element can be assigned to a specific color by the designer. It activates the visualization system [Figure 1], which captures all the positions of the elements in the level and apply a heatmap to show which areas were explored more.
Our query system is implemented on top of everything we previously discussed. It adds a database (MySQL). The database stores events and performs searches related to them. An event in VGDL happens whenever two sprites collide, it is up to the designer to define what kind of event will be fired at the moment of the collision. It can be a killSprite, a cloneSprite, and more than 20 other ones. In other words, the events are the rules of the games. The use of a database makes the implementation of the storage and the searching process easier and straightforward. The queries responsible for searching answers for the questions What, Where, When and Who are provided to the users as a GUI so that they can insert the query’s parameters [Figure 2].
Quantitative User Study Kwiri was also used in a
quantitative user study
We employed a qualitative method in order to understand, from our users, the benefits of Kwiri, as well as to solicit suggestions for future additions and improvements.
We created one inconsistency in the rules for each of three different VGDL games: Zelda, Aliens, and FireStorms. Their official rules are listed below, as well their inconsistencies.
Zelda - An action game (a clone of the cave levels of Zelda: A Link to The Past)
Can kill enemies with a sword; Changes its sprites when it gets the key; To win the level needs to get the key and access the gate.
Can occupy the same sprite of another enemy, a key and a gate; Cannot kill other enemies;
Kill Player when colliding with it.
The inconsistency in this game is that some enemies can kill other enemies.
Aliens - A Space Invaders clone
Kill enemies and barriers by shooting towards them; Cannot go through walls; To win the level needs to kill all the enemies.
Are destroyed when hit by enemy bombs, by player bullets or when colliding with enemies. The user tasks consisted of finding the inconsistency in each one of the three games. To perform the tasks they were allowed to use Kwiri, and obviously, combine it with SeekWhence and the visualization system.
The study had nine participants, all of them male, eight of them were enrolled in a university program (7 Ph.D. students and one undergraduate student), and one was a digital media professional. Just two of them did not study/work with games/gaming. The other ones had mixed experience between industry and academic fields, varying from two to fifteen years. The most cited engines and frameworks by them were Unity, Phaser, and GameMaker. They were recruited through the department e-mailling list.
The first step of the study was to ask the participants to fill out a form about their demographic data and game development experience. After that, we asked if they would be comfortable with their voices being recorded. Then we explained how SeekWhence and Kwiri work. The explanation took less than five minutes. In the end, we asked the users if they wanted to ask any question or skip it to use the tools for a quick warm up.
The second step started by introducing the users to what would be their tasks during the experiment. We informed them that we would give them three different tasks and that they should use the set of features to find a design inconsistency in each one of them. For each task, first, we handed the users a sheet of paper with all the rules of the game in the evaluation. After they read the rules, we ran the game with the agent ”adrienctx”. We choose this agent because it is a former winner of the GVGAI competition. It is able to play the games well, and most of the times can beat the levels it is playing. Then, all the data were available to the user to actually start to work with them in order to try to solve the task.
The third and last step of the procedure consisted of a conversation where the users could express their opinions about the tested features and compare it with others they used before.
We checked in our users after five minutes of working on a task. We decided on this number after three pilot tests executed with a preliminary version of the system. However, this time is not a measure of success. It was only used to verify if the user is feeling tired and/or uninterested in continue after five minutes of effort. We gave them the option to keep trying or move to another task.
We used three sources of data in order to collect the users’ activities: direct observation, audio recordings of test sessions, and a design questionnaire.
Direct observation As participants interacted with three different games and were allowed to speak about their findings, problems, and suggestions related to the tools, they had their voices recorded, and written notes were made of their overall patterns of use and verbal comments. Attention was paid to participants’ interaction between the different systems (SeekWhence, Kwiri, and visualization), how hard they had to work to find the inconsistency, what design features attracted their attention, and whether, at any stage during the study, they seemed to lose interest in the activity.
voice recorded. This helped us to validate our written notes since we used them as tags to pay attention when listening to their recordings. Also, we used this source to clarify some actions they performed which were not initially clear to us. Design questionnaire Our design questionnaire is based on semi-structured interview and required factual (e.g., ”When I was querying for what a sprite was doing.”), perceptual (e.g., ”I think that the query results annoyed me with too much information.”) and comparative (e.g., ”It would be a plus to have these features in the tools I have used before”). We started the questionnaire by asking general questions in order to let the user feel comfortable like ”Which are your thoughts about these features?”. Then we moved to specific ones, many of them influenced by our notes, like: ”In the second task, could you explain what you were trying to do when you asked if you could type your queries?”. Finally, we asked if the users had more suggestions besides the ones they suggested during the tasks.
considered suitable to analysis text materials, which varies
from media products to interview data
As explained in the previous section, we did a qualitative study to know from our users which are the significant gains of our tools and which are the points of improvements in a next iteration of our design process. We will present the data first with a general discussion of our findings by observing the users. Then we will present our conclusions based on the users’ point of view.
Most of the users were able to complete all three of the tasks with success. Just two gave up, one during the Zelda task, and another during the Aliens task.
They started by exploring their options, right after reading the game rules given to them. Some started by using SeekWhence and tried to figure out the problem just by replaying the game frame-by-frame. Then they switched to the query system to filter based on their suspicions. Other users did the opposite, starting by querying and then switched to SeekWhence. This exploratory step was skipped by two users who decided to do a warm-up during the explanation of how the system works.
One fact that grabbed our attention was that five of our users could identify one inconsistency that we were not aware of. In the game Alien they could observe that a bomb was destroying a barrier in a position it was not designed to. Figure 3.
Our data analysis is based on text transcriptions of the design questionnaire discussed in the previous section. We used a classical procedure, Qualitative Content Analysis since it is In general, the users agreed that Kwiri was a valuable resource for finding game design inconsistencies. Some of them pointed to personal experiences with situations in which they needed tools like the one presented here, but instead, they ended up hand-rolling their own.
One of the users explained that he had to write his system in order to evaluate how an AI was behaving. ”I liked these features. Some time ago, I had some issues with an AI that I was developing for playing a game. The tool I was using didn’t have this feature about navigating and use visualizations. Then I had to write my own system to do exactly what we were doing here! Because here, I really can see what the agent is doing.”
Another user, similarly, does not have the appropriate tools to inspect his game projects. ”It is a really cool feature, pretty deal! I would like to use it on my work now, especially this frame navigating tool. But I’m having to implement everything from scratch.”
A third user revealed that his work consisted of play his game to figure what was causing the problems. ”I developed a game once, but I did not have how to record it and play it again as I did here. So every time that I found a bug, the only way to debug it was by playing it again. If I had a tool like that, at least my debug process would be way easier”. Figure 4
Some users said they had problems in understanding the queries, for example, one of them stated that at some point it pops up too much information making the process not so attractive to follow. However, another user stated the opposite. In his speech, information volume does not seem to be a problem. The way that the UI leads to them is what concerns him. ”I really liked these tool, but I would like to see more information, however with less clicks.” Still, about the query system, one user said that he was not sure about the roles of a sprite in an event. ”I would like to know who is killing who in a ’killSprite’ event. The query helped me to confirm my suspicion that an enemy was killing another, but the panel in the Who area should say who is the one doing the action.”. Overall, as pointed in this section, the main issues were related to the query system. Design them as a filter tool does not seem enough. At least for the tasks evaluated, despite the majority of the users were able to use it correctly, it should provide details about the sprite roles in the game and be more explicit about the event as a user stated: ”I wasn’t sure about what this event - transfrom to - mean. I was doing assumptions based on what I have seen before on other tools like GameMaker. Fortunately, it is similar, but I would like to have this information before.”
We got many suggestions from our users that we will add in
future iterations of the system. One user said that he would
like to see small ticks on the timeline bar. The ticks would be
used to let a user know that one or more event is happening
on that part of the gameplay session. To assure what he was
suggesting we used the Youtube
Another user suggested us to present the query options as a tree visualization where he could filter based on options available in each branch of the tree. It is an interesting suggestion, and we think it can reduce user efforts and present a better way to lead them to what they are trying to find.
Finally, another user said that the same color patterns used to represent the agents that does (green) and suffers (red) the actions should be available on the panel used to make the queries. He said that in general the queries help when one has to identify who is involved in an event. However, it is also important to know who started the cause and who is getting the consequences. It was also stated as an issue by another user and is something straightforward to fix in the query UI panel.
We have presented a system for querying game events in space and time. This system is a new addition to Cicero, a general mixed-initiative game design assistance tool. We evaluated our tool in the context of finding design inconsistencies/bugs. The users were able to solve their given tasks promptly and provided us with valuable suggestions; notably, they found bugs we were not even aware of, speaking to the usefulness of the system. To our surprise, the users were significantly attracted to the replay analysis system (SeekWhence) than to the query one (Kwiri) to solve their tasks. It contradicts a previous quantitative study that shows users having much better results, in similar tasks, by using a query system than a replay analysis.
We plan to test the system further, on a large audience of game designers. We also want to design specific tasks for evaluating each one of the features separately and in different kinds of combinations. Besides finding inconsistencies, we will also focus on agent evaluation and game balancing.
Tiago Machado is supported by the Conselho Nacional de Desenvolvimento Cient´ıfico e Tecnolo´gico (CNPQ), under the Science without Borders scholarship 202859/2015-0