In this research, we explore the problem solving strategies of both humans and AI agents in the open-ended domain of video games. We utilize data collected from several human-level performing AI agents, that follow a given policy, and data from expert human players, that follow a set of strategies, for two Atari 2600 console games. We compare both types of data streams using a visualization technique to gain insights about how each player type, AI or expert human, go about solving the given games. Analyzing the action sequences of the two, we demonstrate how closely the agent policies resemble the real-world problem solving of a human player, and explore how we might extract human-level strategies for agent policies. We reflect on the benefits of using data from both AI agents and expert humans to instruct learners, model their behaviour, and how strategies may be more apparent and easier to adopt from human play. Finally, we hypothesize the benefits of combining both types of data for learning these complex tasks within open-ended domains.
The process of building expertise, especially in complex tasks, has
been an area of study for some time in education [
In the development of these agents, it is the human encoding the
strategy into the AI using their knowledge of the game. The
majority of game-playing agents, however, make use of deep
neural nets to develop their policies, which makes them black box
and often difficult to interpret by a human. Recent work has
looked at making policies developed this way programmatically
interpretable, but much work remains for humans to be able to
clearly articulate what many of these agents have learned from
their training [
While previous work continues to reduce the amount of training
data required to develop successful agents via self-learning, others
look to use human games to seed agents. One such study found
that training on human data, they could achieve comparable scores
to state-of-the-art reinforcement learning techniques and even beat
the scores using just the top 50% of their collected data for more
complicated games, such as pinball [
In complex tasks humans generate strategies which can be applied in many different situations. Combinations of strategies that lead to optimal outcomes can lead to expertise in a domain, although there is still no consensus among researchers as to what makes a person an expert and how expertise is defined. In this research we explore the interactions of policies and strategies, then look at how both relate to expertise in the context of these two games. Our long term goal is to see how humans can help teach agents and agents can help teach humans in a continuous loop hence the idea of “teachable humans and teachable agents.” Specifically in this work, the main contribution is a start to this goal with a novel comparison of agent policies, generated with two different state of the art techniques on several complex game domains, and strategies generated from human players. We do this through the use of visualizing both types, expert human and agent, of collected gameplay data using t-SNE diagrams of the state spaces as a means to compare the two. We believe this work can help lead to a better understanding of human strategies and expertise, while also contributing to data mining techniques which can further be used in the context of explainable AI for educational systems. The visualization and comparison techniques used can be extended to more traditionally educational games, to gain a sense of any strategies being enacted. Additionally, it is beneficial to see if the agents are solving the game in a natural way, using a human-like strategy, as many similar systems are often designed to playtest such games and act as tutors to the users. 2.
Expertise has been the subject at the crossroads of Psychology and
Computer Science for some time. One of the first compiled works
came from Glaser et al., The Nature of Expertise [
The question of what exactly defines someone as an expert is still
an open question and has a lot to do with the particular domain
that is being studied. In chess, Chase and Simon posited that it
takes 10,000 hours of study to become an expert in chess [
In the case of learning systems, we often define mastery using
some form of knowledge tracing. These systems often set
“mastery” as a probabilistic value that a learner knows a particular
skill. The value of mastery varies on skills and domains, but often
a value of 90% or 95% are assumed to have achieved mastery [
Artificial intelligence has been used now for decades to create
agents that mimic human behavior. These agents are generally
driven by a policy created by some form of machine learning such
as reinforcement learning [
In education, agents driven by policies have long been a
foundational part of data-driven intelligent tutors and adaptive
learning. Work has been done in modeling learning at a partially
observable Markov decision process (POMDP), and using a
policy generated to predict what a student knows and what the
next best instructional lesson is for a particular student [
Visualizations of gameplay data are widely popular, often
being used by players to compare their performance against
others and to make sense of how they played the game [
Many games are making use of such game-playing agents
and procedurally generated content methods to develop
both the game environment and to play-test the games [
The Arcade Learning Environment (ALE) provides a framework
consisting of over fifty Atari 2600 games that can used to evaluate
competency in deep reinforcement learning (DRL) agents and
other types of AI [
In the game Space Invaders, depicted in Figure 1, the player or agent controls a ship at the bottom of the screen that can navigate along a single dimension of left or right. The goal of the game is to destroy all the enemy units above the user’s ship, gaining points for each enemy destroyed, while also avoiding any projectiles from them. If the player is struck by an enemy projectile they lose one of their three lives. To destroy these enemy units, the player’s ship can fire a projectile that goes directly up, damaging or destroying an enemy unit on contact. Additionally, the player can hide behind three objects at the bottom of the screen to avoid the enemy fire. The only valid controls for this game are left and right to move the player agent and the fire button to shoot.
In the game Seaquest, depicted in Figure 2, the ultimate goal is to
retrieve as many scuba divers from under the water as possible.
The player or agent controls a submarine that can navigate in all
directions around the screen and faces the front of the ship in the
direction of movement, either right or left. This submarine has an
oxygen tank gauge that slowly diminishes over time, the player
must surface their ship at the top of the screen to refill it. As they
navigate around the screen, collecting the divers, they also must
dodge enemy ships and sharks that navigate across the map. If
their submarine collides with an enemy unit or the oxygen gauge
reaches zero, they lose one of their three lives. To combat these
enemies, they are able to shoot a projectile from the front of the
ship, which damages or destroys these enemy units. Killing an
enemy results in a point increase, but the main increase in points
comes from saving the divers. In order to receive points for the
collected divers, the submarine must surface by navigating to the
very top of the map. All valid button combinations for the Atari
2600 controller work for this game, such as up-left-fire, right-fire,
and down.
As the ALE provides a framework for testing DRL agents, we
selected two higher performing agents implemented in the
environment using value-based DRL algorithms. The first agent
utilizes a Deep Q-network (DQN) and has achieved a level
comparable to a human professional in almost fifty games,
including the two we investigate [
The data for both of these agents come from the benchmarks used
in the Atari Zoo , an open-source set of trained models for six
major DRL algorithms at varying benchmarks, collected from the
ALE [
DQN 2380
Using the ALE, we collected expert human data for both Space
Invaders and Seaquest. To collect the human game play data, we
modified the ALE code to record the RAM state at each frame of
gameplay, so that it could be compared to the agent data from the
Atari Zoo . Using Atari 2600 data collected from the Atari Grand
Challenge project as a baseline for Space Invaders, our collected
expert human data ranks in the top 1% based on scores [
A popular technique used for dimensionality reduction and
visualization of high-dimensional data used with large
reinforcement learning datasets is t-SNE [
Human
1685
1745
1845
12590
14220
16880
different DRL algorithms. As these datasets were quite large for
both games, we pre-processed them using Principal Component
AnalysiS (PCA) to a dimensionality of 50, then followed that with
300 t-SNE iterations with a perplexity of 30 [
Utilizing the code provided from the Atari Zoo [
Plotting the DQN, Rainbow, and expert human data from Space Invaders via t-SNE, we can see both similarities and differences in the clustering. Figure 3 depicts a t-SNE embedding of nine Space Invaders games in total, three from each agent and the expert human. The agent data, green depicting Rainbow and blue for DQN, overlaps more throughout the graph than the human data points, represented by red. A majority of the human data clusterings are on the bottom half of the t-SNE, where there only appears to be a single Rainbow and DQN cluster. There is an equal separation of high scoring points, depicted by darker shades of the color, for all three parties. High scoring human points of dark red are scattered about, while the dark blue DQN data is grouped toward the upper center. Above that is the dark green Rainbow data, that is grouped between the 100 and 150 points of the y-axis. Ultimately while there is similar clustering of the Rainbow and DQN agents across all three games, it does not hold true for when the game is coming to an end and a higher score has been achieved. Additionally, regardless of the game’s score, the human data does not seem to have much overlap with either agent.
To further identify any interesting clustering of the points, we selected a single game play from the two agents and the human data, so the t-SNE would show one from each for a total of three, instead of the aforementioned nine. The resulting t-SNE for this is shown in Figure 4, along with screenshots that are representative of the major clusters. We included screenshots for six clusters, two from each, that are darker in color corresponding to a higher score and being further along in the game. Since this depicts a later point in the game, any key moves or strategies are more visible since they have had time to be enacted. With a single game for each agent or human depicted, the representative clusters stand out even more.
Following the same steps of the Space Invaders data, we plotted the collected Seaquest gameplay data via t-SNE. Figure 5 depicts the t-SNE embedding of nine Seaquest games in total, three from each agent and three collected from the expert human. Similar to the Space Invaders t-SNE embedding, the two Rainbow and DQN agents, represented by green and blue respectively, overlap more with one another than the expert human data, represented by the red points. However, all three types of points about the diagram are much less clustered into groups and more spread out throughout the given range, indicating a greater variance of game states between the two agents and the expert human. One notable clustering resulting from all nines games is a grouping in the center, where the Rainbow agent, shown in green, almost perfectly overlaps the DQN agent, shown in blue. For this cluster, the points for both agents are also darker, indicating they are for higher scoring states that occur later during the game play.
As the resulting data appears to be fairly scattered for all nine games of Seaquest, we selected just the third play through for both agents and the humans and displayed it via t-SNE, shown in Figure 6. With just a single game from each source displaying, several clusterings became more apparent. The Rainbow agent has three distinct clusters, two of which overlap with the DQN agent. Screenshots from these two clusters depict the player unit, the yellow submarine, towards the center of the screen with no enemies around. The representative screenshots depicting the expert human data, via the red points, show the shit less towards the center and with more enemy units about. This suggests a potential difference in gameplay between the agents and the human, that we elaborate on in the following discussion.
Plotting the game data via t-SNE provides a concise visualization of such high-dimensional data. However, they are only beneficial to us if their clusterings detail any patterns that might be indicative of a strategy or interesting behaviour. One immediate clustering that caught our eye was for the t-SNE depicting a single game of Space Invaders for the two agents and humans. As Figure 7 shows, there is a clear dark red cluster of expert human data toward the center, indicating there are many similar game states here and ones with a high game score comparatively. Examining the points on this curving cluster, we noticed the screenshots representing the game states at the time had a clear similarity. The states in this cluster were for when a single enemy ship was left on the map, the point right before the player can advance to the next stage. It became clear that human player had difficulty hitting the last few enemies, as they move fast and requires precise aiming when there are not many left. A nearby cluster from the Rainbow agent, represented in green and highlighted in Figure 7 too, depicts a similar set of states. However, there are not as many points for the agent in this set of states as there are the human, suggesting the agent can more accurately hit the fast moving last remaining enemies.
While this is not a particular strategy, it does provide insights into similar difficulties both agent and human have in the game. It also aligns with the maximum scores both agent and human achieved, as the agent spent less time on this phase and could advance through the game more rapidly, achieving a higher score in the allotted time, which one might equate to expertise in this domain. It is a case where the Rainbow agent is reflecting a difficulty also encountered by the human. If this was in the context of an educational game, we could use the agent’s data to gain insights into where a hint or other feedback might be the most optimal, as it is a clear point of difficulty. Additionally, the DQN agent did not demonstrate such difficult. If just the DQN agent’s data was used for such playtesting, this area of struggle may have been missed altogether. This insight was provided through a brief visual inspection enabled via t-SNE, that may not be as readily clear from parsing log data.
Another analysis of the same t-SNE plot for a game of Space Invaders provided insights into a distinct shooting strategy the two agents and human each had. When we inspected clusters and points that represented the game at a halfway point, where half the enemy units on the screen were destroyed and the other half alive, we noticed an interesting pattern in the configuration of the remaining enemies. As Figure 8 shows, the Rainbow and DQN agents target enemies either horizontally across the bottom or in a diagonal pattern. However, the expert human destroys the enemy units starting from the left column and working right. While each of these represents a different shooting strategy for the given player, the expert human’s strategy is debatably the most optimal. For Space Invaders, the enemy units move about the screen horizontally, and once they reach the edge of the screen they move down a single row, and continue moving the opposite horizontal direction. This means that if there are few enemy columns, it takes the enemies a greater time to horizontally traverse, allowing the player more time to fire at them. There are trade-offs for this strategy though, as if the bottom row of the enemy units comes into contact with the ground, the game is over. This may be the reason why the deep reinforcement learning trained agents shoot in a horizontal or diagonal pattern, so that they keep the bottom row higher up to avoid the game over condition, something they must have encountered quite often during their early training phases. However, this does not translate to an optimal strategy as the expert human data reveals. Teaching a player the game using such agents could lead to the adoption of this firing strategy, which would be suboptimal compared to that of the human’s. Such a case could also readily apply to more educational game contexts, as an agent or tutor that learned to play or solve the problem may be doing so in a non-optimal way compared to that of an expert human. Even though the “score” for a given game is greater, ultimately learning the better strategy would have a greater pay off in the long run.
Examining the t-SNE for a game of Seaquest also revealed insights into the differing navigational strategies used by both agent and human. For this t-SNE plotting, there was less clustering compared to the Space Invaders ones. However, as we investigated the different points and viewed the screenshots for representative states, a pattern with the player-controlled submarine emerged. For both the DQN and Rainbow agent, the submarine remained towards the bottom of the map and stayed centered on the y-axis, unless they were briefly moving to rescue a diver. However, the human points showed the submarine in a variety of positions that were far from the center or bottom axis, even without the presence of these scuba divers. As Figure 9 depicts, the human made use of more free moving navigational behavior, traversing the entire map and getting towards the edges to allow themselves more time to position and fire at enemies. The agents, who presumably had better accuracy from their mass training, could remain towards the center and only leave the bottom when they had to move upwards to fire at an enemy or surface for oxygen.
Similarly to the Space Invaders strategy of creating different enemy configurations, the agents for Seaquest had their own strategy that differed from the expert human gameplay. In this game’s context, there is not necessarily a clear benefit of one navigational strategy over the other. However, the one used by the agents might be better suited for a player who has better aim and does not need to get their avatar close to the enemy units. If a human user learned from the actions of these agents though, they might not move around the map as the expert human gameplay did. While not necessarily impacting the score, it could impact their enjoyment of the game, as they will be making less movements and have less control over their avatar, compared to treating it as basically fixed along the y-axis. This is another consideration of using just an agent to playtest or learn from, as even when it might not impact performance, other factors like enjoyment might be impacted from the enactment of certain agent performed strategies.
This research represents our initial exploratory work into understanding expertise of complex tasks in open ended domains using a combination of human and artificial intelligence agents. We have begun by plotting expert human and human-level agents using t-SNEs to provide a way for us to visualize data. We can see from the plotted t-SNEs that expert human data does have some overlap with data from high performing DRL agents, however, gaps exist where humans clusters are far away from the agent data. Nevertheless strategies from both human and agent data emerge in the visualizations and allow for some interesting comparisons between the two. There are clear implications of using just an agent’s gameplay, as the enacted strategies may be optimal, but limiting to a user’s play. They also might demonstrate a clear strategy, such as the firing configuration in Space Invaders, yet such a strategy could actually be sub-optimal for a human to enact.
While these two games are not traditional educational ones, the
implications of the techniques used and insights gained are still
applicable to ones in such a context. Eliciting strategies,
regardless of coming from an AI system or human, is challenging
and such visualizations provide one way to search for and
understand them. At present, the use of agents using similar
mechanisms and reinforcement learning methods to solve
problems then instruct students agents [
Our primary goal in this work is to explore expertise, in this case in the context of games. In such games, prior work often uses the score as a measure of how expert a player, either human or agent, is at the game. We believe in addition to the score, the strategies used to solve the game impact how expertise, in this domain, can be quantified. To gain insights into such strategies we visualized gameplay data of a high scoring and long time playing human, deemed an expert, and high scoring agents gameplay data via t-SNE. Analysis of the resulting t-SNEs yielded insights into both shared and differing strategies the two parties had. Even between agents, there existed similar and dissimilar strategies, in addition to their score variance. Taking into account these gameplay differences and how realistic an enacted strategy might be for a human to learn from or mimick is important for game and tutor developers to keep in mind when using agents as playtesters or instructors. A strategy might be seem beneficial, yet compared to a different one it may not be as optimal nor practical for a player or learner to utilize in their own gameplay.
As we continue this work, we want to extend it to more games other than Space Invaders and Seaquest, particularly ones in the educational space that also have accompanying agent-collected data. Further inspection remains to be done to draw more strategies from the accompanying visualizations. Following this, we will further look into how they cluster, indicating the performance of similar strategies based on their policies. One key area we plan to explore is adding a temporal aspect to the t-SNE graphs. Although not represented in our current visualizations, we do have the screenshots numbered temporarily, so we expect that we can connect the paths to show the progression of game play. Additionally, visualizing novice human data, in addition to the expert and agent data, could provide useful strategy comparisons. This could help developers of educational games find where their novice learners seem to struggle the most, from a visual standpoint.
The research reported here was supported in part by a training grant from the Institute of Education Sciences (R305B150008). Opinions expressed do not represent the views of the U.S. Department of Education.