Through deep reinforcement learning, a computer can learn to play simple video games under the same conditions as human players - by watching the pixels on the screen and issuing the button-press type of actions. In this paper, we investigate how quickly a computer can reach and surpass human performance in a simple two-player video game, given no background knowledge of how to play the game. We implement a Deep Q-Learning (DQN) algorithm using Python and integrate it with the Atari 2600 emulator that runs the Pong game. We train the neural network for up to 3 million training steps and evaluate its performance after every 100.000 steps. As a reference, we measure the performance of 18 human players and take their average rating as the human-level baseline. We propose an evaluation metric that considers the obtained game points and the player's endurance during the game. We find that the Deep Q-Learning algorithm can surpass beginner-level human players in playing the Pong game after about 4 hours of training.
With the advances of deep learning [
In this paper, we aim to answer the research question of how quickly DQN can reach and surpass human performance in a simple two-player video game. We present the results of a video game tournament that we organized to evaluate the human progress in playing the classic Pong game on the Atari 2600 platform. We propose an evaluation metric that encodes the player’s endurance and the achieved score, which we compare with the values achieved by the computer at diferent stages of training.
We implemented the DQN reinforcement learning algorithm [
We trained the DQN for 3 million steps, which accounted for 797 played games. The game runs natively at 20 steps per second, which takes 41.67 hours of real-time gameplay to reach 3 million steps. In our case, the training process lasted about 49 hours on a desktop computer without GPU support. The training started with a high exploration factor to obtain diverse playing experiences, which means that most actions were chosen randomly. The ratio of actions chosen randomly vs. through the learned policy was linearly reduced with the number of training steps until it reached the ratio of 1:10 at 1 million steps and remained constant from that point on. We saved the weights and biases of the trained Q-network every 100,000 steps, thus obtaining 30 diferent AI players, each later one supposedly performing better than the preceding ones. All 30 AI players were then given to play 100 full games against the computer opponent to evaluate their average rating.
To evaluate the performance of human players, we organized a tournament in playing the Pong game, which was attended by 18 students of the University of Primorska. None of the participants were proficient in playing this game before; all could be considered complete beginners. Each participant played three full games against the computer opponent in an Atari 2600 emulator on a laptop computer. Players used a standard computer keyboard and played the game on the same system. They were not allowed to practice the game beforehand. A single game typically lasted about 15 minutes. Compared to today’s standard of computer gaming, Pong is not a particularly engaging game. We observed that the participants started feeling bored and demotivated after finishing three games, which is why we limited the experiment to playing only three games. Our experiment, therefore, measures the performance of beginner players.
We argue that the final game score alone is not a good indicator of the player’s performance. For example, suppose that two players both lose the game without hitting a single goal, but one player manages to play the game for a longer time than the other. Although they both received the same final score, the player with more extended gameplay demonstrates more proficiency. We, therefore, define the player’s rating measured in one gameplay as: rating = 103 · ∑︀=1 · − 1
, (1) where is the number of episodes (either side scores a goal, and a reward is collected), = ± 1 is the award received by the agent, and is the duration of the episode in steps. The constant 103 is chosen to scale the rating into an easily readable interval. This rating is used the same for AI players and humans.
The ratings (1) achieved by human players in all 54 played games are shown in Table 1. A negative rating indicates that the human player performs worse than the built-in computer opponent, and a positive value indicates that the human player performs better. As seen from the table, participants lost all the games but were improving in performance with each game. The average rating over all 54 games was − 8.03, with a standard deviation of 2.84. Therefore, we conclude that human player rating at the beginner’s level is typically between − 10.87 and − 5.19. This interval is depicted on the plot in Figure 1 by a green strip.
We evaluated the 30 trained AI players separately from the organized tournament, but the ratings are comparable because we used the same evaluation metric. The red plot in Figure 1 shows the average rating over 100 played games for each level of AI training, with the yellow strip representing the standard deviation. The rating converges relatively quickly towards zero, which means that the trained AI player reaches the performance of the computer opponent but does not surpass it. By observing the hard-coded strategy of the computer opponent, it becomes clear that it follows a near-optimal defense strategy by always keeping the paddle aligned with the vertical position of the ball. This makes the opponent hard to defeat but not dificult to g n i t a r r e y a l P 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Number of training steps × 105 match by inventing a similar defensive strategy. The training is also not stable, which we can see on the plot as occasional drops in performance along the training timeline. However, these instabilities are quickly compensated with further training. Human performance was surpassed in about 300,000 training steps, which is roughly 4 hours of real-time gameplay.
We researched the performance of deep learning against humans in playing a simple video game. Our research question was how quickly computers could reach and outperform human players in action gaming when playing under the same sensory conditions as humans. We chose to conduct our experiment on the classical Atari 2600 Pong game and measured the performance of 18 people who played 54 games altogether. The number of games was determined after a discussion with the participants, who declared that after the third game, they were demotivated to carry out the experiment longer. We argued that the final score alone is an insuficient measurement of performance and proposed a rating that also incorporates the player’s endurance. The comparison between the measured human ratings and the ratings achieved through deep reinforcement learning showed that the beginner’s human level of performance in playing Pong could be surpassed by AI if trained for about 4 hours or longer. However, this may not be the case with more proficient players. Neither the AI player nor any of our participants ever managed to surpass the performance of the built-in computer opponent, but an experienced human Pong player might.
The participants were not versed in playing this game, so if they were allowed to play more games or to practice before attending the experiment, the measured human rating would probably be higher. We observed that the average ratings improved with every game played. The biggest advancement was made between the first and the second gameplay, mainly because the participants were getting used to controlling the paddle during the first game, and only in the later games focused more on the strategy of playing. A significant drawback was also the use of a keyboard instead of the original analog controller for which the Pong game was originally made.
In this research, we only considered playing against the computer opponent. The idea of a human playing against a trained AI player is an interesting one, but it is not easy to come up with a feasible approach. Expecting the participants to play for hours with a reinforcement learning algorithm is not practical. It is also likely that the training would take a long time because humans tend to experiment with more complex playing strategies and can be inconsistent in their decisions. If trained in advance against the computer opponent, the AI player overfits to the hard-coded strategy and fails to play against humans. Another possibility is to implement adversarial training of two AI players that play against each other. However, the strategy space the two agents would have to explore is vast, and there is no guarantee that the training would be stable. Moreover, the learned adversarial strategies may not work well against human players.