We consider deep reinforcement learning algorithms for playing a game based on the video input. We discuss choosing proper hyperparameters in the deep Q-network model and compare with the modelfree episodic control focused on reusing of successful strategies. The evaluation was made based on the Pong video game implemented in Unreal Engine 4.
Reinforcement learning (RL) is a eld of machine learning, that is dedicated to
agents acting in the environment in order to maximize some cumulative reward.
Actions of an agent are rewarded by the environment to reinforce correct
behaviour of the agent. In this case the agent is able to automatically learn the
optimal strategy. Methods of reinforcement learning appeared to be useful in
many areas where AI is involved: robotics [
An RL agent interacts with an environment over time. At each time step t the
agent is situated in a state st . The agent selects an action at from some action
space A , following a policy (atjst) . This policy is a probability distribution
describing the agent behavior, i.e., a mapping from state st to actions at . Then
agent receives a scalar reward rt from the environment, and transitions to the
next state st+1 according to the environment dynamics or model: reward
function R(s; a) and state transition probability P (st+1jst; at) respectively. In an
episodic RL problem [
To determine agent preference over states, a value function is introduced as
a prediction of the expected accumulated and discounted future reward. The
action-value function Q (s; a) = E[Rtjst = s; at = a] is the expected return
for selecting action a in state s and following policy afterwards. An optimal
action value function Q (s; a) is the maximum action value achievable by any
policy for state s and action a . We could de ne a state value V (s) function
and the optimal state value V (s) in the same way [
Temporal di erence (TD) learning is the key idea in RL. It learns a value function V (s) online directly from the experience with TD error, bootstrapping in a model-free and fully incremental way.
In TD learning the update rule has the following form:
V (st)
V (st) + [rt + V (st+1)
V (st)] where is a learning rate, and rt + V (st+1) V (st) is called TD error.
Similarly, a Q-learning agent learns action-value function with the update rule
Q(st; at)
Q(st; at) + [r + max Q(st+1; at+1) at+1
Q(st; at)]
In contrast to Q-learning which is o -policy approach, SARSA is an on-policy control method, with the update rule
QEC (st; at) (
Rt max QEC (st; at); Rt if (st; at) 2= QEC otherwise (1) (2) (3) (4) 1.2
While Q-networks improves performance after gradient-based optimization
procedures, model-free episodic control (EC) [
In order to remember most successful strategy, EC agent makes use of additional structure { QEC (s; a) table where the best action-values for each state are kept. At each time-step in the particular state of the environment the agent peeks the action with the maximal value of QEC (s; a) . At the end of each episode the value is updated in the following way:
Q(st; at)
Q(st; at) + [r +
Q(st+1; at+1)
Q(st; at)]
SARSA re nes the policy greedily with respect to action values. TD-learning, Q-learning and SARSA converge under certain conditions. From optimal actionvalue function one can derive an optimal policy for RL agent.
To carry out maximization's of QEC (s; a) over all states, it is regarded as a nearest neighborhood model. Then of particular interest is the mapping of observations onto states as observations ot tend to have high dimensionality than drastically slows down KNN search. 2
To illustrate application of RL methods, we implemented them in the Pong
Game environment [
The main goal of this work is to prove that it is feasible to apply methods
of RL inside UE4. We show that it is indeed possible to e ciently train and
apply model built with the modern machine learning framework (in our case
TensorFlow [
We achieve this by patching a plugin to support Python scripting inside UE4, implementing C++ module for capturing game screenshots, and creating TensorFlow-based Python controller for the player paddle. 2.1
We use physics based UE4 implementation of classical Pong environment.
This screenshot from the environment shows main elements of the game: 1. Paddles. Every player controls a solid rectangle called paddle that can reect the ball. 2. Ball. The ball is a rigid body that moves with a constant speed and re ects from the walls and paddles. 3. Walls. The walls are present on the top and on the bottom of the screen. 4. Goals. The left and right sides of the screen represent goals. In order to score a point, the player needs to hit the opposite goal with the ball. 5. Scores. The top part of the screen shows two numbers that are equal to the number of points each player have scored.
The purpose of the game is to maximize the di erence between your score and opponent score, while making it positive during time limit.
The human player uses raw pixels (screenshots, 5 frames per seconds) as an input during play, and outputs three types of actions using the keyboard: { Key up: move paddle up { Key down: move paddle down { Idle: paddle stays at the same place
The game comes with the built-in AI controller that gets high-level features such as position and speed of the ball, position of his and opponent paddle as an input, and uses rule-based approach to control the paddle. 2.2
In Unreal Engine 4, the standard way to implement game logic is through writing C++ modules or using internal visual scripting language called Blueprints. The C++ approach is more powerful and is used to implement core game logic and reusable modules, though in many cases it is overly verbose, and does not allow to quickly iterate on the solution due to slow compilation speeds. On the other hand, blueprints are simple to create and understand, provide faster development cycles, but yet not expressible enough in many cases.
When it comes to the modern machine learning engines, they are usually written in C++, while the provided public API comes in form of Python bindings. Although it is possible to use TensorFlow C++ API, it limits the reuse of openly available RL algorithms implemented in Python, and slows down the research process due to the nature of the C++ language.
Because of all these reasons we looked into alternative languages support
for UE4 and identi ed two candidates: Python and Lua. Both languages are
supported through third-party plugins available on GitHub, UETorch [
As the authors were more familiar with Python and TensorFlow, it was chosen as the primary language.
The UnrealEnginePython plugin allows users to write Python scripts that interact with UE4 by being able to access and mutate the internal state of the game. This can be used to obtain current position of the ball and paddle, and giving the commands to move the paddle.During this work the original plugin was extended to support Python-based controllers in UE4. 2.3
We implement AI controller using TensorFlow machine learning framework [
We use Deep Q-Network (DQN) [
We have built a Deep RL based controller that outperforms the standard rulebased AI, while using raw pixels as an input. The training takes 6 hours on GPU and it takes 10 million iterations to beat the built-in scripted AI.
However, with regard to Episodic Control it was noticed that Euclidean
distance is impractical way to measure frames' similarity. In case of binary images,
Euclidean distance scores only sum of di erent pixels but not their relative
proximity. In other words, using Euclidean distance it is impossible to choose frames
where a ball is closer to its current position. Later, the authors found a result
on too long applicability of episodic control in similar tasks [
The code implementation is available on GitHub ([