The development of arti cial intelligence is one of the most common problems in the video games industry. In most cases, the behavior of computer characters is de ned by classical, deterministic algorithms. However, with increasing com-plexity of AI behavior, the complexity of the code describing that behavior also increases. Even more di cult is to create an AI that behaves like a real player. A deterministic algorithm will work e ciently but its behavior may look unnatural and unpleasant. To solve this problem, it is possible to use Machine Learning. Achievements in this eld have found application in di erent niches, and video games have been no exception. This study explores the creation of a convincing and e ective AI by simultaneous use of Imitation Learning and Reinforcement Learning. Also this study explores the tools for creating the Learning Environ-ment and learning AI agents, the principles of writing the program code for AI agents, and gives practical recommendations to speed up the learning of AI and improve its e ciency. As a practical example, for the purposes of this study, will be created an agent to control the tank, capable of maneuvering and ghting with several opponents simultaneously.
Since the creation of video games, the problem of arti cial intelligence development, which would make games more interesting to play, has always been relevant. Often arti cial intelligence was inferior to players in skills, so video games became too simple for them. In this case, the developers usually resorted to various tricks to adjust Copyright c by the paper's authors. Copying permitted for private and academic purposes. the forces of man and computer opponent. If we are talking about a racing simulator, then the so-called "catchup" is used. The characteristics of a car belonging to a computer are arti cially overstated. Thus, regardless of how skillful the player is, the computer can play with him as an equal. Another example - games of the RTS genre, where the computer takes advantage of the multiplier for the extracted resources. However, such tricks look like a fraud and reject the player. That is why online games are so popular, because playing with a real person is much more interesting. Thus, the problem is the extremely low or unreasonable level of intelligence of computer opponents, which is inferior to the human. Using Machine Learning will solve this problem and greatly expand the capabilities of AI in games. One of the most famous studies on this topic is the paper of DeepMind Technologies employees. With the help of Q-Learning, they managed to implement an algorithm capable of playing simple Atari 2600 video games without knowing anything about them, except for the pixels on the screen. If more information is provided to the neural network (such as, for example, the coordinates of game objects), the application of AI expands. Gathering information about the state of the game world is a fairly simple task if the neural network is intended for in-game AI, which implies the existence of game sources, in contrast to the example with Atari games. In this study the creation and learning within the game agent, which is able to control the tank in a three-dimensional video game, will be considered. The agent's tasks include maneuvering to avoid enemy shells, e ective shooting at enemy tanks, turning the turret and using additional game mechanics to gain an advantage on the battle eld.
As a tool for creation of Learning Environment was chosen the Unity game engine. To train game agents was used the Tensor ow. To use trained agents in the game, the project with open source Unity Machine Learning Agents was used. For learning, two approaches will be used simultaneously: Reinforcement Learning and Imitation Learning. 2.1
The main idea of Reinforcement Learning is that the t-agent exists in a certain S-environment. At any time, the agent may perform an action (or several actions) from the set of A-actions. In response to this action, the environment changes its state and the agent receives the r-reward. Based on this interaction with the environment, the agent must choose the optimal strategy that maximizes his reward.
Reinforcement Learning is especially good for solving problems associated with a choice between long-term and short-term bene ts. It has been successfully applied in various elds, such as robotics, telecommunications, elevator management. Also Reinforcement Learning is a good way to develop AI in games. In the case of games, the game character acts as an agent and the world around him and his opponents act as an environment. Every time the character performs an action that approximates him to win, he receives a reinforcing reward. For example, the car agent on a racing track receives a reward over time if the distance to the nish line is reduced. This is also works the other way around - when performing ine ective actions, the agent receives a punishment. In order for the agent to perform e ective actions, it is necessary for him to receive an array of data characterizing the state of the environment. The amount of this data should be su cient to ensure that the agent receives all the necessary information about the environment, but not be too large for the agent to train more e ectively. Also, it is necessary to normalize the input data, so that the values of the signals arriving to the agent were within the range of [0; 1] or [-1; 1]. Examples of the input signal for a car include speed and position on the racetrack. An array with action signals is the result of the agent's work. As well as input signals, they require normalization. Examples of the input signal for a car include the gas pedal [0; 1] and the steering [-1; 1].
As opposed to Reinforcement Learning, which works with a reward/punishment mechanism, Imitation Learning uses a system based on the interaction between a Teacher agent (performing the task) and a Student agent (imitating the teacher). This is very useful in situations where you don't want your AI to have machine-like perfection, but want it to behave like a real person instead. Imitation Learning Support was implemented in ML-Agents v0.3 Beta. This functionality is a powerful mechanism which allows to develop AI with complex behavior using a little e ort.
From the outside, the process of AI development looks like this: there are two agents, one is a Teacher and another is a Student. Another neural network, a person or a deterministic algorithm may act as a Teacher. The most e ective results are achieved if the Teacher is a real person. Next, the learning process begins. The Teacher plays for a while. The timing varies depending on the task complexity. For simple tasks, it takes about 5 minutes. For complex tasks, it is required up to 2 hours. The learning is that while the Teacher plays, the Student watches his actions and makes attempts to imitate him. 2.3
Reinforcement learning and Imitation Learning might be used simultaneously. The Student will watch the Teacher and receive a reward depending on his actions. Simultaneous use of the two approaches makes it possible to achieve much better results than the separate use of them. Initially, only Reinforcement Learning was used in the practical example of this study, and the results were unsatisfactory. After 10 hours of learning, the tank, controlled by an agent, was able to ght a maximum of two opponents. The e ectiveness of the tank with more number of opponents, was signi cantly reduced. However, after Imitation Learning support was added, the tank was able to lead an equal ght with 3-4 opponents after only 40 minutes of learning 2.4
As a practical example, a battle eld with 1 tank under the agent's control and belonging to the green team and 5 tanks under the simple, deterministic algorithm control and belonging to the red team is used. After completing training of the neural network, the green team's tank must be able to resist the tanks of the red team - this will be the criterion of a successfully trained agent. As a battle eld, a at rectangular area that simulates the surface of the earth, is used. In total, 20 battle elds were established, located one above the other at a distance of 50 meters. This approach allows to signi cantly accelerate the training of agents due to their number and more intensive accumulation of information.
Each round is as follows: there are all the necessary tanks on the battle eld and they are placed in random order. Then the battle begins and continues until the tanks of only one team are left. After the battle ends, all the remaining tanks are destroyed, then the round begins again. The game uses the mechanics of "charge accumulation". Its essence is that if you do not shoot for a long time, then the accumulation of "energy" begins. With full accumulation of the scale, it becomes possible to make a shot with a burst of ve volleys. The trained neural network should also be able to use this opportunity competently and make the right choice between an immediate single shot or waiting and risk to get ve shots. 3 3.1
For proper operation of the neural network, an important factor is the correct choice of its architecture and parameters. In this case, based on the existing information on the experience of using di erent types of neural networks for di erent tasks, several di erent con gurations were selected and tested. The network parameters that showed the best results are as follows:
The signal is transmitted inside the agent using the AddVec-torObs function built into ML-Agents. To achieve
the best results during training, the input signals must be normalized to the range of [
Another 18 signals re ect information about the nearest two enemies, per 9 signals each: The angle between the shell of the enemy tank and the position of the agent
Previous value, but converted. This is described in more detail below.
The remaining two signals contain information about the third enemy (the distance and angle relative to the shell). Information about distant enemies is not transmitted, since it is insigni cant. Also, some signals were pre-processed, so that the neural network would more e ectively perceive them. So, for the angle of rotation of the turret to the enemy, there are two signals that have di erent shapes (See Fig. 5).
The left signal has a linear dependence and allows the neural network to quickly direct the turret to the enemy. The right signal has a non-linear dependence and allows to target more accurately. Due to the shape of the curve, the neural network reacts much more strongly to minor rotation angles of the turret. In other words, it is a program analogue of the telescopic sight. In the course of the experiments, it was found that such an approach allows to increase the number of hits of projectiles by 35%. It is recommended to apply this approach to any signals containing small but important ranges where it is necessary to detect the slightest changes. The output signals of the neural network allow to control the tank. At the output, the neural network of the practical example has 4 signals:
ShotSignalAction = Mathf.Clamp(vectorAction[0], 0, 1);
AccelAction = Mathf.Clamp(vectorAction[
For the correct work of Reinforcement Learning algorithm, it is necessary to use the reward/punishment system at training. This is a very important point; it in uences the result behavior of the neural network. Rewards are issued using the AddReward (value) method. The ML-Agents documentation states that rewards should be issued only according to the nal result, not the actions, but in this practical example, the issuance of small rewards for certain actions positively a ected the results. The nal reward system is as follows: Damage - +0.1
4 4.1
After creating the Learning Environments and writing the code for the agents, you can begin to train the neural network. However, before that, it is worth taking several actions. One of the agents should be appointed as a Teacher and it must be managed by a real person. To do this, it is necessary to use the Brain component from the ML-agents project and assign it a Player type. Then it is necessary to appoint the control. Having any of the famous game controllers like the Xbox 360, you may get some advantage. In contrast to the keyboard control, the use of the controller allows to transmit an analog signal instead of a discrete one, because the controller has analog triggers and joysticks. Unfortunately, the CoreBrainPlayer handles signals from the keyboard only. However, the Xbox 360 controller support is a trivial task, all it takes is to add several code lines to the input handler: [System.Serializable] private class JoyAxisActions { public string axis; public int index; public float scale = 1f; } [SerializeField] [Tooltip("The list of axis actions.")] private JoyAxisActions[] axisActions; <..> foreach (JoyAxisActions axisAction in axisActions) { var axisValue = Input.GetAxis(axisAction.axis); axisValue *= axisAction.scale; if (Mathf.Abs(axisValue) > 0.001) { action[axisAction.index] = axisValue; } } 4.2
After all the settings have been completed, it is possible to build an executable le that will be used by the Tensor ow environment. In order to start the learning process, it is necessary to insert the following command: python python/learn.py build/build.exe --run-id=0133 --train -slow A window with the game will open, and the person using the previously assigned commands will be able to control the tank. It is very important to prepare, you need to be able to play well - the better the Teacher plays, the better the neural network learns. For a game scene, it is desirable to prepare two cameras in advance. One camera (main) will be aimed at the teacher's tank. The second one (auxiliary) - at the tank of one of the trained agents - it may be placed on a screen quarter in one of the corners. This method will allow to monitor the progress of learning (see Fig. 6)
It's worth displaying all the debug information on the screen, because it's easy to make a mistake in the code and it's much better to learn about it right away and not after a failed learning attempt. It is also very helpful
This study explores the opportunities and bene ts of simultaneous use of Reinforcement Learning and Imitation Learning in arti cial intelligence development for video games. Tools for creating the Learning Environment and learning AI agents have been considered. Practical recommendations, allowing to optimize the parameters and characteristics of the neural network and to conduct more e ective training, were given. In the nal result, a video game agent, which controls the tank, e ectively uses the available game mechanics and whose behavior is similar to a human, was created and trained.