This paper introduces TotalBotWar, a novel pseudo-real-time multi-action challenge for game AI. Additionally, it includes initial experiments that assess the framework's performance with various agents. TotalBotWar is inspired by the real-time battles found in the popular TotalWar game series, where players command armies to defeat their opponents. In this game, each turn comprises a set of orders to control one's units. As the game progresses, the number and specific orders that can be executed in a turn change. An intriguing aspect of TotalBotWar is that if a unit doesn't receive an order in a turn, it continues the action it performed in the previous turn. This characteristic results in a rapidly increasing turn-wise branching factor, making it challenging for traditional algorithms. Furthermore, the game's partial observability of the game state makes it a valuable platform for testing modern AI algorithms.
CEUR ceur-ws.org
In recent years, games have proven to be important test-beds for Artificial Intelligence (AI).
For instance, deep reinforcement learning has enabled computers to learn how to play games
such as Chess [
In this paper, we propose TotalBotWar, a new pseudo-real-time challenge for game AI. The game is inspired by the real-time battles of the popular TotalWar game series1, where two players control respective armies with the objective of defeating each other. On each turn, the agent must decide where the unit must move to. When two opposite units collide, they will start to fight. The result of the combat depends on the type of units and their attributes. If during a turn a unit does not receive any order, it will continue its movement following the CEUR Workshop Proceedings previous one, or it will stand still if none was given. This introduces unknown information on the state: it is possible to know that an enemy unit is moving, but not its destination. The game has a high number of possible actions in a turn (≈ 6.77 ) and also a huge number of possible states (≈ 3.329 ), which provides a significant challenge for AI agents. The game, implemented using the CodinGame SDK2, has already been made available online at this platform3.
An initial set of experiments is also presented, where five diferent agents are benchmarked
to give a baseline to future researchers. Three of them are primary agents where a) units never
move (but can fight), b) always move forward, or c) move to a random localisation. The two
remaining are more sophisticated. The first one applies human knowledge by using a heuristic
function. The last one implements the Online Evolutionary planning (OEP) algorithm proposed
by Justesen et al. [
Summarising, this paper presents three main novelties: • We present TotalBotWar a new pseudo-real-time multi-action challenge for game AI. • As far as we know, this is the first work implementing real-time TotalWar -style battles as a game AI challenge. • We assess the performance of five agents, including the Online Evolutionary Planning algorithm, in the proposed game AI challenge.
The rest of the paper has been organised as follows. Section 2 presents the main characteristics of the game including how agents interact with the game engine. A set of baseline agents and some preliminary experiments are shown in Sections 3 and 4. Finally, the most important conclusions drawn from this work are summarised in Section 5.
TotalBotWar is a 1 vs 1, pseudo-real-time, multi-action game, partially inspired by the real-time battles of the Total War games series. In our game, both players start with the same number of military units and their objective is to defeat the other player. The winner is the player who ifrst destroys all the opponent’s units or the one with more units alive on the battlefield when the maximum number of turns is reached, which is set to 400. There are four diferent types of units: Swordsmen, Spearmen, Archers and Knights (see Figure 1). The game uses a classical rock-paper-scissors combat scheme, where swordsmen are good against spearmen, spearmen are good versus knights and finally, knights are good against swordsmen. Archers are an exception: they can attack from a distance but are very weak in face-to-face combat.
Each unit has an attribute vector modelling its behaviour. The attributes are Health Points, Attacking Strength, Defence, Charge Power, Charge Resistance, Moving Speed and defence against Arrows. Besides, archers also have Throwing Distance and Arrow Damage. Table 1 shows the values assigned to each attribute for each unit type. 2https://www.codingame.com/ 3https://www.codingame.com/contribute/view/486222077fe22e3aa6bcdc0f729dd46223bb
Units can move to any place on the battlefield. Two units from the same agent can’t overlap, and they will fight if they belong to diferent armies. If a unit reaches the limits of the battlefield, it stops. Archers always shoot arrows at enemy troops into the attacking range. Troops sufer friendly fire if they are close to an opponent unit receiving arrows.
The game has three diferent leagues or levels. When using the CodinGame platform, the player has to first implement a bot to defeat the system bot of the first league. After, he/she must implement a new one to defeat the system bot of the second league before passing to the third one. In the third league, the CodingGame platform allows testing the player’s bot versus the bots implemented by many other players. Alternatively, the three leagues can be used isolated from CodingGame platform to test AI algorithms and to rule AI competitions.
In the first league (see Figure 2), the army of each player consists of just one unit of each type and units start in predefined initial locations on the battlefield. The second league introduces the draft phase (see Figure 3) where, in the first 9 turns of the game, the agents must select how many units of each type will be part of their armies and their initial positions on the battlefield. Therefore, the total number of units is 9. There is no restriction on the number of units of each type, i.e. the army can be composed of 9 archers if this is the decision of the agent. On each turn, players select units simultaneously, knowing only the units selected by both players in all previous turns. In the third league (see Figure 4), the army is composed of 30 units, therefore the draft lasts until the 30ℎ turn of the game. Additionally, the third league introduces the General unit, which is a highly important unit that can be crucial in the game. All units within a distance of 150 pixels from their general increase (multiplicatively) their attributes by 1.25. However, if the general is dead, all units decrease their attributes by 0.75. The general is always the first unit selected in the draft and can be of any type.
The size of the battlefield is 1920 × 1080 pixels. The battlefield is completely flat. The size of the units is 150 × 150 pixels in leagues 1 and 2, and 75 × 75 pixels in the third league.
The main characteristics of game AI are as follows: • It is a 1 vs 1 game. • It is (pseudo) real-time. Although the game engine performs actions in the order indicated in the turn, the efect of this order is practically negligible. Similarly, the efect of which player performs the actions is minimal.
• Not all information is known in the state. The state contains information about the actual position of the enemy units and if they are moving or not, but it does not provide information about the final target where they are moving. • It is multi-action since in the same turn more than one action can be performed for each diferent unit owned on the battlefield. • The agents have just 200ms to decide the actions to be executed on each turn. This is a restriction of the core of CodingGame engine. • It has a very large number of possible actions in a turn (≈ 6.77 ) and possible states (≈ 3.329 ).
This work has two principal motivations. On the one hand, the game has been included in CodinGame platform to be used as a tool to learn programming skills fascinatingly. CodinGame platform allows the use of many programming languages, and it is possible to see the efect of the source code used for the agent developed. That can help beginners to learn programming languages faster than through a more traditional style of teaching. The first league of the game is perfect for this purpose. On the other hand, the second and third leagues are dedicated to the development of new game AI agents due to their complexity.
The development of a new game AI challenge using the CodinGame SDK has three main advantages with respect to completely developing it from zero: 1. Developers can take advantage of the framework which contains useful code that can make it easier to develop a new game. 2. Users can program their agents in their preferred programming language instead of being restricted, as usually happens, to use just the one used to develop the game. 3. Sometimes starting with a new game AI challenge is hard since, for instance, users do not always have installed the correct libraries to run the game. The use of CodinGame platform avoids this kind of problem.
However, some constraints must be accomplished as the maximum size of the battlefield, the amount of thinking time per turn for the agents and the maximum number of turns, among others.
The game is inspired by Total War games since they are very popular with the general public.
Similarly to other popular games AI challenges such as StarCraft AI competition [
On each turn, the current player can provide an action for each one of their units. An action consists of moving a unit a particular number of pixels in both and directions and the movement normally takes several turns to be completed. If in a turn the player does not indicate an action for a particular unit, it continues the movement following the previous action performed on this unit.
An action has the following format: “ID ” where: • ID is the unique ID of the unit. • is the number of pixels we want to move the troop on the X axis.
• is the number of pixels we want to move the troop on the Y axis.
Note that and are not the global coordinates to move the unit to, but how many pixels the unit must move with respect to its current coordinates. The coordinates are relative to the unit location to be independent to the position of the agent (up or down) in the battlefield.
For instance, some actions that can be played are: • “1 100 50”: Unit with ID 1 will move 100 pixels to its right (east in the display if the agent plays in the bottom part of the battlefield or west, otherwise) and 50 to the front of the battlefield (upwards or north in the display if the agent plays in the bottom part of the battlefield or downwards or south, otherwise). • “3 -100 -10”: Unit with ID 3 will move 100 pixels to its left and 10 pixels backward. • “5 0 0”: Unit with ID 5 will stop.
On each turn, a player can perform more than one action using a string separated by semicolons. For instance, to perform the three previously described actions in the same turn, the player would provide the following multi-action string: “1 100 50; 3 -100 -10; 5 0 0”.
The system provides information about the player and the opponent’s units. First, the game indicates the total number of units for each player’s army. Then, the system provides the following information for each one of the player and opponent’s units: • ID: Unique ID of the unit. • Location: , coordinates indicating the actual position of the unit on the battlefield. • Direction: a number indicating where the unit is looking for. It can be Northwest (0),
North (1), Northeast (2), East (3), Southeast (4), South (5), Southwest (6) and West (7). • Life: amount of health points (See Table 1). The unit is dead when its life reaches 0. • Type: unit type for swordsmen (0), spearmen (1), cavalry (2) and archers (3). • Moving: Indicates if the unit is moving (1) or not (0). • Target: , coordinates indicating where the unit is going to stop, only for friendly units (for opponent units, no target information is provided).
Therefore, the state has 1 + 9 + 7 elements, where is the number of units for each player’s army.
The number of possible actions that can be played on each turn is huge in both the draft and battle phases, due to the large battlefield size ( 1920 × 1080). It also depends on the league, 1 to 3, selected. One possibility to handle its complexity is to artificially reduce the places where the units can be moved. According to the size of the units, we suggest defining two grids, the ifrst one of 13 × 7 (1920/150 ≈ 13, 1080/150 ≈ 7) and the second one of 26 × 14 (1920/75 ≈ 26, 1080/75 ≈ 14). Note that the units can always be moved to any place on the battlefield. The use of the grid is just for reducing the complexity of the game and it must be handled by agent programmers. It is suggested to be used in the first stages of the implementation of the agent, or for beginners.
Tables 2 and 3 show the number of actions in both phases with respect to the size of the battlefield in the three sizes proposed: 1920 × 1080, 26 × 14 and 13 × 7. The number of actions in the draft phase depends on the size of the battlefield ( , ) and the existing number of unit types ( = 4 ). This number can be calculated as: × × (1)
For instance, when the smallest grid is used ( = 13 , = 7 ), the number of actions in the draft phase is 13 ∗ 7 ∗ 4 = 3.62 .
The number of actions in the battle phase depends on the size of the battlefield ( , ) and the number of units in each league ( ). is 4, 9 and 30 in leagues 1, 2 and 3, respectively. The number of actions can be calculated as: ( × × × × × × × ) × × ( × × × × × ) ×
× × × × ×
For instance, when the smallest grid is used ( = 13 , = 7 ) and for the third league ( = 30 ), the number of actions in the battle phase is 13 ∗ 7 ∗ 30 = 2.73 .
Table 4 shows the number of existing army combinations that can be obtained in each league. This number depends on the size of the battlefield ( , ), the existing number of unit types ( = 4 ), and the number of units in each league ( ). In this case, the formula is:
For instance, when the complete battlefield is used ( = 1920 , = 1080 ) and for the second league ( = 9 ), the number of actions in the battle phase is 1920 ∗ 1080 ∗ 4 ∗ 9 = 7.57 . Note that in the first league, there is just one possible combination since there is no draft phase and the initial configuration of the army is always the same.
Table 5 shows the number of states for the three leagues and proposed battlefield sizes. This number depends on the size of the battlefield ( , ), the number of diferent directions ( = 8 ), the number of health points ( ) (for simplicity, we assume in these calculations that all units have the same number of health points = 100 , see Table 1), the existing number of unit types ( = 4 ), if the unit is moving or not ( = 2 ), and the number of units on each league ( ). The number of states can be calculated as: (2) (3) (4)
Note that the second term, corresponding to the opponent units, does not have a second element × since the final target of the opponent units is unknown. For instance, when the complete battlefield is used ( = 1920 , = 1080 ) and for the first league ( = 4 ), the number of actions in the battle phase is (1920 × 1080 × 8 × 100 × 4 × 2 × 1920 × 1080) × 4 × (1920 × 1080 × 8 × 100 × 4 × 2) × 4 = 5.827 .
One of the more interesting features of CodinGame is that it is possible to replay the game. Therefore, it is possible to study how some actions have afected the game at a particular moment of the game. A set of assets has been designed for a better representation of the game. The units are based on the middle age and have a cartoon style (see Figure 1). The game also includes animations for each state in which each unit can be found. The states are: idle, running, attacking and dead. Furthermore, there is an animation for when a unit is under an arrow attack. As an example, Figure 5 shows the diferent sprites of some of the animations used in the game.
Several AI agents have been developed as baselines for the proposed game. They are briefly explained as follows:
• StayStatic ( ): All units stand still during the battle. A predefined army is always selected in the draft. The knights are in the flanks, spearmen and swordsmen in the middle and archers behind. The units never move but they can fight when colliding with an opponent.
Besides, the archers can shoot arrows at opponent troops into the attacking range. • AlwaysForward ( ): All units always move forward. The predefined army is the same as in . • Random ( ): All units select random destinations. The predefined army is the same as in . 3.2. Heuristic (Λ) It uses human knowledge in both phases. In the draft, the agent tries to pick the unit to have an advantage against the opponent. For instance, if the opponent selected in the previous turn a Knight, it will pick a Spearman. The agent has some rules to avoid choosing too many units of the same type. The agent selects the position in front of the troop that can be defeated by the selected one.
For the battle, for each unit, a heuristic function is used to estimate a value indicating how good is to attack each enemy unit. The enemy unit with the biggest value is the one selected as the target. The heuristic function has been designed as the average of 5 factors ( ∈ [0, … , 1], ∀ and ∈ [1, … , 5] ) as follows: = 1 + 2 + 3 + 4 + 5 5 (5) where: • 1 provides higher values if the player’s unit belongs to a type with an advantage with respect to the opponent one, taking into account the rock-paper-scissor combat system. It can be 1.0, 0.5 or 0.0 when the opponent unit type is worse, the same or better, respectively. • 2 is 1.0 if the player’s unit avoids getting into the opponent archer attacking range; 0.0 otherwise. • 3 benefits from having more health points than the opponent unit. It can be 1.0, 0.5 or 0.0 when the opponent unit has less, the same or more health points, respectively. • 4 is 1.0 in case of a flank attack, i.e. the attacking direction is not frontal, and 0.0 otherwise.
• 5 will be higher the closer the player’s unit is to the enemy’s.
In the third league, a new factor 6 is added with a value of 1.0 if the opponent unit is a general and 0.0 otherwise.
This algorithm, proposed by Justesen et al. [
The OEP agent implemented for TotalBotWar uses the method described for the Heuristic
agent for the draft phase and for seeding the initial population in the battle. Each individual
contains genes, where each gene corresponds to a unit owned by the agent and their values
are the IDs of the opponent’s unit to attack, i.e. the number of genes is the number of units
of the army in each league. For instance, in the first league ( = 4 ), a genome [
Several games have been played using the agents developed and described in Section 3. These experiments have been performed outside the CodinGame platform. Table 6 shows the win rate of the agents playing as the first player. In all cases, results are reported using the third league, the complete battlefield size (i.e. no grid is used) and 100 games. As expected, Λ and OEP agents overcome the simplest baselines. Surprisingly there is a tie between Λ and OEP agents. This is likely due to the OEP agent not having enough time (with the time limit constrain of CodingGame platform) to evolve stronger action selections, being unable to find better recommendations than the ones initially provided by the Λ agent.
We have also tested both algorithms (OEP vs Λ) in leagues 2 and 1, obtaining a win rate for the OEP of 0.65 and 0.90, respectively. In these cases, the game is less complex than in the case of the league 3. Therefore, the OEP agent performs more iterations in the allowed budget time and it is able to find better move recommendations.
This paper presents a new game for game AI: TotalBotWar. The game introduces interesting features and challenges for AI, as it presents a pseudo-real-time decision-making problem in a large and continuous state and action space. It also provides an interesting challenge for drafting policies, army composition and tactical planning. The paper suggests diferent possibilities for discretizing the state space and also benchmarks a state-of-the-art algorithm, Online Evolutionary Planning (OEP), which shows good results in the simpler scenarios but can’t outperform domain-knowledge rule-based agents in the complex ones due to the time limitation constrain in CodingGame platform.
Future work can span in multiple directions. Regarding the game, more complex units, rules and terrain features could be added. In terms of agents, sophisticated agents and techniques can be objects of research to outperform the proposed baselines. Finally, we plan to propose this benchmark as a new competition in the future for game-playing AI research.