Strategic Pattern Discovery in RTS-games for E-Sport with Sequential Pattern Mining

Abstract

Electronic sport, or e-sport, denotes the extreme practice of video games where so-called cyber-athletes compete in world-wide tournaments. As for any sport, such professionals are surrounded by sponsors and practice within professional teams. These professional games are even broadcast by commentators over specialized TV channels. StarCraft II (Blizzard Entertainment) is one of the most competitive video game and has now its own world-wide players ranking system (based on the well-known ELO system) and annual world cup competition series (WCS) with a US$1.6 millions prize pool for the year 2013. Each match between two opponents can be recorded as an exhaustive and noisy sequence of actions. Analyzing these sequences yields an important outcome for game strategy prediction and in-depth game understanding. In this work we report a preliminary study on StarCraft II professional players strategies' discovery based on sequential pattern mining.

Introduction

The progressing digitization of modern societies is witnessed in many domains and has irrevocably impacted all aspects of human behaviors. In the wake of these radical digital and social evolutions, the entertainment industry underwent quick mutations to bene t from the widely spread usage of electronic devices (computers, play stations, smart-phones, etc.). Over the last decade, the video games industry became one of the most lucrative business ever developed with many examples such as Ubisoft budgeting 40 millions euros to develop \Assassin's creed III" or \Angry birds" (Rovio Entertainment) accounting more than 600 millions players. The competition spirit appeared in the early beginnings of video game developments [ 9 ]. Naturally, some people easily acquire (gaming) skills among which agility (mouse, keyboard interaction), creativity, reactivity and strategical thinking. To push those skills to the highest, they perform an intense daily practice. These individuals compete in international tournaments, highlighting impressive skills that non daily practitioners cannot achieve. This new notion of competition over electronic games and its associated community is coined by the terms \electronic-sport" or \e-sport" and started in South Korea 15 years ago before spreading to Europe and North America over the last three years [ 9 ]. Esport in general is composed of professionals, amateurs, teams, championships, commentators and sponsors. The main di erence with classical sport is the usage of an electronic device as a support to compete1. This new sport paradigm is poised to challenge classical sports in term of audience in the next decade [ 5 ]2. E-sport should attract many industrial actors and researchers in the next few years around video game analytics, or e-sport analytics, just like with traditional sports (as discussed during 2012 MIT Sloan Sport Conference, eSports: The Future Of Competition). In this context of digital competition, both sport analytic and arti cial intelligence should meet to answer several problems. Indeed, as digital, e-sports easily leave (interaction) traces in great numbers on the Web (e.g. records of the games, statistics of the matches, spectators opinions and audience on social TV [ 5 ]). Match records, also called replays, contain all actions generated by the players. Strategies are hidden in such replays and eliciting them automatically can help for modeling players behaviors, predicting strategies, improving online match-making systems, commercial targeting, etc [ 2, 4, 8, 3, 12, 10, 11, 7 ]. In this article, we propose an approach for automatically discovering strategic patterns for one of the most competitive real-time strategy game (RTS), StarCraft II (Blizzard Entertainment, 2010). We investigate how sequential pattern mining [ 6, 13 ] can help strategy discovery from replays. In our case, a pattern represents a frequent series of game actions done by the two players extracted from a large database of games. We introduce a measure that describes the balance of a pattern (or a strategy), i.e. in which proportion it leads to victory. In pattern mining, this can be formalized with so-called emerging patterns when each object of the dataset is labeled either positively or negatively [ 1 ]. We deal with a hard problem, not studied in the literature, to the best of our knowledge, where the class information (either positive, or negative, for a win or a loss) needs to be encoded within the object description to discriminate both players, in contrast with a single class value for each sequence in the classical settings. We also show with an empirical evaluation that the patterns (and their balance measure) we found in about 100,000 replays can be helpful for game balance study, players modeling and winning prediction.

The rest of the paper is as follows. The principles of the StarCraft II game are presented in Section 2. Section 3 recalls notions on sequential pattern mining. We then present the problem of mining strategic patterns and our methodology in Section 4. We present experiments in Section 5 before concluding. 2

Principle of the RTS video game StarCraft II

A game of StarCraft II involves two players. Each player chooses a faction among the \Zergs" (Z), \Protoss" (P) and \Terrans" (T): there are 6 di erent possible 1 \E-sport is one of the most popular trend in video games and is rapidly attracting the core 18-34 male demographic in greater numbers than any other medium or category", said Jim Lanzone, president of CBS Interactive. 2 \Major League Gaming (MLG)'s live audience is growing in popularity to rival some of the most popular traditional sporting events with a highly engaged fan base worldwide", said Sundance DiGiovanni, CEO and Co-Founder of MLG. combinations of matches. Each combination involves di erent strategies. During a game, two players are battling on a map (aerial view), controlling buildings and units to gather resources, train an army with the nal goal of winning by annihilating the opponent's forces3. Such actions (training, building, moving, attacking) are done in real-time, see Figure 1 for an example. Each faction (Z,P,T) allows di erent units and buildings with distinctive weaknesses and strengths following a \rock-paper-scissors" principle. A strategy is hidden in large sequences of actions generated by players called replays. Actually, the name \replay" comes from the fact that it materializes all actions allowing the game engine to replay the game anytime. A high proportion of the actions are noisy: a replay does not store the result of an action, i.e. if/how it changed the state of the game. For example, if a player orders the construction of a building while it does not have the resources to realize it, this action is still stored in the replay but the state of the game has not changed since construction is impossible. Moreover, players are constrained by the \fog of war" which reduces the player's eld of vision of the game by only revealing terrain features and enemy units only if they pass near a player unit or building. Areas of the map which are out of the eld of vision are subject to a shroud through which only terrain is visible, but not changes in enemy units or bases. Then, scouting the enemy's base is an action a player often needs to perform to adapt his strategy according to the opposing player's one. The fog of war is very similar to the \imperfect information" notion that appears in game theory or economics.

Preliminaries on sequential pattern mining

We use the notations from [ 13 ]. Let I be a set of items. An itemset is a subset of I. A sequence s is a non empty ordered list of itemsets: s = ht1t2:::tni where tj I for all j 2 f1; :::; ng. A sequence = ha1a2:::ani is a sub-sequence of an other sequence = hb1b2:::bmi, denoted as v , if and only if 9j1; j2; :::; jn such as 1 j1 < j2 < ::: < jn m and a1 bj1 ; a2 bj ; :::; an bjn . Given a sequence database D = fs1; s2; :::; sng, the support of an arbitrary sequence (over I) is de ned as the id of sequences in D which contain : supportD( ) = fs j v s; s 2 Dg. A sequence is said frequent in a database D if, given a minimum support threshold minSupp, jsupportD( )j minSupp. Pre xspan is one of the reference algorithm for extracting frequent sequential patterns [ 13 ].

3 http://en.wikipedia.org/wiki/Real-time strategy

Example 1. The table on the right presents a sequence id description database where the set of items is I = fa; b; c; d; e; f g. s10 hafabcgfacgdfcfgi The sequence = habci is a sub-sequence of the sequences ss3200 hhffeafdggfcfabbcggffdafeggcibi s10 and s40 but is not a sub-sequence of sequences s20 s40 hegfafgcbci and s30. Moreover, the sequence = hfdf gci is only a subsequence of the sequence s30. With minSupp = 3, the sequence = hacci is a frequent sequential pattern because supportD( ) = fs10; s20; s40g. However the sequence = hafbcgai is not frequent since jsupportD( )j = 2 < minSupp. 4

Mining strategic patterns as sequential patterns

Given a corpus of StarCraft II replays, we consider the problem of extracting strategic patterns and evaluating (i.e., measure) how they lead or not to victory. To answer this problem, our methodology is decomposed as follows: (i) we encode a set of replays as a database of sequences, (ii) we mine frequent sequential patterns then (iii) we post-process these patterns to characterize and contrast their ability to discriminate winning from losing strategies.

We abstract the notion of replay in a way that is similar to AI planning : it represents two agents trying to succeed by performing inter-dependent actions in time. At the end of their interaction, one will fail, the other will succeed. The fact that actions of one agent are dependent of the other agent's actions (e.g. knowing the last action of the opponent can change one's strategy) leads us to derive one sequence with both agent actions instead of one sequence per agent. To be able to characterize if an agent is successful, we include success or failure in the sequence alphabet: an action a can appear in a sequence as leading to success, or to failure as nal outcome. Hence, our sequence alphabet is composed of elements of the Cartesian product Actions fsuccess; f ailg. Furthermore, the window of time at which a typical action happens is very important in our application in competitive gaming. Hence, we de ne the alphabet I of our sequences as follow.

De nition 1 (Alphabet). Consider a set of actions A available for both agents and a set of timestamps T , i.e. windows of time in which the actions occur. The alphabet of the sequences is de ned as I = A T R where R = fsuccess; f ailg. Example 2. Thanks to this alphabet, we translate the replay sample given in Figure 1 as follows. We consider windows of 30 seconds and the player LiquidHuK is the winner of this game. We obtain s = hf(P ylon; 2; success)g f(Gateway; 3; success); (Spawning; 3; f ail)g f(Assimilator; 4; success); (P ylon; 4; success)g f(Hatchery; 5; f ail); (Cybernetics Core; 5; success); (Assimilator; 5; success)gi where the item (Hatchery; 5; f ail) means that the player who lost performed an action to build a Hatchery during the fth window of time.

Consider now a database of sequences encoded from raw replays thanks to the alphabet presented above: a frequent sequential pattern gives us frequent series actions made by the agents, one of the agent will succeed at the end, the other will fail. In order to assess the interestingness of the resulting patterns, we de ne a measure that re ects the success or failure ability of the strategy materialized by the pattern. We de ne the notion of dual sequence rst.

De nition 2 (Dual sequence). The dual of a sequence s composed of items (a; t; r) 2 I is the same sequence where any (a; t; r) 2 I is replaced by (a; t; Rnr). Consider a pattern s = h(a; 3; f ail); (b; 4; success)i which means that an agent succeeds by doing the action b after the other agent did the action a. On the other hand, we have s~ = h(a; 3; success); (b; 4; f ail)i which characterizes the opposite scenario: both agents do respectively the same actions, but the agent succeeding is not the same. The balance measure we introduce now represents the potential of strategy s to lead to success or failure.

De nition 3 (Balance measure). Consider a frequent sequential pattern s and its dual s~. We de ne the balance measure of s as: balance(s) =

jsupportD(s)j jsupportD(s)j + jsupportD(s~)j The measure balance returns values in [0; 1]. When balance(s) = 0:5 the sequence s does not discriminate either success of fail: it is a balanced strategy. When balance(s) = 1 (resp. 0), the sequential pattern s is found only winning or losing. Note that balance(s) + balance(s~) = 1. Note also that symmetric patterns, i.e. sequences where each itemset that contains the item (a; t; r) 2 I also contains the dual item (a; t; Rnr) 2 I, have a balance of 0:5 by de nition. Algorithm. To compute the set of frequent patterns given a minimal support we use the Pre xSpan algorithm [ 6 ]. To compute the balance measure in postprocessing, it is naively enough to compute supportD(s~) for each frequent pattern s by scanning the database D. This approach is however not scalable as shown in the experiments. To allow a much faster computation of the balance measure, we proceed as follow. We build an auxiliary data structure when producing the sequences dataset from the rough replays, storing for each item i 2 I the identi ers of the sequences that contain it, i.e. D I in the worst case. Then, when building the tree-structure storing the patterns in Pre xSpan, we maintain for each pattern (node) the support of the dual sequence: at the rst step by looking up in our data structure, in the other steps by intersecting the support of the dual item expanding the tree with the support of the previous dual sequence. As such, one can notice that there are redundant patterns: it happens that both s and s~ are extracted and their balance computed. To avoid redundant patterns, we traverse the pattern tree in a dual way: when visiting a node, we visit also the dual node and ag it as already processed (not output). Note that from an extracted pattern the support of the dual can be retrieved as follows : jsupportD(s~)j = jsupportD(s)j

bbaallaannccee((ss~)) .

Experimental study

We experiment our approach to assess its computational feasibility and, helped by a high level StarCraft II player, the quality of the extracted patterns. These patterns are useful for several tasks among which on-the- y win prediction and game balance study. The source code and the data sets are available4. Raw replays collection. StarCraft II replays are easily accessible on the Web; several websites o er high level replays in great numbers5 from which we extracted 371; 267 replays. We are interested in very high level players, since casual (by opposition to professional) players are not able to follow speci c strategies. We kept replays involving at least one player in the highest o cial league (Grandmaster) or playing in average more than 200 actions per minutes (APM). At the end, our dataset involves 90; 678 games between 30; 678 players that played an overall total of 3:19 years in game time. The average length of a game is about 20 minutes. Figure 2 (left) gives the average APM during each minute of a game. Figure 2 (right) gives the repartition of the di erent kinds of actions done in average in time.

APM 600 500 400 300 200 100 0

Average + Standard deviation

Average

Average - Standard deviation 0 5

Deriving sequences datasets from raw replays. In StarCraft II, they are di erent types of games, called match-up, depending on the pair of factions involved, e.g. PvZ describes a game involving a Protoss player against a Zerg player. Each match-up is characterized by di erent strategies: a Protoss does not play the same strategies when facing either another Protoss player or a Zerg opponent, or even a Terran opponent. As such, we divided the 90; 678 replays into six di erent sequence datasets, one for every match ups (since there are 3 factions). Buildings are one of the key elements of a strategy [ 11 ], since they allow di erent kinds of units production: From each replay, we derive a sequence where the items represent the buildings the players chose to produce in real time, and itemsets denote windows of time w (See Example 2). We set w to 30

4 http://liris.cnrs.fr/mehdi.kaytoue/sc2

5 http://wiki.teamliquid.net/starcraft2/Replay Websites

PvP PvT PvZ TvT TvZ ZvZ

Build Build + Upgrade + Occurrence Item Seq. IS I/IS Item Seq. IS I/IS 1,160 6,668 11.5 2.0 5,624 6,668 11.6 2.1 3,655 18,754 19.0 2.6 24,019 18,754 19.4 2.7 3,748 22,784 19.6 2.7 27,913 22,784 20.1 2.8 2,201 7,457 20.7 2.8 15,572 7,457 21.2 2.9 4,492 23,637 22.5 2.8 33,050 23,637 23.1 2.9 1,689 9,554 14.2 2.2 9,139 9,554 14.7 2.3

Table 1. Sequence database characteristics. seconds since the ordering of two items over this period of time is insigni cant in terms of strategy according to the expert. For the Protoss and Terran factions, we ignored the buildings Pylon and Supply depot that are insigni cant in terms of strategy as well (expert opinion). Table 1 gives the main characteristics of these datasets. We also build a second series of datasets, which includes research upgrades (important elements of strategy) and the number of times the same action appeared, which can better help characterize strategies according to the expert, i.e. I is de ned on A T R N. We wrote a plugin for the Sc2Gears software6 to parse the binary replay les. This plugin is freely available7. Quantitative experiments. We experimented on 2.5GHz and 4GB of RAM machines. We slightly modi ed the original C++ algorithm Pre xSpan for extracting frequent sequential patterns [ 6 ]. Figure 4 (left) shows for the data set PvP that our optimization for computing balance speeds up the computation of the balance measure by several orders of magnitude, comparing to the naive approach, making our approach usable for large datasets. It also shows that the time needed to avoid redundancy is negligible, and that the amount of redundant patterns tend to decrease with the minimal support (yet not monotone). Figure 4 (middle) shows that the naive post-processing makes the approach not scalable. In contrast, our optimization allows low minimal supports (Figure 4 (right)). Finally, Figure 3 plots extracted patterns (including redundant) according to their balance measure and support, for the PvP dataset and a minimum support set to 0:1%. Typically, jumping patterns can be observed with a measure equals to 1. Note the \symmetric" aspect of the points cloud, where y = 0:5 is the line representing balanced patterns. Note also that empirically, the more frequent the patterns, the closest to 0.5 the balance measure.

Discovering openings. Like in chess game, StarCraft II openings are generally well-known and codi ed (given a name). We observe that well-known openings in StarCraft II are translated to the most frequent patterns. Starting with the dataset TvZ, we extract frequent sequential patterns with minimum support of 10%. As expected most of the 64 patterns returned are very balanced with values close to 0:5. Openings are per se balanced (otherwise the game would not be attractive for the numerous spectators). We ltered the 64 patterns to retain only the 20 that involve actions of both players. We remark for example

6 https://sites.google.com/site/sc2gears/

7 http://liris.cnrs.fr/mehdi.kaytoue/sc2/MLSA-PKDD13/Sc2Gears4DM.zip

Fig. 4. Execution times and pattern counts hf(1; W; Barracks)g; f(4; L; Hatchery); (4; L; Sp:P ool)gi (and its dual) which is the well-known opening called Fast-expand in the TvZ match-up. Pattern s = h(1; L; Barracks)(4; W; Hatchery)(6; L; ReactorBarracks)(7; L; CCenter)i with balance(s) = 0:5 denotes also a classical opening of game that leads to a so-called macro-game, where players focus on building a strong economy before attacking the opponent. In the PvZ dataset, 49 patterns are extracted with a minimum support set to 10%, and only one pattern involves the two players: the forgeexpand strategy for the Protoss player (with a balance of 0:506), which is one of the most typical opening in that match-up (supported by 37% of the games). Discovering unbalanced strategies. Balance is the most important aspect in e-sport: in a match, all opponents should have exactly the same chances to win given the initial conditions. In StarCraft II for example, a faction should not been advantaged compared to another. When a balancing problem is detected, either by the game developers or by the players themselves, the game properties are adjusted to correct this balance issue. We asked the expert for a balance example problem in StarCraft II. The Bunker-rush in the TvZ matchups consists in building, very rapidly, a defensive structure in the opponent's base which causes strong if not fatal damages when not detected in time. We used the second encoding (involving the number of times the bunker has been built until its last appearance) to check that hypothesis and query among the 17; 990 patterns (minSupp = 1%). Only 359 patterns involves the Bunker building and both players. The top 50 patterns according to balance have all a support cardinality higher than 300 and a balance measure higher than 0:61. For example, the 5th pattern clearly indicates that the bunker rush is an imbalanced strategy, even if it clearly appears that the opponent tried to defend against it: hf(1; W; Barracks; 1)g; f(4; L; SpP ool; 1)g; f(6; W; Bunker; 1); (6; L; SpCrawler; 1)gi. This balance issue has been corrected in a patch of the game by changing some properties of the game element. 6

Conclusion

We presented a preliminary work for extracting strategic patterns from replays of the RTS video game \StarCraft II". We introduce a three-steps approach: (i) encode player actions into sequences, (ii) mine sequential patterns, and (iii) compute the balance of each resulting strategy. Through a series of experiments, we show that it is feasible to extract interesting patterns for many e-sport tasks in reasonable time. We believe that the methodology and the results are interesting to help RTS video games editor to balance their games, but also for the players/team managers for analyzing the strategies of a given player over a selected set of replays (e.g. to prepare a competition). Among others, our perspectives of further research include to study the use of other types of discretization techniques for building the sequences, or the use of other types of sequential patterns such as episodes. We will investigate how to achieve on-the- y game result prediction, which consists in using the extracted patterns for predicting the game outcome in real-time. Also, build orders consist in less than 1% of the information contained in a replay. We plan to use other types of actions, and this interestingly comes with problems of pattern mining in noisy data and game state estimation (remembering that the state of the game is never stored in a replay).

References

[1] K. Deng and O. R. Za ane. An occurrence based approach to mine emerging sequences . In T. B. Pedersen , M. K. Mohania , and A. M. Tjoa, editors, DaWak , volume 6263 of Lecture Notes in Computer Science, pages 275 { 284 . Springer, 2010 .

[2] E. Dereszynski , J. Hostetler , A. Fern , T. Dietterich , T.-T. Hoang, and M. Udarbe . Learning probabilistic behavior models in real-time strategy games . In Seventh Arti cial Intelligence and Interactive Digital Entertainment Conference , 2011 .

[3] J. Hostetler , E. W. Dereszynski , T. G. Dietterich , and A. Fern . Inferring strategies from limited reconnaissance in real-time strategy games . CoRR, abs/1210.4880 , 2012 .

[4] J.-L. Hsieh and C.-T. Sun. Building a player strategy model by analyzing replays of real-time strategy games . In Proceedings of the International Joint Conference on Neural Networks, part of the IEEE World Congress on Computational Intelligence , Hong Kong, China, June 1-6, 2008 , pages 3106 { 3111 . IEEE, 2008 .

[5] M. Kaytoue , A. Silva , L. Cerf , W. Meira Jr. , and C. Rassi. Watch me playing, i am a professional: a rst study on video game live streaming . pages 1181{1188 , 2012 .

[6] J. Pei , J. Han, B. Mortazavi-Asl , H. Pinto , Q. Chen , U. Dayal , and M. Hsu . Pre xspan: Mining sequential patterns by pre x-projected growth . In D. Georgakopoulos and A . Buchmann, editors, ICDE , pages 215 { 224 . IEEE Computer Society, 2001 .

[7] G. Synnaeve and P. Bessiere . A Bayesian Model for Opening Prediction in RTS Games with Application to StarCraft . In Proceedings of 2011 IEEE CIG , 2011 .

[8] G. Synnaeve and P. Bessiere . A dataset for starcraft ai & an example of armies clustering . CoRR, abs/1211.4552 , 2012 .

[9] T. L. Taylor . Raising the Stakes: E-Sports and the Professionalization of Computer Gaming . MIT Press, 2012 .

[10] B. G. Weber and M. Mateas . Case-based reasoning for build order in real-time strategy games . In C. Darken and G. M. Youngblood, editors, Proceedings of the Fifth Arti cial Intelligence and Interactive Digital Entertainment Conference, October 14-16 , Stanford, California, USA. The AAAI Press, 2009 .

[11] B. G. Weber and M. Mateas . A data mining approach to strategy prediction . In P. L. Lanzi, editor, Proceedings of the 2009 IEEE Symposium on Computational Intelligence and Games , pages 140 { 147 . IEEE, 2009 .

[12] S. Wender , A. Cordier , and I. Watson . Building a trace-based system for real-time strategy game traces . In Proceedings of EXPPORT: EXperience reuse: Provenance , Process-ORientation and Traces , ICCBR 2013 , Saratoga Springs , NY , USA, 8 - 11 July 2013 ., 2013 .

[13] X. Yan , J. Han, and R. Afshar . Clospan: Mining closed sequential patterns in large databases . In D. Barbara and C. Kamath, editors, SDM. SIAM , 2003 .