INTRODUCTION

The e-sports industry has grown significantly in the past decade. Exemplified with one of the most successful games, League of Legends (LoL), in 2012, the number of concurrent online viewers in an LoL championship for a single event exceeded one million. 1 From 2016 to 2019, the peak viewer counts rose from 28.26 (2016) to 106.27 (2017) to 205.11 million (2018) viewers. 2 The video game streaming industry competes with traditional sports events for top viewership counts. 3 E-sports has been compared to traditional sports a lot. Like in traditional sports, leisure activities, like playing soccer for fun, may lead to professional training and organized competitions for athletes. In addition, Freeman and Wohn [5] posit that e-sports is defined by the spectatorship and the governing bodies like the ESL Gaming Network. 4 Hamilton et al. [6] found that streaming games focuses on social engagement and community building, which contrasts traditional sports, where the focus lies on the highest levels of play. E-sports appears to be in-between game streaming and traditional sports. Though it depends on a participatory community, especially streams with a large amount of viewers struggle to maintain meaningful social engagement. Seo and Jung [11] see e-sports at the heart of consumer communities with consumers also being players interacting with the game beyond the game interface.

In-game streaming and e-sports, a lot of content is created. Besides videos, platforms like Twitch.tv or YouTube allow spectators to interact with the players, influencing their gameplay. Altogether, the data streams that can be collected for an individual player include video and audio, commentaries, game data and statistics, interaction traces, viewer-to-viewer communication. The level of detail of the data available, as well as its heterogeneity, render video game streams a challenging subject to multimedia research, allowing for a manifold of research questions.

New research fields like game analytics [2,3] have emerged, investigating the highly interactive and narrative nature of video games [1]. Relating to summarization, while a lot of work has been done on videos and multi-modal summarization [7,10,12,16], video games have hardly been investigated so far. At MediaEval 2018, GameStory was organized for the first time [8,9]. Here, participants were given the multi-player plus commentary streams for an esports match of Counter-Strike: Global Offensive (CS:GO), 5 a game with a solid population of players and viewers, and the task was to generate an entertaining summary.

At GameStory 2019 we built on top of the 2018 dataset by adding a particular subproblem of e-sports match summarization. Since the data are comprised of video streams from multiple perspectives on a given match, as well as a commentators' stream, we ask participants to identify critical moments by finding replays in the commentators' stream and by aligning them with the source video clip in the players' streams. Then, as a second task, following our 2018 summarization task, participants are asked to create a multi-modal summary that provides a captivating story of the match's progress. The latter task is open-ended in the sense that there is no ground truth, but participants can be creative and identify critical moments themselves, whereas the former task may guide participants in solving the latter.

BACKGROUND AND DATASET

CS:GO is a first-person shooter (FPS) game and, as an e-sports game, it has very strict rules of-play. Two teams, the terrorists and the counter-terrorists, with five players each, compete in a virtual 3D world, called map. Matches consist of several rounds and players only re-spawn in between rounds. Depending on the success of players and teams, players get awarded virtual money. With that money, players can outfit their avatars with weapons, ammo, tools, and armor in-between rounds. 6Our dataset was recorded at the CS:GO Intel Extreme Masters (IEM) ESL tournament in Katowice 2018. According to the ESL rules, a typical CS:GO match is decided in a best-of-30 fashion. After 15 rounds, the teams switch sides, i.e., terrorists become counterterrorists and vice versa. If two teams end up with a draw after 30 rounds, the teams play overtime to determine the winner. Strategies typically span over multiple rounds, including different sets of constraints, e.g., teams can afford to buy what they need, teams have to save money, etc. In general, a winning condition is to eliminate all opponents. In a DE_Map round, as we have them in our dataset, the goal for the terrorist team is to plant a bomb at one of some specific locations and protect it from being defused (until the end of the round), while the counter terrorist team has to defuse the bomb. In these maps, a bomb going off or the prevention of that event is an additional winning condition.

The data consists of twelve video streams along with metadata. Ten files give the view of the players with the in-game audio streams. One file gives the commentator stream, where a professional cutter selected the parts of the player views to be shown as well as videos from the audience, the teams' players, and the commentator provides the spoken content mixed with recordings from the game and audience cheering. The last one shows the map from above with icons indicating the position of players. A metadata file indicates the start and end of games and the content in the commentators' stream. JSON files, one for each match in the dataset, capture player activity and events in addition to the raw video. Events range from kills, deaths, starts, and ends of rounds to what the players bought at the beginning of rounds and when a bomb was set or a grenade was thrown and went off. The data covers three days of the tournament and is split into training (two days) and test (one day) sets. A ground truth for where to find replays in the commentators' stream is given for the first day of the training dataset. Additionally, for all the video streams, we provide synchronization data as the actual video is off up to 40 seconds from the time stamps given in the metadata.

TASKS AND EVALUATION

Compared to sports summaries [4,[13][14][15], video games are not focused on a small number of attention points like, for instance, the ball and the two goals on a soccer field. Rather, video games comprise multiple views and concurrent events within a well-defined game world (map), which can change in-between games, but stays the same in one single game. Typically, game statistics only convey active and obvious events, but miss those with semantics on a tactical level, including fake tactics, intentional misses, intentional risk-taking (e.g., a player avatar's death or re-spawn), an the like.

With that in mind, we defined two tasks for GameStory 2019:

(1) Find all replays in the commentators' streams and locate the source clips in the respective player streams. (2) Create a short and captivating summary with a maximum length of five minutes of a single match.

The evaluation of the first task is based on the overlap of the found clips with the ground truth. To determine if a given replay has been successfully found, we employ the Jaccard index in terms of video frames:

J (A, B) = A ∩ B A ∪ B ,

where A denotes the set of consecutive frames identified as a replay clip, and B the set of consecutive frames of the actual replay from the ground truth. We consider a replay to be successfully found if J (A, B) > t for threshold t, using two thresholds t 1 > 0.5 and t 2 > 0.75. Using the Jaccard index, we calculate precision, recall, and the F1 score for the set of replays to be identified. Should a given replay be identified more than once, only the clip with the highest Jaccard index is counted. In a second step, we determine the goodness of the match between true positive replay clips and original player streams. Again for we use the Jaccard index to determine the overlap of the source segment from the ground truth with the segment found in the run. To quantify the degree of overlap for all found replays we average for all found replay segments.

The evaluation of the second task is based on a jury of experts, including CS:GO players and game researchers. The jury members watch the summaries individually and independent of each other with the tasks of summarizing and arguing both strong and weak points of each submission, as well as rating them on a 5-point Likert scale (strongly agree to strongly disagree) concerning the following statements:

(1) The submission gives a summary of the match at hand.

(2) The submission is entertaining.

(3) The submission provides the flow and peak of a good story.

(4) The submission provided an innovative way to present a summary of a CS:GO match. (5) A summary like this submission can be applied to games other than CS:GO.

DISCUSSION AND OUTLOOK

GameStory is at the forefront of research on data-driven analysis of video games and game streaming, a domain that has hardly been addressed within multimedia computer science to date. Here, huge amounts of content and data are generated by millions of players and viewers daily, by both amateurs and professional creators and producers. Their highly interactive nature makes the outcome of games mostly unpredictable. Research in this area is still in its infancy but has the potential for a high social impact. Players and viewers are often young, and games have become an important part of youth culture, sometimes having a strong and long-lasting influence on people's lives. With GameStory, we seek to explore this exciting new direction of research. In the coming years, our goal is to diversify and grow the tasks with a combination of objective, quantitative analysis tasks (e.g., finding kill streaks, synchronizing streams, or identifying relations between consecutive wins and economy in the game streams) and freestyle synthesis tasks, where the analysis technology can be readily employed.