This paper introduces an algorithm for automated replay detection in CS:GO tournaments. Firstly, it uses a content-based approach for replay identification in the commentator stream and then the metadata to synchronize them to the player streams. The provided metadata for events cannot be used for detecting events in the streams, because the timestamps of these events are very inaccurate and difer up to 40 seconds. Furthermore, the algorithm uses a content-based approach along with the synchronization data to detect the equivalent scenes in the matching player stream.
E-sports are competitions which are only held virtually via video
games. These sports are becoming increasingly popular nowadays.
One of these games is Counter-Strike: Global Ofensive (CS:GO),
which is a multiplayer first-person shooter game. The CS:GO World
Championship tournament was placed in Katowice in 2018. Each
game is played between two teams of five players. Such events
produce an enormous amount of multimedia data and metadata.
During the tournament, the multimedia data of each game is
composed of the commentator stream and ten player streams. It is not
possible to process the material manually and therefore Mediaeval
2019 defined one task: GameStory: The 2019 Video Game Analytics
Challenge [
The tournament the GameStory challenge focuses on ran its course over three days and there is video material of each gaming day, which includes the commentator stream and the player streams. Every player stream shows the perspective of a specific player. The commentator stream includes live scenes, replays of interesting events, statistics and much more. This video has a frame rate that difers from the player streams. Therefore, the algorithm adjusts the frame rate to 59 fps. This correction enables an acceptable basis for the following detection. According to detection speed, I resized the width to 450 pixels. In addition, the system converts color frames to gray scale images for further analysis.
A replay contains a specific scene with a pair of logo transitions
and a sweeping of efect at the beginning and the end [
The replay start point detection is always identical. In contrast, there are five variants of the replay end.
The detection of a starting point is a full frame comparison based on the structural similarity index, which OpenCV provides. If the score is higher or equal than the threshold of 0.99, the algorithm found the start logo. The next step in this process is adding a predeifned number to the just found frame number, to get the first frame after the logo transition. After this step, the algorithm starts at the adapted frame number to check four regions of interest (ROIs).
The first ROI is the name of the current shown player. The second one is the small Intel-Logo at the right side of the frame. If a replay is running, this logo will not be displayed. The third ROI analyses the current health, which is a number from 100 down to 0. The algorithm uses this ROI to detect the kill of the currently shown player. The last ROI analyzes the time.
While detecting the replay starting point it is possible, that no replay is found after the start replay logo. Because of this reason, the algorithm checks the first and second ROI. If that scene is no replay, the found start frame can be deleted and the search process for the start logo begins again. If the frame diference limit is reached, the algorithm will stop the search for the ending logo. If a correct start point for a replay is found, the algorithm will check the replay end scenarios.
The algorithm includes five end scenarios. The first three scenarios just stop the replay. The last two stop the current replay and start a new one. These five scenarios are: • The standard stopping case for a replay is the logo transition at the end. This detection is the same as the start detection, only the adaptation of the frame number is different. The algorithm reduces the frame number with a predefined number. • A replay will stop if the current shown player is killed. This happens, when the health of a player reaches zero. The algorithm uses the L1-norm of the normalized reference and the ROI values. If the norm drops below the threshold value, the current shown player will be killed, and the replay will stop at this frame. • A hard shot, every pixel in the frame is black, stops the replay too. The black frame occurs after a normal frame and after the black one, a normal frame is shown again but from another scene. The algorithm detects this case with the OpenCV method absdif and stops the replay. The next replay logo is not marked as a starting point, because it is the end of the prematurely stopped replay. • The algorithm for detecting a player change uses the first ROI and the structural similarity index. The found frame is the last of the current replay and the next one is the beginning frame of the new starting replay. • The system identifies a time jump based on the time ROI and compares the current time with the time from the 13th frame before.
The results of replay detection are the frame numbers for begin and end of a replay and the shown player. The metadata processing uses these interim results. 3
The metadata processing uses the result from the detecting replay and generates a list with the roundbegin and next_round_begin frame numbers for the matching player stream. The Java program reads the replay_begin, replay_end and replay_source values from the commentator stream results and looks up the suitable match and round. After that, the algorithm uses the matching match and round to find the correct round_begin frame number of the player stream. This process uses the sync-points, but some rounds have no frame numbers. For that reason, it searches the metadata.csv for the specific day. This metadata is not usable because it is very inaccurate. As a result of using the metadata.csv the working time for the next step is longer. Fortunately, this metadata.csv was never used, because all detected replays are in rounds with sync-points. 4
The replay source detection is the last step of the algorithm and is implemented with Python and OpenCV. This step uses the round_begin numbers to start the detection.
The algorithm compares two ROIs. First, it checks the time of the found replay starting frame of the commentator stream with the current time of the player stream until it fits. If the times match, the middle region of the same frame is checked until the suitable frame precision recall avg. overlap is found. The algorithm does the same with the replay and frame of the commentator stream. The detection stops if the matching start and end frame in the player stream is detected or the next round begins. The time ROI has one disadvantage: the program must resize the time of the commentator stream in order to compare it with the time ROI of the player stream, because the time ROI has diferent size in the commentator and the player streams.
The result of this step includes the replays of the commentator stream and the begin and end frame number of the accurate player stream. 5
The approach I have chosen was mainly based on content-based methods. Only the generation of the synchronization part uses another approach. A Python script and a groundtruth file for the training dataset was available to check and verify the algorithm.
The next step was to check the algorithm with the test dataset. All parameters and configurations are the same as in the implementation step with the training dataset. The described approach detects 39 replays for the test dataset and the possible matching source scenes for that. The organization group checked this with a Python script. Table 1 shows the result.
This task is evaluated in 2 steps. Both, the found replays in the commentator stream and the found scenes in the player stream were checked. For the first step, the used methods are precision, recall and the balanced F1-score. For the second step an average overlap of the found sequence was computed. One detected replay is labeled with result ‘None’, because the video stream files of P7 and P10 shows in match 1 and round 1 to 4 the same player on that day.
Special thanks to the organization group especially Mathias Lux and
thanks to Michael Wutti for the provision of the synchronization
points [