Replay Detection and Multi-stream Synchronization in CS:GO Game Streams Using Content-based Image Retrieval and Image Signature Matching Van-Tu Ninh1* , Tu-Khiem Le1* , Duc-Tien Dang-Nguyen2 , Cathal Gurrin1 1 Dublin City University, Ireland 2 University of Bergen, Norway tu.ninhvan@adaptcentre.ie,tukhiem.le4@mail.dcu.ie,ductien.dangnguyen@uib.no,cathal.gurrin@dcu.ie ABSTRACT detect replay scenes in sports [12]. Later, Bai Liang, Song-Yang Lao, In GameStory: The 2019 Video Game Analytics Challenge, two et.al. proposed the Perception Concept Network-Petri Net (PCN- main tasks are nominated to solve in the challenge, which are PN) model to search and locate interesting events in sports videos replay detection - multi-stream synchronization, and game story [5, 10], which is also a potential approach to detect replays as they summarization. In this paper, we propose a data-driven based ap- are usually critical events in the match. In 2019, Ali Javed, et.al. pro- proach to solve the first task: replay detection - multi-stream syn- posed a novel approach for key-event detection and summarization chronization. Our solution aims to determine the replays which using Confined Elliptical Local Ternary Patterns (CE-LTPs) to ex- lie between two logo-transitional endpoints and synchronize them tract feature of motion history image for each key-event candidate with their sources by extracting frames from videos, then applying lying between the beginning and end of a gradual transition, then image processing and retrieval remedies. In detail, we use the Bag applying Extreme Machine Learning (EML) to learn the pattern to of Visual Words approach to detect the logo-transitional endpoints, detect the key-events in sports videos [3]. The results of this work which contains multiple replays in between, then employ an Image can be used for both replay detection and story summarization of Signature Matching algorithm for multi-stream synchronization different kinds of sports videos. and replay boundaries refinement 1 . The best configuration of our proposed solution manages to achieve the second-highest scores in 3 APPROACH all evaluation metrics of the challenge. In the spirit of previous efforts in the field of video event detection and segmentation, we propose a straightforward method to detect 1 INTRODUCTION replays in videos which are bounded by Intel Extreme Masters logo transition scenes. The logo in these frames is shown in a plain Game analytics is a new research area in recent years that has be- blue background and occupies approximately 80% of its image. We come popular due to the explosion of both modern live streaming propose to use a Bag-of-Visual-Words (BOVW) approach to detect technologies (e.g., Twitch, Discord, Youtube) and e-sports devel- these two endpoints of each replay segment and use image signature opments [2, 4]. Although e-sports has gained significant attention matching to synchronize multi-stream players’ views. from audiences across a variety of ages, not much research has been conducted to analyse the game play to provide the spectator 3.1 Logo-bounded Video Detection with either a quick look into a match (e.g., player’s performance statistics, highlights, critical moments, summary) or a deep insight 3.1.1 Frame Extraction and Filtering: Firstly we use the ffmpeg into a game (e.g., whole game statistics, outstanding team’s playing tool to extract frames from commentator stream. To capture the style analysis, playing strategy trending analysis). frames in which the Intel Extreme Masters logo is clearly visible, In GameStory: The 2019 Video Game Analytics Challenge, the we perform frame extraction at fps=2. As the video length is long organisers use the same dataset used in the previous year, which (approximately 12 hours), many redundant frames are generated. was provided by ZNIPE.tv [8]. More details of the data and task We reduce the number of extracted frames by using a two pointers descriptions can be found in [8]. technique to eliminate consecutive frames which have similar ORB features [11] and the degree of similarity of a gray-scaled color 2 RELATED WORKS histogram. For the ORB features, we only consider the top 500 features in total for comparison. Specifically, at frame i, we iterate A lot of research in sports videos was conducted to analyse the through its next consecutive frames until we reach a frame j (j > i) highlights, the replays, and the story of the match. Jinjun Wang, such that the number of matched ORB features does not exceed et.al. performed a shot classification followed by a proposed scene α and the L2 distance between the two corresponding histograms transition structure analysis on the labels of the classified shots to is greater than or equal to β. In our implementation, we choose 1 https://github.com/nvtu/gamestory_mediaeval2019 α = 200 and β = 0.8 (β ≤ 1), which means that 60% of the ORB features are different and the degree of similarity between the two * These two authors contributed equally. Copyright 2019 for this paper by its authors. Use color histograms is small. The ones between the frame i and j are permitted under Creative Commons License Attribution redundant and do not provide any extra information for the next 4.0 International (CC BY 4.0). steps. MediaEval’19, 27-29 October 2019, Sophia Antipolis, France MediaEval’19, 27-29 October 2019, Sophia Antipolis, France Van-Tu Ninh, Tu-Khiem Le, et al. 3.1.2 Logo-bounded Video Detection using a Retrieval Approach: From the result in section 3.1.1, we use this filtered data to construct a dictionary of visual words (codebook). We run K-Means clustering [6] on the rootSIFT-feature data points [1, 7] extracted from im- ages in the same corpus. Before processing through the remaining BOVW’s steps, frames are center-cropped to reduce background noise. Another purpose is to force the model to focus on certain parts of the frame, which have a high probability of containing the Intel Extreme Masters logo only. After having retrieved the desired frames, we can then select the proper endpoint pairs by assuming that the replay’s length would not be more than 20 seconds. One problem in this stage is that we can only determine the logo-bounded videos, which might contain many other replays Figure 1: Evaluation of all team’s runs at Jaccard-Index from either the same perspective or multiple perspectives. As the threshold = 0.5 transitions in these videos are smooth, we propose to use the results generated from the section 3.2 to refine the replay detection result. 3.2 Multi-stream Synchronization With the result obtained from the section 3.1.2, we extract frames of these logo-bounded videos with their original fps, which is 59 for the commentator stream. We apply the same process to the players’ videos with their corresponding fps. The problem now is to find in the indexed database the image which is a near duplication of the query one. Therefore, the players frames are then encoded by using [13], added to the database, and then indexed for searching with Elastic Search engine 2 . Instead of searching the whole database for the nearly duplicate images, we determine the match and shape (roundness) of the logo-bounded videos based on provided metadata Figure 2: Evaluation of all team’s runs at Jaccard-Index and their time in the commentator stream. Thereby, we can both threshold = 0.75 narrow down the search space and locate the start frame of each replay. of multi-stream synchronization. The evaluation shows that our 3.3 Refine Replay Detection Result algorithm works well to retrieve correctly more than 50%, but less As stated in section 3.1.2, for a logo-bounded video with multi- than 75% of a replay. Our result is lower than the best approach by player view, the problem could be solved easily using multi-stream the AAU-Mekul team by around 20.98% [9]. As we detect replays synchronization output. However, it would be harder for the one based on two logo-transitional endpoint pairs, our algorithm cannot with single-player view only as we now need to detect abrupt scene handle the case that misses one endpoint. changes in the video to split it into proper replays. By using the There are many situations that we mistakenly split the replays searched rank list output from the section 3.2, for two consecu- of a player due to wrong image signature matching results. For tive frames, we choose the most similar frame from the indexed instance, in case that the player stands still and shoots, then moves database, compute the absolute time difference in the synchronised a little bit and finally returns to the original point and shoots again, player stream, and then set a threshold. Thereby, we could reuse most of the frames will all find its source at one time point, while the result of the previous step to detect abrupt scene transitions a sudden player’s movement yields a different time point, which between two replays in the stream with single-player view and causes our algorithm to consider it as a scene change and split re- refine replay detection output based on the source time of similar plays. Or when the player’s view is affected by smoke/blind grenade, frames in player streams set. the features of the white scene are not distinctive enough to deter- mine its source. These cases all lead to the wrong replay splits that 4 RESULTS AND ANALYSIS reduces our detected replays’ length, which decreases our score. It can be seen from the graph shown in figure 1 that the best run Our approach works perfectly with logo-bounded streams contain- of our team (DCU-Computing) manages to gain a precision of ing only one replay. For other cases, we can generate acceptable 73.17% and a recall of 68.18%, yielding F1-score of 70.59%, which results which might have few incorrect splits. is the second-highest score among the submissions for evaluation at Jaccard threshold of 0.5. For evaluation at Jaccard threshold ACKNOWLEDGMENTS of 0.75 shown in figure 2, all of our scores decrease significantly This publication has emanated from research supported by Sci- (approximately 23.53%), except for the average overlapping score ence Foundation Ireland under grant numbers SFI/12/RC2289 and 2 https://github.com/EdjoLabs/image-match 13/RC/2106. GameStory: Video Game Analytics Challenge MediaEval’19, 27-29 October 2019, Sophia Antipolis, France REFERENCES [1] R. Arandjelović and A. Zisserman. 2012. Three things everyone should know to improve object retrieval. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. 2911–2918. https://doi.org/10.1109/ CVPR.2012.6248018 [2] Nathan Edge. 2013. Evolution of the Gaming Experience: Live Video Streaming and the Emergence of a New Web Community. Elon Journal of Undergraduate Research in Communications 4 (2013), 2153–5760. [3] Ali Javed, Aun Irtaza, Yasmeen Khaliq, Hafiz Malik, and Muham- mad Tariq Mahmood. 2019. Replay and key-events detection for sports video summarization using confined elliptical local ternary patterns and extreme learning machine. Applied Intelligence 49, 8 (01 Aug 2019), 2899–2917. https://doi.org/10.1007/s10489-019-01410-x [4] Mark Johnson and Jamie Woodcock. 2018. The impacts of live stream- ing and Twitch.tv on the video game industry. Media Culture Society (12 2018). https://doi.org/10.1177/0163443718818363 [5] Bai Liang, Song-Yang Lao, Alan Smeaton, Noel O’Connor, David Sadlier, and David Sinclair. 2009. Semantic Analysis of Field Sports Video using a Petri-Net of Audio-Visual Concepts. Liang, Bai and Lao, Songyang and Smeaton, Alan F. and O’Connor, Noel E. and Sadlier, David and Sinclair, David (2009) Semantic analysis of field sports video using a petri-net of audio-visual concepts. The Computer Journal, 52 (7). pp. 808-823. ISSN 0010-4620 52 (10 2009). https://doi.org/10.1093/ comjnl/bxn058 [6] S. Lloyd. 1982. Least squares quantization in PCM. IEEE Transactions on Information Theory 28, 2 (March 1982), 129–137. https://doi.org/10. 1109/TIT.1982.1056489 [7] David G. Lowe. 2004. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 60, 2 (01 Nov 2004), 91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94 [8] Mathias Lux, Michael Riegler, Duc Tien Dang Nguyen, Marcus Lar- son, Martin Potthast, and Pål Halvorsen. 2019. GameStory Task at MediaEval 2019. In Proceedings of MediaEval 2019. [9] Kevin Mekul. 2019. Automated Replay Detection and Multi-Stream Synchronization. In Proceedings of MediaEval 2019. [10] Wolfgang Reisig. 1985. Petri Nets: An Introduction. Springer-Verlag, Berlin, Heidelberg. [11] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. 2011. ORB: an efficient alternative to SIFT or SURF. Proceedings of the IEEE International Conference on Computer Vision, 2564–2571. https: //doi.org/10.1109/ICCV.2011.6126544 [12] Jinjun Wang, Eng Chng, and Changsheng Xu. 2005. Soccer replay detection using scene transition structure analysis. IEEE International Conference on Acoustics, Speech, and Signal Processing 2, ii/433 – ii/436 Vol. 2. https://doi.org/10.1109/ICASSP.2005.1415434 [13] Chi Wong, Marshall W. Bern, and David Goldberg. 2002. An image signature for any kind of image. Proceedings. International Conference on Image Processing 1 (2002), I–I.