Usage of Video Analysis Algorithms for Goalball Scounting Purposes Zygimantas Matusevičiusa , Julius Gudauskasa a Faculty of Informatics (Kaunas University of Technology), Kaunas, Lithuania Abstract In the 21st century, IT innovations in sports have become very impactful. Highly innovative equipment allows athletes to stay efficient, reach their best results and even predict the opponent’s strategies. Despite the growing interest in sport innovations, adaptive sports for disabled persons still feels the lack of modern solutions. We introduce the solution to analyze the video stream of goalball and using computing power to calculate the total performed throws and help to gather statistical data about the game Keywords Goalball, Video analysis, Transformation algorithms, Ball tracking 1. Introduction analyzer for teams and athletes, β€œHomeCourt” app [5] for basketball players skills development), game strat- In the 21𝑠𝑑 century, IT innovations in sports have be- egy prediction and even face recognition. Most of the come very impactful. Highly innovative equipment video tracking approaches are based on points of inter- allows athletes to stay efficient, reach their best re- est tracking [6], throughout video frames or color de- sults and even predict the opponent’s strategies. De- tection in three-dimensional color matrix. Video anal- spite the growing interest in sport innovations, adap- ysis is a complex process and the traditional analysis tive sports for disabled persons still feels the lack of techniques usually requires a lot of calculation and be- modern solutions. comes very costly. Kanade–Lucas–Tomasi (KLT) fea- Contacting the Lithuanian goalball team revealed, ture tracker [7] is one of the approaches to feature ex- that nowadays game innovations are a must to have traction that is proposed mainly for the purpose to deal an advantage against opponents, but software solu- with the video analysis special intensity and minimize tions are very expensive or not available. To main- the cost. The method is widely used in face recog- tain high results with the minimal cost of hardware, nition or motion tracking, where the different frames a system, which tracks the ball in real-time or from share a large number of tracking points. It was tested other video sources and gives the statistical informa- on eight broadcasts and achieved a 94% face track de- tion about ball positioning and movement is needed. tection rate (the average processing speed was 3.8 ffs) To achieve this result, the most important task is to [7]. Meanwhile the KLT algorithm based on points solve a video source distortion from the camera posi- of interest, other popular tracking approach based on tioning problem. a color filtration. A color based video filtration uses the spectrum of three dimensional color matrix and the filtration executed by the desired color code. It 2. Overview requires less calculation and it’s easier to implement, Object recognition from video footage and images is but in some cases it does not provide enough data to widely used technology [1, 2]. Projects like football fully track or recognize an object. The other prob- gate line and goals detection (such as β€œOnce video ana- lem that the video analysis is facing is a particle de- lyzer” [3] that helps to explain a game concepts by cre- tection and noise reduction. To retrieve statistical in- ating animated and professionally designed graphics), formation from video sources requires the most accu- basketball ball detection (such as β€œHudl” [4] – sports rate video analysis with minimal distortion level. Par- ticle filtering and noise reduction use adaptive target IVUS 2020: Information Society and University Studies, 23 April 2020, model with prediction and correction calculations [8]. KTU Santaka Valley, Kaunas, Lithuania To achieve the most accurate results, the color dis- " zygimantas1997@gmail.com (Z. Matusevičius); tribution between different gradient positions in the gudauskas.julius@gmail.com (J. Gudauskas) image should be minimal and pixels nearest the cor-  Β© 2020 Copyright for this paper by its authors. Use permitted under Creative ners should weight less. The effectiveness of the algo- Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) rithm depends on the density of pixels in video source, but the high level of intrinsic robustness ensures the will guarantee the same order, regardless of the order, proper performance and accurate filtration. After ana- they were marked. Once we know the coordinates of lyzing the possible image processing algorithms, tools, each game area corner, the calculation can be done for and technical requirements for preferred material, it the bottom edge midpoint of the game area and for the was decided that the C# programming language and bottom edge length - these parameters will be neces- EmguCV [9] (a cross-platform .Net wrapper to the Open sary for further calculations. Now we can formulate CV image processing library) contains a needed and equations describing the edges of an untwisted play- adaptable functionality to the project. An image pro- ing area. These equations will be used in filtering out cessing library can do plenty of things. EmguCV func- points outside the playing area. tionality provides a prepared video source filtration in different color formats and let us filter out objec- π‘Œπ΅π‘œπ‘‘π‘‘π‘œπ‘šπ‘…π‘–π‘”β„Žπ‘‘πΆπ‘œπ‘Ÿπ‘›π‘’π‘Ÿ βˆ’ π‘Œπ΅π‘œπ‘‘π‘‘π‘œπ‘šπΏπ‘’π‘“ π‘‘πΆπ‘œπ‘Ÿπ‘›π‘’π‘Ÿ tionable objects. These are the main image-processing π‘ π‘–π‘›π‘Ž = (1) operations that provide the majority of needed infor- πΏπ‘’π‘›π‘”π‘‘β„Žπ‘œπ‘“ π‘π‘œπ‘‘π‘‘π‘œπ‘šπ‘ π‘–π‘‘π‘’ mation that could be used in further calculations and π‘‹π΅π‘œπ‘‘π‘‘π‘œπ‘šπ‘…π‘–π‘”β„Žπ‘‘πΆπ‘œπ‘Ÿπ‘›π‘’π‘Ÿ βˆ’ π‘‹π΅π‘œπ‘‘π‘‘π‘œπ‘šπΏπ‘’π‘“ π‘‘πΆπ‘œπ‘Ÿπ‘›π‘’π‘Ÿ π‘π‘œπ‘ π‘Ž = (2) problem-solving. In this case, the main information πΏπ‘’π‘›π‘”π‘‘β„Žπ‘œπ‘“ π‘π‘œπ‘‘π‘‘π‘œπ‘šπ‘ π‘–π‘‘π‘’ source needed from the video stream is ball position- ing coordinates in each frame. Using the calculated parameters and the marked area corner coordinates, we can calculate the sin and cos values of the rotation angle for the playing area. Us- 3. Goalball Ball Movement ing the calculated trigonometric values, we apply the rotation transformation to the coordinates of the play- Tracking Algorithm ing area corners. In order to get a real playing field proportion coor- dinates of the ball positioning and movement in the π‘₯ β€² = π‘₯ π‘π‘œπ‘  π‘Ž + 𝑦 𝑠𝑖𝑛 π‘Ž (3) playing field, it is necessary to use a transformation β€² algorithm. Requirements for the method: 𝑦 = βˆ’π‘₯ 𝑠𝑖𝑛 π‘Ž + 𝑦 π‘π‘œπ‘  π‘Ž (4) 1. Sufficient vector recognition > 90% In order to improve the accuracy of the algorithm, 2. Real-time performance we adjust the coordinates of the game zone upper cor- To ensure a correct method performance regardless ners: the ordinates of the game zone upper corners of camera position, a playing zone calibration is needed. must coincide and the abscissa must be symmetrical Selected corners of the area will later assist in calculat- to the vertical line passing through the middle point ing the position of the object and determine the suit- of the game zone lower edge. Once we have the exact ability of the coordinates. coordinates of the corners, we can formulate equations The movement of the object within the calibrated that describe the rotated playing area edges. These area is monitored by examining each frame of the video boundary equations will later be used to perform co- material and subsequently systematizing the data. ordinate transformations. Equations that used to cal- The method pseudocode: culate transformation coefficients are formed as well. The method pseudocode: Calibrate game zone. Initialize game zone While frame isnβ€˜t Order corners by y null. Then order corners by x Read frame. Find middle point of bottom side line Get coordinates of object in video Calculate length of bottom side line frame. Calculate rotation sin and cos Convert video coordinates to real game Make equations for side lines in video zone coordinates. frame Group coordinates into vectors. Rotate corners Fix top corners 3.1. Game Zone Preparation Make equations for side lines in rotated game zone Since the order of the marked playing zone corners Make equations for calculating scale points is unknown, we arrange them in their y coordi- coefficients nate and then in the x coordinate. Sorting of the points 2 Figure 1: Game zone distortion from camera view. 3.2. Receiving of Object Coordinates Once the game zone calibration is done, a ball track- ing algorithm will begin to track the ball in the video stream. Object tracking is performed using every frame of the video stream and color filtration. In order to rec- ognize objects by color, a frame has to be converted into HSV format. Then, applying the color filter, we can get a binary image where the color of interest is Figure 2: Game zone view after calibration. replaced by white and everything else is black. The binary image may have a lot of "noise" and unneces- sary objects. To reduce it, we apply "noise" reduction playing field proportions. This transformation elim- operations [10, 11]. The frame processed in this way inates the asymmetry of the camera angle. Secondly, is already suitable for searching objects in it. the coordinate reference point is moved to the mid- The method pseudocode: point of the playing field lower edge. Then a vertical and horizontal scaling factors are calculated and ap- While frame isnβ€˜t null plied (in this case, coefficient equations are used). The Read frame scaling factors help to "stretch" the trapezoidal area of Convert frame from RGB to HSV playing field from the footage into a rectangle form Convert frame from HSV to binary by that is proportional to the real playing field (Fig. 2). filtering target color The method pseudocode: Reduce noise in binary video frame Get coordinates of object in binary Rotate point video frame Match up starting point with middle If object size between MinSize and point of bottom side line MaxSize Calculate vertical scale coefficient If object is in game zone Apply vertical scale coefficient Collect objectβ€˜s coordinates Calculate horizontal scale coefficient Apply horizontal scale coefficient 3.3. Coordinates Transformation 3.4. Movement Vectors Analysis As a result of video processing, we receive the coordi- nates of the ball in the distorted video frame (Fig. 1). To Once the ball coordinates in the playing area are ob- get coordinates in proportion to the real playing field, tained, they need to be grouped into vectors. The main we have to perform a sequence of geometric transfor- goal is to add received points into the vectors with the mations. condition: point is added to a vector only when the First of all, a rotation transformation is performed, distance between the point and the end of the vector is which transfers the coordinates of a point from the appropriate. If a new point is added to the vector, then designated coordinate system of the playing area to the state of the vector is updated: the vector equation the rotated coordinate system that represents a real is corrected and an attempt to complete the vector is 3 executed . Finally, all the completed and eligible vec- tors moved to the list of completed vectors. If the point cound not be added to the vector, a new vector is cre- ated, whose starting coordinates become the point that could not be added to a vector The method pseudocode: Foreach vector in remaining vectors Try add point into vector If distance between point and end of vector is suitable Add point into vector Figure 3: Results of total and recognized throws in video Adjust equation of vector source (RIO 2016, ALG vs GER (Male)). Try to complete vector If point was not added into any vector If ordinate of point is between MinVal and MaxVal Create new vector Add point to new vector Else Filter completed vectors Filter suitable vectors from completed vectors list Add suitable vectors into finished vectors list Remove completed vectors from remaining vectors list Figure 4: Accuracy of the recognized vectors in video stream (RIO 2016, ALG vs GER (Male)). 4. Results and Discussion For the research and testing of the software solution, The comparison results and calculated accuracy was two different video sources were selected, each was displayed in graphs (Fig. 5, Fig. 6, Fig. 7, Fig. 8). rendered into different frames per second ratio – 15 Other important factor of the algorithm accuracy fps, 30 fps and 60 fps. is the time duration that it takes to calculate throws The main goal of the analysis is to investigate the from video stream. Analyzing the first video from Rio accuracy of the algorithm, regarding the requirements Olympics 2016, male match Algire vs. Germany, the mentioned in third section. First selected video for the time duration that it took to analyze the entire video research was male match between Algeria and Ger- in 15 fps (original video length 20:36), was 17:08 (video many teams in Rio Olympics, 2016. To get the accu- stream had a 120% of the original speed), meanwhile in racy of the algorithm, we calculated the total throws the 30 fps video it took 26:40 (video stream had a 77% of performed by each team by hand and compared the the original speed). The analysis in 60 fps took 1:00:24 results with the output after software processing. The (video stream had a 49% of original video speed). comparison results displayed in chart (Fig. 3). Analyzing the second video from Rio Olympics 2016, From the following data, we can calculate the accu- female match USA vs. Japan, the time duration that racy of recognized vectors and get the average accu- it took to analyze the entire video in 15 fps (original racy in this match. Results was calculated in different video length 46:18), was 31:55 (video stream had a 145% frame ratio and displayed in chart (Fig. 4). A second of the original speed), meanwhile in the 30 fps video selected video for the research was female match be- it took 58:24 (video stream had a 79% of the original tween USA and Japanese teams in Rio Olympics, 2016. speed). The analysis in 60 fps took 1:56:39 (video stream Using the same calculation principle, the number of had a 40% of original video speed). throws was calculated by hand and compared with the result of the software analysis in different frame ratio. 4 Figure 5: Results of total and recognized throws in video source (RIO 2016, USA vs JPN (Female)). Figure 8: Time comparison (RIO 2016, ALG vs GER (Male)). lari, D. Giardino, M. Matta, A. Nannarelli, M. Re, An efficient hardware implementation of rein- forcement learning: The q-learning algorithm, Ieee Access 7 (2019) 186340–186351. [3] Once football video analyser, 2020. URL: https: //www.once.de. [4] Hudl application, 2020. URL: www.hudl.com. [5] Home court app, 2020. URL: https://www. homecourt.ai/. Figure 6: Accuracy of the recognized vectors in video [6] C. Tomasi, T. Kanade, Detection and tracking stream (RIO 2016, USA vs JPN (Female)). of point features, School of Computer Science, Carnegie Mellon Univ. Pittsburgh (1991). [7] D. Chatterjee, S. Chandran, Comparative study of camshift and klt algorithms for real time face detection and tracking applications, in: 2016 Second International Conference on Research in Computational Intelligence and Communication Networks (ICRCICN), IEEE, 2016, pp. 62–65. [8] A. H. Sayed, P. M. DjuriΔ‡, F. Hlawatsch, Dis- tributed kalman and particle filtering, in: Co- operative and Graph Signal Processing, Elsevier, 2018, pp. 169–207. [9] Emgu cv wiki page, 2020. URL: http://www. Figure 7: Average accuracy of recognized vectors in each emgu.com/wiki/index.php/Main_Page. video stream. [10] G. Capizzi, S. Coco, G. Lo Sciuto, C. Napoli, A new iterative fir filter design approach using a gaussian approximation, IEEE Signal Processing References Letters 25 (2018) 1615–1619. [1] G. Capizzi, G. Lo Sciuto, C. Napoli, D. Polap, [11] R. DamaΕ‘evičius, C. Napoli, T. SidekerskienΜ‡e, M. WoΕΊniak, Small lung nodules detection based M. WoΕΊniak, Imf remixing for mode demixing in on fuzzy-logic and probabilistic neural network emd and application for jitter analysis, in: 2016 with bio-inspired reinforcement learning, IEEE IEEE Symposium on Computers and Communi- Transactions on Fuzzy Systems 6 (2020). cation (ISCC), IEEE, 2016, pp. 50–55. [2] S. SpanΓ², G. C. Cardarilli, L. Di Nunzio, R. Fazzo- 5