Over the past few years, Action Recognition task has drawn considerable interests, leading to intensive researches. This is mainly due to the variety of related applications, from autonomous car to human behavior analysis. Up to now, most of researches aim to identify various sport actions such as UCF-101 dataset[11], but, due to the exponential number of online videos and the necessity to be more and more accurate, the need of finer analysis arises. In this working note, results for the MediaEval 2019 Sports Video Annotation "Detection of Strokes in Table Tennis" task [9] are presented. As in sport videos displacement flow appears to be one of the most useful information for stroke identification, especially to diferentiate quite similar strokes, this proposal relies on a combination of spatial information and Optical Flow's singularities identification. As a result, most relevant regions of video frames for the classification task are detected.
The selected task requires to analyze a single sport, which means that the analysis has to be even more precise than high interclass variance datasets. The dataset, aiming at representing real-life sportsman training situations, is made up of videos recorded using standard cameras with unbalanced number of training samples for each stroke. No depth maps or data issued from motion capture suits are available.
This working note provides a description of the methods
proposed by the team MIA on this task. Only handcrafted features
extracted from video frames and optical flow are used: Histogram
of oriented Gradients (HoG)[
The great success and popularity of Deep Learning methods for
2D images recognition tasks, led many researchers to adapt these
architectures to video analysis using 3D filters instead of 2D filters
commonly known as 3DCNN[
For both manual and deep learning methods, the Optical Flow
was also proved relevant, with the arrival of two-stream network
architectures[
As already said, the proposed approach relies on dense accurate
Optical Flow. Nowadays, one of the most popular method is
probably the Farneback [
More recent methods are trying to overcome this drawback,
especially, the PWC-Network [
Given the horizontal and vertical components U and V of the optical lfow, regions of high rotation or divergence are detected by the following stage. For each frame, using a sliding window, the optical lfow is locally approximated using a Legendre polynomial basis. The polynomial basis P is defined as: PK, L (x1, x2) = ÍkK=0 ÍlL=0 x1k x2l
To obtain precise results, a small sliding window of 50 pixels is chosen. The resulting computational cost is therefore limited as a one-dimensional polynomial basis is precise enough in such a case
After the projection, the two components are eficiently calculated on a canonical basis by approximating U and V flows as follows :
UV ≃ A xx21 + b = aa2111xx11 ++ aa2122xx22 ++ bb21
Each pixel region is then represented by a 2x 2 matrix made of canonical projection coeficients of the flow. Significant region are selected by a simple threshold: ∆ (A) = tr (A)2 − 4 ∗ det (A), ∆ (A) < 0.05 2.2
The classification task follow the Bag of Word (BoW) approach: KMeans are used to classify the various singularities (each singularity being originally represented by the four projection coeficients) into six clusters.
Except for the first run, the relative spatial positions of the singularities in the frames are also used. The frames are divided in four-squared grids and the number of singularities on each of these four regions are analysed.
For the last two runs, HoG Features, as represented by a height bins BoW, are also used but only on regions where significant singularities have been selected. This aims at quantifying the relative importance of optical flow-based and gradient based features.
As a result, each stroke is represented by an histogram with at most 18 bins (6 singularities, 8 HoG, and 4 spatial regions).
Classification is done by a cross-validated linear SVM[
The given dataset being seriously unbalanced, a balanced SVM is used on the last run, giving penalties for the most common classes, to increate the retention rate of rare strokes. 3
The proposed method leads to four runs, using only singularities for the first one, and adding additional information like HoG or the position of the singularities region for the others. The accuracy of the four runs are presented in Table 1 for both training and testing set.
The last three runs with the singularities and spatial/pixel information have pretty similar results for the test set, but the run using only the projection coeficients gives a lower global accuracy. That proves that using movement-based analyze, without using other data is not suficient to have a good enough interpretation of a stroke, and focusing only on the flow information results in high information loss.
The second and third run, with singularity positions and unbalanced SVM have similar results both in terms of overall accuracy and predicted classes. This behavior is unexpected as one of the run uses Hog features, while the others does not. Maybe, because only one sport is present in the dataset, the players edges are not suficient to diferentiate strokes. We used HoG on each frame, knowing that one frame alone isn’t enough to know what stroke class it belongs to. We stacked them over the whole sequence without taking into account the temporal data, and that’s probably why the HoG have no impact on the results overall.
On the other hand, the only run with balanced SVM provides a better overall accuracy. As said in the introduction, the dataset is heterogeneously balanced. Standard unbalanced SVM predicts the classes to increase the overall result. On this dataset, it overpredicts the most frequent classes. By using weights, balanced SVM increases its accuracy on the rare classes, resulting in a worst overall result, but in better results on rare classes. 4
This paper presents an approach for the Sports Video Annotation on single-sport dataset task. Due to the dificulty of the task, the rare classes samples, missing metadata about right or left handed players, and diferent camera viewpoints, didn’t achieved high performance scores, but it gives an insight of what is missing in the proposed Optical Flow’s Singularities features.
There is a still rooms for improvement, mostly due to the lack of long term temporal information and the variations between two optical flows of the same stroke class when recorded by cameras on diferent viewpoints.
Optical Flow Singularities for Sport Video Annotation