This paper presents the baseline method proposed for the Sports Video task part of the MediaEval 2021 benchmark. This task proposes a stroke detection and a stroke classification subtasks. This baseline addresses both subtasks. The spatio-temporal CNN architecture and the training process of the model are tailored according to the addressed subtask. The method has the purpose of helping the participants to solve the task and is not meant to reach stateof-the-art performance. Still, for the detection task, the baseline is performing better than the other participants, which stresses the dificulty of such a task.
49 30 60 30 Pool 24 Conv (3x3x3) ReLU 80 (3Cxo3nxv3)Pool ReLU 1
FC ReLU 500
Most recent action detection and classification methods developed in the literature have been using deep learning approaches and high-dimensional spaces. In the domain of image classification, a specific kind of Neural Network has become very popular: the Convolutional Neural Networks (CNNs). Since the breakthrough at the 2012 ImageNet Challenge, CNNs have demonstrated a great improvement for image classification.
For video applications in general and action recognition in
particular, the first models proposed were a direct extension of image
classification methods [
As part of the task organization, for the first time since the
beginning of the Sports Video task (in 2019 [
In order to perform classification and detection, we consider the
model architecture presented in Fig. 1. For each subtask, a distinct
model has been trained on the train set. We train both using a
stochastic gradient approach with a Nesterov momentum of 0.5 [
At each epoch, the model is validated on the validation set. The model performing the best on this set is saved and then evaluated on the test set. The model is fed with the video frames resized to 120 × 120 and staked successively in cuboids of length 98, representing approximately 0.82 seconds.
For the detection task, we inferred Non-stroke segments from the annotated Stroke segments. We considered only segments between two consecutive strokes greater than 200 frames. Such a segment is divided in successive blocks of 200 frames, non overlapping, and added has a negative sample for training the model. The split using 200 frames allows a correct number of negative samples: from the 783 train and 234 validation segments, we inferred respectively for each set 1196 and 260 negative segments. No negative segments have been inferred from the test set. Stroke detection is tackled 1https://github.com/ccp-eva/SportTaskME21 as a classification task by considering two classes: Stroke on Nonstroke. From the test set, which has no temporal boundaries, we created window proposals of length 150 every 150 frames for all the videos. This size was chosen empirically and meant to be revised to achieve good performance. For the classification task, all the classes were not represented in the dataset but we still consider all the 20 possible stroke classes.
To train the model, we inputted the RGB cuboids composed of the successive frames from the starting frame of the considered segment. The desired output is the class vector summing to one and binary at training time. Its length is the number of considered classes: 2 for detection and 20 for classification. Each element represents the probability of belonging to a class. During inference, we follow a similar procedure, and the class decision is the argmax of the output vector. 3
This section presents the results per subtask according to the
metrics presented in [
The detection subtask was tackled as a classification task, considering the strokes and non-strokes samples. After 500 epochs, the model reached 98.3% and 75.7% of accuracy, respectively, on the train and validation sets. On the test set, the model is evaluated using the mAP metric. This metric takes into account the number of actions detected and their overlapping with the ground truth. The baseline achieves an mAP of 0.0173, which the two participants of this subtask did not outperform.
Runs are also evaluated using a global IoU that considers only the frame-wise overlap of the detected strokes with the ground truth annotations. The number of strokes detected is no longer taken into account in the evaluation. The baseline achieves a Global IoU of 0.144, which was outperformed by one participant.
The method’s performance is quite low due to the method being relatively simple. It also relies on a straightforward and non-eficient window proposal to segment the strokes without fusing the output decision. Indeed, two consecutive windows, part of the same stroke and classified as strokes, will be classified as two diferent strokes and not a single one, and will therefore have an impact on the mAP metric. The method can easily be improved by considering better proposals and fusing the output decisions. 3.2
The results for the stroke classification subtask on the test set are reported in the table 1. This table is divided into diferent sections for considering diferent refined classifications. After training, the model reached only 25.2% and 28.9% of accuracy, respectively, on the train and validation sets.
• “Global” consider all the 20 classes • “Type” consider only the type of the stroke: Defensive,
Ofensive or Service • “Hand-Side” consider only Forehand and Backhand superclasses • “Type and Hand-Sided” consider the intersection of the two last clusters leading to 6 classes.
The confusion matrix of the “Type” and “Hand-Side” are also depicted in Fig. 2 for further analysis.
Pierre-Etienne Martin
From table 1, we can state that the performance of the baseline,
considering all classes, is limited. This may be improved by
further analysis of the corpus and further training. Indeed, only 18
classes over the 20 possible were present in the corpus this year,
which simplifies the complexity of the task and could have been
taken into account in the model’s design. Fig. 2.a reveals that the
services have not been learned at all, which is undoubtedly due to
the input processing during training which considers only the 100
ifrst frames and is therefore unable to capture features from these
longer strokes. Finally, Fig. 2.b underlines the main weakness of
the model: being unable of distinguishing Forehand and Backhand
strokes. The pipeline’s method could consider higher level
categories, following a cascade method, to improve the performance.
Two of the three participants have outperformed by far the baseline
performance [
This baseline intends to help the participants solving the Sports Video Task. The baseline performance remains limited, but its publicly available implementation allows the participants to not start from scratch. Many aspects of the method may be improved, such as the data processing: a spatial and temporal ROI may increase the performance. Similarly with the architecture of the model, which was kept very simple, or the training method that could have merged the train and validation sets before inferring on the test set.
The detection subtask seems to be challenging. No participants were able to beat the baseline performance with regard to the mAP metric, which is the ranking metric. This subtask is new in the Sports Video Task, which also explains the low results obtained. However we believe much improvement can be obtained since our method has tackled it as a classification task. The window proposal is also very crude and can easily be improved.
The classification subtask has gathered more participants with, overall, more successful performance. This may be explained by the task’s non-novelty in the history of the MediaEval benchmark and the more active investigation in this field.
Next year we plan to gather ideas from this year’s submissions to improve the baseline and give a more substantial base to the new participants joining the Sports Video Task.