The SportsVideo task of the MediaEval 2021 benchmark is made up of two subtasks: stroke detection and stroke classification. For the detection task, participants are required to find some specific frame intervals broadcasting the strokes of interest. Subsequently, this can be utilized as a preliminary step for classifying a stroke that has been performed. This year, our HCMUS team engaged in the challenge with the main contribution of improving the classification, aiming to intensify the efectiveness of our previous method in 2020. For our five runs, we proposed three diferent approaches followed by an ensemble stage for the two remaining runs. Eventually, our best run ranked second in the Sports Video Task with 68.8% of accuracy.
In the Multimedia Evaluation Challenge 2021, there are two main
sub-tasks: detection and classification. Specifically, the latter
speciifes video boundaries as inputs to perform classifying stroke
categories. About the dataset, strokes are categorized into the same
20 classes as that of last year, with an addition of new and more
diverse samples [
We conducted three experiments with diferent model
architectures and submitted five runs in total. Generally, the first, second,
and fifth runs were independent methods. Turning to the other
versions, the third run is the ensemble of the first and the fifth
runs, while the first and second runs are used for ensembling the
fourth run. For the first run, we employed a rudimentary method to
handle video classification by spatially stacking images in a video
sequence to form a super image, as such a simple idea is proven to
be eficient in [
In this run, we stacked images in sub-clips spatially to create a super image with size × as a representation for the full clip and treat the video classification task as an image classification problem. After that, a classification head was used for making prediction about stroke categories. 2.2
Run 02 In this run, we decomposed the original classification problem into three sub-classification branches, with the sub-categories split for each classifier described in Table 1. This mechanism was based on our motivation to disentangle the existing ambiguity of the raw labels. It would be more relevant to discriminate among serve, ofensive, and defensive strokes rather than serve, forehand, and backhand types. Moreover, our proposed sub-categories classification method by breaking the raw labels into many sub-classes can supplement more training samples for each category in the classifiers, as the collection of some strokes in table tennis are still limited. Eventually, each classifier utilized both shared and exclusive features useful for the corresponding tasks.
The first and third components utilized the shared features
ℎ_13 which were constructed by performing concatenation
between the temporal visual features and temporal pose features. A
3D-CNN architecture implemented by [
However, another significant feature should be incorporated when handling with the third classifier (Forehand, Backhand). We ifrst performed cropping the original image based on the boundaries of the hands’ region, which can be extracted by selecting the coordinates of key points that satisfy a plausible position for human hands. After that, the concatenation of two hand images of shape 1 × 1 × were supplied into a diferent 3D-CNN branch
First Serve, Ofensive, 3 Component Defensive
Second Forehand, 2 Component Backhand
Backspin, Loop,
Third Sidespin, Topspin, 8 Component Hit, Flip,
Push, Block Table 1: Three splitted sub-categories for three classifier types to produce another temporal visual hand feature for the third classification branch
After that, three multi-layer perceptrons (1) were designed for each branch of classification with diferent number prediction heads shown in Table 1. The loss function of each branch was then aggregated for serving the final multi-task learning loss L .
ˆ = ( ( ))
Finally, we formulated the joint probabilities (1, 2, 3) (2) of predicting three independent sub-categories using prior knowledge. By conducting a thorough analysis about the co-existence of three sub-categories, we concluded that the existence of the second component label was independent of the first and third component label. On the other hand, it was possible to narrow down plausible labels of the third component given the prior knowledge about the categories of the first component. In Table 2, we summarize the relation of existence between the first and third components that we have investigated so far.
(1, 2, 3) = (3, 1 |2) · (2) = (3, 1) · (2) = (3, 1) ·ˆ22 The second term can be referred to the ℎ value of ˆ2 (1) of 2 the second classifier. Meanwhile, the first term (3, 1) (3) was factorized into two terms.
(3, 1) = (3 |1) · (1)
= ˆ 33 ·ˆ11
Given the prior knowledge tables, we first construct a binary referenced matrix ∈ 3×8, which encodes the co-existence of labels between the first and third component. Then, we perform Hadamard product on the two vectors (1) ∈R1×8 (where (1) = {0, 1, 2} represents the true index of 1) and ˆ3 ∈R1×8 to produce the refined probability ˆ 3 ∈R1×8 (4). Finally, it is normalized before being multiplied with the ℎ value of ˆ1 (1): 1 ˆ 3 = (1) È
ˆ3 (3, 1) = (3 |1) · (1) =
ˆ 33 Í==18 ˆ 3 ·ˆ11
Table 2: Prior Knowledge tables Trong-Tung Nguyen et al.
Backspin, Loop, Sidespin, Topspin
Hit, Loop, Flip Push, Block, Backspin (1) (2) (3) (4) (5) 2.3
Run 05
We made a small modification on the second run by replacing our
designed classifier with a more powerful model architecture for the
action recognition problem, which we have utilized last year [
In the first run, the final score is the average score of two
subclips in the video. All of the images were resized to shapes of
224×224. We passed the super image to the ResNet-50 [
Run ID Accuracy
Run 1 61.99%
Run 2 44.80%
Run 3 68.78%
Run 4 60.63%
Run 5 67.87%
This work was funded by Gia Lam Urban Development and Investment Company Limited, Vingroup and supported by Vingroup Innovation Foundation (VINIF) under project code VINIF.2019.DA19.