Sports action detection and classification is one of the most researched topics in video analytics. It is very useful in order to make ifne tuned athletic training and get a better analysis of the athlete's performance. We present a model to detect and classify table tennis strokes made by players as a part of the MediaEval 2021 benchmark. Our approach extracts features using a YOLOv5 model trained on the MediaEval Fine Grained Action Detection and Classification of Table Tennis Strokes dataset provided to us, to detect and classify the moves/actions made.
Action recognition is the task where predefined set of actions will
be associated with the video. An automatic analysis of actions in
the videos is the need of the day. In this paper we have proposed
a method to detect and classify strokes in a dataset consisting of
various strokes in table tennis performed during a match or during
practice. Localization of the objects and identifying them followed
the classification is the sequence of tasks involved in the action
recognition. Strategic decisions can be taken once the actions are
detected and classified. The dataset consists of 20 diferent classes [
Sports action classification is a topic in which there has been a lot
of research been carried out which tend to focus on recognising
a large number of actions using spatio-temporal models, using
videos. Feature extraction, dictionary learning , and classification
[
The dataset had many videos which consisted of actions and moves
made by players which had very subtle diferences amongst them.
So we took into account temporal information in the frames in
an efective manner. Since the actions had very subtle diferences
with the low inter-class variability, it was a dificult task to handle.
CNN models found to be optimal to classify if the data is highly
spatial with proper discrimination among the classes. We decided
to study this with object detection and recognition deep learning
framework, YOLOv5 architecture [
The YOLO model takes fixed input sizes for each mini batch. The frames were downscaled to 512 × 512 in order to keep the size of the ifles manageable. CVAT (Computer Vision Annotation Tool) was used to annotate the actions of the players, by drawing bounding boxes over the body, focused on the hand holding the bat, in the videos as per the given frame number in the dataset. The strokes annotated are of varying duration with some being very short while others more lengthy. This meant we had to ensure the extracted frames had information on the entire move in it, irrespective of the duration. Since there were two diferent annotation data sets i.e training and classification, we observed that the detection frame sequences were overlapping with the classification ones, we annotated only using the detection data set using the stroke-classes when present, or marked it as just "stroke" if no class was present. We then split the annotated files in the required two file types. This saved us a lot of work, as we did not have to annotate the same video twice or draw two bounding boxes. 3.2
Our approach uses the complete RGB frame sequence of the whole video that consisted of some subset. The YOLOv5 architecture is a modified version of the YOLOV4 implemented by Ultralytics. YOLOv5 has three functional components namely the Backbone, CSPDarknet, its Neck, PANet and the Head, Yolo Layer. CSPDarknet helps to extract the features from the frames of the table tennis videos. Feature pyramids are constructed using the PANet stage that helps in generalizing with diferent sized objects. By applying the bounding boxes on the features the head layer performs the object detection task.
The YOLOv5 model has been trained for 15 epochs in order to detect the strokes, classify them and find their respective bounding
(a) Detected as stroke (b) Detected as ofensive forehand flip boxes from the frames. The model is trained to detect 20 diferent classes of strokes. It has obtained a training and validation loss of about 0.0039% and 0.0021% with loss function mentioned in Table 1. Hyper-parameters adopted by our approach is listed in Table 1. As we considered the whole action sequence during detection, overiftting was a major problem as even a still position of the player was fed to the model with a positive label which caused over-fitting of the model. This could have been prevented by taking only the frame sequences where a move was performed. In the classification part, this model seemed to perform better than the detection part as the classes were in lesser amount in the dataset compared to the detection part. 4
The model was able to classify 13 out of the 20 classes as we could not annotate the videos which had the other 7 classes. We achieved an accuracy of 9.95% on the 13 classes where some of the classes were predicted with a good accuracy and some of them had poor accuracy. It has been observed with respect to per class accuracy that the model learnt some of the moves well than the others. A sample images after detection are shown in Figure 1a and 1b.
But a very poor performance in the test set of the detection
(mAP=0.000525 G-IoU=0.247) showing that using YOLOv5 with
current training was not a probable model for this dataset. The
GIoU is better than mAP shows that the detection is moving towards
the ground truth, that if we have trained the network with diferent
hyper-parameters for more epochs the detection would have been
better. The baseline results for this dataset can be seen in [
As we processed the data where each move/action was considered for a very large frame sequence, it resulted in over-fitting. Thus over-fitting could have been avoided if the moves/actions were precisely annotated in the video dataset and considered correctly when fed into the model. We learnt that data of this kind needs to have precise annotations after pre-processing which could result in better results. Thus our model could not show comparable accuracy when compared with the baseline model which was provided for reference. The performance could have been enhanced further by annotating all the videos and by training for more number of epochs. Conv3d with diferent hyper-parameters other than the baseline model can be attempted to study the performance.