This work presents a Table Tennis stroke classification approach through a siamese spatio-temporal convolutional neural network SSTCNN. The videos are recorded at 120 frames per second with players performing in natural conditions. The frames are extracted, resized and processed to compute the optical flow. From the optical lfow, a region of interest - ROI - is inferred. The SSTCNN is then feed by RGB and optical flow ROIs stream to give a probabilistic classification over all the table tennis strokes.
In the scope of video processing, action recognition and
classification is one of the main challenge. In the Sport task of MediaEval 2019
[
To deal with the low inter-variability of the classes in TTStroke-21
and avoid over-fitting on this sample of the dataset, we decided
to use cuboids of optical flow in addition to cuboids of RGB
images with spatio-temporal convolutions processed simultaneously
through a Siamese architecture as presented in [
As shown in [
RGB and Optical Flow are spatially segmented using a region of
interest - ROI - of center Croi = (xr oi , yr oi ) estimated from the
maximum of the optical flow norm and the center of gravity of all
pixels [
Cmax = (xmax , ymax ) = arдmax (||D||1)
x ,y Cg = (xд, yд ) = Í δ1(C) Í Cδ (C)
C∈Ω C∈Ω with δ (C) = if ||D||1(C) , 0 otherwise (1) xr oi = α fωx (xmax , W ) + (1 − α ) fωx (xд, W ) yr oi = α fωy (ymax , H ) + (1 − α ) fωy (xд, H ) with parameters α = 0.6, Ω = (ωx , ωy ) = (320 × 180) the size of the resized video frames, (W , H ) the size of the data inputted to our network. The function fω (u, V ) = max (min(u, V − ω2 ), ω2 ) allows to have input data extracted within the boundaries of our data. To avoid jittering, we apply a Gaussian blur along the time dimension to average the center position using a kernel of size 40 and scale parameter σblur = 4.44. 2.3
The RGB image channels are normalized by their theoretical
maximum value, 255 in our case, to map them into interval [
v N (i, j) = vSI′G(i,Nj)(v ′(i, j)) iofth|ve′r(wi,ijs)e| .< 1 (2)
This normalization method maps the values into interval [
v v ′ = µ +3×σ 2.4
Our Siamese Spatio-Temporal Convolutional Neural Network SSTCNN, see Fig. 1, is constituted of 2 branches with three 3D convolutional layers with 30, 60, 80 filter response maps, followed by a fully connected layer of size 500. They take respectively cuboides of RGB values and optical flow computed from them of size ( W × H × T )= (120 × 120 × 100). The 3D convolutional layers use 3 × 3 × 3 spacetime filters with a dense stride and padding of 1 in each direction. The two branches are fused through a final fully connected layer of size 21 followed by a Softmax function to output a probabilistic classification. Data augmentation is made online and is diferent for each epoch. Each stroke feed our SSTCNN once per epoch. For each stroke, we extract one video sample of size (W × H × T ). The T successive frames from the RGB and Optical Flow are extracted following a normal distribution around the center of our stroke with standard deviation of σ = ∆ t6−T . We also spatially augment the data by applying random rotation in the range ±10◦, random translation in range ±0.1 in x and y directions, random homothety in range 1 ± 0.1 and a 0.5 chance flip in horizontal direction and random channel swaps on the RGB data. We take extra care of applying those changing on the Optical Flow by updating its values according to the transformations. Transformations are applied and centered on the region of interest avoiding crops outside of the camera range. 2.6
All models were trained from scratch. We used firstly 250 epochs with the data samples split randomly between all strokes and then split using only two videos for validation. However we noticed the results obtained by splitting the dataset between videos were not satisfying. After looking at the dataset in detail, this is due to the fact that most of the videos contain only one kind of stroke performed by the same player. So the model will over-fit easily to the player appearance and not the characteristics of the stroke itself. With such a limited dataset and a limited time window we preferred to focus on the random distribution of the strokes among our training and validation sets. The two first runs are the classiifcation obtained with the model trained on the split dataset and saved on the minimum loss obtained on the validation set with two diferent flows presented in section 2.1. The other two runs are the same models but retrained from scratch using all data samples with the number of epochs used for obtaining best performance on the ifrst validation set. 3
On the left side of the Table 1 we can see results of the first two runs from the models trained on the split database with 250 epochs; and on the right side two others runs obtained from the models trained with all the data.
Compared to what has been obtained in previous work [
Furthermore, if we consider the confusion matrix of our best run, Fig. 2, and group strokes in larger classes as: ’Forehand’, ’Backhand’ or ’Service’, ’Ofensive’, ’Defensive’ or their intersection (6 classes), we respectively get accuracies of 76.8%, 65.8% and 54.8%. 4
Despite a strong over-fitting, by grouping strokes together in larger
classes, we can notice that some characteristics to recognize strokes
are still learned. Furthermore, the work on TTStroke-21 [
This work was supported by Region of Nouvelle Aquitaine grant CRISP and Bordeaux Idex Initiative.