In this paper, we briefly introduce the solution of our team HFUT-VUT for the Micros-gesture Classification in the MiGA challenge at IJCAI 2023. The micro-gesture classification task aims at recognizing the action category of a given video based on the skeleton data. For this task, we propose a 3D-CNNsbased micro-gesture recognition network, which incorporates a skeletal and semantic embedding loss to improve action classification performance. Finally, we rank Challenge, surpassing the second-place team in terms of Top-1 accuracy by 1.10%. Micro-gesture, action classification, skeleton-based action recognition, video understanding Micro-gesture Analysis for Hidden Emotion Understanding (MiGA) is a Challenge at IJCAI 2023. It is launched based on the iMiGUE [1] and SMG [2] datasets and requires understanding emotion based on the micro-gestures (MGs). The micro-gesture classification challenge aims to recognition MGs from short video clips based on the skeleton data. The iMiGUE dataset were collected from post-match press conferences. Compared to ordinary action or gesture recognition, MGs present more challenge. MGs encompass more refined and subtle bodily movements that occur spontaneously during real-life interactions. Additionally, there is an imbalanced distribution of MGs, where 28 out of 32 categories accounted for 57.8% of the data. In this challenge, we adopt the skeleton-based recognition model PoseC3D [3] as the baseline model, and introduce semantic embedding of action label [4, 5, 6, 7] to supervise the action classification.
Raw video 2D Pose data 3D Heatmap Volumes (joints/limbs) )(xxxTCHW v1Cno trsLyaeee2RN trsLyaeee3RN trsLyaeee4RN )//xxxT8218HW lirggaaeePnboooV 512 (5 l
G
FC !"# 1.0
prob • We proposed a CNN-based network for micro-gesture classification. Specifically, we incorporate skeletal and semantic embedding loss for action classification. • For the micro-gesture classification challenge, our method achieves a Top-1 accuracy of 64.12 on the iMiGUE test set. For the SMG dataset, our proposed method achieves 68.03 and 94.76 of Top-1 and Top-5 accuracy, respectively. The experimental results indicate that our method efectively captures subtle changes of micro-gestures.
Considering that there are no lower body actions in the iMiGUE dataset, only the 22 key
points extracted by OpenPose toolbox [
After getting the 3D heatmap volumes, here, we use 3D-CNNs to capture the spatiotemporal
dynamics of skeleton sequences. Specifically, we first use the SlowOnly [
∈ ℝ300. Here, we use a semantic loss L to make close to L = L
+ ⋅ L ,
L
= ||
− ||2,
(1)
(2)
where is a hyper-parameter to balance the two losses, and we will discuss it in the experiment.
L is MSE loss to supervise the semantic embedding. L = L is the cross-entropy loss
to supervise the skeletal embedding. In addition, L also serve as the classification loss.
iMiGUE [
For the micro-gesture classification challenge, we calculate the Top-1 Accuracy to assess the prediction results. For the micro-gesture classification challenge, we implement our approach with the PYSKL toolbox [11]. The model is trained with SGD with momentum of 0.9, weight decay of 3e−4. We set the batch size to 32, set the initial learning rate to 0.2/3. In addition, the model is trained 100 epochs with CosineAnnealing learning rate scheduler. The SlowOnly model is adopted as the 3D-CNN backbone. For the ensemble model (Joint&Limb), we use the weighted summation of scores for two modalities with a ratio of 2:3.
As shown in Table 1, we report top-3 results on the test set of the iMiGUE dataset. Our team achieves the best Top-1 Accuracy of 64.12, which is higher than the runner-up by 1.10%. In addition, we also compare our approach with diferent skeleton-based action recognition methods on the iMiGUE and SMG datasets. At first, we investigate the impact of hyperparameter in Eq. 1. As shown in Table 2, the proposed method achieves the best Top-1 accuracy when = 20 . Thus, we set = 20 as the optimal setting in the following experiments. Secondly, as shown in Table 3, on the iMiGUE dataset, our method achieves the best Top-1 and Top-5 accuracy of 64.12 and 91.10, respectively. Compared with the baseline model PoseC3D, our method exhibits 2.74% improvements in Top-1 accuracy with the joint feature as input. On the SMG dataset, our method also achieves the best performance (i.e., 68.03 and 94.76 of Top-1 and Top-5 accuracy). Compared with the PoseC3D model, our method achieves 2.63% improvements in Top-1 accuracy in terms of joint feature. In addition, we can see that the ensemble model (Joint&Limb) also shows significant performance improvement ( i.e., 1.96% and 2.96% improvements on Top-1 and Top-5 accuracy compared with ‘Joint’). 1The Codalab competition page: link
In this paper, we present our solution developed for the MiGA challenge hosted at IJCAI 2023. Our approach adopts the PoseC3D model as a baseline, incorporating both skeletal embedding loss and semantic embedding loss. By leveraging the joint and limb modality data, our approach achieved the first place with the top-1 and top-5 accuracy of 64.12 and 91.10, respectively. In the future, we plan to address the issues in this challenge from other perspectives, e.g., more robust network for human pose estimation, data augmentation for imbalanced data learning, RGB-based visual feature for micro-gesture recognition, and temporal context modeling [16, 17] for capturing subtle changes of MG.
This work was supported by the National Natural Science Foundation of China (72188101, 62020106007, and 62272144, and U20A20183), and the Major Project of Anhui Province (202203a05020011). Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6202–6211. [10] J. Pennington, R. Socher, C. D. Manning, Glove: Global vectors for word representation, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 2014, pp. 1532–1543. [11] H. Duan, J. Wang, K. Chen, D. Lin, Pyskl: Towards good practices for skeleton action recognition, in: Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 7351–7354. [12] S. Yan, Y. Xiong, D. Lin, Spatial temporal graph convolutional networks for skeleton-based action recognition, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. [13] L. Shi, Y. Zhang, J. Cheng, H. Lu, Skeleton-based action recognition with multi-stream adaptive graph convolutional networks, IEEE Transactions on Image Processing 29 (2020) 9532–9545. [14] Y. Chen, Z. Zhang, C. Yuan, B. Li, Y. Deng, W. Hu, Channel-wise topology refinement graph convolution for skeleton-based action recognition, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 13359–13368. [15] H. Duan, J. Wang, K. Chen, D. Lin, Dg-stgcn: Dynamic spatial-temporal modeling for skeleton-based action recognition, arXiv preprint arXiv:2210.05895 (2022). [16] D. Guo, S. Wang, Q. Tian, M. Wang, Dense temporal convolution network for sign language translation, in: Proceedings of the International Joint Conferences on Artificial Intelligence, 2019, pp. 744–750. [17] K. Li, D. Guo, M. Wang, Proposal-free video grounding with contextual pyramid network, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2021, pp. 1902–1910.