In this paper, we briefly introduce the solution developed by our team, HFUT-VUT, for the track of Microgesture Classification in the MiGA challenge at IJCAI 2024. The task of micro-gesture classification task involves recognizing the category of a given video clip, which focuses on more fine-grained and subtle body movements compared to typical action recognition tasks. Given the inherent complexity of microgesture recognition, which includes large intra-class variability and minimal inter-class diferences, we utilize two innovative modules, i.e., the cross-modal fusion module and prototypical refinement module, to improve the discriminative ability of MG features, thereby improving the classification accuracy. Our solution achieved significant success, ranking 1st in the track of Micro-gesture Classification. We surpassed the performance of last year's leading team by a substantial margin, improving Top-1 accuracy by 6.13%. Micro-gesture, action classification, multi-model action recognition, video understanding The 2nd Workshop & Challenge on Micro-gesture Analysis for Hidden Emotion Understanding, Aug 3-9, 2024, Jeju, ∗Corresponding author.
CEUR ceur-ws.org
Micro-gestures (MGs) [
Cross-Modal Fusion Module
n Conv Res2 Conv
Res3
Res3 #"#$ #&'()
Conv3d
Conv3d 1,0,0*,'…#+,,0(,0,0 1,0,0, … , 0,0,0 #,
Compared to ordinary action or gesture recognition, MGs present more challenges. MGs encompass more refined and subtle bodily movements that occur spontaneously during real-life interactions. Additionally, micro-gestures sufer from small inter-class diferences and large intra-class diferences. Therefore, micro-gestures that have unclear action boundaries, large individual diferences, high similarity, and are easily influenced by the environment tend to be misclassified, which we refer to as ambiguous samples.
In this challenge, we adopt PoseConv3D [3] as the baseline model, and the main contributions of our method are summarized as follows: • We proposed a multi-modality-based network for micro-gesture classification. Specifically, we incorporate the cross-modal fusion module and prototypical refinement module for action classification. • For the micro-gesture classification challenge, our method achieves a Top-1 accuracy of 70.254 on the iMiGUE test set. The experimental results indicate that our method efectively captures subtle changes in micro-gestures.
The main structure of the proposed method is illustrated in Figure 1. We adopt the PoseConv3D [3] network as the backbone, which enables more eficient learning of spatio-temporal features and greater robustness in noise handling. Concretely, the proposed method consists of a two-pathway 3D CNN-based backbone network, where the upper path is responsible for processing RGB data, and the lower path processes skeleton data.
The inputs to the model are
= ( 1, … , represent the number of frames, we set ) and to 8 and = ( 1, … ,
), where to 32. The backbone network and comprises the Cross-Modal Fusion Module and Prototypical Refinement Module. Specifically, the Cross-Modal Fusion Module aims to explore the correlations between diferent modalities. The Prototypical Refinement Module defines prototype representations of each fine-grained micro-gesture category during training and forces the model to calibrate ambiguous samples among diferent micro-gesture categories. These feature vectors are subsequently mapped to the probability distribution of candidate categories through a fully connected (FC) layer with softmax activation. Finally, the probability distributions of the two modalities are fused in a 1:1 ratio to obtain the model’s final prediction for the micro-gesture category.
Due to the subtle and complex nature of micro-gestures, diferent categories of micro-gestures may have diferent advantages and disadvantages in terms of expression in diferent modalities. To efectively leverage the characteristic features of diferent modalities within the backbone network, we propose the Cross-Modal Fusion module.
Assuming the features of the module are ̂ ∈ ℝ 1× 1× 1× 1 and ̂ ∈ ℝ 2× 2× 2× 2. Firstly, these features are mapped to the same hidden dimension through two 3D convolutional layers, and then a maximum pooling layer is used to compress the spatial features to (1,1). At this point, ̂ , ̂
∈ ( 1, ′), where ′ represents the hidden dimension. Subsequently, we utilize the cross-attention fusion mechanism to fuse the features learned from RGB and Skeleton modalities in a multi-level adaptive manner. We define the cross-attention fusion mechanism as: The variables in Equation 1 are defined as follows:
Attn ∈ ( , , ), where ∈ { RGB, Pose}.
= ̂ ′ , = ̂ ′ , = ̂ ′ . processing multimodal data. dimension and then summed:
In contrast to existing methodologies that implement attention [4, 5, 6] and cross-attention mechanisms [7, 8, 9, 10] along spatial dimensions, this paper introduces a novel approach that applies cross-attention based on channel dimensions. This technique enables the model to execute information fusion across diferent modalities, specifically at the channel level, and facilitates the learning of correlations between channel-dimensional information across these modalities. Additionally, it efectively minimizes redundant and noisy data within the channels. By focusing cross-attention on channel dimensions rather than the entire feature map, this method significantly reduces computational complexity, ofering a more eficient solution for Finally, the resulting attentional weights are multiplied with the inputs in the channel {
″ ̂
″ ̂ = ( 3( = ( 3( ̂′ ) ⋅ ̂ ) + ̂ , are spliced through lateral connections: ̂ ); ̂ P = (1 − ) ⋅ 1 ∑ + ⋅ P
∈ , where[ ; ] denotes the concatenation of two features.
Micro-gesture recognition sufers from the problem of large intra-class diferences and small inter-class diferences. In order to solve this problem, we utilize the prototypical refinement module inspired by [11], which uses prototype-based contrastive learning, and is dedicated to discovering and calibrating the ambiguous samples between these categories. As shown in preliminary categorical predictions are calculated as ∈ ℝ × , where represents the and then we can get ′ ∈ ℝ × , where is the hidden dimension. number of action categories for micro-gestures. The ground-truth is also obtained as ∈ ℝ × . The extracted feature is then fed into a maximum pooling layer and a fully connected layer, Ambiguous Samples Discovery. Given an action label , using and , we can find the confidence samples and ambiguous samples in the training phase as follows. If a sample is predicted correctly, namely as a True Positive (TP), we consider it a confident sample to distinguish it from ambiguous samples. If a sample of action is misclassified as another category, it is called a False Negative (FN). If samples from other categories are misclassified as action , it is called False Positive (FP). We find these ambiguous samples in a batch and compute their center representations as follows:
= 1 ∑ ,
∈ = 1 ∑ . ∈
Prototype Clustering. We created a prototype representation for each micro-gesture category and randomly initialized it before training. Take the action category as an example, the prototypes P serve as a stable estimate of the clustering center for action category and can be continuously optimized during the training process through confidence samples (namely true positive TP samples) of category . With online training, the prototype P of action category can be updated by exponential moving average (EMA): (4) (5) (6) (7) and is empirically set to 0.9. where is the feature of sample , P
is the prototype before updating. is a momentum term
Error Calibration. To calibrate the prediction of ambiguous samples, we use the confidence sample of action as an anchor and compute the two auxiliary terms in the feature space, which are and for FN and FP samples, respectively, and are defined as follows: = { 1 − ( , 1 + ( , 0, ℎ where (, )
denotes the cosine distance between two features. , as the compensation term for the set of FN samples, should be closer to the confidence samples in the feature space by minimizing , motivating the model to correct the ambiguous samples as action . , as the penalty term for the FP samples, should be further away from the confidence samples in the feature space by minimizing , preventing the model from identifying the ambiguous samples as action . Finally, its prototypical refinement loss is defined as follows:
L () = − log (
, )/−(1− ) ( , )/−(1− ) + ∑(
, )/ − log (
, )/−(1− ) ( , )/−(1− ) + ∑(
, )/ ≠ ≠ ,
,
L = L + ⋅ L , where is the predicted probability score of sample for the micro-gesture category .
Finally, the total loss in the training process can be defined as follows: (8) (9) (10) where L is Cross-Entropy loss and is a hyper-parameter. 3. Experiments
iMiGUE [
As shown in Table 1, we first report top-3 results on the test set of the iMiGUE dataset. Our team achieves the best Top-1 Accuracy of 70.254. Compared to the team of “ywww11”, our method outperforms it by 1.94%. In addition, we also compare our approach with skeleton-based and RGB-based action recognition methods on the iMiGUE datasets.
As shown in Table 2, we explore the action recognition methods using diferent modalities on the test set of the iMiGUE dataset. For the RGB-based methods, compared to TSM [13], Video Swin Transformer [14] is able to capture the local feature information better by local self-attention operation and achieves an excellent Top1-Accuracy result of 61.73. For the skeleton-based method, Li et al. [20] incorporates a semantic embedding loss to improve action classification performance and achieves the Top-1 accuracy of 64.12. Huang et al. [21] propose an ensemble hypergraph-convolution Transformer equipped with an auxiliary classifier to mitigate the impact of imbalanced data. In contrast, our method utilizes a multi-modal fusion strategy based on RGB and skeleton modalities to bridge the information diferences between 1The Kaggle competition page: https://www.kaggle.com/competitions/2nd-miga-ijcai-challenge-track1/leaderboard modalities and capture the information of micro-gestures more comprehensively. Since there are very similar features between diferent micro-gesture categories, the prototypical refinement module is used to calibrate these ambiguous micro-gesture samples. As a result, our method achieves 67.91 on the Top-1 accuracy. To incorporate the advantages of RGB-based and skeletonbased methods, we perform a model ensemble on the results of the Video Swin Transformer [14] base version and Li et al. [20] and our method. Finally, our method achieves 70.25 on the Top-1 accuracy, which is 6.13% higher than the results of MiGA’23 1st place.
In this paper, we present an innovative approach based on prototype learning for micro-gesture recognition in the MiGA challenge hosted at IJCAI 2024. Our approach adopts the PoseConv3D [3] model as the baseline, incorporating the cross-modal fusion module and prototypical refinement module for action classification. The RGB modality contains rich texture information, and the pose modality provides geometric information about the human structure. Through the cross-modal fusion module, the model can establish a correlation between the two modalities and combine the information of both modalities. The prototypical refinement module can provide a stable and clear representation of each category by creating a prototype for each micro-gesture action category. This module efectively mitigates intra-category diferences in micro-gestures and makes the distinction between categories clearer. Eventually, the proposed approach reached 70.254 on the test set of the iMiGUE dataset.
In future work, we will address the issues in this challenge from other perspectives. One promising direction is the application of video motion magnification techniques [ 22, 23] to magnify the subtle changes of micro-gestures for better recognition. Additionally, we plan to pre-train our model on the large-scale micro-action dataset [24, 25] to investigate the impact of prior knowledge for micro-gesture recognition. We also plan to exploit the weakly supervised method [26] or unsupervised method [27] to alleviate the dificulty of data annotation in micro-gesture recognition.
This work was supported by the National Key R&D Program of China (No. 2022YFB4500601), the National Natural Science Foundation of China (No. 62272144,72188101,62020106007 and U20A20183), the Fundamental Research Funds for the Central Universities (No. 226-2022-00051, JZ2024HGTG0309), and the Major Project of Anhui Province (No. 202203a05020011). [2] H. Chen, H. Shi, X. Liu, X. Li, G. Zhao, Smg: A micro-gesture dataset towards spontaneous body gestures for emotional stress state analysis, International Journal of Computer Vision 131 (2023) 1346–1366. [3] H. Duan, Y. Zhao, K. Chen, D. Lin, B. Dai, Revisiting skeleton-based action recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2969–2978. [4] K. Li, J. Li, D. Guo, X. Yang, M. Wang, Transformer-based visual grounding with crossmodality interaction, ACM Transactions on Multimedia Computing, Communications and Applications 19 (2023) 1–19. [5] S. Tang, R. Hong, D. Guo, M. Wang, Gloss semantic-enhanced network with online backtranslation for sign language production, in: Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 5630–5638. [6] K. Li, D. Guo, M. Wang, Vigt: proposal-free video grounding with a learnable token in the transformer, Science China Information Sciences 66 (2023) 202102. [7] Z. Wu, C. Sun, H. Xuan, Y. Yan, Deep stereo video inpainting, in: Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 5693–5702. [8] Z. Wu, C. Sun, H. Xuan, G. Liu, Y. Yan, Waveformer: Wavelet transformer for noise-robust video inpainting, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 2024, pp. 6180–6188. [9] Y. Zhang, Z. Wu, Y. Yan, Proceedings of the pfta-net: Progressive feature alignment and temporal attention fusion networks for video inpainting, in: IEEE International Conference on Image Processing, 2023, pp. 191–195. [10] H. Xuan, Z. Wu, J. Yang, B. Jiang, L. Luo, X. Alameda-Pineda, Y. Yan, Robust audio-visual contrastive learning for proposal-based self-supervised sound source localization in videos, IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (2024) 4896–4907. [11] H. Zhou, Q. Liu, Y. Wang, Learning discriminative representations for skeleton based action recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10608–10617. [12] M. Contributors, Openmmlab’s next generation video understanding toolbox and benchmark, https://github.com/open-mmlab/mmaction2, 2020. [13] J. Lin, C. Gan, S. Han, Tsm: Temporal shift module for eficient video understanding, in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 7083–7093. [14] Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin, H. Hu, Video swin transformer, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 3202–3211. [15] S. Yan, Y. Xiong, D. Lin, Spatial temporal graph convolutional networks for skeleton-based action recognition, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. [16] H. Duan, J. Wang, K. Chen, D. Lin, Pyskl: Towards good practices for skeleton action recognition, in: Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 7351–7354. [17] L. Shi, Y. Zhang, J. Cheng, H. Lu, Skeleton-based action recognition with multi-stream adaptive graph convolutional networks, IEEE Transactions on Image Processing 29 (2020) 9532–9545. [18] Y. Chen, Z. Zhang, C. Yuan, B. Li, Y. Deng, W. Hu, Channel-wise topology refinement graph convolution for skeleton-based action recognition, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 13359–13368. [19] H. Duan, J. Wang, K. Chen, D. Lin, Dg-stgcn: Dynamic spatial-temporal modeling for skeleton-based action recognition, arXiv preprint arXiv:2210.05895 (2022). [20] K. Li, D. Guo, G. Chen, X. Peng, M. Wang, Joint skeletal and semantic embedding loss for micro-gesture classification, arXiv preprint arXiv:2307.10624 (2023). [21] H. Huang, X. Guo, W. Peng, Z. Xia, Micro-gesture classification based on ensemble hypergraph-convolution transformer, Micro-gesture Analysis for Hidden Emotion Understanding 2023 (2023) 9. [22] F. Wang, D. Guo, K. Li, M. Wang, Eulermormer: Robust eulerian motion magnification via dynamic filtering within transformer, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 2024, pp. 5345–5353. [23] F. Wang, D. Guo, K. Li, Z. Zhong, M. Wang, Frequency decoupling for motion magnification via multi-level isomorphic architecture, arXiv preprint arXiv:2403.07347 (2024). [24] D. Guo, K. Li, B. Hu, Y. Zhang, M. Wang, Benchmarking micro-action recognition: Dataset, methods, and applications, IEEE Transactions on Circuits and Systems for Video Technology 34 (2024) 6238–6252. [25] K. Li, D. Guo, P. Liu, G. Chen, M. Wang, Mmad: Multi-label micro-action detection in videos, arXiv preprint arXiv:2407.05311 (2024). [26] J. Zhou, D. Guo, Y. Zhong, M. Wang, Advancing weakly-supervised audio-visual video parsing via segment-wise pseudo labeling, arXiv preprint arXiv:2406.00919 (2024). [27] Y. Wei, Z. Zhang, Y. Wang, M. Xu, Y. Yang, S. Yan, M. Wang, Deraincyclegan: Rain attentive cyclegan for single image deraining and rainmaking, IEEE Transactions on Image Processing 30 (2021) 4788–4801.