Micro-gesture is becoming a fundamental clue of emotion analysis and achieves more attention in this ifeld. The studies are mainly focused on the task of micro-gesture classification which predicts the categories of micro-gesture while no works have been reported for spotting the micro-gestures. As a preliminary step for classification, the micro-gesture online recognition (spotting) that predicts the temporal location and category has achieved limited attention. In this context, we propose a novel deep network for micro-gesture online recognition, which incorporates the graph-convolution and multiscale transformer encoders. Specifically, we utilize a graph-convolution based Transformer module to extract motion features of 2D skeleton sequences, which are then processed by a feature pyramid module to obtain hierarchical multiscale features. We further employ a local Transformer module to model the similarity between micro-gesture frames, and decouple the classification and regression branches to achieve accurate location and category. These Transformers are trained in a two-stage strategy and combined to perform the spotting. Our proposed method is validated on the iMiGUE dataset and has achieved the first ranking in the task of online recognition (Track 2) of the MiGA2023 Challenge.
In daily interactions, the human usually rely on body gestures to perceive emotions, which
plays a crucial role in facilitating communication and understanding between individuals. With
the increasing demand for intelligent systems, such as robots and other human-computer
interaction systems, the ability to recognize and respond to users’ emotions based on their body
gestures has become a critical component [
The micro-gesture analysis by computer vision techniques has attracted much attention
in recent years. Micro-gesture analysis can mainly be divided into two classes [
In this paper, to locate and recognize the micro-gestures from a long skeleton sequence, we propose a deep network for detecting micro-gestures by integrating the graph-convolution and multiscale Transformer encoders. We utilize a graph-convolution Transformer module based on hypergraphs and hyperedges to extract motion features of 2D skeleton sequences. Then the hierarchical multiscale features are obtained by a feature pyramid module. We further employ a multiscale Transformer module to model the similarity between micro-gesture frames. The classification and regression branches are finally decoupled to achieve accurate location and category. These Transformers are trained in a two-stage strategy and combined to perform the spotting. The main contributions of this paper can be summarized as: • We design a deep network for MiG online recognition for the first time, which integrates the graph-convolution and multiscale Transformer encoders. • We explore a graph-convolution Transformer as a feature extractor and a combination of feature pyramid and local Transformer to locate MiGs, which are trained separately in a two-stage way.
• We achieve the first ranking in the Track 2 of MiGA2023 challenge for online recognition.
For spotting the micro-gestures, we propose a Transformer based network, which mainly consists of four important components: graph-convolution Transformer, hierarchical feature extractor, local Transformer and micro-gesture estimator. The overall architecture is shown in Fig. 1. Given a long sequence, the framework outputs the temporal positions (the starting and ending indexes in a long sequence) and categories of micro-gestures. In the graph-convolution Transformer, motion features are extracted from the long sequence by the graph convolution on hypergraphs and hyperedges. Then, the extracted features are further processed by the
local window_1 (II) Hierarchical Feature Extractor (Ⅳ) Micro-gesture Estimator Concatenation ….. …..
….. 4Upsample 3Upsample 2Upsample 1
local window_n
CLS RE G Transformer C h Local
Local Transformer
Local Transformer
Local Transformer a n n e l A tt e n t i o n
(Ⅰ) Graph-convolution Transformer (ⅠII) Local Transformer
Global Pooling Temporal Conv
×L HyperSA 1 …
C o n c a t e n a it o n
clip_1 clip_n … … raT yH n ep 1 fs … rreom rrgahp … (I) graph-convolution Transformer, (II) hierarchical feature extractor, (III) local Transformer, and (IV) micro-gesture estimator. hierarchical feature extractor to obtain multiscale encoding features, which is fed to local Transformer for modelling the correlation between frames within an inner window. Finally, the interval of micro-gesture is predicted by decoupling classification and regression branches.
The performance of micro-gesture online recognition depends on the ability to capture subtle
motion information in spatial and temporal dimensions. Therefore, the selection of backbone
model plays a critical role in determining the detection performance. In the field of image
processing, it is widely recognized that pretrained classification models can serve as the backbone
of downstream tasks to extract features, such as object detection. Drawing inspiration from this,
we also choose a video recognition model as the backbone for our proposed method. Although
any graph-convolution network can be used as the backbone, in order to efectively process
the 2D skeleton sequence, we choose to exploit the hypergraph Transformer [
The use of hypergraphs and hyperedges in hypergraph Transformer (Hyperformer) allows for a more comprehensive representation of the input data, enabling the model to capture subtle nuances in the skeletal point relationships and structures that are crucial for accurate microgesture detection. Currently, hypergraph Transformer is primarily utilized for macro-action recognition, while there exists the significant diference between micro-gesture and behavioral action. To address this issue, we choose to train the Hyperformer on the iMiGUE dataset in the first stage, which uses the sequence clips of the recognition task (Track 1) to learn the parameters. The trained model is then utilized as a one-stage feature extractor to extract the motion-aware features, which can be matched with various micro-gesture spotting networks to achieve precise micro-gesture location. Given a pre-segmented clip with frames, the feature extracted by the trained Hyperformer with the input ∈ R × × can be embedded as ∈ R(/8)× , where and represent the number and characteristic dimensions of skeletal points, respectively. We choose to use fixed-length sliding window for solving the problem of varying sequence lengths. Therefore, we concatenate the features of diferent small fragments in the time dimension to obtain the motion features in a sliding window, which is fed into the subsequent module. The concatenation operation is given by: = (1, 2, · , ) (1) where is the number of clips in a sliding window, can be embedded as ∈ R(× /8)× .
As the diferent durations of micro-gestures exist in a long sequence, hierarchical feature pyramid is beneficial to capture diferent temporal window lengths (multiscale information). The block module of the pyramid in our network shown in Fig. 1 (II) is similar to the C3 module in YOLOv5 1 but with some key diferences. Specifically, we utilize 1D convolution with kernel size of 3 × 1 and stride of 1, followed by layer normalization and SiLU activation function. In order to ensure that the model can capture micro-gesture features with a short duration, the stride of the first layer in the feature pyramid is set to 1, other layers are set to 2. Subsequently, we acquire multiscale features through linear upsampling and concatenation operations, which enable the integration of rich contextual information and enhance the representation can be embedded as { ∈ R( ′ /2− 1)× , = 1, 2, 3, 4}. ability of the features. Given a feature ∈ R ′ × extracted from the previous module,
Since the occurrence of micro-gesture is often inseparable from the contextual frames, we
employ the attention mechanism to measure the similarity between frames and model the
dependency of frames. Transformer [
Given ∈ R × , is utilized to project encoded representations of Query(Q), Key(K), and Value(V) by using ∈ R× , ∈ R× , ∈ R× , which are given by: = × , = × , = × (2) 1https://github.com/ultralytics/yolov5 Then multi-head attention (MHA) will be applied in a local window by the following operations: (, , ) = (ℎ0, . . . , ℎ) , ℎ = max ︂( )︂ √ (3) (4) where is the number of heads, is the parameter matrix, , and respectively represent , and in the -th local window. Then the results of each window MHA are concatenated in the time dimension to obtain the encoded results, which is given by: = ∑︁ ( (, , ))
dimension. where ∈ R × and (· ) the concatenation of MHA results in the time
Finally, the encoded features of local Transformer are fed to the estimator module to predict the
location and category of micro-gestures. The estimator module consists of decoupling regression
and classification branches, which is shown in Fig. 1 (IV). The former predicts the distance
to the starting and ending frames of the micro-gesture at each point in the time dimension,
while the classification branch is responsible for identifying the category to which it belongs. In
order to obtain classification and regression related feature information, we employ the channel
attention mechanism and apply it before sending the features to the head. To prevent overfitting,
we enforce weight sharing among these attention layers. To further achieve accurate localization
of the gesture interval for micro-gesture detection, we adopt the approach proposed by [
R × 2, and the classification branch can be embedded as ∈ Given an encoded feature ∈ R × , the output of the regression branch can be embedded as R × , where is the number of categories of micro-gestures. In the second stage of model learning, the local Transformer and estimator are jointly trained by the training data of online recognition.
Dataset. Micro-Gesture Understanding and Emotion analysis (iMiGUE) dataset [
⋂︀ ℎ ⋃︀ ℎ
≥ 1 − =
2 2 + +
Metric. The true positive (TP) per interval in one sequence is defined based on the intersection between the spotted interval and the ground-truth interval. The spotted interval is considered as TP if it fits the following condition: (5) (6) where takes 0.3, and ℎ represents the ground truth of the micro-gesture interval (onset-ofset). F1-socre is then used to evaluate the performance of the model, which is given by: where and represent the false positive and false negative, respectively.
The hypergraph Transformer model is firstly trained using pre-segmented micro-gesture data
from the iMiGUE dataset with a total of 200 epochs trained. Then, the fully connected layer
of hypergraph Transformer is removed, and the model is utilized as the feature extractor for
detection network. In Fig. 1 (I), the number of layers in graph-convolution Transformer is
10. The length of one clip fed into feature extractor is 8, the overlap value is 2. We set the
length of the sliding window to 512. In local Transformer, the local window size is set to 8,
the overlap value is set to 4. The local Transformer and estimator are secondly trained for
200 epochs with a cosine learning rate schedule and 5 warmup epochs. We use Adam as an
optimizer, where the initial learning rate is 1 − 4. The mini-batch size is 32, and the weight
decay is 5e-4. None-Maximum Suppression (NMS) [
As no baseline approach has been reported in the past, we only report the final performance on
iMiGUE to evaluate the influence of two important components, i.e., the hypergraph Transformer
and multiscale Transformer. Table 1 presents the performance of using other components to
replace the above two components for micro-gesture online recognition. When HD-GCN
[
The visualization results of micro-gesture online recognition are demonstrated through two sets of skeletal sequences in Fig. 2. The first set shown in Fig. 2 (a) displays accurately spotted micro-gesture, while the second set depicted in Fig. 2 (b) demonstrates micro-gesture that is spotted incorrectly with an IoU value of 0.15. The results show that our method still faces challenges in accurately locating some samples.
... ...
… … ✓ 7493 5669
In this paper, we proposed a Transformer based model for micro-gesture online recognition, which integrates graph-convolution and multiscale Transformers. Our proposed method achieved excellent performance on the iMiGUE dataset, but it is important to note that the development of micro-gesture online recognition is still in its early stages and there is much room for improvement in terms of detection accuracy.
This work is partly supported by the Natural Science Foundation of Chongqing (No. CSTB2022NSCQ-MSX0977), and the Key Research and Development Program of Shaanxi (Nos. 2021ZDLGY15-01 and 2023-ZDLGY-12).