This paper addresses the “Behavior-Based Emotion Recognitio” task in the IJCAI 2025 Micro-Gesture Hidden Emotion Understanding Challenge (MiGA) by proposing a tennis match outcome prediction method that integrates an emotional prior module, dual-channel image modeling, and a class balancing strategy. The task involves predicting the match outcome (win or loss) from post-match interview videos, where challenges include complex emotional expressions, subtle behavioral cues, and severe class imbalance. To address the controlled emotional expressions and occluded facial cues, we first introduce a DeepFace-based emotion recognition module that performs frame-by-frame facial analysis to extract multidimensional emotional distributions and generate semantic tendency scores as prior information for the main model. On this basis, a dual-channel image feature extraction structure is designed, modeling global behaviors and local expressions separately using a shared ResNet34 backbone, thus enhancing the model's perception and discrimination of key visual cues. To mitigate the severe imbalance-over 70% of the training samples belong to the “win” class-we propose an intra-class partitioning strategy. The majority class is split into two subsets, each paired with the minority class to train two structurally identical but parameter-independent image classifiers, thereby implicitly achieving class balance at the data level. Finally, the two image classifiers are integrated with the emotional prior module, and the final prediction is determined via majority voting, significantly improving system robustness and generalization. Experimental results demonstrate the proposed method's efectiveness in addressing challenges such as class bias, emotional concealment, and sparse signals, validating its potential in behavior modeling and complex emotion understanding scenarios.
With the continuous advancement of artificial intelligence, video-based human behavior analysis and
emotion understanding have become pivotal research areas in computer vision, showing great value in
real-world applications such as sports analytics, psychological assessment, educational feedback, and
media analysis [
The IJCAI 2025 Micro-Gesture Hidden Emotion Understanding Challenge (MiGA) introduces a highly realistic task setting—predicting athletes’ match outcomes (win or loss) based on post-match interview videos. Essentially a behavior-driven video classification problem, the task must be solved under significant uncertainty and data bias, making it extremely challenging. Specifically, the task presents two major dificulties: first, there is significant individual variation in emotional expression, and some losing athletes still maintain a positive demeanor, making it dificult to directly correlate facial emotion signals with match outcomes. Second, the training set is severely imbalanced, with over 70% of the samples labeled as “win”, which easily leads to model overfitting and bias during training.
To tackle these challenges, this paper proposes an integrated prediction method that combines
an emotional prior module, a dual-channel image feature extraction mechanism, and an intra-class
partitioning strategy, aiming to improve the model’s capacity to identify latent emotions. The overall
architecture is illustrated in Figure 1. Specifically, we extract both full-body and facial image frames
from interview videos to construct two separate channels for learning behavioral and facial expression
features, respectively. Temporal features from both channels are extracted using a shared ResNet34
backbone. A DeepFace-based emotion recognition module [
The main contributions of this paper are as follows: - We introduce a DeepFace-based emotional prior module to improve the recognition of complex and implicit emotional states;
- We construct a dual-channel image feature modeling framework incorporating both full-body behavior and facial expressions, and apply ensemble voting to achieve robust classification under high uncertainty and noise;
- We propose an intra-class partitioning and dual-model training strategy to efectively alleviate overfitting and prediction bias caused by severe class imbalance;
Experimental results demonstrate that our method performs well in modeling realistic complex interactions between behavior and emotion, validating its potential in multi-source information fusion and imbalanced data learning scenarios.
In recent years, video-based behavior analysis and emotion recognition have become prominent research areas in computer vision, with wide applications in human-computer interaction, intelligent surveillance, and afective computing. In unstructured environments, emotional expressions are often subtle and complex, posing significant challenges to model accuracy and decision stability. Three major obstacles commonly arise: (1) class imbalance in training data, (2) implicit and ambiguous emotional signals, and (3) instability in model decisions under noisy conditions. To address these issues, prior studies have explored solutions from three main perspectives: class balancing techniques, emotional prior integration, and ensemble modeling strategies.
To address class imbalance, SMOTE (Synthetic Minority Over-sampling Technique) [
Emotion-related cues have been leveraged to support behavior recognition, particularly when primary
features are ambiguous. Zhao et al. [
Ensemble learning has proven efective in improving model robustness and generalization in expression
recognition tasks. Hasani and Mahoor [
The overall architecture is illustrated in Figure 1. Through the coordinated operation of multiple modules, the framework systematically addresses challenges posed by complex emotional expressions, weak behavioral cues, and imbalanced class distributions. The key components are described as follows.
The preprocessing pipeline consists of three stages: frame sampling, temporal segmentation, and multi-scale feature construction. To reduce redundancy from the original 25 fps videos, we adopt sparse sampling at 2.5 fps, preserving temporal context while lowering computational cost. The downsampled video is split into 128-frame windows—overlapping or not—as training samples, each inheriting the original label. To enhance temporal representation, each segment is divided into 16 intervals with one frame randomly selected per interval. Additionally, we extract two crops from each sampled frame: a global full-body image and a localized face region detected via OpenFace. This dual-view strategy supports the later dual-channel modeling and boosts sensitivity to both macro behaviors and ifne-grained emotional cues.
In post-match sports interview scenarios, emotional states often correlate with match outcomes: winners tend to exhibit joy, excitement, or relaxation, while losers may show sadness, tension, or suppression. This emotional-outcome association serves as a natural prior for classification.
Let denote the frequency of the -th emotion in the video, and +, − be its positive and negative weights. The total positive and negative tendency scores are calculated as:
Positive Score = ∑︁ + ·
Negative Score = ∑︁ − · (2) If the Positive Score exceeds the Negative Score, the emotional prior module predicts a win; otherwise, a loss.
This emotional prior provides semantically rich external knowledge, ofering supplementary judgment signals that improve the system’s discrimination capability and generalization performance.
While emotional priors often correlate with match outcomes, there are scenarios where this relationship is disrupted due to emotional masking or individual diferences. For example, some athletes may maintain a positive appearance after losing due to professionalism, while others may appear emotionally subdued despite winning. To enhance the model’s capability in recognizing such latent behavioral signals, we design a dual-channel image modeling module that jointly processes full-body and facial images using a shared ResNet34 backbone. Each channel extracts temporal features independently, with shared weights to reduce model complexity and enforce feature consistency.
The scene channel encodes macro-level visual cues such as posture, movement rhythm, and body language. The face channel focuses on micro-level expression features and subtle emotional variations.
Let fscene ∈ R and fface ∈ R denote the feature vectors extracted from the scene and face channels, respectively. These features are fused via element-wise addition:
The fused representation is then passed through a linear classifier for binary prediction: ffused = fscene + fface ˆ = · f fused + where ∈ R1× and ∈ R are the parameters of the output layer, and ˆ ∈ R denotes the predicted score for the win/loss classification.
This design efectively integrates both global and local visual cues, enhancing the model’s robustness and discriminative power under complex emotional masking conditions. (1) (3) (4)
In the iMiGUE dataset used for the MiGA challenge, a severe class imbalance exists—“win” samples far outnumber “loss” samples—leading to training bias and degraded generalization performance. To address this issue, we propose an intra-class partitioning strategy at the data level, combined with a dual-model training scheme. Specifically, the majority class (“win”) is randomly divided into two disjoint subsets of approximately equal size. Each subset is then paired with the complete “loss” class to form two balanced training datasets. Two independent ResNet34 models are trained on these datasets, which we refer to as BalanceNet A and BalanceNet B.
This strategy avoids the distributional distortion introduced by traditional oversampling or undersampling techniques. Moreover, by training on distinct data partitions, it introduces model diversity, which enhances the efectiveness of ensemble learning and reduces the risk of overfitting. In summary, the proposed strategy mitigates class imbalance at both the data and model levels, significantly improving fairness and generalization across the framework.
To further enhance robustness, we adopt a majority-voting based multi-model ensemble consisting of three components: BalanceNet A, trained on “win” subset A and the full “loss” class; BalanceNet B, trained on “win” subset B and the full “loss” class; and EmotionScoreNet, an emotional prior module that generates predictions based on DeepFace emotion distribution scores.
Each model independently predicts the match outcome for a given video. The final decision is determined through majority voting: if at least two out of the three models agree on a class label, that label is taken as the final prediction.
This ensemble approach combines the complementary strengths of behavioral modeling and semantic emotion priors , significantly enhancing the system’s robustness in noisy or ambiguous conditions. Experimental results confirm that the ensemble consistently outperforms any individual model, demonstrating the efectiveness of the proposed architecture in complex emotion-behavior modeling scenarios.
iMiGUE dataset [
The task is formulated as a binary classification problem. A label of 1 indicates a “win”, and 0 indicates a “loss”. Classification accuracy is adopted as the primary evaluation metric.
All experiments are conducted using the PyTorch framework. Input images are uniformly resized to 224 × 224 pixels. The classification head comprises two fully connected layers and is optimized with the Adam optimizer. The learning rate is set to 3 × 10 −4 for the classifier and 3 × 10 −5 for the ResNet-34 backbone.
The training is conducted in two phases: for the first 100 epochs, all parameters are trainable; in the ifnal 30 epochs, only the classifier is fine-tuned. During inference, the final prediction is determined by majority voting across the three models.
Our method ranked third in the IJCAI 2025 MiGA Challenge, demonstrating its strong capacity to tackle complex, real-world emotion recognition tasks. To analyze the contribution of each module, we conduct a comprehensive ablation study across three aspects: class balancing strategy, dual-channel feature modeling, and multi-model ensemble.
Given an approximate class imbalance between the “win" and “loss" categories, naive oversampling—which duplicates minority class samples to balance class counts—leads to a decline in accuracy to 53.85%. This suggests that oversampling may distort the original data distribution and introduce overfitting.
Loss reweighting slightly improves upon the baseline, reaching an accuracy of 56.73% by assigning greater gradient weight to the minority class during training. While this method provides some relief, its impact remains limited and indirect, particularly for deep models.
In contrast, the proposed intra-class partitioning strategy divides the majority class into two disjoint subsets and trains two identical yet independent models on the resulting balanced data. This method neither alters the data distribution nor relies on synthetic samples. It achieves the highest accuracy of 61.54%, demonstrating superior efectiveness and robustness in behavior-based emotion recognition for tennis match outcome prediction.
To evaluate the impact of dual-channel image features, we compare three input settings—scene only, face only, and combined scene+face—under both balanced and imbalanced training. Results are shown
In the balanced setting, dual-channel input significantly outperforms both single-channel variants, reaching 61.54%. This validates our feature modeling strategy, where the ResNet34 backbone jointly encodes scene-level actions and facial micro-expressions. Their fusion efectively captures complementary behavioral signals.
Even under class imbalance, the dual-channel input remains more robust than either single modality, achieving 55.77%. These results highlight the synergy between proper class balancing and dual-view feature design.
It is worth noting that the scene-only input yields the lowest performance across both settings, likely due to sparse behavioral cues and background interference. The face-only model improves over scene-only, but still struggles when athletes deliberately suppress facial expressions. In contrast, combining face and scene provides a balanced and expressive representation, demonstrating superior reliability.
We further assess the contribution of ensemble learning by comparing the individual model accuracies with the ensemble result. Table 4 presents the results.
Both BalanceNet A and B are trained on diferent partitions of the majority class but share identical architectures. Their accuracy reflects consistency and diversity. The EmotionScoreNet, based on DeepFace, provides semantic-level priors and performs comparably.
When combined via majority voting, the ensemble model achieves 63.46% accuracy, surpassing all individual models. This demonstrates that ensemble learning successfully leverages structural diversity and complementary information, especially under ambiguous emotional signals.
This paper presents a behavior-based emotion recognition framework for class-imbalanced video data, proposed for the IJCAI 2025 MiGA Challenge. The goal is to perform emotion recognition and match outcome prediction in post-match interview scenarios. The method comprehensively addresses realworld challenges such as imbalanced data distribution, emotional masking, and weak behavioral signals, by constructing a complete framework that integrates intra-class partitioning, dual-channel image feature extraction, emotional prior modeling, and multi-model ensemble decision-making.
Specifically, the proposed intra-class partitioning strategy alleviates class imbalance without altering the original data distribution, efectively improving the model’s capacity to learn from minority class samples. Meanwhile, the dual-channel feature extraction mechanism, based on full-body actions (scene) and facial expressions (face), integrates both global and local temporal cues, enhancing the model’s sensitivity to complex behaviors and fine-grained emotional signals. In addition, an external emotional prior module based on DeepFace is incorporated to provide complementary decision cues to the visual pathway. Finally, a multi-model voting ensemble is used to improve overall robustness and generalization. Experimental results show that the proposed method achieves strong performance in the MiGA Challenge and exhibits significant advantages in all ablation studies, demonstrating its efectiveness and practicality in complex real-world tasks.
During the preparation of this work, the author(s) used ChatGPT-4 and Qwen in order to: grammar and spelling check. After using these tools, the author(s) reviewed and edited the content as needed and take full responsibility for the publication’s content.
The work is jointly supported by the National Natural Science Foundation of China under grants No.62472342, and 62272364, the National Science and Technology Major Project under grant No.2022ZD0117103, the provincial Key Research and Development Program of Shaanxi under grant No.2024GH-ZDXM-47, the Research Project on HigherA Education Teaching Reform of Shaanxi Province under grant No.23JG003, the Fundamental Research Funds for the Central Universities under grant No.QTZX25037.
- Our code is available at the GitHub repository: https://github.com/wudidaluobo/MiGA-IJCAI