Memes, as a new means of creative expression on social networks, provide an appealing multi-modal form of communication. However, some memes are being used to express hatred, which can take a toll on people's mental health and on societal cohesion. This year's Memotion 3.0 challenge provides an English and a mixed Hindi-English meme dataset for three classification tasks: Task A is sentiment analysis to classify a given meme as positive, negative, or neutral. In Task B, emotion classification, a meme should be identified as humorous, sarcastic, ofensive, or motivational. Finally, Task C asks to predict the intensity of the emotion classes in Task B. Both text and image data play a role in the identification and classification of hateful memes. While such multi-modality can be helpful in many contexts, here, it also increases the challenge of the classification tasks due to the nature of memes, which often achieve their humorous efects through juxtaposition and irony. To address this dificulty, we adopt a multi-headed self-attention mechanism to integrate the text and image information in a learned, task-adapted manner. The gradient blending algorithm prevents overfitting issues in the multi-modal model. Our uni-modal models, which feed into the attention mechanism, are based on the CLIP model due to its outstanding performance on zero-shot classification tasks. Ultimately, with an ensemble strategy of our two best-performing models, our submission only reaches a 0.3289 weighted F1 score on sub-task A, but it ranks 1st on the two final Tasks B and C, with respective scores of 0.7977 and 0.5982. 1 1Our code will be made available at: https://github.com/wentaoxandry/Memotion3.0_challenge.git De-Factify 2: 2nd Workshop on Multimodal Fact Checking and Hate Speech Detection, co-located with AAAI 2023. 2023 Washington, DC, USA $ wentao.yu@rub.de (W. Yu); dorothea.kolossa@tu-berlin.de (D. Kolossa) https://cognitive-signal-processing.de/index.php/team/ (W. Yu); https://www.tu.berlin/en/mtec (D. Kolossa) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CPWrEooUrckResehdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CEUR Workshop Proceedings (CEUR-WS.org)
It is well-known that multi-modal machine learning can vastly outperform uni-modal learning,
at least when the system is set up appropriately. For example, in audio-visual speech recognition,
visual information can complement speech signals to significantly improve recognition rates [
In this work, we consider transfer learning to customize two multi-modal models based
on the Transformer model: the CLIP model [
The paper is organized as follows: Section 2 introduces the related solutions to the task. Our system framework is described in Section 3, followed by the experimental setup in Section 4. Finally, our results are shown and conclusions are drawn in Sections 5 and 6.
The transformer model [
Recently, contrastive learning has drawn much attention due to its outstanding performance
on zero-shot prediction [
F, = encoder(T),
F, = encoder(I),
where and are the attention dimension of text and image encoder, respectively. The
classiifcation features are mapped to the same dimension and normalized with an l2 regularization.
The contrastive logits x are derived as their scaled, pairwise cosine similarities:
x = (‖W · F,‖2 · ‖ W · F,‖2 ) × ,
where is a learnable parameter. As in [
CE CE and image classification features and multi-modal models. where CE is the cross-entropy, ̂y︀ = softmax(x) and ̂y︀ = softmax(x). Finally, the learned text =0 =1 F,, F, are used in our proposed CLIP-based text, image, (1) (2) (3)
Classi er1 Fcls;t xt + Classi er2
h RNN
Et CLIPtext
T (a) Text model are utilized in the multi-modal model, where is the number of patches of the image.
Figure 2 depicts the proposed multi-modal model. The sequence embeddings from the text and image model in Figure 1 are concatenated along the sequence dimension as multi-modal embeddings
E = [E; E], where E ∈ R(+)× . x, where
The complete embedding sequence E is fed into six multi-head attention (MHA) blocks. We removed the MHA block's residual connection and dropout layer based on our experimental results. Finally, the classifier, which has the same structure as the text and image classifiers, uses the multi-modal classification embedding F ∈ R2 to obtain the multi-modal logits F = [Ẽ︀[0, :], Ẽ︀[0, :]]. (4) (5) (6) Ẽ︀[0, :] and Ẽ︀[0, :] are the first classification embeddings after 6 MHA blocks.
xmulti
The Memotion 3.0 challenge has three sub-tasks. Task A is to classify a meme as positive,
negative, or neutral. In Task B, a given meme should be identified as humorous, sarcastic,
ofensive, or motivational. It is a multi-label classification task, so that a meme can have more
than one category. Finally, Task C asks to predict the intensity of the emotion classes in Task B.
In our work, we only optimized the models for Task A and Task C. Task B results are obtained
1https://github.com/schesa/ImgFlip575K_Dataset
from Task C. For example, we consider a meme as Humorous if it is classified as F, VF, or H
(detailed in Table 2) and vice versa. Since the Memotion3 dataset contains English as well as
mixed Hindi-English memes, we perform back-translation to Hindi and then to English with
the Python translators package.
All models are trained using the PyTorch library [
In this work, we adopt Gradient-Blending [24] (GB) to reduce the efect of overfitting. The multi-modal model (Figure 2) is based on the text and image model (Figure 1). Therefore, the text and image model logits x and x are also available in the multi-modal model. Taking the gradient of the blended loss = ∑︁ CE, (7)
where ∈[text, image, multi-modal], produces the blended gradient. It should be emphasized that the multi-modal predictions are only obtained from the multi-modal logits x. Finally, Table 3 gives an overview of the use of the loss functions in training all models.
We use the Python RAY 2 package to find the best-performing hyperparameters. The training process is carried out on NVIDIA's Volta-based DGX-1 multi-GPU system, using 3 TeslaV100 GPUs with 32 GB memory each.
2https://github.com/ray-project/ray
This work considers the CLIP-text, CLIP-image (in Figure 1), CLIP-multi (in Figure 2), and OSCAR models. For better performance, majority voting is adopted to ensemble diferent models' decisions. Ensemble-1 fuses the prediction decisions of the candidate models CLIP-text0, CLIP-image0, CLIP-multi0, and OSCAR0, while Ensemble-2 also takes CLIP-text1, CLIP-image1, CLIP-multi1, and OSCAR1 into consideration. We iterate over all possible model combinations and adopt majority voting on the validation set to find the best performance model combinations. Then, these combinations are used to fuse the test set predictions.
Table 4 lists the weighted F1 score on the validation set. For Task A ("Overall" column in Table 4), the CLIP-text model performs better than the CLIP-image model. The score of the CLIP-multi setup lies between those of the former two models. Ensemble-2 improves the weighted F1 score to 0.4453. The model for motivation classification has scores above 0.9, because the binary classification dataset is imbalanced. Comparing the best-performing text and image models (CLIP-text0 and CLIP-image0), the image model shows a slightly better performance in Task C. The CLIP-multi0 model without GB training performs far worse than its gradient-blending counterpart. Overall, Ensemble-2 shows the best performance in Task A and Task C. Ultimately, the strategy of ensembling the top two models yields a 0.3289 (5th) weighted F1 score on Task A, 0.7977 (1st rank) on Task B and 0.5982 (also 1st rank) on Task C.
This work proposes a multi-modal CLIP-based meme classification system, which owes its capabilities on this rather small dataset to the outstanding zero-shot performance of the CLIP model. The text model combines the CLIP model text encoder with 2 BiLSTM layers; the image model is fine-tuned on the Memotion 3.0 dataset. The proposed multi-modal model integrates the text and image embeddings from the text and image encoders in 6 multi-head self-attention blocks. Gradient blending prevents the fusion model from overfitting. The OSCAR model is used both as a baseline model and as a participant model in our ensemble strategy, which further serves to improve the system performance. Our ensembe of the top two models yields a clearly better accuracy than one single model, winning Task B and C in the Memotion 3.0 challenge. The experimental results of the challenge do indicate, however, that sentiment analysis in memes is dificult for machine learning. The next goal of our work is therefore to develop mechanisms for understanding multi-modal, contrasting information, e.g. conveying irony, to improve sentiment classification performance for memes and social media posts.
The work was supported by the PhD School ”SecHuman - Security for Humans in Cyberspace” by the federal state of NRW, and partially funded by the Deutsche Forschungsgemeinschaft (DFG – German Research Foundation) [Project-ID 429873205] and by the German Federal Ministry of Education and Research [”noFake”, Grant No: 16KIS1519]. The authors are responsible for the content of this publication. N. Gimelshein, L. Antiga, et al., Pytorch: An imperative style, high-performance deep learning library, Advances in Neural Information Processing Systems (2019). [22] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101 (2017). [23] T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollár, Focal loss for dense object detection, in:
Proc. ICCV, 2017, pp. 2980–2988. [24] W. Wang, D. Tran, M. Feiszli, What makes training multi-modal classification networks hard?, in: Proc. CVPR, 2020, pp. 12695–12705.