Memes are commonly used in social media platforms for humour. Generally, memes consist of an image and embedded text. Memes can be used to spread hate or misinformation, hence it is important to study them. The Memotion task [1] conducted at SemEval 2020, released a data of 10k memes annotated with sentiment label (task A), emotion label (task B) and emotion intensity label (task C). It received ≈ 30 run submissions and 27 papers. However, the best f1 scores were only 0.35, 0.51 and 0.32 respectively for task A, task B, and task C, which shows the need for more extensive research on this topic. In this paper, we release a new dataset, Memotion 2 which has 10k annotated memes along the same directions as Memotion 1.0 This paper detailed baseline system on the Memotion 2.0 data.
Memes, usually created with image and/or text, have become a highly popular medium of
communication on the internet. Over the past decade, memes have become an integral part of
the internet culture, giving communities the power to make their voices heard to large audiences
in a matter of few hours. Hence, to understand a community’s opinions and alignment with
their causes, a good start would is to be conscious of the memes shared by the community [
Fig 1 shows the rapid growth in using memes over the past few years and its success can be attributed to the increased usage of internet by people from all walks of life. Many memes are used for just fun and games, but they are also a powerful medium to voice a community’s opinion (or alignment with a cause). With the freedom to express opinion anonymously comes a varying definition of ”normal”. As dificult as it already is to classify content from unimodal data like simple texts/tweets; including multi-modal data (eg. memes) into the mix only further complicates the problem, given the fact that they are highly unconventional and don’t have a ifxed template or modality.
A flip side of this powerful medium is that it can be misused to spread hatred in the community.
The growing trends in hate speech on social media commensurate with the increased quotidian
usage of memes. Considering the efects hate speech might have on individuals, detecting and
preventing such content from being spread is important. Williams et al. 2016 [
Humans show various emotions like rage, disgust, grief, serenity, fear, etc. and each exhibits them in varying levels of intensity. Our dataset, Memotion 2.0, focuses on quantifying emotion and their intensities into discrete labels. It also has labels corresponding to the sentiment of a meme. Memotion 2.0 adds to the previous iteration (Memotion 1.0) by providing another set of 10k memes from various social media websites. In this iteration, the collected memes are more widespread in terms of topics ranging from history to world wars, politics, etc. Popular social media sites like Reddit, Facebook, Imgur, Instagram were used as sources for memes. These websites consist of individuals from diferent countries, religions and ethnic groups. Hence, we believe that this enables the analysis of multi-modal memes with respect to sentiment analysis and emotion detection using our dataset more closer to general human perception.
The paper is organised as follows: We describe the related work and the task in section 2 and 3 respectively; Section 4 contains the details of the dataset we collected for memotion analysis: Memotion 2.0 ; followed by a brief description of baseline models 5 and their results in 6. We conclude with the mention of future work and limitations, in section 7.
Past years have seen a lot of work related to analysing social media content and detecting
emotions, profanity and other such attributes. Majority of hate speech datasets are of textual
modality [
Due to their reliance on identifying n-grams, phrases or textual patterns, these datasets do
not account for subjective bias [
The growing ubiquity of Internet memes on social media platforms suggests that it is more than important to consider such multi-modal content. MMHS150K [16] is a dataset collected from Twitter using Hatebase terms. It contains 150,000 tweets and images manually annotated into six classes based on the type of hate speech. Haoti et. al. [17] and Hosseinmardi et. al. [18] provide annotated datasets from Instagram with posts and comments targeted towards combating cyberbullying. Sentiment analysis datasets such as [19] and [20] classify videos or image-text pairs into ”positive” or ”negative” labels.
While there are several datasets that facilitate computation of social media data, analysing
memes has received relatively less attention. MultiOFF [21] is an annotated dataset with 743
memes from Kaggle. While it does have image and text captions, the dataset only has binary
labels i.e. ”ofensive” and ”non-ofensive”. Another notable dataset is hateful memes dataset by
facebook [22]. The memes are from social media groups in the United States and they were
annotated using a specific definition of hate speech. Each meme can have multiple labels and
the labels are defined for multimodal and unimodal hate speech separately. The dataset is of
size 10k but has some reconstructed i.e. artificial memes. The Memotion 1.0 task at SemEval
2020 [
We release a dataset of 10k annotated memes. Each data point consists of of an image and
text associated with it along with labels for each sub-tasks. Similar to Memotion 1 [
the type of emotion expressed. The categories should indicate if the meme is humorous, sarcastic, ofensive and motivational. A meme can belong to more than one category. • Task C: Scales/Intensity of Emotion Classes - The third task is to quantify the extent to which a particular emotion is being expressed. Fig 2 mentions about intensities of each emotion.
Tasks B and C can be clearly understood by looking at the meme in Figure 4.
In this section we describe the data collections and data annotation process along with a brief summary of the data distribution.
As established in prior sections, memes are highly complex form of data, and it is necessary that we collect them from a wide range of categories. We shortlisted several topics of interests: like politics, religion, sports etc.- and manually downloaded the memes. We have also used a Selenium based web-crawler for a part of data collection, followed by extensive cleaning of the data. All the memes have been collected from public domains, and in order to avoid copyright claims, the source-urls have been attributed in the dataset. We used the Google Vision API1 to extract the OCR text from the images.
After collecting data, we turned to Amazon Mechanical Turk(AMT) 2 workers to get it annotated. An annotator has to choose from the following and annotate each meme for all the said ifelds. For this purpose, they use an interface built by us, as shown in Fig 2. For task A, the annotators were asked to judge what the person who created the meme intended it to be (positive/negative/neutral). For task B and C, the annotators were asked to mark their opinions on the emotion of the meme. The true afect of a meme on an individual depends on their perception of several aspects within the society, and could vary a lot from another individual. We solve this problem with each meme being annotated by 3 diferent workers. Based on the majority voting scheme, the final annotations are adjudicated.
The dataset consists of 10,000 images divided into a train-val-test split with 8500-1500-1500 images. Each meme is annotated for its Overall Sentiment (positive, neutral, negative), Emotion (humour, sarcasm, ofense, motivation) and Scale of Emotion (0-4 levels). Along with the said attributes, we also introduce a new attribute called ”classification_based_on”, which denotes if the annotation has been made on the basis of image only, text only, or both image and text data. Fig.5. shows the distribution of memes across all the 20 labels.
The statistical summaries in Fig. 6 and Fig.7 show the overlapping emotions in memes, which proves the aforementioned challenges. Several interesting points can be inferred from the tables like many ofensive memes are funny. It can also be observed that many of the memes are funny and non-motivational.
In this section, we describe the various baseline models that we tried.
Memes are known to convey information with both image and text. However, it is noticed that many memes have the same template (image) but diferent text and by implication, diferent messages to convey. Recognizing of the emotion induced in such memes would require accurate modelling of the textual influence. For this purpose, we evaluate the afect of textual features using LSTM. The additional controlling knobs in LSTM, makes it more suitable than simple RNNs for these classification tasks. Table 1. shows the baseline Weighted F1 scores of the model on all three memotion tasks.
Experiments are carried out considering the multi-modal features of memes i.e both image and OCR text. BERT is a widely known attention model that provides State-Of-The-Art results on various text related tasks. We make use of BERT to extract features from the OCR text. The text and image features are represented by the CLS output of BERT and the final MLP layer output of ResNet-50 respectively. We concatenate the obtained features and use two layers of MLP for classification. Scores on Tasks A, B and C are reported in table 1.
Baseline results in Table 1 show Weighted F1 scores for each task and sub-task. It can inferred from the table that multi-modal Image+Text models with scores 43.90%, 73.72% and 51.08%, perform better than the Text-Only models with scores 28.29%, 31.38%, 50.94% for Task A, Task B, and Task C, respectively. The table shows the inconsistency of performance throughout Task B and Task C sub-tasks upon using Text-Only model, unlike Image+Text model. Although it can’t be concluded, one can clearly see the importance of using multi-modal data for the said tasks. This dataset will be publicly available and we leave it to the future works, to come up with novel methods which dig deeper into Memotion Analysis and provide statistical insights to multi-modal relations of a Meme.
Task Task-A Task-B Task-C
Class Sentiment Humour Sarcasm Ofensive Motivation Average Humour Sarcasm Ofensive Motivation Average
In this paper, we introduce a novel task of identifying memes and classifying them using a multimodal setting. To the best of our knowledge, this is the first large-scale multimodal dataset for meme classification. In order to provide a fine-grained and extensive analysis of tweets We provide gold-data for three diferent verticals namely- sentiment analysis, emotion classification and intensity of emotion. We also provide text only and multimodal baselines for each of these tasks. While the text-only model uses LSTM, the multi-modal model uses ResNet-50 + BERT, which is a recent SOTA model on many popular image-text tasks like captioning, VQA, phrase grounding etc. The performance of these models indicate that incorporating both images and text for all the tasks improves performance, however, it must be noted that our models are only preliminary and more innovative methods will improve performance furthermore. In the future, we intend to extend our work by releasing datasets for memes of other languages, identification of targets of hate and difusion patterns etc. [16] R. Gomez, J. Gibert, L. Gomez, D. Karatzas, Exploring hate speech detection in multimodal publications, 2019. arXiv:1910.03814. [17] H. Zhong, H. Li, A. Squicciarini, S. Rajtmajer, C. Grifin, D. Miller, C. Caragea, Contentdriven detection of cyberbullying on the instagram social network, 2016. [18] H. Hosseinmardi, S. A. Mattson, R. I. Rafiq, R. Han, Q. Lv, S. Mishra, Detection of cyberbullying incidents on the instagram social network, 2015. arXiv:1503.03909. [19] L.-P. Morency, R. Mihalcea, P. Doshi, Towards Multimodal Sentiment Analysis: Harvesting Opinions from The Web, in: International Conference on Multimodal Interfaces (ICMI 2011), Alicante, Spain, 2011. URL: http://ict.usc.edu/pubs/Towards%20Multimodal% 20Sentiment%20Analysis-%20Harvesting%20Opinions%20from%20The%20Web.pdf. [20] A. Hu, S. Flaxman, Multimodal sentiment analysis to explore the structure of emotions, Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery |& Data Mining (2018). URL: http://dx.doi.org/10.1145/3219819.3219853. doi:10.1145/3219819.3219853. [21] S. Suryawanshi, B. R. Chakravarthi, M. Arcan, P. Buitelaar, Multimodal meme dataset (MultiOFF) for identifying ofensive content in image and text, in: Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, European Language Resources Association (ELRA), Marseille, France, 2020, pp. 32–41. URL: https://aclanthology.org/2020. trac-1.6. [22] D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, D. Testuggine, The hateful memes challenge: Detecting hate speech in multimodal memes, 2021. arXiv:2005.04790.