English. In this paper, we describe and present the results of meme detection system, specifically developed and submitted for our participation to the first subtask of DANKMEMES (EVALITA 2020). We built simple classifiers, consisting in feed forward neural networks. They leverage existing pretrained embeddings, both for text and image representation. Our best system (SNK1) achieves good results in meme detection (F1 = 0.8473), ranking 2nd in the competition, at a distance of 0.0028 from the first classified.
1.1
DANKMEMES
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
For our participation to the first subtask of DANKEMES, we built simple classification models for meme detection.
The main challenge is to effectively combine textual and image inputs. We tried to exploit the ability of pretrained embedding to represent the information present in text and images, paying a limited computational cost.
To quickly build various prototypes of
neural networks, we used Uber Ludwig framework
We trained our models using Google Colaboratory, a hosted Jupyter notebook service, which provides free access to GPUs, with some resource and time limitations. 1.2
The dataset provided for the first subtask has the following features: • File: the name of the .jpg image file. • Date: when the image has first been posted on Instagram. • Picture manipulation: entails the degree of visual modification of the images. Nonmanipulated or low impact changes are labeled 0. Heavily manipulated, impactful changes are labeled 1. • Visual actors: the political actors (i.e. politicians, parties’ logos) portrayed visually, regardless whether edited into the picture or portrayed in the original image. • Engagement: the number of comments and likes of the image. • Text: the textual content of the image. • Meme: binary feature, where 0 represents non meme images and 1 meme images. This is the target label.
The dataset also includes image embeddings.
We discarded Date feature, because it seems irrelevant for meme detection.
Picture manipulation and Meme are simple binary features and do not require preprocessing.
We chose to scale Engagement feature, using min-max normalization.
Visual actors feature was preprocessed using Ludwig approach for sets. We report an extract of the official framework documentation1: “Set features are expected to be provided as a string of elements separated by whitespace.
The string values are transformed into a binary valued matrix of size n x l (where n is the size of the dataset and l is the minimum of the size of the biggest set and a max_size parameter) [...]
The way sets are mapped into integers consists in first using a tokenizer to map from strings to sequences of set items. Then a dictionary of all the different set item strings present in the column of the dataset is collected, then they are ranked by frequency and an increasing integer ID is assigned to them from the most frequent to the most rare (with 0 being assigned to PAD used for padding and 1 assigned to UNK item).”
For text representation, we chose to use pretrained word embeddings for the Italian language.
Our first model used fastText word
representations
Our second classifier used BERT word
representations
For image representation, we used the embeddings
provided in DANKMEMES dataset. The vector
representations are computed employing ResNet
Figure 1 shows a block diagram of system architecture, which is very simple. Picture manipulation, Visual actors, Engagement, Image vector and Sentence vector (obtained from word embedding) were combined by concatenation. The resulting multimodal feature vector was fed as input into a feed-forward neural network with two hidden layers of 256 and 16 neurons respectively, with a ReLU activation function. The last single neuron predicts whether the image is a meme or not. 2 2.1
To train our neural networks, we chose crossentropy loss as the objective function. As defined in the subtask, the metrics of interest are precision, recall and F1 score. In the following, all metrics 2https://github.com/idb-ita/GilBERTo reported were calculated using the officially provided evaluation script3.
We used Adam optimizer with the following parameters: 1 = 0:9, 2 = 0:999, " = 10 8. We set an early stop of 5 epochs, in order to avoid overfitting.
Hyperparameter optimization was manually conducted and we tried various combinations of learning rate and batch size: our final models have learning rate of 10 5 and batch size of 10.
During our experiments, we studied the impact of a multimodal analysis, compared to using language or vision only.
We trained various models, including different combinations of basic features (Picture manipulation, Visual actors and Engagement), text representation (fastText or GilBERTo) and image representation (ResNet). 2.2
random baseline Basic Features BF+fastText BF+GilBERTo BF+ResNet
BF+fastText+ ResNet (SNK1) BF+GilBERTo+ ResNet (SNK2)
Pr 0.525 0.8732 0.8253 0.7685 0.8341 0.8515
Re 0.5147 0.6078 0.6716 0.7647 0.8382 0.8431 0.8317 0.848
We observe that basic features are quite informative: the model based only on them far outper3https://github.com/gianlucalebani/ dankmemes2020 forms the random baseline.
Models based on basic features and visual representations perform meme detection well. It should be noted that unimodal vision models perform significantly better than textual models. As Sabat et al. (2019) pointed out, an obvious reason is that the dimensionality of the image representation (2048) is much larger than the linguistic one (fastText: 300; GilBERTo: 768), so it has the capacity to encode more information. It would be interesting to conduct further experiments to investigate less obvious motivations and understand if the image representation actually conveys features of the visual scene, which are specific and distinctive of a meme.
As shown by Beskow et al. (2019), multimodal classifiers are considerably better than textual models and provide some improvement over unimodal vision models, which nevertheless provide solid performance in meme detection.
A2 SNK1 B2 A1 SNK2 B1 ... baseline
With reference to the competition, model SNK1 (Basic features + fastText + ResNet) ranked 2nd, at a short distance from the first classified. Model SNK2 (Basic features + GilBERTo + ResNet) ranked 5th.
In this paper, we have presented simple multimodal systems for meme detection, based on a neural network classifier; they leverage existing pretrained embeddings to represent both text and image. Our systems achieve good performance, providing improvements over unimodal classifiers. In the first subtask of DANKMEMES (EVALITA 2020), our models ranked 2nd and 5th.
Based on our experiments, it is observed that pre-trained embeddings can be used effectively and with little effort to represent information conveyed by visual and textual components. While we haven’t explicitly included irony or other distinctive aspects derived from text or image among the features, it is understood that the vectors generated by the embeddings express them implicitly.
Starting from the simple model used, it could be
interesting to conduct in-depth analyzes to
understand which of the basic features are most
important. Furthermore, we could build saliency maps
The proposed model could be improved. With more time and computational resources, a broader experimentation campaign could be conducted, using Bayesian hyperparameter optimization; we could try different numbers of neurons in hidden layers and other neural network architectures. To improve the classifier without much effort, we could also make an ensemble of our best performing models.
In our classifier, we used BERT powerful language model to get text vectors. We could do BERT fine tuning, in order to obtain better textual embedding, aimed at meme detection task.
Finally, to overcome the limits of this simple
model, we could look for a more explicit way to
encode the irony present in the text, drawing
inspiration from IronITA