English. We introduce ArchiMeDe, a multimodal neural network-based architecture used to solve the DANKMEMES meme detections subtask at the 2020 EVALITA campaign. The system incorporates information from visual and textual sources through a multimodal neural ensemble to predict if input images and their respective metadata are memes or not. Each pre-trained neural network in the ensemble is first fine-tuned individually on the training dataset to perform domain adaptation. Learned text and visual representations are then concatenated to obtain a single multimodal embedding, and the final prediction is performed through majority voting by all networks in the ensemble.
Italiano. Presentiamo ArchiMeDe, un’architettura multimodale basata su reti neurali per la risoluzione del subtask di “meme detection” per DANKMEMES a EVALITA 2020. Il sistema unisce informazione visiva e testuale attraverso un insieme multimodale di reti neurali per prevedere se immagini e rispettivi metadati corrispondano a meme o meno. Ogni rete neurale pre-allenata all’interno dell’insieme e` inizialmente adattata al dominio specifico del dataset di training. In seguito, le rappresentazioni di ogni rete per immagini e testo vengono concatenate in un unico embedding multimodale, e la previsione finale e` effettuata tramite un voto di maggioranza effettuato da tutte le reti nell’insieme.
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
In recent years, the democratization of data
collection procedures through web scraping and
crowdsourcing has led to the broad availability of
public datasets spanning modalities like language and
vision. Contemporary state-of-the-art machine
learning models can leverage those resources to
achieve highly accurate and often superhuman
performances using millions or even billions of
parameters
In this paper, we present ArchiMeDe, a
multimodal system leveraging pre-trained
language and vision models to compete in the
DANKMEMES
Images
Sentence embeddings + Image embeddings + Raw metadata
Dense Layers Dense Layers Dense Layers
Is this a meme? Majority Vote task of DANKMEMES, aimed at discriminating memes from standard images containing actors from the Italian political scene. Task organizers extracted a total of 1600 training images from the Instagram platform, and data available from each dataset entry – text, actors and user engagement, among others – were leveraged to train an ensemble of multimodal models performing meme detection through majority-vote. The following sections present our approach in detail, first showing our preliminary evaluation of multiple modeling approaches and then focusing on the final system’s main modules and the features we leverage from the dataset. Finally, results are presented, and we conclude by discussing the problems we faced with some inconsistencies in the data. Our code is made available at https://github.com/ jinensetpal/ArchiMeDe 2
ArchiMeDe is composed of a multimodal
learning ensemble, with the final output being the
result of a majority vote. Figure 1 visualizes our
approach. First, the transcript associated with
each image is fed to an UmBERTo
The remaining part of this section contains an in-depth description of our ensemble’s components, focusing on the input features that were used and how those were preprocessed to best suit learning. Moreover, we also include transfer learning specifications with some details about their impact on the overall system accuracy. 2.1
Engagement User engagement per post is expressed as a numeric integer value. We scale and standardize engagement values to obtain a distribution centered in 0 with = 1. This procedure is a standard practice to avoid passing extreme absolute values as inputs for the neural network. Date We decided to leverage temporal information in our system, building upon the intuition that memes often rely on a small set of templates that undergo a significant variation in popularity through time. Temporal information may thus provide our system with additional cues about an image’s meme status in a specific time-frame. In the training dataset, dates for each post has been presented in the yyyy-mm-dd format. This date was compared with the predetermined date, 1st January 2015, to derive a numeric value representing the number of days from the date of reference. Min-max scaling is then applied to the numeric values, further deriving float numeric values between in the range [0,1], subsequently fed into each training model.
Manipulation The manipulation field provides boolean information about whether an image has been manipulated before being added to the dataset. We found this information noisy and a weak predictor of meme status; therefore, it was dropped as input.
Visual Actors Each entry was additionally provided with a list of names of the visual actors present in the frame. In the specific case of the DANKMEMES shared task, visual actors can be especially useful to identify meme images. For example, we can hypothesize that politicians who maintain a strong public presence by making claims that produce a high level of public engagement are more likely to be the subject of meme images. Moreover, some combinations of actors may be particularly likely for memes e.g. politicians belonging to parties at the political compass’s antipodes. In order to produce a unified representation of visual actors for our system, we perform a one-hot encoding of all the actors occurring in the training set: if a specific politician is present in an image, the corresponding entry is true; conversely, if no such actor is present, the binary field is set to false. Actors that were not present in the training set are disregarded during evaluation: while this step is required given the context, we assume that this may significantly impact the outcome in images for which new actors were introduced. 2.2
The analysis of textual content in meme images is
critical to the success of the overall system.
Indeed, ironical or satyrical comments may deeply
affect the users’ interpretation of an image that
would otherwise be classified as normal. We
note that this problem cannot be approached
similarly to standard textual analytic frameworks since
memes are elucidated in short, concise phrases and
do not necessarily comply with standard
grammatical rules. They also tend to contain slang
and vernacular expressions, which, albeit
conveying the intended meaning to the reader, greatly
increase the need for high model capacity and
adhoc training data. For this reason, we selected
UmBERTo
Transformers framework
1umberto-commoncrawl-cased-v1 in the
HuggingFace’s model hub
Due to the inherently large variance in meme
images’ styles and contents, it is impractical to
expect a single framework to effectively describe
each distinguishable feature and utilize it to
classify an entry. Hence, we split the representational
burden across multiple pre-trained model
architectures. Each of them uses a fundamentally
different approach to extract image embeddings,
making the resulting ensemble predictions more
flexible in general settings. The three networks we
used for producing image embeddings are:
ResNet Residual Networks, or ResNets
DenseNet Pre-trained models such as ResNet
and AlexNet use a large number of hidden
layers. While the increase in depth allows
for better feature abstraction, it often leads
Run #
to vanishing-gradient problems during training.
DenseNet
The aim of using multiple vector embeddings was to cumulatively cover a significant portion of possible meme combinations and templates. As a result, is Section 4 we show how the ensemble of systems using different image embeddings leads to significant increases in validation accuracy. 3
Table 1 presents the system ranking for the meme detection subtask. Our system placed 7th in terms of F1 score,2 impeded primarily by inconsistent recall performances but significantly better than the random baseline (+0:2466 F1).
Results suggest that ArchiMeDe has developed inductive biases for specific image features that strongly influence the classification outcome. By inspecting validation folds over training data, we observe that most false negatives produced by the system involve distinct facial characteristics of scene actors. Inversely, ArchiMeDe effectively classifies images containing text bubbles and evident manual edits. Another notable failure case we identified is due to face-swapping. This failure is especially relevant since face-swapping is com2The F1 score is the harmonic mean between precision and recall, commonly used to evaluate classification systems. .83/.77 .87/.83 .83/.79 .80/.84 .87/.85 .79/.81 .84/.85 .85/.83 .82/.79 .84/.87 .86/.86 monly used to add an ironic component to meme images, but it is hardly detectable due to missing real-world context. 4
As a complementary perspective on our experiments’ nature, in this section, we present other approaches tested in the context of meme detection and that were finally disregarded in favor of the ArchiMeDe approach presented in the previous section.
CNN without Metadata Preliminary runs on the DANKMEMES dataset relied solely on the use of standard convolutional neural networks. The target architecture was fed the image itself without associated metadata to ensure that the standalone impact of the architecture was shown. The system performed poorly, performing only slightly better than the baseline scores. Additional measures to optimize this network were not taken since we assumed that this naive approach would not lead to substantial gains in performances over the baseline.
working with an ensemble, we estimated the
performances of its components in performing meme
detection. Besides the three models that we
finally included in ArchiMeDe, we also tested
ResNeSt
Multimodal Ensemble Following the
complementary viewpoints of different encoders, we
decided to evaluate the performances of an
ensemble. Table 2 shows that our ArchiMeDe ensemble
outperforms single systems in terms of both
precision and recall when considering both classes,
compensating the weaknesses of individual
systems. The resulting majority-vote ensemble was
optimized and used as the final system for our
submission. Multiple experimental iterations showed
that an increase in depth, followed by a
reduction in layers’ width, led to increased accuracy
scores. Each model was trained with a batch size
of 64 sets, 100 epochs fitted with test accuracy
callbacks, and an early stopping strategy with a
five epochs’ patience value. Each model utilized
the Adam optimizer
Given the relatively small size of the available training dataset and since popular classification models are often trained using thousands if not millions of images, we tested some data augmentation strategies to improve our system’s generalization performances. We applied random changes for each image to augment data, modifying it with random brightness, rotation, and zoom in a reasonable margin to keep it distinguishable. 9 augmented images were produced for every initial image entry. As a result, the training dataset is increased from 1280 to 12800 images.
Every augmented image is associated with the same metadata as the original, varying only in the visual embedding itself. The result we aimed for was an increase in generalization performances, as the model fits better to the general rule of recognizing memes. However, our results showed the opposite behavior: the system would easily overfit individual observation when data augmentation was used. We think this was partly due to augmentations not pertinent to the general meme template and partly because of the significant increase in the number of entries having the same associated metadata.
An extensive set of augmentation strategies was tested over the dataset, modifying factors, ranges, and augmentation count. No iteration significantly and consistently improved the system’s performance, and thus the augmentation process was determined noisy, relatively inconclusive, and therefore dropped from the training procedure. 5
In this paper, we presented ArchiMeDe, our multimodal system used for participating in the DANKMEMES task at EVALITA 2020. The results produced by the system are promising, even if the systems do not encode inductive biases that are specific neither for multimodal artifact recognition nor to meme detection in particular. The entry is not far behind in terms of precision from the best-performing systems, and several paths display considerable potential for improving its performances. The paper effectively highlights the crucial impact of transfer learning on the success of this system. Notably, ArchiMeDe can be easily trained with standard consumer-level GPUs.
A direction that can be explored to improve the current system would be to modify the recall threshold, obtaining a better precision-recall balance for predictions. Another possibility involves introducing an aggregator network on top of the ensemble instead of using majority vote: in this way, the network can learn whether the predictions of a single subnetwork are reliable, regardless of it being part of the majority. The ensemble could also include more varied models with differing architecture to further accentuate differences in feature representations. Above all, we believe that leveraging additional data (not necessarily in Italian) could significantly improve the system’s performance at the cost of increased time and computational costs.
Memes today are one of the most formidable modes of portraying one’s idea while building a strong interpersonal connection between creators and users. The informality of memes, combined with their ease of making and distribution, has greatly accentuated their growth in the last few years. To be able to interpret memes effectively is a task far deeper than what can be intuitively thought. As humans continue to unravel their minds and derive ingenious computational methods, we realize the importance of slang and how it relates directly to the core human principle of community belonging. A piece of our culture, memes are the best represented and documented cultural artifacts we have today, and to effectively interpret them would mean to cross a significant milestone for the field NLP, with lasting impacts on our society as a whole.