Identifying the topics in online images such as memes and advertisements, and the sentiments evoked by them is a challenging task. It requires consideration of both visual and textual content in the images, and what they mean together. Both these modalities of information contain a cognitive component (what is going on in the image) and an afective component (what might be the emotional efect of the image). The main idea of this work is to extract such cognitive and afective concepts from visual and textual modalities using distinct pre-trained neural extractors and applying a language model (BERT) to reason over it for multimodal classification tasks. We benchmark the performance of our approach on the topic and sentiment classification tasks of the CVPR 2018 advertisement dataset and achieve state-of-the-art (SOTA) on topic classification with improvement in terms of mAP and F1-O score over current SOTA for sentiment classification. We also use it on the hate detection task of the Facebook hateful memes dataset and report a decent performance. We demonstrate that BERT can utilize textual inputs from diferent neural extractors in diferent formats, to obtain a performance improvement.
Expressing hate to a large audience has gotten easier with the freedom of reach ofered by social media platforms like Twitter, Facebook, and Instagram. Hence it is being increasingly important to understand, monitor, and filter such hate content. Memes are one such medium of expressing hate that incorporates both image and text data. An understanding of both modalities is required to detect the hate in them. A similar understanding is required to detect the topics and the sentiments inspired by advertisement (ad) images. This understanding can be used in ad placement based on its match with a given context, sensitive ad filtering, etc. The viewer’s perception of such images is a function of its cognitive and afective components from both modalities. (a) (b)
In this work, we exploit pretrained neural networks to extract such cognitive and afective
concepts relevant to the task in an advertisement or meme image and use a language model
(BERT) [
With this approach, we achieve SOTA results on the ads topic classification task and an improvement in mean average precision (mAP) and F1-O scores over the current SOTA on the ads sentiment classification task. On the Hate Memes dataset, our approach achieves similar performance to other approaches that use multi-modal transformer-based architectures. Our approach is flexible as one can add more neural extractors to improve performance.
There have been active eforts in recent years to collect and analyze memes datasets [
Understanding image advertisements also requires consideration of both visual and textual
modalities. Zhang et al. [25] address the topic and sentiment classification tasks on the ads
dataset [
The Facebook Hateful Memes dataset [
The tasks in hand may require both cognitive (Objects, their attributes, and relations) and afective (image valance arousal scores, object attributes such as happy, angry, calm, etc.) type of semantic information. We use a set of neural modules to extract this information we term as image features (table 1), from visual and textual modalities. These neural modules provide noisy outputs on our datasets as they are used in a zero-shot setting. We concatenate combinations of these features to form a single string and provide it as an input to BERT for a sentence classification task. For the task of binary classification of the memes dataset, we pass the [CLS] pin output to a fully connected layer with two output neurons followed by a softmax function. For all the experiments, we use a batch size of 16, learning rate of 2e-5, AdamW [31] optimizer, cross-entropy loss, and train the models for 5 epochs. In the case of classification on ads dataset, we pass the [CLS] pin output to a fully connected layer followed by a sigmoid function. We consider the prediction to be positive if the output is greater than 0.5.
Our approach is modular and eficient as one can use of-the-shelf neural extractors to extract information relevant to the task from both visual and textual modalities. A powerful language model is then used to reason over this information extracted in the form of text.
We report the results for both topic and sentiment classification tasks on the ads dataset in table
3. On the hateful memes dataset, we report the results on the ‘seen’ test and validation sets (table
2) for the results to be comparable with [
We perform experiments with individual outputs of neural modules (features) for each task to
identify the contribution of each feature. For the memes dataset, we perform experiments with
the scene-text provided in the dataset (GSTX) to benchmark against [
To avoid important information getting lost due to the truncation beyond the maximum token length (512) allowed by BERT architecture, we decide the feature order to be fed to BERT as follows. Scene-text being the most contributing feature, it is the first to appear in order. We consider the top 5 ANP predictions, which are 10 words. On average we get 4 MRCNN objects per sample, which are 4 words. Length of concatenated densecaps may go beyond the allowed max sequence length. To avoid truncation of ANPs and/or MRCNN Objects, we append them after the scene-text and before densecaps with descending order of their confidence. We concatenate text valence-arousal scores at the beginning so that the positions of these values remain the same across examples. This allows BERT to utilize the positional encodings to treat this as a distinct signal. We follow the same order for both datasets.
We try several combinations of text valence-arousal score, scene-text, ANPs, MRCNN Objects, and densecaps to find out which combination gives the best results on the validation split of each dataset. With scene-text+densecaps as input, we achieve SOTA on the ads dataset for the topic classification task and also on sentiment classification task in terms of F1-O score. After adding ANPs as an additional feature to this combination, we see an improvement in mAP and F1-C scores of sentiment classification. A multi-task setting as in [ 25] gives us inferior results compared to the single task. The feature combination, scene-text+densecaps also achieves a decent performance on the memes dataset.
Within features from just the visual modality (ANPs, densecaps, MRCNN Objects) we find that the combination of ANPs+densecaps gave the best results for both datasets. Within features from just the text modality, text valence-arousal+extracted scene-text gave us the best results for the ads dataset and the ground-truth scene-text gave the best results on the memes dataset. The diference in the performances of models on just the text modality and just the visual modality suggests that the text modality might be contributing more to the tasks compared to the visual modality. For the ads dataset, adding text valence-arousal scores and ANPs individually to scene-text gave an improvement over just scene-text. This shows that we can utilize even numeric or word phrase information from neural modules as input to BERT to get better performance. We fine-tune ResNet152 [ 32] pretrained on ImageNet to have a benchmark on endto-end approaches that operate directly on the image features. We observe that this approach performs moderately well on the tasks but the results are always inferior to our approaches that utilize the scene-text. However, when compared with our approaches that utilize just the visual input features (densecaps, ANPs, MRCNN Objects), it performs better indicating some loss of information during feature extraction from the visual input.
To understand the role of BERT’s pre-training, we perform experiments on tasks of ads and memes dataset by using BERT without the pre-trained weights [BERT NP]. For this experiment, we use features that gave us the best results for each task. The significant reduction in scores on both tasks indicates that BERT’s pre-training plays a prominent role in obtaining the performance.
We demonstrate a powerful approach that utilizes the language model BERT to achieve a good performance on distinct multimodal tasks. Extracting semantic information from the visual input in the form of text by using neural modules and using BERT to reason over this information gives us comparable results with approaches that operate directly on object or image features despite some loss of information. We show that a language model like BERT can utilize inputs in the form of numbers, categorical labels, or phrases to get an improvement. In the future, we would like to validate our approach with more language models and neural modules. International Publishing, Cham, 2020, pp. 104–120. [19] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, K.-W. Chang, Visualbert: A simple and performant baseline for vision and language, in: Arxiv, 2019. [20] F. Yu, J. Tang, W. Yin, Y. Sun, H. Tian, H. Wu, H. Wang, Ernie-vil: Knowledge enhanced vision-language representations through scene graphs, Proceedings of the AAAI Conference on Artificial Intelligence 35 (2021) 3208–3216. [21] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, J. Dai, Vl-bert: Pre-training of generic visuallinguistic representations, in: International Conference on Learning Representations, 2020. [22] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, C. L. Zitnick, Microsoft coco captions: Data collection and evaluation server, 2015. arXiv:1504.00325. [23] P. Sharma, N. Ding, S. Goodman, R. Soricut, Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Melbourne, Australia, 2018, pp. 2556–2565. doi:10.18653/v1/P18-1238. [24] C. Sharma, D. Bhageria, W. Scott, S. PYKL, A. Das, T. Chakraborty, V. Pulabaigari, B. Gamback, Semeval-2020 task 8: Memotion analysis – the visuo-lingual metaphor!, 2020. arXiv:2008.03781. [25] H. Zhang, Y. Luo, Q. Ai, Y. Wen, H. Hu, Look, Read and Feel: Benchmarking Ads Understanding with Multimodal Multitask Learning, Association for Computing Machinery, New York, NY, USA, 2020, p. 430–438. [26] S. Pilli, M. Patwardhan, N. Pedanekar, S. Karande, Predicting sentiments in image advertisements using semantic relations among sentiment labels, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 408–409. [27] K. Kalra, B. Kurma, S. V. Sreelatha, M. Patwardhan, S. Karande, Understanding advertisements with bert, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 7542–7547. [28] A. P. Bradley, The use of the area under the roc curve in the evaluation of machine learning algorithms, Pattern recognition 30 (1997) 1145–1159. [29] K. Yi, J. Wu, C. Gan, A. Torralba, P. Kohli, J. Tenenbaum, Neural-symbolic vqa: Disentangling reasoning from vision and language understanding, in: Advances in neural information processing systems, 2018, pp. 1031–1042. [30] J. Wang, Y. Yang, J. Mao, Z. Huang, C. Huang, W. Xu, Cnn-rnn: A unified framework for multi-label image classification, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2285–2294. [31] I. Loshchilov, F. Hutter, Fixing weight decay regularization in adam (2018). [32] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.