Deep learning algorithms trained on huge datasets containing visual and textual information, have shown to learn useful features for other downstream tasks. This implies that such models understand the data on different levels of hierarchies. In this paper we study the ability of SOTA (state-of-the-art) models for both texts and images to understand the emotional attitude caused by a situation. For this purpose we gathered a small size dataset based on IMDB-WIKI one and annotated it specifically for the task. In order to investigate the ability of pretrained models to understand the data, the KNN clustering procedure over representations of text and images is utilized in parallel. It's shown that although used models are not capable of understanding the task at hand, a transfer learning procedure based on them helps to improve results on such tasks as image-captioning and sentiment analysis. We then frame our problem as the task of image captioning and experiment with different architectures and approaches to training. Finally, we show that adding additional biometric features such as probabilities of emotions and gender probabilities improves the results and leads to better understanding of emotional attitude.
2. A set of experiments on the tasks of image-captioning and sentiment analysis, based on features extracted from highlighted models. It’s also shown that adding biometric features as gender and emotions distribution improves the performance of image-captioning models.
The training procedure was conducted using tensorflow [
The actual data needed to include both images and their captions. As the main intent was to capture the emotional attitude, the images would have to contain people and explicit or implicit information about the cause of their emotional state. The captions should have contained an exhaustive unbiased description of the situation. Based on highlighted requirements, the first idea was to make a dataset from the subset of existing image-captioning datasets.
Image-captioning is the process of generating textual description of an image. The task implies
that the relevant dataset consists of image-text pairs. One of the most popular datasets for the
discussed task is COCO [
The other variant was to gather a dataset from the very beginning and annotate it. The images were
taken from the IMDB-WIKI [
In order to categorize the dataset, sentiments related to captions were added using Vader [
New sentiment category was used for analysis of clustering and for solving the task of sentiment analysis given the captions.
In order to analyze the ability of pretrained models to understand such difficult information as emotional attitude, recent SOTA models trained on big datasets of textual and visual information were chosen. 3.1.
ResNet, introduced by Kaiming He et.al, is a deep convolutional architecture, which suppressed
previous results on Imagenet benchmark and showed to be pretty successful for object detection by
obtaining a 28% relative improvement on the COCO object detection dataset. Main advantage of such
architecture is the addition of residual connections that help to fight the problem of vanishing
gradients, which is typical for deep neural networks. This advantage gave a possibility to train a very
deep network, each layer of which learned different useful features. In our work ResNet152V2
pretrained on the Imagenet dataset was used. We also experimented with ResNet50 trained on FER
[
EfficientNet, introduced by Tan et.al, is a deep convolutional neural network architecture and
scaling method that uniformly scales all dimensions of depth/width/resolution using a compound
coefficient. It achieves state-of-the-art 84.3% top-1 accuracy on ImageNet and transfers well to other
tasks, reaching state-of-the-art accuracy on CIFAR-100 [
Word2Vec, introduced by Mikolov et.al [
The exact setup of experiments and description of layers using which the data representation was derived along with experimental results is discussed further in the paper
Image understanding is the process of interpreting regions/objects to figure out what's happening
in the image. This may include figuring out what the objects are, their spatial relationship to each
other, etc [
As it was stated by
Kovenko et.al [22], by solving the problem of data reconstruction, autoencoders tend to learn low-level features, which are useful for transfer learning. Based on this idea we train the deep convolutional autoencoder on our dataset and use latent code produced by the encoder part for encoding images in image-captioning task. Also the experiments include the output of 4th block of ResNet, along with the logits of ResNet as the encoders. In order to compare this transfer learning approaches, we also experiment with custom not pretrained convolutional encoder.
The decoder part is represented by the embedding layer and LSTM (Long-short-term-memory) [23] network. LSTM is capable of learning long-time dependencies, which is especially useful when working with sequential data. As the embedding layer, for all the experiments, Word2Vec was used. For all the approaches, layer normalization [24] after LSTM was used. As it was stated by Xu et.al [25], the attention mechanism applied to image-captioning tasks can greatly improve results. Nezami et.al [26] showed that usage of additional features of emotions helps to improve results on the imagecaptioning datasets that include emotional aspects. Based on these ideas, we experimented with using attention and conditioning LSTM on additional features. Different from Nezami’s approach, gender features were also used and the emotional ones were encoded as probability distribution. Specifically, YoloV3 is used to extract face regions from the images and EfficientNet trained on Age-Gender dataset along with ResNet trained on FER one are used to predict gender and emotions.
Gender features are produced using predicted probabilities for each face presented on the image (formula 1). (1) (2) of an argmax operation over prediction probability vector for specific face i. where G - number of unique genders, g - gender, N - number of faces presented on the image, 1
- normalized vector with gender probabilities, - identifier of Pibeing equal to specific g, - result =
, ∑
= ∑ 1 =1 , = ( )
Emotional features are produced as normalized probability distribution of the sum of probability vectors for each face presented on the image (formula 2).
= ∑ , =
∑ where
- vector of averaged emotion probabilities, N - number of faces presented on the image, predi - prediction probability vector for specific face i, M - number of unique genders.
The data was splitted in the same way as for sentiment analysis. The approaches were validated based on the test set performance using beam search technique with the beam size of 5. BLEU score along with perplexity were used as the main metrics. For all the experiments RMSprop optimizer was used, with the initial learning rate of 0.0001. In order not to overfit, the loss reduction technique was used. If there was no improvement in validation perplexity for two epochs, the loss was reduced by the factor 10. All the models were trained with a batch size of 64 for 30 epochs (Fig. 3).
Analyzing the results it’s obvious that the transfer-learning procedure gives better results than training from scratch (ordinary) w.r.t BLEU on a test set. It’s also clear that ResNet representation tends to give better results than autoencoder’s one, possibly because of a deeper architecture and better learned features. Attention didn’t work well for all the approaches, probably because of the low number of samples in the dataset and small amount of epochs. So far an approach that utilized logits output of ResNet for encoder part of the network along with Word2Vec embeddings and additional features of emotions (resnet_logits_w2v_emotions) gave the best results on test data w.r.t to averaged BLEU score. The other appomodelrach, which is also worth paying attention to is the one, which incorporates both emotions and gender features. Despite resnet_logits_w2v_emotions_gender didn’t achieve the best performance on test BLEU, it reached the best balanced performance on all the data splits, and thus was chosen as the best one. The architecture of the overall prediction pipeline is shown in Fig. 4.
As it can be seen from Fig. 4, the overall pipeline is dependent on the face pre-processing step along with the detection of emotions and gender. Obviously, if the performance of highlighted steps is poor, the final output will be at least biased. The example of such bias is represented in Fig. 5.
During error analysis, it was found that the model suffers from slight overfitting on most frequent words and phrases (like “man is flirting with a woman” presented in Fig. 5), which is a problem caused by a small diversity of the dataset. Despite the fact that the collected data is a noisy one, as each image was annotated by a different expert, which is not very suitable for the task of imagecaptioning, the model succeeds to give adequate results on average (Fig. 6). b) Figure 6 (a, b). Example of generated captions. T - true caption, greedy - result of greedy decoding, beam - result of beam search decoding. Captions which are fully inappropriate are marked with bleu. c) e) Figure 6 (c – e). Example of generated captions. T - true caption, greedy - result of greedy decoding, beam - result of beam search decoding. Captions which are fully inappropriate are marked with bleu.
It’s important to note that the longer training would probably give better results.
In this paper we analyzed the ability of deep learning models to understand the emotional attitude driven by the situation. For this purpose, a new dataset with image-text pairs was presented. In result of pretrained SOTA models analysis, it was concluded that some of them can be used in the process of transfer-learning. Through the experiments it was shown that the dataset can be used to solve the problem of sentiment analysis. It was then theorized that the problem of understanding the emotional attitude, can be transferred to the task of imagecaptioning. Empirical results have shown that addition of emotional and gender features along with transferlearning based on ResNet network and Word2Vec embeddings improve the overall captioning performance. Our approach gives pleasant results on average, confirming that deep learning models are able to understand emotional attitude if they are trained to. It's important to note that such an approach has many downsides, as it’s dependent on the performance of three additional models for face, emotions and gender detection. The other problem that was faced is the noisy nature of the dataset and small variation of phrases in it. In future work it’s planned to gather a bigger dataset, label each image with 5 captions and fix current problems.
We would like to thank Oleksii Abdullaiev, Dmytro Tarasovskyi and Dmytro Maliovanyi for their contribution in terms of the dataset creation.
[20] [20] Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H. (2019). A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CsUR), 51(6), 1-36. [21] Bisikalo, O., Bogach, I. & Sholota, V. (2020). The Method of Modelling the Mechanism of Random Access Memory of System for Natural Language Processing. In 2020 IEEE 15th International Conference on Advanced Trends in Radioelectronics, Telecommunications and Computer Engineering (TCSET) (pp. 472-477). doi: 10.1109/TCSET49122.2020.235477. [22] Kovenko, V., & Bogach, I. (2020). A Comprehensive Study of Autoencoders' Applications
Related to Images. In IT&I Workshops (pp. 43-54). [23] Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780. [24] Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450. [25] Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., ... & Bengio, Y. (2015, June).
Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning (pp. 2048-2057). PMLR. [26] Nezami, O. M., Dras, M., Anderson, P., & Hamey, L. (2018, September). Face-cap: Image captioning using facial expression analysis. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 226-240). Springer, Cham.