Multi-modal neural models that are able to encode and process both visual and textual data are becoming more and more common in the last few years. Such models enable new ways to learn the interaction between vision and text, and thus can be successfully applied to tasks of varying complexity in the domain of image and text classification. However, such models are traditionally oriented to learn grounded properties of images and of the objects they depict and less suited to solve tasks involving subjective characteristics, such as the emotions they can convey in viewers. In this paper, we provide some insights in the performances of the recently released OpenAI CLIP model for an emotion classification task. We evaluate the model both under zero-shot settings and via fine tuning on an image-emotion dataset. We compare the performances of CLIP both in a zero-shot and fine-tuning setting on (i) a standard benchmark dataset for object recognition (ii) an image-emotion dataset. Moreover, we evaluate to which extent a CLIP model adapted to emotions is able to retain general knowledge and generalization capabilities.
The ever-increasing production and spread of multi-modal content over the internet requires new analytical tools to deal with them. Although many issues related to the multi-modal analysis of text and images have already been addressed in the literature, it is still unclear whether and to what extent state-of-the-art multi-modal systems can be exploited to explore the afective characteristics of the visual contents.
Several multi-modal resources, systems and architectures have been proposed
in the literature to approach a wide range of natively multi-modal tasks, such
as Image Captioning [
On the contrary, the field of Natural Language processing has addressed
problems related to the afective properties of texts for many years. The literature
is filled with approaches dealing with sentiment, opinion and afect. For example,
several studies have been proposed to analyze the sentiment and the emotions
expressed and evoked by texts from several perspectives [
The sentiment encoded in images has attracted a lot of interest due to its
various applications [
This aspect is also very relevant in the field of Industry 4.0. Companies are in fact expected to constantly communicate with their customers using new and efective forms of communication, such as the visual ones. On the one hand, it is important to study the emotional content conveyed by an image. On the other hand, especially for web marketing purposes, it is crucial to analyse the emotions “elicited” by images in viewers.
To the best of our knowledge, a fully multi-modal dataset that includes
realworld image samples addressing this issue is still lacking. However, a large scale
visual dataset labelled for the emotions evoked by images has been proposed
in [
To study aspects related to the emotions evoked by images, we decided to
base our experiments on the recently released and well-known OpenAI CLIP
model [
In this context, the motivation of our choice to adopt CLIP is twofold. On
the one hand, the model has been trained to eficiently learn visual concepts
by exploiting natural language supervision. We can argue that it may directly
encode latent emotive concepts. On the other hand, CLIP authors claim that
it can be used to nearly arbitrary visual classification tasks [
We are conscious that analyzing afect elicited by images is a very
challenging task because people with diverse social and cultural backgrounds may have
diferent emotional reactions to the same image [
In this work, we propose to exploit and analyze the performances of CLIP
for the task of image emotion recognition. CLIP can be leveraged either as a
pre-trained model for zero-shot classification, as intended by the authors [
The contributions of this paper are the following: – We evaluate the zero-shot performances of CLIP on two diferent benchmark datasets, namely (i) a dataset for image emotion recognition and (ii) a dataset for a more standard image classification problem; – We evaluate CLIP in a fine-tuning setting on two diferent tasks, namely (i) image emotion recognition and (ii) image classification, and compare the obtained results; – We evaluate to which extent CLIP is able to retain general knowledge and generalization capabilities to other tasks after being fine-tuned.
The rest of this paper is organized as follows. Section 2 thoroughly describes the performed experiments. In Section 3 the results of the experiments are presented and discussed in order to shed some light into the capabilities of CLIP for image emotion recognition. Finally, Section 4 draws some conclusions and discusses future work.
In order to provide some insights into the capabilities of CLIP for emotion recognition we perform several diferent experiments. The experiments are devised in order to fulfil two goals: first, we want to assess the performances of CLIP under zero shot settings; second, we want to evaluate the impact of fine-tuning in the performances of the model, both for the specific task and in its generalization capabilities. In addition to this, we try to address the diferences between the performances of the CLIP model on more abstract and more concrete tasks across all of the performed experiments.
For the image emotion recognition task we employ the dataset described in
[
Our experiments are organized as follows. First we perform zero-shot classi-fi cation on the two dataset using the pre-trained ViT-B/32 CLIP model. Second, we fine-tune the CLIP model on the two datasets, and evaluate its performances in a cross-validation experiment. Third, we again perform zero-shot classification on each of the two datasets using the model fine-tuned on the other one. This means that the model fine-tuned on the Image-Emotion dataset is applied to CIFAR100 and vice versa. This aims to understand how fine-tuning afects zero-shot performances on other tasks.
All the experiments are performed by exploiting the CLIP python library3 and the oficial pre-trained available models. In the following, we thoroughly describe the experiments and show the obtained results. 2.1
In the first set of experiments, we simply employ a pre-trained CLIP model to
classify images under zero shot settings. Following the original CLIP paper [
For both experiments, we encode all the images and all the captions with the CLIP model. Specifically, we use the ViT-B/32 pre-trained model. Then, we compute cosine similarity between the representations of each image and each caption. To obtain the final label, we simply assign to each image the caption (label) with the highest cosine similarity to it. 2.2
For the second set of experiments, our goal is to evaluate how much improvement could be obtained on downstream tasks by fine-tuning a base CLIP model. More specifically, we focus on the two downstream tasks of (i) image emotion classification on the Image-Emotion dataset and (ii) image classification on the CIFAR100 benchmark.
In order to obtain reliable and comparable estimations of the performances, we use 10-fold cross validation during training. However, since the two datasets are diferent in terms of size, number of classes, and distribution of classes, we also perform some hyperparameter tuning to obtain the best possible results on both datasets. For the sake of brevity we leave out the details of parameter tuning. However, in this regard it is very interesting to notice how the process of finetuning CLIP is extremely sensitive to diferent hyperparameters. For example, a slight change in learning rate or number of training epochs may lead to a decrease in performances of up to 0.20 in weighted and macro average F1-Score.
First, we experiment with the Image-Emotion dataset. We refer to the resulting model as Emotion-CLIP. As previously mentioned, we perform 10-fold cross validation on the whole dataset. Each fold is composed of 20,000 training examples and 3,500 test examples. The model is evaluated by predicting the most likely label for each image by means of cosine similarity with respect to the generated captions, as under zero-shot setting described in Section 2.1. As for the hyperparameters, we train each fold for 3 epochs with a batch size of 256. We use an Adam optimizer with a learning rate of 2e-5 and a 0.2 weight decay. Training each epoch took roughly 3 minutes on a Nvidia Titan RTX GPU. To obtain the final results for the classification, we average performances on each fold.
In order to further evaluate how the process of fine-tuning can be helpful also for zero-shot capable models, we propose to exploit a simpler and more grounded task of image classification on the CIFAR100 dataset. We refer to the trained model as CIFAR100-CLIP. As for the previous experiment, we perform 10-fold cross validation on the entire dataset (i.e. the concatenation of train and test set), with the predictions obtained by means of cosine similarity between images and captions. Each fold is composed by 54,000 training samples and 6,000 test samples. Note that the distribution of classes on the whole dataset is perfectly balanced (i.e. each label is associated with exactly 6,000 images). After tuning the parameters, we chose to train the model on each fold for 1 epoch with a batch size of 256. The same learning rate and optimizer used for Emotion-CLIP are employed also in this case.
While fine-tuning is a viable strategy for applying CLIP to downstream
classiifcation task, the original goal of CLIP is to take advantage of the interaction
between natural language and images to perform image classification tasks
without the need of direct optimization for the dataset at hand [
In order to pursue this goal, we propose the following experiments. We first ifne-tune the Emotion-CLIP model on the whole Image-Emotion dataset, and test it under zero-shot settings on the CIFAR100 dataset for image classification. Then, we do the opposite, i.e. we train CIFAR100-CLIP on the CIFAR100 dataset and test it for image emotion recognition on the Image-Emotion Dataset.
Both Emotion-CLIP and CIFAR100-CLIP are trained on their respective datasets with the same parameters employed for the cross-validation experiments described in Section 2.2. The only diference is that, in this case, the model is trained on the whole dataset. As for testing, the models are deployed in zeroshot setting and labels for both the CIFAR100 and Image-Emotion Dataset are obtained by means of cosine similarity between images and generated captions. 3
In this Section, we provide the results obtained for each of the performed experiments and discuss them to shed some light on the performances of CLIP with the diferent settings and datasets. 3.1
First, we evaluate the performances of the CLIP model under zero-shot settings both for the Image-Emotion dataset and for the CIFAR100 benchmark. As described in Section 2.1, for both the experiments the original CLIP ViT-B/32 pre-trained model was asked to compare the cosine similarity between the generated captions and the images. As for the Image-Emotion dataset, we used the following captions: “an image that evokes the emotion of < emotion >”, where < emotion > stands for one of the eight emotion classes in the dataset. As for the CIFAR100 benchmark, the captions were of the form “a photo of a < label >”, where < label > is one of the 100 labels in CIFAR100.
CIFAR100
Image-Emotion Accuracy Precision
Recall F1-Score
Results for the two datasets are shown in Table 1. We report accuracy, weighted-average and macro-average precision, recall, and F1-score for each dataset. We can see that, as expected, despite the much higher number of classes in the CIFAR100 dataset, the CLIP model under zero-shot settings is better able to predict its labels with respect to the emotion elicited by the image in the Image-Emotion dataset. We can argue that this is due to the fact that the training data for CLIP is much more akin to the CIFAR100 classification task. However, it is interesting to notice how the baseline model is nevertheless fairly able to face also a more complex and more abstract task such as emotion recognition out-of-the-box.
For the sake of completeness, we also report on class-level performances for the Image-Emotion dataset in Table 2.
We notice that there is a high variance in performances among classes, that is however not directly related to the sample size on each class. In fact, it seems that some emotions such as Disgust and Sadness are harder to model for the CLIP base model. 3.2
In the second set of experiments, we evaluated the performances of fine-tuning the CLIP model for specific downstream tasks on the Image-Emotion dataset and on the CIFAR100 benchmark. The implementation details for the experiments are described in Section 2.2.
Table 3 reports on the results of the Emotion-CLIP model. For completeness, we also report the performances for each class.
It is interesting to notice how performances drastically improve by means of leveraging a fine-tuned model trained on images and small captions that describe and mention the emotion that is likely to be elicited when watching that image. Interestingly, the model and nfie-tuning process is also rather sensitive to the input captions that describe the labels. During the experiments we noticed in fact that captions that use more complex words, such as for example “an image that elicit < emotion >”, or that are more direct in describing the image (e.g. “this image is about < emotion >”) are consistently outperformed by models trained on a more simple yet specific and clear caption. While diferences in performances are in the order of a few percentage points, i.e. 3-5%, it is nonetheless an interesting issue that could be explored further and more in-depth. Another interesting aspect that can be taken into account is the fact that performance vary rather widely across the diferent emotions. This may be due to the fact that describing (and thus recognizing) images eliciting certain emotions, such as Fear and Sadness, may be harder than with emotions such as Amusement and Disgust that may have more prominent visual features in the images. In addition to this, the size of the dataset and distribution of the labels must be taken into account as well. Interestingly, Disgust was the class for which performances were the worst in the zero-shot setting. Thus, in this case, it appears that the fine-tuning was rather helpful in pinpointing visual features of the emotion. Figure 1 shows some examples that highlight the diferences between the zero-shot and the fine-tuned model. Specifically, we considered each caption (i.e. emotion) and show the top-8 images associated with that caption in the dataset extracted using zero-shot CLIP (top) and Emotion-CLIP (bottom). From the images, it is first and foremost clear that fine-tuning is very efective in learning better representations for the captions, and thus it is closer to images that actually represent the emotional content. Second, it is also interesting to notice that while the performances for classes such as Fear and Sadness are sub-par with respect to other emotions, the top-8 images actually represent them quite well. This may serve as an indication that fine-tuned CLIP models may be extremely helpful also for retrieval purposes.
Table 4 reports instead on the results of the CIFAR100-CLIP model. In this case, due to space concerns we report only the overall average performances of the model.
It is clear from the results that, even after only 1 epoch of fine-tuning, the model is closer to solve the CIFAR100 dataset with respect to the baseline CLIP model, with performances above 0.80 on all the considered metrics. (a) Amusement (b) Anger
(c) Awe (d) Contentment
(e) Disgust (f) Excitement (g) Fear (h) Sadness
It is also very interesting to notice that if we compare the results of Emotion-CLIP with those of CIFAR100-CLIP, we see that the diferences in performances before and after the fine-tuning are similar for both experiments, with an improvement of around 0.20 across all metrics. This is interesting considering that the original model is much better suited to perform image classification tasks similar to the one of CIFAR100. We could speculate that, given a zero-shot capable model such as CLIP, the improvements in performances on downstream tasks and benchmark data may be limited by the architecture of the model itself. 3.3
In the final experiments, we evaluated the zero-shot capabilities of CLIP after ifne-tuning on a diferent dataset, i.e. the extent to which fine-tuning on specific data may afect the zero-shot performances on diferent dataset. Recall that in order to do so, we first trained Emotion-CLIP and CIFAR100-CLIP on their respective dataset, with the same settings described in Section 2.2. Then, we exploited the fine-tuned models to perform classification on the other considered dataset. The details of the experiments are described in Section 2.3.
Results of the experiments are shown in Table 5.
If we analyze the results of leveraging fine-tuned CLIP for diferent tasks, we can identify an interesting trend. We saw in Section 2.2 that fine-tuning for a specific task is efective in improving performances. In this case, both fine-tuned models perform worse than the ViT-B/32 CLIP pre-trained model on a task they are not fine-tuned on. This is clearly expected as the models’ weights are shifted towards the end goal of the downstream tasks. However, it is interesting to notice that both the experiments show rather similar degradation of the performances. In fact, both the models lose between 15 and 20% of F1-Score when tested on a diferent benchmark. This is interesting if we consider the nature of the training set of CLIP and its performances on simpler tasks with respect to more complex and/or abstract ones. The CIFAR100 dataset is definitely more akin to the original training set with respect to the Image-Emotion dataset, thus the resulting model should be more similar to the original one in terms of weights, i.e. it has to learn less about the classes. On the other hand, addressing image emotion classification starting from a pre-trained model requires a deeper adaptation of the model. This is also proven by the fact that CIFAR100-CLIP needed only a training epoch to learn the dataset, while Emotion-CLIP needed three. However, the relative closeness between CIFAR100-CLIP and the original CLIP model does not avoid the performance degradation in zero-shot settings on the image emotion classification. Notably, such a degradation is similar to the one detected by performing zero-shot classification on the CIFAR100 dataset starting from a model specialized on detecting emotions. 4
In this paper, we have provided an evaluation of CLIP for the detection of emotions elicited by images. We experimented with the model both under zero-shot settings and by leveraging a fine-tuning strategy, and evaluate the advantages and drawbacks of both also in comparison with a more straightforward computer vision task. Exploiting CLIP as a zero-shot classifier provides good and rather inexpensive out-of-the-box performances on image classification, while for image emotion recognition the obtained results still show a wide margin of improvement. By leveraging fine tuning, we saw a significant improvement, similar in both considered tasks, but at the cost of generalization. A fine-tuned model on a specific downstream task performs worse than the base CLIP model on a benchmark it is not trained on.
The obtained results provide an early insight into exploiting state-of-the-art
multi-modal models to characterize the emotions elicited by images, and thus
on more abstract and subjective tasks. In the future, we plan to extend this
line of research by leveraging diverse models and datasets. To this extent, we
plan to create a new dataset in which the emotive labels associated with images
are provided with textual information describing the choice of the labelling,
according to the annotation schema adopted in the ArtEmis [