The Visual Sentiment Analysis task is being ofered for the first time at MediaEval. The main purpose of the task is to predict the emotional response to images of natural disasters shared on social media. Disaster-related images are generally complex and often evoke an emotional response, making them an ideal use case of visual sentiment analysis. We believe being able to perform meaningful analysis of natural disaster-related data could be of great societal importance, and a joint efort in this regard can open several interesting directions for future research. The task is composed of three sub-tasks, each aiming to explore a diferent aspect of the challenge. In this paper, we provide a detailed overview of the task, the general motivation of the task, and an overview of the dataset and the metrics to be used for the evaluation of the proposed solutions.
An enormous amount of multimedia content is generated and
shared over the internet daily, especially after the introduction
of several social media platforms, such as Twitter, Instagram, and
Facebook. Among other types of content, visual content is used
extensively to deliver a specific message to the viewer, implying
the common phrase “a picture is worth a thousand words” [
Disaster-related images can be challenging to interpret,
especially when analyzing them using current visual analysis techniques.
These images generally contain important details in both the
background and foreground, making no single part the main point of
interest [
In order to facilitate future work in the domain, a large-scale dataset is collected, annotated, and made publicly available. For the annotation of the dataset, a crowd-sourcing activity with a large number of participants has been conducted. 2
The literature reports several interesting works aiming to minimize
and mitigate the loss of lives and infrastructure damage. Diferent
types of solutions rely on ground sensors and satellite imagery
for efective disaster handling [
As a first step, in creating this benchmark dataset [
After data collection, the next step was data annotation, which was crowd-sourced using Microworkers. The challenging part for crowd-sourcing was designing the task and selecting proper keywords to represent diferent categories of the annotated data. The three keywords namely Positive, Negative, and Neutral are the most widely used sentiment labels in the literature. However considering the applications and the complexity of sentimental analysis of disaster-related images, it is quite complicated to interpret and represent the sentiment and emotion in these three categories. We rather need a more specific and larger set of sentiment categories/labels to interpret the emotions associated with natural disasters. There is an interesting recent work in psychology on emotions and sentiments representation, where 27 diferent emotion categories are reported. Based on this study, we set up four diferent types of annotations, which are expected to better represent the sentiments and emotions associated with natural disasters by covering diferent aspects of natural disasters. These sets of labels are described below.
(1) Positive, negative, and neutral. (2) Relax/calm, normal and simulated/excited. (3) Joy, sadness, fear, disgust, anger, surprise, and neutral. (4) Anger, anxiety, craving, empathetic pain, fear, horror, joy, relief, sadness, and surprise.
As a first step towards visual sentiment analysis of natural disaster multimedia content, the scope of the task is limited to images from diferent types of natural disasters. Overall, 4, 003 images were selected for the crowd-sourcing study. To ensure quality and consistency, each image was annotated by five diferent persons. The study was disseminated through multiple channels. A total of 10, 010 responses were received during the study, and 2, 338 participants participated from 98 diferent countries. 4
It is the first time we are hosting this task in MediaEval. The task is
closely related to previous tasks namely ”The 2017-2019 Multimedia
Satellite Task: Emergency Response for Flooding Events” [
Images from the dataset provided for the task are filtered to check whether there is a majority consensus among the annotations provided by the crowd-sourcing participants. The resulted dataset consists of 3, 631 images. The development set is composed of 2, 432 images, and the test set contains 1, 199 images. The task is divided into three subtasks, which are described below. 4.1
This is a multi-class single label classification task, where the images are arranged in three diferent classes, namely positive, negative, and neutral. There is a strong imbalance towards the negative class, given the nature of the topic. 4.2
This is a multi-class multi-label image classification task, where the participants are provided with multi-labeled images. The multilabel classification strategy, which assigns multiple labels to an image, better suits our visual sentiment classification problem and is intended to show the correlation of diferent sentiments. In this task, seven classes, namely joy, sadness, fear, disgust, anger, surprise, and neutral, are covered. 4.3
This task is also multi-class multi-label, however, a wider range of sentiment classes are covered. Going deeper in the sentiment hierarchy, the complexity of the task increases. The sentiment categories covered in this task include anger, anxiety, craving, empathetic pain, fear, horror, joy, relief, sadness, and surprise. 5
The proposed solutions will be evaluated using a weighted 1 score. It is a more preferable metric when dealing with imbalanced datasets as it is more sensitive to data distribution. It also provides a good balance between precision and recall as it is a harmonic mean of these two metrics. The weighted 1 score can be calculated using the following four equations. (1) (2) (3) (4) =
True Positives (True Positives + False Positives ) =
True Positives (True Positives + False Negatives ) 1 Score = ℎ 1 = 2 ∗ ∗ ( + ) Í=1 (1 Score ∗ ) Í=1 ( )
Equation 1, 2, and 3 shows the calculation of precision, recall and 1 score for each class, respectively. Equation 4 shows the calculation weighted 1 score where is the number of instances of a particular class. 6
Though the literature reports some initial eforts on the topic, several important aspects need to be addressed yet. We believe that extracting and representing sentiments from disaster-related visual content will benefit several stakeholders, such as news agencies, Government and non-government, and other humanitarian organizations. From a research point of view, visual sentiment analysis of natural disaster-related visual content is not limited to object recognition. Instead, it requires a more complex identification analysis and establishing a connection among salient objects, scenes, expressions, and color schemes. We hope the task will result in a joint efort towards this important topic, and it also will open new research directions.