In case of a disaster, a large number of relevant and irrelevant images are propagated through social networks. In this case, the sentiment identification of such disasters is important to speed up relief work in the afected region. This paper describes the contribution of FASTNU-DS team for the Visual Sentiment Analysis: A Natural Disaster MediaEval Use-case [4] held at MediaEval 2021. Various pre-trained deep learning-based models for the single-label classification and multi-label classification tasks have been used for feature extraction and classification. Data augmentation techniques to over-sample minority classes were used to deal with the inherent imbalance nature of the dataset. For a single-label and multi-label classification of tasks, VGG16 proved to be more useful than ResNet50. In this work, we achieved a 0.65 weighted F1 score for the first single label classification subtask and 0.54 and 0.41 weighted-F1 scores for the second and third multi-label subtasks, respectively. Visual Sentiment Analysis: A Natural Disaster Use-case task of MediaEval, 2021, involves multi-class and multi-label classification tasks. Various similar studies have been performed which has focused on visual sentiment analysis.
Disaster creates a dificult situation to handle, which may harm
valuable resources and loss of human lives. Government, NGOs and
the public use diferent social networks to propagate relevant and
irrelevant information about the natural calamity in the form of
images, videos, and posts to aware others. In literature, researchers
have more focused on text-based sentiment analysis using NLP.
However, images and videos sentiment analysis using ML models
is an open research problem and need attention to identify
sentiments communicated through images. These images may reveal
emotional responses. Hence, careful identification of disaster the
sentiment is important to stop, aware and control any miss lead.
For example, this image-based sentiment analysis may be used for
rapid identification of situational awareness during disaster and
assistance in restoration activities. In addition, the categorization of
such images may further be used to understand the adversity of the
situation. In this work, we have worked on the "Visual Sentiment
Analysis: A Natural Disaster Use-case at MediaEval 2021". The
research has performed single-label and multi-label classification to
identify visual sentiments that occurred during disasters [
The research efort has focused on disaster-related images from
social media and also implemented various deep learning-based
methods [
Another research efort has proposed a framework that considers
both text and image-based visual sentiment analysis [
Similarly, a class-specific residual attention module (CSRA) has
been proposed, which has an extremely simple and eficient model,
It requires fewer resources for training and achieved the state of
the art results for various datasets of multi-label classification of
images[
The dataset for the task of "Visual Sentiment Analysis A Natural
Disaster Use-case" at MediaEval, 2021, contains 2432 images[
The proposed method for the first subtask of single label classification has been designed by performing diferent experiments to select image augmentation technique and appropriate deep learning model.
As the dataset used for the first subtask is imbalanced. The negative class contains 1695 images, and the positive class includes 648 images. However, the neutral class includes 89 images, which is significantly less than the positive and negative classes. Three diferent methods are utilized to manage the challenge of class imbalance, including weight assignment to classes, oversampling, and image augmentation. In the first attempt, diferent weights are allocated to all three classes, so that they can be balanced. Another class balancing efort has been performed by increasing the number of images by simply using oversampling. The oversampling technique increases copies of instances of minority class and makes them equal to majority class. The method has proved better in comparison to the weight balancing technique. In last, the class imbalance has been reduced by using the data augmentation technique, in which variants of a single image are created by using diferent augmentation techniques. Moreover, augmentation techniques including random shift, random flip, random brightness, and random zoom are applied to increase the number of images in minority classes and make them equal in quantity. The data augmentation technique has provided the best results in tackling class imbalance problems compared to oversampling and class weight assignment techniques.
There are two deep learning based pre-trained models are
selected for the experiments, including Visual Geometry Group (VGG)
[
To improve the eficiency of experiments, all the experiments are initially performed by converting images into grayscale, which has reduced the processing time for the method. After selecting optimal values for hyperparameters, coloured images are used to further improve the method’s performance. The trained model is then applied for the prediction of test set instances, which are 1199 images. 3.2
Subtask 2 and Subtask 3 are aimed to classify images based on multi-label image classification, where one image can be assigned to various classes, according to its depicted emotions. According to subtask 2, the image may belong to one or more classes, including anger, disgust, joy, fear, neutral, surprise, and sadness. However, for subtask 3, the image may acquire one or more classes, and it is also a multi-label classification task. The diference between subtask 2 and subtask 3 is the number of classes containing 7 and 10 classes, respectively.
The dataset is imbalanced for subtask 2 and subtask 3, and few classes contain more images than other classes. The oversampling technique is utilized to increase the number of images in minority classes to reduce the class imbalance.
After balancing the classes, the VGG16 pre-trained on ImageNet is fine-tuned on the dataset for subtask 2 and subtask 3. Moreover, the sigmoid activation function is used to predict multi-label classification. 4
At the initial stage, VGG16 and ResNet50 have been experimented for subtask 1. The implementation has been performed on three different data balancing techniques: weight balancing, oversampling, and image augmentation. The weighted F1 scores on training data for the first subtask are shown in Table 1.
The training experiments have proved VGG16 as a better
pretrained model and image augmentation as the best class balancing
technique. By considering better performance, the VGG16 has been
trained on the whole dataset and used to predict test data. The image
augmentation technique is used for class balancing in subtask 1.
However, for subtask 2 and subtask 3, the oversampling technique
increases the number of minor classes. The results for test-set are
visualized in Table 2.
The research has proposed a deep learning-based model for
singlelabel and multi-label classification tasks to analyze visual
sentiments during disastrous conditions. The approach has tried various
class balancing techniques and pre-trained models. Furthermore,
the research can be extended by using weights from the Places
dataset [
This work was supported in part by the Smart Video Surveillance Lab, an afiliated Laboratory of (NCBC), FAST-National University of Computer Emerging Science.