In this paper we propose a multi-modal deep learning approach to detect floods in social media posts. Social media posts normally contain some metadata and/or visual information, therefore in order to detect the floods we use this information. The model is based on a Convolutional Neural Network which extracts the visual features and a bidirectional Long Short-Term Memory network to extract the semantic features from the textual metadata. We validate the method on images extracted from Flickr which contain both visual information and metadata and compare the results when using both, visual information only or metadata only. This work has been done in the context of the MediaEval Multimedia Satellite Task.
The growth in smart phone ownership and the almost omnipresent
access to Internet has empowered the rapid growth of social
networks such as Twitter or Instagram, where sharing comments and
pictures has become part of our daily lives. Using the vast amount
of data from social media to extract valuable information is a hot
topic nowadays [
The dataset used in this work was introduced for the MediaEval 2017
Multimedia Satellite Task [? ], and contains 6600 images extracted
from the YFCC100M-Dataset [
In this section we will discuss the deep learning algorithm design
for the task of flood evidence retrieval in social media posts. The
problem will be approached under a probabilistic framework. As
explained in Section 2, the posts contain an image and/or
metadata. To extract rich visual information we apply the convolutional
InceptionV3 network, using the pre-trained weights on ImageNet
[
We have divided the development set in training (3960 + 989 extra
lfood images) and validation (1320). As for the optimizer we have
chosen RmsProp [
Since the dataset does not have a very large number of training data it is common to run into overfitting problems. In order to avoid this problem we have used the validation set to determine when to stop the training. Thus, it is stopped when the performance on the validation set stops increasing or starts decreasing over the last two epochs. Then we have used that number to retrain the system using the training and the validation set. We have followed this procedure for all the experiments.
We have trained the system in four diferent configurations: 1) having images and metadata as input, 2) having only images as input and 3) having only the metadata as input, and 4) having images and metadata in addition to the extra images obtained from Google Similar Images. The results on the test set of these four experiments are given in Table 1. The system has been evaluated as a retrieval task. All the posts from the test set have been given a probability of containing evidence of flood and have been put in order from higher probability to lower. In the first column of Table 1 we show the Average Precision (AP) of posts which have been classified as containing evidence of flood in the first 480 retrieved posts. In the second column we show the mean over average precision when evaluated on the first 50, 100, 250 and 480 posts. 5
As can be seen in Table 1 the metadata that we have selected for the task is certainly relevant to retrieve information about the evidence of flood-related images in social network posts, reaching over 70% mean precision over 4 retrieval cutofs. Since the classification of the posts as containing evidence of flood or not has been manually done using only the images, the image information should be enough for the retrieval problem. However the performance of the algorithm using only the image goes down to 66% in mean over average precision at diferent cutofs. This shows that although images should be more discriminative for this task, due to the dificulty of processing images in comparison to text, the metadata analysis gives better performance. There is also a clear improvement when combining both types of information, reaching almost a 84% accuracy in mean over the average precision in several cutofs which shows that the metadata and the image complement each other quite well. Surprisingly, when training the system with extra images, the Mean AP drops to 76%, since the images have been manually inspected to make sure that there were no noisy images added to the dataset, this makes us suspect that that result degrades when adding images without metadata, as this performs the weakest among all experiments, however it should be further studied before drawing additional conclusions. 6
In this paper we have proposed a multi-modal deep learning approach to retrieve posts from social networks containing valuable information about floods. The system can work using only visual information, only text or combining both types of information.
It has been shown that combining both types of information improves greatly the performance of the system. For future work it would be interesting to check if other type of metadata could also provide useful information for the task, as for example the location or time where the image was taken since there are regions and seasons which are more prone to flooding. It would also be interesting to study why adding more images to the training set has worsened the performance of the system and how well does the system generalize to images outside of the dataset.
This work was partially supported by the Spanish Grants TIN201675404-P AEI/FEDER, UE, TIN2014-52072-P, TIN2016-79717-R, TIN201342795-P and the European Commission H2020 I-REACT project no. 700256. Laura Lopez-Fuentes benefits from the NAERINGSPHD fellowship of the Norwegian Research Council under the collaboration agreement Ref.3114 with the UIB. Marc Bolaños benefits from the FPU fellowship.