In this paper we study and compare several approaches to detect lfoods and evidence for passability of roads by conventional means in Twitter. We focus on tweets containing both visual information (a picture shared by the user) and metadata, a combination of text and related extra information intrinsic to the Twitter API. This work has been done in the context of the MediaEval 2018 Multimedia Satellite Task.
Social media are becoming, year by year, ever more popular and
used for sharing people daily activities. This massive adoption
brought to a large availability of contents, such as texts and pictures,
in the most varied sectors, making the social media a great source of
information. The large availability of data is precious for extracting
knowledge and it lays the basis for several applications. One of
them falls in the context of emergency management related to
natural disasters, in which computer vision and machine learning
techniques are investigated [
In this work we focus on analyzing social media posts in order to
extract valuable information about roads afected by flood. In more
detail, we propose a multi-modal deep learning network which
processes flood-related social media pictures and related metadata (e.g.,
Twitter, Flickr, YFCC100M), both to provide (a) evidence of roads
and (b) whether they are also passable. The approach presented in
this paper takes his inspiration (and it wants to be an extension)
from the work [
Recent literature approaches leverage on satellite [
The dataset used in this work was distributed by MediaEval 2018
Multimedia Satellite Task [
To properly deal with heterogeneous data, we opted for a "divideet-impera" approach: we created a model for each kind of data. In detail we developed: (i) a classification model using only the metadata information, (ii) three classification models using only the images, (iii) a model which combines the metadata and the visual information. In this section we will briefly describe the three diferent systems. 4.1
We processed metadata (i) filtering out properties not available
in the whole dataset, (ii) studying the correlation among the
remaining ones and selecting the most relevant ones. As a result
we kept the text written by the user, the language in which the
tweet was originally written, the number of retweets and the
number of persons who had favourited the tweet. As a pre-processing
step we have translated all texts to English, removed emojis, urls
and special characters; then we used lemmatization and
tokenization techniques. We represented the text features using a word
embedding initialized with Glove [
Both tasks have been trained in parallel. Initially, we set all the images which had no evidence of flood as having no passability issues either. However, this strategy introduced a big imbalance to an already imbalanced dataset which made the training more dificult, so we finally decided to use only the images containing evidence of flood to train the passability classifier, while still doing the training in parallel. 4.2
As a pre-processing step for the images we applied several data augmentation techniques: image rotation, width and height shifts, horizontal flip and zoom. We designed two diferent systems to process the images, which we will compare.
Double-ended classifier with compact loss: We used the
Inception V3 [
Network staking: We used 9 state-of-the art networks (InceptionV3, Xception, VGG16, VGG19, InceptionResNetV2, MobileNet, DenseNet121, DenseNet201, NASNetLarge) all of them pre-trained on ImageNet. They were separately trained for both problems in 5 diferent train-validation folds, which generated 90 networks (45 per task). The output of each network is a number between 0 and 1 which represents the probability of the picture containing evidence of roads in flooded regions and evidence of passable roads, respectively. Being n the number of networks and pi the probability of the picture corresponding to class 1, we define p as the average of pi for all 1 ≤ i ≤ n. We define votinд(p1, ..., pn ) = |{i/pi > 0.5, 1 ≤ i ≤ n}|, where |.| is the set cardinality. We used two diferent methods to aggregate the results from the networks: (i) Average aggregation: pred(p1, ..., pn ) = (p > 0.5), (ii) Average and voting aggregation: 1 pred(p1, ..., pn ) = 0 if p > 0.5 and votinд(p1, ..., pn ) > n2 − 2 or 6 p > 0.45 and votinд(p1, ..., pn ) ≥ n2 , otherwise. 4.3
To combine the metadata and the images, we merged the Metadataonly and the Double-ended classifier with compact loss approaches. The two networks were taken without their respective doubleended fully-connected (FC) layers and merged with two newer FC layers (one per task) with a softmax classifier. 5
To get a first idea of the upper-bound for our task we asked 4 persons to do the task on a subset of 50 images, the results are given in Table 1. In the subsequent rows we have included the results for the 5 diferent approaches introduced in this paper. As it can be seen, our approach for flood evidence classification using metadata obtains very poor results but close to the results obtained by human annotators, which means that the metadata was not very discriminative for this task. Since the error is cumulative, the results of both the humans and the metadata classifier drop significantly for the passability detection, being the F-score again very close to one another. All the image classification approaches achieve similar results on the first task, while the network stacking achieves a small improvement with respect to the double-ended classifier with compact-loss network. Furthermore, the aggregation of the networks using average and voting slightly improves the aggregation compared to using only the average. However, there is a big gap between the performance of the double-ended classifier with compact loss and the network stacking approaches. When we decided to use the same network body to perform both tasks we thought that since the tasks were very related, one task could benefit from the others knowledge. However, since one task was more dificult than the other, the double-ended network seems to have specialized on the easy task while leaving aside the dificult task. It is also remarkable that the network stacking algorithms achieve significant better results than the human annotators, probably because: (i) road passability is subjective in several cases and (ii) while the network learnt over the whole training set, the human annotators were not given any examples about the task.
Finally, combining metadata with images does not provide much improvement or it even worsens the results due to the lack of discriminative features in the metadata.
In this paper we studied several approaches to perform flood and road passability detection. We proposed several approaches to deal with textual and visual information. According to our tests, we discovered that when a network tries to accomplish several tasks with diferent dificulties, even if they are related, it focuses on one of them (presumably the simplest one), achieving good performance in one case, but bad in the latter one.
This work was partially supported by the Spanish Grants TIN201675404-P AEI/FEDER, UE, TIN2014-52072-P, TIN2016-79717-R, TIN201342795-P and the European Commission H2020 I-REACT project no. 700256. Laura Lopez-Fuentes benefits from the NAERINGSPHD fellowship of the Norwegian Research Council under the collaboration agreement Ref.3114 with the UIB.