This paper describes the approaches used by our team (MultiBrasil) for the Multimedia Satellite Task at MediaEval 2017. For both disaster image retrieval and flood-detection in satellite images, we employ neural networks for end-to-end learning. Specifically, for the first subtask, we exploit Convolutional Networks and Relation Networks while, for the latter, dilated Convolutional Networks were employed.
Natural disaster monitoring is a fundamental task to create
prevention strategies, as well as to help authorities to act in the control
of damages. In its first appearance at MediaEval, the Multimedia
Satellite Task [
For the DIRSM task, we employed Convolutional Networks (CNN) [
In runs 1, 2, 3, and 4, we used neural networks and trained them for classification, with the positive class being a flooding event. In those runs, the final ranking was created by sorting the test set with respect to the classification score, from highest to lowest. Thus, ideally, images with flooding events should have higher score and appear first in the ranking. None of the runs used additional datasets.
Run 1: This run, which focuses only on visual data, employed
GoogleNet [
Run 2: A relevant portion of the available metadata are tags
and descriptions that are not necessarily well-written sentences.
In addition, the amount of available data also discourages the use
of recent neural networks designed for learning from text data,
as many of them are large architectures based on convolutional
or recurrent neural networks, thus requiring larger datasets of
structured sentences. With this in mind, we hypothesized that the
co-occurrence of words is still valuable evidence of a flooding event
and is easier to learn from than structured sentences. For runs 2
and 3, which use text data, we extracted a set of words from each
image metadata and used a neural network to learn from that set
how to classify if an image describes or not a flooding event. This
set is the union of the set of words extracted from the description
and the set of words occurring in tags. To obtain the set of words
from the description, we remove any HTML, non-letter symbols,
stop words, and then apply a standard stemming procedure [
We designed a RN to learn from the sets of words for run 2. In order to create a representation for words, we built a word dictionary from all words in the training data, assigning an integer ID to each word. The first layer of the network is a fully connected layer that takes one-hot encoded vectors representing words as input (ID of the word is the index of the ‘1’ value) and outputs a 32-dimensional vector. Using this strategy, vector representations improve as learning goes on, while not requiring large word datasets.
Run 3: Since we had access to both images and their metadata,
we use an architecture that incorporates a CNN for image data and
a RN similar to the one used for run 2 for the metadata. The CNN
is a ResNet-18 [
Run 4: We proposed an alternative solution for run 1. We split the training images into 5 disjoint subsets, which are combined by taking 4 out of 5, covering all possible combinations of the 5 sets (similar to a 5-fold cross validation). This process results in 5 distinct and combined sets (each one composed of 4 subsets), which are used to fine-tune 5 independent GoogLeNets. In the prediction phase, we average the scores of being a flooding event given by each network and then rank the test set with them.
Run 5: We used a metadata-based approach based on an IR ranking solution, ranking test samples based on their estimation of being flood. That estimation comes from a metric evaluation over a ranked list for each test sample, computed as follows. Let D = {d1, d2, . . .} be the dev set, and T = {t1, t2, . . .} be the test set. Also let ⟨M, F ⟩ be a pair of a representation model M and a distance function F , in which M is applied over a sample s (the query from an IR perspective) and produces M(s), and F is applied over a pair of samples previously modeled by M, so F (M(s1, s2)) = fs1,s2 corresponds to the distance of s1 to s2 with respect to M and F . With ⟨M, F ⟩ and a test sample t , we generate a ranked list r (t ) that contains up to |D | pairs of ⟨di , ft,di ⟩, where di ∈ D, whose pairs are sorted by ft,di .
We tested ⟨M, F ⟩ pairs, then selected the ones who performed
best on dev set. For the three best pairs, we produce three ranked
lists for a sample. We use a graph-based rank-aggregation technique
to produce an unified ranked list. By applying the same procedure
to the dev samples as well, as if they were also queries, we generate
graphs that combine their ranked lists, thus we end up with graphs
for every sample. Given a test graph, we compare it to the dev
graphs and produce a final ranked list. A graph-based dissimilarity
function [
The three best ⟨M, F ⟩ pairs were ⟨RF, WGU⟩, ⟨bigrams-TF, cosine⟩,
and ⟨BoW-TF, cosine⟩, where TF is a weighting functions, RF
(relative frequency) is a graph-based text representation model [
For the FDSI task, we employed CNNs with dilated (or a-trous)
convolution [
In this subtask, we proposed 4 CNNs. The most important one, which was exploited in all runs, is composed of 6 dilated convolution layers and the final fully-connected one, which is responsible for the classification. There are no poolings or normalizations inside the network. The first two convolutions have 5 × 5 filters with dilation rate 1. Convolutions 3 and 4 have 4 × 4 filters but larger rate 2. Finally, the last convolutions have smaller filters ( 3 × 3) but larger dilation rate 4. In a pre-processing stage, we normalized the images using the mean and standard deviation of each image band.
Run 1: We trained the aforementioned CNN by using overlapping patches of size 25 × 25 extracted from all training images. In the prediction phase, we also extracted overlapping patches with the same resolution from the testing images and averaged the probabilities outputted by the network.
Run 2: We processed the images exactly as in run 1 but using a larger patch, with 50 × 50 pixels, which tends to aggregate more context that could improve the learning process.
Run 3: We combined the features extracted from several distinct
CNNs using a linear SVM. Specifically, the SVM receives as input
features extracted from CNNs trained in run 1, 2, and 5, as well
as: (i) a dilated CNN with pooling layers (but that do not reduce
the resolution giving the padding), and (ii) two networks based
SegNet [
Run 4: We combined all networks presented in run 3 using a majority voting scheme.
Run 5: We trained a specific dilated CNN (using patches of 25 × 25) for each of the six locations, i.e., we had one network specialized for each location. The prediction is similar to run 1, except for the use of each CNN in its respective location. For the new location, we combined the features extracted from each CNN using a linear SVM, just like run 3. 4
Table 1 presents our results for the DIRSM subtask. The best results considering AP@480 was achieved by run 3 (95.84%), the neural network solution that combines textual and visual data. However, considering the MAP@[50,100,250,480] the visual only approach that combines results from 5 fine-tunned networks from GoogLeNet (91.59%) stood out (e.g., run 4). In both cases, the results of neural networks surpassed by far those yielded by the IR approach (run 5).
We thank FAPESP, FAPEMIG, CNPq, and CAPES.