This paper presents the solutions of team EVUS-ikg for the Multimedia Satellite Task at MediaEval 2019. We addressed two of the subtasks, namely multimodal flood level estimation (MFLE) and city-centered satellite sequences (CCSS). For MFLE, a two-step approach was proposed, which retrieves flood relevant images based on global deep features and then detects severe flood images based on self-defined distance features, which can be extracted from human body keypoints and semantic segments. For CCSS, a neural network, which combines CNN and LSTM, was used to detect lfoods in satellite image sequences. Both methods have achieved a good performance on the test set, which shows a great potential to improve the current flood monitoring applications.
Flood, as one of the great natural disasters, endangers people’s
safety and their property. Satellite images are one of the most often
used data sources for flood mapping. However, this is not suficient
to obtain enough evidence for the estimation of local flood severity.
Crowdsourcing, as a rapidly developing method for data acquisition,
has been proved to be beneficial for such a purpose. From social
media, flood relevant posts can be retrieved with image and text
classifiers trained from deep neural networks (e.g. [
In MFLE, corresponding image and text pairs were annotated with
binary labels, which indicate whether the image contains at least
one person standing in water above the knee. For run 1, where only
visual information is allowed, a two-step approach was proposed. A
ifrst classifier was trained for extracting flood relevant images and a
second classifier was then used to detect the images containing
people standing in water above the knee from these relevant images.
We concatenated the features extracted from four CNN models,
namely InceptionV3 [
For run 2, where only textual information is allowed, we used
a TextCNN model [
In CCSS, the sequences are collected from 12-bands Sentinel-2 satellite images with date and time. As a pre-processing step, we performed a normalization by calculating the Z-score (subtracting the mean and then dividing by the standard deviation) for each band individually and then clipped the normalized image into the range from -1 to 1. We used DenseNet121 as a feature extractor, and then connected the features using a LSTM with 32 cells in the temporal direction (Figure 2). We used a many-to-many LSTM, where we required an output for each input image individually. The weights of this DenseNet were initialized with the weights pre-trained on (1) Overlay of semantic segmentation and body keypoints (2) Valid area created by body (3) Extract valid connecting keypoints and image bottom boundary points using convex hull (4) Abstraction with the height of lowest boundary point (5) Extraction of distance feature ImageNet. Since the image sequence lengths are diferent and do not exceed 24 layers, we padded the sequence to 24 layers with same size tensors with zeros and generate a mask indicating which layers are padded. Then, we excluded these padded images when calculating the categorical cross-entropy loss and accuracy metric. During the training, we used only the RGB channels with a reduced image size 256 × 256. Since there are 267 sequences available in devset, we used 170 for training, 43 for validation and 54 as an internal testset. Since some of the images in the sequence may be broken or incomplete, the field FULL-DATA-COVERAGE in the timeseries files can help us to filter these images.
We trained this model with diferent annotation settings. Each
sequence is annotated with binary labels (hereinafter called
seqlabel), where the field FLOODING in the timeseries file of each
sequence provides the layer-level labels (hereinafter called
layerlabel). Our primary observation of the training data shows that,
the layer-labels indicate a strong correspondence to the seq-labels,
where only the sequences annotated with all negative are annotated
with negative in the seq-labels. Thus, in run 1, we simply used these
layer-labels to train this model. Subsequently, we tested diferent
pseudo labels generated from the seq-labels. A repetition of
seqlabels in run 2 (i.e. seq-label is 1, layer-labels are all 1; seq-label
is 0, layer-labels are all 0). Since we observed a strong pattern in
the positive labeled sequences, that the first half of the seq-labels
are negative while the latter half negative. Thus, we followed this
pattern to generate pseudo layer-labels in run 3, where seq-label is
1, layer-labels are [
For MLFE (Table 1), our image based approach can achieve an averaged F1-score of 68.16% on test set. The text based method performs significantly worse than the image based model. Combining both textual and visual information did not improve the F1-score in our case. In run 4, f1-score was improved slightly by introducing additional images for training of the flood relevance classifier. We further exam the failure examples. They can be categorized into three types, namely failed pose detection, failed semantic segmentation and failed water level estimation, where most of them are caused by drawing a wrong water line. This leads to many false positive detections. For CCSS (Table 2), comparing the results from the first three runs using many-to-many LSTM and many-to-one LSTM in run 4, the improvement is obvious. Regarding the diferent annotation settings, run 3 achieved the best performance, where the pseudo labels followed annotation patterns in layer-labels. Run 2 has also achieved a better performance than run 1, which indicates the exact layer-labels may not be necessary to predict if a sequence has flood or not. 4
In this paper, separate solutions have been proposed for subtask MFLE and CCSS. Both models can solve the tasks properly according to their performance on test set. For MFLE, the water line estimation can be further improved in order to reduce false positive detections. For CCSS, the robustness of model can be tested on further events.
This work is supported by project TransMiT (BMBF, 033W105A). The computational resource is provided by ICAML (BMBF, 01IS17076).