This paper presents the various algorithms that the CERTH-ITI team has implemented to tackle three tasks that relate to the problem of flood severity estimation, using satellite images and online media content. Deep Convolutional Neural Networks were deployed to classify articles as flood event-related based on their images, but also to detect flooding events in satellite sequences. Remote sensing indices play a key role in the machine learning approach to identify changes between satellite imagery, while visual and textual features were exploited to estimate whether an image shows people standing in flooded areas.
News websites now play a crucial role in the field of public
information, turning into a rich and open source of articles and images
that cover numerous events. At the same time, the high availability
of satellite data induces an alternative source of imagery. This data
can be exploited in the domain of natural disasters, e.g. to detect a
lfooding incident or to estimate the severity of a flood. Several
ongoing H2020 projects follow this direction: beAWARE [
The Multimedia Satellite Task is a challenge of MediaEval that
consists of the following subtasks. News Image Topic
Disambiguation (NITD) entails an image classifier that is able to identify whether
or not an image belongs to a flood-related article. Multimodal Flood
Level Estimation (MFLE) calls for a classifier that receives visual
and/or textual information from articles and predicts whether or
not an image contains people standing in water above the knee.
Finally, City-Centered Satellite Sequences (CCSS) asks participants
to detect a flooding incident by using sequences of satellite images.
For further details on the subtasks and the respective data sets, the
reader is referred to [
The next section presents the algorithms proposed by the CERTHITI team for each subtask, followed by the results of their evaluation and a short discussion with conclusions. 2.1
We aim to classify news articles’ topics judging from the images
that appear in them. One challenge of this task is that inside the
images flooded areas may be completely out of view. Even more
challenging are the instances where a flooded area is clearly shown
in the image but the article’s topic is not relevant to a flood event.
Also, in some instances water is present but not in the context of
lfoods (e.g. a beach). In order to examine the performance of
stateof-the-art image classification techniques [
We acquire the VGG architecture pre-trained on the Places365
dataset [
The estimation of flood level involves checking whether or not an
image contains people standing in water above the knee, and it is
realized by considering machine learning techniques on visual and
textual information. Regarding the visual information, a 22-layer
GoogleNet network was fine-tuned and the dimension of the
classiifcation layer was set equal to 345 [
Regarding the textual information, we followed a well-established
approach in text classification called word2vec [
Finally, a simple late fusion approach was followed in order to consider both visual and textual information, so the outputs of the above two modules are considered for deciding the fused approach prediction. If the output of the two SVM binary classifiers coincide, then their common label defines the label of the fused module; otherwise, only the output of the visual module is considered. 2.3
The first approach to detect flood events using satellite sequences
involved the use of a deep learning model which was trained on
two diferent datasets of three-channel images with the diferences
of two days within an event. The first dataset was created by
combining the Red-Green-Blue (B02-B03-B04) bands and the second by
combining the Red-Swir-Nir (B02-B03-B04). Then, the three bands
were stacked and converted to JPEG. Within each event, the unique
diferences between its days were calculated. Next, pre-trained
networks on ImageNet [
An additional change detection approach based on the remote
sensing water index of MNDWI [
As a third approach, outlier detection was also performed on water body masks, produced by zero thresholding of the MNDWI index. Counting the water pixels of each day of an event generated time series of integers. Then, Z-score was calculated per each point as x − m / σ , where x is the value of each point and m and σ are the mean and standard deviation of all points in the time series. If a point exceeded a threshold γ , it was considered an outlier and thus the complete sequence of images was classified as an event. 3
The complete results in the dev set and the test set for all three subtasks of the Multimedia Satellite Task can be seen in Table 1, where it is evident that the DCNN approaches in NITD and the image diferencing technique in CCSS really stood out. In detail:
NITD Examining the errors, we observe that the article classifier is mainly producing False Positives and very little to none False Negatives. Many of the FP cases actually show flooded areas, although the article topic is negative to a flood event. On the test set, the 2019 model performs better than the 2017 model reaching an accuracy of 90.2%. We hypothesise that it is performing better because it has learnt correlations beyond the obvious: a flooded area in an image is a strong sign of flood relevancy in the article but certain groups of people appearing may also be a positive flag, like authorities or politicians. This is expected to hold true, especially if the training and the test set are taken from a single event where the same people appear frequently on the news articles.
MFLE The exploitation of visual information reaches a ∼65% F1-score, due to the significant number of FP, since the concept detection focused on the identification of humans and water and it didn’t restrict to images of people standing in water above the knee. The textual information features performed slightly lower to the visual ones, while the fusion of visual and textual features performed equally to the visual, which can be easily explained by the aforementioned description of the approach.
CCSS Detecting the outliers on the diferences of MNDWI consecutive images achieved a 76.47% F1-score. The image diferencing technique proved adequate to detect changes relative to flood events, using the σ and minimum water _ratio values that were calculated on the annotated dev set. Using DCNN provided decent results (70.58%), showing its ability to learn to detect flood patterns even with a small training set. On the other hand, outlier detection on water masks, using MNDWI index and setting γ to 2, did not accomplish a high F1-score (54.41%), possibly due to the fact that all the remote sensing information was limited to a binary mask. 4
Through the participation in the Multimedia Satellite challenge, the CERTH-ITI team gained the opportunity to examine various methodologies for the problem of flood detection. Results for the NITD task indicate that it is possible to classify flood event articles with good accuracy using either a generic flood detector or by annotating a specific dataset. However, the second approach looks more promising when dealing with articles concerning a single event. Results of the MFLE task show that visual features perform better than the textual ones, but they could be further improved if a segmentation step was applied on top of the proposed approach for recognising whether water covered people below the knee. Finally, results of the CCSS demonstrate the ability of the combined method of image diferencing and water relative index of MNDWI to detect lfood events, showing better robustness with balanced FP and FN rates, compared to the DCNN approach , whereas the three extra layers of VGG19 don’t show any impact on the learning process.
This work was supported by EC-funded projects H2020-700475beAWARE and H2020-776019-EOPEN.