This paper presents the method proposed by team UTAOS for MediaEval 2018 Multimedia Satellite Task: Emergency Response for Flooding Events. In the first challenge, we mainly rely on object and scene level features extracted through multiple deep models pre-trained on the ImageNet and Places datasets. The object and scene-level features are combined using early, late and double fusion techniques achieving an average F1-score of 60.59%, 63.58% and 65.03%, respectively. For the second challenge, we rely on a convolutional neural networks (CNNs) and a transfer learning-based classification approach achieving an average F1-score of 62.30% and 61.02% for run 1 and run 2, respectively.
Natural disasters, such as floods, earthquakes and droughts, may
cause significant damage to both human life and infrastructure. In
such adverse events, an instant access to information may help
to mitigate the damage [
This paper provides detailed description of the methods proposed by team UTAOS for MediaEval 2018 Multimedia Satellite Task: Emergency Response for Flooding Events. The challenge is composed of two parts, namely (i) flood classification for social multimedia and (ii) flood detection in satellite imagery. The first task is further divided in two sub-tasks aiming to predict (a) whether there are evidences of a flood in a given social media image or not and (b) if evidences of flood exists in the image, whether it is possible to pass through the flooded road (passability). The second task aims to analyze the roads from satellite images, and predict whether or not it is possible for a vehicle to pass a road. 2.1
To tackle the first challenge, based on our previous experience [
To be able to fuse the complementary information (i.e., object and scene-level features), we use three diferent fusion techniques, namely early, late and double fusion. In the early fusion, we simply concatenate the features extracted through diferent models. In the late fusion, we simply average the results obtained through the individual models. In our third fusion technique, we combine the results obtained from the first two techniques in an additional late fusion step by assigning them equal weights. For classification purposes, we rely on Support Vector Machines (SVMs) in all of the submitted fusion runs. 2.2
For the FDSI part of the task, we initially tried to apply the
wellperforming Generative Adversarial Network (GAN) approach
introduced in our previous works for the flood detection satellite
imagery [
We conducted an exhaustive set of experiments, but unfortunately could not achieve a roads passability detection performance better than random label assignment would achieve. The reason for that is the limited size of the dataset (only 1,437 samples were provided in the development set). This, in combination with the large variety of landscapes, road types, types of obstacles and weather conditions, etc., prevents the GAN-based approach from adequate training and finding key visual features required to reliable distinguish between flooded and non-flooded roads.
Thus, we decided to fall-back to another convolutional neural
network (CNN) and a transfer learning-based classification approach,
which has been mainly introduced for the medical images
classification in our previous work [
For the here presented work, we froze all the basic convolutional
layers of the network and only retrained the two top fully
connected layers after random initialization of their weights. The fully
connected layers were retrained using the RMSprop [
As the input for the CNN model, we used the image patches, extracted from the full images using the provided coordinates of the target road end points. Visual inspection of the training-datasetgenerated roads’ patches showed relatively good coverage for the road-related areas and enough coverage of the neighbourhood areas together give enough visual information for the following CNNbased analysis and classification.
Moreover, in order to increase the number of training samples, we also performed various augmentation operations on the images. Specifically, we performed horizontal and vertical flipping and change of brightness in the interval of ±40%. After the model has been retrained, we used it for a multi-class classifier that provides the probability for each of two classes: passable and non-passable. 3 3.1
During the experimentation on the development set, we observed that the classifiers trained on scene level features extracted through models pre-trained on the places dataset perform better compared to the ones pre-trained on Imagenet. However, we also observed that combining the object and scene level features leads to better results compared to the individual models. In order to better utilize the scene and object level features, we used three diferent techniques in each run. Our first run is based on late fusion where we used equal weights for each model. In the second run, we concatenate the features extracted through the individual models. An SVM classifier is then trained on the combined feature vectors. In the final run, we use double fusion by combining the results of the first two runs in a late fusion way. Table 1 provides the evaluation results of our proposed methods in terms of F1 scores for each of the runs. Overall, better results are obtained with double fusion while least results are obtained with an early fusion of the features. For the experimental setup of the FDSI task, we decided to perform only two mandatory runs which are utilizing the task-provided training data only. Due to a limited amount of training samples available, the usage of the additional road network detection methods is not possible for these runs. Thus, we decided to perform two types of training for our transfer-learning detection approach.
First, we implemented a pipeline for classification that difers from common procedures. This process was involving all the training samples into the training process as both training and validation sets. Usually, for classification tasks, this would result into overiftting of the model and inability to correctly classify the test samples. However, for this specific task, the limited number of training epochs and significant training data augmentation in conjunction with a high variety of road patch samples resulted in normal training process. This allowed to correctly retrain the last layers of the network and produce reasonable classifiers even on such a limited training set.
The oficial F1-Score metric (see table 2) on the non-passable road class for the first "All-train" run is 62.30%. To verify our idea of the usability of using all the training data for both training and validation, we also performed a normal network training with a random 50/50 development/validation data split. This second Halftrained run resulted in F1-Score of 61.02% which is slightly lower comparing to the All-trained run. This is confirming the validity of our idea of using the complete training dataset and heavy data augmentation to improve road patches classification performance. 4
This year, the social media and satellite task introduced a new and important challenge of detecting the passibility of roads, which can be vital for the people lining in the flooded regions. In the social media image analysis, we mainly relied on deep features extracted through diferent pre-trained deep models. During the experiments, we observed that the scene-level information, extracted through models pre-trained on the places dataset perform better compared to the ones pre-trained on Imagenet. However, the object-level information well complement the scene-level features when combined. We also observed that double fusion performs slightly better than the early fusion on the provided dataset. However, it needs to be investigated in more detail. In the future, we aim to analyze the task with more advanced early and late fusion techniques.
On the other hand, in the satellite subtask, we found that just a normal image segmentation approach cannot help, and we implemented an task-oriented CNN and transfer-learning-based approach. This approach was able to classify image patches with roads and achieved an F1-Score of 62.30% for the non-passable road class. In the future, we plan to implement an advanced road network and lfooding detection and segmentation using a combined CNN- and GAN-based approach pre-trained on the existing annotated road network and flooded areas datasets.
Task name as it appears on htp://multimediaeval.org/mediaeval2018