This paper presents the contribution of Team MC-FHSTP to the multimedia satellite task at the MediaEval 2018 benchmark. We present two methods, one for the estimation of the passability of roads from social media images due to flooding and one method that estimates passability from satellite images. We present the results obtained in the benchmark for both methods.
Flood threats have prompted many researchers to develop
technologybased solutions for the precise and autonomous exploration of flood
areas. Such solutions should enable the assessment of the impact
of hazards as well as the immediate response to disasters. This can
be done by analyzing satellite images. In the work of Pradhan et
al. [
In this paper we are aiming at determining road passability
from images as proposed in the Multimedia Satellite Task of the
MediaEval 2018 benchmark [
The following two sections describe our approaches for both subtasks. The idea behind both approaches is to provide a simple, reproducible and straight-forward baseline for the respective tasks. 2.1
The dataset provided for sub-task one consists of twitter textmessages with accompanying images and is therefore a multimodal data-source. Our initial idea consisted of training two diferent classifiers for the image data and a separate one for the text data and then merging the predictions. This should allow the information from both inputs to be used for the final decision.
For the textual data, we investigated several methods. All tweets that were not in English or Spanish were excluded from further processing. Then the Spanish tweets (a representative minority in the data) were translated into English using the Google Translate API. We generated Bag-of-Words(BOW) descriptors on the N = 100 and N = 50 most common words and used them separately for training. Since the above mentioned representation did not deliver promising results on the training data, we chose to calculate TFIDF representations.
For the image data, two approaches were developed. First, we extracted a Bag-of-Words descriptor based on the responses (tags) retrieved by the clarifai1 API for the twitter images and fed it into a Support Vector Machine (SVM). Since preliminary results were, however, little promising on the development data, we excluded the approach from the final evaluation. We assume, that the main problem with the clarifai tags was that they were not specific enough to solve our task. In a second visual classification approach we leveraged a deep neural network for image description and classification (ResNet50 pre-trained on ImageNet2). The original network was extended with two densely connected layers (of size 512 and 256, respectively) and with two dropout layers (likelihood 0.5) in between and a softmax layer on top. Initially, in all experiments only the added layers were trained for 20 epochs. Depending on the run (see below), in a further step the whole network was fine-tuned within a variable number of epochs. Prior to network training the provided twitter images were pre-processed in two ways: first, the images were rescaled (non-uniformly) to the required input size of the network (224x224), and second, a central patch was cut out from 1https://www.clarifai.com/ 2http://www.image-net.org/ the images to put more focus on the image center. Furthermore, we applied rotation and mirroring as data augmentation steps. 2.2
For the prediction of road passability between two given points in a satellite image we made a simplifying assumption. We assumed, that at least one of the two given points is under water in case the road segment defined by the two points is not passable. Consequently, we assumed that the patches around points under water share visual properties. We modelled our assumption as follows. Patches: From each satellite image, we extracted a patch of 50x50px around each of the two given points. Visual features: From each patch, we extracted RGB histograms with 16 bins per channel. Training and Classification : We trained SVMs. For the evaluation of our approach on the development data we used 10-fold cross validation. For the test data we trained the SVMs with the complete development data set. We used three aggregation methods for the RGB histograms: (1) Concatenation (concat): We concatenated the RGB histograms of the two patches of a satellite image to a feature vector with 96 dimensions and trained an SVM. (2) Separation (sep): We trained two SVMs, one with the patches of the first point of each image and the second with the patches of the second points, respectively. We predicted, that a satellite image contains a passable road, if the predictions for both patches of the image are passable. (3) Joined (join): As in aggregation (sep), we kept the RGB histograms of the two patches per satellite images separate. Instead of training two SVMs, we trained one SVM with all patches, i.e. we used a training data set with double the number of samples. We predicted as in the (sep) case. In a first baseline experiment, we evaluated this assumption. The results were surprisingly useful, consequently we decided to use the approach. 3 3.1
Since the initial data set was unbalanced, we divided the data into a balanced training set to avoid any bias towards a class during training and an unbalanced validation set (that contains the remaining development data). The training set contained 874 samples per class. The results for the training, validation and test phase in terms of F1-score and classification accuracy (in percent) are summarized in Table 1. In Run 1 we directly used the predictions of the ResNet50 network, which was trained on the balanced training set and evaluated on the unbalanced validation set. We trained the model for five epochs, with a batch size of 32, the Adam optimization algorithm and a learning rate of 0.0001. In Run 2 we employed the SVM model trained with the TFIDF representations which are based on a high min_df value of 120. We skipped Run 3 because we could not gain better results by combining visual and textual information. In Run 4 and Run 5 the predictions where obtained from the ResNet50 network trained on the entire development data for three epochs (Run 4) and six epochs (Run 5). In the training phase Run 2 performed best regarding the F1-score, and Run 5 regarding the accuracy. However, it should be noted that generally a relatively high accuracy was achieved, but a rather low F1-score, which assumably stems from the class imbalance and indicates that the models could not learn all classes equally well. On the test set Run 5 clearly performs best (F1-score = 0.35). The weakest result is obtained by Run 4 which indicates that the training was stopped too early. A prolonged training phase (more than 6 epochs as in Run 5) could further improve the result. Overall, it is notable that the best run (run 5) also has the strongest generalization ability (i.e. F1 on test set is larger than on validation set). 3.2
This work was supported by the Austrian Research Promotion Agency (FFG), Project No. 856333.