This paper presents the method proposed by team UTAOS for the Mediaeval 2017 challenge on Multi-media and Satellite. In the first task, we mainly rely on diferent Convolutional Neural Network (CNN) models combined with two diferent late fusion methods. We also utilize the additional information available in the form of meta-data. The average and mean over precision at diferent cut-ofs for our best runs are 84.94% and 95.11%, respectively. For challenge two, we utilize a Generative Adversarial Network (GAN). The mean Intersection-over-Union (IoU) for our best run is 0.8315. Linking social media information to remote sensed data holds large possibilities for society and research [1-3]. The Multimedia and Satellite task in Mediaeval 2017 [4] aims to integrate information from both sources, sensed data and social media, to provide a better overview of a disaster. This paper provides a detailed description of the methods developed by the UTOS team for the Mediaeval 2017 Multimedia Satellite Task. The challenge consists of two sub tasks, (i) Disaster Image Retrieval from Social Media (DIRSM) and (ii) Flood Detection in Satellite Images (FDSI).
2.1
To tackle challenge (i), we rely on Convolutional Neural Network
(CNN) features. In detail, we first extract CNN features for seven
diferent models from state-of-the-art architectures pre-trained on
the ImageNet [
All in all, we extract eight feature vectors through four diferent network architectures from the same image. AlexNet and VGGNet16 provide a feature vector of size 4096 while GoogleNet and Resnet provide feature vectors of 1024 and 2048, respectively. Subsequently, the extracted features are fed into ensembles of Support Vector Machines (SVMs), which provide classification scores in
2.2
For the challenge (ii), we started from the visual analysis of the provided development set. We observed that it is not possible to use any already existing open-source framework due to the nature of the provided satellite data. Furthermore, we observed that the used four-channel 16-bit TIFF file format is too specific and cannot be correctly processed and even viewed by existing libraries.
To perform the visual analysis we developed a conversion code which provide a conversion from geo-TIFF to a pair of images: RGB and infrared (IR). For the RGB images we used the per-threechannels normalization which fits all the R, G and B pixel values of the input geo-image into standard 0-255 RGB region. Normalization coeficients are the same for all three channels to achieve real color balance even in cases of low variations in one of the components. The normalization of the IR component is performed separately. rдbmin = min(min ri , min дi , min bi )
i ∈R i ∈G i ∈B rдbmax = max(max ri , max дi , max bi ) i ∈R i ∈G i ∈B irmin = min irk , irmax = max irk
k ∈I R k ∈I R ∀i ∈ {R |G |B} {r |д|b }i∗ = ({r |д|b }i − rдbmin ) ∗ 255
rдbmax − rдbmin ∀k ∈ I R iri∗ = (irk − irmin ) ∗ 255
irmax − irmin
Moreover, we performed a human-expert-driven visual
analysis of the images and found them all to be non-contrast, blurry
and color-range-limited. From our previous experience [
As the basis for our method we selected a neural network
architecture used for retinal vessel segmentation in fundoscopic images
with generative adversarial networks (V-GAN) 4. The V-GAN
architecture is designed [
V-GAN is implemented in Python on top of Keras with Tensorflow GPU-enabled back-end. We have modified the network architecture by changing the top-layers configuration in order to support four-channel floating-point geo-image-compatible input. The final generator network output layer used for creation of probabilistic output segmentation image was extended by the simple threshold activation layer to generate the binary segmentation map.
First, we have performed experiments with the development set only and found that the modified V-GAN is able to perform the segmentation of the provided satellite images, but the estimated performance metrics were below the expected level. Additional visual analysis of the converted RGB and IR images showed that sometimes IR component of the sourced geo-images was irrelevant to the flooding areas that probably caused our GAN to bias during training process and prevent it from the correct flooding areas properties extraction. Thus, we have decided to exclude IR component from the model input and process only the RGB components of the converted normalized geo-images. This resulted in the significant performance improvement and correct segmentation most of the developments set flooding areas except for the some images taken in not-common lighting and cloudy conditions. 3 3.1
For DIRSM, we submitted five diferent runs. Table 1 provides the oficial results of our methods in terms of average precision at cutof 480 and mean over precision at diferent cutof ( 50, 100, 250, 480). Run 1 and run 4 are mainly based on visual information extracted with seven diferent CNN models and jointly utilized in PSO and IOWA based fusions, respectively. As it can be seen in Table 1, the PSO based fusion method outperforms IOWA with a significant gain of 3.79% and 5.34%. On the other hand, run 2 is based on metadata achieving the worst results among the all runs. Similarly, run 3 and run 5 represents two diferent variations of our method used for combining meta-data and visual information. Run 3 is based on IOWA while run 5 represents our PSO based fusion of meta-data and
visual information. Again, PSO based fusion performs better. One of the main limitations of IOWA based fusion is its mechanism of assigning more weight to a more confident model. In this particular case, we noticed that our classifier trained on meta-data provides more confident decisions with high probabilities causing significant reduction in the performance. This can also be concluded from the results on run 2 where the meta-data obtain worst results. The degradation in the performance due to the inclusion of meta-data shows that the additional information available are not much useful. This paper provides a detailed description of the methods proposed by UTAOS for the Mediaeval 2017 challenge on Multimedia and Satellite. During the experimental evaluation of sub-task 1 (DIRSM), we noticed that visual information seems more useful compared to meta-data for the retrieval of disaster images. For sub-task 2 (FDSI), we rely on a Generative Adversarial Network where better results are obtained in 3 and 4. Based on the experiments conducted in this work we believe that a proper fusion of social media information and satellite data can provide a better story of a natural disaster.