This paper provides a description of the MediaEval 2017 Multimedia Satellite Task. The primary goal of the task is to extract and fuse content of events which are present in Satellite Imagery and Social Media. Establishing a link from Satellite Imagery to Social Multimedia can yield to a comprehensive event representation which is vital for numerous applications. Focusing on natural disaster events in this year, the main objective of the task is to leverage the combined event representation withing the context of emergency response and environmental monitoring. In particular, our task focuses this year on flooding events and consists of two subtasks. The first Disaster Image Retrieval form Social Media subtask requires participants to retrieve images from Social Media which show a direct evidence of the flooding event. The second task Flood Detection in Satellite Images aims to extract regions in satellite images which are afected by a flooding event. Extracted content from both tasks can be fused by means of the geographic information. The task seeks to go beyond state-of-the-art flooding map generation towards recent approaches in Deep-Learning while augmenting the satellite information at the same time with rich social multimedia.
Recent advances in earth observation are opening up a new
exciting area for exploration of satellite image data. Programs like
ESA Copernicus, NASA Landsat, and private companies like
PlanetLabs or Digital Globe provide access to such imagery, for the first
time. Large-scale datasets such as the EuroSAT-Dataset [
One challenge when solely relying on remote sensing is the sparsity problem of satellite data over time. Due to the delayed receiving time of satellite imagery and low temporal revisit time of a particular location by satellites, locations are often sparsely sensed with missing information. In the context of natural disaster monitoring, where efects are often present at multiple locations at the same time, missing information represents a crucial problem since humanitarian organizations and rescuer eforts need to rely on up-to-date disaster maps.
In order to overcome this problem and provide an accurate and
comprehensive view of the event, the objective of this task is to fuse
satellite imagery with real-time multimedia content from Social
Media. Our approach is motivated by previous work in [
The goal of the first subtask is to retrieve all images which show direct evidence of a flooding event from social media streams, independently of a particular event. The objective is to design an algorithm that given any collection of multimedia images and their metadata (e.g., YFCC100M, Twitter, Wikipedia, news articles) is able to identify those images that are related to a flooding event. Please note, that only those images which convey a visual evidence of a lfooding event will be considered as True Positives. Specifically, we define images showing ”unexpected high water levels in industrial, residential, commercial and agricultural area“ as images providing evidence of a flooding event.
The main challenges of this task lie in the proper discrimination of the water levels in diferent areas (e.g., images showing a lake vs. showing a flooded street) as well as the consideration of diferent
types of flooding events (e.g., coastal flooding, river flooding, pluvial lfooding). Participants are allowed to submit 5 runs: • Required run 1: using visual data only • Required run 2: using metadata only • Required run 3: using metadata-visual data only fused without resources other than those provided by the organizers • General run 4, 5: everything automated allowed, including using data from external sources (e.g. Twitter, Flickr)
The aim of the second subtask is to develop a method that is able to identify regions in satellite imagery which are afected by a flooding. Participants are given a set of satellite image patches for multiple instances of flooding events along with corresponding segmentation masks for the flooding to train their models. Participants report for the unseen image patches a segmentation masks of the flooded area. Participants are allowed to submit 5 runs: • Required run 1, 2, 3: using satellite data only • General run 4, 5: everything automated allowed, including using data from external sources
The dataset for the first subtask consists of 6,600 Flickr images. All
images were extracted from the YFCC100M-Dataset [
Images with the tags of flooding, flood and floods were selected and additionally refined by human annotators according to the strength of the evidence of flooding that they depict: very strong non-evidence of a flooding (0), non-evidence of a flooding (1), direct evidence of a flooding (4), very strong direct evidence of a flooding (5), or with “don’t know” answer (3). The definition of relevance was available to the annotators in the interface during the entire process. The annotation process was not time restricted. The scores were collected from two annotators and the final ground truth label was determined as flooding if both annotators rated the image with 4 or 5 and as non flooding for scores of 0 or 1. To cover a broader diversity of images, we injected additional distractor images in the dataset.
For each image, image metadata from YFCC100M and visual feature descriptors are provided to participants. Visual features were extracted with the open-source LIRE library2 using default parameter settings. A overview of the provided features is given 2LIRE, http://www.lire-project.net/ in Table 1. The dataset is separated with a ratio of 80/20 into the following two sets: • Development-Set contains 5,280 images, along with features and class labels (1=evidence of a flooding event and 0=no evidence) • Test-Set contains 1,320 images and features
The dataset for the second subtask consists satellite image patches
which have been derived from Planet’s 4-band satellites [
The oficial metric for evaluating the correctness of retrieved images from Social Media is Average Precision at k (AP@k) at various cutofs, k=50,100, 200, 300, 400, 500. The metric measures the number of relevant images among the top k retrieved results and takes the rank into consideration.
In order to assess performance of generated segmentation masks for flooded areas in the satellite image patches, the intersectionover-union metric (Jaccard Index), is used for the oficial evaluation: IoU = TP / (TP + FP + FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively, determined over the whole test set. The metric measures the accuracy for the pixel-wise classification.
We would like to thank Planet for providing us with high resolution satellite images for this task. Additionally, this work was supported BMBF project MOM (Grant 01IW15002).