In this paper, we present an approach and first results for the MediaEval 2019 sub-task on “Multimodal Flood Level Estimation from News” from the “2019 Multimedia Satellite Task”. The water level is measured by detecting people standing in water and using the human body as a reference. We focus only on the visual modality and propose a combination of a ResNet-based water detector and pose estimation to solve the task. First results are promising and show that our approach is clearly performing above baseline. 1http://www.multimediaeval.org/mediaeval2019/multimediasatellite, 01.10.2019
The assessment of natural disasters by automated media analysis
becomes increasingly important as, on one hand, the amount of
user-generated media has been rising with the availability of smart
phones and, on the other hand, the likelihood of disasters increases
e.g. due to the ongoing climate change. The availability of (social)
media data represents an opportunity to automatically detect and
assess disasters to better guide first responders and emergency
forces. The types of disasters targeted in this work are floods and
the particular task to solve is flood level estimation [
Disaster recognition based on social media images has been a rising
topic recently. Flood level estimation is technically challenging as
shown by Hostache et al. [
Since the provided dataset is multimodal, our initial idea consisted of training two diferent classifiers for the image data and a separate one for the text data and then fusing the predictions. Due to insuficient text data we decided to provide predictions based only on visual data. We developed two classification approaches. The first approach (see Figure 2a) relies on detecting water within the whole image and detecting at least one person with obscured lower body parts. The second approach (see Figure 2b) performs local water detection. To this end, for each human body detected, a patch that also contains the local neighbourhood of the human body is taken into account for water detection. If at least for one patch in the image our model detects obscured lower extremities and water in the vicinity, the image is assigned to the positive class. Both proposed approaches build upon three main components: (i) a water detector that predicts whether a certain image or image region contains water, (ii) a pose estimator that detects people and ifts skeletons into their bodies and (iii) a rule-based fusion module that combines the information from the water detector and the pose estimator to make a final decision.
Water detector: we build upon ResNet50, which is pre-trained on ImageNet and fine-tuned for water/no water detection using images showing either water or not. Images are resized, using nearestneighbor interpolation, to the network’s input size (227x227) while keeping the original aspect ratio. Horizontal flipping, brightness variations and non-uniform re-scaling of the images are applied for data augmentation. The top five layers are fine-tuned (for 6 epochs, batch size 256) before the whole network (for 10 epochs, batch size 32) is trained using the Adam optimizer (learning rate of 10−4).
Pose estimator (OpenPose): we employ OpenPose [
Rule-based Classifier : to determine whether the lower extremities of a detected skeleton are visible we employ the following heuristic rule: CU /max(CL, 10−4) > T , with CU and CL being the mean detection confidence for the upper and lower body and T an empirically determined threshold of 1.5. The max operator prevents division by zero.
Final decision rule: A positive detection of a person standing in water is declared when both the rule-based classifier and the (local or global) water detector predict positively. 4
We train our models on 80% of the training data and use 20% from
each class for validation (randomly selected). For Run 1, we use
only the data provided by the organizers, but we manually label
the images regarding whether or not they contain water. In all
other runs, we further use the data from the Multimedia Satellite
Task from 2018 [
The performance of our approach is almost the same on our validation and the benchmark test set, which shows that our approach generalizes well. The overall performance is around 60% for all runs and does not show a significant diference between the local and global approaches. Similarly, the fusion of both (Run 3) does not outperform our baseline (Run 1). The random baseline for this task depends on the class cardinalities in the test set and is thus unknown to the authors. It has, however, an upper limit of 50% for the task due to the use of the macro averaged class-wise F1-scores as performance measure. Our approach outperforms this baseline, which shows that it learns useful patterns related to the target task, although there is room for improvement. A closer analysis of the results shows several directions for improvement. While the water detection is quite robust (classification accuracy of 0.88; model is trained on data from last year’s task as well as 700 images from this year’s task and evaluated on 200 images from this year’s task), we observe numerous false and missed detections of OpenPose. Furthermore, reflections of the human body on the water surface represent a problem, i.e. for the detection of lower extremities. In several cases, OpenPose added body parts for the lower extremities, which were actually under water (see right image in Figure 1). 5
In this paper, we presented our contribution to the MediaEval 2019 task on flood level estimation from news media images. Our approach combines a pose detector and a water detector to find images showing people standing in water above the knee. First results show a promising generalization ability. Concerning the overall performance, improvements are possible. A promising approach to increase robustness is the use of several human (pose) detectors trained on diferent data (e.g. urban and rural setting). A limitation of our approach is that not only water but also other objects can obscure the lower extremities of a person or that only torso or head are shown in the picture. As a result, the lower extremities cannot be detected and if water is present, our approach may fail. In order to compensate for these efects, pixel-wise segmentation of water and humans could be advantageous. Additionally, pixel accurate data could help to detect false detections by OpenPose, e.g. detected body parts protruding out of the segmented area, which represents a human body, should not be considered.
The work in this article was supported by the Austrian Research Promotion Agency FFG under grant no. 856333.