In this paper, we describe our approach for the Multimedia Satellite Task: Emergency Response for Flooding Events at the MediaEval 2019 Challenge. Specifically, for the Multimodal Flood Level Estimation subtask, we employ a combination of ResNet-50 trained on Places365 dataset for features extractor, OpenPose for pose estimation and Mask R-CNN for segmentation to predict if an image has at least one person standing in water above the knee. Our approach has achieved the highest results for Multimodal Flood Level Estimation subtask. In this Multimedia Satellite Task, we take part in two subtasks: Image-based News Topic Disambiguation (INTD) and Multimodal Flood Level Estimation (MFLE). We propose using EficientNet features [8] for training a water-related image classifier in the first subtask. For the second task, we use both EficientNet and ResNet50 features. Then, we employ Faster R-CNN[7] to detect if there are people in the image. We also combine binary mask from Mask R-CNN [3] and pose from OpenPose [2] to predict whether the image contains at least one person standing in water above the knee. We also implement a language model for article's content and title contains the image. To evaluate our method, we use F1 score. Full details of the challenge tasks can be found in [1].
2.1
Firstly, the input image will be segmented to get the background. After that, we use EficientNet architecture to extract image features of both the original image and background image. We use the extracted features on multiple convolution layer and concatenate them together. By using the original image and the background image we have two extracted image features of the same size. Finally, we concatenate these two features together and feed into fullyconnected layers to estimate the final result.
For the second task, our proposed method contains four stages: water-related scene recognition, person detection, pose estimation and prediction based on paired mask and pose. Our system pipeline for this subtask shown in the Figure 1.
For the first stages, we label all the images of the training set into
two categories: water-related and non-water-related scene. Then,
we use the result of the average pooling layer from ResNet-50
trained on Places365[
For the second stage, we use Faster R-CNN to eliminate images that do not contain a person inside on water-related images. Both positive and negative images will be estimated pose by OpenPose to detect the swimming person in the next stage.
In the third stage, we use the OpenPose with COCO output
format [
For the last stage, we will make a prediction based on the paired mask and pose. Firstly, to detect all the swimming persons that contain in the image we extract poses with shoulders upwards only (including arms). In most swimming case, OpenPose gives very well result. After extract poses, we crop 50 x 100 pixels areas below the pose bounding box that calculated from the previous step then feed into WaterClassifier to predict whether a person is swimming or not. According to the observation, we realized that in some cases swimming persons that only have head and upward cannot be detected or misclassified by Faster R-CNN. Therefore, we also applied this to the negative result from the person detection stage to make sure we do not miss any swimming person.
We also use Mask R-CNN to get binary mask and bounding box (bbox) of each person in the images. Then, we conducted a pairing between pose and binary mask of each person in all the waterrelated images that have at least one person. We calculate the IoU score of bounding box mask and bbox pose of all pose and binary mask pairs included in the image. After that, we match pose and mask with IoU score from high to low with each pose having only one mask and vice versa. We also eliminate cases where the person is on a vehicle or boat removing the paired mask and pose from the image. After matching pose and mask of each person in each image, we conduct resolve special flooded cases: Knee keypoint outside of the person (Figure 2.c), Hip keypoint outside of the person (Figure 2.b), Keypoint fit with the person but the ratio of thighs is very small compared to the upper body (Figure 2.d), Knee keypoint close to the submerged part (Figure 2.e) by crop a rectangular area suitable rectangle for each case. All the rectangular areas cropped from the above cases will be extract features using ResNet-50 trained on Place365 as the input of the WaterClassifier to predict whether the person’s knee above the water or not. All remaining images not classified with water or that do not meet the above cases are classified in class 0.
For this subtask, we also implement language model base on
the article’s content and title. We employ both LSTM and CNN to
extract features of preprocessed text. Then, we use GloVe [
into these two modules, we summarize their output on the output layer and feed in the full connected layers to classify. 3 3.1
For the INTD subtask, we have submitted 3 runs, as below: - Run 1: Randomly split the train set with 9:1 ratio into train and val set. After training, we also run more some epochs on the entire training and validation set before predicting on the test set.
- Run 2: Same model as Run 1 with additional photos in the training set of MFLE subtask.
- Run 3: Same model as Run 2 but adjusted some threshold. For the MFLE subtask, we have submitted 5 runs, as below: - Run 1: Model described at Section 2.2. - Run 2: Text model described at the end of the Section 2.2. - Run 3: Combine the results Run 1 and 2 with class 1 only. - Run 4, Run 5: Same as Run 1 and Run 3 but adjusted some threshold of visual model. 3.2
In this paper, we employ a combination of ResNet-50 trained on Places365 dataset and EficientNet for features extractor, OpenPose for pose estimation and Mask R-CNN for segmentation to predict an image has at least one person standing in water above the knee. Our methods show potential results and achieve the highest rank at the MFLE subtask at the challenge. For the future works, we think we can improve both water-related image classifier and water classifier to increase accuracy.
Research is supported by Vingroup Innovation Foundation (VINIF) in project code VINIF.2019.DA19. We would like to thank AIOZ Pte Ltd for supporting our team with computing infrastructure.