This paper presents the contribution of NUCES DSGP team for the Multimedia Satellite Task at MediaEval 2019. The essential tasks include News Image Topic Disambiguation (NITD) and Multimodal Flood level Estimation (MFLE) from news images. An ensemble based deep learning method has been applied to the News Image Topic Disambiguation task, where data augmentation and transfer learning were used for binary classification of images. During training, the challenge of class imbalance is managed by using data augmentation technique and selection of equal sample size from each class. For Multimodal Flood Level Estimation task, person's lower body keypoints were detected along with image flood probability scores from two deep convolutional network architectures, namely ResNet50 and VGG19. The confidence scores of detected keypoints and the convolutional networks' output probabilities were combined and were passed to a Random Forest classifier for a ifnal prediction score. The evaluation of the proposed methods for the test set of NITD task revealed a 0.895 F1-Micro score (3rd best score), while the evaluation of MFLE task provided 0.734 F1-Macro score (2nd best score).
Efective and eficient flood response system requires timely
information about the event. The risk of damage could be reduced
by appropriate actions on the basis of inferred information. The
data collected through diferent mediums including news articles
is easily available and could be used for disaster response system.
The NITD task of "Multimedia Satellite Task at MediaEval, 2019"
[
In the MFLE task, we had to determine whether at least one
person in the image is standing in water, and whether or not the level
of water is above the knee. To solve this problem, we considered
both global and local perspective of an image. Our approach was
to first detect and localize all person(s) present in an image. Then
a dataset was prepared which contains persons’ bounding boxes
from all of the images. The ResNet50 Convolutional Neural
Network was fine-tuned to extract flood probability of the bounding
box. The label of the image was attached to all bounding box(es) of
person(s) present in the image. Then we stacked ResNet50
probability with confidence score of four lower body keypoints of interest:
namely Right Knee, Left Knee, Right Ankle, Left Ankle of a person
and a score that indicated belief of calculated confidence scores.
We also added an image level flood evidence probability obtained
from VGG19 pre-trained on the Places365 dataset and fine-tuned
on Multimedia Satellite 2017 flood images. This flood probability is
same for all the detected person(s) in an image. For inference, we
trained a random forest on keypoints scores and flood probability
scores obtained for each person bounding box. If the person with
the maximum probability exceeds 0.50, the image is classified as
positive instance. Our inspiration for this work comes from eforts
on text and image based data from diferent sources to identify type
and intensity of disasters [
Two diferent deep learning based approaches were used for both
tasks. The details of approaches are given below:
News Image Topic Disambiguation (NITD): An ensemble based
deep learning approach has been adopted for the task of
Imagebased News Topic Disambiguation, MediaEval, 2019. The dataset
for the task contains 5181 images for training and 1296 images
for testing purpose. There were 564 images related to first class
and 4617 images for second class. An ensemble based method has
been implemented for the specified binary image classification task.
Initially, class imbalanced problem has been solved by using data
augmentation technique. The Augmentor [
Mu. ltimodal Flood Level Estimation from news (MFLE): For
the prediction of at least one person standing in water level above
the knee in an image, we hypothesized that "a detected person
in the flooded region with low knees and ankles visibility is a
positive case". For this purpose, we prepared a separate dataset
which consists of all the detected persons’ bounding boxes from
their respective images. The label attached to a person bounding
box was same as that of the parent image. For person’s bounding
box detection, the Faster-RCNN model [
Afterwards, for the persons bounding boxes dataset, we obtained
the visibility confidence scores for four lower body keypoints i.e
Right Knee, Left Knee, Right Ankle, Left Ankle of a person and an
overall score indicating belief in the keypoints calculation. These
probabilities help in estimation of the visibility part of our
hypothesis. We utilized Fang et. al [
Then a Random Forest [
For all runs of the NITD task, VGG16 ensemble was used. Run 1 and
2 use class weights to handle the class imbalance, Run 3, 4 and 5
use data augmentation. Run 5 uses EasyEnsemble [
Table 1 shows the performance of various runs on test set. For NITD, the ensemble of five diferent VGG16 (Hybrid) models has produced a micro-averaged F1 score of 0.895. For MFLE, the proposed method has produced F1 (macro-averaged) score of 0.734. We got an F1-Score (+ve class) of 0.694 on the MFLE validation set. Various approaches for NITD task have been applied. An approach without data augmentation has been implemented by balancing the minority and majority classes through weights. Also, features with the length of 1365 have been extracted for each image and EasyEnsemble classifier is applied. However, the best performing approach utilized data augmentation technique to increase the quantity of minority class. We did an ablation study of the MFLE inference model. First, when VGG19 global level flood estimation component was removed, the performance deteriorated. Same hold true for the ResNet local level flood estimation and for the Ankle keypoints. For the Random Forest probabilities, we computed diferent statistics such as mean and median, but maximum of probabilities resulted in better performance on the validation set.