Flood detection using Social Media Data and Spectral Regression based Kernel Discriminant Analysis

Abstract

Natural disasters destroy valuable resources and are necessary to recognize so that appropriate strategies may be designed. In recent past, social networks are very good source to gather event specific information. This working notes paper is based on the task of Disaster Image Retrieval from Social Media dataset (DRISM), as a part of MediaEval, 2017. The Dataset of images and their relevant metadata is taken from various social networks including Twitter and Flicker. An ensemble approach is adopted in this paper where diferent visual and metadata features are integrated. Kernel Discriminant analysis using spectral regression is then used as dimensionality reduction technique. Mean Average Precision (MAP) at various cutofs are reported in this paper.

INTRODUCTION

The solution is submitted in Multimedia Satellite Task at MediaEval, 2017, on the task “Disaster image retrieval from social media" [ 1 ]. The goal of the challenge is to retrieve images, having direct evidence of flood in descending order of probability value for the task.

APPROACH

Information is provided in two forms, first includes visual features, while other includes textual metadata of each image. For processing of visual data, features are ensembled to perform classification at the later stage. Figure 1 shows our proposed approach. In this approach, all visual features provided by organizer [ 1 ] including AutoColor Correlation, Edge Histogram, Tamura etc are integrated. Kernel Discriminant Analysis using Spectral Regression [ 2 ] is then used during classification. Spectral methods have now established as a great technique for both manifold learning and dimensionality reduction and [ 2 ]. Spectral Regression in combination with Kernel Discriminant Analysis (SRKDA) has proved successful in many classification tasks such as multilabel classification, action recognition [ 3, 4 ]. Large matrix decomposition becomes less complex in this method due to spectral graph analysis. We have also investigated various machine learning techniques such as Random Forest, Support Vector Machine. On validation data and using 10 Fold Cross Validation, the best results were obtained using SRKDA and thus adopted in this task.

In metadata, most valuable features of ’usertags’ are selected for prediction. To enhance efectiveness, stop words are removed from selected features. Afterwards, for each image, Term Frequency Inverse Document Frequency (TFIDF) of every user tag is calculated and resultant matrix of TFIDF vectors are saved [ 5, 6 ]. The matrix is analyzed with SRKDA to predict the confidence of respective class of each image, as shown in Figure 2.

Moreover, a combination of metadata and all visual features are ensembled to produce and processed through SRKDA, as shown in Figure 3. 3

RESULTS AND ANALYSIS

Flood prediction is performed by using datasets of visual features, metadata and combination of both visual features and metadata of each image using SRKDA algorithm. 3.1

Parameter Tunings using Training Data

The most important parameter in our approach is the selection of kernel. Leave one out cross validation is used to select the best kernel in KDA including cityblock, euclidean and chi-squared kernel. Results of visual features are collected by tuning diferent parameters of SRKDA and the best results were obtained by using distance as city block and value of Gamma as 0.05 in KDA. Average precision is calculated at diferent values of k = 50, 100, 200, 300 and 400 and 500 as shown in Figure 4 and Table 1. Average precision were quite high as we able to obtained around 0.94 using fusion of visual and metadata. The proposed system is then evaluated on test data and predicted values of all test images are submitted to organizers for independent evaluation. Using visual features, we able to obtain around average precision of 0.649 at 480 cutof point while for metadata, we able to obtain average precision of around average precision of 0.65. The fusion of visual and meta give us around 0.646 average precision which is surprising as we were expecting better results using fusion. It is part of future work to explore the reasons behind poor performance of fusion. Table 2 shows mean average precisions at diferent cutofs. The best results were obtained using visual features which are around 80.98%. Figure 5 shows top 4 images correctly retrieved by the system. Future work aims to identify images that are not correctly retrieved by the system and investigate deep learning approaches to improve the overall system.

The experiments using SRKDA, indicates that fusion of visual and textual features produce better results on validation set. But on test data, the performance drops using fusion of visual and textual features. Particularly for metadata, prediction can be improved by using all provided metadata information including title and description. 4

CONCLUSION

In this paper, we have presented our runs on Disaster Image Retrieval from Social Media task (DRISM). SRKDA technique is investigated to train and test the model using ensemble of 6 diferent features. Our proposed system able to obtain around 0.81 mean average precisions at diferent cutofs (50, 100, 250, 480).

References

[1] Bischke , Benjamin and Helber, Patrick and Schulze, Christian and Venkat, Srinivasan and Dengel, Andreas and Borth, Damian, 2017 . “The Multimedia Satellite Task at MediaEval 2017: Emergence Response for Flooding Events" , Proc. of the MediaEval 2017 Workshop , Dublin, Ireland

[2] Cai Deng , He Xiaofei , and Han Jiawei, 2011 . “ Speed up kernel discriminant analysis" , The VLDB Journal , 20 ( 1 ), 21 - 33 .

[3] Tahir Muhammad Atif , Fei Yan, Peter Koniusz , Muhammad Awais, Mark Barnard, Krystian Mikolajczyk, Ahmed Bouridane, and Josef Kittler, 2013 . A robust and scalable visual category and action recognition system using kernel discriminant analysis with spectral regression . IEEE Transactions on Multimedia , 15 ( 7 ).

[4] Tahir Muhammad Atif , Josef Kittler and Ahmed Bouridane , 2016 . ” Multilabel classification using stacked spectral kernel discriminant analysis . " Neurocomputing 171 , 127 - 137 .

[5] Bramer , Max, 2013 . ” Principles of Data Mining." Springer-Verlag London.

[6] John, Vineet, and Olga Vechtomova, 2017 . “ Sentiment Analysis on Financial News Headlines using Training Dataset Augmentation" , Proceedings of the 11th International Workshop on Semantic Evaluation.