Disastrous situations can be better managed by availability of timely and relevant information. Social media plays very important role in providing information of disastrous events. The working paper is based on "Multimedia satellite task: Emergency response for flooding events", as a part of MediaEval, 2018. The dataset was provided with tweets and their respective images, which may or may not acquire evidence of roads and its status of pass-ability. An ensemble approach is followed in this paper by combining local features and global features of images. Text contents of the tweets were processed by their TF-IDF scores. Moreover, two level classification is performed by applying Spectral Regression based Kernel Discriminant Analysis (SRKDA) on individual feature categories as well as ensemble of diferent feature types. It is observed that F 1 score produced by visual, text and ensemble of both text and visual features for evidence of road remain 74.58%, 58.30% and 76.61% respectively. The average F1 score for evidence of road and its status of passability remain 45.04%, 31.15% and 45.56% for visual, text and ensemble of both visual and text features.
Disastrous situations require efective and timely information. Data could be collected with the help of sensors, but it is expensive to configure sensors on every critical location. Social media could be the best way to collect information in various forms including text, images and videos. Social media data can be utilized in managing disastrous situations including earthquake, fire and flooding situations. Data collected from social media could assist disaster response organizations, so that timely action may be taken.
Response in emergency situations, has recently been helped by
enormous amount of data collected through social media. Various
studies have been performed to collect data from social media and
its significance has been proved in spreading timely awareness
in disastrous situations. There are various situations where social
media data can produce fruitful results, including fire, earthquake
and flooding events. Tweets can produce helpful results for fire
detection, when analysed with Deep Neural Network and Support
Vector Machine (SVM) [
Generally, tweets can contain two modalities such as (visual and
textual), however visual information is not always present. Both
categories of information are processed in two-level hierarchy of
classification. At first level, evidence of the pathway is examined,
while the status that road is actually passable or not is determined
in second level of hierarchy, as shown in Figure 1. Various
classifiers were implemented on training dataset, including Support
Vector Machine (SVM), Multinomial Naive-Bayes, and Random
Forest. However, best classification results were produced by
Spectral Regression in combination with Kernel Discriminant Analysis
(SRKDA) [
Text part of social media dataset was cleaned by eliminating
hyperlinks, punctuations and symbols. Stop words were also removed
from each of the tweet. User tags for each tweet were also extracted
and combined with text of tweet. Then, TF-IDF [
Images are processed by using conventional visual descriptors
provided by organizers of MediaEval, 2018 [
Deep Local Features (DELF): DELF is CNN-based local feature descriptor, specifically used for large-scale image recognition. It provides semantically local features, which are helpful in image retrieval. Furthermore, keypoint selection is also performed by proposed attention mechanism. The pre-trained model based on Convolutional Neural Network is also released, which is optimized for landmark recognition. Delf desriptor is extracted for each image. Delf descriptor is further processed with Gaussian Mixture Model (GMM) and fisher-vector having length of 2560 is generated for each image. 3.3
The ensemble is created by DELF and conventional visual descriptors, along with TFIDF of textual data. The prediction on ensemble of social media data is performed by the use of SRKDA followed by Nearest Neighbour classifier for prediction. 4
Results are extracted for three diferent runs, which includes text, visual and ensemble of both text and visual features. First we will discuss results on validation set which is obtained from training data followed by discussion on test data. Table 1 shows the results on validation data which is obtained by taking 10% of training data. It is observed that DELF features did contribute in improving the accuracy using visual information. For the classification of road passability, around 80% F1 score is obtained. However, for another scenario which is whether road is good enough to be used, score of around 0.60 is obtained.
Moreover, results were extracted by using visual, text and ensemble of both of them by using test dataset. The F1 scores for road evidence classification has produced 74.58%, 58.30% and 76.61% for visual, text and ensemble of both. While, the average F1 scores for road evidence and status of passability has produced 45.04%, 31.15% and 45.56% scores for visual, text and ensemble of both text and visual features, as shown in Table 2.
It is observed that better outcome has produced by using visual features along with SRKDA. It is part of future work to improve the results by deep analysis of local and global features.
It has been observed that results produced for evidence of road are much better than its status of passable or not-passable. It is observed that less training data is available for classifier to check the status of passable roads. In future, data augmentation could be used to increase the quantity of visual training data. Moreover, TFIDF based sparse representation of text could be improved by the use of chi-square feature selection with probabilistic weighting scheme, as Chi-Square features are more discriminating about the model. 6
The paper presented results produced for the "Multimedia satellite task: Emergency response for flooding events". Diferent methods of feature extraction and classifiers are used to produce results. It is observed that an ensemble of conventional visual descriptors and DELF features with SRKDA classifier, has provided best outcome. Multimedia Satellite Task: Emergency Response for Flooding Events