Social media, which provides instant textual and visual information exchange, plays a more important role in emergency response than ever before. Many researchers nowadays are focusing on disaster monitoring using crowd sourcing. Interpretation and retrieval of such information significantly influences the eficiency of these applications. This paper presents a method proposed by team EVUSikg for the MediaEval 2018 challenge on Multimedia Satellite Task. We only focused on the subtask “flood classification for social multimedia”. A supervised learning method with an ensemble of 10 Convolutional Neural Networks (CNN) was applied to classify the tweets in the benchmark.
Crowdsourcing is a rapidly developing method for acquiring information from many users in real time. Many applications nowadays are focusing on monitoring natural disaster events such as earthquakes, fires and flooding. The retrieved information can improve the situation awareness for citizens. At the same time, it helps the rescuers to provide a better emergency response. Flooding is one of the topics which attracts lots of attention. With the development of information retrieval and deep learning techniques, instead of using pre-defined keywords for extracting flood relevant information, deep learning models can achieve much better performance for visual and textual information understanding.
In our previous work [
More information regarding the floods, such as severity, is still
desired. Multimedia Satellite Task at MediaEval 2018 [
In this section, our approach is introduced. All the models are trained using the Tensorflow and Keras frameworks. We randomly selected 10% from the given dataset (582 tweets) as an independent internal test set. Moreover, 60 tweets from each label were randomly selected and used as a validation set. All the remaining tweets were used to train the models. The network architectures and parameters were tuned and compared internally based on the performance of the models on the internal test set. The validation set was used for early stopping during training. Data augmentation, such as rotation, shift, and zoom, was also performed during the training process.
Run 1 allows only visual information to be used for the
classification task. Pre-trained models DenseNet201 [
Input Image
DenseNet201
InceptionV3 InceptionResNetV2 + + 1920 x 1 Concat 2048 x 1 1536 x 1
FC
FC
Softmax
3 x 1
Run 2 allows only metadata information to be used for the
classiifcation task. In this case, only user provided tweet texts are used as
inputs. We applied a classic CNN model [
Run 4 is a general run, where we only used the visual information. Same architecture as Run 1 was used. Since the flood related tweets have only a very small proportion among the daily usersent tweets, we wanted to investigate whether introducing a large amount of negative labeled images can improve the robustness of the classifiers for the current task. Therefore, the 3,349 negative labeled flickr photos from the Multimedia Satellite Task at MediaEval 2017 were used as an extra data source to improve the robustness of the model against the tweets with no evidence.
Run 5 is also a general run, where we only used the visual information. Same architecture as Run 1 was used. We observed that many of the positive labeled images from the Multimedia Satellite Task at MediaEval 2017 were describing severe flood situations. Therefore, in this run we assigned the label (2) with evidence “not passable” to these 1,916 positive labeled photos. Together with the 3,349 negative labeled photos, we trained an image classifier. In this way, we introduced imprecise labeled training examples. Our intention is to investigate how much the performance of classifiers is afected through introducing more data but with imprecise labels.
Since the random initialization of the weights in neural networks often leads to unstable performance of the models, during Run 1, 2 and 3, each model was trained 10 times on the same training set. In this case, we regard these 10 models as weak classifiers. We ensemble the predictions and take the majority voting of all the 10 predictions as the final prediction. In this case, we hope the ensemble learning improves the robustness of the classifiers. For Run 4 and 5, the models were trained only once due to the much longer training time. 3
As for the subtask “flood classification for social multimedia”, we ifrstly tested the performance of the classifiers based on our internal test set. The results are shown in Table 1. The evaluation on the private test set, which was provided by the organizer, is shown in Table 2. The metrics used for evaluation are the averaged F1-scores for the classes (1) with evidence passable and (2) with evidence not
From the results above, the classifiers have generally similar performance on both internal and private test set. The visual based classifier (Run 1) can achieve the best performance, compared to all the other runs. The models trained only on metadata information do not achieve a good performance on both test sets. The fusion of both models did not make any significant improvements according to the evaluation. From the results of Run 4 and 5, we noticed that introducing more negative examples or more examples with imprecise labels, lead to a significantly worse performance. Therefore, we conclude that the balance of the training examples plays an important role for the classifier performance.
Since the textual descriptions from users rarely address the severity of the flood situation, it is very hard to achieve a reasonable classification performance only based on textual information. Textual information may contain informative words or phrases regarding flood evidence. However, distinguishing whether the road is passable or not from a single tweet text, which is less than 280 characters, is a challenging task. In our case, we concluded that introducing textual information did not help for the current task. 4
In this paper, an ensemble of CNN models was trained for retrieving lfood relevant tweets. Our best model is the one trained only on visual information. Using only metadata, the classifiers were not able to produce high quality predictions. Using photographs is reasonable, since nowadays people are likely to share photographs to address their current situation, rather than detailed textual descriptions. The analysis of video sequences would be a promising extension for extracting more information regarding flooding events, such as rainfall intensity, flow speed or even water depth.
The authors would like to acknowledge the support from the BMBF funded research project "EVUS - Real-Time Prediction of Pluvial Floods and Induced Water Contamination in Urban Areas" (BMBF, 03G0846A). We also gratefully acknowledge the support of NVIDIA Corporation with the donation of a GeForce Titan X GPU used for this research.