Social media, such as Twitter, has increasingly afected information dissemination and consumption, demonstrating its potential to alarm the upcoming natural disaster beforehand. This paper describes the design of a novel natural disaster event detection method that used hybrid fusion to utilize multimodal information in tweets, called HFFD (Hybrid Fusion based Flood Detection). The goal of this work is to discover flood-related event when related information spread at the early stage. The performance on the oficial dataset confirms the efectiveness of our model.
With the development of online social media techniques, social media allows people to seek
and share information more efectively and overcome the barriers of traditional communication
such as time lag or geographical constraints. Such characteristic of social media shows its
capacity of detecting natural disaster at early stage when the disaster related information starte
to spread on social media platform, such as Twitter [
This paper discusses the RCTP subtask of MediaEval2022’s DisasterMM task [
The proposed model adopts the hybrid fusion [
Early fusion is more of an early exploration of multimodal research. Early fusion refers to the feature-level fusion of the features of diferent modalities before the decision task, which can better capture the low-level interaction between diferent modal information. However, due to the existence of the modal gap, it is dificult to find a model that can perform transfer learning between more than two modalities, that is, early fusion cannot fully achieve cross-modal feature fusion.
In contrast, late fusion fuses at the final decision level, that is, uses features from diferent
modalities to perform decision-making tasks separately, and finally rationally combining
diferent results according to a cleverly designed mechanism, such as averaging [
Synthetically, the hybrid fusion approach combines the ability of early fusion to capture
feature-level interactions and the ability of late fusion to flexibly cope with complex modal
situations, respectively. Hybrid fusion has been successfully applied in multimodal event
detection (MED) tasks [
The overall flow chart of the proposed model is shown in Figure 1. The model comprehensively utilizes the body text, images, entities (#, urls), and time features in the tweet data to detect whether the tweet is related to the disaster topic.
The image feature extractor uses ResNet101 trained on ImageNet and fine-tuned on the task dataset. Each tweet sample contains varying numbers of images, and we input them into ResNet respectively to obtain corresponding image features F , where i = 1, 2, . . ., n.
The italian dataset is a novel idea, for which we have tried variants of BERT models such as
RoBERTa and multilingual BERT as textual feature extractors. In the end, multilingual BERT
outperformed others. Entities refer to hashtags (#) and urls contained in the tweet text. It is
common to mention related users or organizations in tweets, or use hashtags for topic labeling.
The text in the url attached to the tweet is related to the original text of the tweet, and both
can be regarded as the text content of the tweet [
Since flood-related tweets increase with the time of the rainy season each year, time feature is also important modal feature for detecting flood topics. In order to extract the periodicity of time feature in long periods, the year and date of each tweet’s creation time are extracted separately in the proposed model, and the time feature are encoded in the form of sine feature F and cosine feature F respectively. This, we can mine the periodic characteristics in time information.
Since both the hashtag feature F and the text feature F are the same source text information features extracted by BERT, they are directly concatenated to obtain the text-entities fusion feature F - = F ⊕ F1 ⊕ . . . ⊕ F. After that, we further concatenate the time feature into F - to obtain the text-entities-time fusion feature F -- = F - ⊕ F ⊕ F. Since the number of images contained in each sample is diferent, We OR the prediction results for each image to get the final image result R = R 1 OR R2 OR . . . OR R. Finally, we consider that one of the image and text-entities-time is related to the flood, so we can conclude that the entire sample is related to the flood, so we OR the final result of the image and the text-entities-time result again to get the final result R = R -- OR R .
We tested several diferent textual feature extractors on the Italian dataset, among which
RoBERTa [
As shown in Table 2, we conduct ablation experiments with diferent modality feature extractors. After introducing entities features, the model performs slightly better than relying only on uncleaned text or only cleaned body text. We also found that the model relying only on image feature performs poor. This is because most sample images do not contain obvious floodrelated elements, which makes the image feature extractor undertrained - in fact, the proposed multimodal model has the highest precision and recall on the development set exceeding 0.93, but the precision on the oficial final test set is down to 0.6741. This is because in the proposed model, there is a "path" that only passes through the image classifier, that is, when the image classifier detects a "flood-related" image, the whole model will directly output the final result. At this time, the performance of the whole model will be afected by the image classifier and become unstable. It’s to say, we still need to explore a better late fusion method.
Because the development set data of the RCTP subtask was collected in a short time span (May 25, 2020 to June 12, 2020), time features has no significant impact on the model performance. But the dataset of the LETT subtask has a long time span, in which we derive the periodicity of the number of flood-related tweet creations over time relative to the dates of the rainy season. In addition, some disasters caused by special weather are also related to specific hours. For example, some areas encounter squall line weather, and heavy precipitation will occur in the afternoon and evening. But we did not find hour-level temporal characteristics in the given datasets.
Thanks to the organizers of the MediaEval2022, especially to those organizers for DisasterMM. This work was supported in part by the Scientific Research Leader Studio of Jinan (Grant No. 2021GXRC081), and in part by the Joint Project for Smart Computing of Shandong Natural Science Foundation (Grant No. ZR2021LZH010, ZR2020LZH015, and ZR2022LZH012).