The Flood-Related Multimedia Task at MediaEval 2020 Stelios Andreadis1, Ilias Gialampoukidis1, Anastasios Karakostas1, Stefanos Vrochidis1, Ioannis Kompatsiaris1, Roberto Fiorin2, Daniele Norbiato2, Michele Ferri2 1Information Technologies Institute - Centre of Research and Technology Hellas, Greece 2Eastern Alps River Basin District, Italy {andreadisst,heliasgj,akarakos,stefanos,ikom}@iti.gr {roberto.fiorin,daniele.norbiato,michele.ferri}@distrettoalpiorientali.it ABSTRACT This paper provides a description of the Flood-related Multimedia Task at MediaEval 2020. The primary goal of the task is to analyse and combine textual and visual content from social media data that reflect real-world events. The focus is on natural disasters and especially on flooding incidents, which are frequent around the globe and have large social consequences for communities and individuals. In particular, the task requires participants to identify Twitter posts that are relevant to flood events in a specific area of interest, based on their text and images. The automatic classification of posts as relevant or not relevant will essentially improve the quality of retrieved social media data, so that they can play a more valuable role in the emergency management. 1 INTRODUCTION Figure 1: The area of interest - Eastern Alps river basin dis- trict in North-East Italy The worldwide dominance of social media in the daily life of mod- ern people has led to a vast amount of available crowd-sourced information. The large streams of public social media data have cre- ated new directions in the research community and their analysis has the potential to affect positively several domains. One domain that can significantly benefit is natural disaster management where the exploitation of social media data is able to assist in every stage incidents. The work in [7] proves that the proximity of disaster- of a hazard event: a) they can notify about a possible disaster in the related georeferenced social media images to a disaster event can pre-emergency phase; b) they can provide insights on the evolution be an significant indicator of relevance. The authors of [4] apply of the incident and detect regions in danger during the disaster; machine learning techniques to classify informative versus non- and c) they can assist in the damage control in the post-emergency informative tweets posted during an earthquake, while [6] proposes phase. To this end, the Flood-related Multimedia Task at Media- an approach to decide whether a tweet is relevant to flooding by Eval 2020 focuses on improving situational awareness for flooding combining text classification of the posted message and visual clas- incidents in specified areas of interest. sification of the attached image. Moreover, [5] investigates the However, the acquisition of social media data about a particular usefulness of annotated social media data from a prior disaster natural disaster, such as floods in our scope, raises a number of together with unlabeled data from a current disaster so as to learn challenges. First, keyword-based search can result to social media domain adaptation classifiers. posts that contain keywords related to floods, but their content is Following the Multimedia Satellite Task that was introduced in in fact irrelevant; for example, when a word is used metaphorically. 2017 [3] and continued in years 2018 [1] and 2019 [2], the Flood- Another challenge is that posts often consist of both textual and related Multimedia Task remains focused on the high-impact natu- visual information (i.e. message and attached image) that may have ral disaster of floods, but this year solely through the perspective of different relevance to the disaster. Finally, a third challenge is to social media data. The overall goal of the task is to tackle the afore- filter out posts that refer to past events or incidents occurring mentioned challenges and combine text and images from social outside the area of interest. media streams in order to identify posts about floods in a predefined Towards addressing these challenges, several works aim at es- area. Moreover, the task poses an additional challenge by involving timating the relevance of social media content to natural disaster Italian social media posts in order to encourage researchers to move away from a focus on English. Better ability to separate relevant Copyright 2020 for this paper by its authors. Use permitted under Creative Commons and not relevant tweets will contribute to improving the quality of License Attribution 4.0 International (CC BY 4.0). the incoming information available to support first responders and MediaEval’20, 14-15 December 2020, Online civil protection authorities. MediaEval’20, 14-15 December 2020, Online S. Andreadis et al. Table 1: Keywords used in the acquisition of tweets Italian keywords Translation alluvione flood alluvione vicenza flood Vicenza allagamento flooding bacchiglione Bacchiglione (river) fiume piena full river allerta meteo weather alert sottopasso allagato underpass flooded allerta meteo vicenza weather alert Vicenza esondazione flood livello fiume river level keywords inside the tweet text. The complete list of keywords can be seen in Table 1. All tweets contain an attached image and had to be still online at the time of releasing the dataset. In order to be compliant with the Twitter Developer Policy, only the IDs of the tweets are distributed to the participants. The ground truth data of the dataset consists of one class label for the relevancy of each Tweet ID and has been collected with human annotation. Each tweet has been annotated by a single Figure 2: An example of a tweet that is relevant to floods in person and the annotators work for the Eastern Alps River Basin NE Italy District, who are experts on flood risk management in the Eastern Alps district of NE Italy. Apart from the fact that tweets were in their native language, their expertise allowed them to annotate as 2 TASK DESCRIPTION relevant tweets that referred to specific events, known a priori to The Flood-related Multimedia Task tackles the analysis of social them as real, in their district and to other phenomena that may be multimedia from Twitter for flooding events. In this task, the par- indirectly related to flooding. ticipants receive a set of Twitter posts (tweets) and their associated Ground truth is provided to participants only for the development- images, which contain keywords related to floods in a specific area set as key-value pairs of Tweet ID and ground truth label for the rel- of interest, specifically, the Eastern Alps district in Northeastern evancy (0=not relevant/ 1=relevant), where out of the 5,419 tweets (NE) Italy (Fig. 1). However, the relevance of the tweets to actual 1,140 (21%) are relevant and 4,279 (79%) are not relevant. On the flooding incidents in that area is ambiguous. other hand, ground truth for the test-set is not provided to the The objective of the task is to build an information retrieval sys- participants, since it is used in the evaluation. tem or a classifier that is able to distinguish whether or not a tweet is relevant to a flooding event in the examined area. An example 4 EVALUATION of a relevant tweet can be seen in Fig. 2. The dataset of the task The official metric for evaluating the correctness of retrieved tweets consists of Italian-language tweets, motivated by the common flood for the two classes relevant (1) and not relevant (0) is the F1-Score events in the cities of Eastern Alps (e.g., Venice, Vicenza, Trieste, metric on the test set. F1-Score has been selected, because it is Padua, Pordenone) and surrounding areas. Participants can tackle defined as the harmonic mean between precision and recall. the task using text features, image features, and a combination of both, and are allowed to submit 5 runs: ACKNOWLEDGMENTS • Required run 1: automated, using a fusion of textual and This work has been supported by the EU’s Horizon 2020 research visual data and innovation programme under grant agreements H2020-776019 • Optional run 2: automated, using textual information only EOPEN and H2020-832876 aqua3S. • Optional run 3: automated, using visual information only • General run 4,5: everything automated allowed, including using data from external sources REFERENCES [1] Bischke Benjamin, Helber Patrick, Zhao Zhengyu, Borth Damian, and others. 2018. The Multimedia Satellite Task at MediaEval 2018: 3 DATASET DESCRIPTION Emergency response for flooding events. (2018). The dataset of the task consists of 5,419 Tweet IDs (development-set) [2] Benjamin Bischke, Patrick Helber, Simon Brugman, Erkan Basar, and 2,288 Tweet IDs (test-set) that have been collected from Twit- Zhengyu Zhao, Martha Larson, and Konstantin Pogorelov. The Multi- ter between 2017 and 2019, by searching for Italian flood-related media Satellite Task at MediaEval 2019: Estimation of Flood Severity. Flood-related Multimedia MediaEval’20, 14-15 December 2020, Online In Proc. of the MediaEval 2019 Workshop (Oct. 27-29, 2019). Sophia Antipolis, France. [3] Benjamin Bischke, Patrick Helber, Christian Schulze, Venkat Srini- vasan, Andreas Dengel, and Damian Borth. 2017. The Multimedia Satellite Task at MediaEval 2017.. In MediaEval. [4] Muhammad Imran, Carlos Castillo, Ji Lucas, Patrick Meier, and Sarah Vieweg. 2014. AIDR: Artificial intelligence for disaster response. In Proceedings of the 23rd International Conference on World Wide Web. 159–162. [5] Hongmin Li, Nicolais Guevara, Nic Herndon, Doina Caragea, Kishore Neppalli, Cornelia Caragea, Anna Cinzia Squicciarini, and Andrea H Tapia. 2015. Twitter Mining for Disaster Response: A Domain Adapta- tion Approach.. In ISCRAM. [6] Anastasia Moumtzidou, Stelios Andreadis, Ilias Gialampoukidis, Anas- tasios Karakostas, Stefanos Vrochidis, and Ioannis Kompatsiaris. 2018. Flood relevance estimation from visual and textual content in social media streams. In Companion Proceedings of the The Web Conference 2018. 1621–1627. [7] Robin Peters and João Porto de Albuquerque. 2015. Investigating images as indicators for relevant social media messages in disaster management.. In ISCRAM.