WaterMM: Water Quality in Social Multimedia Task at MediaEval 2021

Abstract

This paper describes the “WaterMM: Water Quality in Social Multimedia” Task at MediaEval 2021. The overall aim of the task is to analyse the textual content of social media data that express real-world issues. The focus is specifically on water quality, safety and security, which is a fundamental part of life sustainability. Participants of this task are required to classify the social media posts of a bilingual dataset as relevant or not relevant to water-related problems, while they can optionally combine textual features with visual. The automatic prediction of posts could enhance the quality of crowd-sourced information, consequently supporting situational awareness in the water sector.

INTRODUCTION

With the rise of social media in the everyday life of people around the world, a very broad range of topics is now discussed online. The widespread availability of public social media posts has paved the way for developing Artificial Intelligence solutions that exploit crowd-sourced information. The scientific community has particularly focused on emergency, disaster and crisis management [ 1, 7 ] where the use of social media data can be really beneficial to detecting threats, monitoring situations, and enhancing response. Despite the fact that research aims attention mostly at sudden crisis, i.e. natural or human-caused disasters that occur without warning, another highly interesting domain is the creeping crisis, i.e. a threat to life-sustaining systems that evolves over time and space and is foreshadowed by precursor events [ 6 ]. Such a type of crisis could threaten, for example, water quality, safety and security.

Among the various topics discussed on Twitter, it is anticipated that users will also post tweets that refer to water quality. The acquisition of posts containing citizen complaints on the condition of drinking water (as an addition to traditional means, e.g. phone calls) or news coverage about water-related issues could support situational awareness in a water distribution network.

However, within the post stream it is expected that a number of posts containing water-quality-related keywords does not refer to actual cases of polluted water. To minimize the incoming noise, automatic prediction of a post’s relevance is required. Filtering out irrelevant posts will improve the quality of the information that interested organisations, such as water utilities or water protection agencies, receive from social media. Estimating the relevance of a tweet faces two further challenges. First, the textual information of a tweet (i.e. Twitter message) may have a diferent relevance to the examined topic in comparison to its visual information (i.e. Twitter image). Secondly, the text of the tweets may be in multiple languages, which requires independent processing and training.

The potential contribution of relevance prediction to situational awareness in the water sector has motivated the organisation of the “WaterMM: Water Quality in Social Multimedia” Task1 at MediaEval 2021. As a continuation of the Multimedia Satellite Task (2017-2019) [ 3–5 ] and the Flood-related Multimedia Task [ 2 ], the WaterMM Task focuses exclusively on social media data and shifts the domain of application from floods to water quality, safety and security. The overall goal of the task is to tackle the aforementioned challenges and use textual information (as well as visual information and metadata) from a bilingual dataset of Twitter posts in order to identify tweets that refer to concerns about water. 2

TASK DESCRIPTION

The WaterMM Task deals with the analysis of social media posts from Twitter with regards to issues of water quality, safety and security. The participants of this task are provided with a set of Twitter post IDs in order to download the text, the attached image (if it exists) and the metadata of tweets that have been selected with keyword-based search that involved words/phrases about the quality of drinking water (e.g. strange color, smell or taste, related illnesses, etc.). Nevertheless, the occurrence of such phrases in a tweet might not necessarily reflect a case of water contamination.

The objective of this task is to build a binary classification system that will be able to distinguish whether a post is relevant or not to water-quality issues. An example of a relevant tweet is shown in Fig. 1, while an irrelevant tweet in Fig. 2. Participants can tackle the task using text features, image features, metadata, or a combination of the above, and they are allowed to submit up to 5 runs: • • • •

Required run 1: automated using textual information only Optional run 2: automated using fused textual and visual information Optional run 3: automated using fused textual and visual information as well as other metadata General runs 4 & 5: everything automated allowed, including using data from external sources 3

DATASET DESCRIPTION

The dataset of the task is a set of social media posts collected from Twitter during one year, i.e. from May 2020 to April 2021, by searching for English and Italian keywords inside the tweet text about water quality (e.g. issues with drinking water, signs of water pollution, illnesses related to water, etc.). The keywords have been proposed by the Eastern Alps River Basin District, who are responsible for hydrogeological defense, which involves the protection of water resources and aquatic environments, in the Eastern Alps partition of North-East Italy. For reasons of brevity, we present here only the most frequently matched keywords for both languages in Table 1, while the complete list is provided to participants along with the dataset in the task’s repository2. The bilingual dataset is separated into two sets: the development-set that contains 8,000 posts and the test-set with 2,000 posts. In order to be fully compliant with the Twitter Developer Policy, only the IDs of the tweets are distributed to the participants. Thus, it was ensured at the time of releasing the dataset that all tweets were still online.

The ground truth of the dataset reflects the relevance of a tweet (relevant / not relevant) and has been manually collected with human annotation. The annotation has been realized again by the Eastern Alps River Basin District. Apart from their valuable expertise on the domain, they were also able to annotate tweets in their native language, i.e. Italian. It should be noted that each tweet has been annotated by a single person and not by multiple annotators.

Initially, solely the ground truth for the development-set is released, since the ground truth for the test-set is used in the evaluation stage and will be available only after the completion of MediaEval 2021. Participants are provided with key-value pairs of Tweet ID and ground truth label for the relevancy (0=not relevant/ 1=relevant). In particular, 1,374 tweets (17.18%) of the developmentset are relevant and 6,626 (82.82%) are not relevant to water quality, showing that it is a quite imbalanced training dataset and participants should consider this issue. 4

EVALUATION

F1-Score is selected as the oficial metric for evaluating the binary classification of tweets as relevant (1) and not relevant (0) on the test set, since this measure is the harmonic mean between precision and recall, taking both metrics into account. Participants are also encouraged to carry out a failure analysis of their results in order to gain insight in the mistakes that their classifiers make.

ACKNOWLEDGMENTS

This work has been supported by the EU’s Horizon 2020 research and innovation programme under grant agreements H2020-832876 aqua3S, H2020-883484 PathoCERT, and H2020-101004157 WQeMS. 2https://github.com/multimediaeval/2021-WaterMM/blob/main/dataset/keywords. json

References

[1] David E Alexander . 2014 . Social media in disaster risk reduction and crisis management . Science and engineering ethics 20 , 3 ( 2014 ), 717 - 733 .

[2] Stelios Andreadis , Ilias Gialampoukidis, Anastasios Karakostas, Stefanos Vrochidis, Ioannis Kompatsiaris, Roberto Fiorin, Daniele Norbiato, and Michele Ferri . 2020 . The flood-related multimedia task at mediaeval 2020 . In Proceedings of the MediaEval 2020 Workshop , Online. 14 - 15 .

[3] Bischke Benjamin , Helber Patrick , Zhao Zhengyu , Borth Damian , and others. 2018. The Multimedia Satellite Task at MediaEval 2018 : Emergency response for flooding events . ( 2018 ).

[4] Benjamin Bischke , Patrick Helber, Simon Brugman, Erkan Basar, Zhengyu Zhao , Martha Larson , and Konstantin Pogorelov . The Multimedia Satellite Task at MediaEval 2019 : Estimation of Flood Severity . In Proc. of the MediaEval 2019 Workshop (Oct. 27 - 29 , 2019 ). Sophia Antipolis, France.

[5] Benjamin Bischke , Patrick Helber, Christian Schulze, Venkat Srinivasan, Andreas Dengel, and Damian Borth . 2017 . The Multimedia Satellite Task at MediaEval 2017 .. In MediaEval.

[6] Arjen Boin , Magnus Ekengren, and Mark Rhinard . 2020 . Hiding in plain sight: Conceptualizing the creeping crisis . Risk, Hazards & Crisis in Public Policy 11 , 2 ( 2020 ), 116 - 138 .

[7] Yan Jin , Brooke Fisher Liu, and Lucinda L Austin. 2014 . Examining the role of social media in efective crisis management: The efects of crisis origin, information form, and source on publics' crisis responses . Communication research 41 , 1 ( 2014 ), 74 - 94 .