=Paper=
{{Paper
|id=Vol-3181/paper4
|storemode=property
|title=WaterMM: Water Quality in Social Multimedia Task at MediaEval 2021
|pdfUrl=https://ceur-ws.org/Vol-3181/paper4.pdf
|volume=Vol-3181
|authors=Stelios Andreadis,Ilias Gialampoukidis,Aristeidis Bozas,Anastasia Moumtzidou,Roberto Fiorin,Francesca Lombardo,Anastasios Karakostas,Daniele Norbiato,Stefanos Vrochidis,Michele Ferri,Ioannis
Kompatsiaris
|dblpUrl=https://dblp.org/rec/conf/mediaeval/AndreadisGBMFLK21
}}
==WaterMM: Water Quality in Social Multimedia Task at MediaEval 2021==
https://ceur-ws.org/Vol-3181/paper4.pdf
WaterMM: Water Quality in Social Multimedia Task at
MediaEval 2021
Stelios Andreadis1 , Ilias Gialampoukidis1 , Aristeidis Bozas1 , Anastasia Moumtzidou1 ,
Roberto Fiorin2 , Francesca Lombardo2 , Anastasios Karakostas1
Daniele Norbiato2 , Stefanos Vrochidis1 , Michele Ferri2 , Ioannis Kompatsiaris1
1 Information Technologies Institute - Centre of Research and Technology Hellas, Greece
2 Eastern Alps River Basin District, Italy
{andreadisst,heliasgj,arbozas,moumtzid,akarakos,stefanos,ikom}@iti.gr
{roberto.fiorin,francesca.lombardo,daniele.norbiato,michele.ferri}@distrettoalpiorientali.it
ABSTRACT
This paper describes the “WaterMM: Water Quality in Social Mul-
timedia” Task at MediaEval 2021. The overall aim of the task is
to analyse the textual content of social media data that express
real-world issues. The focus is specifically on water quality, safety
and security, which is a fundamental part of life sustainability. Par-
ticipants of this task are required to classify the social media posts
of a bilingual dataset as relevant or not relevant to water-related
problems, while they can optionally combine textual features with
visual. The automatic prediction of posts could enhance the quality
of crowd-sourced information, consequently supporting situational
awareness in the water sector.
1 INTRODUCTION
With the rise of social media in the everyday life of people around
the world, a very broad range of topics is now discussed online.
The widespread availability of public social media posts has paved
the way for developing Artificial Intelligence solutions that ex-
ploit crowd-sourced information. The scientific community has
particularly focused on emergency, disaster and crisis management
[1, 7] where the use of social media data can be really beneficial to
detecting threats, monitoring situations, and enhancing response.
Despite the fact that research aims attention mostly at sudden crisis,
i.e. natural or human-caused disasters that occur without warning,
another highly interesting domain is the creeping crisis, i.e. a threat
to life-sustaining systems that evolves over time and space and is
foreshadowed by precursor events [6]. Such a type of crisis could
threaten, for example, water quality, safety and security. Figure 1: A tweet that is considered relevant to water quality
Among the various topics discussed on Twitter, it is anticipated
that users will also post tweets that refer to water quality. The
interested organisations, such as water utilities or water protection
acquisition of posts containing citizen complaints on the condition
agencies, receive from social media. Estimating the relevance of a
of drinking water (as an addition to traditional means, e.g. phone
tweet faces two further challenges. First, the textual information
calls) or news coverage about water-related issues could support
of a tweet (i.e. Twitter message) may have a different relevance
situational awareness in a water distribution network.
to the examined topic in comparison to its visual information (i.e.
However, within the post stream it is expected that a number
Twitter image). Secondly, the text of the tweets may be in multiple
of posts containing water-quality-related keywords does not refer
languages, which requires independent processing and training.
to actual cases of polluted water. To minimize the incoming noise,
The potential contribution of relevance prediction to situational
automatic prediction of a post’s relevance is required. Filtering out
awareness in the water sector has motivated the organisation of the
irrelevant posts will improve the quality of the information that
“WaterMM: Water Quality in Social Multimedia” Task1 at MediaEval
Copyright 2021 for this paper by its authors. Use permitted under Creative Commons 2021. As a continuation of the Multimedia Satellite Task (2017-2019)
License Attribution 4.0 International (CC BY 4.0). [3–5] and the Flood-related Multimedia Task [2], the WaterMM
MediaEval’21, December 13-15 2021, Online
1 https://multimediaeval.github.io/editions/2021/tasks/watermm/
MediaEval’21, December 13-15 2021, Online S. Andreadis et al.
Table 1: Most frequently matched keywords
English Italian
muddy water bottiglia acqua (water bottle)
water not clear acqua rubinetto (tap water)
water bad taste acqua potabile (drinking water)
water pollution sabbia acqua (sand water)
water bad smell colore acqua (water color)
by searching for English and Italian keywords inside the tweet
text about water quality (e.g. issues with drinking water, signs
of water pollution, illnesses related to water, etc.). The keywords
have been proposed by the Eastern Alps River Basin District, who
are responsible for hydrogeological defense, which involves the
protection of water resources and aquatic environments, in the
Eastern Alps partition of North-East Italy. For reasons of brevity,
we present here only the most frequently matched keywords for
Figure 2: A tweet that is considered irrelevant to water qual- both languages in Table 1, while the complete list is provided to
ity participants along with the dataset in the task’s repository2 . The
bilingual dataset is separated into two sets: the development-set that
contains 8,000 posts and the test-set with 2,000 posts. In order to be
Task focuses exclusively on social media data and shifts the domain fully compliant with the Twitter Developer Policy, only the IDs of
of application from floods to water quality, safety and security. The the tweets are distributed to the participants. Thus, it was ensured
overall goal of the task is to tackle the aforementioned challenges at the time of releasing the dataset that all tweets were still online.
and use textual information (as well as visual information and The ground truth of the dataset reflects the relevance of a tweet
metadata) from a bilingual dataset of Twitter posts in order to (relevant / not relevant) and has been manually collected with hu-
identify tweets that refer to concerns about water. man annotation. The annotation has been realized again by the
Eastern Alps River Basin District. Apart from their valuable exper-
2 TASK DESCRIPTION tise on the domain, they were also able to annotate tweets in their
The WaterMM Task deals with the analysis of social media posts native language, i.e. Italian. It should be noted that each tweet has
from Twitter with regards to issues of water quality, safety and been annotated by a single person and not by multiple annotators.
security. The participants of this task are provided with a set of Initially, solely the ground truth for the development-set is re-
Twitter post IDs in order to download the text, the attached image leased, since the ground truth for the test-set is used in the eval-
(if it exists) and the metadata of tweets that have been selected uation stage and will be available only after the completion of
with keyword-based search that involved words/phrases about the MediaEval 2021. Participants are provided with key-value pairs of
quality of drinking water (e.g. strange color, smell or taste, related Tweet ID and ground truth label for the relevancy (0=not relevant/
illnesses, etc.). Nevertheless, the occurrence of such phrases in a 1=relevant). In particular, 1,374 tweets (17.18%) of the development-
tweet might not necessarily reflect a case of water contamination. set are relevant and 6,626 (82.82%) are not relevant to water quality,
The objective of this task is to build a binary classification system showing that it is a quite imbalanced training dataset and partici-
that will be able to distinguish whether a post is relevant or not to pants should consider this issue.
water-quality issues. An example of a relevant tweet is shown in
Fig. 1, while an irrelevant tweet in Fig. 2. Participants can tackle the 4 EVALUATION
task using text features, image features, metadata, or a combination F1-Score is selected as the official metric for evaluating the binary
of the above, and they are allowed to submit up to 5 runs: classification of tweets as relevant (1) and not relevant (0) on the
• Required run 1: automated using textual information only test set, since this measure is the harmonic mean between precision
• Optional run 2: automated using fused textual and visual and recall, taking both metrics into account. Participants are also
information encouraged to carry out a failure analysis of their results in order
• Optional run 3: automated using fused textual and visual to gain insight in the mistakes that their classifiers make.
information as well as other metadata
• General runs 4 & 5: everything automated allowed, includ- ACKNOWLEDGMENTS
ing using data from external sources This work has been supported by the EU’s Horizon 2020 research
and innovation programme under grant agreements H2020-832876
3 DATASET DESCRIPTION aqua3S, H2020-883484 PathoCERT, and H2020-101004157 WQeMS.
The dataset of the task is a set of social media posts collected 2 https://github.com/multimediaeval/2021-WaterMM/blob/main/dataset/keywords.
from Twitter during one year, i.e. from May 2020 to April 2021, json
WaterMM MediaEval’21, December 13-15 2021, Online
REFERENCES
[1] David E Alexander. 2014. Social media in disaster risk reduction
and crisis management. Science and engineering ethics 20, 3 (2014),
717–733.
[2] Stelios Andreadis, Ilias Gialampoukidis, Anastasios Karakostas, Ste-
fanos Vrochidis, Ioannis Kompatsiaris, Roberto Fiorin, Daniele Nor-
biato, and Michele Ferri. 2020. The flood-related multimedia task at
mediaeval 2020. In Proceedings of the MediaEval 2020 Workshop, Online.
14–15.
[3] Bischke Benjamin, Helber Patrick, Zhao Zhengyu, Borth Damian,
and others. 2018. The Multimedia Satellite Task at MediaEval 2018:
Emergency response for flooding events. (2018).
[4] Benjamin Bischke, Patrick Helber, Simon Brugman, Erkan Basar,
Zhengyu Zhao, Martha Larson, and Konstantin Pogorelov. The Multi-
media Satellite Task at MediaEval 2019: Estimation of Flood Severity.
In Proc. of the MediaEval 2019 Workshop (Oct. 27-29, 2019). Sophia
Antipolis, France.
[5] Benjamin Bischke, Patrick Helber, Christian Schulze, Venkat Srini-
vasan, Andreas Dengel, and Damian Borth. 2017. The Multimedia
Satellite Task at MediaEval 2017.. In MediaEval.
[6] Arjen Boin, Magnus Ekengren, and Mark Rhinard. 2020. Hiding in
plain sight: Conceptualizing the creeping crisis. Risk, Hazards & Crisis
in Public Policy 11, 2 (2020), 116–138.
[7] Yan Jin, Brooke Fisher Liu, and Lucinda L Austin. 2014. Examining
the role of social media in effective crisis management: The effects of
crisis origin, information form, and source on publics’ crisis responses.
Communication research 41, 1 (2014), 74–94.