This paper presents our contributions to the MediaEval 2021 task namely ”WaterMM: Water Quality in Social Multimedia”. The task aims at analyzing social media posts relevant to water quality with particular focus on the aspects like watercolor, smell, taste, and related illnesses. To this aim, a multimodal dataset containing both textual and visual information along with meta-data is provided. Considering the quality and quantity of available content, we mainly focus on textual information by employing three diferent models individually and jointly in a late-fusion manner. These models include (i) Bidirectional Encoder Representations from Transformers (BERT), (ii) Robustly Optimized BERT Pre-training Approach (XLMRoBERTa), and a (iii) custom Long short-term memory (LSTM) model obtaining an overall F1-score of 0.794, 0.717, 0.663 on the oficial test set, respectively. In the fusion scheme, all the models are treated equally and no significant improvement is observed in the performance over the best performing individual model.
In recent years, social media has emerged as a valuable tool and
platform to discuss and convey concerns over diferent challenges and
daily life issues [
This paper provides a detailed description of the methods proposed by team CSE-Innoverts for the water quality analysis represented in the MediaEval task. The dataset provided for the task covers multi-modal information including textual, visual, and metadata. However, images are available for very few posts. Moreover, the majority of the available images are not relevant. Thus, we mainly focus on textual information by proposing four diferent solutions as detailed in Section 2. 2
In total, we submitted 4 diferent runs by employing three
diferent Neural Networks (NNs) architectures, namely BERT [
• BERT-based Solution (Run 2): In this proposed solution, we rely on a pre-trained BERT model, which is finetuned on the data development set provided by the task organizers. Before proceeding with fine-tuning the model, necessary pre-processing is performed, using Tensorflow libraries, to bring the data in the required form to be used for training the model. Since it is a binary classification task, we used Binary Cross entropy loss function with Adaptive Moments (Adam) optimizer. • XLM-RoBERTa-based Solution (Run 3): In this approach, we rely on the multilingual pre-trained XLM-RoBERTa model that is fine-tuned on the development set. As a ifrst step, the input text is tokenized in the pre-processing phase. A pre-trained model is then fine-tuned on the preprocessed data using Adam optimizer with a binary crossentropy loss function. • LSTM-based Solution (Run 4): In this approach, we rely on a custom LSTM model. The model is composed of three layers including an input, LSTM, and output layer. We used this model as a baseline for our experiments. However, the model obtained encouraging results on the development and was thus utilized in the fusion scheme.
We also cleaned the data before feeding into the models by removing URLs, account handles, emojis, and unnecessary punctuation.
Input Text
Models
Late Fusion
Predicted_Label Moreover, in all the proposed solutions, we used an up-sampling technique to balance the dataset.
For the evaluation of the proposed methods, we used four diferent metrics, namely (i) accuracy, (ii) micro precision, (iii) micro recall, and (iv) micro F1-score. Precision, recall, and f1-scores are the oficial metrics while accuracy has been used as an additional metric for the evaluation of the methods on the development set.
Table 1 provides the experimental results of our proposed solutions on the development set. To this aim, a separate validation set composed of 1,810 samples is used. Run 1 represents our fusion-based solutions while Run 2, Run 3, and Run 4 represent our solutions based on the individual models namely BERT, RoBERTa, and LSTM, respectively. On the development set, overall better results are obtained with the BERT-based solution obtaining an overall F1-score and accuracy of 0.950 and 0.929, respectively. The least performance in terms of F1-score and accuracy are observed for RoBERTa.
Table 2 provides the oficial results on the test set in terms of precision, recall, and f1-score. Overall better results are obtained for BERT among the individual model-based solutions while the least scores are observed for the LSTM based solution. However, interestingly, no significant improvement in the performance for the fusion-based solution over the best-performing individual modelsbased solution has been observed. One of the possible reasons could be the low-performing models as all the models are treated equally by simply aggregating the obtained posterior probabilities. This limitation could be addressed by using merit-based fusion where weights are assigned to the contributing models based on the performance of the model.
The quantity and quality of the images associated with the social media posts were not good enough to contribute to the task. Thus, we focused on the textual information only by employing several NNs based solutions. In total, four diferent solutions including a fusion and three individual models based solutions. In the current implementation, we used a simple fusion mechanism by simply aggregating the posterior probabilities obtained with each individual model.
In the future, we aim to employ more sophisticated fusion schemes by assigning merit-based weights to the contributing models. We also aim to make use of the additional information available in the form of metadata in our future fusion-based solutions.
WaterMM: Water Quality in Social Multimedia