This paper describes our system submitted to HASOC FIRE 2020. The goal of the shared tasks was to detected hate speech and ofensive content in three languages namely Hindi, English, and German. The ifrst subtask was a binary classification of a sentence into hate and ofensive and normal. In the second subtask, a more granular classification of hate/ofensive sentences was required. So overall there were 6 subtasks, 2 per language for 3 languages. We propose a system that performs all these tasks with a single model by jointly training a multilingual system on a combined corpus for all languages. It is relatively easy to fine-tune a model per task but it can pose various problems during deployment. These days most of the online platform supports multiple languages and it is not practical to deploy one model per language or per task. There are so many languages and tasks to cover and the online system will quickly run into memory and latency issues if there were multiple models handling the same task for diferent languages. Our system is capable of handling all subtasks for three languages with a single deep learning model. On the test set, we achieved a weighted average f1-score of 0.62, 0.85, 0.75 on subtask A and 0.35, 0.51, 0.43 on subtask B for Hindi, English, and German respectively.
With the rising popularity of the internet, there has been the growth of various online platforms and social media platforms being among them. Currently, we have multiple social media platforms operating globally across many countries and regions. There is a need for automatic monitoring systems for these social media platforms to detect unsocial elements such as hate speech and ofensive content which can quickly disrupt the harmony in societies. This problem becomes more challenging when we want a platform to support multiple languages to fulfill the needs and enhance the experience of users from various backgrounds. The solution needs to scalable across multiple languages and multiple tasks. HASOC 2020 provides a platform to test hate speech and ofensive content detection through the competition organized by them. We proposed and evaluated a system capable of supporting multiple languages and performing multiple tasks using a single deep learning model. The requirement of such systems is becoming obvious day by day as many social media platforms started supporting a large number of https://www.cse.iitb.ac.in/~pb/ (P. Bhattacharyya)
© 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). languages and it is very ineficient to have one model per language during deployment as it increases the memory requirement and also adds up the training time.
In HASOC 2019 [
In HASOC: FIRE 2020 [5], organizers have presented two challenges and provided the training and test datasets. These two subtasks were for three Indo-European Languages- Hindi, English, and German. The dataset was curated by collecting tweets and manually annotating the dataset for hate speech and ofensive content. The challenge consists of the following two tasks for each of the three languages mentioned above: • Sub-task A- Identifying Hate, ofensive, and profane content: This was basically a binary classification task where each of the tweets was required to be classified either as a normal tweet or hate speech/ofensive. • Sub-task B- Discrimination between Hate, profane and ofensive posts: This was a further granular classification of hate speech and ofensive tweets. Each of the hate speech or ofensive tweets was required to be classified as either hate or ofensive or profane. This subtask was relatively more challenging than the first one due to an increase in the number of classes and a reduction in dataset size as only tweets labeled as hate speech or ofensive are relevant for training.
We propose a joint multitask learning approach to perform all the six subtasks in the challenge using a single deep learning model. We combined the datasets of all three languages and finetuned a multilingual BERT [6] on two subtask A and B together. The results of this system were pretty competitive and reduced the resource requirements during training and inference. In section 2 we explain our system and training method in detail. In section 3 we report the performance of our system for various subtasks on the test dataset.
In this section, we provide the dataset description and statistics, deep learning network architecture, and its joint training on multilingual corpora on both subtasks.
Organizes have collected tweets and annotated them for subtask A subtask B in all three languages. This dataset was made available to the participants along with the dataset of HASOC: 2019. However, we used only HASOC 2020 dataset to train and evaluate our system. Since, the dataset was for Hindi, English, and German languages so it contained both Roman and Devanagari scripts. Hindi language dataset also contained some amount of code-mixing and transliteration. For subtask A, each tweet was labeled as either (NOT) Non-Hate-Ofensive or (HOF) Hate and Ofensive and for subtask B, each of the (HOF) labeled tweets was further categorized into (HATE) Hate speech, (OFFN) Ofensive, and (PRFN) Profane. Table 1 provides details about the number of tweets per class in the training dataset for three languages. This also shows class imbalance in the dataset which is an important issue to tackle while building robust hate speech and ofensive content detection systems. We split the provided dataset into ifve-folds and performed 5-fold cross-validation.
Since the dataset was directly scraped from Twitter and raw tweets contained various unnecessary features we processed the text before feeding it to the deep learning model. As a pre-processing step we performed the following tasks: • Removed @mentions and RT from the tweets • Replaced website URLs with string URL • Removed # character from words used as hashtags • Removed multiple spaces, if present any sentence
As a deep learning model, we choose multilingual BERT [6] and trained jointly on both subtasks on a combined corpus of all three languages. The final hidden state vector of special token [CLS] is taken as an aggregate representation [7] of the entire tweet and this 768-dimensional vector is passed throughout two diferent fully connected neural networks. One neural network ends with a softmax layer having two heads responsible for subtask A and the second neural network ends with a softmax layer having four heads responsible for subtask B. In subtask B we did four-class classification by also considering NOT labeled tweets for training along with HATE, OFF and PRFN labels. Figure 1 depicts the overview of the proposed multilingual deep learning system. We combined the loss from both the network heads and the back-propagate average of these two losses. The entire network was then jointly trained on both subtasks by gradually unfreezing the layers of the multilingual BERT model.
We did extensive hyperparameter tuning to get the best performance from the system. The deep learning model was trained on a combined corpus of all three languages and jointly fine-tuned for both the subtasks of each language. Since there was a class imbalance in the training dataset, hence we employed weighted cross-entropy loss giving more weight to counter the under-representation of some classes.
In table 2, we report the performance of our system on the test set provided by the organizers. Subtask A was a binary classification system having two class labels, NOT and HOF. Subtask B was a four-class classification system with four labels, NONE, HATE, OFF, and PRFN. We used the macro average f1-score as an evaluation metric. We also report accuracy, macro average precision, and recall for each subtask of six languages. In leaderboard scores, on average there was an absolute diference of 0.0356 in the macro average f-score of our system and top-3 bestperforming systems in individual tasks. Given that we have trained a single deep learning model for every subtask this trade-of between f-score and resources (memory and latency) seems promising. In table 3, we compare the the performance of our system with top-3 performing system as per leaderboard published by organizers.
With an increase in diversity and number of users, online platforms have to support multiple languages. This leads to the demand for language scalable solutions if we want to perform hate speech and ofensive content detection. Having one deep learning model per language or per task will be a very ineficient solution in deployment if our platform has to support hundreds of languages and multiple tasks. We purposed and established the eficacy of a multilingual and multitask system that can support three languages and perform two tasks for each language. The performance of our system was very competitive and at par with single individual models ifne-tuned for one task.
In the future, we would like to expand our system to more languages and increase the number of tasks it can perform. We will also explore the use of other multilingual transformer models and do a comparative analysis.
We thank all the organizers of HASOC, FIRE 2020 for arranging this opportunity to push the research in multilingual hate speech and ofensive content detection. We also express our gratitude towards them for their continuous support throughout the competition and for being very accommodating towards the requests from participants. Association for Computational Linguistics, Minneapolis, Minnesota, USA, 2019, pp. 54–63.
URL: https://www.aclweb.org/anthology/S19-2007. doi:1 0 . 1 8 6 5 3 / v 1 / S 1 9 - 2 0 0 7 . [4] M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, R. Kumar, SemEval-2019 task 6: Identifying and categorizing ofensive language in social media (OfensEval), in: Proceedings of the 13th International Workshop on Semantic Evaluation, Association for Computational Linguistics, Minneapolis, Minnesota, USA, 2019, pp. 75–86. URL: https://www.aclweb.org/ anthology/S19-2010. doi:1 0 . 1 8 6 5 3 / v 1 / S 1 9 - 2 0 1 0 . [5] T. Mandl, S. Modha, G. K. Shahi, A. K. Jaiswal, D. Nandini, D. Patel, P. Majumder, J. Schäfer, Overview of the HASOC track at FIRE 2020: Hate Speech and Ofensive Content Identification in Indo-European Languages), in: Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, CEUR, 2020. [6] T. Pires, E. Schlinger, D. Garrette, How multilingual is multilingual BERT?, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, 2019, pp. 4996–5001. URL: https://www. aclweb.org/anthology/P19-1493. doi:1 0 . 1 8 6 5 3 / v 1 / P 1 9 - 1 4 9 3 . [7] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. URL: https://www.aclweb.org/ anthology/N19-1423. doi:1 0 . 1 8 6 5 3 / v 1 / N 1 9 - 1 4 2 3 .