In this paper, we present our submission from the team CAROLL_Passau for subtask 1A of the HASOC 2021 workshop. Our presented model, C-BiGRU, is composed of a Convolutional Neural Network (CNN) together with a bidirectional Recurrent Neural Network (RNN). We utilized word embeddings to allow our model to apprehend the correlation between words in the text. The structure of our model enables it to capture the contextual information along with the long-term dependencies in the text in order to perform binary classification on ofensive text. We evaluated our model on the test data provided by the HASOC organizers. Our model achieved a macro F1 score of 75.04%, accuracy of 77.48%, precision and recall with the scores of 74.63% and 75.60% respectively.
Social media is growing continuously to be one of the powerful means of communication
around the world, spreading various forms of user-generated content containing various sorts
of information[
The remainder of the paper is organized as follows. In the second section, we present previous work related to hate speech detection. We then provide an overview of the baseline model. In the third section, the system description including the experimental setup of our model C-BiGRU is presented. In the fourth section, we report our results and discussion for the challenge. Lastly, we conclude the paper and provide changes and enhancements that can be applied to our approach in the future.
A variety of ideas and algorithms have been proposed over the years to identify and categorize
ofensive language, aggression, hatespeech, and various abusive language phenomena on social
media [
A classification methodology through transfer learning using CNN is proposed by [
In this section, we describe our model C-BiGRU along with the pre-processing steps and
embeddings utilized in the system. The architecture of the model is shown in Figure 1
3.1. Data
We utilized the data provided by the organizers of the HASOC 2021 workshop [
Pre-processing of tweets includes removal of emojis, URLs, and any additional spaces. In addition, the tweets are converted to lowercase, HTML character encodings are replaced with their respective token representation or literal. Apart from this, TweetTokenizer from NLTK is used to split the tokens containing special characters (e.g. ’/’, ’-’).
We begin our model construction with the embedding layer. The pre-processed tweets are fixed to a sequence length of 150 tokens, longer sequences are clipped at the end and shorter sequences are padded with a masking token. Following this, we create a dictionary containing all unique tokens that appear more than once and map them to their number of occurrences in the respective corpus. As a next step, we construct the weighting matrix * to form the embedding layer, where dim is the dimension of the fastText [22] embedding model used in our setup and m the number of unique tokens , ∈ {1, .., }. The word vector of is stored in W if the token is present in the embedding model. If has no pre-trained word vector, we generate a random vector drawn from the uniform distribution within − ︂[ √︁ 6 , as suggested by [23].
Next in line is the convolutional layer where n-gram features are extracted from the sequence of tokens. The (kx128) 1-dimensional filters present in this layer assist in the accomplishment of feature extraction. The value of k varies from 2 to 5 and represents diferent window sizes. The outputs produced by this layer are all of the same sequence lengths which are attained through padding. Furthermore, we also make use of the ReLu activation function. The resulting feature maps are concatenated and then forwarded to the recurrent layer.
GRUs which are the improvised version of standard recurrent neural networks aim to solve the vanishing gradient problem using a gating mechanism. Originally proposed by [24] these are used in capturing long-term dependencies of input-sequence. GRUs have appeared to accomplish practically identical outcomes to LSTM in sequence modeling tasks. They have been reported to outperform LSTM while training on smaller datasets [25]. In our model, we made use of bidirectional GRU as the recurrent layer. As input to one of the GRU layers, the output from the previous layer is received meanwhile, the reversed form of the same output is used for the other GRU layer. The GRU layers return hidden states for each processed feature map. The hidden states from both layers are concatenated. The resultant output of size 150*128 is obtained by setting the hidden layers of both the layers to 64.
The output from the previous layer is then passed through a global max-pooling layer which reduces the output space to (1x128) nodes. Finally, a fully connected layer with 32 neurons that connect to a single output neuron that utilizes the sigmoid activation function is used. In order to prevent overfitting, a dropout layer is introduced with a rate of 0.2 before the single output neuron. Besides that, another dropout layer with a rate of 0.2 is included after the embedding layer.
Moreover, we used cross-entropy as an error function for our model and the Adam optimizer to update our network weights [26]. During the training phase, early stopping is implemented and we perform a split on the data resulting in 10% of the data as validation dataset and 90% of the data for training. The batch size of the gradient update is set to 32 with a maximum of 5 epochs.
To compare the performance of our model with a baseline setup, we used a classification model, Logistic Regression. The model was evaluated using the validation dataset after carrying out the pre-processing steps as described previously in section 3.2. The baseline model after evaluating resulted in a macro F1 score of 70.03%, accuracy of 75.87%, precision and recall with the scores 64.04% and 51.05% respectively. Afterward, the system was tested using the test data and achieved a macro F1 score of 73.46%, accuracy of 78.13%, precision and recall with the scores 76.14% and 72.22% respectively. The scores of the baseline model are compared with our C-BiGRU system in the following section.
To study the performance of our C-BiGRU model, we conducted experiments in the Hindi language to detect hate speech utilizing the train and test data as described in section 3.1. The model was trained and thereafter, evaluated using the validation data which is 10% of the train data. During the validation phase, the system achieved an accuracy of 76.30%, recall, precision, and macro F1 score of 72.61%, 61.49%, and 63.64% respectively. Following this, the model was tested using the test data and the system achieved a macro F1 score of 75.04%, accuracy of 77.48%, precision and recall with the scores 74.63% and 75.60% respectively. The results are displayed in the table 1 Our presented C-BiGRU model successfully performed the classification task with noticeable results and surpassed the baseline model. The model also showed consistent F1 scores during the validation phase and test phase.
The overview of the results and findings of HASOC 2021 is presented in [27] This experiment helped us evaluate the performance of the C-BiGRU model in another language as a follow-up to the previously impressive performances of the model in other languages English, German, Danish, and Turkish [28, 29, 30].
In this submission, we described our approach to detecting hate speech present in social media posts and we presented the architecture of our model. We further provided the results of our model evaluated on the Hindi language with significant performance. Although our model performed well, there are a few limitations such as handling of unknown tokens, and distinguishing between explicit and implicit hate speech, which is an important task that is not easy to overcome, as explained and investigated by [31, 32].
In the future, we plan to extend our approach to work on identifying rhetorical figures and multi-word expressions containing abusive language. We will also work on improving the model to identify the fine-grained diference between implicit and explicit ofensive posts. Apart from this, domain-specific word embeddings can help in handling the unknown tokens. Other potential features such as POS (Parts of Speech) tagging will also be explored.
The project on which this report is based was funded by the German Federal Ministry of Education and Research (BMBF) under the funding code 01|S20049. The author is responsible for the content of this publication. inghe, M. Zampieri, D. Nandini, A. K. Jaiswal, Overview of the HASOC subtrack at FIRE 2021: Hate Speech and Ofensive Content Identification in English and Indo-Aryan Languages, in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021. URL: http://ceur-ws.org/. [22] A. Joulin, E. Grave, P. Bojanowski, T. Mikolov, Bag of tricks for eficient text classification, in: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, Association for Computational Linguistics, Valencia, Spain, 2017, pp. 427–431. URL: https://aclanthology.org/E17-2068. [23] K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification., in: In Proceedings of the IEEE international conference on computer vision, pages 1026–1034., 2015. [24] K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio, Learning phrase representations using RNN encoder–decoder for statistical machine translation, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Doha, Qatar, 2014, pp. 1724–1734. URL: https://aclanthology.org/D14-1179. doi:10.3115/v1/D14-1179. [25] J. Chung, Çaglar Gülçehre, K. Cho, Y. Bengio, Empirical evaluation of gated recurrent neural networks on sequence modeling, ArXiv abs/1412.3555 (2014). [26] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014). [27] S. Modha, T. Mandl, G. K. Shahi, H. Madhu, S. Satapara, T. Ranasinghe, M. Zampieri, Overview of the HASOC Subtrack at FIRE 2021: Hate Speech and Ofensive Content Identification in English and Indo-Aryan Languages and Conversational Hate Speech, in: FIRE 2021: Forum for Information Retrieval Evaluation, Virtual Event, 13th-17th December 2021, ACM, 2021. [28] J. Mitrović, B. Birkeneder, M. Granitzer, nlpUP at SemEval-2019 task 6: A deep neural language model for ofensive language detection, in: Proceedings of the 13th International Workshop on Semantic Evaluation, Association for Computational Linguistics, Minneapolis, Minnesota, USA, 2019, pp. 722–726. URL: https://aclanthology.org/S19-2127. doi:10.18653/v1/S19-2127. [29] B. Birkeneder, J. Mitrovic, J. Niemeier, L. Teubert, S. Handschuh, upinf-ofensive language detection in german tweets, in: 14th Conference on Natural Language Processing KONVENS 2018, 2018. [30] O. Hussein, H. Sfar, J. Mitrović, M. Granitzer, NLP_Passau at SemEval-2020 task 12: Multilingual neural network for ofensive language detection in English, Danish and Turkish, in: Proceedings of the Fourteenth Workshop on Semantic Evaluation, International Committee for Computational Linguistics, Barcelona (online), 2020, pp. 2090–2097. URL: https://aclanthology.org/2020.semeval-1.277. [31] T. Caselli, V. Basile, J. Mitrović, I. Kartoziya, M. Granitzer, I feel ofended, don’t be abusive! implicit/explicit messages in ofensive and abusive language, in: Proceedings of the 12th language resources and evaluation conference, 2020, pp. 6193–6202. [32] T. Caselli, V. Basile, J. Mitrović, M. Granitzer, Hatebert: Retraining bert for abusive language detection in english, arXiv preprint arXiv:2010.12472 (2020).