This paper describes our model submitted for HASOC-2021 as the IIT_Patna team for hate and ofensive content identification in English and Hindi languages. A deep learning model, namely BERT+FastTextGRU, has been developed based on BERT and FastText, followed by GRU with attention. Our proposed model uses a BiGRU-based deep neural network to extract textual features, followed by an Attention layer to focus on the most important phrase of the text. The BERT language model and FastText embedding have been employed to examine the efectiveness of joint embedding representation compared to a single one. We have set up some baselines by varying the RNN architecture (LSTM/GRU) and the word vector representation approach (BERT/FastText). Our model outperforms all the baselines with the highest accuracy values of 76.32% for subtask-1A (EN), 56.73% for subtask-1B (EN), 69.17% for subtask-1A (HI) and 40.45% for subtask-1B (HI).
With the progress of technology and the growing popularity of many social media platforms,
the number of people active on these platforms has risen dramatically. The majority of the
time, these individuals abuse their right to free of speech and violate the forums’ permissible
usage standards. This has prompted the detection of any ofensive or obscene posts, comments,
photos, or other content and the prevention of further distribution in order to limit the impact
on social media. On social media, user-generated content is not always structured according
to the standards. In reality, foul language content has been common on social media in recent
years [
In this work, we have developed a deep learning model(BERT+FastText-GRU) based on BERT
and FastText followed by GRU with attention to solve subtask-1A and subtask-1B in both Hindi
and Engish languages [
Researchers have been studying and reporting their findings and observations linked to online
abuse of social media platforms for quite some time now like in [
The task in HASOC 2021 is divided into sub tasks based on the kind and target of ofences. Detailed train and test set distributions of subtask-1A and subtask-1B in both Hindi and English languages are shown in Table 1.
• Subtask-1A : Hate and Ofensive labels detection Here we have to categorize between hate and ofensive and non hate and ofensive tweets. The class labels are HOF and NOT.
This is for both English and Hindi languages. In Hindi subtask-1A train data, there are total of 4594 instances, of which 3131 instances are of not ofensive while 1433 instances are ofensive. In English subtask-1A train data, there are total 3843 instances of which 2501 instances are of Hate & ofensive while 1342 instances are of Not hate & ofensive. • Subtask-1B : Ofensive labels detection Here we have to categorize between Profane, Hate, Ofensive and none labels on tweets. The class labels are PRFN, HATE, OFFN and NONE. This is also for both English and Hindi languages. In Hindi sub task-1B train data, there are 3161 instances of None, 654 instances of Ofensive, 566 instances of Hate and 213 instances of Profane. In English sub task-1B, there are 1342 instances of None, 1196 instances of Profane, 683 instances of Hate and 622 instances of Ofensive.
This section describes an attention based framework we have developed to identify hatespeech and Ofensive Content in English and Hindi Languages. Figure 1 depicts the overall architecture of our proposed Bert+FastText-GRU model. 3.1. BERT BERT [23] is a Transformer-based [24] language model. Fine-tuning BERT has shown a decent improvement in solving several Natural Language Processing (NLP) tasks like text classification, question answering, machine translation etc. BERT has diferent variances like BERT base model, Multilingual BERT (M-BERT), medical BERT, etc. For English language based task we have considered BERT base model1. We choose the M-BERT2 for the Hindi language task since it has been trained on 104 languages, including Hindi.
The Facebook Research Team developed FastText[25] for eficient word embedding of more than 157 diferent languages. FastText model was trained using CBOW technique with character n-grams of length 5, a window of size 5, and 10 negatives, and it returns 300 dimensional dense vector corresponding to each token. In a social media text, the inclusion of Out-of-Vocabulary (OOV) words is a severe issue. To evade automatic checking, users in social media often perform 1https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1 2https://tfhub.dev/google/bert_multi_cased_L-12_H-768_A-12/1
I N P U T T W E E T
BERT Model FastText Wn w1 w2 w3 Wn
Sequence Output
300 Embedding Matrix
To learn the contextual information of input tweet from both the directions, the word vectors from both BERT and FastText are passed through two separate Bidirectional GRUs [28] layer. To capture long-term dependencies in the tweet, bi-directional GRU sequentially encodes these feature map into hidden states as, ⃖⃗ ℎ = ⃖(⃖⃖⃖⃖⃖⃗ , ℎ−1 ) , ⃖ℎ⃖ = ⃖(⃖⃖⃖⃖⃖⃖ , ℎ+1 ) is a single hidden state representation. where each word vector of sentence i is mapped to a forward hidden state ⃖ℎ⃗ and backward hidden state ⃖ℎ⃖ by invoking ⃖⃖⃖⃖⃖⃖⃗ and ⃖⃖⃖⃖⃖⃖⃖, respectively. ⃖ℎ⃗ and ⃖ℎ⃖ are combined to form ℎ , which [ℎ = ⃖ℎ⃗
The basic principle behind the attention mechanism [29] is to give greater weight to the words that contribute the most to the meaning of the phrase. To produce an attended sentence vector, (1) (2) we leverage the attention mechanism on the Bi-GRU layer’s output. Specifically, = ℎ(
ℎ + ) =
( ∑( = ∑( ∗ ℎ ) ) )) (3) (4) (5) (6) generated by attention layer and attention weight for a particular word is .
Where is the hidden representation of ℎ and is the context vector. is the output
network.
As a loss function, we have used categorical cross-entropy ( , ̂ ) to train the parameters of the ( , ̂ ) = −
1 ∑
∑ =1 =1 ( ̂ ) and the number of tweets respectively.
Where ̂ is predicted label and is true label. and represents the number of classes, 2 ∈ 2
Let ( , 1 , 2 )=1 be a set of N tweets where 1 ∈ 1 (Hate Classes: HOF and NOT) and (Ofensive Classes: PRFN, HATE, OFFN and NONE). 1 , 2 represents the hate and ofensive labels corresponding to ℎ tweet respectively. This Bert+FastText-GRU Framework aims to learn a function that maps an unknown instance to its appropriate hate label 1 and ofensive label 2 .
Let the input sentence =
{ 1, 2, ..... } be a sequence of n input tokens, where n is the maximum length of a sentence. The input text is fed into both the BERT and FastText models. BERT generates two types of outputs: a pooled output of shape [ℎ , 768] the whole input sequences, and a sequence output of shape [ℎ , ℎ, 768] that represents for each input token. Let ∈ ℝ× be the embedding matrix obtained from the BERT model for input where = 768 is the embedding dimension of each token. On the other hand, FastText generates an embedding matrix ∈ ℝ× , where = 300. Outputs from both BERT and FastText are passed through two separate Bi-GRUs (128 hidden units) followed by an attention layer to learn the contextual information and to assign more weightages on the relevant words. The outputs and returned by the attention layers placed on the top of BERT+GRU and FastText+GRU are concatenated to make a joint representation of the input tweet . The concatenated feature vector is passed through a fully connected layers ( 1(100 ) + 2(100 )
) followed by an output layer.
We use Tanh activation in GRU cells and ReLU activation in all fully connected layers (100 neurons each). We add a 25% dropout after the attention and fully-connected layers for all subtasks in both languages. With a batch size of 32, we train our models for 10 epochs. We utilize Adam optimizer and set the learning rate to 0.001 to backpropagate the loss across the network.
This section shows the results of diferent baseline models and our proposed model. All our experiments were conducted on a hybrid cluster of multiple GPUs comprised of RTX 2080 Ti. All the models are implemented using Scikit-Learn 0.22.23 and Keras 2.4.34 with TensorFlow2 2.4.15 as a backend.
Following baselines have been introduced for comparison with our proposed approach. 1. BERT+GRU (Baseline-1): BERT generated word vectors are passed through BiGRU with attention layer. Output from attention layer is then passed to task specific fully connected layers[FC1(100) + FC2(100)] followed by output Softmax layer. 2. BERT+LSTM (Baseline-2): Same as Baseline-1 with one modification: GRU is replaced by LSTM. 3. FastText+GRU (Baseline-3): Same as Baseline-1, the only diference is the embedding approch. Here we have utilized FastText to generate word embedding of the input sentence. 4. FastText+LSTM (Baseline-4): This is identical to Baseline-3, but here GRU is replaced by LSTM. 5. BERT+FastText-LSTM (Baseline-5): This is identical to our proposed model with one modification: GRU is replaced by LSTM.
Table 2 presents the results in terms of accuracy, macro F1-score for all the baselines and the proposed model. From the result table, we can conclude that our model outperform all the baselines with a significant margin. Moreover, all the joint embedding based baselines performs better than any single embedding based baseline. Our model attained highest accuracy value of 76.32% 56.73%, 69.17% and 40.45% for four tasks, i.e, subtask-1A (EN), subtask-1B (EN), subtask-1A (HI) and subtask-1B (HI) respectively.
Out of four single embedding based baselines, FastText+GRU (Baseline-3) achieves higher f1 score of 74.92% and 66.41% for subtask-1A (EN) and subtask-1A (HI) respectively. While, 3https://scikit-learn.org/stable/ 4ttps://keras.io/ 5https://www.tensorflow.org/overview/ FastText+LSTM (Baseline-4) attains the best f1 score of 54.70% and 35.89% for subtask-1B (EN) and subtask-1B (HI) respectively. FastText+GRU achieves 2.73% and 1.55%, respectively, improvements in accuracy values for subtask-1A (EN) and subtask-1B (EN) over the BERT+GRU. On the other hand, FastText+LSTM (Baseline-4) attains the improvements in accuracy values for subtask-1A (EN) and subtask-1B (EN) over the BERT+LSTM (Baseline-2) as 3.26% and 1.59%, respectively. We have also examined that for all the single embedding based baselines when embedded with FastText performs better than the one embedded with BERT.
Experimental results of this work imply that joint embedding representation enhances the hate and ofensive post detection task’s performance compared to the one with single embedding.
With the expansion of digital sphere and advancement of technology, identifying hate, ofensive and profane content from the post is strongly determined by many of the researchers. In this work, we have developed a deep learning model (BERT+FastText-GRU) based on BERT and FastText followed by GRU with attention. This attention mechanism allows us to give more weight to the words that contribute the most to the phrase’s meaning. In the HASOC-2021, our model is scored 32 with a macro F1 score of 75.78% and 29ℎ with a macro F1 score of 56.52% for English subtask-1A and subtask-1B respectively out of all entries. In Hindi language, our model ranked 32 with a test accuracy of 69.17% for subtask-1A and 21 with a test accuracy of 40.45% for subtask-1B. From the results table, we can observe that the FastText-model outperformed the BERT-model in most experiments. Our team’s participation in the HASOC-2021 competition has been a valuable learning experience, and we look forward to learning from the other top-performing submissions. Proceedings of the 2012 conference of the North American chapter of the association for computational linguistics: Human language technologies, 2012, pp. 656–666. [15] I. Kwok, Y. Wang, Locate the hate: Detecting tweets against blacks, in: Twenty-seventh
AAAI conference on artificial intelligence, 2013. [16] J. Cheng, M. Bernstein, C. Danescu-Niculescu-Mizil, J. Leskovec, Anyone can become a troll: Causes of trolling behavior in online discussions, in: Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing, 2017, pp. 1217–1230. [17] B. Gambäck, U. K. Sikdar, Using convolutional neural networks to classify hate-speech, in:
Proceedings of the first workshop on abusive language online, 2017, pp. 85–90. [18] T. Davidson, D. Warmsley, M. Macy, I. Weber, Automated hate speech detection and the problem of ofensive language, in: Proceedings of the International AAAI Conference on Web and Social Media, volume 11, 2017. [19] A. Schmidt, M. Wiegand, A survey on hate speech detection using natural language processing, in: Proceedings of the fith international workshop on natural language processing for social media, 2017, pp. 1–10. [20] Z. Zhang, D. Robinson, J. Tepper, Detecting hate speech on twitter using a convolution-gru based deep neural network, in: European semantic web conference, Springer, 2018, pp. 745–760. [21] P. Liu, J. Guberman, L. Hemphill, A. Culotta, Forecasting the presence and intensity of hostility on instagram using linguistic and social features, in: Twelfth international aaai conference on web and social media, 2018. [22] J. Pavlopoulos, P. Malakasiotis, I. Androutsopoulos, Deeper attention to abusive user content moderation, in: Proceedings of the 2017 conference on empirical methods in natural language processing, 2017, pp. 1125–1135. [23] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in neural information processing systems, 2017, pp. 5998–6008. [25] E. Grave, P. Bojanowski, P. Gupta, A. Joulin, T. Mikolov, Learning word vectors for 157 languages, arXiv preprint arXiv:1802.06893 (2018). [26] Q. Le, T. Mikolov, Distributed representations of sentences and documents, in: International conference on machine learning, PMLR, 2014, pp. 1188–1196. [27] J. Pennington, R. Socher, C. D. Manning, Glove: Global vectors for word representation, in: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532–1543. [28] K. Cho, B. Van Merriënboer, D. Bahdanau, Y. Bengio, On the properties of neural machine translation: Encoder-decoder approaches, arXiv preprint arXiv:1409.1259 (2014). [29] Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, E. Hovy, Hierarchical attention networks for document classification, in: Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, 2016, pp. 1480–1489.