The detection of hate speech online has become an important task, as ofensive language such as hurtful, obscene and insulting content can harm marginalized people or groups. This paper presents TU Berlin team experiments and results on the task 1A and 1B of the shared task on hate speech and ofensive content identification in Indo-European languages 2021. The success of diferent Natural Language Processing models is evaluated for the respective subtasks throughout the competition. We tested different models based on recurrent neural networks in word and character levels and transfer learning approaches based on Bert on the provided dataset by the competition. Among the tested models that have been used for the experiments, the transfer learning-based models achieved the best results in both subtasks.
Hindi and Marathi. The second subtask focuses on the identification of conversational hatespeech in Code-Mixed Languages. The TU Berlin team focuses on the subtasks 1A and 1B. Subtask 1A is a coarse-grained binary classification task where tweets should be classified into two classes: • (NOT) Non Hate-Ofensive: These posts do not contain any hate speech, profane or ofensive content • (HOF) Hate and Ofensive: These posts contain hate, ofensive and profane content Subtask 1B is is a three-class classification task ofered for English and Hindi, where hatespeech, profane and ofensive posts from subtask 1A are further classified into the following categories: • (HATE Hate speech: this class contains posts which hate-speech content • (OFFN Ofensive: posts in this class contain ofensive content • (PRFN Profane: posts in this class contain profane content
In this paper the proposed models for classifying tweets into one of the classes for the
respective subtask are presented. For this purpose, the state-of-the-art NLP methods are applied
to classify the posts and categorize them into the classes. Hereby the team of TU Berlin focuses
on the English dataset. We used transfer learning models based on the BERT language model [
The following section 2 describes some of the state-of-the-art models for the task of hate speech detection in English. Section 3 describes the provided train and test dataset, whereas section 4 contains details about data processing and the experiments and models applied. Furthermore, in section 5 the achieved results are analyzed, and section 6 summarizes and concludes the approaches and results.
In this section, we overview some of the recent approaches for automatic hate speech detection
from English text. Although the automated approaches for hate speech detection could be
categorized into keyword-based, source metadata and machine learning based approaches [
Among the proposed models for the HASOC shared task on 2020, Mishra et al. has been used
a Long Short-Term Memory (LSTM) based model using Glove vectors [
Badjatiya et al. did diferent experiments based on three diferent neural network architecture
to detect hate speech tweets in Twitter [
The English dataset of HASOC 2021 for the subtasks 1A and 1B, contains the text content of tweets in English, IDs, and the labels for subtask 1A and 1B, respectively. the statistics of the training dataset is presented in Table 1. Moreover, the test dataset contains 1281 tweets which should be categorized into one of the classes based on the subtask.
The content would contains hashtags, emojis, links and usernames that refer to a user on
Twitter. A sample of the dataset in diferent categories is presented in Table 2. More details
about the datasets are provided in [
This section contains a short description on the used pre-processing steps and also the developed models and experiments for the task of hate speech detection.
For pre-processing of the raw data, we followed the same procedure as the experiments on
the last year’s competition [
The best performance of the last year HASOC competition for the English dataset have been
achieved by [
Total # of Instances English 3843
Subtask 1A HOF NOT 2501 1342
HATE 683
Subtask 1B OFFN PRFN 622 1196
NONE 1342 This is enough of yours Modi This is not skill India it is kill India @narendramodi #ExitModi #Resign_PM_Modi https://t.co/m9FZyU4Lfg Please, abdicate! You failed us. You failed everyone. Everyone is sufering. EVERYONE! #ModiKaVaccineJumla @Feisty_Waters Ok. What did you do to piss of the universe? @ndtv Nothing gonna help you please #Resign_PM_Modi HOF
HOF HOF
NOT
showed also promising results for similar task. Therefore, the TU Berlin team focuses on BERT
based transfer learning approaches for the proposed subtasks. We also did some experiments
on character level LSTM models which achieved our best results on the last year’s competition
[
We developed two diferent models based on LSTM networks. We developed a smaller, character based architecture, Char_LSTM hereinafter, and a deeper and more complex network based on words, Word_LSTM hereinafter. Since people sometimes do minor changes on the words (e.g., by repeating some characters) when they express hate speech, a word based model may not signal those terms properly. As a result, we also developed a character based model to compare the outcomes of the models.
For the Char_LSTM, we tried out diferent hyper-parameters that includes: In our experiments, the batch size of 32 and, the Adam optimizer [21] and the Binary Cross Entropy (BCE) loss function have been used in both models. In the Word_LSTM model, we tested either using Glove pre-trained vectors and training the embedding layer from scratch. The detailed results of the proposed models are presented in the section 5.
• Embedding dimension [50, 100, 200]
• Hidden dimension [
In addition to the models based on Recurrent Neural Networks (RNNs), we tested two transfer learning based models using BERT language model [22]. In one of the experiments, we finetuned English Bert for the task of hate speech identification. For this purpose, we followed the recommended hyper-parameters by the authors [22].
As the other transfer learning based model, we used Bert for extracting features from textual data. In other words, in this approach, the Bert language model was used to convert text data into vectors. The resulting vectors inputted into a Gated Recurrent Units (GRU) network. Diferent hyper-parameters tested on the data to choose the best parameters. The range of diferent hyper-parameters which had been used in the feature extraction approach are as follow: • Hidden dimension [32, 64, 128, 256, 512] • Dropout [0.25, 0.5, 0.75] Like the LSTM based models, the batch size of 32 and, the Adam optimizer [21] and the Binary Cross Entropy (BCE) loss function have been used in this experiment. We present the detailed results by the diferent architectures in section 5.
In this section the achieved results on the training data are presented. For doing the experiments, the training dataset has been divided into train, validation and test datasets. The train part contains 70% of the whole data, the validation part consist of 10% of the data, and the test part contains the remaining 20% of the provided dataset.
We tested all of the mentioned models with diferent hyper-parameters. The best achieved
results are shown in tables 3 - 5. In order to determine the impact of the pre-processing steps
on the final results, we’ve repeated the experiments with the same hyper-parameters without
applying the pre-processing steps. Although the runs without applying pre-processing could
achieve competitive results in some cases, the experiments based on the pre-processed data
outperforms the other ones in most of the cases. The performance of the submitted models for
both sub-tasks are reported in details in [
The same architectures have been trained on the data for the sub-task 1B. The best achieved results on the second task were applied on the sub-task 1B test dataset and submitted to the shared task.
In this paper, we presented the proposed models on the task 1A and 1B of the shared task on hate speech and ofensive content identification in English. We used a BERT based architecture and word and character based LSTM models for training a model to classify tweets into ofensive and not ofensive categories. Our experiments show that Bert based model outperform the other approaches.
Model name
Pre-processed
Hyper-parameters Embedding dimension Hidden dimension dropout Char_LSTM Word_LSTM yes yes yes yes no no yes yes yes no 0.75 0.78 0.76 0.79 0.75 0.77 0.81 0.83 0.80 0.79
F1
Since over-fitting was one of the main issues for training diferent models during the competition, enriching the training data by adding data samples from diferent resources could be a possible solution for improving the results. Moreover, the proposed transfer learning based results could be compared with the results from the the other state-of-the-art language models like GPT-3 to check if there is a significant diference in the performances.
We would like to thank the organizers of HASOC2021 shared task for organizing the competition and taking time on the inquiries. M. Mitra (Eds.), Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, 2020, volume 2826 of CEUR Workshop Proceedings, CEUR-WS.org, 2020, pp. 823–828. URL: http://ceur-ws.org/Vol-2826/T10-3.pdf. [20] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. Zettlemoyer, Deep contextualized word representations, in: M. A. Walker, H. Ji, A. Stent (Eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), Association for Computational Linguistics, 2018, pp. 2227–2237. URL: https://doi.org/10.18653/v1/n18-1202. doi:10.18653/v1/n18-1202. [21] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: Y. Bengio, Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL: http://arxiv.org/abs/ 1412.6980. [22] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, CoRR abs/1810.04805 (2018). URL: http://arxiv. org/abs/1810.04805. arXiv:1810.04805.