The current paper describes the system submitted by team AI_ML_NIT_Patna. The task aims to identify ofensive language in code-mixed dataset of comments in Indo-European languages ofered for English, German, Hindi collected from Twitter. We participated in both Sub-task A, which aims to classify comments into two class, namely: Hate and Ofensive (HOF), and Non- Hate and ofensive (NOT), and Sub-task B, which aims to identify discrimination between Hate (HATE), profane (PRFN) and ofensive (OFFN) comments. In order to address these tasks, we utilized pre-trained multi-lingual transformer (BERT) based neural network models and their fine-tuning. This resulted in a better performance on the validation, and test set. Our model achieved 0.88 weighted F1-score for English language in Sub-task A on testing dataset, and got 3 rank on the leaderboard private test data having F1 Macro average of 0.5078.
With the increase in usage of Internet, appealing to the propagation of thoughts, and
expressions of an individual, there has been a tremendous increase in the spread of online hate speech
[
The Hate Speech and Ofensive Content Identification in Indo-European Languages (HASOC) shared tasks of 2019, and 2020 focused on Indo-European languages in three diferent languages: English, German, and Hindi. The shared tasks have two sub-tasks: Sub-task A, and Sub-task B focused on multi-lingual posts. Therefore, it gave us an opportunity to address the multi-lingual issues associated with social media posts. In order to address this, we applied pre-trained transformer based neural network (BERT) models, which are publicly available in multiple languages, and the model supports fine-tuning for specific tasks. In addition to this, its multi-lingual feature allows for us to analyse sentiment for the comments which have multiple language words, and sentences, for example HINGLISH (which is combination of English, and Hindi, commonly used as an expression medium) in Indian social media. We participated for both sub-tasks, and all the three languages. We have tried several neural networks architecture but found that our BERT model is performing better than other models. We achieved 3 rank for English Sub-task A.
The organization of the paper as follows: the previous works in these scenarios are discussed in Section 2, the data description is explained in Section 3. The models description and their training are presented in Section 4 and Section 5, respectively. The results of various experiments are explained in Section 6 and the current paper summarizes the most important findings in Section 7.
Lot of researchers and practitioners from industry and academia have been attracted towards the problem of automatic identification of hate speech, cyberbullying, cyber-aggression and ofensive languages.
The dynamics of the definition of cyberbullying, cyber-aggression, hate speech and its
undoubted potential for social impact are addressed by authors [
The data supplied by the organizing team, are the posts collected from Twitter. The posts are
in the following three languages: English, German, and Hindi. The competition has two
subtasks for each language, i.e. Sub-task A, and Sub-task B. Sub-task A consists of labelling with
Hate and Ofensive (HOF) if the content of the post contains any hate speech, otherwise the
label should be Not Hate-Ofensive (NOT). Sub-task B has a more fine-grained data which also
pointed to discriminate between Hate, Profane, and Ofensive posts by labelling with HATE,
PRFN, and OFFN. We utilized the testing dataset released by the organizers. The details of the
collection, and labelling of datasets are discussed in paper [
We used BERT as our pre-trained language model because of its success in recent as well as availability in multiple languages. We finely tuned the already trained model on our training dataset, and generated the submission model.
We used BERT, specifically BERT BASE UNCASED, which is a transformers model pretrained on a large corpus of data in a supervised fashion. It has 756 encoding length with 12 transformer layers, 12 attention heads, and 110 million parameters. Sentences are tokenized by BertTokenizerFast. Before padding up to 256 sequence length, Tokens are converted into Sequences. Two tokens were added to each input text (CLS, and SEP) marking the beginning, and the end of the sequence. These sequences are passed into the model. After that, we used BERT BASE UNCASED model from TFBertModel. It contains vocabulary, and pre-trained model. We used optimizer as Adam with Categorical Cross-entropy for Sub-task B, and Binary Cross-entropy for Sub-task A as a loss function.
We have trained the BERT model on English, German, and Hindi datasets of HASOC 2020 for Sub-task A. In addition to HASOC 2020 datasets, we also used HASOC 2019 datasets to train our models for Sub-task B. Training involves batch of size 64 with 20 epochs (on an average). Adam is used with the learning rate of 5e-05. For training, we have utilized 80% samples from training data provided by the organizing team, and rest 20% was used for validation. After training the model, we evaluated our model with the provided testing data. We experimented with diferent number of epochs, and analysed the training model with each epoch. It was observed that on an average after 20 epochs, the training accuracy did not improve rather it was decreasing, and hence the accuracy started to oscillate with its highest being at 20 epochs. The ifnal leaderboard is calculated with approximately 15% of the private test data by the organizing teams. F1 Macro average was used which was for the deciding parameter that can be observed from the classification report.
In this competition, teams are ranked on performance in terms of F1 Macro average on final leaderboard results. The Table 2 presents our results on testing samples provided by the organizing team in terms of Weighted F1-Score and F1 Macro average as performance metrics. The Table 3 presents the final leaderboard results for all the languages with their respective sub-tasks. According to the oficial results in Sub-task A, the best Marco F1 score of our model are 0.5078, 0.4768, and 0.4561 respectively, for English, German, and Hindi languages ranked 3 place for English Sun-task A.
In this paper, we presented our approach based on fine-tuning monolingual, and multi-lingual transformer networks (BERT model) to classify Twitter posts in three Indo-European languages: English, German, and Hindi, for hate-speech, and ofensive content identification. Using transfer learning with the pre-trained BERT model on English Sub-task A, we achieved rank 3 on leaderboard private test data. Overview of the HASOC track at FIRE 2020: Hate Speech and Ofensive Content Identification in Indo-European Languages), in: Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, CEUR, 2020. [12] T. Mandl, S. Modha, P. Majumder, D. Patel, M. Dave, C. Mandlia, A. Patel, Overview of the hasoc track at fire 2019: Hate speech and ofensive content identification in indo-european languages, in: Proceedings of the 11th Forum for Information Retrieval Evaluation, 2019, pp. 14–17.