Online hatred is a growing concern on many social media platforms. To address this issue, diferent social media platforms have introduced moderation policies for such content. They also employ moderators who can check the posts violating moderation policies and take appropriate action. Academicians in the abusive language research domain also perform various studies to detect such content better. Although there is extensive research in abusive language detection in English, there is a lacuna in abusive language detection in low resource languages like Hindi, Urdu etc. In this FIRE 2021 shared task - “HASOC - Abusive and Threatening language detection in Urdu” the organisers propose an abusive language detection dataset in Urdu along with threatening language detection.
A
In recent years, social media has become an important means of communication. It has allowed
people to share their ideas and opinions instantly. Unfortunately, abusive, threatening,
aggressive, etc., languages continue to be used online 1 and endanger the well-being of millions of
people. In some cases, it has been reported that online incidents have already turned into crimes
against minorities [
To this end, several studies have tried to come up with methods to automatically detect
abusive content [
Recently, diferent shared task like HASOC 2019 [ 12] have been launched to identify Hate and ofensive content in languages other than English. There is also one sub-task in HASOC 20209 which aimed to identify ofensive post in code-mixed dataset. Extending that task further, the organisers of this shared task [13] have build two datasets of 3400, 9950 posts to detect abusive and threatening language in Urdu. Twitter’s definition has been followed to describe , whether a post is abusive/non-abusive 10, and threading/non-threatening 11.
In this paper we explore several machine learning models for classification in these sub-tasks. We find that for both the sub-task A & B, our Transformer based model 12 ranked 1 at the shared task. The rest of the paper is organized as follows: In Section 2 we cover some of the related works; We discuss the Dataset Description in Section 3; In Section 4 we present the System Description. Then we discuss about the results, some observation in section 5 and 6. Finally we finish our paper with Conclusion.
Detection of abusive language in natural languages has recently gained significant attraction among the research community. However, the space of abusive language is vast and has its own subtleties.
In 2017, Waseem et al. [
Davidson et al. [
In 2018, Pitsilis et al. [14], tried recurrent neural networks (RNNs) to identify ofensive language in English and found that it was quite efective in this task by achieving 0.9320 F1score using ensemble methods. RNN’s remember the output of each step the model conducts. This approach can capture linguistic context within a text which is critical to detection. While RNN’s have been projected to work well with language models, other neural network models, such as the CNN. LSTM have had notable success in detecting hate/ofensive speech [ 15, 16].
Recently, Transformer based [17] language models such as, BERT, m-BERT [18] are becoming
quite popular in several downstream task, such as classification, span detection etc. Previously,
it has been identified that Transformer based models have been outperformed several deep
learning models [
The shared tasks [19] present in this competition are divided into two parts. Where in one part participants have to focus on detecting Abusive language using twitter tweets in Urdu language (Subtask A)13 and in other part mainly focusing on detecting Threatening language using Twitter tweets in Urdu language (Subtask B)14. The presented data has been collected and annotated from Natural Language and Text Processing Laboratory15 at Center for Computing Research16 of Instituto Politécnico Nacional, Mexico.
This task is a binary classification task in which tweets need to be classified into two classes, namely: Abusive and Non-Abusive. Training data has total 2400 instances and Testing data has total 1100 instances, which is already annotated as Abusive and Non-Abusive. The mentioned dataset is balanced between Abusive and Non-Abusive classes. The dataset description for Urdu Abusive Task has been represented in Table 1.
13https://ods.ai/competitions/Urdu-hack-soc2021 14https://ods.ai/competitions/Urdu-hack-soc2021-threat 15https://nlp.cic.ipn.mx/ 16https://www.cic.ipn.mx/index.php/en/
This is also a binary classification task of identifying/detecting Threatening language in Urdu. Training data is having 6000 instances and Testing data has 3950 instances which is annotated as Threatening and Non-Threatening. As per the data statistics it not properly balanced data (ratio is 1:5). We have presented the dataset distribution in Table 2.
In this section, we will explain the details regarding the features and machine learning models used for the task. We have attempted several models for abusive and threatening detection task. In both of the cases as a baseline we tried XGBoost[20] and LightGBM[21] with pre-trained Urdu laser embedding. Later we have tried Transformer-based pre-trained architecture of multilingual BERT[18]. The beauty of the mBERT is it is pretrained in unsupervised manner on multilingual corpus. Besides we have used another m-BERT based model which is previously fine-tuned on Arabic hate speech date set. The model has been referred as “Hate-speech-CNERG/dehatebertmono-arabic’[22] model. The motivation of using the following model in Arabic language because it is origin of Urdu 17, so further fine-tuning the model with the Urdu dataset may yield better performance.
Both Subtask A and B is a binary classification problem. We fine tuned BERT Transformer and classifier layer on top and used binary target labels for individual classes. We have used multilingual BERT (mBERT) and dehatebert-mono-arabic for abusive language classification. Binary 17 https://en.wikipedia.org/wiki/Urdu
mBERT dehatebert-mono-arabic
0.76072
The main challenge in any classification problem is the imbalance in data. This imbalance in data may create a bias towards the most present labels, which lead to a decrease in classification performance. From Table 1 and Table 2 we observe that, the dataset for abusive tweet detection is almost balanced, but the threatening detection dataset is highly imbalanced. Lots of research has been done in this domain to make the data balance [23]. Oversampling and undersampling are very much popular data balancing methods, but they have coherent disadvantages also. Another method of handling an imbalanced dataset is by using class weight so that the instances which are less in the dataset get more importance while calculating the average loss in the model and we follow this method for threatening tweet detection.
For the classical model such as XGBoost and LGBM, we have used the default setting and trained in on the provided dataset for both tasks. For Transformer-based models, we have run the models for 10 epochs with Adam optimizer[24] and initial learning rate of 2e-5. For the abusive tweet detection, we divided the training data points into 85% and 15% split and used the 15% as a validation set. We predict the test set for the best validation performance. For threat detection, we have used the full dataset for training due to data imbalance and predicted the test set after the completion of 10 epochs.
The results of all the models for both sub-tasks are presented in Table 3 ( We have shown the classification result of the private leaderboard). By using mBert based ”dehatebert-mono-arabic” model and further fine-tuning it with the Urdu datasets, our model got the first position for both sub-task A & B, with F1 score of 0.8806 for abusive tweet detection and 0.5457 for threatening tweet detection.
We found that our model was able to achieve good performance for abusive tweet detection (sub-task A), but was not able to perform well in threatening tweet identification (sub-task B). One of the issues with the dataset of sub-task B is, it is highly imbalanced, and because of that the models are not able to learn the diferent dimensionality of threatening posts.
In this shared task, we have experimented with classical machine learning model as well as Transformer based model. We have used XGboost and LGBM classifier with pre-trained laser embedding. We finetuned both mBERT and ”dehatebert-mono-arabic” model and observed insteading of finetuning a model from scratch, it is better to use existing finetune model in the same domain. Our model occupied the top position for both sub-task A & B. European Language Resource Association, Marseille, France, 2020, pp. 40–47. URL: https: //aclanthology.org/2020.osact-1.6. [9] M. Amjad, N. Ashraf, A. Zhila, G. Sidorov, A. Zubiaga, A. Gelbukh, Threatening language detection and target identification in urdu tweets, IEEE Access 9 (2021) 128302–128313. doi:1 0 . 1 1 0 9 / A C C E S S . 2 0 2 1 . 3 1 1 2 5 0 0 . [10] M. Das, B. Mathew, P. Saha, P. Goyal, A. Mukherjee, Hate speech in online social media,
ACM SIGWEB Newsletter (2020) 1–8. doi:1 0 . 1 1 4 5 / 3 4 2 7 4 7 8 . 3 4 2 7 4 8 2 . [11] M. Amjad, N. Ashraf, G. Sidorov, A. Zhila, L. Chanona-Hernandez, A. Gelbukh, Automatic abusive language detection in urdu tweets, ACTA POLYTECHNICA HUNGARICA (2021). [12] T. Mandl, S. Modha, P. Majumder, D. Patel, M. Dave, C. Mandlia, A. Patel, Overview of the hasoc track at fire 2019: Hate speech and ofensive content identification in indo-european languages, in: Proceedings of the 11th forum for information retrieval evaluation, 2019, pp. 14–17. [13] M. Amjad, A. Zhila, G. Sidorov, A. Labunets, S. Butt, H. I. Amjad, O. Vitman, A. Gelbukh, Urduthreat@ fire2021: Shared track on abusive threat identification in urdu, in: FIRE 2021: Forum for Information Retrieval Evaluation, December 13-17, 2021, India, ACM, 2021. [14] G. K. Pitsilis, H. Ramampiaro, H. Langseth, Detecting ofensive language in tweets using deep learning, ArXiv abs/1801.04433 (2018). [15] Y. Goldberg, A primer on neural network models for natural language processing, Journal of Artificial Intelligence Research 57 (2015). doi: 1 0 . 1 6 1 3 / j a i r . 4 9 9 2 . [16] G. L. De la Pena Sarracén, R. G. Pons, C. E. M. Cuza, P. Rosso, Hate speech detection using attention-based lstm, EVALITA Evaluation of NLP and Speech Tools for Italian 12 (2018) 235. [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in neural information processing systems, 2017, pp. 5998–6008. [18] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171–4186. [19] M. Amjad, A. Zhila, G. Sidorov, A. Labunets, S. Butt, H. I. Amjad, O. Vitman, A. Gelbukh, Overview of the shared task on threatening and abusive detection in urdu at fire 2021, in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021.
URL: http://ceur-ws.org/. [20] T. Chen, C. Guestrin, Xgboost, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016). URL: http://dx.doi.org/10. 1145/2939672.2939785. doi:1 0 . 1 1 4 5 / 2 9 3 9 6 7 2 . 2 9 3 9 7 8 5 . [21] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, T.-Y. Liu, Lightgbm: A highly eficient gradient boosting decision tree, in: NIPS, 2017. [22] S. S. Aluru, B. Mathew, P. Saha, A. Mukherjee, Deep learning models for multilingual hate speech detection, 2020. a r X i v : 2 0 0 4 . 0 6 4 6 5 . [23] G. E. Batista, R. C. Prati, M. C. Monard, A study of the behavior of several methods for balancing machine learning training data, ACM SIGKDD explorations newsletter 6 (2004) 20–29. doi:1 0 . 1 1 4 5 / 1 0 0 7 7 3 0 . 1 0 0 7 7 3 5 . [24] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, 2019. a r X i v : 1 7 1 1 . 0 5 1 0 1 .