With social media becoming an important medium for spreading information in people's daily lives, the detection of threatening speech has become increasingly necessary. Recently, FIRE 2022 has released “EmoThreat: Emotions & Threat Detection in Urdu” task which include two subtasks. We have chosen to participate in “Task B: Threatening Language Detection Task in Urdu”. Several BERT models based on transformers were used to accomplish this task. As a result, we came 4th in subtask 1 and 5th in subtask
FIRE 2022
Today basically anyone can communicate and share information, opportunities and ideas in
real time on Internet social media, opportunities and ideas in real time. Social media has
revolutionized how humans interact, providing them with unprecedented opportunities to
satisfy their social needs [
Recently, FIRE 2022 has released “EmoThreat: Emotions & Threat Detection in Urdu” task
which include two subtasks [
∗Corresponding author
The BERT model is now widely used in solving text classification problems. A taxonomy that classifies the state-of-the-art of tweet-based bot detection techniques was provided in [ 7], they also describe the shallow and deep learning techniques for tweet-based bot detection, along with their performance results. [8] explored several machine learning models such as XGboost, LGBM, m-BERT based models for abusive and threatening content detection in Urdu based on the shared task. In [9], a machine learning model is developed for detecting threats from Twitter tweets, accordingly, the Naïve Bayes classifier and flask microweb framework were used to build the model by using the python programming language. [10] approached the task in HASOC 2021 using the Transformers-based Roberta model which has shown remarkable results in NLP tasks such as sentence classification. [ 11] introduce and release a new dataset for threatening language detection in Urdu tweets to further research in this language. Previous study [12] presents guidelines for data annotation process and developed two linguistic resources: (i) Annotated corpus in Roman Urdu Language for cyberaggression and ofensive language detection, the process of data annotation involved bilingual annotators instead of crowdsourcing, (ii) Processing textual information for NLP tasks involves Stop-word elimination as a sub phase, stop words carry least semantic information and increase feature space as compared to the other tokens and index terms in corpora. [13] create a well-balanced dataset by adding a neutral class to a benchmark dataset consisting of four emotions: fear, sadness, joy, and anger, on this extended dataset, we investigate the use of Support Vector Machine (SVM) and Bidirectional Encoder Representations from Transformers (BERT) for emotion recognition.
4.1. Subtask 1 The organizers provided a total of 3564 tweets in Urdu, of which 1782 were marked as threatening tweets, 1782 were marked as non-threatening tweets, while 1341 of the 1782 threatening tweets were marked as threatening tweets for groups and the remaining 441 tweets were marked as threatening tweets for individuals [14] [15] [16].
For subtask 1 of task B, we tried two diferent BERT-based pre-training models from the huggingface community [17]. One is called “Hate-speech-CNERG/dehatebert-mono-arabic” [18], a model used in the HASOC 2021 task of FIRE@2021, and the other model is called “Hatespeech-CNERG/urdu-abusive-MuRIL” [19], two both models are from the same open source model contributor.
For the “Hate-speech-CNERG/dehatebert-mono-arabic” model, we tried the combination of batch size = 24, epochs = 10, while for the “Hate-speech-CNERG/urdu-abusive-MuRIL” model, we tried for the “Hate-speech-CNERG/urdu-abusive-MuRIL” model, we tried the combination of batch size=24, epochs = 10 and batch size=24, epochs = 15, and we can see that the results is improving in Table 1. 4.2. Subtask 2
Run Run1 Run2 Run3
F1 Considering that if the labeled test set is directly classified into two categories, threatening tweets and non-threatening tweets, based on subtask 1, subtask 2 can again be considered as a binary classification task if the labeled threatening tweets are then classified to target individuals or to target groups.
We simplified subtask 2 to a binary classification task based on the prediction results we obtained in subtask 1, and fed it into the “Hate-speech-CNERG/dehatebert-mono-arabic” model as a simple experiment. As we did in subtask 1, diferent combinations of batch size and epochs were tried one after another, and the results were obtained according to the feedback as shown in Table 2.
We ended up with F1 scores of 0.681, Accuracy scores of 0.722, and ROC-AUC scores of 0.679 for subtask 1, and F1 scores of 0.41, Accuracy scores of 0.666, and ROC-AUC scores of 0.588 for subtask 2. As shown in Table 3.
Subtask Subtask1 Subtask2
F1 0.681 0.41
Accuracy
ROC-AUC 0.722 0.666 0.679 0.588
We were ranked 4th in subtask 1 and a 5th in subtask 2.
Regarding subtask 1, at first we were concerned about the occurrence of overfitting, so we chose smaller epochs. However, we can see that the accuracy improves significantly only as the epochs increase, indicating that there is still much room for parameter tuning. Regarding subtask 2, we basically just considered it as a dichotomous classification task after simply trying it out. Training it with a more suitable model may give better results.
Transformer-based BERT models have been used very widely and efectively in NLP tasks. This helps one to reduce much of the workload, and instead of building and tuning a model from scratch, we can choose to directly call an of-the-shelf model that has been proven to be efective in similar problems. We simply called the pre-trained BERT model from the huggingface community and completed the task.
Thanks to the providers of the open source model, to the huggingface community and to the organisers of the task, without whom we would not have been able to perform the task with ease. [6] M. Amjad, A. Zhila, G. Sidorov, A. Labunets, S. Butt, H. I. Amjad, O. Vitman, A. Gelbukh, Overview of the shared task on threatening and abusive detection in Urdu at FIRE 2021, in: FIRE (Working Notes), CEUR Workshop Proceedings, 2021. [7] A. Derhab, R. Alawwad, K. Dehwah, N. Tariq, F. A. Khan, J. Al-Muhtadi, Tweet-based bot detection using big data analytics, IEEE Access 9 (2021) 65988–66005. doi:10.1109/ ACCESS.2021.3074953. [8] M. Amjad, A. Zhila, G. Sidorov, A. Labunets, S. Butt, H. I. Amjad, O. Vitman, A. Gelbukh, UrduThreat@ FIRE2021: Shared track on abusive threat identification in Urdu, in: Forum for Information Retrieval Evaluation, 2021, pp. 9–11. [9] S. H. Sadi, M. R. H. Pk, A. M. Zeki, Threat detector for social media using text analysis,
International Journal on Perceptive and Cognitive Computing 7 (2021) 113–117. [10] S. Kalraa, M. Agrawala, Y. Sharmaa, Detection of threat records by analyzing the tweets in urdu language exploring deep learning transformer-based models (2021). [11] M. Amjad, N. Ashraf, A. Zhila, G. Sidorov, A. Zubiaga, A. Gelbukh, Threatening language detection and target identification in urdu tweets, IEEE Access 9 (2021) 128302–128313. [12] A. Dewani, M. A. Memon, S. Bhatti, Development of computational linguistic resources for automated detection of textual cyberbullying threats in roman urdu language, 3 c TIC: cuadernos de desarrollo aplicados a las TIC 10 (2021) 101–121. [13] I.-A. Albu, S. Spînu, Emotion detection from tweets using a bert and svm ensemble model, arXiv preprint arXiv:2208.04547 (2022). [14] M. Amjad, N. Ashraf, A. Zhila, G. Sidorov, A. Zubiaga, A. Gelbukh, Threatening language detection and target identification in urdu tweets, IEEE Access 9 (2021) 128302–128313. doi:10.1109/ACCESS.2021.3112500. [15] N. Ashraf, A. Rafiq, S. Butt, H. M. F. Shehzad, G. Sidorov, A. Gelbukh, Youtube based religious hate speech and extremism detection dataset with machine learning baselines, Journal of Intelligent & Fuzzy Systems (2022) 1–9. [16] N. Ashraf, R. Mustafa, G. Sidorov, A. Gelbukh, Individual vs. group violent threats classiifcation in online discussions, in: Companion Proceedings of the Web Conference 2020, 2020, pp. 629–633. [17] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al., Huggingface’s transformers: State-of-the-art natural language processing, arXiv preprint arXiv:1910.03771 (2019). [18] S. S. Aluru, B. Mathew, P. Saha, A. Mukherjee, Deep learning models for multilingual hate speech detection, arXiv preprint arXiv:2004.06465 (2020). [19] M. Das, S. Banerjee, A. Mukherjee, Data bootstrapping approaches to improve low resource abusive language detection for indic languages, arXiv preprint arXiv:2204.12543 (2022).