As humans, we express sadness, anger, happiness, frustration, bullying, etc., in both physical and virtual worlds. In the virtual world, i.e., social media, we use textual ways to express ourselves. Due to the lack of ofensive and threatening language detection mechanisms aggressive behavior in social media is not always followed by an immediate consequence. But the impact of these posts on the victim can cause prolonged mental illness and instigate fear for social media platforms. This paper aims to identify threatening posts using deep learning transformer-based models such as Roberta. The Urdu tweet dataset used in this study has been provided by HASOC-2021 which aims to identify Hate speech and ofensive remarks without human assistance. We submitted our model in its subtask B of the 4th subtrack(Abusive and Threatening language detection in Urdu), secured 2nd position on the public leaderboard, and obtained Weighted f1 of 0.5346 and ROC AUC of 0. 8199.
With the expansion of the Internet, Social media has become a nursery of toxic and unethical content over the years. It is flooding with manipulative and hateful information leading to disharmony and violence in society. A lot of research has gone into identifying threatening and ofensive language by many social media giants. Detecting hate speech using well-defined algorithms will go a long way in filtering out inappropriate language from powerful platforms used for general discourse and prevent the psychological and physical consequences of these abusive comments on its victims. HASOC 2021 is searching for technology to identify Hate speech and ofensive remarks without human assistance. The challenge is divided into four subtracks. We participated in the 4th subtrack, which aimed at identifying Abusive and Threatening language in Urdu1 and code is available in the github repository. 2 https://www.bits-pilani.ac.in/pilani/yash/profile (Y. Sharma) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings 2https://github.com/Kalra-Sakshi/HASOC-Subtask-B-files.git
This subtrack is further divided into two subtasks. Subtask A focuses on detecting Abusive language and subtask B on threatening language using Twitter tweets in Urdu. This paper aims to describe the methodology used for subtask B. There is a subtle diference between abusive and threatening language. The current definition of abusive speech is anything that directly attacks people based on what is known as their “protected characteristics” — race, ethnicity, national origin, religious afiliation, sexual orientation, sex, gender, gender identity, or severe disability or disease. For example. “[Religious Group] are viruses; they are making this country sick.” On the other hand, a threat is a communication intended to inflict harm or loss on another person, such as “Let’s kill these cops cuz they don’t do us no good / pullin’ out your Glock out ’cause I live in the ‘hood” and “I’ma jam this rusty knife all in his guts and chop his feet.”
We approached the task using the Transformers-based Roberta [
A lot of research has already been done in identifying hate speech on social media platforms.
Waseem and Hovy [
Kamble et al. [
Chen et al. used NLP methods to propose sentence lexical and syntactic features for ofensive
speech detection [
The subtask B in the HASOC challenge for Urdu is a binary classification task. We need to categorize the sentences in the Urdu tweet dataset into Threatening (THR) and Non-Threatening (NON-THR) categories. There are a total of 6000 tweets in the dataset. The dataset is imbalanced, with 4929 tweets categorized as Non-Threatening (NON-THR) and 1071 as Threatening (THR). The dataset distribution of the task can be seen in fig 1.
One approach to address the problem of data imbalance in the training dataset is to resample
it randomly. There are two methods to resample the dataset randomly: undersampling, i.e.,
deleting examples from the majority class, and oversampling, i.e., duplicating samples from
the minority class[
Transformers-based models are state of the art in various NLP tasks such as Machine Translation, Question Answering Systems, Rumour Detection, Fake News Detection, etc. They perform better than previous methods as they are bidirectionally trained and have a deeper understanding of the language. One of the most favorable features of these models is that they could be pretrained on a large corpus of raw texts and later fine-tuned on downstream tasks with lesser train instances. We used a pre-trained Roberta model from the HuggingFace library, trained on Urdu news corpus in an unsupervised manner, enabling the model to develop contextual embedding representations of diferent words in the language. These representations can be used to initialize the weights of our model. We added a classification head that is randomly initialized for fine-tuning the model for sentence classification. We fine-tuned this model on our task (detecting Threatening language using Twitter tweets), transferring the knowledge of the pre-trained model to it (which is why doing this is called transfer learning). Figure 2 shows the flow of the proposed architecture. Figure 3 shows the fine-tuning of the Roberta model using the given training dataset and figure 4 shows the classification of a tweet from the test dataset using fine-tuned Roberta model.
For the proposed task, various experiments have been done on Urdu language. We used the “Urduhack/roberta-Urdu-small”model from the HuggingFace library and its corresponding tokenizer to maintain consistency with what was used during the model pre-training. The model was fine-tuned using Adam optimizer with weight decay. The maximum sequence length in each batch sent into the model was set to 256. Label smoothing cross-entropy was used for the task—a maximum learning rate of 5e-05was used for training.
The learning rates were warmed up from 0 to their maximum values and then decayed from this set maximum using a linear schedule.Table 1 lists the various hyperparameters used and their description. Table 2 lists the model performance on the local validation set. Table 3 lists the performance of the model on the leaderboard. The organizers used weighted f1 and ROC AUC scores to evaluate the models and rank us on the leader board.
In this paper, we presented our solution to subtask B of subtrack “HASOC - Abusive and Threatening language detection in Urdu” of HASOC 2021. We fine-tuned an Urdu Roberta model on the provided training dataset. Before training, we did some pre-processing to tackle data imbalance and normalizing Urdu text. We were placed second on the public leaderboard for the subtask. For future work, we can try with the diferent transformer-based architecture to get more accurate results. We can take this hate speech detection task as a multimodal aspect by targetting both images and text and getting the visual elements for better feature extraction. We will try to extend the model by adding the multilingual aspects.
• Huggingface.