As social media platforms grow in popularity and importance, the consequences of their misuse become more severe. Numerous posts containing abusive language directed at specific users worsen users' experiences on such platforms. In this paper, we look at the task of detecting Abuse in the Urdu Language. We experiment with diferent machine learning algorithms and Transformer based models to achieve the best results on this one-of-a-kind task of Abusive language detection in Urdu. We got accuracy equal to 0.93607 on the test dataset using the soft voting technique with the help of 3 transformer based-techniques such as Urduhack, BERT, and XLM-Roberta.
With the advent of social media, anti-social and abusive behavior has become a prominent
occurrence online. Undesirable psychological efects of abuse on individuals make it an important
societal problem of our time [
Online communities, social media enterprises, and technology companies are investing
heavily and encouraging research in this area by organizing tasks and workshops. One such
community is FIRE, which has been actively organizing the HASOC tasks since 2019 [
© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings Task in Urdu. This is a binary classification task in which participating systems are required to classify tweets into two classes, namely: Abusive and Non-Abusive.
• Abusive This Twitter post contains any abusive content.
• Non-Abusive This Twitter post does not contain any abusive or profane content.
Techniques for abuse detection have gone through several stages of development, starting with
extensive manual feature engineering and then turning to deep learning. Early approaches
experimented with feature extraction from speech text like a bag of words or n-grams [
The datasets for the tasks are provided by the organizers of HASOC ’212 and the code is available in the github repository 3 The data consists of tweets in Urdu annotated for a binary classification task: Abusive, Non-Abusive. Abusive - This Twitter post contains any abusive content. Non-Abusive - This Twitter post does not contain any abusive or profane content. Table 1 lists the statistics of the dataset. According to Twitter, the definition describes abusive comments toward individuals or groups to harass, intimidate, or silence someone else’s voice. The dataset was collected and annotated in Natural Language and Text Processing laboratory at the Center of Computing Research of Instituto Politécnico Nacional, Mexico, by Ph.D. candidate Maaz Amjad, a native Urdu-speaker4.
The paper describes various approaches and draws out a comparison between them. The first approach extracts N-grams features from the tweets, which are weighted according to TF-IDF values. Then, models using machine learning algorithms are trained upon these features. Fig 2 shows the proposed architecture using machine learning-based techniques such as Logistic Regression, Random Forest Classifier, and Support Vector Machine. The second approach uses pre-trained transformer-based models and their associated tokenizers. Three pre-trained models are used for this task. Urduhack Roberta-Urdu-small5: Trained on news data from Urdu news resources in Pakistan BERT (checkpoint : bert-base-multilingual-cased6) : Trained on 104 diferent languages XLM-Roberta 7: Trained on 2.5TB of newly created clean Common Crawl data in 100 languages. Fig 3 shows the proposed architecture using transformer-based techniques.
The primary evaluation metric for evaluating the applied machine-Learning and Transformer based models is the F1 score, and ROC AUC is the secondary evaluation metric used.
Here, we use three machine learning algorithms: Logistic Regression, Support Vector Classifier, and Random Forest Classifier available in the ’scikit-learn’ package. While training, a 5-fold grid search is performed on the entire train dataset to find the best set of hyperparameters.
For initial experimentation, pre-processing is carried out in Normalization8, but results without the Normalization are significantly better. Hyper-parameter tuning for the models is carried out using RAY TUNE. Population-Based Training scheduler is used for all three models, with train batch size in (2,4,8,16). The learning rate was set to a uniform log distribution between 5e-6 and 5e-5. Table 1 and 3 lists the Hyperparameter description. For the multilingual Bert and Urduhack model, train epochs are selected between 2,3,4. Given the large size of XML-Roberta, train epochs are fixed at 2. Finally, soft voting is carried out, taking the average of each model’s
output scores and predicting the target class.
The following results are obtained on the test set made public at the end of the competition and described in Table 2.9 All models used the best parameters obtained through a 5-fold grid search. Submission for the competition has been made using the Urduhack model with Normalization and results are listed in Table 4. Further soft voting is carried out using the three transformer-based models without Normalized the tweets, using the following parameters listed in Table 5. The following results are obtained on the entire test set listed in Table 6.
This paper started with experimentation using classical machine learning models such as Logistic Regression, SVM, and Random Forest Classifier. We then moved on to leveraging recent advances in large-scale Transformer-based pre-trained language models. The larger pre-trained models still outperform the classical models while performing well. Pre-processing performed using the UrduHack library did not necessarily yield better results, which could lead to why punctuations and diacritics add information valuable to Abuse detection. Our model is getting 0.9340 accuracies on the public data with normalization of the tweets and 0.9360 without normalization. For future work, we can try out diferent multilingual transformer-based models to get a more robust model.
• Huggingface.