With the advent of social media, anti-social and abusive behavior has become a prominent occurrence online. Undesirable psychological effects of abuse on individuals make it an important societal problem of our time [1]. Pew Research Centre, in its latest report on online harassment [2], revealed that 40% of adults in the United States had experienced abusive behavior online, of which 18% have faced severe forms of harassment, e.g., that of sexual nature. These statistics stress the need for automated detection and moderation systems. Hence, a new research effort on abusive language detection has sprung up in NLP in recent years.
Online communities, social media enterprises, and technology companies are investing heavily and encouraging research in this area by organizing tasks and workshops. One such community is FIRE, which has been actively organizing the HASOC tasks since 2019 [3]. The Urdu language has more than 230 million speakers worldwide with vast social networks and digital media representation. This paper 1 will contain details regarding the subtask A -Abusive language using Twitter tweets in Urdu language of Abusive and Threatening Language Detection Task in Urdu. This is a binary classification task in which participating systems are required to classify tweets into two classes, namely: Abusive and Non-Abusive.
• Abusive This Twitter post contains any abusive content.
• Non-Abusive This Twitter post does not contain any abusive or profane content.
Techniques for abuse detection have gone through several stages of development, starting with extensive manual feature engineering and then turning to deep learning. Early approaches experimented with feature extraction from speech text like a bag of words or n-grams [4], lexical and linguistic features [5] and, and user-specific features, such as age [6]. With the advent of deep learning, the trend shifted, with great work focusing on neural architectures for abuse detection. Initially witnessing an extensive use of CNNs [7] and then moving on to LSTMs [8]. Most recently, the use of pre-trained transformer-based architectures such as BERT [9] has given state-of-the-art results. Amjad et al. [10] describe the first shared task for fake news detection in the Urdu language. The dataset consists of news articles from five domains with 900 annotated articles for the training and 400 annotated news articles for the testing part. In this shared task, nine teams submitted their results, and the best performing system achieved an F-score value of 0.90. Amjad et al. in [3] introduced a new dataset for classifying threatening and non-threatening language in the Urdu language. The recommended dataset comprises 3,564 tweets manually annotated by human specialists. They applied different models based on Machine and Deep Learning-based techniques. They compared the three forms of text representations. Their research reveals that an MLP classifier with the combination of word n-gram features outperformed other classifiers. [11], [12] has also performed well in the Abusive language detection in the Urdu language.
The datasets for the tasks are provided by the organizers of HASOC '21 2 and the code is available in the github repository 3 The data consists of tweets in Urdu annotated for a binary classification task: Abusive, Non-Abusive. Abusive -This Twitter post contains any abusive content. Non-Abusive -This Twitter post does not contain any abusive or profane content. Table 1 lists the statistics of the dataset. According to Twitter, the definition describes abusive comments toward individuals or groups to harass, intimidate, or silence someone else's voice. The dataset was collected and annotated in Natural Language and Text Processing laboratory at the Center of Computing Research of Instituto Politécnico Nacional, Mexico, by Ph.D. candidate Maaz Amjad, a native Urdu-speaker 4 .
The paper describes various approaches and draws out a comparison between them. The first approach extracts N-grams features from the tweets, which are weighted according to TF-IDF values. Then, models using machine learning algorithms are trained upon these features.
The primary evaluation metric for evaluating the applied machine-Learning and Transformer based models is the F1 score, and ROC AUC is the secondary evaluation metric used.
Here, we use three machine learning algorithms: Logistic Regression, Support Vector Classifier, and Random Forest Classifier available in the 'scikit-learn' package. While training, a 5-fold grid search is performed on the entire train dataset to find the best set of hyperparameters.
For initial experimentation, pre-processing is carried out in Normalization8 , but results without the Normalization are significantly better. Hyper-parameter tuning for the models is carried out using RAY TUNE. Population-Based Training scheduler is used for all three models, with train batch size in (2,4,8,16). The learning rate was set to a uniform log distribution between 5e-6 and 5e-5. Table 1 and 3 lists the Hyperparameter description. For the multilingual Bert and Urduhack model, train epochs are selected between 2,3,4. Given the large size of XML-Roberta, train epochs are fixed at 2. Finally, soft voting is carried out, taking the average of each model's
The following results are obtained on the test set made public at the end of the competition and described in Table 2. 9 All models used the best parameters obtained through a 5-fold grid search. Submission for the competition has been made using the Urduhack model with Normalization and results are listed in Table 4. Further soft voting is carried out using the three transformer-based models without Normalized the tweets, using the following parameters listed in Table 5. The following results are obtained on the entire test set listed in Table 6.
This paper started with experimentation using classical machine learning models such as Logistic Regression, SVM, and Random Forest Classifier. We then moved on to leveraging recent advances in large-scale Transformer-based pre-trained language models. The larger pre-trained models still outperform the classical models while performing well. Pre-processing performed using the UrduHack library did not necessarily yield better results, which could lead to why punctuations and diacritics add information valuable to Abuse detection. Our model is getting 0.9340 accuracies on the public data with normalization of the tweets and 0.9360 without normalization. For future work, we can try out different multilingual transformer-based models to get a more robust model.
The implementation of different pre-trained BERT-models are available at • Huggingface.