The lack of clarity in rules imposed on discussions on social media and the lack of critical eyes on discussions in regional languages, unlike the languages with a greater audience like English, the vulgarity of most of the discussions go unnoticed. This demands an automated model to classify abusive and threatening messages to maintain decorum in social media platforms. Here in this work we have used classic models from Sklearn library to classify the data given in task HASOC 2021 - Abusive and Threatening language detection in Urdu. It has been observed that the best model for abusive classification was MLP with paraphrase multilang v1 encoding and for threatening language dataset, the best model observed was an nu-SVM.
The boom in social media platforms has led to a inevitable freedom of self-expression, especially among communities sharing the same native language and script.This rise of access of various native language communities to the means of self-expression via the Internet raised the need for detecting threatening and abusive language in their native scripts. The myriad of variations in the meaning for the same scripts in a diferent language removes the possibility for one model classifier for all languages. This creates a need for classifiers in each language,Roman Urdu, where Urdu is written in English script has seen a lot of input. Here we have created a model to classify the threatening and abusive nature of a sentence in native Urdu script. Independent models based on Multi layer perceptron, Logistic Regression, SVM, KNN were used to perform this classification task.
Identifying hate speech in social media is one of the most essential tasks to prevent spreading
hatred nowadays. Detection of such hate speech is tedious and the organisations hosting social
media platforms are taking steps to prevent the hate speech spreading through their platforms[
Variety of machine learning algorithms such as, Linear regression, SVM, Random Forest,
Naive Bayes and SGD classifier are applied on custom Roman Urdu [
Diferent models with n-gram pre-processing have been used for Ofensive classification in
Urdu sentences[
Propaganda Spotting in Online Urdu Language (ProSOUL) is designed to identify the sources
of propaganda in Urdu language [
Hate Speech Roman Urdu 2020 (HS-RU-20) corpus has been created in order to classify the
Roman Urdu tweets into three classes namely, Neutral - Hostile, Simple - Complex, and Ofensive
classes [
Classification
Train set
Test set Abusive Not Abusive Total
In [
Multilayer perceptron (MLP) classifier is a multilayer neural network. It uses back propagation to tune its weights and learns from the loss function in each iteration. MLP works for even linearly-unseparable problems. The model used for both the tasks contains 2 hidden layers with 256, 128 neurons respectively, and the neural weights were adjusted through 300 epochs with learning rate of 0.001. The default ReLU activation function was used for all layers.
Logistic Regression is a classical statistical analysis approach that relies on prior observation.
Logistic regression is usually used for classification sort of problems. It uses the sigmoid on the
given parameters to perform binary classification.[
1 1 + − (1)
nu-SVC a Support Vector Machine (SVM) classification algorithms was tested and trained with the given datasets.The best F1-score was achieved for a RBf kernel (where the classes marked on higher dimensions are governed by a Gaussian radial basis function) for regularization parameter value in the range 0.3 to 0.5. 4.4. KNN K Nearest Neighbors (KNN) classification is a clustering algorithm that labels a point with the class of majority of its neighbours. Our model analyses 50 neighbours for every node to predict the class of the node. Since the training set was not biased towards any one class, this algorithm gave reasonable results for both the the classification tasks. The KNN model requires high dimensional vectors as input which was derived using Tf-Idf vectorization method available in the sklearn library version 0.0, default version provided in google colab[11]. The parameters passed were n_neighbors=50, weights=’uniform’, algorithm=’auto’.The same model configuration was used for both the classification tasks.
4.5.1. Embeddings for MLP Two embeddings from the SentenceTransformers module , a Python framework for state-ofthe-art sentence, text and image embeddings were used to fetch corresponding embedding of the training set tweets[12]. The train data was lemmatized using the lemmatizer in urduhack, an NLP library for Urdu language. The lemmatized sentences are transformed to similar embedding using the below mentioned transformers: ’distiluse-base-multilingual-cased-v2’ and ’paraphrasexlm-r-multilingual-v1’ and trained separately . The results of training the model on encodings of the lemmatized versions of the sentences was better than just training on raw data. This might be because of an internal working of the pretrained models used for tokenizing the sentences. 4.5.2. Tf-Idf for KNN, LR, SVM Tf-Idf Vectorization was used to vectorize the sentences along with a character 10-gram with a max features of 50000. Character n-gram gave better results than word n-gram. It might be because of the complex morphology of Urdu that character n-gram works better extracting features from the samples.[13] While vectorizing, the sentences are converted to lower case in order to avoid the confusion caused by the case of words in learning the context of the sentence. Lemmatizing the words did not help in improving the accuracy. It might be because the root word of any Urdu word does not necessarily have the same meaning in the context or the derived word has a more precise meaning which helps the model understand the context of the sentence better. So only the Tf-Idf vector with 10-word gram was fed as input to these models. 4.6. Hardware specification and link to computation The training and testing of the models was done in google colab. A general purpose RAM size of 8GB was allotted with a 2.3GHz Intel Xenon CPU was used for training of the above models. Python note books associated with the abusive and Threatening tasks are given in the link. 1
The above algorithms with the aforementioned extracted features are trained with k fold cross validation and the best models and their parameters are tabulated below for each task.
The performance of the proposed system for abusive language detection using training data are tabulated in Table 3 and training data performance of threatening language detection is shown in Table 4.
The training performance shows that MLP-paraphrase and Nu-SVC have performed well for Abusive language detection and threatening language detection with 89% and 93.2% respectively. The MLP models trained on the 2 diferent encodings gave almost similar results on training but paraphrase-xlm-r-multilingual-v1 encoding worked better than the rest on the test data.
The performance of the proposed system for abusive language detection using test data are tabulated in Table 5, threatening language detection performance is tabulated in Table 6. 1https://github.com/ask-1710/Abusive-and-Threatening-Language-Detection-Task-in-Urdu
Model distiluseMLP KNN Nu-SVC LR
MLP-paraphrase Model MLP-distiluse KNN Nu-SVC kernel:RBF LR MLP-paraphrase
F1-Score 0.82 0.82 0.84 0.83 0.81 distiluseMLP KNN Nu-SVCrbf LR
MLP-paraphrase Model distiluseMLP KNN Nu-SVC, kernel:RBF LR MLP-paraphrase private F1 private ROC_AUC public F1 public ROC_AUC
From Table 5 and Table 6, it has been noted that for both abusive and threatening language detection task, paraphrase-xlm-r-multilingual-v1 embeddings with MLP models produces better results than other approaches.
Spreading hatred to the community on the basis of ethnicity, race, religion and gender is a menace to the society. Social media applications nowadays serve as a unintended medium for enabling transfer of such abusive and hatred messages. Techniques have to be developed to curtail such abusive messages from spreading. In this work, five machine leaning approaches have been explored to detect the abusive and threatening Language Task in Urdu Language.
Classical machine learning were able to come close to the MLP for both tasks in terms of F1scores. This work can be enhanced further by exploring the linguistic features of Urdu and also other deep learning approaches can be employed with fine tuned parameters for this task. [11] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al., Scikit-learn: Machine learning in python, Journal of machine learning research 12 (2011) 2825–2830. [12] N. Reimers, I. Gurevych, Making monolingual sentence embeddings multilingual using knowledge distillation, arXiv preprint arXiv:2004.09813 (2020). URL: http://arxiv.org/abs/ 2004.09813. [13] M. P. Akhter, Z. Jiangbin, I. R. Naqvi, M. Abdelmajeed, M. T. Sadiq, Automatic detection of ofensive language for urdu and roman urdu, IEEE Access 8 (2020) 91213–91226. doi:10.1109/ACCESS.2020.2994950.