The growing popularity of user-generated material on social media has increased the quantity of offensive language used online. The tendency of user-generated material on social media is growing, giving rise to offensive language on these platforms. The offensive language negatively impacts individuals and affects society as a whole, which is why it is a dire need of time to identify vulgar remarks in languages used online. 'Urdu' is one of the many languages used on the internet that faces the same issue. Manually labeling the text as abusive on social media platforms is unattainable due to the production of a large amount of daily content. Therefore, automation (machine learning) is used to create the solution. This study uses machine learning algorithms, namely logistic regression, bagging algorithms, decision trees, and artificial neural networks (ANN), to detect abuse in the text. The F1 score is used as the primary metric, along with accuracy, precision, recall, and AUC-ROC, to measure the performance. Based on the evaluation, the bagging and logistic regression perform equally with an 83% F1 score. However, logistic regression is better for this use case because it is computationally less expensive and requires less effort than the bagging classifier.
Historically, mass communication mediums were utilized under ethical and moral obligations
dictated by societal standards. In this digital age, the wide acceptance of social media continues to be
fueled by the prevalence of internet connection and mobile technologies, particularly smartphones and
tablets [
2021 Copyright for this paper by its authors.
Urdu is South Asia's resource-scarce language [
The rise in the use of social media has given birth to numerous problems, one of which is abusive
behavior on social media platforms. Each platform has its policies to detect such behavior. For example,
Twitter defines abusive behavior as an attempt to intimidate, harass, or silence someone's voice. Based
on this definition, Twitter can classify a tweet as abusive or non-abusive. Recently, the computational
linguistics community has focused on detecting abusive language and hate speech from various online
social media platforms, such as Twitter [
The dataset is adapted from HASOC abusive and threatening language detection in Urdu competition [20][21]. The dataset was gathered and labeled in the natural language and text processing laboratory at the center for computing research of Instituto Politécnico Nacional. The collector and annotator of the datasets are native Urdu speakers. The dataset used in the study contains two columns tweet and target with 2400 rows. Each row represents a tweet and its corresponding labels. There are two labels, 0 and 1, '0' represents a neutral text while '1' shows abusive text. Table 1 shows the distribution of the dataset on the label count. We have two labels in the dataset, abusive and nonabusive, and 1187 abusive tweets and 1213 non abusive tweets, which balances the dataset.
The problem we are tackling in this research is a binary classification problem, and the following algorithms were used: 1. Logistic Regression 2. Decision Tree 3. Bagging classifier 4. Neural Network 4.1.
Logistic regression is a powerful technique of simulated results of a binary classification. It is used
to allocate data to a discrete label, and being a classification method, it relies on probability [
The prediction function is represented by equation 2 ( ) =
1 1 − ℎ ( ) = ( ) =
1
1 + −
(1)
(2)
The value of has a unique significance; it indicates the likelihood that ℎ ( ) is 1 [
In this study, all the hyperparameters for the logistic regression are kept default because of getting the best results. 4.2.
The decision tree is a technique for successfully supervising inductive learning through the
generation of rules from data. Typically, one event might trigger two or more subsequent events, each
with a distinct outcome. The decision tree's structure is top-down as a result of this characteristic. This
structure resembles a flowchart. Every branch of a tree corresponds to a new decision result. The
children node on each node corresponds to the corresponding attribute test. This child node's ID,
generated from decision-making algorithms, is passed on [
The bagging technique (bootstrap aggregation) generates a collection of classifiers. A bootstrapped duplicate of the original dataset is created for each classifier by randomly selecting N instances with replacement. When a new input is desired to be classified, the number of classifiers that anticipate the instance's class value is counted for every label state. The number of votes and the state with the most votes are projected to win the instance. In this study, we are using bagging for bootstrapping different logistic regression classifiers. 4.4.
A neural network comprises a linked group of artificial neurons that analyses data in a connectionist
fashion. In general, an ANN is a self-organizing system that fine-tunes its organization in response to
external or internal data that flows through the network throughout the learning process. They are
typically used to describe complex connections between inputs and outputs or to deduce patterns from
data. ANN has been successfully utilized in a variety of applications. For instance, ANNs have been
successfully utilized in predictions, handwritten character recognition, and assessing home values [
steps: 1. Importing Dataset:
Cleaning Dataset: data is cleaned data.
3.
Extracting Features: To extract features, TF-IDF is used. The TF-IDF algorithm (term frequency-inverse document frequency) is an enhancement to the DF technique. It is a type of statistical technique used to determine the significance of a word within a file collection. The significance of a word is related to its frequency of occurrence in the text and inversely related to its frequency of occurrence in the total document collection.
, is the rate of input in document , as shown by equation (3) is represented by equation 4 = log
| | |{ : ∈ }| , = , (3) (4) (5) Here | | is the total number of files and indicates the total number of occurrences of a word.
, is represented by equation 5. 4. Splitting Dataset:
Using test train split for creating two sets of datasets—one for training and the other for testing. The train test ratio is 75:25. That is, 75% of the dataset is used for training and 20% for evaluating results. 5.
Apply Machine Learning Algorithm:
In this step, machine learning algorithms are applied to the training set to create a classification model. The algorithms used in this study are logistic regression, decision tree, bagging ensemble classifier, and ANN.
Evaluate Machine Learning Model:
The last step is where inference is performed on 25% of the dataset to determine how well the classifiers are performed. The classification metrics used in this study are accuracy, precision, recall, F1 score, and AUC ROC.
The evaluation parameter taken for this study is accuracy, precision, recall, F1 score, and AUC ROC. To decide the best-performing model, the key parameter used is the F1 score.
Accuracy is an evaluation parameter that shows the degree to which a classifier fits all classes. It is helpful since it treats all classes equally. It is measured as the fraction of correct predictions to all predictions. Equation 6 represents accuracy in mathematical terms.
in equation 6 represent true positive, is true negative, is false positive, and is false (6) (7) negative. 6.1.2. Precision 6.1.3. Recall
+ ( + + + )
( + )
The precision is determined as the fraction of Positive samples accurately classified to all positive cases classified. Precision is a measure that shows a model's predictive accuracy in classifying a sample as positive. It is determined by equation 7.
in equation 7 represent true positive and is false positive. (8) (9)
The recall is calculated as the ratio of correctly classified positive samples to all accessible, positive occurrences. The recall parameter specifies the model's ability to detect positive samples. The higher the recall, the more positive samples are discovered. Equation 8 represents mathematical representation. in equation 6 represent true positive and is false negative.
( + ) 2 ∗
∗ + 6.1.4. F1 Score: 6.1.5. AUC ROC:
The F1 score is a statistic that indicates how accurate a model is on a given dataset. It is used to assess binary classification systems that categorize examples as either positive or negative. Equation 9 represents mathematical representation.
The AUC - ROC curve is a benchmarking tool for classification problems using a variety of threshold values. The receiver operating characteristic (ROC) curve denotes the extent or measure of separability. In contrast, the area under the curve (AUC) indicates the level or degree of separability. It indicates the degree to which the model is adept at differentiating between classes. The larger the AUC, the more accurately the model predicts 1 class as '1' and 0 class as '0'. For example, the greater the AUC, the more accurate the model discriminates between abusive and non-abusive texts. 6.2.
Three fundamental parameters for deciding which classifier performed best are accuracy, precision, and recall. The classifier was created using all four algorithms. Table 2 shows the accuracy, precision, and recall value for all the algorithms.
difference between logistic regression and bagging classifier, which shows the use of extra effort in bagging classifier is not worth it. The ANN was the second least performing model with 79.1%, 74.1%, and 70.5% accuracy, precision, and recall. 6.3.
The main parameter used in this study is the F1 score, which is considered a reliable metric in classification tasks due to the involvement of both precision and recall. Figure 2 represents the F1 score for all the algorithms. F1 score for bagging and logistic regression are the same, 83%. Due to the extra effort being put into the bagging classifier, logistic regression is considered the better choice for the task. It provides the same F1 score with less effort.
F1 score
ANN s hm Bagging Classifer t i r oLogistic Regression g l A
The area under the curve for the receiver operation curve is an important parameter to judge the algorithm's performance. The AUC ROC values for each of the algorithm is shown in Figure 3. Looking at figure 3, we can see that bagging has the best value of 91.4, followed by logistic regression with 90.7%, which is not a significant difference compared to the extra efforts put into the bagging classifier. ANN also proved to be a good algorithm with a value of 87%. The algorithm with the lowest value of AUC ROC is the decision tree having 75.5% AUC ROC.
100 80 CO 60 R CU 40 A 20 0 75.5 90.7
The receiver operating characteristic (ROC) curve illustrates the relationship between TPR and FPR at various categorization levels. Reduce the classification threshold, and more items are classified as positive, increasing both true and false positives. Figure 4 shows the ROC curve for the decision tree, logistic regression, bagging classifier, and ANN. The more the area under the curve, the better the model. According to curves, it can be seen that the best performing models are logistic regression and bagging classifier. Although the ROC curve is not bad, compared to logistic regression and bagging classifier, it does not cut the best algorithms for this case due to the limitation of the dataset.
To eradicate the problem of the use of abusive language on social media, machine learning is employed to detect abusive remarks in Urdu tweets. The dataset used in this study was obtained from the text processing laboratory at the center for computing research of Instituto Politécnico Nacional. The detection of abusive language in Urdu text is performed as a classification task. The dataset has only two labels which makes it a binary classification problem. In order to solve this problem, the algorithm chosen were logistic regression, decision tree bagging algorithm, and ANN, which could work well on binary classification. Accuracy, precision, recall, F1 score, and AUC ROC were used as evaluation metrics. The key parameter for deciding the best algorithm was the F1 score. Based on the F1 score, logistic regression, and bagging performed equally well. However, logistic regression was chosen as the best performing model with an 83% of F1 score. ANN did not perform well due to the limitation of the dataset. All these evaluations were made on 25% of the test split.
For future work, different embedding layers will be trained on Urdu data. Different pre-trained models will be tuned for this use case to improve accuracy. After acquiring a model with better results, a chrome extension can be created to detect abusive Urdu text on social media.
[20] Amjad, Maaz, Alisa Zhila, Oxana Vitman, Sabur Butt, Hamza Imam Amjad, Grigori Sidorov, Alexander Gelbukh. "Overview of the shared task on threatening and abusive detection in Urdu at fire 2021." In CEUR Workshop Proceedings. (2021). [21] Amjad, Maaz, Alisa Zhila, Oxana Vitman, Sabur Butt, Hamza Imam Amjad, Grigori Sidorov, Alexander Gelbukh. "UrduThreat@ FIRE2021: Shared Track on abusive threat Identification in Urdu." In Forum for Information Retrieval Evaluation. (2021). [22] Amjad, Maaz, Noman Ashraf, Alisa Zhila, Grigori Sidorov, Arkaitz Zubiaga, and Alexander Gelbukh. "Threatening Language Detecting and Threatening Target Identification in Urdu Tweets." IEEE Access. (2021). [23] Maaz Amjad, Noman Ashraf, Grigori Sidorov, Alisa Zhila, Liliana Chanona-Hernandez, Alexander Gelbukh. "Automatic Abusive Language Detection in Urdu Tweets." Acta Polytechnica Hungarica. (2021).