Major setbacks for detection of abusive South African tweets are inadequacy of annotated corpus and high cost of annotation, which semi-supervised learning solves. Semi-supervised learning techniques enhance training data by combining labelled and unlabelled data. However, existing approaches have skewed classification of unlabelled data towards labelled data despite class imbalance of labelled data and unmatched feature distribution between labelled and testing data, that is common in abusive texts. This paper presents a reliable semi-supervised learning approach that reduces the noise in training data by combining features of unlabelled data with varying sizes of important features of labelled data. Chi-square statistics is used for the feature selection, while k-means algorithm is used for clustering of data points. By majority voting rule, reliable labels are assigned to the data points. Classifications with Support Vector Machine and Logistic Regression classifiers show that the proposed approach improves prediction performance.
The rise in the act of racial and social media conflicts and their negative
consequences means that improved detection of abusive languages on social networks
cannot be over-emphasised. One of the major setbacks to improvement of
detection of abusive languages in social networks is inadequacy of lexical resources for
many languages [
Over the last decades, South Africa has experienced upsurge in various
degrees of violence such as violent protests and xenophobic attacks, which have
led to loss of human and material resources. Many of these violent incidents
could be attributed to fast spread of inciteful and abusive comments,
perpetrated through social networks. However, there has not been any infrastructural
measure to check the soaring volumes of such communications. Recently, South
African government promulgated laws to address incidents of hate speech [
As at January 2019, there was twenty-three million active social media users
in South Africa, out which twenty percent, about five million have subscribed
to Twitter [
Machine Learning is a reliable technique that has been used for abusive tweet
detection [
South Africa is a multilingual society, but English is mostly used to
communicate on the social media. The existing English corpora for abusive language
detection have been labelled using crowd-sourcing tools or by annotators that
are familiar with the contexts of the abusive discourses. However, studies have
shown that language use varies across societies, contexts and individual [
This paper therefore focuses on development of inexpensive and reliable Semisupervised learning approach, which combines large unlabelled tweets and few labelled tweets to automatically detect abusive tweets. 2
Semi-supervised learning (SSL) techniques are machine learning techniques that rely on both labelled and unlabelled data for classification tasks. Here, machines learn from fewer labelled data points with the help of large number of unlabelled data points.
Several research works have used unlabelled data to enhance the performance
of classification models. These can be categorised into Self-Learning Approach
and Active Learning Approach. In Self-learning approach, unlabelled data is
automatically annotated and the instances with high confidence are added to
the training datasets iteratively. It can be categorised into Self-training
methods [
In this work, we are interested in Cluster-then-Label Generative method
because they were developed to address missing data problem. Kumar et al. [
Leng et al. [
The above semi-supervised learning approaches have relied on the labels of the labelled data to assign labels to the unlabelled data despite the classimbalance nature of the labelled data and partially matched features of the labelled and testing data, which is often the case in real-life. 3
In this section, we formalise the approach used to detect South African abusive tweets. 3.1
The Semi-supervised learning method proposed in this work is motivated by the following assumptions which indicate that labelling decision cannot be skewed towards labelled data, in real-life.
– Features of the testing data might not match exactly the features of either labelled or unlabelled data.
– Datasets of similar contexts share asymmetrical features.
Let X be set of n tweet samples xi X. Given a binary-class classification problem with l very low labelled instances and u large unlabelled data such that U > L; the set of labelled instances L = {(x1, y1), . . . ., (xl, yl)} and the set of unlabelled instances U = {xl+1, . . . , xl+u}, where y = (0, 1) are the class values of the data.
Since the objective of semi-supervised learning is to build a classification model based on the training dataset, then we define our approach as presented in equation (1).
y = CX(x) : y {0, 1} (1)
The schematic diagram in Figure 1 depicts the semi-supervised learning process. This involves three procedures: labelling of unlabelled data as described in section 3.3 and 3.4, training of merged unlabelled and labelled data, and testing with test data as described in section 4.3.3. (3) 3.4
n order to assign labels to the unlabelled data, majority vote rule is applied to every instance of R0 in A such that VT > 0.
The selected label yi = Cu(max(Vt)) (4)
Where R0 = {rl+1, . . . , rl+u} , VT = labels for each instance for all tested sizes of features.
The pseudocode is presented below: Data: Labels for unlabelled data instances, VT Result: Majority Voting labels, Vm for T = 1, . . . , t do max count = Max (AllCounters) if max count > t/2 then
Vm =class label corresponding to max count 4 4.1
end end
Total of 21,350 tweets of South African discourses on Twitter between the period
of May 5, 2019 and May 13, 2019 were collected using Twitter Archival tool, a
Google Sheets plugin, that works based on Twitter Search API. The collection
targeted tweets related to 2019 South African national elections, popular South
Africa individuals and trending issues such as land reclamation, Orania and white
communities. Tweets that contained non-English words were removed, except
names of individuals, towns, people, and organizations. Retweets and repeated
tweets as well as tweets with empty word characters were also removed. Following
these steps, the number of tweets remaining was 10,245. The tweets were divided
into three data samples of labelled, unlabelled and testing data. Total of 1,737
tweets were randomly selected for annotation while the remaining 8,548 tweets
were not annotated. The selected samples were annotated by two annotators as
either ‘abusive’ (A) or ‘non-abusive’ (NA), from which 1,730 tweets were selected
because of agreement on their labels. The Cohen’s Kappa agreement score [
Three major stages of process were involved, which include feature engineering, clustering and classification.
Feature Engineering The texts in the tweets were transformed into Term
Frequency and Inverse Document Frequency (TF-IDF) feature space, where weights
were created as indicated in equation (5) [
T F − IDF (t, d) = T F (t, d) ∗ IDF (t)
(5)
when IDF (t) = log[n/(DF (t) + 1] , n = total number of documents in the
document set; DF (t) = document frequency of t We also made use of word and
character n-gram models. The word n-gram include as Unigram (1), combination
of Unigram and Bigram (1, 2) and combination of Unigram, Bigram and Trigram
(1, 2, 3), while character n-gram include character n-gram with length sizes from
2 to 6 (2-to-6), 3 to 7 (3-to-7) and 4 to 8 (4-to-8)
Clustering We employed TF-IDF vectorization on the features of the labelled
and unlabelled data, without over or under-sampling. The important features of
the labelled samples were extracted using the Chi-Square statistics [
Classification We applied n-gram features weighted by TF-IDF vectorization
on the combination of semi-supervised labelled data and the originally labelled
data samples. They were used because of their effective performance in
previous text classification problems [
In order to select the best training model, Grid-search Hyperparameter Tuning approach was implemented over 10-fold Cross Validation. For the Support Vector Machine classifier (SVM), different C-regularization values ranging from 0.001 to 1000 were tested. For the Logistic Regression (LogReg), both L1 and L2 penalty functions with np.logspace values over -4, 4 and 20 were tested. 4.4
Precision, Recall, F-Measure, Accuracy and Mean Accuracy are metrics used to evaluate the performance of the proposed Semi-supervised Learning Method. The performance metrics are defined as presented in equations (6) to (10). The equations rely on the true positive (TP), which is the number of correctly predicted abusive tweets; true negative (TN), which is the number of correctly predicted non-abusive tweets; false positive (FP), which is the number of incorrectly predicted abusive tweets; false negative (FN), which is the number of incorrectly predicted non-abusive tweets.
Precision P =
Recall R =
T P T P + F P
T P
T P + F N F-Measure F1 =
Accuracy A = 2X(Recall X P recison) (Recall + P recision)
T P + T N
T P + F P + T N + F N Mean Accuracy = 10 i=1 Ai 10 (6) (7) (8) (9) (10) 4.5
The proposed semi-supervised learning approach (SSL) was compared with two supervised learning approaches.
– Method A: This is a supervised learning method, in which only labelled data was used for training data. – Method B: This is a supervised learning method with unmatched training data distribution consisting of labelled data and unlabelled data, which pseudo-labels was obtained by K-means clustering of unlabelled data without the features of the labelled data. 5
We evaluated the SSL for word n-gram and character n-gram features with Support Vector Machine and Logistic Regression classifiers and compared the performances with method A and method B. The testing results of accuracy and mean accuracy for SVM and LogReg over 10-fold cross validation are presented in Table 2 and Table 3, respectively. Also, the results of Precision, Recall and F-Measure for SVM for word and character n-gram features are presented in Table 4, while the results of LogReg are presented in Table 5. The comparisons of accuracy and F-measure of the SSL, method A and method B are presented in Figure 2.
The results of the performance of the SSL, method B and method A in Table 2 showed that the SSL recorded the highest accuracy of 0.9585 and 0.9667 for word n-gram and character n-gram, respectively followed by method A. The difference in the accuracy of the SSL and method A were 0.0942 (≈0.1) and 0.0968 (≈0.1) for word and character n-gram, respectively. In Table 3, the SSL recorded the highest accuracy of 0.9578 and 0.9696 for word n-gram and character n-gram, respectively followed by method A. The difference in the accuracy of the SSL and method A were 0.1228 and 0.0924 (≈0.1) for word and character n-gram, respectively. The same ranges of differences were recorded in mean accuracy, which showed clearly that the SSL convincingly outperformed method A and method B, in terms of accuracy. Method B recorded the lowest accurcy in both Table 2 and Table 3.
In Table 4, the value of precision, recall and F-Measure for SSL were higher than both method A and method B. While 0.96 and 0.95 precisions were recorded by the SSL in word n-gram evaluation for non-abusive and abusive, respectively, method A recorded 0.80 and 0.80, and method B recorded 0.89 and 0.90. In character n-gram evaluation, the SSL recorded 0.97 and 0.94, respectively, method A recorded 0.89 and 0.76, while method B recorded 0.80 and 0.00. In the case of recall, the highest values of 0.99 and 0.80 were recorded by the SSL for nonabusive and abusive, respectively, followed by 0.91 and 0.53 for method A. In character n-gram, the SSL recorded the highest recall of 0.99 and 0.87, respectively, followed by method A. The 1.00 recall against 0.00 for method B showed bias against abusive class during training. In the case of F-Measure, the SSL recorded the higest performance of 0.87 and 0.90 for abusive tweet detection for word n-gram and character n-gram, respecively followed by method A. The value of precision, recall and F-Measure for the SSL in Table 5 were also higher than both method A and method B.
In character n-gram evaluation, the SSL recorded 0.98 and 0.93, respectively, method A recorded 0.89 and 0.80, while method B recorded 0.80 and 0.00. For recall, while 0.97 and 0.89 precision was recorded by proposed SSL for nonabusive and abusive, respectively, method A recorded 0.92 and 0.49 and method B recorded 1.00 and 0.00. For character n-gram, the SSL recorded 0.99 and 0.89, respectively, method A recorded 0.96 and 0.52, while method B recorded 1.00 and 0.00. The 1.00 recall against 0.00 for method B, showed bias against abusive class during training. In terms of F-Measure, the SSL recorded the higest performance of 0.90 and 0.91 for abusive tweet detection for word n-gram and character n-gram respecively followed by method A. The lowest F-Measure was recorded by method B.
The bar chart in Figure 2(a) showed that there was consistent increase in the width of the charts from method B through method A to the SSL. There was also slight increase from word n-gram to character n-gram. In Figure 2(b), drastic rise was observed from the bar of method B to method A. So also was drastic rise from method A to proposed SSL. These outcomes indicated that the proposed SSL addressed the problems of class imbalance and unmatched distribution. We have developed a semi-supervised learning approach that combined both labelled and unlabelled data, without skewness towards labelled data for improved detection of abusive tweets in binary classification model. The approach reduced the impact of class imbalance and unmatched distribution among labelled, unlabelled and testing data features.
Matrix multiplication was used to fuse the labelled and unlabelled features; K-Means algorithm was used to cluster the fused features; majority voting rule was applied to select reliable labels for the unlabelled samples. The labelled and the previous unlabelled samples were used as training data. The performance of the approach was evaluated using word n-gram and character n-gram features as well as support vector machine and logistic regression classifiers. The results showed that our semi-supervised learning approach performed better than supervised learning approaches, with few training data or noisy training data. In future, classification of abusive language will be considered.