With the development of the Internet, more and more people use the social medias to share their daily life. However, there are various problems existing in the online community. One of these problems is that some people would like to post hate speech and ofensive contents. How to identify hate speech and ofensive contents is a serious problem. “Hate Speech and Ofensive Content Identification in English and Indo-Aryan Languages” is a track which is committed to solve this problem. We used three diferent models consisting of SVM, CNN and BERT to do experiments about English texts. Among them, BERT has the best performance. Our team called QQQ get a Macro-averaged F1 score of 0.7374.
A lot of methods have been used in the detection of hate speech and ofensive content such as
TF-IDF[
A number of recent studies show that the methods based on the deep learning have better
performance than the traditional machine learning. Convolutional Neural Networks (CNN)[
√
) ,
Where is the dimension of the query and key vectors. Based on these attention scores, each
word is given a weighted vector representation that captures contextual information[
In this paper we choose to use the model of BERT[
In this section, we will introduce the datasets and the method we process them.
The training dataset is provided by HASOC which consists of ids, texts and labels of subtask 1A
and 1B[
• (NOT) Non Hate-Ofensive:
This post does not contain any Hate speech, profane, ofensive content. • (HOF) Hate and Ofensive:
This post contains Hate, ofensive, and profane content. • (HATE) Hate speech: Posts under this class contain Hate speech content. • (OFFN) Ofensive:
Posts under this class contain ofensive content.
• (PRFN) Profane: These posts contain profane words.
The data is mainly from Tweet and Facebook in English, Hindi and German. In this paper, we choose to use the English dataset to perform the experiment. There are total 3790 texts in the English training dataset and 1268 texts in the test dataset. In addition to the dataset of HASOC 2021, for the subtask 1A, we also used the datasets of HASOC 2020 and HASOC 2019 which consist of 12305 texts in English in total. The size of test datasets is 1268 posts. The Table 1 shows the datasets in detail. We need to use the training data to train our models. Then use the models to predict the labels of test data.
The data format in the dataset is presented in the Table 2.
Text Technically that’s still turning back the clock, dick head
https://t.co/jbKaPJmpt1 @krtoprak_yigit Soldier of Japan Who has dick head damn damson asked for some transparency and they made him deactivate
Task_1
Task_2 HOF HOF NOT
OFFN OFFN NONE
We combined the datasets of HASOC 2021, HASOC 2020 and HASOC 2019 into the training data and disrupted the order. At the same time, we delete the labels of task_2. After that, we divide the training data into a train set and a development set according to a 4:1 ratio.
The test set is released by HASOC including 1268 posts.
To clean the data, we removed some useless strings such as URL, id and emoji. We also removed some special symbols. These strings may reduce the accuracy of experimental results. So we removed them in advance to improve the result of the experiment.
In this section, we are going to introduce the methods that we used in the experiments. The methods are SVM, CNN and BERT.
Support vector machine[12] is a binary classification model whose basic model is a linear classifier defined by maximizing the interval on the feature space, which distinguishes it from a perceptron. SVM also includes kernel tricks, which make it essentially nonlinear classifier. The learning algorithm of SVM is the optimization algorithm for solving convex quadratic programming.
CNNs are often used in image recognition systems. Besides that,CNNs have also been explored for natural language processing. CNN models are efective for various NLP problems and achieved excellent results in semantic parsing, sentence modeling, search query retrieval, classification, prediction and other traditional NLP tasks[ 13].
A convolutional neural network consists of an input layer, hidden layers and an output layer. In any feed-forward neural network, all of the middle layers are called hidden because their outputs and inputs are masked by the activation function and final convolution. In this paper, the input layer is the embedding layer. The embedding layer encodes the word used in the comments. We used fastText embedding as the embedding layer. After feeding the embedded comment to the CNN layer, we used four layers of convolution and one layer of max-pooling.Finally, we used the flatten layer followed by a dense layer. At the dense layer, sigmoid and softmax are activation functions which is good at binary class problems.In the hidden layers, activation function is the Rectified Linear Unit (ReLU). 4.3. BERT BERT is a pre-training model based on self-attention which has good performance in many NLP tasks[14]. It evolved from the transformer model which is a kind of seq2seq model consisting of Encoder and Decoder.
In our experiment, we used the pre-trained models of ”BERT-Base, Uncased” released by Google which has 12-layer, 768-hidden, 12-heads, 110M parameters. We set max_seq_length=200, train_batch_size=32, learning_rate=2e-5 and num_train_epochs=3.0. In our task of identifying the hate speech, we only need to classify the text which is usually less than 200 words. We can regard the text as a sentence. We classify a sentence as follow:
Input = damn damson asked for some transparency and they made him deactivate Label = NOT
results.
tion of the evaluation: In this section, we will show our results of the experiments. We choose SVM and CNN as the baseline. We wiil find how much the Bert model can improve the experimental efect. And will there be diferences between the data with preprocessing and without preprocessing in the In this paper, we used Macro-averaged F1 score as the evaluation measure.Here is the definiCondition positive (P):the number of real positive cases in the data.
Condition negative (N):the number of real negative cases in the data.
There are true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN) in the result of binary classification. Precision and recall are then defined as: = = =
= ∑
+ + 2 ×
× + = ∑
(1) (2) (3) (4)
We used SVM, CNN and BERT to perform the experiment. The evaluation measure is Macroaveraged F1. The result presents in the follow table.
With preprocessing
Without preprocessing SVM CNN BERT 0.6042 0.6744 As expected, the preprocessing of texts can also improve the evaluation of the result.
In recent years, the spread of hate speech has become more widespread. It has become a serious problem in the social medias. In this paper, we make a little contribution to this problem. We used the models of SVM, CNN and BERT to classify the hate speech. Among them, BERT has the best performance whose Macro-averaged F1 score is 0.7374.
Unfortunately, the training data is a little bit small. We may improve the model if we have more training data. Besides that, the number of two kinds of label is not equal.The gap between them is a bit big. This also has a bad efect on the experiment. In the future work, we will try some improved models based on BERT such as ERNIE[15] and RoBERTa[16]. We will also search for more data to enhance the training data. identification for low resource languages: The case of marathi, in: Proceedings of RANLP, 2021. [11] S. Satapara, S. Modha, T. Mandl, H. Madhu, P. Majumder, Overview of the HASOC Subtrack at FIRE 2021: Conversational Hate Speech Detection in Code-mixed language , in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021. [12] A. Wright, A. B. McCoy, S. Henkin, A. Kale, D. F. Sittig, Use of a support vector machine for categorizing free-text notes: assessment of accuracy across two institutions, Journal of the American Medical Informatics Association 20 (2013) 887–890. [13] W. Wang, J. Gang, Application of convolutional neural network in natural language processing, in: 2018 International Conference on Information Systems and Computer Aided Education (ICISCAE), IEEE, 2018, pp. 64–70. [14] M. Mozafari, R. Farahbakhsh, N. Crespi, Hate speech detection and racial bias mitigation in social media based on bert model, PloS one 15 (2020) e0237861. [15] Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, H. Wang, Ernie 2.0: A continual pretraining framework for language understanding, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 2020, pp. 8968–8975. [16] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019).