Online social media platforms provide convenience for people to communicate, but the harm caused by online hate speech and ofensive language accompanying them is also significant. At the same time, it is a challenge to identify indirect insults such as metaphor and irony, so it is necessary to understand the semantic information of the text in depth. This paper describes the approach our team is using at HASOC2020: Hate Speech and Ofensive Content Identification in Indo-European Languages. In Subtask A and Sub-task B for English, we fine-tune ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, and add a customized network structure that enables the model to take advantage of the semantic information extracted by ALBERT to complete the classification task, and use StratifiedKFold to ensemble. We achieve Marco F1 of 0.4994 and 0.2412 in Subtask A and Subtask B for English language, ranked 15th and 11th.
The network platform constructs a brand new living and cultural space, promotes the
communication and exchange among netizens, and makes all kinds of information and speech grow
exponentially in the network space. Some individuals or groups with ulterior motives take the
opportunity to spread illegal information, which is not conducive to social stability or may
infringe upon the legitimate rights and interests of others. Among them, online hate speech
and ofensive language are a kind of undesirable speech that does great harm and attracts wide
attention. Online hate speech and ofensive language not only make people feel
uncomfortable, but also make the victim sufer from severe depression, causing irreversible psychological
harm. In addition to psychological harm, such toxic online content can also lead to real hate
crimes[
HASOC2020[
A: Identifying Hate, ofensive and profane content for English language and Sub-task B:
Discrimination between Hate, profane and ofensive posts for English language. We use a model
based on ALBERT: A Lite BERT for Self-supervised Learning of Language Representations [
As online social media platforms face great challenges in identifying hate speech and ofensive
language, the NLP community has done a lot of work on identifying hate speech, ofensive
language, cyberbullying, abusive content, and organized some shared tasks on these topics,
such as HASOC, HatEval[
In HASOC 2019, IRLab@IITBHU[
ALBERT: ALBERT is a simplified model designed by Google on the basis of BERT, mainly to solve the problem of BERT with too large parameters and too slow training. ALBERT overcame the obstacles to pre-training model expansion through two parameter reduction techniques: Factorized embedding parameterization, the large word embedded matrix is decomposed into two small matrices, so as to separate the size relation between the hidden layer and the dictionary, the two are no longer directly related, so that the node number expansion of the hidden layer is no longer limited. Cross-layer parameter sharing, this prevents the number of arguments from increasing with the depth of the network. Both techniques significantly reduce the number of arguments without significantly afecting their performance. The ALBERT conifguration is similar to the BERT-Large level, but with an 18-fold reduction in the number of arguments and a 1.7-fold increase in training speed. At the same time, parameter reduction also plays a regularization role, which greatly enhances the generalization ability of the model.
For this task, we use data sets provided by HASOC2020, mainly from Twitter. This task is divided into two subtasks. Sub-task A: Identifying Hate, ofensive and profane content. Sub-task A focus on Hate speech and Ofensive language identification ofered for English, German, Hindi. Sub-task A is coarse-grained binary classification in which participating system are required to classify tweets into two class, namely: Non- Hate and ofensive (NOT): This post does not contain any Hate speech, profane, ofensive content , and Hate and Ofensive (HOF): This post contains Hate, ofensive, and profane content. Sub-task B: Discrimination between Hate, profane and ofensive posts. This sub-task is a fine-grained classification ofered for English, German, Hindi. Hate-speech and ofensive posts from the sub-task A are further classified into three categories. Hate speech (HATE), Ofensive(OFFN) and Profane(PRFN), here, we need to add another category, (NONE) : posts that do not contain any of the above. In this task, we take part in the English task. For English language, the training dataset has a total of 3708 data, of which there are 1856 of HOF and 1852 of NOT, and there are 159 of HATE, 321 of OFFN, 1377 of PRFN and 1852 of NONE.
Our model structure is shown in Figure 1. First, we input the preprocessed text into the model through the input layer, and then vector representation is carried out. Vector representation is divided into three parts: word vector representation, text vector representation and position vector representation. ALBERT model converts each word in the text into the vector by querying the word vector table, namely, word vector representation; the value of this vector is automatically learned in the model training process, which is used to describe the global semantic information of text and integrate with the semantic information of words, namely, text vector representation; because the semantic information carried by words appearing at diferent positions in the text is diferent, the ALBERT model attaches a diferent vector to the words at diferent positions to make a distinction, namely, position vector representation. Then ALBERT model takes the addition of word vector, text vector and position vector as input, which is further processed by the Transformer Encoder module, we then input the output of the last hidden layer of ALBERT into BiLSTM, and concatenate the output of BiLSTM with the output of the last hidden layer of ALBERT, and then, through the Relu function, map the splited vector to the lower dimension in a nonlinear way. After that, max-pooling will be used to take the maximum value of each position in the vector on all timing sequence to obtain the feature vector. Finally, the feature vector will concatenate with the original ALBERT output and input into the classifier.
In this experiment, we use the StratifiedKFold ensemble method to train diferent data sets during each training process, and extract diferent features during the process of extracting features from the model, so as to enhance the generalization ability of the model and achieve the purpose of enhancing the performance of the model. The idea of StratifiedKFold is K-fold cross-segmentation. Each type of data in the initial training set is divided into K sub-samples, a single subsample is retained as the data of the verification model, and the other K-1 samples are used for training, that is, to ensure the hierarchical sampling, and the proportion of all kinds of samples in the training set and test set is the same as that in the original data set, as shown in Figure 2.
In order to eliminate the interference of irrelevant information in the tweet, so that the model can better extract text features and train the model more eficiently, we preprocessed the original training data provided by the oficial with NLTK tool. The main processing steps are as follows: • Remove number. Numbers generally have no meaning in text analysis, so they need to be removed before further analysis. • Steming. The process of reducing the derived form of a word to its stem, which will greatly shorten the word list and improve the eficiency. • Remove link address, whitespace, etc.( e.g https://t.co/5u8Di1waFC.) • Remove special characters. Characters that occur frequently but are meaningless for training need to be removed. (Remove the ’RT’ from the sentence.) • Converts all characters to lowercase.
The pre-training model we use in this experiment is albert_base_v2 from the ALBERT series model. In the process of fine-tuning, the maximum sequence length is set to 150, the learning rate is set to 2e-5, the gradient accumulation steps and batch size are both set to 4. By using 5-fold crossvalidation on the training data, we set the epoch to 10 for training, and the best weight is saved during the training.
The organizers evaluate the classification system by calculating the Marco F1 score. According to the oficial results, our team’ s Marco F1 score is 0.4994, ranked 13th place in Subtask A for English language, as shown in Table 1. And in Subtask B for English language, our team’ s Marco F1 score is 0.2412, ranked 10th place, as shown in Table 2.
In this paper, we describe our work in HASOC2020: Identifying hate speech and ofensive language. Our model uses ALBERT pre-training model to extract the semantic information features of text, and further processes the output features by using customized network structure. Finally, StratifiedKFold ensemble is used to improve the generalization ability of the model, and there are fewer model parameters, making the model lighter and easier to train. Our model achieves satisfying performance. In further research work, we will try to fine-tune ALBERT’s more hidden layers to make the model more suitable for specific training tasks, thus further improving the model performance.
We would like to thank the organizers for organizing this shared task and the teachers for their help. Finally, we would like to thank the school for its support to my research and the future reviewers for their patient work. for multilingual hate speech and ofensive language identification., in: FIRE (Working Notes), 2019, pp. 199–207. [14] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).