This paper describes the system submitted to HASOC 2020. This task aims to identify hate speech and ofensive content in Indo-European languages . We only participate in the English part of subtask A, which aims to identify hate speech and ofensive content in English. To solve this problem, we propose an ALBERT-based model , and use the self-training and max ensemble to improve model performance. Our model achieves a macro F1 score of 0.4976 (ranks 20/35) in subtask A.
In recent years, due to the rapid development of the mobile Internet and social media platforms, people have begun to use various social media to share their lives, such as Facebook and Twitter. People share their views on life on social media, and this behavior may receive good or bad comments. Some bad comments slowly evolved into ofensive language. Social media is flooded with a lot of ofensive language [
HASOC 2020 [
In this task, we only participate in subtask A of English language : Identifying hate, ofensive
and profane content. In the task, the main problem to be solved is how to get the best task
performance. In order to solve the problem that afects task performance, this paper proposes a
method that combines two efective strategies. First, we introduced an external data set [
The rest of our paper is structured as follows. Section 2 describes data preparation. Methods are described in Section 3. Experiments and evaluation are described in Section 4. The conclusions are drawn in Section 5.
2.1. Data The organizers provided training and test datasets, containing 3708 and 814 sentences respectively. We counted the number and distribution of labels in the dataset. The number of labels in the dataset is shown in Figure 1.
where NOT means that This text does not contain any Hate speech, profane, ofensive content and HOF means that its contains hate, ofensive, and profane content.
Some text in the tweets has no efect on the meaning expressed. Tweets are processed using
the tweettokenize tool [
• Tokens are converted to lower case.
In Natural Language Processing (NLP), using diferent data in the same domain to train a model
is a form of model training. This training method is called self-training [
In this paper, we use self-training to train the model. The self-training process of our model
is shown in Figure 2. Our self-training method uses the idea of ”Teacher and student”. ”Teacher
and student” refers to the same training process. The beginning of student training is the end
of teacher training, which can deepen the learning of the model. We use the ”Teacher-Student”
method to design model self-training. The ”Teacher” part uses an additional external dataset,
which is the task dataset [
Google has introduced a new language representation model called BERT [
The ALBERT [
In this task, we use the ALBERT model to get good performance. Our model is shown in Figure 3.
In this paper, we hope to make full use of the ALBERT model through better fine-tuning strategies, so as to achieve the best task performance. In fact, fine-tuning the performance of ALBERT is usually sensitive to diferent random seeds and orders of the training data. In order to alleviate this situation, ensemble methods can be used to reduce overfitting and improve model generalization ability. Therefore, ensemble [10] methods are widely used to combine multiple fine-tuned models. The ensemble ALBERT model usually has higher performance than a single ALBERT model.
We know that the common ensemble method is based on voting [11]. In this paper, we ifne-tune multiple ALBERT models with diferent random seeds. For each input, we will output the best prediction and probability derived from the fine-tuned ALBERT, and summarize the predicted probability of each model. The output of the ensemble model is the prediction with the highest probability. This ensemble method is called the max ensemble. The formula for the max ensemble [12] we used is shown below ∑ ( =1 (; ) = ( )) (1) where (
) represents a fine-tuning of the ALBERT model.
In this task, we use self-training and max ensemble ALBERT-based model. For the ALBERT model, the main hyper-parameters we focused on are the training step size, batch size, warm steps, and learning rate. After learning the hyper-parameter adjustment for similar tasks, we ifne-tune the model hyper-parameters. As is shown in Table 1. means that only max ensemble is not used. ( ℎ) and max ensemble ALBERT-based model.
From this table, we can see that the self-training and the max ensemble method can efectively optimize the efectiveness of ALBERT model. So, for this task, our method can get a good performance. means that we use self-training
In this task, our main consideration is how to get a good task performance. In other words, we need to adopt methods to optimize the performance of our models. We mainly use the ALBERT model. Based on the model, we also adopt the method of self-training and max ensemble. Experiments prove that our method can achieve the best performance.
However, in the ranking, the performance of our model is still not satisfying. In the future, we will improve our methods by adjusting our model and trying more ensemble methods.
This work was supported by the Science Foundation of Yunnan Education Department under Grant 2020Y0011. [10] S. Avidan, Ensemble tracking, IEEE transactions on pattern analysis and machine intelligence 29 (2007) 261–271. [11] A. Onan, S. Korukoğlu, H. Bulut, A multiobjective weighted voting ensemble classifier based on diferential evolution algorithm for text sentiment classification, Expert Systems with Applications 62 (2016) 1–16. [12] Y. Xu, X. Qiu, L. Zhou, X. Huang, Improving bert fine-tuning via self-ensemble and self-distillation, arXiv preprint arXiv:2002.10345 (2020).