In this paper, we introduce the system for the Hate Speech and Ofensive Content Identification in Indo-European Languages (HASOC) 2020 Challenge, which is submitted by our team. We use a lot of social media in our daily life, but now social media is full of hate speech and ofensive language, so the detection of hate speech and ofensive language has become an essential task. The task is available in English, German, and Hindi, but there is a lot of work done in the English languages, with limited work reporting posts in Hindi and German, so we chose the German task to complete. The BERT-Ger model could not meet our requirements for semantic information characteristics, we modify the upper layer structure of BERT-Ger. Finally, our system wins second place in German subtask A and tenth in German subtask B.
With the rapid development of network information technology, we are more and more used
to express our opinions on social media, such as Facebook, Twitter, etc., but there is a lot of
hate speech and ofensive language in these published contents. Hate speech is published in
cyberspace and spread through the network media, aiming at groups with specific identities to
carry out ofensive and harmful speech1[]. The biggest diference between internet violence
and the traditional behavior of defamation lies in the fact that the objects of hate speech on the
Internet are specific groups with high identifiability [
The rest of the paper is organized as follows: In Section 2, we introduce the related work of hate speech and ofensive language detection. Section 3 gives a description of our proposed model and a summary of the dataset. Section 4 introduces the experiment and results, and section 5 gives the conclusion.
Some works have been done on the classification of hate speech and ofensive language. In
the following, we will briefly introduce and discuss the work of these researchers. On social
networks, people can interact without face-to-face, and more importantly, people often have
diferent backgrounds and perspectives. In the anonymous environment, some users use hate
speech to cause controversy to gain a sense of security. So you’ll find that anonymous users on
some sites are making hate statements. Nascimento et al3.][ classified Brazilian Portuguese
texts to detect hate speech. In social media, people are attacked by hate speech and ofensive
language for various reasons, such as gend4e,r5[], diferent nationalities [
In this section, we first analyze the data used and describe the distribution of the data. Then we describe the model we used.
We take part in the German language task. The German-language data set is collected from Twitter. Subtask A and subtask B use the same dataset. For subtask A, the dataset contains two labels: Hate and Ofensive (HOF) and Non-Hate and ofensive (NOT). For subtask B, the dataset contains four labels: (HATE) Hate speech: posts under this class contain Hate speech content, (OFFN) Ofensive: posts under this class contain ofensive content, (PRFN) Profane: these posts contain profane words and (NOT) Not: Non-Hate and ofensive. For details about the task, we refer the reader to the shared task publicati2o1n]. [Because of the relevance of label classification, it makes the classification task more dificult. Table 1 shows the detailed statistics in the dataset. As can be seen from table 1, the given dataset is imbalanced.
BERT[22] is a bidirectional encoder representation from Transformers. It is a new language model developed and released by Google at the end of 2018. BERT model plays an important role in many natural language processing tasks, such as question answering, named entity recognition, natural language reasoning, text classification, and so on. We use the BERTGer(bert-base-german-case1d)as our pre-trained model. The BERT-Ger trained 810k steps with a batch size of 1024 for sequence length 128 and 30k steps with sequence length 512. In training, it takes about 9 days. As training data, BERT-Ger uses the latest German Wikipedia dump (6GB of raw text files), the OpenLegalData dump (2.4 GB), and news articles (3.6 GB). In the classification task, the last layer hidden state of the first token of the sequence (CLS token) is processed by a linear layer and a Tanh activation function to get the output of BERT-Ger (pooler output). However, the pooler output’s summary of input semantic content is often inadequate. To let the model gain more features of semantic content, we try to solve this problem with the model architecture in Figure 1. First of all, the sequence of hidden states at the output of the last layer of the BERT-Ger is given to us, also known as the _ℎ _ 2. Then, we input the _ℎ _ into Bi-LSTM and Bi-GRU to get the output of Bi-GRU and the hidden state of Bi-GRU(ℎ_ ). Thirdly, we get th e _ by average-pooling and th e_ by max-pooling after getting the output of Bi-GRU. Finally, we concatenat e_ , _ , ℎ_ and _ into the classifier.
1https://huggingface.co/bert-base-german-cased 2https://huggingface.co/transformers/model doc/bert.htmlbertmodel
In this section, we mainly introduce the steps of our experiment and the results of the competition.
First of all, we get the new validation set and the new training set by using the stratified 5-fold cross-validation. The data set used is the training set provided by the competition organizers, with a total of 2373 pieces of data. This method of stratified sampling ensures that the sample proportion in each dataset remains the same. Then, in each fold of the data set, we choose the model with the highest F1 score in the validation set to predict the test set. And we get the prediction results of the model by averaging the probability of the five prediction results. Thirdly, we input data into one to four models for training with the training set and predict one to four results with the test set. Finally, we get the final result by combining the four results by hard voting.
In the experiment, we use the triangular learning rate, and the parameter learning rate is set to 1e-5. The learning rate is gradually increased through warm-up, and the linear learning rate is gradually reduced through linear learn rete decay. This experimental setup significantly improves the training efect. To save GPU memory, the gradient accumulation steps are set to 4 and the batch size parameter of GPU in fine-tuning is set to 4. The hyperparameters for each German language subtask are shown in Table 2.
The oficial ranking is based on the test scores of private test sets. We also get the F1 Macro average according to the test sets given by the oficial. The specific scores are given in Table 3. After the fine-tuning of the BERT-Ger model, the ability of our model to obtain semantic information characteristics has been improved. This enables us to get a good result in the German language task. Our method ranks 2nd (2/25) in German subtask A and the F1 Macro average score is 0.5225. In German subtask B, our method ranks 10th (10/19) and the F1 Macro average score is 0.2579.
This paper introduces the model and final results of the Simon team in Hate Speech and Ofensive Content Identification in Indo-European Languages (HASOC) 2020 Challenge. The model uses BERT-Ger, Bi-GRU, Bi-LSTM, and so on, which improves the ability of the model to obtain semantic information features. The identification of hate speech and ofensive content in social media has a positive efect on the development of society. In the future, we will participate in more such tasks and contribute to the identification of multilingual hate speech.
First of all, we would like to thank the HASOC sharing task organizers for giving us a happy experience and congratulations on the success of the sharing task. Secondly, I would like to thank BY for his valuable suggestions. [21] T. Mandl, S. Modha, G. K. Shahi, A. K. Jaiswal, D. Nandini, D. Patel, P. Majumder, J. Schäfer, Overview of the HASOC track at FIRE 2020: Hate Speech and Ofensive Content Identification in Indo-European Languages), in: Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, CEUR, 2020. [22] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. URLh:ttps://www.aclweb.org/ anthology/N19-1423.doi:10.18653/v1/N19- 1423.