With the development of information technology, social media provides people with more and more convenient forms of communication. People can freely express their opinions on social networks. At the same time, some people will use social networks to release their one-sided emotions to guide the public to attack the innocent, undermine objective discussions, and intensify conflicts. An immeasurable language environment is the foundation of a harmonious atmosphere and the cornerstone of universal progress [1]. To purify the Internet environment, we need to identify the negative emotions on the Internet. Identifying hate speech and insulting, derogatory or obscene content and another negative emotional speech on social networks belongs to the research direction of emotion classification in natural language processing. Sentiment classification [2] is one of the main tasks in classification tasks in natural language processing, and it is also a hot spot in domestic and foreign research in recent years. The task of sentiment classification is to help researchers quickly obtain, organize, and analyze relevant text information, and analyze, summarize, and infer the emotion contained in the text. Traditionally, regular text language is more beneficial to the analysis, processing, induction, and reasoning of natural language processing. This is because the regular text used conforms to specific rules, it is critical for natural language processing to find the rules in the document.
However, in emotional classification, the language used is more perceptual rather than rational, and many sensitive texts are somewhat different from regular texts. Most of the languages used on social networks are related to personal life experience, and these texts are even affected by the personal national language in different ways of expression. Although the texts on social network media are in the same language, its meaning is quite different. Besides, the language text on social media is updated very quickly which is because frequently updated hot events on social media make netizens reach a consensus on a certain event. Besides, the language text on social media is updated very quickly. Facing the ultra-fast updating of social media texts, some scholars of natural language processing still insist on studying the characteristics of hate texts. From the initial manual feature extraction to the rule-based feature extraction of machine learning, and now, the feature extraction of neural networks. The feature extraction has experienced a long and complicated text research process.
HASOC competition, which is an evaluation task to identify hate speech and offensive content in Indo-European languages [3,4]. This year, HASOC provides 2 subtasks with each for Langusge. Subtask A: Coarse-grained classification, offensive and profanity content. Subtask B: Fine-grained classification, the distinction between profanity and offensive posts.
We participated in subtasks A and B in English and German, and the datasets discussed in this article come from HASOC. Based on the method of deep learning, we have developed an end-to-end neural network model, which takes Multi-Top π Self-Attention as the core and joins K-Max pooling. During training, K-Folding that can alleviate data imbalance and data overfitting, and the approach of fitting generation is used for batch training. This model has achieved an excellent result of subtasks A and B in English and German of HASOC 2020.
The structure of this paper is as follows: In Second 2, we introduce the related work of identifying hate speech and offensive language. In Section 3, we describe the data set and the model structure in detail. In Section 4, we describe the experimental result and data analysis.
The research of natural language processing on hate speech on social networks can be traced back to 2010. Gries et al. collected and annotated the text on social media(tweets), completed a love hate data set about social media speech, and provided these data sets to scholars who need to do research [5]. Provided these datasets to scholars who needed to do research. The scholars who made the dataset also held regular competitions to improve the dataset regularly to encourage more people to participate in this task. Dhillon et al. proposed the SAS model [6]. SAS provides a direction for studying computer network methods, technologies and systems. SAS uses natural language processing technology to identify sentence components (such as subject, verb, and object), disambiguate and identify entities so that SAS can identify whether there is a basic relationship. SAS provided one or more user interface tools for sentiment analysis. In 1996, Hochreiter et al. proposed Long Short Term Memory (LSTM) [7]. The emergence of LSTM made the technology of natural language processing an epoch-making milestone. Malmaisi et al. combined the method of feature representation (character n-gram, word n-gram and word skip-gram) with the Support Vector Machine (SVM) classifier to distinguish hate speech on social media from general profanity, and the accuracy of 0.78 was obtained [8,9]. Feature representation plays an important role in natural language processing. In 2018, Malmaisi et al. learned from the previous research, and based on the feature representation model, they tried to integrate multiple natural language classification models and achieved an accuracy of 0.80 on the three classification task [10].
The study of hate speech is not limited to the study of feature representation [11,12]. In the negative polarity and emotional intensity [13,14], the research of hatred classification features [15,16] has also made great progress. In deep learning, there are some similarities between natural language processing and image processing. Some neural network researchers apply the attention mechanism originally used for image processing to natural language processing, such as [17,18]. It is found that the attention mechanism is highly adaptive in natural language processing, and can be applied to various tasks, and achieve better than the neural network model originally used for natural language processing. In the attention mechanism, in addition to determining the 'value', the initial values of the 'key' and the 'query' must be manually set. The manual operation has not only increased the experimental error. In 2017, the self-attention mechanism was first proposed by Lin et al [19], and they set the value, key and query to the same value, thereby reducing the external influence and making the model self-optimizing. The self-attention mechanism is used in various tasks. In 2019, based on the study of self-attention mechanism, Child et al. proposed a sparse self-attention mechanism [18]. While reducing the calculation time of the algorithm, the model uses the filtering function to optimize the self-attention research, and the model also maintains a good effect on various tasks.
We preprocess the data set using word representation (Fasttext) to convert the text into vector that the computer can process. Two Multi-Top π Self-Attention were applied to refine the representative features of the text. After model using the K-Max pooling and Tanh functions get text characteristics, finally, softmax and tanh function were used to calculate scores of each classification. The model is shown in figure 1
The input layer accepts the preprocessed text data. Put the sorted data into the model, which data are in accordance with the format of data processed by the model
This layer accepts the input of the Input layer and vectorizes the words in the existing dictionary, which are input by the pre-training vector model.
In 2019, the results of Transformer [20] are obvious to all, but the problems of Transformer are also very obvious. The transformer takes a non-discriminatory approach to feature rep- here π is the first dimension of Embedding. π, πΎ, and π all come from input, but the weights of the matrix of a linear transformation are different. Then we multiply π and πΎ by dot product to get the dependency between the input word and the word. Finally, all values are mapped to a space with dimension π. We get the score for the attention mechanism, and π contains all the characteristic merit scores. At this point, our model is the same as that of the Transformer.
Then we start to filter the feature score of π.
The π β² here is the filtered matrix. we use the heap algorithm [21], maintains a small top heap of size πΎ and puts the data into the heap in turn. When the heap size is full, we only need to compare the top element with the next number. If it is larger than the top element, the current top element is discarded and inserted into the heap. Finally, all of the π‘ππ π are in the heap. But this p prime right here is the same thing as π. The purpose of this π‘ππ π operation is simply to mark π‘ππ π elements in π. We maintain a matrix called π π * π .
When π(π, π) is in the range of π‘ππ π , π(π, π) is 1, and the reverse is negative infinity. Then we filter out the values of the π‘ππ π .
π ππ π ππππ π΄π‘π‘πππ‘πππ = πππ π‘πππ₯(πππππ) Γ π
At this moment, we have completed an Attention block. We use the output of the Attention block as the input for the next Attention block. Finally, all values are combined.
In the previous operation, we connected parameters from two Multi-π ππ π Self-Attention. At this point, we take advantage of the π ππ π filtering mechanism again. But the difference is, here's π ππ π , we use the algorithm is the merge algorithm [22], take the first π value. The idea of the integration algorithm is to combine and sort the parameters on the dimension by using the division method. That is.
Concerning the output layer, in order to improve training efficiency, we first use the π ππβ function and finally use π ππβ function and softmax function to calculate the probability output of each class.
In this experiment, we took part in subtasks A and B in English languages and German languages provided by HASOC. We have completed a total of 4 subtasks. We submit the result on the platform given by HASOC [4], use F1-macro as the evaluation criteria. In these tasks, We get the macro F1-score of our model is 0.5042 of subtask A in English, 0.2396 of subtask B in English.0.5121 of subtask A in German,0.2736 of subtask B in German. The result and ranking as shown in table 1.
The distribution of the data is shown in table 2. All data is sourced from the data provided by the HASOC platform. Subtask A in English and German, identification hate or none hate positions, Hate speech and offensive posts in Subtask A fall into two categories: hate speech(hate), normal text(NOT). Subtask B in English and German, identification hate, profanity and offensive positions. This subtask is a fine classification of English, German, and Hindi datasets. Hate speech and offensive posts in Subtask B fall into four categories: hate speech(hateful), (OFFN) offensive, (PRFN) profanity, (NONE) normal text [4]. We can see that the total amount of data is small and unbalanced distributed, which can easily lead to model overfitting. Due to the small amount of data, we can obtain too few representative features during the training process. So that the model to learn that weak representation or only individual text has characteristics. These features are often irrelevant to the expected features that need to be extracted. To address the irregular amount and distribution of data, we use cross-validation. We messed up the data set into five parts, validate five times, each time taking 1/5 of the dataset as the validation set.
In this paper, we describe the attention mechanism model based on Multi-Top π Self-Attention and K-Max pooling, which used to subtask A and B in English and German of HASOC 2020. And in these subtasks,we have achieved a good result. Unbalanced data distribution will lead to poor model generalization and easy over-fitting. Therefore, in the future, we will focus on research on data imbalance processing.