Hate speech and o ensive language detection are receiving more and more attention in recent years. The RALIGRAPH team participated in the Shared Task on the Identi cation of O ensive content for Indo-European languages within the FIRE conference. This paper describes our approach VGCN-BERT model for all three sub-tasks of hate language and o ensive language detection in English. VGCN-BERT takes into account both local and global information, by combining the Graph Convolutional Networks (GCN) and the Self-Attention Encoder (BERT). Our approach produced good results in the experiments.
In recent years, an increasing number of users are subjected to o ensive languages or have witnessed abusive and hateful texts online, which is related to sexism, racism or other types of aggressive behaviors and cyberbullying. Governments have started to enact laws, and major social platforms such as Facebook and Twitter are also censoring o ensive posts with the assistance of arti cial intelligence technologies, human reviewing processes, user reporting processes, and so on. However, it seems the problem is still far from being successfully resolved.
A lot of research methods for o ensive language detection have been proposed
in the past few years [
However, the existing deep learning models may capture well semantic and
syntactic information in local contexts, but may ignore global information such as
word co-occurrences in a corpus which carries non-consecutive and long-distance
dependencies [
In our research on the task of classifying o ensive language, we intend to
leverage both the local information captured in BERT and the global
information on the whole vocabulary in a language. To obtain global information, we
use Graph Convolutional Networks (GCN) [
In section two, we describe some related work, such as self-attention mechanisms, BERT model and graph convolutional networks. In section three, we describe our approach, including the structure and training settings of VGCNBERT model, and the method of datset preprocessing. In section four, we list the results and make some comments. 2 2.1
Self-Attention [
QKT Attention(Q; K; V ) = softmax( p dk )V; (1) where the denominator is a scaling factor used to control the scale of the attention score, dk is the dimension of the query and key vectors. Using these attention scores, every word can get a weighted vector representation which captures contextual information.
The structure of BERT [
BERT is a pre-trained model. The goal of pre-training is to provide a good initialization for model training, which has been widely used in image classi cation and NLP. BERT is trained on 800M words from BooksCorpus and 2,500M words from English Wikipedia, and uses two unsupervised task to improves the pre-training: { Masked Language Model. Instead of the traditional n-gram language model, BERT randomly selects words to mask them out and then tries to predict the masked words from their context. { Next Sentence Prediction. BERT uses sentence classi cation as a pre-training task to determine if a sentence is the real next sentence or another randomly picked sentence.
A typical input to BERT is a pair of sentences as follows [
Input = [CLS] the man went to [MASK] store [SEP] he bought a gallon [MASK] milk [SEP]
Label = IsNext where [CLS] is the BOS (begin of sentence) for a pair of sentences; [SEP] is used as a separator of two sentences or as EOS (end of sentence); [MASK] is used to mask out the predicted words in the masked language model. The nal state of [CLS] embedding after encoding is used as the aggregated sequence representation for classi cation tasks. In our o ensive language detection task, we only need to classify every individual document, which is usually as short as a tweet, with only one or two sentences. So we treat the document as a sentence. Below is an example of the sentence classi cation:
Input = [CLS] Stupid ass coward b*tch f*ggot racist piece of sh*t. [SEP] Label = O ensive 2.2
A GCN [
Usually, people use the adjacency matrix A and its degree matrix D to represent graph G, where Dii = Pj Aij . The diagonal elements of A are set to 1 because of self-loops. For one convolutional layer, the formula is H = A~XW; (2) where X 2 Rn m, n is the number of nodes, m is the dimension of the feature, A~ = D 12 AD 12 is the normalized symmetric adjacency matrix and W 2 Rm h is a weight matrix. 3 In order to distinguish from notations (v; V; jV j) of vocabulary, this paper uses notations (p; P; jP j) to represent the point(vertex) of the graph.
Usually, we use two layers of GCN to capture information about direct and
indirect neighbors [
Text GCN [
The idea we explore in our work is to combine the graph's global informationawareness capabilities (global receptive elds) with the self-attention local informationawareness capabilities (local receptive elds). Speci cally, we want each document to get some global information about the vocabulary through a vocabulary graph before doing self-attention training. On the other hand, BERT is able to take into account local dependencies between words in a sentence. Our approach combines GCN with BERT.
For vocabulary graph, we take inspiration from Text GCN [
1 log p(i; j) log p(i; j) p(i)p(j)
After building the vocabulary graph, we construct the vocabulary graph embedding module and insert it into BERT in the word embedding phase. To obtain the vocabulary graph embedding, we modify the formula 2 to the following form:
Gembedding = XA~W; where X is the word embedding matrix which comes from word embedding layer of BERT, and A~ is the normalized symmetric adjacency matrix of vocabulary graph, and the output dimension of W controls the number of vocabulary graph embedding whose dimension is the same as every word embedding.
Therefore, the original sentence represented by word embeddings will be enriched by the vocabulary graph embedding. Then we feed all the embedding vectors to the self-attention encoder, and the encoder module will pay attention to all word embeddings as well as graph embeddings. Finally, we use the embedding produced for [CLS] for classi cation. (4) (5) 3500 3000 2500 2000 1500 1000 500 0
Task A Task B Task C 0.0 0.5 1.0 1.5 2.0 2.5
3.0 The training dataset of HASOC has a total of 5852 texts, and its labels are divided into three sub-tasks. Figure 1 shows the data distribution on the training set and nd that the categories of the dataset are unbalanced, especially Task B and Task C.
Therefore we use the weighted cross entropy as loss function. The weight of each of the classes (Wc) is calculated by
Wc =
#dataset #classes #one class ; (6) where #dataset is the total size of dataset and #classes is the number of classes and #one class is the count of one class.
In addition to the dataset provided by HASOC, for sub-task A, we also used
Founta et al.'s dataset [
To clean the text, we removed URL string and @-mentions for every text, then the text was lower-cased and tokenized using NLTK's TweetTokenizer 6. We use bertTokenizer function to split text, so that the vocabulary for GCN is always a subset of pre-trained BERT's vocabulary.
To build the vocabulary graph, we set the window size as 20 when calculating NPMI, and set the threshold of NPMI as 0.2 to lter out non meaningful relationships between words.
In the VGCN-BERT model, the graph embedding output size is set as 16, and the hidden dimension of graph embedding as 128. We use the bert-baseuncased version of pre-trained BERT, and set the max sequence length as 200. The model is then trained in 15 epoch-es with a dropout rate of 0.2.
The following are other parameter settings for the three tasks. { Sub-task A: While using Founta et al.'s dataset to pre-train the model, we set learning rate as 4e-6 and mini batch size as 12 and L2 loss weight decay as 1e-4. While ne-tuning the model using dataset of sub-task A, we set learning rate as 1.9e-6 and mini batch size as 16 and L2 loss weight decay as 0.02. { Sub-task B: we set learning rate as 4e-5, and L2 loss weight decay as 0.07. { Sub-task C: we set learning rate as 1.4e-5, and L2 loss weight decay as 0.05.
We use the original BERT model and a 2-layer MLP model as the two baselines. The parameter settings of BERT are the same as the VGCN-BERT for all three tasks, except that there is no graph embedding output dimension. We also use Founta et al.'s dataset to pre-train the BERT model for sub-task A. For the 2 hidden layers MLP, we use the term-frequency (TF) as inputs feature, and we set the rst hidden layer dimension as 512 and the second hidden layer dimension as 100, learning rate as 1.5e-3 and L2 loss weight decay as 2e-5, batch size as 64, total train epoch as 100, early stopping as 10. 4
Before getting the test set from HASOC, we take out small parts of the training set as validation set for model selection. The shu ed training set are divided into three sets with a ratio of 80:5:15, which represent the training set, the valid1 set, and the valid2 set, respectively. Table 1 lists the f1-score performance of the valid2 set when the performance of the valid1 set gets the best. It should be noted that we can't ensure that such parameter settings are optimal for all three 6 http://www.nltk.org/api/nltk.tokenize.html models, because we found that the parameters of learning rate and L2 decay have a great in uence on the nal performance of the models, and sometimes small changes of them will cause large uctuations. We believe the small dataset and the uneven data distribution are the main causes. Despite this, we nd that in most cases, the performance of VGCN-BERT is better than the other two models, especially on task C.
Table 2 lists the results on test set provided by the HASOC. Our team got third place on Task B and C and scored the second highest weighted F1-score on the Task C. 5
In this paper, we report the work we performed in the FIRE hate language detection tasks. Our main idea is to complement the local information captured by BERT with global information on the vocabulary. The combination of BERT with GCN seems to produce good results, and our submissions are ranked quite high.
As we do not have much training data speci c for the tasks, the model can be further improved with more training data. In addition, as future work, we will also consider using di erent relationship measurements to construct various vocabulary graphs. Also, more layers of GCN networks needs to be explored to get a sense of the in uences of the network structures when introducing BERT embeddings into VGCN.