In recent years, people are more concerned about hate speech identification and identification than ever. This paper describes our system for English and German Sub-Task A in HASOC2020. For these subtasks, we fine-tune the XLM-RoBERTa pre-training model for sentence embedding and extract the layer with the best performance for slicing and splicing. In order to make full use of both English and German corpus, we propose a multi-task method to optimize two classification tasks at the same time. Our model has achieved 0.9076 for F1 score in English Sub-Task A and 0.8165 in German Sub-Task A.
https://619496775.github.io/ (L. Xu) © 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Detecting abusive language in a sea of data on social media is a dificult and arduous work, and
researches have only conducted in recent years. Some research shows that the deep learning
model with word-embedding can achieve better results in text classification tasks. As a result,
Word2Vec is commonly used to obtain semantic information and attributes in the text through
an unsupervised word embedding method. Common machine learning algorithms include
Logistic Regression(LR), Support Vector Machine(SVM), Random Forest, etc. For deep learning
methods, most of them are based on Long-Short Term Memory network (LSTM), convolutional
neural network (CNN) or a recursive neural network (RNN). Some of the early work use features
like bag of words, word and character n-grams with relatively machine learning classifiers for
detection (Dinakar et al.[
More recently, BERT released by Google gains more attention in the research community, as it can capture both long-distance reliance and true bidirectional context information compared with the traditional way of deep learning.
BERT [
RoBERTa [
As for the XLM-RoBERTa model, it combines these advantages of both. We fine-tune XLMRoBERTa and use multitask training to improve the performance of prediction. This enables our model to achieve good results in the category of hate text.
The dataset of HASOC task A contains over 10,000 annotated tweets respectively composed of userID, tweets, and labels from Twitter. Through our analysis on the dataset, English dataset has a uniform distribution, while German dataset is uneven.
We took part in the English and German Sub-Task A, which involves building a coarse-grained binary categorization model to test whether a text is ofensive or insulting (HOF). If a text contains any form of unacceptable verbal, aggression and profanity, it will be considered as
Our solution is specifically divided into data preprocessing, feature extraction and model structure.
Because there was noise in tweets in oficial dataset, which would afect the performance of model training, we cleaned the data before training. The steps are as following: • Keep the label • Handle user name and @ beginning uniformly as ”username” • Separate conjunction • Turn Emojis expression into the corresponding phrase • Remove words that have no emotional meaning • Remove all URLs • Convert text to lowercase • Numbers are normalized to strings as ”number”
After data preprocessing, we combined the English and German datasets as one dataset. In order to improve the generalization ability of model, let the model learn the real data characteristic distribution, the method of stratified sampling is adopted to ensure the training set and the valid set have the same data distribution.
Besides, we divide 20% as the valid set, 80% as the train set. Table 1 shows the distribution of HOF and NOT in the dataset. The test set is given by oficial.
XLM-RoBERTa embeddings: We utilized XLM-RoBERTa[
[CLS] Tok1 Tok2 Layer1
E[CLS]
E1 [SEP] E[SEP]
Tok1 E1'
TokM
EM' E2 T2
XLM-RoBERTa Layer12
C
T1
T[SEP]
T1'
TM' 32×64×768
Slice
32×768 Sentences De and En
Labels De
Labels En
Softmax Classifier
Concatena te
Tc 32×768
After getting the embedded vectors of the texts, we fine-tuned XLM-RoBERTa to make it more suitable for the downstream task of hate speech identification. XLM-roBERTa has a total of 12 layers which learn diferent semantic information. Generally speaking, the shallower the layers, the more word level semantic information is learned. The deeper layers, the more generalized semantic information is learned. Whereas ([CLS]vectors) contains the semantic information for classification of the entire sentence, we try to combine with vectors at a certain layer to improve training performance. The influence of each layer is given in Table 2.
Global semantic information is more helpful for binary classification such as sub-Task A. The hidden layer of the XLM-RoBERTa model is 768 dimensions, with 12 layers of Transformer. Because the shape of is [32,768] and the hidden vector of the 12th layer’s shape is [32,60,768], we take out 12th layer as dim= (0,2) and splice together with , then passed it to classifier and send the result to Softmax. We trained both English and German tasks by feeding the processed data into the model.
Multitask learning can learn useful information from similar tasks. All tasks will share XLMRoBERTa’s layers. In order to take full advantage of each classification corpus, two classification tasks are combined to carry out multi-task training. In this way, the number of datasets can also be expanded.
LR is a machine learning method to solve the problem of categorization (0 or 1). We use it as the baseline classifier for both English and German datasets. The configuration is as follows: we use L2 regularization with the hyper parameter C=1.2 (Inverse of regularization strength) and use TF-IDF features of word n-grams(1,6) for training the classifier.
SVM is a binary classification model [
BiLSTM[
XLM-RoBERTa model and other model’s accuracy and F1 macro-average score in English and German are showed in Table 3 and Table 4. To verify the efectiveness of the fine-tuning strategies, ablation experiments are conducted. We also using BERT base model and − as the baseline.
In the two subtasks, BERT and XLM-RoBERTa models performance better than LR, SVM and BiLSTM models. It may because deep learning has great advantages over traditional machine learning methods in document classification. For traditional machine learning methods, it is not easy to extract text features. Moreover, these features cannot well represent the semantics and syntax of the document, and a large part of useful information is lost. Deep learning is to hand over the feature extraction to deep network for automatic completion. Higher computational costs in exchange for more comprehensive and better text features. So deep learning methods performance better in our hate speech identification tasks.
These experiments prove that our fine-tune XLM-RoBERTa model is efective for both German and English tasks. Table 3 and 4 show the comparison between diferent models where deep learning models performs better than traditional machine learning models.
Our system ranked 16th in the German subtask and 34th in the English subtask, F1 macroaverage Score for German subtask was 0.4968 (Top team was 0.5235) under the oficial private dataset, F1 macro-average score for English subtask was 0.4856 (Top team was 0.5152) under the oficial private dataset.
We have proposed a neural solution with fine-tuning XML-RoBERTa for hate speech identification. Particularly, the output and hidden layers are slicing and splicing, which solves the sparsity of data and increases the generalization ability of the model in a multi-task way. Experiments have proved the competitiveness of our method.
This work is supported by the National Natural Science Foundation of China (61962061), partially supported by the Yunnan Provincial Foundation for Leaders of Disciplines in Science and Technology, Top Young Talents of ”Ten Thousand Plan” in Yunnan Province, the Program for Excellent Young Talents of Yunnan University.
• GitHub