We describe our English, German and Hindi SVM and BERTbased hate speech classi ers, which includes the top-performing model for the German sub-task B. A special focus is laid on the exploration of various external corpora, the lack of mutual compatibility and the conclusions that arise from this.
In the participatory web, there is an ongoing in ux of user contents. Depending on the policies of a web page, the netiquette allows {and disallows{ a set of online behaviors towards others. This situation is further enforced by current governmental initiatives against online abuse and hate speech demanding direct action by the operators of an online service in case of law-infringing user contents.1 However, as the amount {and psychological weight{ of the data is overwhelming for human moderators, there is a growing interest in automating the identi cation of abusive comments online.
As a consequence, the recent years have seen a growing emergence of corpora
attempting to capture hate in various facets with the aim of providing training
data for text classi ers. While some corpora focus on di erentiating between
di erent targets of hate (i.e. sexism vs. racism) [
In the following sections, we will give a brief introduction to related work in hate speech classi cation (2), followed by a description of the data (3) and the models (4, 5). At the end, we present our results (6) as well as future work (7). 2
In the last years, a variety of standard text classi cation procedures have been
applied to the task of hate speech detection. These range from statistical
methods such as naive-bayes [
Due to the comparatively large amount of neography in user comments,
subword features such as character n-grams [
While most features used for training hate speech classi ers focus on textual
data, there is a recent interest in features that go beyond this by including user
information via embedded user graphs [
While the pre-processing for the BERT-based models is performed using the pre-de ned tokenization pipelines of each pre-trained model, the data provided to the SVM underwent various external pre-processing steps including tokenization, the removal of stopwords (excluding negations), lowercasing, stemming and lemmatization.
We explore various external hate-speech corpora and their e ect on the classi cation performance. However, as most corpora focus on di erent facets of hate, a one-to-one correspondence between labels is not always given. In such cases a mapping between similar labels was performed, which are described in table 1 along with the class distributions for task A and B for each corpus. Kaggle Davidson Founta TRAC TRAC en en en
Twitter en Facebook hi Facebook Corpora Lang. Source
Task A
Task B Mappings (A) Mappings (B) GermEval de Twitter
For English, we use four di erent external corpora. The Kaggle (KA) [
The Davidson (DA) [
Lastly, we used the TRAC (TR) [
For German, we explored the GermEval 2018 (GE) [
Most of the corpora have unbalanced classes. For task 1, the NOT class is often over-represented, which in its extremes leads to a ratio of 1:8835 hate to non-hate labels in the case of KA. However, for DA and TR, this unbalance is reversed, where more samples are marked as hateful than not. This unbalance is also present in task 2, where PRFN is heavily under-represented, followed by HATE for most corpora except GE. This unbalance, which can also be observed in the 2 https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/ data 3 https://github.com/t-davidson/hate-speech-and-offensive-language 4 https://github.com/ENCASEH2020/hatespeech-twitter 5 https://sites.google.com/view/trac1/shared-task 6 https://github.com/uds-lsv/GermEval-2018-Data o cial HS training data, leads to special di culties when training a classi er on these datasets. 4
We use pre-trained BERT [
To deal with the unbalanced nature of the training data, we perform randomized weighted re-sampling of the data at each epoch. The weights given to a class is calculated such that underrepresented classes are given a larger weight and vice versa. 5
SVM
We have used a linear SVM classi er for task 1. Here, we explored di erent
features including tf-idf, word and character n-grams as well as byte pair encoding
(BPE) [
Table 2 depicts the best combination of features for each selection, used for the SVM submissions in sub-task A:
Lang.
English German Hindi
Feature Combination tf-idf + BPE + word n-grams(1, 3) + stopword tf-idf + BPE + word n-grams(1, 3) + stopword tf-idf + BPE + word n-grams(1, 3) We train both BERT-based and SVM classi ers on di erent combinations of corpora, while 10-fold cross validation is performed on the o cial HASOC training data only. The results are reported in table 3.
When high-quality monolingual pre-trained models are available, these often yielded better results than their multilingual counterparts, i.e. +0:02 for HSen and HSde in task A, with the biggest gain in Macro F1 being +0:14 in 7 https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_
A-12.zip 8 https://deepset.ai/german-bert 9 https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_ H-768_A-12.zip Lang. Model
Data
F1
Task A
Macro Micro
F1
Task B
Macro Micro en de hi the case of the monolingual HSde as opposed to its multilingual counterpart in task B. This comes to show that high quality monolingual models {if language model training data is available in abundance{ can lead to great improvements over multilingual baselines. In fact, the usage of a high-quality monolingual pretrained model applied to the severely low-resourced task B, yielded top results for German. Nevertheless, for HSen, we observe a slightly better performance of the multilingual model in task B. One reason for this may be due to the nature of the training data, as the HSen contains India-related content as well as some Hinglish and code-switched sentences. This, together with the general enforced data sparsity in task 2, might have lead to the slight gain in macro F1 for the multilingual model.
For task A, adding external data either lead to slightly improved or unchanged results. For English, adding KA yielded an improvement of +0:02, which given the large size of the KA corpus is a modest increase. For German we observe a large increase in macro F1 (+0:06) when adding GE. This is most likely due to the larger amount of HOF-labeled data in the otherwise very similarly de ned GE corpus. In general, the simplicity of the binary decision task still allows for external data to be of use {or at least not destructive{ for the described task. However, when moving to the more complex task of identifying di erent shades of hate, external data quickly becomes reduced to additional noise during training, leading to either decayed or unchanged results for all external data in task B. This is especially interesting for GE, which has a very similar three-class corpus design (profane, insult and abuse). This comes to show that, as de ni
Task Language Run Model F1
Macro Micro A B tions of hate and its sub-classes di er, and nal annotations depend not only on the de nitions provided but also on subjective choices of the annotators, di erent hate speech corpora become incompatible, thus enforcing the data sparsity in this eld.
For all models in task A, the SVM models are outperformed by their BERT counterparts by margins between +0:03 (English and Hindi) and +0:09 (German). 6.1
Submitted Models For task A, both BERT and SVM models as well as an ensemble of both are submitted. For each language, run 1 is the ensemble of all 10 folds of the topscoring BERT model. As the recall of the NOT class is generally low, it was boosted by labeling a test sample as NOT whenever any of the folds suggested this label. For run 2, an SVM version trained on the whole dataset was submitted. Lastly, run 3 is the ensemble of all ten BERT folds and the SVM, using the same voting scheme as for run 1.
For task B, only BERT models were taken into consideration. As the test data provided still contained non-hateful comments, run 1 uses the BERT ensemble from task A (run 1) to pre-select hateful comments, which are then further classi ed by an ensemble of the 10 folds of the top scoring BERT model in task 2. Here, a majority vote approach was taken, such that the label with the most votes is accepted. For run 2, alternative models trained on HS data only but also covering the NONE label have been trained and were ensembled using the majority vote approach. Finally, run 3 is the ensemble of both run 1 and 2. The results of each run are reported in table 4. 7
We have trained both SVM and BERT-based classi ers on several combinations of external corpora. Primarily, we observed that ner-grained detection of di erent types of hate does not bene t from external corpora due to the incompatibility between di erent de nitions of hate {and its subtypes{ as well as the subjectivity of the matter, reducing external resources to added noise during training. This further enforces the data sparsity in the eld of hate speech detection also for higher-resourced languages with several corpora available. We therefore want to underline two directions for future research in abusive language detection and similar elds: a) A special focus on low-resource text classi cation for improved results despite the lack of large amounts of mutually compatible labeled data and b) creating corpora of hate speech which go beyond ambiguously de ned sub-categories of hate. For the latter, we plan to create a corpus which focuses on identifying di erent objective features within a comment {i.e. the targets of a sentiment (positive or negative), pragmatic cues such as the existence of an accusation, swear words or capitalization etc., | which in their sum will help to identify hateful content based on di erent subsets of such features.