We present three diferent transformer-based approaches to contextual hate speech detection, submitted respectively to the three tasks of the HASOC 2022 competition [1]. To deal with the scarce dataset we use an ensemble of cross-validation trained models (CV ensemble) that makes use of all available data while still having validation sets to apply early stopping. We try diferent methods such as ensemble of ensembles, cross validation trained ensembles and embedding isotropy optimization. Furthermore, we compare diferent base models for the ensembles as well as diferent solution proposals to the tasks. Our team, hate-busters, ranked 3rd in task 1, 5th in task 2, 3rd in task 3A, 1st in task 3B and 4th in task 3C.
The automatic detection of hate speech in internet forums and platforms has the capacity to significantly improve the online safety of millions of users. Yet, abusive text detection is a challenging task.
Firstly, language is dynamic, especially conversational language in social media, where words are often generated or given new meanings. The constantly changing vocabulary requires an ofensive language detection system to have an architecture capable of dealing with unknown words.
Secondly, the rise of social media in recent decades has also led to an increase in global inter-connectivity and a rich exchange of cultures and languages. It is not uncommon to find "macaronic" texts, namely texts written in hybrid mixtures of more than one language. While some natural language processing systems are already able to recognise and classify comments in multi-lingual settings, this code-mixed language in single text samples is an added dificulty for current algorithms. Furthermore, dialects and what are known as low-resource languages, often lack the large amounts of annotated data required to train deep neural networks, which leads to unbalanced levels of moderation and protection against hate speech across languages.
Thirdly, language and semantics are both very complex and highly contextual. Sometimes, external knowledge is needed to understand the meaning of a sentence. In the case of a conversation, a machine should take into account the previous text inputs when classifying a sample since they could have an efect on the samples label. For instance, a text with only neutral content should be moderated if it is a reply agreeing with a hateful comment. Because of this, a good hate speech detection system should include contextual knowledge. Finally, an extra task for abusive text detection is identifying the target of the hate speech, which provides more fine-grained information of the text.
Facing all of these issues is important if we are to fully solve the problem of automatic hate
speech detection and moderation. This is one of the main purposes behind the HASOC 2022
competition [
The first task is the (1) Identification of Conversational Hate-Speech in Code-Mixed Languages
(ICHCL) [
The second task is (2) multiclass ICHCL for hinglish text and it consists in identifying whether
a sample is standalone hate, contextual hate or no hate [
The final task is (3) Ofensive Language Identification in Marathi and it addresses the problems
for low-resource languages as well as the added complexity of identifying targeted hate speech
[
This work is structured into six sections. Section 1 is the introduction. Section 2 outlines the related work. Section 3 describes the data for every task and section 4 gives an overview of the processing steps and system architectures per task. The results are presented in section 5, followed by the conclusion in section 6 and the Appendix.
The automatic detection of hate speech is a research area that has been growing over the last few
years. There have been numerous surveys detailing the available data and methodologies used
in hate speech detection [
Due to its complexity and relevance, there have been a series of data challenges posted
online to encourage researchers to solve particular hate-speech detection tasks. Among these
challenges were: Prolifing Hate Speech Spreaders on Twitter - PAN 2021 [
The data used was provided by HASOC 2022 [
The data for task 3 [
Table 1 presents a data summary of the languages, number of train samples, number of test samples and class labels per task and subtask.
1 2 3A 3B 3C
For the first task, the team worked on 3 diferent solutions in parallel and then merged them into one ensemble model as can be seen in Figure 1.
The first solution used an XLM-Roberta huggingface model [
5 fold CV Ensemble
XLM Ensemble
Isotropy Model
The second solution used the same XLM-Roberta huggingface model and tokenizer as the ones used in the isotropy model. In this case, the samples were also concatenated with all parent texts yet the data was not normalized - nothing was removed. Furthermore, since the amount of data for training such a large model was quite scarce, it was decided that this model would use the entirety of the available data for training and spare no samples for validation. The model was trained for 5 epochs. This model architecture was then run using 5 diferent seeds. The scores of the five trained models were averaged resulting in an ensemble, which we refer to as the XLM Ensemble.
The final solution used a diferent XLM-Roberta huggingface base model and tokenizer,
namely Microsoft’s InfoXLM [
The scores of all three solutions are averaged into one ensemble. Since two out of the three models are ensembles by themselves we call this last ensemble the ensemble of ensembles.
The system architecture for the model submitted to task 2 is a 5-fold CV Ensemble as described in section 4.1 and uses the same data pre-processing steps. Diagram 4 in the appendix shows Task 2’s model architecture. 4.3. Task 3: Ofensive Language Identification in Marathi The third task is divided into three subtasks as described in section 3. Our aim is to train a joint model capable of solving all three subtasks at once. To achieve this we can look at every subtask as a categorical label, and the labels within that subtask as classes. From this point of view task 3 is reduced to a multi-label, multi-class problem.
We can further simplify the task by merging all labels into a single categorical label, thus converting the multi-label, multi-class problem into a single-label, multi-class problem. Since only the HOF class is used in subtask B and only TIN is used in subtask C, the combinations of labels across subtasks is small and the new label will only have five possible classes: NOT, OFF-UNT, OFF-TIN-IND, OFF-TIN-GRP and OFF-TIN-OTH. Figure 2 shows the data distribution across classes and subtasks with the original labels. We can compare this distribution to that of Figure 3, which shows the data distribution after merging the labels into one.
Once the labels are merged, we tokenize the data and feed it into our CV Ensemble architecture.
For task 3 we tested two diferent huggingface models: InfoXLM [
The results for task 1 are shown in table 2. The 5 fold CV ensemble received the best scores among the diferent architectures. The two most notable diferences in this system are on the 2000 le 1750 b lra 1500 e p lse 1250 p am 1000 s f reo 750 ubm 500 N 250 0
Label Distribution in Task 3
Label Distribution of Merged Labels NOT OFF TIN UNT IND GRP OTH
2000 le 1750 b lra 1500 e p lse 1250 p am 1000 s f reo 750 ubm 500 N 250 0 Subtask A
Subtask B Category labels per Subtask
Subtask C
NOT
OFF-TIN-IND
OFF-TIN-GRP OFF-TIN-OTH
OFF-UNT Merged Labels ifrst hand the model used - InfoXLM and on the other hand, the cross validation training format, which allows the use of early stopping while still applying the entire dataset for training. Due to a lack of submission tries we were not able to test the performance for the cross validation training across diferent models, which would give us more insight into the architectures efects on the task. Also due to the lack of submission tries we did not test the isotropy model by itself.
The other systems tested in task 1 were also ensembles and achieved very similar scores to the 5 fold CV ensemble results. The final ensemble model (the ensemble of ensembles) included the 5 fold CV ensemble, the XLM ensemble and the isotropy model yet it did not achieve better results than the 5 fold CV ensemble by itself (+0.3% Macro F1), proving that this form of ensemble stacking is not efective.
Our team achieved 3rd position in this task with a F1 diference of 4.73% to the top submission and a diference of 0.11% to the second position.
Table 2 also shows the results for task 2. The overall results of the submissions presented by all teams are lower than those from task 1 due to the increased complexity of the task. Our team ranked 5th and achieved a score of 46.51%, which was 2.88% lower than the top score (49.39%). Within our submissions, we compare our 5 fold CV ensemble method with a single InfoXLM model and we observe an increase of almost 3% when using the cross validation ensemble.
For task 3 we compare between three diferent architectures tested on all three subtasks. When comparing the performance of the 5 fold CV ensemble using InfoXLM as base against the L3cube we see a relatively significant increase in performance for the marathi trained L3Cube across all tasks. The increase is of +4.21%, +12.78% and +8.19% in F1 for tasks A, B, and C respectively.
The second comparison we can observe is the diference between the 5 fold CV ensemble and the 10 fold CV ensemble. Increasing the value of k was beneficial across all tasks, yielding improvements of +1.17%, +8.71% and +12.13% in F1 for tasks A, B and C. In this case, the diference in the magnitude of the improvements could also be due to the change in the number of samples available per task (see figure 3). Subtask A has almost 3 times as many samples as subtask B and almost 4 times as many as subtask C. Taking away a fifth of the data for validation vs a tenth makes a significant diference for such scarce data.
Another observation is the degrading performance as tasks become more fine-grained with the overall performance of submissions for task A being higher than those for task B, who are themselves higher than those for task C. Withal, the results from task 3 indicate clearly that both language-specific training and a higher value of k in the k-fold CV ensemble consistently improve model performance for said tasks.
For subtask A our team ranked 3rd scoring 1.77% F1 lower than the top submission and 0.21% F1 lower than the second submission. For subtask B, our team ranked 1st with an improvement of 0.58% F1 over the second position. And finally, for subtask C, our team ranked 4th. The variation of results for this task was larger than for the other tasks with the top result achieving a score 16.78% F1 higher than the second result. Our team scored 23.52% F1 lower than the top rank.
We have presented transformer-based ensemble solutions for each of the tasks in the HASOC 2022 competition. Our method of cross validation (CV) training has shown to be beneficial compared to our XLM Ensemble approach in this low resource setting for hate speech detection. The advantage of this CV method is that - by alternating the validation and training sets - it allows us to use the entire available data for training as well as to implement early stopping. From our experiments with InfoXLM and L3Cube in task 3 we can also conclude that using the evaluation language in the training set increases performance significantly, therefore using language-specific models instead of state of the art cross lingual and multilingual models is still in several cases a better option.
Data 5-fold Cross Validation sampling of
train and validation sets. 5 fold CV ensemble - InfoXLM 5 fold CV ensemble - L3cube 10 fold CV ensemble - L3cube
10 fold CV ensemble - L3cube Run Name infoxlm_large-5folds-joint task3-5fold-concat_a task3-10fold-concat_avg_a