This paper describes the WLV-RIT entry to the Hate Speech and Ofensive Content Identification in Indo-European Languages (HASOC) shared task 2020. The HASOC 2020 organizers provided participants with annotated datasets containing social media posts of code-mixed in Dravidian languages (Malayalam-English and Tamil-English). We participated in task 1: Ofensive comment identification in Code-mixed Malayalam Youtube comments. In our methodology, we take advantage of available English data by applying cross-lingual contextual word embeddings and transfer learning to make predictions to Malayalam data. We further improve the results using various fine tuning strategies. Our system achieved 0.89 weighted average F1 score for the test set and it ranked 5ℎ place out of 12 participants.
Ofensive content is pervasive in social media putting users of various platforms at risk [
Recent studies have addressed many types of ofensive content such as online abuse [
In this paper we describe the WLV-RIT entry to the the HASOC 2020 shared task which
featured Malayalam-English code-switched data. Building on the experience of recent high
performing models submitted to the OfensEval competitions, we use a transformer-based
architecture described in detail in Section 3. The HASOC 2020 code-switching dataset is a
particularly challenging one for state-of-the-art ofensive language detection systems and we make
use of transfer learning techniques that have been recently applied to project predictions from
English to resource-poorer languages with great success [
The goal of this task is to identify the ofensive language of the code-mixed dateset of
comments/posts in Dravidian Languages (Tamil-English and Malayalam-English) collected from
social media. Each comment/post is annotated with an ofensive language label at the
comment/post level. The dataset has been collected from YouTube comments [
In addition to the dataset provided by the organisers we also used an English Ofensive
Language Identification Dataset (OLID) [
B: Categorization of ofensive language - targeted insult or thread vs. untargeted profanity;
We adopted the transfer learning strategy similar to previous recent work [
The methodology applied in this work is divided in two parts. Subsection 3.1 describes traditional machine learning applied to this task and in Subsection 3.2 we describe the transformer models used.
The motivation behind our methodology is the recent success that the transformers had in
wide range of NLP tasks like language generation [
In the first part of the methodology, we used traditional machine learning models. We
experimented with three models; Multinomial Naive Bayes [
Data Preprocessing We performed three preprocessing techniques; removing punctuations,
removing emojis and lemmatising the English words. This was done with the use of the NLTK
(Natural Language Toolkit) library [
Hyper Parameter Optimisation Optimisation of hyper parameters was performed on SVM and random forest only. For SVM, the hyper parameters fine-tuned were alpha, random state and max iteration, where alpha represents regularisation, random state is used for shufling of the data and max iteration denotes number of passes through the training data which is also known as epochs. Optimal values achieved were alpha=0.001, random state=5, max iteration=15. For random forest, only one hyper parameter was used which is n-estimator that denotes number of decision trees created. Optimal value achieved for number of trees was 500.
As the second part of the methodology, we used Transformer models. Transformer
architectures have been trained on general tasks like language modelling and then can be fine-tuned
for classification tasks [
1HuggingFace model repository - https://huggingface.co/models
Transfer Learning The main idea of the transfer learning strategy is that we train a
classiifcation model on a resource rich language, typically English, using a transformer model and
perform transfer learning on a less resource language. We trained the classification model on
the first level of OLID [
Data Preprocessing The data preprocessing for this task was kept fairly minimal to make it
portable for other languages too. We only followed one data preprocessing technique;
converting emojis to text. Emojis are found to play a key role in expressing emotions in the context of
social media [
Fine-tuning To improve the models, we experimented diferent fine-tuning strategies:
majority class self-ensemble, average self-ensemble, language modelling, which are described
below. These fine tuning strategies have shown promising results in recent shared tasks [
• Majority-class SE (MSE) - As the majority class, we computed the mode of the classes predicted by each model. Given a data instance, following the softmax layer, a model predicts probabilities for each class and the class with highest probability is taken as the model predicted class. • Average SE (ASE) - In average SE, final probability of class is calculated as the average of probabilities predicted by each model as in Equation 2 where h is the ifnal hidden state of the [CLS] token. Then the class with highest probability is selected as the final class.
( |ℎ) =
∑ =1 ( |ℎ) (2) 2. Language Modelling (LM) - As language modelling, we retrained the transformer model on task dataset before fine-tuning it for the downstream task; text classification. This training is took place according with the model’s initial trained objective. Following this technique model understanding on the task data can be improved.
Implementation
We used a Nvidia Tesla K80 GPU to train the models. We mainly fine tuned the learning rate and number of epochs of the classification model manually to obtain the best results for the validation set. We obtained 1 − 5 as the best value for learning rate and 3 as the best value for number of epochs for all the languages. Training for English language took around 1 hour while training for Malayalam took around 30 minutes.
In this section, we report the experiments we conducted and their results. As informed by the task organisers, we used Weighted Average F1 score to measure the model performance. We also report Precision, Recall and F1 score for each class label as well the Macro F1 score in the results tables. Results in Tables 1 - 5 are computed on validation dataset. Finally, in Section 4.1 we report the results provided by organisers to our models, for the test set. learning outperformed other models. Also we could notice that transfer learning improved both models; BERT and XLM-R.
Model Random Forest Linear SVM Mult. Naive Bayes The self ensemble methods were experimented using all the transformer models and obtained results are summarised in Tables 3 and 4. In most experiments, ASE has given a higher F1 than MSE and it improved the results over the default settings. With that fine tuning strategy too XLM-R with transfer learning outperformed all the other models.
The language modeling fine tuning strategy were experimented using all the transformer models and obtained results are summarised in Table 5. These experimented were done on top of ASE fine tuning strategy since it provided better results than the default settings. Results show that language modeling clearly improved the results. In fact, the best result from our experiments were shown when XLM-R model with transfer learning fine tuned with ASE and language modeling.
Considering the evaluation results on the validation set, we selected the fine-tuned XLM-R(TL) model with ASE + language modeling as our oficial submission to the HASOC task. According to the results provided by the organisers, our best model has scored 0.89 weighted average F1 score on the test set and ranked 5ℎ out of 12 participants.
In addition to the experiments described in this paper, we carried out a qualitative analysis on the dataset to find interesting patterns and observations. In the training data out of 3,200 tweets only 567 were labelled ofensive and the remaining 2,633 were labelled as not-ofensive. The use of English words were minimal although there are many tweets which are in Malayalam language but written in Roman script. When analysing the tweets labelled as ofensive, we observed that there are many tweets in the dataset which are actually not-ofensive but labelled as ofensive. Free English translations of some examples include: (1) Spent 4 years proclaiming to be a Royal Mech. (2) There are 25k dislikes from Ikka (Mammooty) fans, you are free to unlike and cry. (3) Nice, looks like a TV drama series from SuryaTV (a Malayalam channel). (4) Have you no shame defaming a reputed hospital? We observed that between 20% and 25% of the tweets which are labelled as ofensive are similar to the example shown above which has certainly impacted the performance of the models.
In this paper we have presented the system submitted by the WLV-RIT team to the HASOC
2020 - Ofensive Language Identification - Dravidian Code Mix Task 1 at FIRE 2020.
Following a recent study [
We would like to thank the HASOC organizers for running this interesting shared task and for replying promptly to all our inquiries. We further thank the anonymous reviewers for their insightful feedback.