In recent years, there has been a lot of focus on ofensive content. The amount of ofensive content generated by social media is increasing at an alarming rate. It created a greater need to address this issue than ever before. To address these issues, the organizers of “Dravidian-Code Mixed HASOC-2021” have created two challenges. Task 1 involves identifying ofensive content in Malayalam data, whereas Task 2 includes Malayalam and Tamil Code Mixed Sentences. Our team participated in Task 2. We used multilingual BERT to extract features in our proposed model, and we used two diferent classifiers, Support Vector Machine (SVM) and Deep Neural Network (DNN), on the extracted features. In addition, we used the proposed data to evaluate the performance of a monolingual BERT classifier. Our best performing model monolingual Bert received a weighted F1 score of 0.70 for Malayalam data, ranking ifth; we also received a weighted F1 score of 0.573 for Tamil Code Mixed data, ranking twelfth.
The availability of smartphones and the internet has created a lot of interest in social media
among today’s youth. These applications give a huge platform for users to connect with the
outside world and share their ideas and opinions with others. With these benefits comes a
disadvantage: many people misuse the platform under the name of freedom of expression to
publish inflammatory content on social media. This inflammatory information typically targets
a single person, a group of people, a particular faith, or a community [
Various studies on hate speech, harmful content, and abusive language identification in social
media have been conducted during the previous decade. The majority of these studies were
focused on monolingual English content, and a large amount of English language cuprous
has been created [
The rest of the article is arranged in the following manner: Section 2 provides a brief summary of previous work, while Section 4 describes the proposed model in full. Section 5 concludes by providing information on the outcome.
Many researchers and practitioners from industry and academia have been attracted to the
subject of automatic identification of hostile and harmful speech. [
Ofensive content detection from tweets is part of some conferences as well as competition
tasks. Ofensive 2020 was provided by SemEval in 2020 as a task in five languages: English,
Arabic, Danish, Greek, and Turkish [
We have taken data set from HASOC subtask, ofensive language identification of Dravidian
CodeMix[
Task 1: A binary classification problem with message-level labeling for ofensive and nonofensive information in Malayalam CodeMixed YouTube comments.
Task 2: Given Romanized Tanglish and Manglish tweeter or YouTube comments, the system must classify them as ofensive or non-ofensive.
Our team took part in Task 2 for identifying ofensive information in the Tanglish and
Manglish data sets. According to the organizer, Tanglish data is collected from Twitter tweets
and comment on the hello APP. Whereas Manglish data is taken from YouTube comments [
Our team has proposed three submissions based on three diferent models. In the first two models, mBERT embeddings are passed through SVM and DNN classifiers, while in the third model, monolingual BERT is employed as a classifier. Each of them is designed using the general architecture shown in Figure 1. Thus, our model consists of three stages, each of which is discussed in the preceding subsections.
The data set provided by the organizer contains many unwanted information. A few data
preprocessing steps were undertaken on both text and label fields to convert the data suitable for
model building. Digits, special characters, hyperlinks, and Twitter user handles were omitted
from the data set because they were not helping us improve the performance of our model.
Furthermore, the social media data provided by the organizer did not follow grammatical norms;
hence, data lemmatization is performed to convert the data to its usable base form. For example,
the word ate, eaten, and eating were converted to their base form eat. Converting text to lower
case is also done to eliminate redundant terms. All of this preprocessing was done with the
help of the NLTK toolbox from the Python library [
1https://huggingface.co/bert-base-multilingual-cased
To obtain contextual embeddings from Code-Mixed data, we used the multilingual Bidirectional
Encoder Representation (mBERT) model [
Our proposed model experimented with three diferent classifiers: SVM and DNN classifiers with mBERT embedding in models 1 and 2 and pre-trained language model BERT in model 3. The descriptions of these models are presented in the subsections that follow. The intuition behind selecting these proposed models is that they outperformed other models such as Logistic Regression (LR), Random Forest (RF), and Naive Bayes (NB) in our preliminary trials.
We experimented with traditional machine learning algorithms such as Support Vector Machine
(SVM) with ten-fold cross-validation. Experiment results for the suggested model demonstrate
that kernel value ”1” and solver ”lbfgs” produce the best results. Experimental trials are used
to determine these hyper-parameter values. This model accepts mBERT embeddings as input
and produces labels that are either ofensive or non-ofensive. The model was developed using
Python’s sci-kit-learn library [
Later, we experimented with the Deep Neural Network (DNN) model, a second model in our proposed methodology. The DNN model comprises several dense layers that are designed to extract more significant features from input embeddings. We used dense layers of 1000, 500, 100, and 50 neurons in our model. Each dense layer follows a dropout rate of 0.4 to prevent the overfitting problem. The optimum grid search value determines the dropout rate of 0.4, and it remains constant throughout the experiment. To normalize activation data, we additionally employed a batch normalization layer. The output from these layers is then classified using the sigmoid layer.
In our last model, we experimented with transformer-based language models such as BERT. Transformer architectures are trained on generic tasks such as modeling language and then ifne-tuned for classification. The underlying model for our classification is Bert-base-uncased 2, which BERT developers provide. We did not use ten-fold cross-validation to evaluate monolingual BERT since it is more computationally expensive. Implementation details of all three proposed models are provided in GitHub repository3.
To evaluate the presented models, the organizers have provided a weighted F1 score. Among the proposed models, our top-performing monolingual BERT received a sixth-place for ofensive content recognition in the Mangalish data set and eleventh in the Tanglish data set. Table 2 and Table 3 provide the list of top-performing models with weighted F1 scores for Manglish 2https://huggingface.co/transformers/model_doc/bert.html 3https://github.com/shankarb14/dravidian-codemix and Tanglish data set respectively (The result of our proposed model is shown in bold letters). Among our proposed models, BERT outperformed other classifiers, reaching 70% accuracy for the Mangalish data set and 57% accuracy for the Tanglish data set. Finally, we compared the results of our proposed models in Table 4. We trained our proposed models on a Tanglish data set comprising 4000 comments from the training set and tested them on 940 comments from the test set. For the Manglish data set, 4000 train comments and 1000 test comments are used.
We investigated the behavior of proposed models on sample test sentences to evaluate their performance. We discovered that our best-performing model monolingual BERT classifier could accurately classify all of the test samples based on our experimental observations. However, multilingual BERT models such as mBERT+SVM and mBERT+DNN could not classify test samples 3 and 2, respectively. Table 5 summarises the results of the findings.
In our work, we presented a model submitted by our team IIITD-ShankarB for ofensive content identification in the shared task “Dravidian-CodeMixed HASOC-2021”. Our proposed work experimented with three distinct models: a machine learning-based model, a Deep Neural Network model, and a transformer-based language model. Our model is one of the top-performing models, ranking sixth on the Manglish data set and eleventh on the Tanglish data set. In future work, we can improve the eficiency of the suggested model by including domain-specific embeddings.