Social media platforms have seen a significant rise in user engagement in recent years. More and more people are expressing their views and ideas on social platforms. There is an ardent need to develop an accurate system to classify text based on sentiments. In this paper, our team IRLab@IITBHU presents a solution architecture submitted to the shared task “Sentiment Analysis and Homophobia Detection of YouTube Comments in Code-Mixed Dravidian Languages" organized by DravidianCodeMix 2022 at Forum for Information Retrieval Evaluation (FIRE) 2022. to reveal how sentiment is expressed in code-mixed scenarios. For task A: we used mBERT model and word-level language tag to classify YouTube comments into positive, negative, neutral, or mixed emotions. And for Task B: we performed basic preprocessing steps and built mBERT model to identify homophobia, transphobia, and non-anti-LGBT+ content from the given corpus. For Task A, our proposed system achieved the best result, securing the first rank for Malayalam-English and Kannada-English code-mixed datasets with the 1 score of 0.72 and 0.66 respectively.
scripts. People are generally inclined to write texts in roman-scripts. We have systems trained on monolingual data, but those specialized systems will fail to give us a satisfactory result on code-mixed data. Hence we need a model which can analyze code-mixed textual data available to us.
Sentiment analysis refers to a sub-task in Natural Language Processing which uses computational methods to analyze, process, and better understand the emotions of the users behind a text or interaction. It categorizes users’ opinions into various classes of sentiments. It allows organizations to gain insights from a vast volume of unstructured data and help them remodel their strategies that are better focused on the target market. Companies like Amazon and Flipkart can analyze their product reviews to understand the customer’s responses and leverage that information to improve their performance.
Given the freedom to express views and beliefs publicly, there comes a challenge to regulate the ofensive content present on the internet. A certain section of the community, like LGBTQ+, is being targeted and subjected to many forms of abuse on social media platforms. In a survey of more than 1,100 LGBT+ people for Galop’s Online Hate Crime Report 2021 2, 64% reported experiencing antiLGBT+ violence or abuse. The daily experience of violence or abuse was reported by 16% of the respondents. Verbal abuse was the most prevalent (92%) and was followed by online abuse (60%) in frequency. Recent studies have shown how this style of abuse can hamper their mental state and hurt them. Hence there is a need for a system to identify the contents which are homophobic or transphobic in nature.
DravidianCodeMix organized the shared task on Sentiment Analysis and Homophobia detection of
YouTube comments in Code-Mixed Dravidian Languages [
In this paper, we propose a method to analyze the real-world data and perform sentiment analysis on Code-Mixed datasets of YouTube comments in Dravidian languages (Tamil-English, MalayalamEnglish, and Kannada-English). Along with the identification of Homophobic and transphobic speech contents at diferent levels.
The rest of the paper is arranged in the following fashion. First, we discuss the related work in section 2. Section 3 describes the dataset. The proposed methodology, which includes pre-processing and model architecture, is described in Section 4. In Section 5, we report our results and analysis. Finally, we conclude in Section 6.
This section summarizes earlier work on sentiment analysis and homophobia detection.
2https://galop.org.uk/wp-content/uploads/2021/06/Galop-Hate-Crime-Report-2021-1.pdf
Sentiment analysis is one of the top study fields targeted to analyze people’s feelings and views on
a specific subject. Numerous studies have been conducted in various languages, focusing on
monolingual languages. Code-Mixed languages, however, are an exception to this rule. Few code-mixed
language pairings have been attempted in the past. In addition to sentiment analysis, the Forum
for Information Retrieval Evaluation (FIRE) has carried out many code-switching tasks. The tasks
include Code-Mixed Question Answering, sentiment analysis for code-mixed Indian languages [
There are not many datasets that combine Kannada and English for sentiment analysis. A
KannadaEnglish [
Recently, it was widely noted worldwide that postings or comments on social media involving hate
often devolve into violence. Numerous methods have been developed to identify hate speech, and
automatic hate speech recognition has attracted much interest. SemEval, HASOC, and HatEval are
just a few of the latest shared tasks used to identify hate speech on social media. Hate speech language
detection can be formulated as a classification task. In general, it is a binary classification problem that
determines if something is hateful or not. In the future, though, if there is a more detailed description
of hate speech, it may be handled as a multi-class classification problem. The methods are divided into
two categories. One is the use of classifiers in machine learning models such as SVM [
Comments that are homophobic or transphobic are typically seen to be hate speech directed
toward LGBT+ people. Concerns about this occurrence are mounting. The first dataset on
homophobia and transphobia in multilingual comments in Tamil, English, and Tamil-English was created by
Chakravarthi et al. [
The organizers provided two datasets of Training, Development, and testing for two separate shared tasks. The first share task is sentiment detection of code-mixed text in Dravidian languages. There was three pair of code-mixed datasets, Tamil-English, Malayalam-English, and Kannada-English. The training dataset consists of 35,656 Tamil-English [13] and 6,212 Kannada-English [14] and 15,880 Malayalam-English [15] YouTube video comments. Table 1 summarizes the statistics of the training, development, and testing data sample collection and their distribution by class.
Detecting homophobia, transphobia, and non-anti-LGBT+ content from the provided corpus is the
goal of the second share task [
Data preprocessing was kept to a bare minimum to keep it adaptable for both the shared tasks. We saw in the task report [16], which was previously given that eliminating continuous repetitions did not result in any appreciable performance diferences. This year, we limited the repeated characters to two contiguous repeats. This reduces the whole sequence length to below 512. Then, we eliminate hashtags, punctuation, URLs, and numbers and mentions that do not have a clear semantic significance. We replace emojis with their proper semantic text. And stripped of any white spaces and extra spaces.
Non-anti-LGBT+ content
Homophobic
Transphobic Non-anti-LGBT+ content
Homophobic
Transphobic Non-anti-LGBT+ content
Homophobic
Transphobic Non-anti-LGBT+ content
Homophobic Transphobic
Total Class Total Class Total Class Total
The most well-known NLP technology in recent memory is likely word embedding. It captures a word’s semantic characteristics. We used bert-base-multilingual-cased (mBERT) pre-trained models 3 to get a vector as an embedding for the sentence that we can use for classification.
mBERT is A transformer architecture that is an encoder-decoder network that uses self-attention on the encoder side and attention on the decoder side. The models are pre-trained on large text corpora such as Wikipedia and produce state-of-the-art results with necessary fine-tuning on several downstream tasks. The contextual language representation model BERT (Bidirectional Encoder Representations from Transformers) has been used for the downstream task of code-mixed language identification. Multilingual BERT or mBERT ( bert-base-multilingual-cased4) is pre-trained on cased text in the top 104 languages with the largest Wikipedias and has a total 179M parameters with 12 transformers blocks, 768 hidden layers and 12 attention head. This model takes a special 3https://huggingface.co/transformers/pretrained_models.html 4https://github.com/google-research/bert/blob/master/multilingual.md [CLS] token as input first, followed by a sequence of words as input. It then passes the input to the next layer. [CLS] here stands for Classification. Each layer applies self-attention and passes the result through a feedforward network to the next encoder.
For the implementation, HuggingFace’s transformers library was utilized. A Python package called HuggingFace transformers ofers pre-trained and adaptable transformer models that may be used for various NLP tasks. The implementation environment is the PyTorch library, which supports GPU computation. The mBERT models were run using Google Colab. We trained our classifier across 2-4 epochs with a batch size of 32. The AdamW optimizer is used, and the dropout value is set at 0.1. The learning rate is 2e-5. For tokenization, we utilized the hugging face transformers’ pre-trained BERT tokenizer. We utilized the HuggingFace library’s BertForSequenceClassification module for tinkering and sequence classification.
Initially, we considered task A as a multi-class classification problem. But after carefully studying the dataset description provided by the organizers, we realized that the given tag "Not in intended Language" is independent of the remaining tags. According to the original description of the dataset, any statement which does not contain words from the specified language is to be labeled as not<language>. In this manner, the above Task A could be considered as a multi-label classification where statements labeled as not-<language> can further be classified into positive, negative, neutral, or mixed emotions. But according to the given shared task description, we approached it as a multiclass classification problem and simultaneously classified these comments into two classes, namely language and not-<language>.
To do a binary classification between language and not-<language>, we deployed our language identification (LID) module, which provides word-level language identification for the text. To find the LID for each word, we used Googletrans 5 which is a free and unlimited python library that implemented Google Translate API. It uses the Google Translate Ajax API to make calls to such methods as detect and translate. Based on the LID of the words, we built a rule-based system that labels the statement as not-<language> if the given statement does not contain any word in the specified language else, label it as language. After that, we utilized a function similar to the OR function that depends on the output of the LID module. The mBERT prediction will remain the same if the output is <language>. Otherwise, not-<language> will be used in its replacement.
After that, we combined the predictions from two modules, mBERT and the Rule-based system (See Figure 1).
For task B, we had to develop a system to predict whether the given comments are homophobic/transphobic in nature. We used the mBERT model to classify the comments into two classes Homophobic and non homophobic.
In this section, we present the evaluation of our model and submitted results for both tasks: sentiment analysis and homophobic identification for Dravidian languages.
The performance of our proposed models is examined using evaluation measures including accuracy, recall, macro averaged F1-score, and weighted average F1-score for sentiment analysis of code-mixed text in Dravidian languages. The organizers gave the test data for the three Dravidian languages. Based on the training and validation data, we fine-tuned our model, and we then submitted our prediction file for the test data. First, we have improved the mBERT model to create the system and forecast the sentiment polarity for the Tamil-English, Malayalam-English, and Kannada-English 5https://pypi.org/project/googletrans/
mBERT Positive not-<language>
Negative Mixed_feelings unknown_state
Input text
Language Identification <language> not-<language> Positive
Negative
Mixed_feelings unknown_state not-<language> languages, as indicated in this study [17]. For all three language pairings, we have demonstrated positive outcomes. Following that, we utilized a rule-based approach that divided everything into two categories: language and not-<language>. After receiving this binary classification, we combine the predictions with the mBERT prediction and then provide the final prediction. Table 3 and 4 display the performance of the test outcomes for the mBERT model and mBERT + Ruled-based systems for the two languages. For Kannada-English and Malayalam-English test data we got 0.66 and 0.72 1 score respectively and stand top of the rank list for both of the language pairs. Here we observed that our approach outperforms existing mBERT based model for both language pairs. We can find the class wise performance improvement in table 5 for Kannada-English and table 6 for Malayalam-English pair.
For Homophobic detection our proposed model did not perform as expected. And we could only achieve the 1 score of 0.333 for Tamil-English (See Table 7), 0.289 for Tamil (See Table 8), 0.337 for English (See Table 9) and 0.427 for Malayalam (See Table 10).
We also looked at the confusion matrix, which is depicted in Figure 2. This was a crucial analytical tool that allowed us to examine which classes are misclassified by which classes. Figures 2a and 2b demonstrate how well our model categorizes not-Kannada and not-Malayalam. Only one nonMalayalam language qualifies as positive, while the remaining three are classified as unknown_state. Figure 2d shows that our approach could not categorize the transphobic class. There were only six labels of transphobic in the training data. Therefore, this is probably the reason for that. If we see all four confusion matrices for task B (Figure 2c, 2d, 2e and 2f), We discovered that our model predominantly misclassified data as non-anti-LGBT+. Furthermore, the enormously unbalanced dataset may be the cause. Consequently, our model overfitted and labeled the majority of the classes as non-antiLGBT+.
Govt should take action Police pls take action Government take this type of peoples civiliar actions
neutral neutral neutral
not-Tamil not-Tamil not-Tamil
This paper presents the methodology for identifying the sentiment polarities and Homophobia detection from YouTube social media comments in Tamil, Malayalam, and Kannada code-mixed languages. Our group only employed a few preprocessing methods. We experimented with a pre-trained Multilingual BERT model with input variations for shared tasks A and B in all languages. It is evident from the evaluation that optimizing mBERT architecture receives better scores. For job A, our model mBERT + Ruled-based systems works admirably in two languages. Our model has achieved the leading position for both Kannada-English and Malayalam-English language pairings. Our model under (a) mBERT + Ruled-based systems
(b) mBERT + Ruled-based systems (c) mBERT
(d) mBERT (e) mBERT (f) mBERT performed on task B. In Tamil, English, English-Tamil, and Malayalam, we placed fourth, fifth, sixth, and eighth rank, respectively. After addressing the class imbalance, we may use several deep learning methods to enhance task B’s performance (homophobia detection). To prevent miss-classification, we will expand this work to additional languages and enhance eficiency by managing the indirect code-mixed comments. comments, in: Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 369– 377. URL: https://aclanthology.org/2022.ltedi-1.57. doi:10.18653/v1/2022.ltedi-1.57. [13] B. R. Chakravarthi, V. Muralidaran, R. Priyadharshini, J. P. McCrae, Corpus creation for sentiment analysis in code-mixed Tamil-English text, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 202–210. URL: https://www.aclweb.org/anthology/2020.sltu-1. 28. [14] A. Hande, R. Priyadharshini, B. R. Chakravarthi, KanCMD: Kannada CodeMixed dataset for sentiment analysis and ofensive language detection, in: Proceedings of the Third Workshop on Computational Modeling of People’s Opinions, Personality, and Emotion’s in Social Media, Association for Computational Linguistics, Barcelona, Spain (Online), 2020, pp. 54–63. URL: https://www.aclweb.org/anthology/2020.peoples-1.6. [15] B. R. Chakravarthi, N. Jose, S. Suryawanshi, E. Sherly, J. P. McCrae, A sentiment analysis dataset for code-mixed Malayalam-English, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 177–184. URL: https://www.aclweb.org/anthology/2020.sltu-1.25. [16] S. Chanda, S. Pal, Irlab@ iitbhu@ dravidian-codemix-fire2020: Sentiment analysis for dravidian languages in code-mixed text (2020). [17] S. Chanda, R. Singh, S. Pal, Is meta embedding better than pre-trained word embedding to perform sentiment analysis for dravidian languages in code-mixed text?, in: FIRE, 2021.