In the recent advancement of technology and social media hatred towards LGBTQ+ community is also growing. Homophobia/transphobia refers to the actions resulting in threat, dread, dislike, discomfort or mistrust of lesbian, gay, transgender or bisexual person [1]. Social media, as it provides medium for communication, allowing the users to express their views, ideas and feelings on anything at any time. The power of sharing resources, materials to support their views are also enabled using social media platforms [2] [3]. The abundance of data available online can enable researchers to use natural language processing to interpret, quantify, and monitor the user behavior, propagation of information across different communities and events influence by these online information [4].
Internet is home to a wide verity of racist, sexist, homophobic, trans phobic and all sorts of unpleasant content.The increase in the quantity of such contents have appeared as a problem for online communities [5]. The wide verity of data available on social media platforms such as YouTube, Facebook and others are ever changing and are influencing the way people think, talk and connect with each other.Social media platforms also provide a great avenue to venture into the darker side of internet, like, share and support the violent, sexist, homophobic content creation and sharing [6].
With increasing content available on the internet, the computer scientists, linguists and researchers have an opportunity to build and use automated solutions that can mitigate or ban anti LGBTQ+ harmful content, and try to make internet a place of equality, diversity and inclusion. While much work has been put into the domain of aggression identification [7], misogyny [8] [5], and racism [9], homophobic or transphobic verbal abuse, on the other hand, was given as far less important than racist or other prohibited issues Recent advancements in the attention mechanism used in transformers, which are becoming very popular in low resource Dravidian languages like, Tamil, Malayalam, Kannada among others. Lack of language carpus available to train makes it difficult to train the models without using embedding where transformers are acting as a solution. Bert [10] and XLnet [11], which are the two highly popular models used for the text classification and which are pose to have drawbacks which are overcome with MPNet.
Wast availability of data on the internet attracted many researchers and computer scientist to develop and research possible solutions to tackle the hatred towards LGBTQ+ communities. One of the early studies towards identifying offensive comment identification in dravidian languages (Tamil) [12] [13] followed by DravidianLangtech [14] shed light towards possibilities in bringing equality and diversity for LGBTQ+ people who are also ill-treated in these part. Dataset for HASOCDravidianCodeMix which consisted of 4000 comments which were collected from twitter and other social media platforms. Similar work DravidianLangTech comprised of 30 thousand YouTube comments, which were annotated by multiple volunteers. Both these datasets are code mixed Datasets. Based on these two datasets k [14]. Inspired by these works, our previous work on dravidian code mix dataset (troll-meta) [15], created a hybrid deep learning model which performed classification of given images to one of the 2 classes. The work focused on classification of data into offensive speech and neutral ones.
Works by Ljubešić et al. [16] constructed lexicons of several languages including Croatian, Dutch and Slovene. And using these lexicons to identify texts containing socially unacceptable words towards topics of migrants and LGBTQ+. Even though this is a great work, but it fails to meet the end goals as it was in the early stages of research, it lacks confidence in classification task.
DravidianCodeMix, a recent work proposes a multilingual model which tries to establish a baseline model to conduct further research [1]. The corpera included comments collected from Youtube, belonging to 4 major dravidian languages, Kannada, Tamil and Malayalam. The data set has Kannada-English, Tamil-English and Malayalam-English datasets.Which are annotated by human volunteers.
Our work focus on sentiment analysis and classification of homophobic and transphobic comments that are collected by YouTube.
The dataset is provided as part of shared task on "Sentiment Analysis and Homophobia detection of YouTube comments in Code-Mixed Dravidian Languages" [17]. The data set consists of annotated data for sentiment analysis and offensive language identification for a total of more than 60 thousand individual comments on YouTube videos.
The dataset count for individual languages and tasks are tabulated in Tables 1, 2, 3 and 4. The dev dataset for Homophobia/Transphobia detection contained 526 Non-anti-LGBTQ+ content, 103 homophobic and 37 transphobic comments for Tamil, 532 non-anti-LGBTQ+ content, 58 homophobic and 2 transphobic comments for English, for Malayalam 692,133 and 41 comments for Non-anti-LGBTQ+ content, homophobic and transphobic labels respectively. Tamil-English dev data consisted of 862 non-anti-LGBTQ+ content, 66 homophobic and 38 transphobic comments.
The approach to the solution started with cleaning the data, making it free from special characters, converting the Kannada-English, Tamil-English and Malayalam-English to lowercase.
Emoji's as the name suggests, which is used to express emotions in the form of graphics, images, pictogram or ideogram embedded with text. We consider these as one of the major driver in finding emotions such as sarcasm, sadness, happiness etc. They play major role in finding or classifying emotions and analysing the sentiments in the given comments. We used the Python library (https://pypi.org/project/emoji/) to convert the emoji's to text.
For Example: Original: Idha pathutu road la students kathi vaichi kitu sanda poduvanunga 🤦 Translated: Idha pathutu road la students kathi vaichi kitu sanda poduvanunga Face Palm The study involves 2 tasks which are Sentimental Analysis and Homophobia-Transphobia Detection which will be referred to as Task A and Task B henceforth. Task A involved 3 different languages which are Tamil, Kannada, and Malayalam whereas for Task B, 4 languages i.e. Tamil, English, Malayalam, and Tamil-English(combination of both, often called as Tanglish in colloquial language) were in consideration. For both the tasks, two methods were experimented. The first method involved building a text classifier using MP-Net [19] and the second method involved word generating embedding using fastText [20], followed by dimension-wise averaging, finally classifying using LightGBM [21] to obtain the required results.
For LightGBM, 15 num_leaves, min_child_weight of 1e-1, subsample of 0.8 and random state of 42 give the better results for the task of Homophobia/Transphobia detection.
Similar to this, MPNet trained with a learning rate of 2e-5, weight decay of 0.01 and batch size of 8 for the task of sentiment analysis.
For Sentimental Analysis, MP-Net method was used, and for Homophobia-Transphobia Detection, for Tamil and Malayalam, fastText+LightGBM method was used whereas for English and Tamil-English, MP-Net was used.
The research activity performed for sentiment analysis as well as classifying the comments into homophobic/transphobic comments or not. The model involved two different machine learning model for the classification problem, part-A is a decision tree based LightGBM [21], where as the part-B was a hybrid model of masked language modeling and permuted language modeling [19].
To analyze the results of the model, a confusion matrix was constructed, and the weighted f1 is calculated.
The results for the conducted research activity are tabulated in the table 5 and 6.
The research activity carried out shed the light towards the setbacks faced during training and evaluation steps. One of the important reason for the performance of the models is lack of data pertaining to dravidian languages compared to other, as it can be seen in the result of classifying homophobia/transphobia. Wherein, the rank in English task is 1, which is because the MPNet is trained on larger English language data corpus. That is the reason why we explored fastText as an alternate option. The other models in the same task were trained on fastText+LightGBM, even though fastText were also trained on Tamil/Malayalam language corpus, due to the differences in colloquial language versus formal language in which the models were trained on, the results were poor. Tamil-English performed poor in ranking compared to others, which is likely as pre-trained models seldomly comes across language code-switch, thus failing to provide better representation embedding. Dataset size is another aspect that we analyze the results. The size of the dataset available for Tamil language is more compared the others, due to which performance is better, which can be clearly seen in the results. With the minimum precision, Malayalam language had least quantity of data next to Kannada. Even with embeddings from transformers, the quantity of data were not enough for better generalization.
In terms of of improvement, pre-training or fine-tuning these aforementioned models on the available dataset, will significantly increase the quality of predictions. Also we will have to explore other methodologies to handle low-resource constraints and strive to achieve best results.
We experimented with both fastText along with LightGBM and MPNet which were able to provide some improvements over the baseline models. Even with considerable improvements the models experienced some shortcomings [22] [18].
In the future works we are considering building custom transformers and enhanced architectures to gain a better results.