Internet users frequently express themselves through posts, comments, and articles. The examination of such posts/comments has recently attracted the research community's attention. Sentiment analysis and the identification of homophobic comments are two key research areas in this field. Sentiment analysis reveals that people's emotions reflect positive, negative, or mixed feelings about a certain topic or article. Further, Homophobia refers to a wide range of attitudes and feelings toward people who identify as homosexual, transgender, lesbian, gay, or queer. To encourage research in this direction, the organisers of the Dravidian LangTech shared task as part of FIRE 2022 have set two shared tasks. Task A consists of a message-level polarity detection problem, in which the given YouTube comments system has to recognise positive, negative, and mixed emotions. Task B involves detecting transphobic and homophobic YouTube comments. Our team participated in both subtasks; we worked on the Kannada dataset for sentiment analysis, and our best-performing model secured 11th place among the participating teams. For Task B, we participated in all four languages (Tamil, English, Malayalam, and Tanglish) and received 6, 6, 2, and 4th positions, respectively. In our proposed approach, we employed several Machine learning models, the Ensemble method and Deep learning models to achieve the desired result.
Social media websites, blogs, and microblogging sites have become very prominent in today’s
world, where people can easily share their thoughts and opinions on various real-time scenarios.
These websites have also become a source of all kinds of information. Naturally, these comments,
posts, and articles tend to infer diferent things for diferent people across the world. The
comments which are good for some people may not be in the best interest of others. Hence
there are various emotions on the same topic, post or issue. These sentiments can be classified
into Positive, Negative, Mixed feelings or Unknown states. Analysing each comment, post or
article in these categories is known as Sentiment Analysis. Nowadays, sentiment analysis [
LGBTQ+ community refers [
Several studies on sentiment analysis have been undertaken in recent years; however, most
of these studies have focused on high-resource languages such as English [
The rest of the article is organized as follows. The next section, 2 gives the brief overview of the existing work. Further, section 3 provides the details of the given tasks and dataset statistics. This is followed by the description of model used for experimentation in Section 4. The results are explained in the Section 5.
Several studies on sentiment analysis and the moderation of homophobic content on social media networks have been conducted; however, the majority have focused on high-resource languages such as English. To organise the related work, we divided the background study into two parts: section 2.1 provides a brief description of the model proposed for sentiment analysis, and part 2.2 describes the model proposed for homophobic content moderation. 2.1. Models proposed for Sentiment analysis [10] developed a novel framework for assessing the rating of internet reviews. The suggested method detects polarity in online reviews by combining text processing and feature extraction methods. The authors claim that their proposed strategy outperforms existing deep learning methods. [11] used code-mixed text data from social media to identify sentiment. Their study made use of two code-mixed datasets: English-Bengali and English-Hindi. They grouped the data based on the statement’s polarity conflict, such as positive, negative, or neutral. The translation and transliteration-based transformer model was developed by [12] to detect hateful comments from social media networks [13, 14, 15]. [16] presented a novel Framework for predicting discrepancies in Google App text comments and ratings using Deep Learning approaches. The framework is divided into two phases. In the first step, the polarity of reviews is predicted using a sentiment analysis algorithm. In the second step, star ratings are predicted from the text format of reviews after deep learning models have been trained on the ground truth obtained in the first phase.
To extract homophobic information from social media data, [17] first convert code-mixed text to monolingual, utilising a data augmentation and transliteration-based approach. [18] used transformer-based XLM-Roberta to identify homophobia and transphobia data.TF-IDF vectorizer combined with SVM model is used by [19] to identify homophobia content. The number of monolingual and multilingual transformer models were experimented with data augmentation by [20] for homophobia detection.
DravidianLangTech organised the shared task on sentiment analysis and homophobia
identification in YouTube comments [
The datasets for the competitions were made available in phases. Task A and task B training and validation datasets were released initially; later, Test data was made available. The dataset is collected from comments on popular YouTube channels. The dataset contains two fields: Text and Label. The complete statistics of the data we investigated in our work are presented in Table 1,2. 4. Methodology The current paper used the multi-class classification approach for sentiment analysis and homophobic and transphobic text detection. Several conventional machine learning models, and ensemble methods were used to realise the goal. A detailed description of all the methods is presented in the subsection below.
The datasets were preprocessed before being fed into the models. The preprocessing is carried out on the Text field. The numbers, punctuation, and symbols have been deleted from the text because they do not help us predict the label. We also deleted white spaces; finally, the lower casing of text is performed to avoid redundant data. The cleaned texts are then tokenized and encoded into a series of token indexes.All of this preprocessing was done with the help of the NLTK toolbox from the Python library 1. Furthermore, TF-IDF vectorization (n-gram vectors) is performed, and vectorized data is used as input for diferent models. We also applied SMOTE on vectorised data to balance the overall dataset.
We used diferent ensemble techniques, and traditional machine learning classifiers in the proposed approach to predict the outcomes. The following sections provide comprehensive details of each of these models.
Initially, we experimented with diferent conventional machine learning models such as Logistics Regression, Passive Aggressive classifier, Support vector machine (SVM), Random Forest and Naïve Bayes to classify the text into their respective categories. We have used default parameters provided by the sci-kit-learn library to train the models. The input for all these models was taken from TF-IDF vectors created from the cleaned text. The model was developed using Python’s sci-kit-learn library 2.
We employed an ensemble setup in the model to increase the performance of classic machine learning models. Three diferent ensemble approaches were used to classify the text: gradient ensemble, stacking ensemble, and model selection ensemble. As weak learners, the stacking ensemble included logistic regression,k nearest neighbour classifier, decision tree classifier, Support vector Machine (SVM), and naive Bayes classifier. The logistic regression, random forest classifier, and SVM were employed in the model selection and gradient boosting. The TF-IDF vectoriser is used as the input for all of these models. The detailed Architecture of the proposed model is illustrated in Fig 1.
All experiments were conducted in the Keras and sklearn environments. To read the datasets, we utilised the pandas library. The dataset was prepared using Keras preprocessing methods and nltk library. Using sentiment and homophobic data provided by the task organisers, we used K-fold cross-validation to train our proposed models. Experimental trials are used to select the hyperparameter value K=5. Table 3 illustrates the findings of the sentiment analysis performed on the Kannada dataset, and Table 4 provides homophobia results.
For sentiment analysis using the Kannada dataset the best model was found to be the model using stacking ensemble with the accuracy of 0.515. The stacking ensemble consisted of Logistic Regression, KNeighbors Classifier, Decision Tree Classifier,SVM and Gaussian Naive Bayes as the base models and Logistic Regression as the meta learner model. Diferent models were used to detect homophobia for diferent datasets. The stacking ensemble produced the best results on the Tamil dataset, with an accuracy of 0.762. In stacking ensemble Logistic Regression, K nearest neighbours Classifier, Decision Tree Classifier, SVM, and Gaussian Naive Bayes were included as base learners, with Logistic Regression serving as the meta learner model. Similarly, the English dataset has given better results using the stacking ensemble model with an accuracy of 0.966. On the other hand, the Malayalam dataset performed best with the Passive Aggressive classifier, with an accuracy of 0.927. The model chosen for the Tamil English dataset was gradient boosting, which produced an accuracy of 0.891.
The organisers provided a weighted F1 score to evaluate the presented models. Our topperforming Stacking ensemble model was ranked 11th and 6th among the participating teams on Kannada, Tamil, and English datasets. Similarly, Passive Aggressive and gradient boosting performed better on Malayalam and Tanglish data, ranking second and fourth, respectively. Table 5 illustrates the final ranking of our proposed models among the participating teams.It also includes the best F1 scores achieved among the participating teams.
In our work, we presented a model submitted by our team for Sentiment analysis and Homophobia content identification on You Tube comments in the Fire 2022 shared task. Our proposed work evaluated two distinct models: a machine learning-based model and an ensemble setup with machine learning classifiers as base learners. The experimental findings demonstrate that ensemble models outperform diferent baseline models for stance detection. We can increase the eficiency of the suggested modes by using context-aware domain-specific embeddings. [10] G. S. Budhi, R. Chiong, I. Pranata, Z. Hu, Using machine learning to predict the sentiment of online reviews: a new framework for comparative analysis, Archives of Computational Methods in Engineering 28 (2021) 2543–2566. [11] S. Ghosh, S. Ghosh, D. Das, Sentiment identification in code-mixed social media text, arXiv preprint arXiv:1707.01184 (2017). doi:https://doi.org/10.48550/arXiv.1707.01184. [12] S. Biradar, S. Saumya, et al., Fighting hate speech from bilingual hinglish speaker’s perspective, a transformer-and translation-based approach., Social Network Analysis and Mining 12 (2022) 1–10. [13] S. Saumya, A. Kumar, J. P. Singh, Ofensive language identification in dravidian code mixed social media text, in: Proceedings of the first workshop on speech and language technologies for Dravidian languages, 2021, pp. 36–45. [14] A. K. Mishra, S. Saumya, A. Kumar, Iiit_dwd@ hasoc 2020: Identifying ofensive content in indo-european languages., in: FIRE (Working Notes), 2020, pp. 139–144. [15] A. Kumar, S. Saumya, J. P. Singh, Nitp-ai-nlp@ hasoc-fire2020: Fine tuned bert for the hate speech and ofensive content identification from social media., in: FIRE (Working Notes), 2020, pp. 266–273. [16] S. Sadiq, M. Umer, S. Ullah, S. Mirjalili, V. Rupapara, M. Nappi, Discrepancy detection between actual user reviews and numeric ratings of google app store using deep learning, Expert Systems with Applications 181 (2021) 115111. [17] B. R. Chakravarthi, A. Hande, R. Ponnusamy, P. K. Kumaresan, R. Priyadharshini, How can we detect homophobia and transphobia? experiments in a multilingual code-mixed setting for social media governance, International Journal of Information Management Data Insights 2 (2022) 100119. [18] J. García-Díaz, C. Caparrós-Laiz, R. Valencia-García, Umuteam@ lt-edi-acl2022: Detecting homophobic and transphobic comments in tamil, in: Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, 2022, pp. 140–144. [19] N. Ashraf, M. Taha, A. Abd Elfattah, H. Nayel, Nayel@lt-edi-acl2022: Homophobia/transphobia detection for equality, diversity, and inclusion using svm, in: Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, 2022, pp. 287–290. [20] V. Bhandari, P. Goyal, bitsa_nlp@lt-edi-acl2022: Leveraging pretrained language models for detecting homophobia and transphobia in social media comments, arXiv preprint arXiv:2203.14267 (2022). [21] K. Shumugavadivel, M. Subramanian, P. K. Kumaresan, B. R. Chakravarthi, B. B, S. Chinnaudayar Navaneethakrishnan, L. S.K, T. Mandl, R. Ponnusamy, V. Palanikumar, M. Balaji J, Overview of the Shared Task on Sentiment Analysis and Homophobia Detection of YouTube Comments in Code-Mixed Dravidian Languages, in: Working Notes of FIRE 2022 - Forum for Information Retrieval Evaluation, CEUR, 2022.