Social networking as become the irreplaceable platform like never before. We are more up to date with the issues whether they are good or bad around the globe. The oversharing happening on social media leads to cyberbullying. In this study we are going to compare and analyze methods for comment-level text polarity classification task using the Dravidian-CodeMix-FIRE2021 data-set. Techniques such as TFIDF, Count vectorizer and multilingual transformer based encoded features. The features are trained with diferent machine learning models such as Multi layer perceptron, SVM, Random forest. Our models scored F1 scores of 0.588, 0.69 and 0.63 for the Tamil-English, Kannada-English and the MalayalamEnglish code-mixed test data respectively.
The paper is organized as follows: The section 2, explains work related with sentiment analysis. The dataset descriptions are given in Section3.1 Section 3.2 details the experimental setup and various features used for this task. Section 4 provides a subjective analysis and comparison of the performance of various models on the development and test data. Finally, Section 5concludes the paper.
In the recent years,Sentiment analysis on multilingual code mixing text is an active research area [6]. For Kannada - English code mixed sentiment analysis text, distributed representation is used[7]. They have compared their results with diferent machine learning and deep learning techniques. The authors [8] reported that machine learning and neural networks algorithms achieves better accuracy for code-mixed social media for the systems relying on hand-crafted features. For sentiment and ofensive language detection for Kannada language KanCMD [ 9] dataset is used.
This data set consists of YouTube comments with emotional polarity tags at the message level. The categories of assignments include positive, negative,mixed feelings, unknown-state or if comments are not in the language of the label. This becomes a multi-class classification task. The average sentence length contained in single comments is 1. But there is a certain imbalance in the class because it simulates real life scenarios. More details about the dataset is found in [5]and [4]. The data distribution is tabulated in Table 1.
The experimental structure of the task can be divided into two stages: feature extraction stage and classifier stage. In the feature extraction stage, technologies such as count vectorization, TFIDF vectorization [10], multilingual transformer based encoding etc. were analyzed, and diferent classifiers such as support vector machine, logistic regression, multilayer perceptron, naive Bayes, Knearest and random forest classifier were compared. The features extracted from the first stage are used to train the machine learning model in the second stage, and their performance is compared using the F1 score and the accuracy score. The measurement methods in the scikit learn package are used to measure performance. The code for the sentiment analysis task is available here 1
The content of the text annotation is a mixture of various languages, their grammar and switching between diferent symbols. It becomes dificult to capture the consistent intensity of comments using existing pre-trained models. Therefore, the model of words and characters based on bag-of-words is realized and analyzed by changing the range of ngram. The ngram ranges of 2-3, 1-5 and 2-3 gave the better results for Tamil-English,Malayalam-English and KannadaEnglish corpus respectively. Subtask gives the best results in devset. The term frequency inverse document frequency model helps to assign a weight of less than to the mediocre words in the corpus. This technique emphasizes unique terms in the corpus more than repeated words and provides a better model.
The YouTube comments selected for the research contained text from the fused English and Dravidian languages. This becomes the main issue to consider when applying the mono language pre-training model and adjusting it for this specific task. However, by unsupervised selection of pre-trained models in a large number of languages, can fine-tune these multilingual models to fit well with Codemix applications. Since the multilingual model fastText and BERT [ 11] have shown fruitful results, is considered in this experiment. The sentence transformers such as paraphrase-xlm-r-multilingual-v1 ,stsb-xlm-r-multilingual, paraphrase-multilingual-mpnetbase-v2 encodes the sentence into 768 dimensional dense vector space [12]. These sentence transformers were used in Tamil-English, Malayalam-English code mixed text. The dimension for distiluse-base-multilingual-cased-v2 and distiluse-base-multilingual-cased-v2 is 512 used in Kannada-English code mixed text.
On account of all the analyzed models the TFIDF gives the best performance for the TamilEnglish code mixed text for development data, if we consider the Malayalam-English Corpus that TFIDF and countvectorizer techniques generated equal performance and if we consider Kannada-English Corpus TFIDF gives the best performance. The performance details for each model on development set is listed in the Tables 2,3, 4.
From Table 2, it has been noted that count vectorizer with MLP and SVM is giving equal performance using F1-score.The multilingual transformer model’s performance is lower than TFIDF and count vectorizer features.
From Table 3, it has been noted that count vectorizer with MLP and TFIDF with MLP and SVM is giving equal performance using F1-score.The multilingual transformer model’s performance is lower than TFIDF and count vectorizer features.
From Table 4, it has been noted that TFIDF with Random forest and SVM is giving better performance then other approaches for Kannada - English code mixed text.The multilingual transformer model’s performance is lower than TFIDF and count vectorizer features.
The performance of code-mixed corpus using test data is tabulated in Table 5. The results submitted for this task where the best models stood 1st, 8th and 11th ranks in the Kannada, Malayalam and Tamil tasks respectively.
This proposed work summarizes the machine learning techniques used for sentiment analysis in the last periods. The impact of applying data transformation can improve the implementation of the classification method, but the type of transformation depends on the dataset and the language it contains. Therefore, check the details, select characteristics, apply transformation and filter the least relevant data, generalize and the machine learning methods are efective, because the computers of today have limitations and cannot process all the data without previously processing any formal review. In this study, we have analyzed a variety of feature extraction techniques and conclude that the Count, TFIDF based vectorization, and multilingual transformer encoding technique performs well on code-mix polarity labeling task. With these features, we reach a weighted F1 score of 0.588 for the Tamil-English task, 0.69 for the Malayalam-English task and 0.63 for the Kannada-English tasks respectively. [2] B. R. Chakravarthi, P. K. Kumaresan, R. Sakuntharaj, A. K. Madasamy, S. Thavareesan, P. B, S. Chinnaudayar Navaneethakrishnan, J. P. McCrae, T. Mandl, Overview of the HASOC-DravidianCodeMix Shared Task on Ofensive Language Detection in Tamil and Malayalam, in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021. [3] R. Priyadharshini, B. R. Chakravarthi, S. Thavareesan, D. Chinnappa, T. Durairaj, E. Sherly, Overview of the dravidiancodemix 2021 shared task on sentiment detection in tamil, malayalam, and kannada, in: Forum for Information Retrieval Evaluation, FIRE 2021, Association for Computing Machinery, 2021. [4] B. R. Chakravarthi, N. Jose, S. Suryawanshi, E. Sherly, J. P. McCrae, A sentiment analysis dataset for code-mixed Malayalam-English, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 177–184. URL: https://aclanthology.org/2020.sltu-1. 25. [5] B. R. Chakravarthi, V. Muralidaran, R. Priyadharshini, J. P. McCrae, Corpus creation for sentiment analysis in code-mixed Tamil-English text, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 202–210. URL: https://aclanthology.org/2020.sltu-1.28. [6] B. R. Chakravarthi, R. Priyadharshini, V. Muralidaran, S. Suryawanshi, N. Jose, E. Sherly, J. P. McCrae, Overview of the track on sentiment analysis for dravidian languages in code-mixed text, in: Forum for Information Retrieval Evaluation, 2020, pp. 21–24. [7] K. Shalini, H. B. Ganesh, M. A. Kumar, K. P. Soman, Sentiment analysis for code-mixed indian social media text with distributed representation, in: 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI), 2018, pp. 1126– 1131. doi:10.1109/ICACCI.2018.8554835. [8] P. Mishra, P. Danda, P. Dhakras, Code-mixed sentiment analysis using machine learning and neural network approaches, CoRR abs/1808.03299 (2018). URL: http://arxiv.org/abs/ 1808.03299. arXiv:1808.03299. [9] A. Hande, R. Priyadharshini, B. R. Chakravarthi, KanCMD: Kannada CodeMixed dataset for sentiment analysis and ofensive language detection, in: Proceedings of the Third Workshop on Computational Modeling of People’s Opinions, Personality, and Emotion’s in Social Media, Association for Computational Linguistics, Barcelona, Spain (Online), 2020, pp. 54–63. URL: https://aclanthology.org/2020.peoples-1.6. [10] A. B. Nitin Nikamanth, B. Bharathi, J. Bhuvana, Ssncse_nlp@dravidian-codemixifre2020:sentiment analysis for dravidian languages in code-mixed text, in: Working Notes of FIRE 2020- Forum for Information Retrieval Evaluation, CEUR, 2020, pp. 4–12. [11] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [12] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2019. URL: http://arxiv.org/abs/1908.10084.