Sentiment analysis is highly important in social media monitoring since it helps us to see how the general population feels about a certain issue. Several studies have been published in recent years that attempt to extract sentiment from social media messages. However, the majority of the work is verified using just English language datasets. Machine learning algorithms do not perform equally well when social media posts are written in multilingual and code-mixed script. This paper presents an ensemble-based model to classify Kannada-English, Malayalam-English, and Tamil-English social media postings into five diferent sentiment classes using character-level TF-IDF features as input. The proposed ensemble-based model achieved the weighted 1-scores of 0.62, 0.73, and 0.62 for KannadaEnglish, Malayalam-English, and Tamil-English datasets, respectively. The code for the proposed models is available at: https://github.com/Abhinavkmr/Dravidian-Sentiment-Analysis-.git
Sentiment analysis helps in the recognition of opinions or responses on a given topic. Due to its
enormous influence on companies such as e-commerce, recommendation systems, hate speech
detection [
The sentiment analysis of code-mixed language has recently caught the interest of the research community [8, 9]. Kumar et al. [9] proposed a hybrid CNN and Bi-LSTM Network to classify social media posts into diferent sentiment classes. Mahata et al. [ 10] proposed bi-directional LSTM with language tagging to classify Tamil-English and Malayalam-English code-mixed social media posts into diferent sentiment classes. Sharma and Mandalam [ 11], on the other hand, employed sub-word level representation to capture text sentiment and implemented an LSTM network to classify Tamil-English and Malayalam-English social media posts into the diferent polarity classes. Patra et al. [ 12] presented a model for Bengali-English code mixed data using a support vector machine with character n-grams features. To extract emotions from Hinglish and Spanglish (Spanish + English) data, Advani et al. [13] utilised logistic regression using handcrafted lexical and semantic features. Similarly, On Hinglish data, Goswami et al. [14] presented a morphological attention model for sentiment analysis.
This paper presents an ensemble-based model that uses character-level TF-IDF features to classify Kannada-English, Malayalam-English, and Tamil-English social media posts into five diferent sentiment classes. The proposed model is validated on the dataset provided in the DravidianCodeMix FIRE 2021 [15, 16] track. The dataset includes five distinct sentiment classes, including ”positive,” ”negative,” ”mixed feelings,” ”unknown state,” and ”if the post is not in the mentioned Dravidian languages.”
The rest of the sections are organized as follows: the proposed methodology is explained in Section 2. The experimental findings are listed in Section 3 and Section 4 concludes the paper by highlighting the main findings of this study.
The systematic diagram of the proposed ensemble-based model for the Kannada-English language can be seen in Figure 1 whereas, the proposed model for Malayalam-English and TamilEnglish can be seen in Figure 2. The proposed model is validated with the datasets given in the DravidianCodeMix FIRE 2021 competition [16]. The overall data statistic for Kannada-English [17], Malayalam-English [18], and Tamil-English [19] can be seen in Table 1.
Extensive experiments were carried out with a variety of popular machine learning classifiers using various combinations of one-to-six gram word-level and character-level TF-IDF features. a i d e M l a i c o ishS tss lgn oP E a d a n n a
K h s i l g n -E s il t m so aT aP / sh ied i lg M -nE ila m co laa S y a l a M r e t rac rse raahCm taeFF
u )-G -ID -61 FT ( r e t rac rse raahCm taeFF
u )-G -ID -61 FT (
We found that the ensemble of Support Vector Machine (SVM), Logistic Regression (LR), and Random Forest (RF) performed best on the Kannada-English dataset, while the ensemble of SVM and LR performed best on the Malayalam-English and Tamil-English datasets. The proposed models are described in detail in the following sections.
• Kannada-English: An ensemble-based model is proposed containing SVM, LR, and RF in parallel (see Figure 1). This ensemble-based model uses one to six-gram character TF-IDF features to predict the probability for each of the classes. To choose the suitable character n-gram range, extensive experimentation was performed with one-gram to six-gram character-level TF-IDF features. We found first 50,000 one to six-gram character-level TF-IDF features were performed better than the other combination of character-level n-gram TF-IDF features. The probabilities of all the three classifiers are then averaged class-wise to get the final class probability. The final class for the post is assigned that has the highest average class probability.
Negative
Positive not-Kannada unknown state
Confusion matrix not-Kannada
unknownstate Predicted label 300 250 200 150 100 50 0
Precision, recall, and the 1 -score are utilised to assess the suggested ensemble-based model’s performance. The confusion matrix and AUC-ROC curve are also presented to highlight the 1.0 micro-average ROC curve (area = 0.90) macro-average ROC curve (area = 0.86) Mixed feelings (AUC = 0.78) Negative (AUC = 0.91) Positive (AUC = 0.84) not-Kannada (AUC = 0.93) unknown state (AUC = 0.84) 0.4 0.6 False Positive Rate 0.8
1.0
Confusion matrix Mixed_feelingNsegative
Positivenot-malayalam Predicted label unknown_state model’s performance in addition to these measures. Table 2 shows the outcomes of the suggested model for the Kannada-English, Malayalam-English, and Tamil-English languages.
The suggested ensemble-based model has a weighted precision of 0.64, recall of 0.65, and 1 -score of 0.62 for the Kannada-English dataset. Figures 3 and 4 show the ROC curve and confusion matrix for the Kannada-English dataset, respectively. The suggested ensemble-based model had a weighted precision, recall, and 1 -score of 0.73 for the Malayalam-English dataset. Figures 5 and 6 show the confusion matrix and ROC curve for the Malayalam-English dataset, respectively. The suggested ensemble-based model achieved a weighted precision of 0.61, recall Mixed_feelings
Negative 0.4 0.6 False Positive Rate Mixed_feelingNsegative
unknown_state Predicted label 2000 1500 1000 500 of 0.65, and 1 -score of 0.62, respectively, on the Tamil-English dataset. Figures 7 and 8 show the confusion matrix and ROC curve for the Tamil-English dataset, respectively.
Sentiment analysis of social media messages is an essential task in natural language processing, which analyses social discussions and feedback to discover the deeper context as they 1.0 0.0 0.0 0.2 micro-average ROC curve (area = 0.90) macro-average ROC curve (area = 0.84) Mixed_feelings (AUC = 0.76) Negative (AUC = 0.84) Positive (AUC = 0.83) not-Tamil (AUC = 0.94) unknown_state (AUC = 0.82) 0.4 0.6 False Positive Rate 0.8 1.0 pertain to a topic, brand, or theme. This work proposes an ensemble-based model to classify Kannada-English, Malayalam-English, and Tamil-English social media postings into five different sentiment classes. The use of one to six-gram character-level feature performed best with the other combinations of n-gram character-level features. For the Kannada-English, Malayalam-English, and Tamil-English datasets, the suggested ensemble-based model achieved weighted 1 -scores of 0.62, 0.73, and 0.62, respectively. To improve performance, a robust deep ensemble-based model can be developed in the future by integrating character-level and word-level features. J. P. McCrae, Dravidiancodemix: Sentiment analysis and ofensive language identification dataset for Dravidian languages in code-mixed text, arXiv preprint arXiv:2106.09460 (2021). [7] A. Hande, S. U. Hegde, R. Priyadharshini, R. Ponnusamy, P. K. Kumaresan, S. Thavareesan, B. R. Chakravarthi, Benchmarking multi-task learning for sentiment analysis and offensive language identification in under-resourced Dravidian languages, arXiv preprint arXiv:2108.03867 (2021). [8] B. R. Chakravarthi, R. Priyadharshini, V. Muralidaran, S. Suryawanshi, N. Jose, J. P. Sherly, Elizabeth McCrae, Overview of the track on Sentiment Analysis for Davidian Languages in Code-Mixed Text, in: Working Notes of the FIRE 2020. CEUR Workshop Proceedings., 2020. [9] A. Kumar, S. Saumya, J. P. Singh, NITP-AI-NLP@ Dravidian-CodeMix-FIRE2020: A hybrid CNN and Bi-LSTM network for sentiment analysis of Dravidian code-mixed social media posts., in: FIRE (Working Notes), 2020, pp. 582–590. [10] S. Mahata, D. Das, S. Bandyopadhyay, Sentiment classification of code-mixed tweets using bi-directional rnn and language tags, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, 2021, pp. 28–35. [11] Y. Sharma, A. V. Mandalam, Bits2020@ Dravidian-CodeMix-FIRE2020: Sub-word level sentiment analysis of Dravidian code mixed data., in: FIRE (Working Notes), 2020, pp. 503–509. [12] B. G. Patra, D. Das, A. Das, Sentiment analysis of code-mixed indian languages: An overview of SAIL_code-mixed shared task@ ICON-2017, arXiv preprint arXiv:1803.06745 (2018). [13] L. Advani, C. Lu, S. Maharjan, C1 at SemEval-2020 Task 9: SentiMix: Sentiment analysis for code-mixed social media text using feature engineering, arXiv preprint arXiv:2008.13549 (2020). [14] K. Goswami, P. Rani, B. R. Chakravarthi, T. Fransen, J. P. McCrae, ULD@ NUIG at SemEval2020 Task 9: Generative morphemes with an attention model for sentiment analysis in code-mixed text, arXiv preprint arXiv:2008.01545 (2020). [15] B. R. Chakravarthi, R. Priyadharshini, S. Thavareesan, D. Chinnappa, D. Thenmozhi, E. Sherly, J. P. McCrae, A. Hande, R. Ponnusamy, S. Banerjee, C. Vasantharajan, Findings of the Sentiment Analysis of Dravidian Languages in Code-Mixed Text, in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021. [16] R. Priyadharshini, B. R. Chakravarthi, S. Thavareesan, D. Chinnappa, D. Thenmozi, E. Sherly, Overview of the DravidianCodeMix 2021 shared task on sentiment detection in tamil, malayalam, and kannada, in: Forum for Information Retrieval Evaluation, FIRE 2021, Association for Computing Machinery, 2021. [17] A. Hande, R. Priyadharshini, B. R. Chakravarthi, KanCMD: Kannada CodeMixed dataset for sentiment analysis and ofensive language detection, in: Proceedings of the Third Workshop on Computational Modeling of People’s Opinions, Personality, and Emotion’s in Social Media, Association for Computational Linguistics, Barcelona, Spain (Online), 2020, pp. 54–63. URL: https://www.aclweb.org/anthology/2020.peoples-1.6. [18] B. R. Chakravarthi, N. Jose, S. Suryawanshi, E. Sherly, J. P. McCrae, A sentiment analysis dataset for code-mixed Malayalam-English, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 177–184. URL: https://www.aclweb.org/anthology/ 2020.sltu-1.25. [19] B. R. Chakravarthi, V. Muralidaran, R. Priyadharshini, J. P. McCrae, Corpus creation for sentiment analysis in code-mixed Tamil-English text, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 202–210. URL: https://www. aclweb.org/anthology/2020.sltu-1.28.