Identification of sentiment analysis from social media contents has more attention in the past decades. Such social media contents are code-mixed in nature and peoples find easier to express in this format. They can bind their mother tongue with English. This task deals with identifying sentiment analysis from code-mixed Dravidian languages. Dataset provided by the organisers are in Tamil-English, KannadaEnglish and Malayalam-English languages. Our system uses the three diferent approaches viz: machine learning(MNB and DTC), neural-network(ANN and CNN) and transfer learning(BERT, mBERT). For Malayalam-English, MNB, trained using TF-IDF found best. For Tamil-English, ANN and for KannadaEnglish CNN performed better.
most of the peoples in India rely on code-mixed or code-switched languages, which leads to diferent sentiments such as hate speech [ 2], insult [3], ofensive comments [ 4] and trolling [5]. If these kind of negative sentiments are not taken care in time, can harm communal health and can turn into devastating events [6]. Specially many code-mixed Indian languages and under-resourced languages [7], needs serious attention. With this motivation, the researchers have initiated their work on identifying diferent sentiments occurring on social media. In this context, the shared task on sentiment analysis for Dravidian Languages in Code-Mixed Text has been organized [1] and authors have participated. The details about this task and its purpose is given in Section 3.
From the above discussion we can understand diferent types of sentiments occurring in codemixed text content on social media. Though significant amount of sentiment related text in code-mixed Dravidian languages could be found on social media, very less sentiment analysis work has been done till date. Bharathi Raja et al.[8], attracted attention of researchers by organizing special track on Sentiment Analysis for Dravidian Languages in Code-Mixed Text in year 2020. Similarly Bharathi Raja et al. [9], has organized a Shared Task on identifying Ofensive Language from Dravidian languages viz.: Tamil, Malayalam, and Kannada in year 2021. In both these task, the organizers have provided training, validation and test datasets with appropriate annotations. In these tasks, many researchers have participated and have applied diferent machine learning and transfer learning techniques.
Our study shows that prior to the initiatives taken for Dravidian languages, Patra et al., [10] took eforts for code-mixed Hindi-English and code-mixed Bengali-English languages for sentiment analysis. They have organized shared task in ICON 2017 and prepared the dataset using Twitter API. In the year 2018, Aditya Bohra et al. [11], contributed to this research area and they have developed the dataset in code mixed Hindi-English using Twitter Python API. Similarly Anita Saroj and Sukomal Pal [12], have created the dataset in English and code-mixed Hindi languages by using Facebook and Tweeter media in year 2020. This data was collected from parliamentary election of India (PEI-2019) event and they implemented diferent machine classifiers.
The goal of this shared task is to categorize the posts/comments of the Dravidian code-mixed dataset collected from YouTube comments into diferent sentiments polarity. The dataset provided by organizers are Malayalam-English, Tamil-English and Kannada-English languages, containing code-mixed sentences viz. Inter-Sentential switch, Intra-Sentential switch and Tag switching [13]. The Dataset is consisting of 09 tsv files. For each language, the dataset has 03 ifles for Training, Validation and Testing respectively. Both Training and Validation dataset have three columns namely id, comment/post and sentiment polarity respectively. The polarity for respective comment/post, is annotated with five classes viz. Mixed feeling, Negative, Positive, not-language and unknown state. The test dataset has single text column that contains YouTube comments/posts. The participants have been asked to develop a system that can identify the appropriate class of respective comment and annotate the test data accordingly [14]. The dataset details, contain total number of comments of five classes. These details of training and development datasets given by the organisers is shown in the Table 1. From this table, we can see that the classes are highly imbalanced.
As the datasets are collected from social media, they are noisy in nature. So pre-processing is required, which is done at the initial stage of this experimental work. Later features are extracted from datasets(training and validation), using TF-IDF and Keras Tokenizer API 1. On these extracted features, two diferent approaches, Machine learning and Neural network, are applied. Results of these two approaches, are submitted to HASOC 2021 shared task. To improve the system further, we have applied a Transfer Learning, of which results were not be submitted to task organiser. Our system is thus based on three approaches as mentioned below. The code is available on GitHub 2.
• Machine learning. • Neural network.
• Transfer learning.
The performance of all these models is evaluated for development dataset, using weighted average F1-score. For all the three Dravidian language, same methodology is followed. The working of each approach is presented in this article in detail.
We have removed the white spaces, digits, special characters, extra spaces and emojis etc. English stop-words are also removed. For Malayalam language, we have used ml2en algorithm3. The ml2en algorithm, transliterates Malayalam script to Roman script (’Manglish’). 1https://keras.io/ 2https://github.com/sayprasad1/KBCNMUJAL-HASOC-2021 3https://nadh.in/code/ml2en/
We have applied diferent n-gram ranges of TF-IDF word, TF-IDF character and TF-IDF combined word and character for extracting the features. The selection of diferent ranges is done, by testing at what range the classifier is producing the highest f1-score. For every language, we have tested the diferent n-gram range, and finally, have used the following n-gram range .
• For Malayalam word n-gram of (
As the classes have severe skew in distribution, both the classifiers failed to categoriz the comments among the five classes for all the Dravidian languages. Hence SMOTE 4 technique, is applied on TF-IDF extracted features. SMOTE is the most popular oversampling method. While applying SMOTE on TF-IDF extracted features sampling strategy is kept to auto and thus we get the re-sampled TF-IDF features according to the majority class. On re-sampled features MNB and DTC were applied. While applying DTC for Tamil language we have selected 10,000 features because training and development dataset have bigger size as compared to Malayalam and Kannada. The class-wise performance of both the classifiers on this re-sampled features, is given in the Table 2, 3, 4.
We have implemented an Artificial Neural Network (ANN) and Convolution Neural Network (CNN) in our work. For ANN we created tokenization of datasets using one-hot encoded matrix. For one-hot encoded matrix, we have used keras text_to_matrix method. For CNN, we have used keras text_to_sequences method for the tokenization. The extracted tokens are used to construct the vocabulary base of the respective language.
4imblearn.over_sampling.SMOTE
After tokenizing the text, ANN is built by using three dense layers. The first input (dense) layer has 512 nodes and input vocabulary of size 10,000 is provided. This is followed by activation layer using relu activation function, which is followed by dropout layer of 0.3. The second dense layer also has the 512 node followed by the activation layer having relu function. This is followed by a dropout layer with dropout of 0.3. The last output (dense) layer has 5 nodes because we have to be categorized the comments/post into five classes. Dense layer is followed by activation layer using sigmoid as activation function. The proposed ANN trained with categorical_crossentropy loss function and Adam optimizer. The training has the batch size of 32 with 10 epochs. The same ANN network is implemented for all the three Dravidian languages.
For implementing CNN, we have applied diferent parameters for all the Dravidian languages. We have extracted total unique words(vocab_size) and length of longest sentence(max_length) from the dataset for all languages. We found 40230, 69675 and 15800 unique words from Malayalam, Tamil and Kannada languages respectively. For Malayalam, Tamil and Kannada maximum length of comment/post is 195, 124 and 92 respectively. Every input comment padded with a maximum length(max_length) of that language. For every language, embedding of 100 is used with unique words(vocab_size). So for Malayalam, Tamil and Kannada, we get (40230 X 100), (69675 X 100) and (15800 X 100) dimensional embedding matrix respectively. Thus embedding layer is diferent for each language, but the next set of layers(Conv1D, GlobalMaxPooling1D and dense) are same. Embedding layer treated as input to Conv1D layer. Conv1D provided with number of output filters as 64, and kernel size of 3 with relu as activation function. Conv1D layer is used for generating sequences of text. Conv1D layer is followed by GlobalMaxPooling1D layer and dropout of 0.5. The dense layer has 5 nodes with activation function as softmax. CNN trained with categorical_crossentropy loss function and Adam optimizer. The training has the batch size of 128 with 5 epochs.
Though the classes were highly imbalanced in nature, neural-network approach(ANN and CNN) successfully categorized the post into the five classes for all the Dravidian languages. The class wise distribution of the dataset using ANN and CNN is give in Table 5.
Further to analyse the performance of ANN and CNN on resampled dataset, We have applied SMOTE technique on the tokenized text(features). While applying SMOTE, sampling strategy parameter is set to minority, which resamples the minority class. Thus we have received the resampled features according to the minority class. On resampled features, the same ANN and CNN are implemented. This has enabled to predict the comments among the desired classes. The results are presented in Table 6.
After comparing the result of neural network approaches on tokenized features and on resampled features from Table 5 and Table 6, we can observe that for Malayalam and Kannada language results are very low for resampled features, but for Tamil language ANN performed better on resampled features.
For applying transfer learning, we used Simple Transformers [15] library based on the Transformers library by HuggingFace. Simple Transformers permits to fine-tune Transformer models. Transformers ofers plenty of pretrained models which are used to accomplish diferent tasks like text classification, multi-label text classification, text generation, question answering, summarization, translation, information extraction and more in over 100 languages. Our approach used two diferent variations of BERT [ 16] (Bidirectional Encoder Representations from Transformers) transfer models, for categorizing the comments. The two variations are : • Pretrained BERT base model (bert-base-cased) : 12-layer, 768-hidden, 12-heads, 110M parameters. It is trained on lower-cased English text using a masked modelling technique. • Pretrained BERT multilingual model (bert-base-multilingual-cased) : It has 12 layers, 768 hidden, 12 attention heads with 168M parameters. It is trained on lower-cased text in 104 languages with masked language modeling.
For implementing above models we created instance of ClassificationModel with its parameters and their values. For each language we created two diferent models : model handling class imbalance while other model does not handle class imbalance. The model handling class imbalance has weight attribute, which takes input as list of weight for each label. To calculate weights of the classes we implemented compute_class_weight5 from sklearn. For both the models hyperparameter ’fp16’ is set to ’false’ while rest of the parameters are set to their default values. Both these models are trained by using training dataset with 4 epochs and results were evaluated using development dataset. Table 7 shows the results of BERT models not handling class imbalance while BERT models handling class imbalance are shown in Table 8.
After examining Table 7 and Table 8, we have noticed that, class weights have dropped the performance for Tamil language, whereas for Kannada language, the results show that there is slight improvement for both BERT models. For Malayalam language, both BERT models with class weights and without class weights have produced same result, while BERTmultilingual without class weights have shown better performance than BERT-multilingual with class weights.
The system performance is evaluated in terms of precision, recall, and F1-score for all the ifve sentiment classes. The weighted average of these classes is also given. Among the three 5sklearn.utils.class_weight.compute_class_weight approaches, the one, which has highest weighted average of precision, recall and F1-score, is considered as best. The results of best performing model were submitted to the task organiser. As mentioned earlier, results of transfer learning and neural network approach using resampled features were not submitted.
For Malayalam language, among all the classifiers, MNB using combined word n-gram
range(
In case of Tamil language, ANN using resampled features performed better with F1 score of a
0.65 as compared to tokenized features with F1 score 0.58. CNN using tokenized features have
F1 score 0.57 and using resampled features it shows a degradation of 0.54. MNB using character
n-gram range(
CNN and ANN have shown better results using tokenized features as compared to resampled
features. CNN using tokenized features has weighted average precision and recall values as
0.63 and 0.64 respectively in case of Kannada language. Similarly, for Tamil language, MNB and
DTC with word n-gram range(
From all the above observations, we can conclude that, in case of Malayalam language, machine learning model: MNB using combined word n-gram and character n-gram has scored the highest results. For Tamil language, the neural network model: ANN using resampled features has scored the better results. In case of Kannada language BERT-multilingual using class weights performed better.
From these results, we can report that Roman/Latin script helps in improving the system performance for all the three approaches. Machine learning approach with word n-gram feature categorized the sentiments, only for Roman/Lain script(e.g, Manglish). On the other hand, for code-mixed sentences(e.g.Tanglish and Tamil), it failed miserably. On the contrary, the same machine learning approach with character n-gram feature and combined word and character n-gram feature successfully categorized the sentiments irrespective whether it is Roman/Lain script or code-mixed sentences. Experiment of the system shows that class imbalanced is successfully handled by neural-network and transfer learning, whereas for machine learning approach resampling of features is must. J. P. McCrae, Overview of the track on sentiment analysis for dravidian languages in code-mixed text, in: Forum for Information Retrieval Evaluation, 2020, pp. 21–24. [9] B. R. Chakravarthi, R. Priyadharshini, N. Jose, A. Kumar M, T. Mandl, P. K. Kumaresan, R. Ponnusamy, H. R L, J. P. McCrae, E. Sherly, Findings of the shared task on ofensive language identification in Tamil, Malayalam, and Kannada, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, 2021, pp. 133–145. URL: https://aclanthology.org/2021. dravidianlangtech-1.17. [10] B. G. Patra, D. Das, A. Das, Sentiment analysis of code-mixed indian languages: An overview of sail_code-mixed shared task @icon-2017, ArXiv abs/1803.06745 (2018). [11] A. Bohra, D. Vijay, V. Singh, S. S. Akhtar, M. Shrivastava, A dataset of Hindi-English code-mixed social media text for hate speech detection, in: Proceedings of the Second Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media, Association for Computational Linguistics, New Orleans, Louisiana, USA, 2018, pp. 36–41. URL: https://aclanthology.org/W18-1105. doi:1 0 . 1 8 6 5 3 / v 1 / W 1 8 - 1 1 0 5 . [12] A. Saroj, S. Pal, An Indian language social media collection for hate and ofensive speech, in: Proceedings of the Workshop on Resources and Techniques for User and Author Profiling in Abusive Language, European Language Resources Association (ELRA), Marseille, France, 2020, pp. 2–8. URL: https://aclanthology.org/2020.restup-1.2. [13] B. R. Chakravarthi, P. K. Kumaresan, R. Sakuntharaj, A. K. Madasamy, S. Thavareesan, P. B, S. Chinnaudayar Navaneethakrishnan, J. P. McCrae, T. Mandl, Overview of the HASOC-DravidianCodeMix Shared Task on Ofensive Language Detection in Tamil and Malayalam, in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021. [14] B. R. Chakravarthi, R. Priyadharshini, S. Thavareesan, D. Chinnappa, D. Thenmozhi, E. Sherly, J. P. McCrae, A. Hande, R. Ponnusamy, S. Banerjee, C. Vasantharajan, Findings of the Sentiment Analysis of Dravidian Languages in Code-Mixed Text, in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021. [15] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, A. M. Rush, Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online, 2020, pp. 38–45. URL: https://www.aclweb.org/anthology/2020.emnlp-demos.6. [16] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, CoRR abs/1810.04805 (2018). URL: http://arxiv. org/abs/1810.04805. a r X i v : 1 8 1 0 . 0 4 8 0 5 .